SlowGuess/NAACL_2021_OCR huggingface dataset:
Chu-Cheng Lin\* Aaron Jaech b Xin Li Matthew R. Gormley Jason Eisner
$^{\#}$ Department of Computer Science, Johns Hopkins University  $^{\flat}$ Facebook AI
Machine Learning Department, Carnegie Mellon University {kitsing,lixints,jason}@cs.jhu.edu ajaech@fb.com mgormley@cs.cmu.edu
Standard autoregressive language models perform only polynomial-time computation to compute the probability of the next symbol. While this is attractive, it means they cannot model distributions whose next-symbol probability is hard to compute. Indeed, they cannot even model them well enough to solve associated easy decision problems for which an engineer might want to consult a language model. These limitations apply no matter how much computation and data are used to train the model, unless the model is given access to oracle parameters that grow superpolynomially in sequence length.
Thus, simply training larger autoregressive language models is not a panacea for NLP. Alternatives include energy-based models (which give up efficient sampling) and latent-variable autoregressive models (which give up efficient scoring of a given string). Both are powerful enough to escape the above limitations.
Sequence modeling is a core NLP problem. Many sequence models  $\tilde{p}$  are efficient at scoring strings: given a string  $\mathbf{x}$ , its score  $\tilde{p}(\mathbf{x})$  can be computed in  $O(\mathrm{poly}(|\mathbf{x}|))$ . For example, an RNN (Mikolov et al., 2011) scores  $\mathbf{x}$  in time  $O(|\mathbf{x}|)$  while a Transformer (Vaswani et al., 2017) does so in time  $O(|\mathbf{x}|^2)$ . The score may be an unnormalized probability, and can be used to rank candidate strings.
Many sequence models also make it easy to compute marginal properties of  $\tilde{p}$ . They support efficient sampling of strings  $\mathbf{x}$  (which allows unbiased approximation of marginal expectations). And they support efficient computation of the normalizing constant  $Z = \sum_{\mathbf{x}}\tilde{p} (\mathbf{x})$  (or simply guarantee  $Z = 1$ ) for any value of the model parameters.
How about training? Briefly: If a sequence model can efficiently compute  $\tilde{p} (\mathbf{x})$  (and its derivatives
MY HOBBY: EMBEDDING NP-COMPLETE PROBLEMS IN RESTAURANT ORDERS
Figure 1: Valid answers to hard natural language inference problems can be hard to find (Munroe, 2009), but in many cases can be checked efficiently (e.g. the KNAPSACK problem in the comic). Given a large enough parametric autoregressive model with correct parameters, we can efficiently solve all problem instances with input length  $n$ , and efficiently verify the solutions — but the required model size can grow superpolynomially in  $n$ . (This allows the model to store precomputed results that we can look up in  $O(n)$  at test time.) A main observation of this paper is that assuming NP  $\nsubseteq \mathrm{P} / \mathrm{poly}$ , then without such a superpolynomial growth in model size, autoregressive models cannot even be used to verify answers to some problems where polynomial-time verification algorithms do exist.
with respect to model parameters), then it is efficient to compute parameter updates for noise-contrastive estimation (Gutmann and Hyvarinen, 2010; Gutmann and Hyvarinen, 2012) or score-matching (Hyvarinen, 2005). If sampling  $\mathbf{x}$  or computing  $Z$  (and its derivatives) is also efficient, then it is efficient to compute parameter updates for ordinary MLE training.
Finally, popular sequence models are compact. Usually a fixed-size model is used to score strings  $\mathbf{x}$  of all lengths. More generally, it might be reasonable to use an  $O(\mathrm{poly}(n))$ -sized parameter vector  $\theta_{n}$  when  $\mathbf{x}$  has length  $n$ , at least if parameter vectors can be obtained (perhaps from an oracle) for all needed lengths. In this paper, we investigate what can and cannot be achieved with models that are compact in this sense. This setup allows us to discuss the asymptotic behavior of model families.
Standard autoregressive models have the form  $p(\mathbf{x}) = \prod_{t}p(x_{t}\mid \mathbf{x}_{< t})^{1}$  where each factor is efficient
*Part of this work was done at Facebook AI.
In this paper we use the shorthand  $\mathbf{x}_{< t} \triangleq x_1 \dots x_{t-1}$ .
Model family	Compact parameters?	Efficient scoring?	Efficient sampling and normalization?	Support can be ...
ELN/ELNCP: Autoregressive models (§3.1)	✓	✓	✓	some but not all L ∈ P
EC/ECCP: Energy-based models (§4.1)	✓	✓	X	all L ∈ P but no L ∈ NPC
Lightly marginalized ELNCP: Latent-variable autoregressive models (§4.2)	✓	X	✓	all L ∈ NP
Lookup models (§4.3)	X	✓	✓	anything
Table 1: A feature matrix of parametric model families discussed in this paper. Also see Figure 2 in the appendices.
to compute from a fixed parameter vector. These models satisfy all three of the desiderata above. By using flexible neural network architectures, standard autoregressive models have achieved stellar empirical results in many applications (Oord et al., 2016; Child et al., 2019; Zellers et al., 2019; Brown et al., 2020). However there are still tasks that they have not mastered: e.g., it is reported that they struggle at deep logical structure, even when initialized to huge pretrained models (Wang et al., 2019a).
We point out that, unfortunately, there are certain sequence distributions whose unnormalized string probabilities  $\tilde{p} (\mathbf{x})$  are easy to compute individually, yet whose autoregressive factors  $p(x_{t}\mid \mathbf{x}_{< t})$  are NP-hard to compute or even approximate, or are even uncomputable. Thus, standard autoregressive models are misspecified for these distributions (cannot fit them). It does not help much to focus on strings of bounded length, or to enlarge the model: under the common complexity-theoretic assumption NP  $\nsubseteq \mathrm{P / poly}$  , the parameter size  $|\theta_n|$  must grow superpolynomially in  $n$  to efficiently approximate the probabilities of all strings of length up to  $n$
Indeed, one of our main findings is that there exist unweighted languages  $L \in \mathbf{P}$  for which no standard autoregressive model has  $L$  as its support, i.e., assigns weight  $> 0$  to just the strings  $\mathbf{x} \in L$ . This is downright depressing, considering the costs invested in training huge parametric autoregressive models (Bender et al., 2021). Since  $L \in \mathbf{P}$ , it is trivial to build an efficient scoring function  $\tilde{p}(\mathbf{x})$  with fixed parameters that has  $L$  as its support—just not an autoregressive one. The problem holds for all standard autoregressive models, regardless of how much computation and training data are used to learn the model parameters.
That is, for an NP-hard problem, scoring a string  $\mathbf{x}$  under a standard autoregressive model  $p(\mathbf{x})$  cannot be used to verify a witness. Nor can finding a witness be solved by prompting such a model with a description of a problem instance and sampling a continuation  $\mathbf{x}$  of that string. Such problems are abundant in NLP: for example, surface realization under Optimality Theory (Idsardi, 2006), decoding text from an AMR parse (Cai and Knight, 2013), phrase alignment between two
sentences (DeNero and Klein, 2008), and in general inference for propositional logic (Cook, 1971), which underlies the NP-hardness of general natural language inference, as in Figure 1. In other words, our results imply that standard autoregressive models do not have the right structure to capture important linguistic regularities: e.g., that observed sequences were in fact constructed to be phonologically optimal, expressive of a semantic form, or logically coherent!
Our work is also relevant to autoregressive models of fixed-dimensional vectors, such as NADE (Uria et al., 2016). These can be extended to arbitrary  $n$ -dimensional vectors by providing separate parameters  $\theta_{n}$  for each  $n$ . Our constructions imply that for some distributions,  $|\theta_{n}|$  must grow superpolynomially in  $n$ , even though this would be not be necessary if the models were not autoregressive.
In the remainder of this paper, we formalize our three desiderata for sequence models. We formalize compact autoregressive models and describe some limitations on their expressiveness. We then show that it can help to choose an alternative model family that relaxes any one of the three desiderata (Table 1).
2.1 Weighted languages
An unweighted language  $L \subseteq V^*$  is a set of strings  $\mathbf{x}$  over a finite alphabet  $V$ . A weighted language  $\tilde{p}$  is a function  $\tilde{p}: V^* \to \mathbb{R}_{\geq 0}$ . It may be regarded as specifying an unweighted language  $L = \text{support}(\tilde{p}) \triangleq \{\mathbf{x}: \tilde{p}(\mathbf{x}) \neq 0\}$  along with positive weights for the strings in  $L$ . We say that a weighted language  $\tilde{p}$  is normalizable if its global normalizing constant  $Z \triangleq \sum_{\mathbf{x} \in V^*} \tilde{p}(\mathbf{x})$  is finite and strictly positive. When  $\tilde{p}$  is normalizable,  $p(\mathbf{x}) \triangleq \tilde{p}(\mathbf{x}) / Z$  is a probability distribution over  $L$ . A distribution is any weighted language whose global normalizing constant is 1.
Let  $\hat{\mathbf{x}}\preceq \mathbf{x}$  mean that  $\hat{\mathbf{x}}$  is a prefix of  $\mathbf{x}\in V^{*}$  (not necessarily a strict prefix). If  $\tilde{p}$  is normalizable, then  $Z(\hat{\mathbf{x}})\triangleq \sum_{\mathbf{x}\in V^{*}:\hat{\mathbf{x}}\leq \mathbf{x}}\tilde{p} (\mathbf{x})$  is  $\leq Z$  for any  $\hat{\mathbf{x}}\in V^{*}$ , yielding a marginal prefix probability  $Z(\hat{\mathbf{x}}) / Z$ . If the prefix  $\hat{\mathbf{x}}$  has positive prefix probability, then it admits a local conditional probability  $p(x\mid \hat{\mathbf{x}})\triangleq Z(\hat{\mathbf{x}} x) / Z(\hat{\mathbf{x}})$  for each symbol  $x\in V$ , where the
denominator is interpreted as a local normalizing constant. This is the conditional probability that if a random string starts with the prefix  $\hat{\mathbf{x}}$ , the next symbol is  $x$ . There is also a probability  $p(\mathbb{S}|\hat{\mathbf{x}})\triangleq 1 - \sum_{x\in V}p(x|\hat{\mathbf{x}}) = \tilde{p} (\hat{\mathbf{x}}) / Z(\hat{\mathbf{x}})\geq 0$  that the string ends immediately after  $\hat{\mathbf{x}}$ ; the special symbol  $\mathbb{S}\notin V$  represents "end of string."
2.2 Computation for weighted languages
We define a weighted language  $\tilde{p}$  to be computable if it is defined by a Turing machine (also called  $\tilde{p}$ ) that maps any  $\mathbf{x} \in V^{*}$  to  $\tilde{p}(\mathbf{x}) \in \mathbb{Q}_{\geq 0}$  in finite time. The Turing machine does not have to compute  $Z$ .
While the computable weighted languages allow any computable function as  $\tilde{p}$ , most architectures for defining weighted languages (e.g., RNNs or Transformers) do only a bounded or linear amount of work per input symbol. As a result, they compute  $\tilde{p}(\mathbf{x})$  in time  $O(\mathrm{poly}(|\mathbf{x}|))$  (that is,  $\tilde{p} \in \mathrm{FP}$ ). We refer to such weighted languages as efficiently computable (EC). This does not imply that the normalized version  $p$  is efficiently computable, since finding the denominator  $Z$  requires summing over all of  $V^{*}$ .
If we tried to construct the same normalized distribution  $p$  as in the previous paragraph using a standard autoregressive model, we would model it as a product of local conditional probabilities,  $p(\mathbf{x}) = (\prod_{t=1}^{\left|\mathbf{x}\right|} p(x_t \mid \mathbf{x}_{<t})) p(\mathbb{S} \mid \mathbf{x})$ . Most such architectures again do only a bounded or linear amount of work per input symbol. Yet one suspects that this may not always be enough work to do the job: the local conditional probabilities of the original  $\tilde{p}$  are expensive to compute (unless  $\tilde{p}$  has some special structure making  $Z(\hat{\mathbf{x}})$  tractable).
Indeed, the observation of this paper is that for some efficiently computable weighted languages  $\tilde{p}$ , the local conditional probabilities are expensive to compute or even to approximate well. More precisely, autoregressive models cannot fit the local conditional probabilities unless they are superpolynomial in either their runtime or in their number of parameters (where the parameters may be precomputed at training time). We now explain how to formalize these notions.
2.3 Non-uniform computation
In the machine learning approach to sequence modeling, we usually do not manually design the Turing machine behind  $\tilde{p}$ . Rather, we design a model  $M$  with parameters  $\theta$ .  $M$  is a Turing machine that reads  $\theta$  and outputs a specialized Turing machine  $\tilde{p}_{\theta} \triangleq M(\theta)$  that can score strings  $\mathbf{x}$  and hence defines a weighted language. Without loss of generality, we will express
$\pmb{\theta}$  as a string in  $\mathbb{B}^*$  (where  $\mathbb{B} \triangleq \{0,1\}$ ). For each  $\pmb{\theta}$ , we obtain a potentially different weighted language.
Strings vary in length, and accurate modeling of longer strings may sometimes require more complex computations with more parameters. For example, when  $V$  is a natural language alphabet, a recurrent neural network may require more hidden units to model sentences of the language rather than individual words, and even more units to model whole documents. To accommodate this, we allow an infinite sequence of parameter vectors,  $\Theta = \{\theta_{n}\in \mathbb{B}^{*}\mid n\in \mathbb{N}\}$ , which yields an infinite sequence of Turing machines  $\{\tilde{p}_n\mid n\in \mathbb{N}\}$  via  $\tilde{p}_n\triangleq M(\theta_n)$ . We then define  $\tilde{p}_{\Theta}(\mathbf{x})\triangleq \tilde{p}_{|\mathbf{x}|}(\mathbf{x})$ , so a string of length  $n$  is scored by the  $\tilde{p}_n$  machine. This is known as non-uniform computation. Of course, it is legal (and common) for all of the  $\theta_{n}$  to be equal, or empty, but if desired, we can obtain more power by allowing the number of parameters to grow with  $n$  if needed.
We can now consider how rapidly the parametric and runtime complexity may grow.
- If  $|\theta_{n}|$  is permitted to grow exponentially, then one can fit any weighted language  $\tilde{p}$  (even an uncomputable one).<sup>2</sup> Simply use  $\theta_{n}$  to encode a trie with  $O(|V|^{n+1})$  nodes that maps  $\mathbf{x} \mapsto \tilde{p}(\mathbf{x})$  for any  $|\mathbf{x}|$  of length  $n$ , and design  $M$  such that the Turing machine  $\tilde{p}_{n} = M(\theta_{n})$  has a (large) state transition table that mirrors the structure of this trie. The resulting collection of Turing machines  $\{\tilde{p}_{n} \mid n \in \mathbb{N}\}$  can then compute  $\tilde{p}(\mathbf{x})$  exactly for any  $\mathbf{x}$ , with only linear runtime  $O(|\mathbf{x}|)$  (which is used to traverse the trie).
- Separately, if unbounded runtime is permitted for  $M$ , then one can exactly fit any computable weighted language  $\tilde{p}$ . Simply have  $M$ , when run on  $\theta_{n}$ , compute and return the large trie-structured  $\tilde{p}_{n}$  that was mentioned above. In this case,  $M$  need not even use the parameters  $\theta_{n}$ , except to determine  $n$ .
- Finally, if unbounded runtime is permitted for  $\tilde{p}_n$ , then again one can exactly fit any computable weighted language  $\tilde{p}$ . In this case,  $M$  trivially returns  $\tilde{p}_n = \tilde{p}$  for all  $n$ .
However, if the parameters  $\Theta$  are "compact" in the sense that  $|\theta_{n}|$  grows only as  $O(\mathrm{poly}(n))$ , and also  $\tilde{p}_n = M(\theta_n)$  is constructed by  $M$  in time  $O(\mathrm{poly}(n))$ , and  $\tilde{p}_n$  scores any  $\mathbf{x}$  of length  $n$  in time  $O(\mathrm{poly}(n))$ , then we say that the resulting weighted language  $\tilde{p}$  is efficiently computable with compact parameters (ECCP).<sup>3</sup> We refer
2See our remark on computability in Appendix A.
<sup>3</sup>Since we require  $M$  to run in polytime, it can only look at a polynomial-sized portion of  $\theta_{n}$ . Hence it is not really crucial for
to  $M$  paired with a parameter space of possible compact values for  $\Theta$  as an ECCP model.
Neural models of weighted languages are typically ECCP models. The construction and execution of the neural network  $\tilde{p}_n$  may perform a polynomial amount of total computation to score the string  $\mathbf{x}$ . This computation may involve parameters that were precomputed using any amount of effort (e.g., training on data) or even obtained from an oracle (they need not be computable). However, the exponentially many strings of length  $n$  must share a polynomial-size parameter vector  $\theta_n$ , which prevents the solution given in the first bullet point above.
In practice one takes  $\theta_{n} = \theta$  for all  $n$  and obtains  $\pmb{\theta} \in \mathbb{R}^{d}$  by training. However, we do not consider whether such parameters are easy to estimate or even computable. We simply ask, for a given target language  $\tilde{p}$ , whether there exists a polynomially growing sequence  $\Theta$  of "good" parameter vectors for any parametric model  $M$ . When not, there can be no scheme for estimating arbitrarily long finite prefixes of such a sequence. So for any polynomial  $f$ , any training scheme that purports to return a trained model of size  $f(n)$  that works "well" for strings of length  $\leq n$  must fail for large enough  $n$  — even if unlimited data, computation, and oracles are allowed at training time.
2.4 P, P/poly, and NP/poly
The phrase "efficiently computable with compact parameters" means that without access to those parameters, the ECCP weighted language may no longer be efficiently computable. Indeed, it need not be computable at all, if the parameter vectors store the outputs of some uncomputable function.
Our definitions above of EC and ECCP weighted languages are weighted generalizations of complexity classes P and P/poly, respectively, and their supports are always unweighted languages in P and P/poly, respectively. An unweighted language  $L$  is in P iff there is a deterministic Turing machine that decides in  $O(\mathrm{poly}(|\mathbf{x}|))$  time whether  $\mathbf{x}\in L$ . And an unweighted language  $L^{\prime}$  is in P/poly iff there exist
the parameters  $\theta_{n}^{\mathrm{p}}$  to be compact, but we nonetheless include this intuitive condition, without loss of generality.
4Namely the nonnegative functions in FP and FP/poly.
5Our presentation of P/poly is a variant of Arora and Barak (2009, §6), in which inputs  $\mathbf{x}$  of length  $n$  are evaluated by a polytime function  $M$  that is given an advice string  $\theta_{n}$  as an auxiliary argument. This corresponds to a neural architecture  $M$  that can consult trained parameters  $\theta_{n}$  at runtime. We have replaced the standard call  $M(\theta_{n},\mathbf{x})$  with the "curried" expression  $M(\theta_{n})(\mathbf{x})$ , which we still require to execute in polynomial total time. Here the intermediate result  $M_{n} = M(\theta_{n})$  corresponds to a
Turing machines  $\{M_n : n \in \mathbb{N}\}$  such that  $M_n$  decides in  $O(\mathrm{poly}(n))$  time whether  $\mathbf{x}$  of length  $n$  is in  $L'$ , where each  $M_n$  can be constructed in  $O(\mathrm{poly}(n))$  time as  $M(\theta_n)$ , for some Turing machine  $M$  and some sequence of polynomially-sized advice strings  $\Theta = \{\theta_n \mid n \in \mathbb{N}\}$  with  $|\theta_n| \in O(\mathrm{poly}(n))$ . We define the language class NP/poly similarly to P/poly: the only difference is the family  $\{M_n : n \in \mathbb{N}\}$  consists of nondeterministic Turing machines.
Naturally,  $\mathrm{P} \subseteq \mathrm{P} / \mathrm{poly}$ . But  $\mathrm{P} / \mathrm{poly}$  is larger than  $\mathrm{P}$ : it contains all sparse languages, regardless of their hardness—even sparse undecidable languages—as well as many dense languages. The extra power of  $\mathrm{P} / \mathrm{poly}$  comes from its access to compact advice strings that do not have to be recursively enumerable, let alone efficient to find. This corresponds to statistical modeling, where the trained model has a computationally efficient architecture plus access to parameters that might have taken a long time to find.
2.5 NP-completeness and SAT
NP-complete decision problems have solutions that are efficient to validate but inefficient to find (assuming  $\mathrm{P} \neq \mathrm{NP}$ ). One of the most well-known NP-complete problems is the boolean satisfiability problem (SAT) (Cook, 1971). Given a boolean formula  $\phi$ , SAT accepts  $\phi$  iff  $\phi$  can be satisfied by some value assignment. For example, the formula  $(A_1 \lor \neg A_2 \lor A_3) \land (A_1 \lor \neg A_4)$  is in SAT, since there is a satisfying assignment  $A_{1\dots 4} = 1101$ . We denote the number of satisfying assignments to  $\phi$  as  $\#(\phi)$ .
It is widely believed that no NP-complete languages are in P/poly. Otherwise we would have all of NP  $\subseteq$  P/poly and the polynomial hierarchy would collapse at the second level (Karp and Lipton, 1980).
A capacity limitation of EC/ECCP weighted languages naturally follows from this belief:[6]
Lemma 1. For any  $L \in \mathbb{P}$ , there exists an EC weighted language with support  $L$ . For any  $L \in \mathbb{P} / \mathrm{poly}$ , there exists an ECCP language with support  $L$ . But for any  $L \in \mathbb{NP}$ -complete, there exists no ECCP language with support  $L$  (assuming NP  $\nsubseteq$  P/poly).
In addition to not capturing the support of NP-complete languages, ECCP weighted languages can
trained runtime model for inputs of length  $n$ . Our Turing machines  $M_{n}$  have size polynomial in  $n$  (because they are constructed by  $M$  in polynomial time). They correspond to the polynomial-sized boolean circuits  $M_{n}$  that are used to evaluate inputs of length  $n$  under the classical definition of P/poly (Ladner, 1975). We exposed these intermediate results  $M_{n}$  only to observe in §2.3 and §4.3 that if we had allowed the  $M_{n}$  to grow exponentially, they would have been able to encode the answers in tries.
6All omitted proofs are in Appendix A.
not help solve other NP-hard problems, either. For example, many structured prediction problems in NLP can be formulated as  $\operatorname{argmax}_{\mathbf{x}:\hat{\mathbf{x}}\leq \mathbf{x}}\tilde{p} (\mathbf{x})$ : we are given a prefix  $\hat{\mathbf{x}}$  as input and look for its optimal continuation under  $\tilde{p}$ . But if this problem is NP-hard for a particular  $\tilde{p}$ , then it is not in P/poly (assuming NP  $\nsubseteq$  P/poly), so it cannot be accomplished by any polytime algorithm that queries an ECCP model.
3 Autoregressive ECCP models (ELNCP models) have reduced capacity
In this section we formally define autoregressive ECCP models, and prove that they have strictly less capacity than general ECCP models or even just EC models. Our proofs rely on the construction of a EC model  $\tilde{p}$  where computing the local conditional probabilities  $p(x\mid \hat{\mathbf{x}})$  is NP-hard, so they cannot be computed with compact parameters, if NP  $\nsubseteq$  P/poly.
3.1 ELN and ELNCP models
Many parameter estimation techniques and inference methods specifically work with local conditional probabilities  $p(x \mid \hat{\mathbf{x}})$ . Thus, it is common to use parametric models where such quantities can be computed in time  $O(\mathrm{poly}(|\hat{\mathbf{x}}|))$  (given the parameters). These are the "standard autoregressive models" we discussed in §1. We say that the resulting distributions are efficiently locally normalizable, or ELN.