SlowGuess/NAACL_2021_OCR huggingface dataset:
We may again generalize ELNs to allow the use of compact parameters. For any weighted language  $\tilde{p}$ , the Turing machine  $M^{\mathfrak{q}}$  efficiently locally normalizes  $\tilde{p}$  with compact parameters  $\Theta^{\mathfrak{q}} = \{\theta_{n}^{\mathfrak{q}} \mid n \in \mathbb{N}\}$  if
- the parameter size  $|\theta_n^{\mathbf{q}}|$  grows only as  $O(\mathrm{poly}(n))$
-  $M^{\mathrm{q}}(\theta_n^{\mathrm{q}})$  returns a Turing machine  $q_{n}$  (similar to  $\tilde{p}_n$  in §2.3) in time  $O(\mathrm{poly}(n))$
-  $\tilde{p}$  is normalizable (so  $p$  exists)
-  $q_{n}$  maps  $\hat{\mathbf{x}} x \mapsto p(x \mid \hat{\mathbf{x}})$  for all  $x \in V \cup \{\mathbb{S}\}$  and all prefixes  $\hat{\mathbf{x}} \in V^{*}$  with  $|\hat{\mathbf{x}}| \leq n$  and  $Z(\hat{\mathbf{x}}) > 0$
-  $q_{n}$  runs on those inputs  $\hat{\mathbf{x}} x$  in time  $O(\mathrm{poly}(n))$ 
If there is  $M^{\mathrm{q}}$  that efficiently locally normalizes a weighted language  $\tilde{p}$  with compact parameters  $\Theta^{\mathrm{q}}$ , we say  $\tilde{p}$  is efficiently locally normalizable with compact parameters, or ELNCP. Note that this is a property of the weighted language itself. In this case, it is obvious that  $\tilde{p}$  is ECCP:
Lemma 2. An ELNCP model  $\tilde{p}$  is also ECCP. Likewise, an ELN model is also EC.
If we define ELNCP models analogously to ECCP models, Lemma 2 means that locally normalized models do not provide any extra power. Their distributions can always be captured by globally normalized models (of an appropriate architecture that we used in the proof). But we will see in Theorem 1 that the converse is likely not true: provided that NP  $\nsubseteq$  P/poly, there are efficiently computable weighted languages that cannot be efficiently locally normalized, even with the help of compact parameters. That is, they are EC (hence ECCP), yet they are not ELNCP (hence not ELN).
3.2 ELNCP models cannot exactly capture all EC (or ECCP) distributions
We reducing SAT to computing certain local conditional probabilities of  $\tilde{p}$  (as defined in §2.1). Each decision  $\mathrm{SAT}(\phi)$  (where  $\phi$  ranges over formulas) corresponds to a particular local conditional probability, implying that there is no polytime scheme for computing all of these probabilities, even with polynomially sized advice strings (i.e., parameters).
Without loss of generality, we consider only formulae  $\phi$  such that the set of variables mentioned at least once in  $\phi$  is  $\{A_1,\ldots ,A_j\}$  for some  $j\in \mathbb{N}$ ; we use  $|\phi |$  to denote the number of variables  $j$  in  $\phi$ . We say that a satisfies  $\phi$  if  $\mathbf{a}\in \mathbb{B}^{|\phi |}$  and  $(A_{1} = a_{1},\dots,A_{|\phi |} = a_{|\phi |})$  is a satisfying assignment. Finally, let boldface  $\pmb {\phi}\in \mathbb{B}^*$  denote enc  $(\phi)$  where enc is a prefix-free encoding function. We can now define the unweighted language  $L = \{\pmb {\phi}\mathbf{a}\mid$ $\phi$  is a formula and  $\mathbf{a}\in \mathbb{B}^{|\phi |}$  and a satisfies  $\phi \}$  over alphabet  $\mathbb{B}$ , which contains each possible SAT problem concatenated to each of its solutions.8
We now convert  $L$  to a weighted language  $\tilde{p}$ , defined by  $\tilde{p}(\mathbf{x}) = \tilde{p}(\boldsymbol{\phi}, \mathbf{a}) = \left(\frac{1}{3}\right)^{|\mathbf{x}| + 1}$  for  $\mathbf{x} \in L$  (otherwise  $\tilde{p}(\mathbf{x}) = 0$ ).  $\tilde{p}$  is normalizable since  $Z$  is both finite ( $Z = \sum_{\mathbf{x} \in \mathbb{B}^*} \tilde{p}(\mathbf{x}) \leq \sum_{\mathbf{x} \in \mathbb{B}^*} \left(\frac{1}{3}\right)^{|\mathbf{x}| + 1} = 1$ ) and positive ( $Z > 0$  because the example string in footnote 8 has weight  $> 0$ ). The conditional distribution
An autoregressive model architecture generally defines  $p(\mathbf{x})$  as an efficiently computable (§2.2) product of local conditional probabilities. However, the parametrization usually ensures only that  $\sum_{x\in V}p_{\theta}(x|\hat{\mathbf{x}}) = 1$  for all prefixes  $\hat{\mathbf{x}}$ . Some parameter settings may give rise to inconsistent distributions where  $Z \triangleq \sum_{\mathbf{x}\in V^{*}}p_{\theta}(\mathbf{x}) < 1$  because the generative process terminates with probability  $< 1$  (Chen et al., 2018). In this case, the factors  $p_{\theta}(x|\hat{\mathbf{x}})$  defined by the autoregressive model are not actually the conditional probabilities of the weighted language (as defined by §2.1). It is true that training  $\theta$  with a likelihood objective does encourage finding a weighted language whose generative process always terminates (hence  $Z = 1$ ), since this is the behavior observed in the training corpus (Chi and Geman, 1998; Chen et al., 2018; Welleck et al., 2020). Our definitions of ELN(CP) models require the actual conditional probabilities to be efficiently computable. Autoregressive models that do not sum to 1, whose normalized probabilities can be uncomputable, are not ruled out by our theorems that concern ELN(CP).
For example,  $L$  contains the string  $\phi \mathbf{a}$  where  $\pmb{\phi} = \mathrm{enc}((A_1 \lor \neg A_2 \lor A_3) \land (A_1 \lor \neg A_4))$  and  $\mathbf{a} = 1101$ .
$p(\mathbf{a} \mid \boldsymbol{\phi})$  is uniform over the satisfying assignments  $\mathbf{a}$  of  $\boldsymbol{\phi}$ , as they all have the same length  $|\phi|$ .
$\tilde{p}$  is efficiently computable, and so is  $p = \tilde{p} /Z$  9 Yet deciding whether the local conditional probabilities of  $\tilde{p}$  are greater than O is NP-hard. In particular, we show that SAT can be reduced to deciding whether certain local probabilities are greater than O, namely the ones that condition on prefixes  $\hat{\mathbf{x}}$  that consist only of a formula:  $\hat{\mathbf{x}} = \boldsymbol{\phi}$  for some  $\phi$  . This implies, assuming NP  $\nsubseteq$  P/poly,that no  $(M^{\mathrm{q}},\Theta^{\mathrm{q}})$  can efficiently locally normalize  $\tilde{p}$  with compact parameters. Granted, the restriction of  $\tilde{p}$  to the finite set  $\{\mathbf{x}\in \mathbb{B}^{*}:|\mathbf{x}|\leq n\}$  can be locally normalized by some polytime Turing machine  $q_{n}$  , using the same trie trick sketched in §2.3. But such tries have sizes growing exponentially in n, and it is not possible to produce a sequence of such machines,  $\{q_n:n\in \mathbb{N}\}$  , via a single master Turing machine  $M^{\mathrm{q}}$  that runs in  $O(\mathrm{poly}(n))$  on  $\theta_{n}^{\mathrm{q}}$  . That is:
Theorem 1. Assuming NP  $\nsubseteq$  P/poly, there exists an efficiently computable normalizable weighted language  $\tilde{p}$  that is not ELNCP.
Proof sketch. Take  $\tilde{p}$  to be the weighted language we defined earlier in this section.  $\tilde{p}$  is clearly efficiently computable. We will show that if it is ELNCP via  $(M^{\mathbf{q}},\Theta^{\mathbf{q}})$ , then the NP-complete problem SAT is in P/poly, contradicting the assumption. We must give a method for using  $(M^{\mathbf{q}},\Theta^{\mathbf{q}})$  to decide SAT in polytime and with compact parameters  $\Theta$ . Given  $\phi$ , our method constructs a simple related formula  $\phi'$  such that
-  $\phi'$  has at least one satisfying assignment (so  $Z(\phi') > 0$  and thus  $p(1 \mid \phi')$  is defined)
-  $\phi'$  has satisfying assignments with  $A_1 = 1$  (i.e.,  $p(1 \mid \phi') > 0$ ) if and only if  $\phi$  is satisfiable
Our construction also provides a polynomial function  $f$  such that  $|\pmb{\phi}'|$  is guaranteed to be  $\leq f(|\pmb{\phi}|)$ . We now define  $\Theta$  by  $\theta_{n} = \theta_{f(n)}^{\mathrm{q}} (\forall n)$ . When our SAT algorithm with compact parameters  $\Theta$  is given  $\pmb{\phi}$  of length  $n$ , it can use the polynomial-size advice string  $\pmb{\theta}_{n}$  to ask  $(M^{\mathrm{q}}, \Theta^{\mathrm{q}})$  in polynomial time for  $p(1 \mid \pmb{\phi}')$ .  $\mathrm{SAT}(\pmb{\phi})$  returns true iff that probability is  $> 0$ .
3.3 ELNCP models cannot even capture all EC (or ECCP) supports or rankings
We can strengthen Theorem 1 as follows:
Theorem 2. Assuming NP  $\nsubseteq$  P/poly, there exists an efficiently computable normalizable weighted
language  $\tilde{p}$  where there is no ELNCP  $\tilde{q}$  such that support  $(\tilde{p}) = \mathrm{support}(\tilde{q})$
Proof. Observe that for any two weighted languages  $\tilde{p}$  and  $\tilde{q}$  with the same support,  $\forall \hat{\mathbf{x}}\in V^{*},Z_{\tilde{p}}(\hat{\mathbf{x}}) > 0\iff Z_{\tilde{q}}(\hat{\mathbf{x}}) > 0$  (where  $Z_{\tilde{p}}$  and  $Z_{\tilde{q}}$  return the prefix probabilities of  $\tilde{p}$  and  $\tilde{q}$  respectively). Thus, for any  $\hat{\mathbf{x}}$  with  $Z_{\tilde{p}}(\hat{\mathbf{x}}) > 0$ ,  $p(1\mid \hat{\mathbf{x}})\triangleq Z_{\tilde{p}}(\hat{\mathbf{x}} 1) / Z_{\tilde{p}}(\hat{\mathbf{x}})$  and  $q(1\mid \hat{\mathbf{x}})\triangleq Z_{\tilde{q}}(\hat{\mathbf{x}} 1) / Z_{\tilde{q}}(\hat{\mathbf{x}})$  are well-defined and  $p(1\mid \hat{\mathbf{x}}) > 0\iff q(1\mid \hat{\mathbf{x}}) > 0$ . If  $\tilde{q}$  is ELNCP, then all such probabilities  $q(1\mid \hat{\mathbf{x}})$  can be computed in polytime with compact parameters, so it is likewise efficient to determine whether  $p(1\mid \hat{\mathbf{x}}) > 0$ . But this cannot be the case when  $\tilde{p}$  is the weighted language used in the proof of Theorem 1, since that would suffice to establish that  $\mathrm{SAT}\in \mathrm{P} / \mathrm{poly}$ , following the proof of that theorem.
To put this another way, there exists an unweighted language in  $\mathbf{P}$  (namely support  $(\tilde{p})$ ) that is not the support of any ELNCP distribution.
If they have different support, normalizable languages also differ in their ranking of strings:
Lemma 3. Let  $\tilde{p},\tilde{q}$  be normalizable weighted languages with support  $(\tilde{p})\neq$  support  $(\tilde{q})$  . Then  $\exists \mathbf{x}_1$ $\mathbf{x}_2\in V^*$  such that  $\tilde{p} (\mathbf{x}_1) <   \tilde{p} (\mathbf{x}_2)$  but  $\tilde{q} (\mathbf{x}_1)\geq \tilde{q} (\mathbf{x}_2)$
Therefore, no ELNCP  $\tilde{q}$  captures the string ranking of  $\tilde{p}$  from Theorem 2. And for some  $\tilde{p}$ , any ELNCP  $\tilde{q}$  misranks even string pairs of "similar" lengths:
Theorem 3. Assuming NP  $\nsubseteq$  P/poly, there exists an efficiently computable normalizable weighted language  $\tilde{p}$  such that no ELNCP  $\tilde{q}$  with support  $(\tilde{q}) \supseteq$  support  $(\tilde{p})$  has  $\tilde{p}(\mathbf{x}_1) < \tilde{p}(\mathbf{x}_2) \Rightarrow \tilde{q}(\mathbf{x}_1) < \tilde{q}(\mathbf{x}_2)$  for all  $\mathbf{x}_1, \mathbf{x}_2 \in V^*$ . Indeed, any such  $\tilde{q}$  has a counterexample where  $\tilde{p}(\mathbf{x}_1) = 0$ . Moreover, there is a polynomial  $f_{\tilde{q}}: \mathbb{N} \to \mathbb{N}$  such that a counterexample exists for every  $\mathbf{x}_1$  such that  $\tilde{p}(\mathbf{x}_1) = 0$  and  $\tilde{q}(\mathbf{x}_1) > 0$ , where the  $\mathbf{x}_2$  in this counterexample always satisfies  $|\mathbf{x}_2| \leq f_{\tilde{q}}(|\mathbf{x}_1|)$ .
Theorem 3 is relevant if one wishes to train a model  $\tilde{q}$  to rerank strings that are proposed by another method (e.g., beam search on  $\tilde{q}$ , or exact  $k$ -best decoding from a more tractable distribution). If the desired rankings are given by Theorem 3's  $\tilde{p}$ , any smoothed[11] ELNCP model  $\tilde{q}$  will misrank some sets of candidate strings, even sets all of whose strings are "close" in length, by failing to rank an impossible string ( $\mathbf{x}_1$  with  $\tilde{p}(\mathbf{x}_1) = 0$ ) below a possible one ( $\mathbf{x}_2$  with  $\tilde{p}(\mathbf{x}_2) > 0$ ).
Smoothing is used to avoid ever incorrectly predicting 0 (a "false negative") by ensuring support(  $\tilde{q}$  )  $\supseteq$  support(  $\tilde{p}$  ). E.g., autoregressive language models often define  $q(x\mid \hat{\mathbf{x}})$  using a softmax over  $V\cup \{\mathbb{S}\}$ , ensuring that  $q(\mathbf{x}) > 0$  for all  $\mathbf{x}\in V^{*}$ .
9 Almost. This  $Z$  could be irrational, but at least it is computable to any desired precision. For any rational  $\hat{Z} \approx Z$ , we can say  $\hat{p} = \tilde{p} / \hat{Z} \approx p$  is EC, via a Turing machine  $M^{\hat{P}}$  that stores  $\hat{Z}$ . Further remarks on irrationality appear in Appendix A.
See also the remark on implications for seq2seq models following the proof in Appendix A.
3.4 ELNCP models cannot even approximate EC (or ECCP) distributions
Theorem 2 implies that there exists  $\tilde{p}$  whose local probabilities  $p(x\mid \hat{\mathbf{x}})$  are not approximated by any ELNCP  $q$  to within any constant factor  $\lambda$ , since that would perfectly distinguish zeroes from non-zeroes and the resulting support sets would be equal.[12]
However, this demonstration hinges on the difficulty of multiplicative approximation of zeroes — whereas real-world distributions may lack zeroes. Below we further show that it is hard even to approximate the non-zero local conditional probabilities (even with the additional help of randomness).
Theorem 4. Assuming NP  $\nsubseteq$  P/poly, there exists an efficiently computable weighted language  $\tilde{p} : V^{*}\to \mathbb{R}_{\geq 0}$  such that there is no  $(M^{\mathfrak{q}},\Theta^{\mathfrak{q}})$  where  $\Theta^{\mathfrak{q}} = \{\theta_n^{\mathfrak{q}}\mid n\in \mathbb{N}\}$  that satisfies all of the following properties (similar to §3.1):
- the parameter size  $|\pmb{\theta}_n^{\mathbf{q}}|$  grows only as  $O(\mathrm{poly}(n))$
-  $M^{\mathfrak{q}}(\pmb{\theta}_n^{\mathfrak{q}})$  returns a probabilistic Turing machine  $q_{n}$  in time  $O(\mathrm{poly}(n))$
- there exists  $\lambda \geq 1$  such that for each  $x \in V \cup \{\mathbb{S}\}$  and  $\hat{\mathbf{x}} \in V^{*}$  with  $|\hat{\mathbf{x}}| \leq n$  and  $p(x \mid \hat{\mathbf{x}}) > 0$ , the probabilistic computation  $q_{n}(\hat{\mathbf{x}} x)$  has probability  $>2/3$  of approximating  $p(x \mid \hat{\mathbf{x}})$  to within a factor of  $\lambda$  (that is,  $q_{n}(\hat{\mathbf{x}} x) / p(x \mid \hat{\mathbf{x}}) \in [1/\lambda, \lambda]$ )
-  $q_{n}$  runs on those inputs  $\hat{\mathbf{x}} x$  in time  $O(\mathrm{poly}(n))$ . Moreover, the statement above remains true
(a) when the approximation guarantee is only required to hold for prefixes  $\hat{\mathbf{x}}$  where  $\{\mathbf{x}:\hat{\mathbf{x}}\preceq \mathbf{x}\}$  is finite (so  $p(x\mid \hat{\mathbf{x}})$  is computable by brute force) (b) or, when support  $(\tilde{p}) = V^{*}$
3.5 ELN models are unconditionally weak
Our above results rely on the NP-hardness of computing or approximating an EC distribution's autoregressive factors  $p(\cdot \mid \mathbf{x}_{< t})$ . In Appendix A, we show that these factors can even be uncomputable. In such cases, the distribution cannot be ELN (Theorem 5), though sometimes it is still ELNCP (Theorem 6). This result does not assume  $\mathrm{P} \neq \mathrm{NP}$  or  $\mathrm{NP} \not\subset \mathrm{P/poly}$ .
4 Alternative model families
We now discuss alternative families of sequence distributions that trade away efficiency or compactness in exchange for greater capacity, as shown in Table 1.
4.1 Energy-based models (EBMs)
Energy-based models (LeCun et al., 2006) of discrete sequences (Rosenfeld et al., 2001; Sandbank, 2008; Huang et al., 2018) traditionally refer to the EC models of §2.2. Only the unnormalized probabilities  $\tilde{p}_{\theta}(\mathbf{x})$  are required to be efficiently computable. Lemmas 1 and 2 showed that this model family contains all ELN languages and can achieve any support in P. Theorem 1 shows that it also contains languages that are not ELN or even ELNCP: intuitively, the reason is that the sums  $Z(\hat{\mathbf{x}})$  needed to compute the local normalizing constants (see §2.1) can be intractable.
If we generalize energy-based sequence models to include all ECCP models—that is, we allow nonuniform computation with compact parameters—then Lemmas 1 and 2 guarantee that they can capture all ELNCP languages and furthermore all languages in P/poly (though still not NP-complete languages).
Experiments on different parameterizations. Maximum-likelihood parameter estimation (MLE) can be expensive in an EBM because the likelihood formula involves the expensive summation  $Z = \sum_{\mathbf{x} \in V^*} \tilde{p}_{\theta}(\mathbf{x})$ . This forces us in practice to use alternative estimators that do not require computing normalized probabilities, such as noise-contrastive estimation (NCE) or score matching (\$1), which are less statistically efficient. In pilot experiments we found that both RNN- and Transformer-based EBMs trained with NCE achieved worse held-out perplexity than comparable locally normalized models trained with MLE. $^{13}$
Fortunately, it is possible to infuse a globally normalized architecture with the inductive bias of a locally normalized one, which empirically yields good results. Residual energy-based models (REBMs) (Bakhtin et al., 2021) are a simple hybrid architecture:
$$
p _ {\theta} (\mathbf {x}) \propto \tilde {p} _ {\theta} (\mathbf {x}) \triangleq p _ {0} (\mathbf {x}) \cdot \exp g _ {\theta} (\mathbf {x})
$$
This simply multiplies our previous weight by a new factor  $p_0(\mathbf{x})$ . The base model  $p_0: L \to (0,1]$  is a locally normalized neural sequence model (ELN model) that was pretrained on the same distribution.  $g_{\theta}: V^{*} \to \mathbb{R}$  is a learnable function (with parameters  $\pmb{\theta}$ ) that is used to adjust  $p_0$ , yielding a weighted language  $\tilde{p}_{\theta}$  with the same support  $L$ . We implemented
<sup>13</sup>This might be due to a capacity limitation of the specific globally normalized architectures (i.e., no parameters work well), or excess capacity (i.e., too many parameters work well on the finite sample), or statistical inefficiency of the estimator (the NCE objective on the finite sample, with the noise distribution we chose, does not distinguish among parameters as well as MLE does), or an optimization difficulty caused by local optima in the NCE optimization landscape.
$^{12}$ Dropping the normalization requirement on the approximated local probabilities (so that possibly  $\sum_{x \in V} q(x \mid \hat{\mathbf{x}}) \neq 1$ ) does not help. Otherwise, again, SAT could be solved in polynomial time (with the help of polysize advice strings) by using  $q(1 \mid \boldsymbol{\phi}')$  to determine in the proof of Theorem 1 whether  $p(1 \mid \boldsymbol{\phi}') > 0$ .
REBMs, again with NCE training, and evaluated them on two different neural architectures (GRU- and Transformer-based) and 3 datasets (WikiText (Merit et al., 2017), Yelp (Yelp), and RealNews (Zellers et al., 2019)). In each setting we tried, the REBM slightly but significantly improved the perplexity of the base model  $p_0$  ( $p < 0.05$ ).<sup>14</sup>
4.2 Latent-variable models
Autoregressive models have  $Z = 1$  for any setting of the parameters (or at least any setting that guarantees consistency: see footnote 7). Clearly  $Z = 1$  ensures that  $Z$  is both finite and tractable. Can we find a model family that retains this convenience (unlike EBMs), while still being expressive enough to have any non-empty language in  $P$  as support?
Autoregressive latent-variable models form such a family. As in directed graphical models, the use of latent variables provides a natural way to model partial observations of an underlying stochastic sequence of events. We will model an observed sequence  $\mathbf{x}$  of length  $n$  as a function of a latent string  $\mathbf{z}$  of length  $O(\mathrm{poly}(n))$ . As in EBMs, the probability  $p(\mathbf{x})$  can be computationally intractable, allowing these models to break the expressivity bottleneck of ordinary autoregressive models. However, the intractability no longer comes from exponentially many summands in the denominator  $Z$ , but rather from exponentially many summands in the numerator—namely, the summation over all latent  $\mathbf{z}$  that could have produced  $\mathbf{x}$ . Notice that as a result, even unnormalized string weights are now hard to compute, although once computed they are already normalized.
Formally, we define **marginalized weighted languages.** We say that  $\tilde{p}$  is a **marginalization** of the weighted language  $\tilde{r}$  if it can be expressed as  $\tilde{p}(\mathbf{x}) = \sum_{\mathbf{z}: \mu(\mathbf{z}) = \mathbf{x}} \tilde{r}(\mathbf{z})$ , where  $\mu: S \to V^*$  is some function (the **marginalization operator**). We say it is a **light marginalization** if  $|\mathbf{z}| \in O(\mathrm{poly}(|\mu(\mathbf{z})|))$  and  $\mu$  runs in time  $O(\mathrm{poly}(|\mathbf{z}|))$ . Typically  $\mu(\mathbf{z})$  extracts a subsequence of  $\mathbf{z}$ ; it can be regarded as keeping the observed symbols while throwing away a polynomially bounded number of latent symbols.
Light marginalizations of ELN distributions are a
reasonable formalization of latent-variable autoregressive models. They are more powerful than ELN distributions, and even include some distributions that (by Lemma 1) are not even ELNCP or ECCP:
Theorem 7. There exists a light marginalization  $p$  of an ELN distribution, such that support  $(p)$  is an NP-complete language.
Our proof of Theorem 7 relies on special structure of a certain NP-complete language (SAT) and does not evidently generalize to all languages in NP.
However, light marginalizations of ELNCP distributions are more powerful still,[16] and can have any language  $\in$  NP or even NP/poly (§2.4) as support:
Theorem 8. The following statements are equivalent for any nonempty  $L \subseteq V^{*}$ :
(a)  $L\in \mathrm{NP / poly}$
(b)  $L$  is the support of a light marginalization of an ELNCP distribution.
(c)  $L$  is the support of a light marginalization of an ECCP weighted language.
Theorems 7 and 8 make use of unrestricted latent-variable autoregressive models. There exist more practical restricted families of such models that admit tractable computation of  $p(\mathbf{x})$  (Lafferty et al., 2001; Rastogi et al., 2016; Wu et al., 2018; Buys and Blunsom, 2018). Such models are EC (and indeed, typically ELN)—but this limits their expressivity, by Theorem 1. Both Lin et al. (2019) and Buys and Blunsom (2018) observed that such models yield worse empirical results than models that do not have tractable exact inference methods. The tractability requirement is dropped in "self-talk" (blixt, 2020; Gontier et al., 2020; Shwartz et al., 2020), where a neural autoregressive language model generates an analysis of the prefix  $\hat{\mathbf{x}}$  via latent intermediate symbols before predicting the next output symbol.[17]
We remark that for autoregressive models, the position of the latent variables is significant. Marginalizing out latent variables at the end of the string adds no power. More precisely, if an ELNCP distribution is over strings  $\mathbf{z}$  of the form  $\mathbf{x}\# \mathbf{y}$ , then its marginalization
16The capacity established by Theorem 8 does not need the full power of marginalization. We could similarly define light maximizations of ELNCP distributions,  $\tilde{p} (\mathbf{x}) = \max_{\mathbf{z};\mu (\mathbf{z}) = \mathbf{x}}\tilde{r} (\mathbf{z})$  Replacing sum by max does not change the support.
17Here the marginal distribution of the next observed symbol can require superpolynomial time to compute (if  $\# \mathrm{P} \neq \mathrm{FP}$ , which follows from NP  $\nsubseteq$  P/poly). Theorem 1 could likewise be evaded by other autoregressive approaches that invest superpolynomial computation in predicting the next symbol (Graves, 2016). Each autoregressive step might explicitly invoke lookahead or reasoning algorithms, just as feed-forward network layers can invoke optimizers or solvers (Amos and Kolter, 2017; Wang et al., 2019b).
14We independently conceived of and implemented the REBM idea proposed in Bakhtin et al. (2021). Details of neural architecture choice, model parameter sizes, training regimen, and evaluation (Appendices B-D) differ between our work and theirs, which also reported positive empirical results (on different datasets). We regard the two independent positive findings as a strong indication that the REBM design is effective.