SlowGuess/NAACL_2021_OCR huggingface dataset:
Remark on conditional models. While we focus on modeling joint sequence probabilities in this work, we note that in many applications it often suffices to just model conditional probabilities (Sutskever et al., 2014). Unfortunately, our proof of Theorem 1 above implies that ELNCPs do not make good conditional models either: specifically, there exists  $\phi$  such that deciding whether  $p(1\mid \phi) > 0$  is NP-hard, and thus beyond ELNCP's capability.
Remark on irrationality. In our definitions of ECCP and ELNCP languages, we implicitly assumed that the Turing machines that return weights or probabilities would write them in full on the output tape, presumably as the ratio of two integers. Such a
Turing machine can only return rational numbers.
But then our formulation of Theorem 1 allows another proof. We could construct  $\tilde{p}$  such that the local conditional probabilities  $p(x\mid \hat{\mathbf{x}})\triangleq Z(\hat{\mathbf{x}} x) / Z(\hat{\mathbf{x}})$  are sometimes irrational. In this case, they cannot be output exactly by a Turing machine, implying that  $\tilde{p}$  is not ELNCP. However, this proof exposes only a trivial weakness of ELNCPs, namely the fact that they can only define distributions whose local marginal probabilities are rational.
We can correct this weakness by formulating ELNCP languages slightly differently. A real number is said to be computable if it can be output by a Turing machine to any desired precision. That Turing machine takes an extra input  $b$  which specifies the number of bits of precision of the output. Similarly, our definitions of ECCP and ELNCP can be modified so that their respective Turing machines  $\tilde{p}_n$  and  $q_n$  take this form, are allowed to run in time  $O(\mathrm{poly}(n + b))$ , and have access to the respective parameter vectors  $\Theta_{n + b}^{\mathrm{p}}$  and  $\Theta_{n + b}^{\mathrm{q}}$ . Since some of our results concern the ability to distinguish zero from small values (arbitrarily small in the case of Theorem 6), our modified definitions also require  $\tilde{p}_n$  and  $q_n$  to output a bit indicating whether the output is exactly zero. For simplicity, we suppressed these technical details from our exposition.
Relatedly, in §4.3, we claimed that lookup models can fit any weighted language up to length  $n$ . This is not strictly true if the weights can be irrational. A more precise statement is that for any weighted language  $\tilde{p}$ , there is a lookup model that maps  $(\mathbf{x}, b)$  to the first  $b$  bits of  $\tilde{p}(\mathbf{x})$ . Indeed, this holds even when  $\tilde{p}(\mathbf{x})$  is uncomputable.
Remark on computability. In §2.1 we claimed that any weighted language  $\tilde{p}$  that has a finite and strictly positive  $Z$  can be normalized as  $p(\mathbf{x}) = \tilde{p}(\mathbf{x}) / Z$ . However,  $Z$  may be uncomputable: that is, there is no algorithm that takes number of bits of precision  $b$  as input, and outputs an approximation of  $Z$  within  $b$  bits of precision. Therefore, even if  $\tilde{p}$  is computable,  $p$  may have weights that are not merely irrational but even uncomputable. An example appears in the proof of Theorem 6 below. Weighted language classes (e.g. ELNCP) that only model normalized languages will not be able to model such languages, simply because the partition function is uncomputable.
However, our proof of Theorem 1 does not rely on this issue, because the  $\tilde{p}$  that it exhibits happens to have a computable  $Z$ . For any  $b,Z$  may be computed to  $b$  bits of precision as the explicit sum  $\sum_{\mathbf{x}:|\mathbf{x}|\leq N}\tilde{p} (\mathbf{x})$
for a certain large  $N$  that depends on  $b$ .
Remark on RNNs. Our proof of Theorem 1 showed that our problematic language  $\tilde{p}$  is efficiently computable (though not by any locally normalized architecture with compact parameters). Because this paper is in part a response to popular neural architectures, we now show that  $\tilde{p}$  can in fact be computed efficiently by a recurrent neural network (RNN) with compact parameters. Thus, this is an example where a simple globally normalized RNN parameterization is fundamentally more efficient (in runtime or parameters) than any locally normalized parameterization of any architecture (RNN, Transformer, etc.).
Since we showed that  $\tilde{p}$  is efficiently computable, the existence of an RNN implementation is established in some sense by the ability of finite rational-weighted RNNs to simulate Turing machines (Siegelmann and Sontag, 1992), as well as an extension to Chen et al. (2018, Thm. 11) to a family of RNNs, where each RNN instance also takes some formula encoding as input. However, it is straightforward to give a concrete construction, for each  $n \in \mathbb{N}$ , for a simple RNN that maps each string  $\mathbf{x} \in \mathbb{B}^n$  to  $\tilde{p}(\mathbf{x})$ . Here  $\tilde{p}(\mathbf{x})$  will be either  $(\frac{1}{3})^{n+1}$  or 0, according to whether  $\mathbf{x}$  has the form  $\phi \mathbf{a}$  where  $\phi$  encodes a 3-CNF-SAT formula  $\phi$  that is satisfied by  $\mathbf{a}$ . The basic idea is that  $\phi$  has  $j \leq n$  variables, so there are only  $O(n^3)$  possible 3-CNF clauses. The RNN allocates one hidden unit to each of these. When reading  $\phi \mathbf{a}$ , each clause encountered in  $\phi$  causes the corresponding hidden unit to turn on, and then each literal encountered in  $\mathbf{a}$  turns off the hidden units for all clauses that would be satisfied by that literal. If any hidden units remain on after  $\mathbf{x}$  has been fully read, then  $\phi$  was not satisfied by  $\mathbf{a}$ , and the RNN's final output unit should return 0. Otherwise it should return  $(\frac{1}{3})^{n+1}$ , which is constant for this RNN. To obtain digital behaviors such as turning hidden units on and off, it is most convenient to use ramp activation functions for the hidden units and the final output unit, rather than sigmoid activation functions. Note that our use of a separate RNN  $M_n^{\mathrm{RNN}}$  for each input length  $n$  is an example of using more hidden units for larger problems, a key idea that we introduced in §2.3 in order to look at asymptotic behavior. The RNN's parameter
sequence  $\Theta^{\mathrm{RNN}} = \{\pmb{\theta}_n^{\mathrm{RNN}} \mid n \in \mathbb{N}\}$  is obviously compact, as  $\pmb{\theta}_n^{\mathrm{RNN}}$  only has to store the input length  $n$ . With our alphabet  $\mathbb{B}$  for  $\tilde{p}$ ,  $|\pmb{\theta}_n^{\mathrm{RNN}}| \in O(\log n)$ .
Proof. Suppose that the claim is false, i.e.,  $\tilde{p}$  and  $\tilde{q}$  have the same ranking of strings. Then the minimum-weight strings under  $\tilde{p}$  must also be minimum-weight under  $\tilde{q}$ . WLOG, there exists  $\mathbf{x} \in V^{*}$  with  $\tilde{p}(\mathbf{x}) = 0$  and  $\tilde{q}(\mathbf{x}) = c > 0$ . Then  $c > 0$  is the minimum weight of strings in  $\tilde{q}$ . But this is not possible for a normalizable language  $\tilde{q}$ , since it means that  $Z_{\tilde{q}} \triangleq \sum_{\mathbf{x}' \in V^{*}} q(\mathbf{x}') \geq \sum_{\mathbf{x}' \in V^{*}} c$  diverges.
Proof. Let  $\tilde{p}$  be the weighted language from Theorem 2. Given an ELNCP  $\tilde{q}$ . By Theorem 2, support  $(\tilde{q}) \neq$  support  $(\tilde{p})$ , so there must exist a string  $\mathbf{x}_1$  that is in one support language but not the other. With the additional assumption that support  $(\tilde{q}) \supseteq$  support  $(\tilde{p})$ , it must be that  $\mathbf{x}_1 \in$  support  $(\tilde{q})$ , so  $\tilde{p}(\mathbf{x}_1) = 0$  but  $\tilde{q}(\mathbf{x}_1) > 0$ .
Given any such  $\mathbf{x}_1$  with  $\tilde{p}(\mathbf{x}_1) = 0$  but  $\tilde{q}(\mathbf{x}_1) > 0$ , we must find a  $\mathbf{x}_2$  of length  $O(\mathrm{poly}(|\mathbf{x}_1|))$  with  $\tilde{p}(\mathbf{x}_2) > 0$  but  $\tilde{q}(\mathbf{x}_2) \leq \tilde{q}(\mathbf{x}_1)$ .
To ensure that  $\tilde{p} (\mathbf{x}_2) > 0$ , let us use the structure of  $\tilde{p}$ . For any  $j$ , we can construct a tautological formula  $\phi$  over variables  $A_{1},\ldots A_{j}$ , as  $\phi = (A_{1}\lor \neg A_{1})\land \dots \land (A_{j}\lor \neg A_{j})$ . It follows that  $\tilde{p} (\phi \mathbf{a}) > 0$  for any  $\mathbf{a}\in \mathbb{B}^j$ . We will take  $\mathbf{x}_2 = \boldsymbol {\phi}\mathbf{a}$  for a particular choice of  $j$  and  $\mathbf{a}$ .
Specifically, we choose them to ensure that  $\tilde{q} (\mathbf{x}_2)\leq$ $\tilde{q} (\mathbf{x}_1)$  . Since  $\tilde{q}$  is ELNCP, it is normalizable and hence has a finite  $Z$  . Thus,  $\sum_{\mathbf{a}\in \mathbb{B}^j}\tilde{q} (\pmb {\phi}\mathbf{a})\leq Z$  . So there must exist some  $\mathbf{a}\in \mathbb{B}^j$  such that  $\tilde{q} (\pmb {\phi}\mathbf{a})\leq Z / 2^{j}$  . We choose that  $\mathbf{a}$  , after choosing  $j$  large enough such that  $Z / 2^{j}\leq \tilde{q} (\mathbf{x}_{1})$  . Then  $\tilde{q} (\mathbf{x}_2) = \tilde{q} (\pmb {\phi}\mathbf{a})\leq Z / 2^j\leq$ $\tilde{q} (\mathbf{x}_1)$
To achieve the last claim of the theorem, we must also ensure that  $|\mathbf{x}_2| \in O(\mathrm{poly}(|\mathbf{x}_1|))$ . Observe that  $\tilde{q}(\mathbf{x}_1)$  can be computed in polytime (with access to
18The restriction to 3-CNF-SAT formulas is convenient, but makes this a slightly different definition of  $L$  and  $\tilde{p}$  than we used in the proofs above. Those proofs can be adjusted to show that this  $\tilde{p}$ , too, cannot be efficiently locally normalized with compact parameters. The only change is that in the construction of Theorem 1,  $\phi^{\prime}$  must be converted to 3-CNF. The proof then obtains its contradiction by showing that 3-CNF-SAT  $\in \mathbb{P} / \mathrm{poly}$  (which suffices since 3-CNF-SAT is also NP-complete).
compact parameters), by Lemma 2. But this means that the representation of  $\tilde{q} (\mathbf{x}_1) > 0$  as a rational number must have  $\leq g(|\mathbf{x}_1|)$  bits for some polynomial  $g$ . Then  $\tilde{q} (\mathbf{x}_1)\geq 2^{-g(|\mathbf{x}_1|)}$ , and it suffices to choose  $j = \lceil g(|\mathbf{x}_1|) + \log_2Z\rceil$  to ensure that  $Z / 2^{j}\leq 2^{-g|\mathbf{x}_{1}|}\leq$ $\tilde{q} (\mathbf{x}_1)$  as required above.
But then  $j \in O(\mathrm{poly}(|\mathbf{x}_1|))$ . Also, recall that the encoding function enc used in the construction of  $\tilde{p}$  is guaranteed to have only polynomial blowup (see the proof of Theorem 2). Thus,  $|\mathbf{x}_2| = |\boldsymbol{\phi}| + |\mathbf{a}| = |\mathrm{enc}(\boldsymbol{\phi})| + j \in O(\mathrm{poly}(j)) \subseteq O(\mathrm{poly}(|\mathbf{x}_1|))$  as required by the theorem.
Lemma A.1. The first part of Theorem 4 (without the modifications (a) and (b)).
We first prove the first part of Theorem 4 (which is restated in full below). In this case we will use a distribution  $\tilde{p}$  that does not have support  $V^{*}$  (so it does not prove modification (b)).
Proof. We take  $\tilde{p}$  to be the weighted language that was defined in §3.2, which was already shown to be efficiently computable. Suppose  $(M^{\mathbf{q}},\Theta^{\mathbf{q}},\lambda)$  is a counterexample to Lemma A.1. Choose integer  $k\geq 1$  in a manner (dependent only on  $\lambda$ ) to be described at the end of the proof.
Suppose we would like to answer SAT where  $\phi$  is a formula with variables  $A_{1},\ldots ,A_{j}$ . Define  $\phi^{\prime} = (\neg A_{1}\land \neg A_{2}\land \dots \land \neg A_{j}\land \neg A_{j + 1}\land \neg A_{j + k})\lor$ $(A_{1}\wedge \mathrm{Shift}(\phi))$ . Note that  $\phi^\prime$  augments  $\phi$  with  $k$  additional variables, namely  $A_{1}$  and  $A_{j + 2,\dots,j + k}$ . For  $k = 1$ , this is the same construction as in the proof of Theorem 1. Let  $n = |\pmb {\phi}'|$  and note that  $n$  is polynomial in the size of  $\phi$  (holding  $k$  constant).
The strings in  $L = \mathrm{support}(\tilde{p})$  that begin with  $\phi^{\prime}$  are precisely the strings of the form  $\phi^{\prime}\mathbf{a}^{\prime}$  where  $\mathbf{a}'$  is a satisfying assignment of  $\phi^{\prime}$ . This is achieved precisely when  $\mathbf{a}' = \mathfrak{O}^{j + k}$  or  $\mathbf{a}' = 1\mathbf{a}\vec{b}$  where  $\mathbf{a}$  is a satisfying assignment of  $\phi$  and  $\vec{b} \in \mathbb{B}^{k - 1}$ .
By our definition of  $\tilde{p}$ , all strings in  $L$  that begin with  $\phi'$  have equal weight under  $\tilde{p}$ . Call this weight  $w$ . Clearly  $Z(\phi' \emptyset) = w$ , and  $Z(\phi' 1) = w \cdot 2^{k-1}$  (number of satisfying assignments of  $\phi$ ).
Recall that  $p(\emptyset \mid \phi') = Z(\phi' \emptyset) / (Z(\phi' \emptyset) + Z(\phi'1))$ . Let us abbreviate this quantity by  $p$ . It follows from the previous paragraph that if  $\phi$  is unsatisfiable, then  $p = 1$ , but if  $\phi$  is satisfiable, then  $p \leq 1 / (1 + 2^{k - 1})$ . By hypothesis,  $p$  is approximated (with error probability  $< 1/3$ ) by the possibly random quantity  $(M^{\mathrm{q}}(\theta_{|\phi'|}^{\mathrm{q}})) (\phi' \emptyset)$ , which we abbreviate by
$q$ , to within a factor of  $\lambda$ . That is,  $p \in [q / \lambda, \lambda q]$ . By choosing  $k$  large enough[20] such that  $[q / \lambda, \lambda q]$  cannot contain both 1 and  $1 / (1 + 2^{k - 1})$ , we can use  $q$  to determine whether  $p = 1$  or  $p \leq 1 / (1 + 2^{k - 1})$ . This allows us to determine  $\mathrm{SAT}(\phi)$  in polynomial time with error probability  $< 1 / 3$ , since by hypothesis  $q$  is computable in polynomial time with compact parameters. This shows that  $\mathrm{SAT} \in \mathrm{BPP} / \mathrm{poly} = \mathrm{P} / \mathrm{poly}$ , implying  $\mathrm{NP} \subseteq \mathrm{P} / \mathrm{poly}$ , contrary to our assumption. (BPP/poly is similar to  $\mathrm{P} / \mathrm{poly}$  but allows  $M^{\mathfrak{q}}$  to be a bounded-error probabilistic Turing machine.)
Theorem 4. Assuming NP  $\nsubseteq$  P/poly, there exists an efficiently computable weighted language  $\tilde{p}: V^{*} \to \mathbb{R}_{\geq 0}$  such that there is no  $(M^{\mathfrak{q}}, \Theta^{\mathfrak{q}})$  where  $\Theta^{\mathfrak{q}} = \{\theta_{n}^{\mathfrak{q}} \mid n \in \mathbb{N}\}$  that satisfies all of the following properties (similar to §3.1):
- the parameter size  $|\pmb{\theta}_n^{\mathrm{q}}|$  grows only as  $O(\mathrm{poly}(n))$
-  $M^{\mathfrak{q}}(\pmb{\theta}_n^{\mathfrak{q}})$  returns a probabilistic Turing machine  $q_{n}$  in time  $O(\mathrm{poly}(n))$
- there exists  $\lambda \geq 1$  such that for each  $x \in V \cup \{\mathbb{S}\}$  and  $\hat{\mathbf{x}} \in V^{*}$  with  $|\hat{\mathbf{x}}| \leq n$  and  $p(x \mid \hat{\mathbf{x}}) > 0$ , the probabilistic computation  $q_{n}(\hat{\mathbf{x}} x)$  has probability  $>2/3$  of approximating  $p(x \mid \hat{\mathbf{x}})$  to within a factor of  $\lambda$  (that is,  $q_{n}(\hat{\mathbf{x}} x) / p(x \mid \hat{\mathbf{x}}) \in [1/\lambda, \lambda]$ )
-  $q_{n}$  runs on those inputs  $\hat{\mathbf{x}} x$  in time  $O(\mathrm{poly}(n))$ . Moreover, the statement above remains true
Proof. It remains to show that the statement remains true with modification (a) and with modification (b). For (a), the proof of Lemma A.1 suffices, since it reduces SAT to approximate local probability queries of the stated form. That is, the true local probabilities  $p(x \mid \hat{\mathbf{x}})$  that can be computed with finite summations, thanks to the structure of our example language  $\tilde{p}$ , which guarantees that the prefix  $\hat{\mathbf{x}}$  can only continue with suffixes of a fixed length that is easily determined from  $\hat{\mathbf{x}}$ .
For modification (b), again let  $V = \mathbb{B} = \{\emptyset, 1\}$ . Choose some  $\epsilon > 0$  (any choice will do), and let
$$
\begin{array}{l} \tilde {p} _ {1} (\mathbf {x}) = \left\{ \begin{array}{l l} \left(\frac {1}{3}\right) ^ {| \mathbf {x} + 1 |} & \text {i f} \mathbf {x} = \boldsymbol {\phi} \mathbf {a} \text {w h e r e} \boldsymbol {\phi} = \operatorname {e n c} (\boldsymbol {\phi}) \\ & \text {a n d} \mathbf {a} \text {s a t i s f i e s} \boldsymbol {\phi} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \\ \tilde {p} _ {2} (\mathbf {x}) = \left(\frac {1}{9}\right) ^ {| \mathbf {x} + 1 |} > 0 \\ \tilde {p} (\mathbf {x}) = \tilde {p} _ {1} (\mathbf {x}) + \epsilon \cdot \tilde {p} _ {2} (\mathbf {x}) \\ \end{array}
$$
20It suffices to ensure that  $1 + 2^{k - 1} > \lambda^2$ , so take any  $k > 1 + \log_2(\lambda^2 - 1)$ .
${}^{19}$  Specifically,each such string has length  $n + j + k$  ,so  $\tilde{p}$  gives it a weight of  $w = {\left( \frac{1}{3}\right) }^{n + j + k + 1}$  .
We use  $Z_{1}, Z_{2}$ , and  $Z$  respectively to denote normalizing constants of these three weighted languages. Note that  $\tilde{p}_{1}$  is the weighted language that was previously used in the proofs of Theorem 1 and Lemma A.1. Our new  $\tilde{p}$  is intended to be very similar while satisfying the additional condition (b). It is easy to show that  $\tilde{p}$  is efficiently computable, much as we showed for  $\tilde{p}_{1}$  in Theorem 1. Also,  $\tilde{p}$  is normalizable, since  $Z = Z_{1} + \epsilon \cdot Z_{2}$ , where  $Z_{1} \leq \left(\frac{1}{3}\right) / \left(1 - \frac{2}{3}\right) = 1$  and  $Z_{2} = \left(\frac{1}{9}\right) / \left(1 - \frac{2}{9}\right) = \frac{1}{7}$  are both finite.
The proof proceeds as in Lemma A.1, with  $\phi'$  constructed from  $\phi$  as before. Recall that  $\phi$  has  $j$  variables,  $\phi'$  has  $j + k$  variables, and  $|\phi'| = n$ . We may assume WLOG that the encoding function enc is such that an encoded formula always has at least as many bits as the number of variables in the formula, so  $n \geq j + k$ .
Notice that  $Z_{1}(\pmb{\phi}^{\prime})$  sums over the satisfying assignments of  $\phi^{\prime}$ , and there may be as few as one of these (if  $\phi$  is unsatisfiable). By contrast,  $Z_{2}(\pmb{\phi}^{\prime})$  sums over an infinite number of continuations with positive probability. The faster decay rate of  $\frac{1}{9}$  in  $\tilde{p}_{2}$  was chosen to keep  $Z_{2}(\pmb{\phi}^{\prime})$  small relative to  $Z_{1}(\pmb{\phi}^{\prime})$  despite this. Specifically,
$$
Z _ {1} \left(\phi^ {\prime} \emptyset\right) = \left(\frac {1}{3}\right) ^ {n + j + k + 1}
$$
$$
Z _ {1} \left(\boldsymbol {\phi} ^ {\prime} 1\right) = \left(\frac {1}{3}\right) ^ {n + j + k + 1} \cdot 2 ^ {k - 1}
$$
- (# of satisfying assignments of  $\phi$ )
$$
\begin{array}{l} Z _ {2} \left(\boldsymbol {\phi} ^ {\prime} \mathfrak {0}\right) = \left(\frac {1}{9}\right) ^ {n} \cdot \frac {1}{9} \cdot \left(\frac {1}{9} / \left(1 - \frac {2}{9}\right)\right) \\ = \frac {1}{7} \cdot \left(\frac {1}{3}\right) ^ {2 (n + 1)} \\ <   \frac {1}{7} \cdot Z _ {1} \left(\phi^ {\prime} \mathbb {0}\right) \\ \end{array}
$$
(because  $2(n + 1) > n + j + k + 1$ )
$$
Z _ {2} \left(\phi^ {\prime} 1\right) = Z _ {2} \left(\phi^ {\prime} \mathbb {0}\right)
$$
As in the proof of Lemma A.1, we will show that  $p(\emptyset \mid \phi')$  is much larger when  $\phi$  is unsatisfiable. Recall that  $Z(\hat{\mathbf{x}}) = Z_1(\hat{\mathbf{x}}) + \epsilon \cdot Z_2(\hat{\mathbf{x}})$ . When  $\phi$  has zero satisfying assignments,
$$
\begin{array}{l} p (\emptyset \mid \phi^ {\prime}) = \frac {Z (\phi^ {\prime} \emptyset)}{Z (\phi^ {\prime} \emptyset) + Z (\phi^ {\prime} 1)} \\ = \frac {Z (\phi^ {\prime} 0)}{Z _ {1} \left(\phi^ {\prime} 0\right) + \epsilon \cdot Z _ {2} \left(\phi^ {\prime} 0\right) + \epsilon \cdot Z _ {2} \left(\phi^ {\prime} 1\right)} \\ > \frac {Z (\phi^ {\prime} 0)}{Z _ {1} (\phi^ {\prime} 0) + 2 \cdot \frac {\epsilon}{7} \cdot Z _ {1} (\phi^ {\prime} 0)} \\ \end{array}
$$
whereas if  $\phi$  has at least one satisfying assignment,
$$
\begin{array}{l} p (\emptyset \mid \phi^ {\prime}) = \frac {Z (\phi^ {\prime} \emptyset)}{Z (\phi^ {\prime} \emptyset) + Z (\phi^ {\prime} 1)} \\ <   \frac {Z (\phi^ {\prime} \emptyset)}{Z _ {1} (\phi^ {\prime} \emptyset) + Z _ {1} (\phi^ {\prime} 1)} \\ \leq \frac {Z (\boldsymbol {\phi} ^ {\prime} \boldsymbol {0})}{Z _ {1} (\boldsymbol {\phi} ^ {\prime} \boldsymbol {0}) + 2 ^ {k - 1} Z _ {1} (\boldsymbol {\phi} ^ {\prime} \boldsymbol {0})} \\ \end{array}
$$
This rewrites both probabilities in terms of  $Z.(\phi'0)$  quantities, which do not depend on the number of satisfying assignments. So now we can see that the first probability is at least  $\left(1 + 2^{k - 1}\right) / \left(1 + \frac{2\epsilon}{7}\right)$  times as large as the second probability. Choose  $k$  large enough[21] such that  $[q / \lambda ,\lambda q]$  cannot contain both probabilities, and complete the proof as in Lemma A.1.
Theorem 5. The set  $\{\tilde{p}:\tilde{p}$  is normalizable,  $\tilde{p}\in$  EC,  $\tilde{p}\notin \mathrm{ELN}\}$  is not empty.
Theorem 5 states that some normalizable EC distributions cannot be expressed as ELN distributions. The proof is based on the undecidability of the halting problem, rather than the assumed inefficiency of the Boolean satisfiability problem. Thus, unlike Theorem 1, it does not rely on the assumption that NP  $\nsubseteq$  P/poly, or even on the weaker assumption that  $\mathrm{P}\neq \mathrm{NP}$
Proof. Given any unweighted language  $L \subseteq \mathbb{B}^*$ , we can define a normalizable weighted language  $\tilde{p}$  with support  $L$  by  $\tilde{p}(\mathbf{x}) = 1/3^{|\mathbf{x}| + 1}$  for  $\mathbf{x} \in L$  and  $\tilde{p}(\mathbf{x}) = 0$  otherwise. Moreover, if  $L \in \mathrm{P}$ , then  $\tilde{p} \in \mathrm{EC}$ .
For our purposes, we take  $L$  to consist of all strings of the form  $\mathbf{x}^{(1)}\mathbf{x}^{(2)}$ , for which there exists a deterministic Turing machine  $M$  such that  $\mathbf{x}^{(1)} = \mathrm{enc}(M)$  (where enc is a prefix-free encoding function) and  $\mathbf{x}^{(2)}$  encodes an accepting execution path of  $M$  on an empty input. (Such a path may be represented as a sequence of transitions of  $M$  that begins with an initial state and ends at an accepting state.) Note that any deterministic TM  $\mathbf{x}^{(1)}$  can be paired with at most one accepting execution path  $\mathbf{x}^{(2)}$ , and cannot be paired with any  $\mathbf{x}^{(2)}$  if it does not halt.
Clearly  $L \in \mathbb{P}$ : given  $\mathbf{x} \in \mathbb{B}^*$ , we can decide whether  $\mathbf{x} \in L$  by first checking if  $\mathbf{x}$  can be expressed as a concatenation of strings  $\mathbf{x}^{(1)}$  and  $\mathbf{x}^{(2)}$  of the required form. Then we build  $M$  from  $\mathbf{x}^{(1)}$  and simulate it to check the transitions in  $\mathbf{x}^{(2)}$  on  $M$  step-by-step. This can be done in  $O(\mathrm{poly}(|\mathbf{x}|))$  total time. We conclude that the  $\tilde{p}$  derived from  $L$  is EC.
21 It suffices to ensure that  $(1 + 2^{k - 1}) / (1 + \frac{2\epsilon}{7}) > \lambda^2$ , so take any  $k > 1 + \log_2(\lambda^2 \cdot (1 + \frac{2\epsilon}{7}) - 1)$ .
Now,  $Z(\mathbf{x}^{(1)}) > 0$  iff  $M$  halts on the empty input. But this undecidable problem could be decided if there were an ELN weighted language that had support  $L$ , since then  $Z(\mathbf{x}^{(1)}) / Z$  could be found as a product of local conditional probabilities,  $\prod_{t=1}^{| \mathbf{x}^{(1)} |} p(x_t^{(1)} \mid \mathbf{x}_{<t}^{(1)})$ , that could each be computed by a Turing machine. Therefore  $\tilde{p}$  is not ELN.
We have shown above that a certain unweighted language  $L$  is not the support of any ELN distribution. We conjecture that it is also not the support of any ELNCP distribution;22 a proof of this would strengthen Theorem 5 to become an unconditional version of Theorem 1. However, ELNCP weighted languages do have more power than ELN weighted languages, as we now show.
Theorem 6. The set  $\{\tilde{p}:\tilde{p}$  is normalizable,  $\tilde{p}\in$  EC,  $\tilde{p}\in$  ELNCP,  $\tilde{p}\notin$  ELN} is not empty.
Theorem 6 justifies why this region is drawn as non-empty in Figure 2. Again, it does not rely on the assumption NP  $\nsubseteq$  P/poly or  $\mathrm{P}\neq \mathrm{NP}$ . Note that Theorem 5 can be regarded as a corollary of Theorem 6.