SlowGuess/NAACL_2021_OCR huggingface dataset:
Proof. The weighted language  $\tilde{p}$  constructed in Theorem 5 is not necessarily ELNCP. To fix this, we modify the construction to obtain a weighted language  $\tilde{p}'$  with sparse support  $L'$ . We will again be able to show that  $\tilde{p}'$  is EC and not ELN. To show that  $\tilde{p}'$  is also ELNCP, we will rely on the sparsity of  $L'$ , meaning that prefixes  $(L') \triangleq \{\hat{\mathbf{x}}': (\exists \mathbf{x}' \in L') \hat{\mathbf{x}}' \preceq \mathbf{x}'\}$  contains at most  $O(\mathrm{poly}(n))$  strings  $\hat{\mathbf{x}}'$  of length  $\leq n + 1$ . Thus, we can use  $\Theta_{n}^{\mathrm{q}}$  to store all of those strings  $\hat{\mathbf{x}}'$  in polynomial space, along with their  $Z(\hat{\mathbf{x}}')$  values.[23] Notice that all strings  $\hat{\mathbf{x}}' \notin$  prefixes  $(L')$  have  $Z(\hat{\mathbf{x}}') = 0$ , so they need not be stored. Now for any  $\hat{\mathbf{x}}'$  of length  $\leq n$ , a Turing machine that consults  $\theta_{n}^{\mathrm{q}}$  can compute  $q(x \mid \hat{\mathbf{x}}') = Z_{\tilde{p}'}(\hat{\mathbf{x}}'x) / Z_{\tilde{p}'}(\hat{\mathbf{x}}')$  in time  $O(\mathrm{poly}(n))$  as desired, establishing that  $\tilde{p}'$  is ELNCP.
We may define  $\tilde{p}^{\prime}$  as follows. Let  $\operatorname{sparsify}(\mathbf{x})$  be a version of  $\mathbf{x}$  with many extra  $\mathfrak{o}$  symbols inserted: specifically, it inserts  $2^{t}$  copies of  $\mathfrak{o}$  immediately before the  $t^{\mathrm{th}}$  bit of  $\mathbf{x}$ , for all  $1\leq t\leq |\mathbf{x}|$ . We construct  $\tilde{p}^{\prime}$  so that  $\tilde{p}^{\prime}(\operatorname{sparsify}(\mathbf{x})) = \tilde{p} (\mathbf{x})$ . Specifically, let  $L^{\prime}\triangleq$
sparsify  $(L)$ . The inverse function sparsify  ${}^{-1}(\mathbf{x}^{\prime})$  is defined on exactly  $\mathbf{x}^{\prime}\in L^{\prime}$ , and is unique when defined. For all  $\mathbf{x}^{\prime}\in \mathbb{B}^{*}$ , let  $\tilde{p}^{\prime}(\mathbf{x}^{\prime})\triangleq \tilde{p} (\mathrm{sparsiy}^{-1}(\mathbf{x}^{\prime}))$  if  $\mathrm{sparsiy}^{-1}(\mathbf{x}^{\prime})$  is defined, and  $\tilde{p}^{\prime}(\mathbf{x}^{\prime})\triangleq 0$  otherwise. This can be computed in polytime, so  $\tilde{p}^{\prime}$  is EC. Also, its support  $L^{\prime}$  is sparse as claimed, so  $\tilde{p}^{\prime}$  is ELNCP.
Finally, we claim  $\tilde{p}^{\prime}$  is not ELN. A given deterministic Turing machine  $M$  halts on the empty input iff  $\mathrm{enc}(M)\in$  prefixes  $(L)$  iff sparsify  $(\mathrm{enc}(M))\in$  prefixes  $(L^{\prime})$  iff  $Z^{\prime}$  (sparsify  $(\mathrm{enc}(M))) > 0$  .But as in the proof of Theorem 5, this would be decidable if  $\tilde{p}^{\prime}$  were ELN as defined in §3.1, since then we would have a Turing machine to compute the local conditional probabilities  $p^\prime (\hat{x}_t\mid \hat{\mathbf{x}}_{< t})$  for  $\hat{\mathbf{x}} =$  sparsify  $(\mathrm{enc}(M))$
Proof. We will construct  $p$  such that support  $(p)$  is the NP-complete language SAT of all satisfiable boolean formulas. The idea is to construct an ELN distribution  $r$  that can autoregressively generate any assignment  $\mathbf{a}$  followed by any formula  $\phi$  that is satisfied by  $\mathbf{a}$ . Thus, if we delete the  $\mathbf{a}$  prefixes, the support consists of exactly the satisfiable formulas  $\phi$  (or more precisely, their encodings  $\phi$ ).
To be more precise, we will have support  $(r)$  be the language  $L = \{\mathbf{a}\# \pmb {\phi}|\mathbf{a}\in$ $\mathbb{B}^*$  and  $\phi$  is a formula satisfied by  $\mathbf{a}\}$ . This is defined similarly to the support language  $L$  in §3.2, but with the order of  $\pmb{\phi}$  and  $\mathbf{a}$  crucially swapped:  $r$  will now generate the "solution"  $\mathbf{a}$  before the "problem"  $\pmb{\phi}$ . The alphabet  $V$  of this language contains at least the symbols  $\{\emptyset ,1,\# \}$ , where  $\#$  is a separator symbol, and any other symbols needed to encode  $\phi$  as  $\pmb{\phi}$ . The marginalization operator  $\mu$  maps  $\mathbf{a}\# \pmb{\phi}$  to  $\pmb{\phi}$ .
Let  $j = |\mathbf{a}|$ . As in §3.2, we will require  $\phi$  to use all of the variables  $A_{1},\ldots ,A_{j}$  (and only those variables), implying that  $|\pmb {\phi}|\geq j$ . This ensures that marginalizing over the  $j + 1$  latent symbols is only light marginalization since  $j + 1 + |\pmb {\phi}|\in O(\mathrm{poly}(|\pmb {\phi}|))$ . For convenience, we will also require  $\phi$  to be a CNF formula. These requirements shrink support  $(p)$  but do not affect its NP-completeness.
The remaining challenge is to construct an autoregressive distribution  $r$  whose support is  $L$ . We can think of this distribution as describing an efficient procedure for randomly generating a string from left to right so that the procedure generates the  $t^{\text{th}}$  symbol
22We have not attempted to prove this. Our loose intuition is that the compact parameters of an ELNCP language may help it to memorize some small part of  $L$ , but the halting problem would still be undecidable when restricted to the rest of  $L$  (Myasnikov and Rybalov, 2008).
23More precisely, the first  $b$  bits of  $Z(\hat{\mathbf{x}}^{\prime})\leq 1$  may be stored in  $\Theta_{n + b}^{\mathrm{q}}$  , when ELNCP is defined as explained in our "Remark on irrationality" above.
in time  $O(\mathrm{poly}(t))$ , terminates with probability 1,[24] has positive probability of producing any string in  $L$ , and has zero probability of producing any string not in  $L$ . Below we give such a procedure.[25]
1. First, the procedure generates  $\mathbf{a}\#$  as a sequence of random symbols from  $\{\emptyset, 1, \#\}$ , making a uniform draw at each step. It stops immediately after generating  $\#$  for the first time. The string generated before  $\#$  is called  $\mathbf{a}$  and we let  $j = |\mathbf{a}|$ . For example,  $\mathbf{a} = \emptyset 10$  and  $j = 3$ .
2. Second, the procedure must generate the encoding  $\phi$  of a random CNF formula  $\phi$  that is satisfied by  $\mathbf{a}$ , such as  $(A_{2} \lor \neg A_{3} \lor \neg A_{2} \lor A_{2}) \land (\neg A_{1})$  in our example. This involves generating a random sequence of 0 or more satisfied clauses connected by  $\wedge$ . At each step, the procedure decides whether to generate a new clause or end the formula. The probability of generating a new clause is ordinarily  $1/2$ . However, this probability is 1 if the previous clauses do not yet mention all the variables  $A_{1}, \ldots, A_{j}$ .
How does it generate each satisfied clause? This involves generating a sequence of literals connected by  $\vee$ , at least one of which must be true. At each step of this subroutine, it uniformly chooses an integer  $i \in [1, j]$ , and then flips a fair coin to decide whether to add the literal  $A_i$  or  $\neg A_i$  to the current clause. If the clause is now satisfied by  $\mathbf{a}$  (i.e., at least one of the literals is true), it then flips another fair coin to decide whether to end the clause.
$r$  is ELN because there exists a Turing machine that computes from input  $\hat{\mathbf{x}} x$  — in time  $O(\mathrm{poly}(|\hat{\mathbf{x}} |))$  the probability that the next symbol generated after the prefix  $\hat{\mathbf{x}}$  would be  $x$ , under the above procedure. As discussed in footnote 7, that probability equals  $r(x\mid \hat{\mathbf{x}})$  — which is what our Turing machine is required to return—because the above procedure almost surely terminates (footnote 24), ensuring that  $r$  is a consistent probability distribution over  $V^{*}$  (that is,  $\sum_{\mathbf{x}\in V^{*}}r(\mathbf{x}) = 1$ ).
(a)  $L\in \mathrm{NP / poly}$
(b)  $L$  is the support of a light marginalization of an ELNCP distribution.
(c)  $L$  is the support of a light marginalization of an ECCP weighted language.
Proof. (b) implies (c) since any ELNCP distribution is an ECCP weighted language (Lemma 2). (c) implies (a) by Lemma A.2 below. Finally, (a) implies (b) by Lemma A.3 below.  $\square$
Lemma A.2. For any ECCP weighted language  $\tilde{r}$ , if  $\tilde{p}$  is a light marginalization of  $\tilde{r}$ , then  $\mathrm{support}(\tilde{p}) \in \mathrm{NP/poly}$ .
Notice that this lemma concerns the class NP/poly, not P/poly (see §2.4). The proof is straightforward.
Proof. Suppose  $\tilde{r}$  is ECCP via  $(M^{\tilde{\mathbf{r}}},\pmb{\theta}^{\tilde{\mathbf{r}}})$ , and  $\mu$  is the marginalization operator such that  $\tilde{p} (\mathbf{x}) = \sum_{\mathbf{z}:\mu (\mathbf{z}) = \mathbf{x}}\tilde{r} (\mathbf{z})$ . By the light marginalization assumption, there is a polynomial  $f$  such that  $|\mathbf{z}|\leq f(|\mu (\mathbf{z})|)$ .
To prove support  $(\tilde{p}) \in \mathrm{NP/poly}$ , we must show that there exists  $(M, \Theta)$  such that for all  $n \geq 0$ , a nondeterministic Turing machine  $M_n$  can be constructed as  $M(\theta_n)$  in time  $O(\mathrm{poly}(n))$ , which can in turn decide in time  $O(\mathrm{poly}(n))$  whether  $\tilde{p}(\mathbf{x}) > 0$  for any  $\mathbf{x}$  with  $|\mathbf{x}| = n$ .
Deciding  $\tilde{p} (\mathbf{x}) > 0$  means deciding whether  $(\exists \mathbf{z}\in V^{*})\mu (\mathbf{z}) = \mathbf{x}$  and  $\tilde{r} (\mathbf{z}) > 0$  .But if  $|\mathbf{x}| = n$  ,the first condition  $\mu (\mathbf{z}) = \mathbf{x}$  implies  $|\mathbf{z}|\leq f(|\mu (\mathbf{z})|) =$ $f(|\mathbf{x}|) = f(n)$  .Thus, we need  $M_{n}$  to nondeterministically check only the  $\mathbf{z}$  of length up to  $f(n)$  to see whether  $\mu (\mathbf{z}) = \mathbf{x}$  and  $\tilde{r} (\mathbf{z}) > 0$
How can  $M_{n}$  check a string  $\mathbf{z}$  of length  $m$ ? It can decide the first condition  $\mu (\mathbf{z}) = \mathbf{x}$  in time  $O(\mathrm{poly}(m))$ , since the marginalization operator  $\mu$  is a polytime function. To decide the second condition  $\tilde{r} (\mathbf{z}) > 0$ , it must construct the (deterministic) Turing machine  $M^{\mathrm{f}}(\pmb{\theta}_{m}^{\mathrm{f}})$  and then apply it to  $\mathbf{z}$  to obtain  $\tilde{r} (\mathbf{z})$ : since  $\tilde{r}$  is ECCP, both steps take time  $O(\mathrm{poly}(m)) = O(\mathrm{poly}(f(n)))\subseteq O(\mathrm{poly}(n))$  as required.
However, this means that  $M_{n} = M(\theta_{n})$  must have access to the parameter vectors  $\pmb{\theta}_{m}^{\mathrm{r}}$  for all  $m\leq f(n)$ . We therefore make  $\pmb{\theta}_{n}$  include this collection of parameter vectors. Each  $|\pmb{\theta}_{m}^{\mathrm{r}}|\in O(\mathrm{poly}(m))\subseteq O(\mathrm{poly}(n))$  since  $\tilde{r}$  is ECCP. So  $|\pmb{\theta}_n|\in O(\mathrm{poly}(n))$  as required.
Lemma A.3. For any  $L \in \mathrm{NP/poly}$ , there exists a light marginalization  $p$  of an ELNCP distribution, such that  $\text{support}(p) = L$ .
Lemma A.3 resembles Theorem 7, but it constructs distributions for all  $L \in \mathrm{NP} / \mathrm{poly}$ , not just for one
24 Phase 1 almost surely terminates after a finite number of bits. Phase 2 almost surely terminates after a finite number of clauses, and each clause almost surely terminates after a finite number of literals. "Almost surely" means "with probability 1."
25Our presentation here makes use of an infinite alphabet that includes symbols such as  $A_{i}$  and  $\neg A_{i}$  for all  $i \in \mathbb{N}_{>0}$ , as well as symbols such as  $\emptyset, 1, \wedge, \lor$ . We implicitly invoke some prefix-free encoding scheme to translate each symbol into a fixed string over the finite alphabet  $V$ .
particular  $L\in \mathrm{NPC}$  . The proof is similar but more complicated. In both cases, the goal is to demonstrate how an ELNCP distribution  $r$  can define a left-to-right stochastic string generation process such that the suffix of the generated string must be in  $L$  and can be any element of  $L$
Our string generation process in this case is inspired by rejection sampling, a widely used method for sampling from an energy-based model with support  $L$ . The standard scheme is to first sample a string  $\mathbf{x}$  from a tractable distribution  $q$  such that  $\text{support}(q) \supseteq L$ , then accept the sample with an appropriate probability, which is 0 if  $\mathbf{x} \notin L$ . The process is repeated until a sample is finally accepted. There is no guarantee that this standard scheme will terminate in polynomial time, however. Fortunately, in our setting, we are not trying to match our sampling distribution  $p$  to a given energy-based model, but simply match its support to a given language  $L$ . We make use of the polysize parameter vectors of ELNCP languages to store certain 'fallback strings' that are guaranteed to be in the desired language  $L$ . Wherever ordinary rejection sampling would reject a string and try generating another, we switch to generating a stored fallback string of an appropriate length. This scheme places all of the rejected probability mass on the small set of fallback strings (in contrast to rejection sampling, which in effect throws away this mass and renormalizes). The advantage is that it does not iterate indefinitely. At a high level,  $r$  is a distribution over strings  $\mathbf{z}$  that record traces of this generative story we describe above.
Proof. WLOG we assume  $L$  uses the alphabet  $V = \{\emptyset, 1, \# \}$ . In the case where  $L$  is finite, the result is trivial. We simply define  $r(\mathbf{x}) = \frac{1}{|L|}$  for  $\mathbf{x} \in L$  and  $r(\mathbf{x}) = 0$  otherwise. We then take  $p = r$  (a trivial marginalization). It is easy to show that  $r$  is ELN, and therefore ELNCP as desired, by constructing an appropriate Turing machine that maps  $\hat{\mathbf{x}}x$  to  $r(x \mid \hat{\mathbf{x}})$  in time  $O(|\hat{\mathbf{x}}x|)$ , for any  $\hat{\mathbf{x}}$  that is a prefix of some string in  $L$  and any  $x \in V \cup \{\mathbb{S}\}$ . The finite state table of the Turing machine includes states that correspond to all possible strings  $\hat{\mathbf{x}}x$ , with transitions arranged in a trie. It reads the input string  $\hat{\mathbf{x}}x$  from left to right to reach the state corresponding to  $\hat{\mathbf{x}}x$ . If it detects the end of the input while in that state, it writes  $r(x \mid \hat{\mathbf{x}})$  on the output tape.
Now we consider the case where  $L$  is infinite. For each  $j \in \mathbb{N}_{\geq 0}$ , let the 'fallback string'  $\mathbf{x}^{(j)}$  be some string in  $L$  of length  $\geq j$ . For definiteness, let us take it to be the shortest such string, breaking ties
lexicographically. At least one such string does exist because  $L$  is infinite, so  $\mathbf{x}^{(j)}$  is well-defined.
Also, since  $L \in \mathrm{NP/poly}(\S 2.4)$ , let  $(M, \Theta)$  be an ordered pair and  $f$  be a polynomial such that  $M_j = M(\theta_j)$  nondeterministically accepts  $\mathbf{a}$  within  $\leq f(j)$  steps iff  $\mathbf{a} \in L$ .
As in the proof of Theorem 7, we now describe a procedure for randomly generating a string  $\mathbf{z}$  from left to right.  $\mathbf{z}$  will have the form  $\mathbf{a}\# \mathbf{b}\# c\mathbf{d}$ , where  $\mathbf{d} \in L$  and the latent substring  $\mathbf{a}\# \mathbf{b}\# c$  will be removed by the marginalization operator  $\mu$ .
1. First we generate a random string  $\mathbf{a} \in \mathbb{B}^*$  followed by  $\#$ , just as in the proof of Theorem 7. Again let  $j = |\mathbf{a}|$ .
2. Next, we must consider whether  $\mathbf{a} \in L$ . We generate a random computation path  $\mathbf{b}$  of  $M_{j}$  on input  $\mathbf{a}$  until it either accepts (in which case we then generate  $\#1$  to record acceptance of  $\mathbf{a}$ ) or has run for  $f(j)$  steps without accepting (in which case we then generate  $\#0$  to record rejection).
3. In the former case  $(c = 1)$  we finish by deterministically generating  $\mathbf{d} \triangleq \mathbf{a} \in L$ . In the latter case  $(c = 0)$ ,  $\mathbf{a} \notin L$ , so we fall back and finish by deterministically generating  $\mathbf{d} \triangleq \mathbf{x}^{(j)} \in L$ .
Let  $r(\mathbf{z})$  be the probability that the above procedure generates  $\mathbf{z}$ . support(r) is then the set of strings that can be generated by the above procedure. The marginalized language  $\mu(\mathrm{support}(r))$  keeps just the d parts of those strings. It consists of all strings a that are accepted by at least one path b of  $M_{|\mathbf{a}|}$  (which are exactly the strings in  $L$ ) together with the fallback strings (which form a subset of  $L$ ). Thus,  $\mu(\mathrm{support}(r)) = L$  as desired.
We wish to show that  $r$  is ELNCP. In other words, some Turing machine  $M^{\mathrm{q}}$  efficiently locally normalizes  $r$  with compact parameters  $\Theta^{\mathrm{q}}$ , as defined in §3.1. The parameters will be used to store information about the infinite set of fallback strings.
In particular, for each  $n$ ,  $\theta_{n}^{\mathrm{q}}$  must have enough information to construct a Turing machine  $q_{n} = M^{\mathrm{q}}(\theta_{n}^{\mathrm{q}})$  such that  $q_{n}(\hat{\mathbf{z}} z)$  returns  $r(z \mid \hat{\mathbf{z}})$  for all  $z \in V \cup \{\mathbb{S}\}$  and all  $\hat{\mathbf{z}}$  with  $|\hat{\mathbf{z}}| \leq n$  and  $Z(\hat{\mathbf{z}}) > 0$ . Here  $Z(\hat{\mathbf{z}}) > 0$  means that  $\hat{\mathbf{z}}$  is a prefix of a string  $\mathbf{z} = \mathbf{a}\# \mathbf{b}\# c\mathbf{d}$  that could be generated by the above procedure. The computation  $q_{n}(\hat{\mathbf{z}} z)$  proceeds by simulating the sequence of choices in the above procedure that would be required to generate  $\hat{\mathbf{z}}$ , and then returning the probability that the procedure would generate symbol  $z$  next. That probability equals  $r(z \mid \hat{\mathbf{z}})$  as desired because the above procedure
almost surely terminates (as explained at the end of the proof of Theorem 7).
In general, the computation  $q_{n}(\hat{\mathbf{z}} z)$  may have to construct  $M_{j} = M(\theta_{j})$  and simulate it on a (for  $j = |\mathbf{a}|$ ) if  $z$  falls in the b#c portion of  $\hat{\mathbf{z}}$ , and it may have to look up a character of the fallback string  $\mathbf{x}^{(j)}$  if  $z$  falls in the d portion of  $\hat{\mathbf{z}}$  or terminates that portion with  $z =$  Fortunately  $j < n$ , and fortunately if the computation looks up the  $t^{\text{th}}$  character of  $\mathbf{x}^{(j)}$  then  $t < n$ . Thus, constructing and simulating  $M_{j}$  can be done in time  $O(\text{poly}(j)) \subseteq O(\text{poly}(n))$ , and looking up the  $t^{\text{th}}$  character of  $\mathbf{x}^{(j)}$  can be achieved with access to the first  $n$  characters of each of  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$ , which can be stored by  $\theta_{n}^{\mathrm{q}}$  in space  $O(n^{2})$ . It follows that  $M^{\mathrm{q}}$  can construct and apply  $q_{n}$  in polynomial time with access to compact parameters  $\Theta^{\mathrm{q}}$ , so  $r$  is ELNCP.
B Implementation details of REBMs
B.1 Modeling finite subsets of infinite languages
The experiments of this paper are conducted on datasets where we only observe strings that are finitely long. Given a possibly infinite language  $L$ , we use the notation  $L_{\leq T} = \{\mathbf{x} \mid \mathbf{x} \in L, |\mathbf{x}| \leq T\}$  for the subset of strings that are most  $T$  symbols long. Specific values of  $T$  for datasets used in our experiments are listed in Appendix D.1.
B.2 Design of base models  $p_0$
$p_0$  can be any distribution over  $L_{\leq T}$  provided that we can sample from it, and evaluate  $p_0(\mathbf{x}), \forall \mathbf{x} \in L_{\leq T}$ , both in  $O(\mathrm{poly}(|\mathbf{x}|))$ . In this work, we experiment with two designs of  $p_0$ : GRU- and Transformer-based locally normalized language models. GRU-based models are used in WikiText and Yelp experiments. The GRU-based  $p_0$ 's are parametrized with 2-layer GRUs with 500 hidden units, and word embeddings of dimension size 500.
As for Transformer-based  $p_0$ 's, we make use of Grover models (Zellers et al., 2019), which effectively are GPT-2 models trained on the aforementioned REALNews dataset. In this work, we experiment with the 'base' variant of public available weights, which are 12-layered Transformers, with 12 heads, and 768 hidden units.
B.3 Design of discriminators  $g_{\theta}$
We formulate  $g_{\theta}(\mathbf{x})$  as a summation of scores at positions  $1 \ldots |\mathbf{x}|$ , passed through an activation function  $f$ :
$$
g _ {\theta} (\mathbf {x}) = f \left(\sum_ {i = 1} ^ {| \mathbf {x} |} g _ {t} (\mathbf {x}; \boldsymbol {\theta})\right). \tag {1}
$$
To verify whether lower-bounding  $g_{\theta}$  would help with learning, as we discuss in §4.1, we experiment with two variants of  $f$ :
- tanh:  $f(x) = 2 \cdot \tanh(x)$
- **softplus:**  $f(x) = -\log(1 + \exp(x + s))$
The former one is bounded between  $(-2, 2)$ , while the second one has range  $(-\infty, 0)$ . The offset term  $s$  in the softplus activation function determines initial values of  $Z_{\theta}$ . In this paper we set  $s = 20$ .
The design of  $g_{t}(\mathbf{x};\boldsymbol{\theta})$  follows their base model counterparts: we use Bi-GRU discriminators for GRU base models; and bi-directional Transformer discriminators for Transformer ones. For GRUs  $g_{t}(\mathbf{x};\boldsymbol{\theta}) = \mathbf{h}_{t}\cdot x_{t}$ , For Transformers  $g_{t}(\mathbf{x};\boldsymbol{\theta}) = \sum \mathbf{h}_{t}$  where  $\mathbf{h}_t$  are the hidden states at time step  $t$ . In both cases, the discriminators have access to information of the whole sequence  $\mathbf{x}$  at any timestep: the Bi-GRU discriminators achieve this through the bi-directional RNNs, and the Transformers through the attention mechanism without directional masking.
B.4 Training procedure
As we note in §4.1, MLE-based training methods are generally not feasible for globally normalized models. We therefore opt to train our model using the ranking variant of noise contrastive estimation (NCE) (Ma and Collins, 2018), which does not require samples from  $p_0$  and has a simple form for residual LMs. Using  $p_0$  as a noise distribution, NCE training requires minimizing the following single-sequence loss, in expectation over the true distribution  $p$ :
$$
\mathcal {L} _ {\mathrm {N C E}} (\boldsymbol {\theta}, \mathbf {x}, p _ {0}, K) = - \log \frac {\frac {\tilde {p} _ {\boldsymbol {\theta}}}{p _ {0}} (\mathbf {x})}{\sum_ {k = 0} ^ {K} \frac {\tilde {p} _ {\boldsymbol {\theta}}}{p _ {0}} (\mathbf {x} ^ {(k)})}, \tag {2}
$$
where  $\mathbf{x}^{(0)}\triangleq \mathbf{x}$ $\frac{\tilde{p}_{\theta}}{p_0} (\mathbf{x})\triangleq \frac{\tilde{p}_{\theta}(\mathbf{x})}{p_0(\mathbf{x})}$  and  $\mathbf{x}^{(1)}\ldots \mathbf{x}^{(K)}\sim$ $p_0$  . Since  $\tilde{p_{\theta}} (\mathbf{x}) = p_0(\mathbf{x})\cdot \exp g_{\theta}(\mathbf{x})$  , we have  $\frac{\tilde{p_{\theta}}}{p_0} (\mathbf{x}) = \exp g_{\theta}(\mathbf{x})$  . The NCE minimization objective (2) now reduces to the simple form
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {N C E}} (\boldsymbol {\theta}, \mathbf {x}, p _ {0}, K) \\ = - g _ {\theta} (\mathbf {x}) \\ + \log \left(\exp g _ {\theta} (\mathbf {x}) + \sum_ {k = 1} ^ {K} \exp g _ {\theta} \left(\mathbf {x} ^ {(k)}\right)\right). \tag {3} \\ \end{array}
$$
26Note that since  $p_0$  does not have support over  $L$ , it has to assign  $p(\mathbb{S} \mid \mathbf{x}_{1\dots T}) = 1$ , which is generally not an issue.
Notice that minimizing the expected loss with stochastic gradient descent methods  $\mathcal{L}_{\mathrm{NCE}}$  defined in equation (3) requires only evaluating sequence probabilities under  $g_{\theta}$ , and tuning its parameters, but not the base model  $p_0$ . We only need to generate the noise samples  $\{\mathbf{x}^{(k)} \sim q \mid k \in [K]\}$  from  $p_0$ . This way we do not need to backpropagate through parameters of the base model  $p_0$ , which can speed up training considerably when  $p_0$  is backed by a huge network. In fact, the training of  $g_{\theta}$  can be completely agnostic to the design of  $p_0$ , allowing for the application of finetuning any locally normalized  $p_0$ .