SlowGuess/NAACL_2021_OCR huggingface dataset:
$^{15}$ WLOG,  $\mu$  can be required to run in linear time  $O(|\mathbf{z}|)$ , as it does in our constructions below.
via  $\mu (\mathbf{x}\# \mathbf{y}) = \mathbf{x}$  can be expressed more simply as an ELNCP language. Thus, by Theorem 2, marginalizations of such distributions cannot have arbitrary NP languages as support. Our proofs of Theorems 7 and 8 instead use latent strings of the form  $\mathbf{y}\# \mathbf{x}$ , where all latent variables precede all observed ones (as in Kingma and Welling, 2014). (This simple design can always be used without loss of generality.) Trying to reorder those latent strings as  $\mathbf{x}\# \mathbf{y}$  while preserving their weights would have yielded a non-ELNCP distribution  $p(\mathbf{x}\# \mathbf{y})$  (because if it were ELNCP, then  $p(\mathbf{x})$  would be ELNCP also, and we know from Lemma 1 that it cannot be for any distribution whose support is an NP-complete language).
How about lightly marginalizing ECCP languages instead of ELNCP ones? This cannot model any additional unweighted languages, by Theorem 8. But it may be able to model more probability distributions. One can easily construct a light marginalization  $p$  of an ECCP distribution such that  $\# (\phi) = c_n\cdot p(\phi)$ , where  $\# (\phi)$  is the number of satisfying assignments of  $\phi$  and the constant  $c_{n}$  depends only on  $n = |\phi |$ . We conjecture that this is not possible with lightly marginalized ELNCP distributions.
4.3 Lookup models
$\S 2.3$  noted that with exponential growth in stored parameters, it is possible to fit any weighted language up to length  $n$ , with local probabilities computed in only  $O(n)$  time by lookup. Of course this rapidly becomes impractical as  $n$  increases, even if the amount of training data increases accordingly. However, there has been some recent movement toward storage-heavy models. Such models are typically semiparametric: they use a parametric neural model, such as an autoregressive model, together with an external knowledge base of text strings or factoids that are not memorized in the layer weights. The neural model generates queries against the knowledge base and combines their results. Examples include kNNLMs (Khandelwal et al., 2020) and semiparametric LMs (Yogatama et al., 2021). The knowledge base grows linearly with the training data rather than compressing the data into a smaller parameter vector. It is in fact a copy of the training data, indexed to allow fast lookup (Indyk and Motwani, 1998). (Preparing the index is much cheaper than neural network training.) Access to the large knowledge base may reduce the amount of computation needed to find the local conditional probabilities, much as in the trie construction of  $\S 2.3$ .
Chen et al. (2018) show that it is hard to map RNN parameters to properties of the resulting autoregressive weighted language, such as consistency ( $Z = 1$ ). We focus on cases where the RNN parameters are already known to be consistent, so the RNN efficiently maps a string  $\hat{\mathbf{x}}$  to its local conditional distribution  $p(\cdot|\hat{\mathbf{x}})$ . Our point is that for some weighted languages, this is not possible (even allowing polynomially larger RNNs for longer strings), so consistent RNNs and their ilk cannot be used to describe such languages.
In a Bayes network—which is really just an autoregressive model of fixed-length strings—approximate marginal inference is NP-hard (Roth, 1996). Assuming NP  $\nsubseteq$  P/poly and the grid-minor hypothesis, Chandrasekaran et al. (2008, Theorem 5.6) further showed that for any infinite sequence of graphs  $G_{1}, G_{2}, \ldots$  where  $G_{n}$  has treewidth  $n$ , there is no sequence of algorithms  $M_{1}, M_{2}, \ldots$  such that  $M_{n}$  performs approximate marginal inference in time  $O(\mathrm{poly}(n))$  on graphical models of structure  $G_{n}$ . This remarkable negative result says that in any graph sequence of unbounded treewidth, approximating the normalizing constant for  $G_{n}$  given arbitrary parameters is hard (not  $O(\mathrm{poly}(n)))$ , even with advice strings. Our negative result (Theorem 4) focuses on one particular infinite weighted language, showing that approximating local conditional probabilities given an arbitrary length- $n$  prefix is hard in the same way. (So this language cannot be captured by an RNN, even with advice strings.)
Autoregressive models are suited to those probability distributions whose prefix probabilities are efficiently computable. This efficiency is convenient for training and sampling. But unless we sacrifice it and allow runtime or parameter size to grow superpolynomially in input length, autoregressive models are less expressive than models whose prefix probabilities expensively marginalize over suffixes or latent variables.
All model families we have discussed in this paper can be seen as making compromises between different desiderata (Table 1). Natural follow-up questions include 'Are there model families that win on all fronts?' 'What are other modeling desiderata?'
While some languages  $\in \mathbb{P}$  cannot be supports of ELNCPs, we do not know if the same can be said for most languages  $\in \mathbb{P}$ . This problem seems to be closely related to the average complexity of NP-complete languages, where most questions remain open (Levin, 1986; Bogdanov and Trevisan, 2006).
We thank the anonymous reviewers for their comments. We also thank our colleagues at Johns Hopkins University, Facebook, and Carnegie Mellon University for their comments on earlier versions of the manuscript. This material is based upon work at Johns Hopkins University supported by the National Science Foundation under Grant No. 1718846. It does not represent the views of Microsoft (where Dr. Eisner is also a paid employee, in an arrangement that has been reviewed and approved by the Johns Hopkins University in accordance with its conflict of interest policies).
Brandon Amos and J. Zico Kolter. 2017. OptNet: Differentiable optimization as a layer in neural networks. In ICML.
Sanjeev Arora and Boaz Barak. 2009. Computational Complexity: a Modern Approach. Cambridge University Press.
Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc'Aurelio Ranzato, and Arthur Szlam. 2021. Residual energy-based models for text generation. JMLR, 22(40):1-41.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAccT.
blixt. 2020. Re: Teaching gpt-3 to identify nonsense. https://news.ycombinator.com/item?id=23990902. Online (accessed Oct 23, 2020).
Andrej Bogdanov and Luca Trevisan. 2006. Average-case complexity. Foundations and Trends in Theoretical Computer Science, 2(1):1-106.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Jan Buys and Phil Blunsom. 2018. Neural syntactic generative models with exact marginalization. In NAACL.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In ACL.
Venkat Chandrasekaran, Nathan Srebro, and Prahladh Harsha. 2008. Complexity of inference in graphical models. In UAI.
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. Recurrent neural networks as weighted language recognizers. In NAACL.
Zhiyi Chi and Stuart Geman. 1998. Estimation of probabilistic context-free grammars. Computational Linguistics, 24(2):299-305.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. ArXiv, abs/1904.10509.
Stephen A. Cook. 1971. The complexity of theorem-proving procedures. In STOC.
John DeNero and D. Klein. 2008. The complexity of phrase alignment problems. In ACL.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and C. Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. In NeurIPS.
A. Graves. 2016. Adaptive computation time for recurrent neural networks. ArXiv, abs/1603.08983.
Michael U. Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In *AIS-TATS*.
Michael U. Gutmann and Aapo Hyvarinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. JMLR, 13(11):307-361.
Y. Huang, A. Sethy, K. Audhkhasi, and B. Ramabhadran. 2018. Whole sentence neural language models. In ICASSP.
Aapo Hyvarinen. 2005. Estimation of non-normalized statistical models by score matching. JMLR, 6(24):695-709.
W. Idsardi. 2006. A simple proof that optimality theory is computationally intractable. Linguistic Inquiry, 37:271-275.
Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: Towards removing the curse of dimensionality. In STOC.
Richard M. Karp and Richard J. Lipton. 1980. Some connections between nonuniform and uniform complexity classes. In STOC.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In ICLR.
Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational Bayes. In ICLR.
Richard E. Ladner. 1975. The circuit value problem is log space complete for P. SIGACT News, 7(1):18-20.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc'Aurelio Ranzato, and Fu-Jie Huang. 2006. A tutorial on energy-based learning. In Predicting Structured Data. MIT Press.
Leonid A. Levin. 1986. Average case complete problems. SIAM Journal on Computing, 15:285-286.
Chu-Cheng Lin, Hao Zhu, Matthew R. Gormley, and Jason Eisner. 2019. Neural finite-state transducers: Beyond rational relations. In *NAACL*.
Zhuang Ma and Michael Collins. 2018. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In EMNLP.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. ArXiv, abs/1609.07843.
Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukas Burget, and Jan Honza Cernocky. 2011. RNNLM—Recurrent neural network language modeling toolkit. In IEEE Automatic Speech Recognition and Understanding Workshop.
Randall Munroe. 2009. My Hobby: Embedding NPComplete Problems in Restaurant Orders. Online (accessed May 29, 2020).
Alexei G. Myasnikov and Alexander N. Rybalov. 2008. Generic complexity of undecidable problems. The Journal of Symbolic Logic, 73(2):656-673.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. ArXiv, abs/1609.03499.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In NAACL.
Ronald Rosenfeld, Stanley Chen, and Xiaojin Zhu. 2001. Whole-sentence exponential language models: A vehicle for linguistic-statistical integration. Computer Speech & Language, 15(1):55-73.
Dan Roth. 1996. On the hardness of approximate reasoning. Artificial Intelligence, 82(1-2):273-302.
Ben Sandbank. 2008. Refining generative language models using discriminative learning. In EMNLP.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In EMNLP.
Hava T. Siegelmann and Eduardo D. Sontag. 1992. On the computational power of neural nets. In  $COLT$ .
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS.
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. 2016. Neural autoregressive distribution estimation. JMLR, 17(1):7184-7220.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Po-Wei Wang, P. Donti, B. Wilder, and J. Z. Kolter. 2019b. SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In ICML.
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020. Consistency of a recurrent language model with respect to incomplete decoding. In EMNLP.
Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In EMNLP.
Yelp. Yelp open dataset. https://www.yelp.com/dataset.
Dani Yogatama, Cyprien de Masson d'Autume, and Lingpeng Kong. 2021. Adaptive semiparametric language models. ArXiv, abs/2102.02557.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.
Figure 2: The space of unweighted languages. We assume in this diagram that NP  $\nsubseteq$  P/poly. Each rectangular outline corresponds to a complexity class (named in its lower right corner) and encloses the languages whose decision problems fall into that class. Each bold-italic label (colored to match its shape outline) names a model family and encloses the languages that can be expressed as the support of some weighted language in that family. All induced partitions in the figure are non-empty sets: shape A properly encloses shape B if and only if language class A is a strict superset of language class B. As mentioned in Table 1, standard autoregressive models (ELN models) have support languages that form a strict subset of P (Lemmas 1 and 2, Theorem 5, and §2.4). ELNCP models (§3.1) extend ELN models by allowing the parameter size to grow polynomially in string length, allowing them to capture both more languages inside P (Theorem 6) and languages outside P (including undecidable but sparse languages) that can be characterized autoregressively with the help of these compact parameters. All of those languages belong in the class P/poly. Theorem 2 establishes that energy-based (EC) and ECCP models go strictly further than ELN and ELNCP models, respectively (Theorem 2): they correspond to the entire classes P and P/poly (Lemma 1). However, even ECCP does not capture any NP-complete languages under our assumption NP  $\nsubseteq$  P/poly. Allowing a polynomial number of latent symbols extends the power further still: lightly marginalized ELNCP or ECCP distributions cover exactly the languages  $\in$  NP/poly (Theorem 8). Finally, if we were to drop the requirement that the parameters  $\Theta$  must be compact, we could store lookup tries to model any weighted language (§4.3).
Lemma 1. For any  $L \in \mathrm{P}$ , there exists an EC weighted language with support  $L$ . For any  $L \in \mathrm{P} / \mathrm{poly}$ , there exists an ECCP language with support  $L$ . But for any  $L \in \mathrm{NP}$ -complete, there exists no ECCP language with support  $L$  (assuming NP  $\nsubseteq$  P/poly).
This simple lemma relates our classes EC and ECCP of weighted languages to the complexity classes P and P/poly of their supports, which are unweighted formal languages (§2). It holds because computing a string's weight can be made as easy as determining whether that weight is nonzero (if we set the weights in a simple way), but is certainly no easier. We spell out the trivial proof to help the reader gain familiarity with the formalism.
Proof. Given  $L$ , define a weighted language  $\tilde{p}$  with support  $L$  by  $\tilde{p}(\mathbf{x}) = 1$  if  $\mathbf{x} \in L$  and  $\tilde{p}(\mathbf{x}) = 0$  otherwise.
If  $L \in P$ , then clearly  $\tilde{p}$  is EC since the return value of 1 or 0 can be determined in polytime.
If  $L \in \mathbb{P} / \mathrm{poly}$ ,  $L$  can be described as a tuple  $(M, \Theta)$  following our characterization in §2.4. It is easy to show that  $\tilde{p}$  is ECCP, using the same polynomially-sized advice strings  $\Theta$ . We simply construct  $M^{\tilde{\mathfrak{p}}}$  such that  $M^{\tilde{\mathfrak{p}}}(\theta_n)$  returns 1 or 0 on input  $\mathbf{x}$  according to whether  $M(\theta_n)$  accepts or rejects  $\mathbf{x}$ . Both  $M^{\tilde{\mathfrak{p}}}(\theta_n)$  and  $M^{\tilde{\mathfrak{p}}}(\theta_n)(\mathbf{x})$  are computed in time  $O(\mathrm{poly}(n))$  if  $|\mathbf{x}| = n$ . (The technical construction is that  $M^{\tilde{\mathfrak{p}}}$  simulates the operation of  $M$  on the input  $\theta_n$  to obtain the description of the Turing machine  $M_n = M(\theta_n)$ , and then outputs a slightly modified version of this description that will write 1 or 0 on an output tape.)
For the second half of the lemma, we use the reverse construction. Suppose  $\tilde{p}$  is an ECCP weighted language with support  $L$ .  $\tilde{p}$  can be characterized by a tuple  $(M^{\tilde{\mathbf{P}}},\Theta)$ . It is easy to show that  $L\in \mathrm{P} / \mathrm{poly}$ , using the same polynomially-sized advice strings  $\Theta$ . We simply construct  $M$  such that  $M(\theta_{n})$  accepts  $\mathbf{x}$  iff  $M^{\tilde{\mathbf{P}}}(\theta_n)(\mathbf{x}) > 0$ . Then by the assumption,  $L\notin \mathrm{NP}$ -complete.
Proof. Let  $\tilde{p}$  be an ELNCP language. This implies that  $\tilde{p}$  is normalizable, so let  $p(\mathbf{x})\triangleq \tilde{p} (\mathbf{x}) / Z$  as usual. Specifically, let  $M^{\mathrm{q}}$  efficiently locally normalize  $\tilde{p}$  with compact parameters  $\Theta^{\mathrm{q}} = \{\theta_n^{\mathrm{q}}\mid n\in \mathbb{N}\}$ . It is simple to define a Turing machine
$M^{\mathrm{r}}$  that maps each parameter string  $\theta_{n}^{\mathrm{q}}$  to a Turing machine  $r_n$ , where  $r_n(\mathbf{x})$  simply computes  $\left(\prod_{t=1}^{n} q_n(x_t \mid \mathbf{x}_{<t})\right) \cdot q_n(\mathbb{S} \mid \mathbf{x})$ . Then for all  $\mathbf{x}$  of length  $n$ ,  $r_n(\mathbf{x}) = \left(\prod_{t=1}^{n} p(x_t \mid \mathbf{x}_{<t})\right) \cdot p(\mathbb{S} \mid \mathbf{x})$ , by the definition of local normalization, and thus  $r_n(\mathbf{x}) = p(\mathbf{x})$ .
$M^{\mathrm{r}}$  can be constructed by incorporating the definition of  $M^{\mathrm{q}}$ , so that  $r_n = M^{\mathrm{r}}(\pmb{\theta}_n^{\mathrm{q}})$  can include  $q_{n} = M^{\mathrm{q}}(\pmb{\theta}_{n}^{\mathrm{q}})$  as a subroutine. This allows  $r_n$  to query  $q_{n}$  for local conditional probabilities and multiply them together.
- Since  $M^{\mathrm{q}}$  runs in polytime, it is straightforward for this construction to ensure that  $M^{\mathrm{r}}$  runs in polytime as well.
- Since  $q_{n}(\cdot \mid \hat{\mathbf{x}}) \in O(\mathrm{poly}(n))$ , this construction can ensure that  $r_{n}$  runs in polytime as well.
- We were given that  $|\theta_n^{\mathrm{q}}| \in O(\mathrm{poly}(n))$  (compact parameters).
Since  $p$  is the weighted language defined by  $(M^{\mathrm{r}},\Theta^{\mathrm{q}})$  and  $M^{\mathrm{r}}$  and  $\Theta^{\mathrm{q}}$  have the properties just discussed, we see that  $p$  is efficiently computable with compact parameters (ECCP). Therefore  $\tilde{p} (\mathbf{x}) = Zp(\mathbf{x})$  is also ECCP.
In the case where  $\tilde{p}$  is more strongly known to be ELN (the parameters  $\Theta^{\mathbf{q}}$  are not needed), a simplification of this argument shows that it is EC.
Proof. The proof was sketched in §3.2. Here we fill in the details.
The unweighted language  $\tilde{p}$  defined in that section is efficiently computable via the following simple algorithm that outputs  $\tilde{p}(\mathbf{x})$  given  $\mathbf{x} \in \mathbb{B}^*$ . If  $\mathbf{x}$  has a prefix that encodes a formula  $\phi$ , and the remainder of  $\mathbf{x}$  is a satisfying assignment  $\mathbf{a}$  to the variables of  $\phi$ , then return  $(\frac{1}{3})^{|\mathbf{x}| + 1}$ . Otherwise return 0. This algorithm can be made to run in polynomial time because whether an assignment satisfies a formula can be determined in polynomial time (a fact that is standardly used to establish that  $\mathrm{SAT} \in \mathrm{NP}$ ).
Given a formula  $\phi$  with variables  $A_{1},\ldots ,A_{j}$  , we define  $\phi^{\prime} = (\neg A_{1}\land \neg A_{2}\land \dots \land \neg A_{j}\land \neg A_{j + 1})\lor$ $(A_{1}\wedge \mathrm{Shift}(\phi))$  , where Shift  $(\phi)$  is a version of  $\phi$  in which  $A_{i}$  has been renamed to  $A_{i + 1}$  for all  $1\leq i\leq j$  It is obvious that  $\phi^{\prime}$  and  $p$  have the properties stated in the proof sketch. The strings in  $L$  that begin with  $\phi^{\prime}$  are precisely the strings of the form  $\phi^{\prime}{\bf a}^{\prime}$  where  $\mathbf{a}'$  is a satisfying assignment of  $\phi^{\prime}$  -which happen just when  $\mathbf{a}' = \mathbb{0}^{j + 1}$  or  $\mathbf{a}' = 1\mathbf{a}$  where  $\mathbf{a}$  is a satisfying
assignment of  $\phi$ . At least one string in  $L$  begins with  $\pmb{\phi}'$ , namely  $\pmb{\phi}'\mathfrak{O}^{j+1}$ , so  $Z(\pmb{\phi}') > 0$ . Moreover,  $Z(\pmb{\phi}'1) > 0$  iff  $\phi$  has any satisfying assignments. Therefore the local probability  $p(1|\pmb{\phi}') = Z(\pmb{\phi}'1)/Z(\pmb{\phi}')$  is defined (see §2.1), and is  $>0$  iff  $\mathrm{SAT}(\phi)$ .
Notice that the formal problem used in the proof is a version of SAT whose inputs are encoded using the same prefix-free encoding function enc that was used by our definition of  $L$  in  $\S 3.2$ . We must choose this encoding function to be concise in the sense that  $\phi \triangleq \operatorname{enc}(\phi)$  can be converted to and from the conventional encoding of  $\phi$  in polynomial time. This ensures that our version of SAT is  $\leq_{m}^{P}$ -interreducible with the conventional version and hence NP-complete. It also ensures that there is a polynomial function  $f$  such that  $|\phi'| \leq f(|\phi|)$ , as required by the proof sketch, since there is a polynomial-time function that maps  $\phi \rightarrow \phi \rightarrow \phi' \rightarrow \phi'$  and the output length of this function is bounded by its runtime. This is needed to show that our version of SAT is in P/poly.
Specifically, to show that the existence of  $(M^{\mathrm{q}},\Theta^{\mathrm{q}})$  implies  $\mathrm{SAT}\in \mathrm{P / poly}$ , we use it to construct an appropriate pair  $(M,\Theta)$  such that  $(M(\theta_{n}))( \phi) =$ $\mathrm{SAT}(\phi)$  if  $|\pmb {\phi}| = n$ . As mentioned in the proof sketch, we define  $\Theta$  by  $\pmb {\theta}_n = \pmb{\theta}_{f(n)}^{\mathrm{q}}$ , and observe that  $|\pmb {\theta}_n|\in$ $O(\mathrm{poly}(n))$  (thanks to compactness of the parameters  $\Theta^{\mathrm{q}}$  and the fact that  $f$  is polynomially bounded). Finally, define  $M(\pmb {\theta}_n)$  to be a Turing machine that maps its input  $\pmb{\phi}$  of length  $n$  to  $\pmb{\phi}'$  of length  $\leq f(n)$  then calls  $M^{\mathrm{q}}(\pmb {\theta}_n) = M^{\mathrm{q}}(\pmb{\theta}_{f(n)}^{\mathrm{q}})$  on  $\pmb{\phi}'1$  to obtain  $p(1\mid \pmb{\phi}')$ , and returns true or false according to whether  $p(1\mid \pmb{\phi}') > 0$ . Computing  $\pmb{\phi}'$  takes time polynomial in  $n$  (thanks to the properties of enc). Constructing  $M^{\mathrm{q}}(\pmb{\theta}_{f(n)})$  and calling it on  $\pmb{\phi}'$  each take time polynomial in  $n$  (thanks to the properties of  $f$  and  $M^{\mathrm{q}}$ ).