Before revision:  Our result is conditional on the widely believed hypothesis that NP } \mathrm{P/poly energy-based ) models may sometimes outperform locally normalized (autoregressive) models , as we demonstrate experimentally for language modeling .
After revision:  Our result is conditional on the widely believed hypothesis that NP } Standard autoregressive language models perform only polynomial-time computation to compute the probability of the next symbol. While this is attractive, it means they cannot model distributions whose next-symbol probability is hard to compute. Indeed, they cannot even model them well enough to solve associated easy decision problems for which an engineer might want to consult a language model. These limitations apply no matter how much computation and data are used to train the model, unless the model is given access to oracle parameters that grow superpolynomially in sequence length. Thus, simply training larger autoregressive language models is not a panacea for NLP. Alternatives include energy-based ) models may sometimes outperform locally normalized (autoregressive) models , as we demonstrate experimentally for language modeling .