SlowGuess/ACL_2020_OCR huggingface dataset:
Although end-to-end models  $f = P_{\mathrm{S2S}}(\mathbf{W}|\mathbf{X})$  predict  $\mathbf{W}$  directly from  $\mathbf{X}$ , interpolation with the unconditional  $g = P_{\mathrm{LM}}(\mathbf{W})$  remains helpful (Toshniwal et al., 2018). One explanation comes from cold and simple fusion (Sriram et al., 2018; Stahlberg et al., 2018), which further improve on shallow fusion (Section 3.1) by learning  $g(\mathbf{W})$  first. They argue  $g$  expresses fluency; fixing  $g$  early allows  $f(\mathbf{W},\mathbf{X})$  to focus its capacity on adequacy in encoding the source, and thus specializing the two models. With this perspective in mind, we compare  $\log P_{\mathrm{LM}}$  and PLL as candidates for  $\log g$ .
4.1 Relative linguistic acceptability
In this work we interpret fluency as linguistic acceptability (Chomsky, 1957); informally, the syntactic and semantic validity of a sentence according to human judgments (Schütze, 1996). Its graded form is well-proxied by neural language model scores  $(\log P_{\mathrm{LM}})$  once length and lexical frequency are accounted for (Lau et al., 2017). This can be seen in a controlled setting using minimal pairs and GPT-2 (345M) scores:
$\checkmark$  Raymond is selling this sketch. -40.0,
Raymond is selling this sketches. -45.2.
This example is from the Benchmark of Linguistic Minimal Pairs (BLiMP) (Warstadt et al., 2020), a challenge set of  $67\mathrm{k}$  pairs which isolate contrasts in syntax, morphology, and semantics (in this example, determiner-noun agreement). While its predecessor, the Corpus of Linguistic Acceptability (CoLA), has a training set and asks to label sentences as "acceptable" or not in isolation (Warstadt et al., 2019), BLiMP provides an unsupervised setting: language models are evaluated on how often they give the acceptable sentence a higher (i.e., less negative) score. This is equivalent to 2-best rescoring without sequence model scores  $(\log f = 0)$ . Since most minimal pairs only differ by a single word, the effect of length on log probabilities and PLLs (discussed in Section 4.3) is mitigated.
We compute PLLs on the sentences of each pair using cased BERT and RoBERTa, then choose the sentence with the highest score. Our results are in Table 7. Despite using less than half the data and a
Model (case)	Overall	ANA-AGRARG-STR BINDINGCTRL-D-N AGRELLIPSISFILLER GAPIRREGULARISLANDNPI	QUANTIFIERSS-V AGR	Unacc. PPPLAcc. PPPLatio
GPT-2 (345M)	82.6	99.4	83.4	77.8	83.0	96.3	86.3	81.3	94.9	71.7	74.7	74.1	88.3	-	-	-
BERT (base)	84.2*	97.0	80.0	82.3*	79.6	97.6*	89.4*	83.1*	96.5*	73.6*	84.7*	71.2	92.4*	111.2	59.2	1.88
BERT (large)	84.8*	97.2	80.7	82.0*	82.7	97.6*	86.4	84.3*	92.8	77.0*	83.4*	72.8	91.9*	128.1	63.6	2.02
RoBERTa (base)	85.4*	97.3	83.5	77.8	81.9	97.0	91.4*	90.1*	96.2*	80.7*	81.0*	69.8	91.9*	213.5	87.9	2.42
RoBERTa (large)	86.5*	97.8	84.6*	79.1*	84.1*	96.8	90.8*	88.9*	96.8*	83.4*	85.5*	70.2	91.4*	194.0	77.9	2.49
Human	88.6	97.5	90.0	87.3	83.9	92.2	85.0	86.9	97.0	84.9	88.1	86.6	90.9	-	-	-
Table 7: Unsupervised performance (forced choice accuracy) on BLiMP using log probabilities (GPT-2) or PLLs. Human scores from Warstadt et al. (2020). Values with * denote improvements over GPT-2 of ≥1% absolute.
third of the capacity, BERT (base) already outperforms the previous state of the art (GPT-2) by  $1.6\%$  absolute, increasing to  $3.9\%$  with RoBERTa (large). There are 4 of 12 categories where all four PLLs outperform log probabilities by  $\geq 1\%$  absolute (values marked by  $*$ ), and 7 where three or more PLLs outperform by this margin. Interestingly, PLLs do consistently worse on quantifiers, though all are relatively bad against the human baseline. The ratio of token-level PPPLs between unacceptable and acceptable sentences overall increases with performance, separating the two sentence sets.
RoBERTa improves by around  $10\%$  on filler-gap dependencies, island effects, and negative polarity items (NPIs), largely closing the human gap. This suggests that the difficulty of these BLiMP categories was due to  $P_{\mathrm{LM}}$  decomposing autoregressively, and not intrinsic to unsupervised language model training, as the original results may suggest (Warstadt et al., 2020). For some intuition, we include examples in Table 8. In the subject-verb agreement example, BERT sees The pamphlets and resembled those photographs when scoring have vs. has, whereas GPT-2 only sees The pamphlets, which may not be enough to counter the misleading adjacent entity Winston Churchill at scoring time.
4.2 Interpolation with direct models
We observed that  $\log g = \mathrm{PLL}(\mathbf{W})$  is not unduly affected by unconditional token frequencies; this mitigates degradation in adequacy upon interpolation with  $P_{\mathrm{S2S}}$ . Consider a two-word proper noun, e.g.,  $\mathbf{W} = \text{"San Francisco"}$ :
$$
\begin{array}{l} \log P _ {\mathrm {L M}} (\boldsymbol {W}) \\ = \log P _ {\mathrm {L M}} (\mathrm {S a n}) + \log P _ {\mathrm {L M}} (\text {F r a n c i s c o} \mid \mathrm {S a n}) \\ \ll \log P _ {\mathrm {M L M}} (\mathrm {S a n} | \text {F r a n c i s c o}) \\ + \log P _ {\mathrm {M L M}} (\text {F r a n c i s c o} \mid \text {S a n}) \\ = \operatorname {P L L} (W). \\ \end{array}
$$
It is a highly-fluent but low-probability bigram and thus gets penalized by  $\log P_{\mathrm{LM}}(W)$ . Informally,  $\mathrm{PLL}(W)$  expresses how likely each token is given other tokens (self-consistency), while  $\log P_{\mathrm{LM}}(W)$  expresses the unconditional probability of a sentence, beginning with the costly unconditional term  $P_{\mathrm{LM}}(\mathrm{San})$ . We see this in practice when we take LM to be GPT-2 (345M) and MLM to be RoBERTa (large). Substituting in the actual scores:
$$
\begin{array}{l} \log P _ {\mathrm {G P T - 2}} (\boldsymbol {W}) = - 8. 6 9 3 \\ = (- 7. 7 4 9) + (- 0. 9 4 4) \\ \ll (- 0. 0 0 6) + (- 1. 0 0 0) \\ = - 1. 0 0 6 = \mathrm {P L L} _ {\mathrm {R o B E R T a}} (\boldsymbol {W}). \\ \end{array}
$$
Both give similar probabilities  $P(\text{Francisco} \mid \text{San}) \approx e^{-1.0} \approx 37\%$ , but differ in the first summand.
We examine the interplay of this bias with our sequence models, in cases where the baseline, GPT-2, and BERT gave different top-1 hypotheses (Table 8). In our examples, GPT-2 restores fluency using common and repeated words, at the cost of adequacy:
clasping truth and  $\mapsto$  class in truth and,
Union by the Union Sivities  $\mapsto$
Union by the Union by the Union Civities.
One can view these as exacerbations of the rare word problem due to overconfident logits (Nguyen and Chiang, 2018), and of over-translation (Tu et al., 2016). Meanwhile, BERT rewards self-consistency, which lets rarer but still-fluent words with better acoustic or translation scores to persist:
clasping truth and  $\mapsto$  clasping truth in,
Union by the Union of LiberCivities,
System	Model	Output sentence
BLiMP (S-V agreement)	BERT	The pamphlets about Winston Churchill have resembled those photographs.
GPT-2	The pamphlets about Winston Churchill has resembled those photographs.
BLiMP (island)	BERT	Who does Amanda find while thinking about Lucille?
GPT-2	Who does Amanda find Lucille while thinking about?
LibriSpeech (dev-other)	Baseline	clasping truth and jail ya in the mouth of the student is that building up or tearing down class in truth and jail ya in the mouth of the student is that building up or tearing down clasping truth in jail gagging the mouth of the student is that building up or tearing down clapping truth into jail gagging the mouth of the student is that building up or tearing down
GPT-2
BERT (adapted)
Target
gl→en (test)	Source (gl)	Traballaba de asesora@científica na ACLU, a Unión polas Liberdades Civís .
Baseline	I worked on a scientific status on the ACLU, the Union by the Union Sivilies .
GPT-2	I worked on a scientific status on the ACLU, the Union by the Union by the Union Civities .
BERT	I worked on a scientific status on the ACLU, the Union by the Union by the Union of Liber Civities .
Target (en)	I was working at the ACLU as the organization 's science advisor .
Table 8: Examples of different top-1 hypotheses after ranking the minimal pairs or rescoring hypotheses from 4-best models, with differences highlighted. GPT-2 and BERT both promote fluency, but GPT-2's left-to-right biased scores appear to cause it to overweigh common word sequences at the expense of adequacy.
which preserves the  $p$  sound in the ground truth (clapping) for ASR, and promotes the more globally-fluent Union by the Union of Liber Civities. We also see the under-translation (i.e., omission) of Liber being corrected, without being discouraged by the rare sequence Liber Civities.
Given the differences between PLLs and log probabilities, we explore whether ensembling both improves performance in Appendix D. Similar to the largely-dominant results of MLMs on BLiMP over GPT-2 (Section 4.1), we find that as the MLM gets stronger, adding GPT-2 scores has negligible effect, suggesting that their roles overlap.
4.3 Numerical properties of PLL
PLL's numerical properties make it an ideal foundation for future ranking or scoring schemes. For example, given fixed  $|\pmb{W}|$  one expects  $-\log P_{\mathrm{MLM}}(\pmb{w}_t \mid \pmb{W}_{\backslash t})$  to be in the same range for all  $t$ . Meanwhile  $-\log P_{\mathrm{LM}}(\pmb{w}_t \mid \pmb{W}_{< t})$  decreases as  $t \to |\pmb{W}|$ , the rate of which was studied in recurrent language models (Takahashi and Tanaka-Ishii, 2018). We validate this with GPT-2 (Figure 3) and BERT (Figure 4). In particular, we see the outsized cost of the unconditional first unigram in Figure 3. This also explains why bi-SANLM was more robust than uni-SANLM at shorter and earlier positions (Shin et al., 2019); the difference is intrinsic to log probabilities versus PLLs, and is not due to model or data size.
Figure 4 also shows that domain adaptation (Section 3.5) affects PLL's positionwise cross-entropies. Cased BERT spikes at position 1, as it observes a lowercase word where a capitalized word is expected. All MLMs spike at the final token of an utterance, before our appended period ".". Terminal
Figure 3: Cross-entropy (natural base) of  $\mathbf{w}_t\mid W_{< t}$  versus context length  $(t - 1)$  from GPT-2 models, averaged over LibriSpeech's test utterances.
Figure 4: Cross-entropy (natural base) of  $\mathbf{w}_t\mid W_{\backslash t}$  versus  $t$  from BERT, averaged over LibriSpeech's 189 test utterances of length  $|\pmb {W}| = 19$  (including ".").
words are difficult to predict in general, but here more so as the BERT+LibriSpeech text corpora and the LibriSpeech test set are mismatched; the latter's ground-truth utterances were segmented by voice activity and not punctuation (Panayotov et al., 2015). Otherwise, the averaged cross-entropies are flat. This, plus our success on BLiMP, suggest posi
tionwise scores as a way of detecting "disfluencies" (at least, those in the form of domain mismatches) by observing spikes in cross-entropy; with  $\log P_{\mathrm{LM}}$  spikes are confounded by the curve in Figure 3.
In Appendix C, we plot sentence-level PLLs versus  $|\pmb{W}|$  and observe linearity as  $|\pmb{W}| \to \infty$ , with spikes from the last word and lowercase first word smoothing out. This behavior motivates our choice of  $\alpha = 1.0$  when applying the Google NMT-style length penalty (Wu et al., 2016) to PLLs, which corresponds to the asymptotically-linear  $\mathrm{LP}_{\mathrm{MLM}} = (5 + |\pmb{W}|) / (5 + 1)$ . In contrast, autoregressive scores like  $P_{\mathrm{LM}}(\pmb{W})$  integrate over the inverse power-law curve in Figure 3. We speculate that this explains the effectiveness of their hyperparameter  $\alpha = 0.6$ , widely used in NMT baselines like ours, as there exists  $C$  such that
$$
\mathrm {L P} _ {\mathrm {S 2 S}} (\boldsymbol {W}) = \frac {(5 + | \boldsymbol {W} |) ^ {0 . 6}}{(5 + 1) ^ {0 . 6}} \approx \int_ {0} ^ {| \boldsymbol {W} |} \frac {C}{(5 + x) ^ {0 . 4}} \mathrm {d} x.
$$
Our work extends the closest previous works (Wang and Cho, 2019; Shin et al., 2019) with regards to experiments and tasks, as outlined in Section 2.1. Furthermore, neither work considers the inference cost of masked rescoring, which we address with our maskless scoring approach, or analyze PLL's numerical properties.
Future context. Log probabilities conditioned on past and future context have been used in MT (Finch and Sumita, 2009; Xiong et al., 2011) and perennially in ASR (Shi et al., 2013; Arisoy et al., 2015; Chen et al., 2017) to positive effect. However, these are not "deep bidirectional" as they model interactions between  $W_{<t}$  and  $W_{>t}$  via the forward and backward context vectors, while MLMs model all pairwise interactions  $w_{s}$  and  $w_{s'}$  via dot-product attention (compare ELMo versus BERT). Their PLLs would have different properties from ours (e.g., their cross-entropies in Figure 4 may be convex instead of flat).
Discriminative language modeling. Previous works (Roark et al., 2004; Huang et al., 2018) have explored training language models that directly optimize for a downstream metric (WER, BLEU). While we also eschew using log probabilities from conventional LMs, our approach remains generative. Log probabilities model the joint distribution; PLL does so as well, albeit implicitly
(Appendix B). PLL's summands (conditional probabilities) remain accessible for Gibbs sampling and are not tailored to any metric. The two approaches are complementary; for example, one could use PLL as a "prior" or regularizer for scores given by discriminatively-finetuned BERT models in tasks like passage re-ranking (Nogueira and Cho, 2019).
Language model integration. Beyond finetuning pretrained LMs and MLMs, monolingual pretraining has also improved NMT performance (Ramachandran et al., 2017; Conneau and Lample, 2019). However, modular integration of language representation models remains prevalent for various pragmatic reasons, similar to fusion in ASR. Contemporary examples are the use of finetuned BERT scores in a question-answering pipeline (Nogueira and Cho, 2019), or "as-is" cosine similarity scores from BERT to evaluate generated text (Zhang et al., 2020). For example, one might have no pretrained multilingual LMs for decoder initialization or fusion, as such models are difficult to train (Ragni et al., 2016). However, one may have an M-BERT or XLM for the target language/domain. Finally,  $N$ -best rescoring and pretraining are not mutually exclusive, though pretraining may already go partway to improve fluency.
We studied scoring with MLM pseudo-log-likelihood scores in a variety of settings. We showed the effectiveness of  $N$ -best rescoring with PLLs from pretrained MLMs in modern sequence-to-sequence models, for both ASR and low- to medium-resource NMT. We found rescoring with PLLs can match or outperform comparable scores from large unidirectional language models (GPT-2). We attributed this to PLL's promotion of fluency via self-consistency, as demonstrated by improvement on unsupervised acceptability judgements and by qualitative analysis. We examined the numerical properties of PLLs, proposed maskless scoring for speed, and proposed pseudo-perplexities for intrinsic evaluation of MLMs, releasing a codebase implementing our work. Future work could find additional modular uses of MLMs, simplify maskless PLL computations, and use PLLs to devise better sentence- or document-level scoring metrics.
We thank Phillip Keung and Chris Varano for their thoughtful suggestions on this work.
Roee Aharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. In NAACL-HLT.
Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. 2015. Bidirectional recurrent neural network language models for automatic speech recognition. In ICASSP.
Julian Besag. 1975. Statistical analysis of non-lattice data. The Statistician, 24(3):179-195.
Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, R Cattoni, and Marcello Federico. 2015. The IWSLT 2015 evaluation campaign. Technical report, FBK and KIT.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP.
Xie Chen, Anton Ragni, Xunying Liu, and Mark JF Gales. 2017. Investigating bidirectional recurrent neural network language models for speech recognition. In INTERSPEECH.
Noam Chomsky. 1957. Syntactic structures. Mouton.
Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019. On the use of BERT for neural machine translation. In WNGT.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In NeurIPS.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Andrew Finch and Eiichiro Sumita. 2009. Bidirectional phrase-based statistical machine translation. In EMNLP.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535.
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, et al. 2020. GluonCV and GluonNLP: Deep learning in computer vision and natural language processing. JMLR, 21(23):1-7.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. *Deep Learning Workshop*, NeurIPS.
Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018. Large margin neural language model. In EMNLP.
Frederick Jelinek, Lalit Bahl, and Robert Mercer. 1975. Design of a linguistic statistical decoder for the recognition of continuous speech. IEEE Trans. Inf. Theory, 21(3):250-256.
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In EMNLP.
Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202-1241.
Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. 2019. Jasper: An end-to-end convolutional neural acoustic model. In INTERSPEECH.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In WMT.
Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In EMNLP.
Toan Q Nguyen and David Chiang. 2018. Improving lexical choice in neural machine translation. In NAACL-HLT.
Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. 2018. Parallel WaveNet: Fast high-fidelity speech synthesis. In ICML.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In ICASSP.
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In NAACL-HLT.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Anton Ragni, Edgar Dakin, Xie Chen, Mark JF Gales, and Kate M Knill. 2016. Multi-language neural network language models. In *INTERSPEECH*.
Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In EMNLP.
Brian Roark, Murat Saraclar, Michael Collins, and Mark Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In ACL.
Carson T Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Language Science Press.
Yangyang Shi, Martha Larson, Pascal Wiggers, and Catholijn M Jonker. 2013. Exploiting the succeeding words in recurrent neural network language models. In INTERSPEECH.
Joongbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using BERT for speech recognition. In ACML.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold fusion: Training seq2seq models together with language models. In INTER-SPEECH.
Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018. Simple fusion: Return of the language model. In WMT.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS.
Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2018. Cross entropy of neural language models at infinity—a new bound of the entropy rate. Entropy, 20(11):839.
Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. 2018. A comparison of techniques for language model integration in encoder-decoder speech recognition. In SLT.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In NeuralGen.