SlowGuess/ACL_2020_OCR huggingface dataset:
Chenguang Wang, Mu Li, and Alexander J Smola. 2019. Language models with transformers. arXiv preprint arXiv:1904.09408.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: An efficient data augmentation algorithm for neural machine translation. In EMNLP.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. TACL.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. TACL, 7:625-641.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018. ESPnet: End-to-end speech processing toolkit. In INTERSPEECH.
Thomas Wolf, L Debut, V Sanh, J Chaumont, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. HuggingFace's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Deyi Xiong, Min Zhang, and Haizhou Li. 2011. Enhancing language models in statistical machine translation with backward n-grams and mutual information triggers. In ACL.
Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation. In EMNLP.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In ICLR.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In ICLR.
A.1 Language models
Implementation. English BERT, M-BERT, GPT2, and RoBERTa models were served, adapted, and finetuned via the GluonNLP toolkit (Guo et al., 2020). German BERT and XLM models were served via HuggingFace's Transformers toolkit (Wolf et al., 2019). We release a reference implementation (a language model scoring package) for our work at https://github.com/awslabs/mlm-scoring.
Training. When adapting to a corpus we continue the training scheme for BERT, i.e., MLM + next-sentence prediction (Devlin et al., 2019), on the new dataset only, until the training loss converges. We still perform warmup at adaptation time (ratio of 0.01), but continue to use batches of 256 sequences of contiguous sentences, each with length up to 512.
Scoring. For BERT, M-BERT, and RoBERTa we preprocess and append [CLS], [SEP] tokens. For GPT-2 we preprocess and append  $<\mid$  endoftext  $\mid>$ , the default tokens for unconditional generation, as we found this outperformed other initial conditions (e.g., a preceding "."). For XLM we preprocess and append  $
$  (prepending  $
$  is more proper, but this is due to a bug in Hugging-Face Transformer's XLMTokens that we will fix; changes in results should be negligible). When computing (pseudo-)perplexity (Section 2.3), these special tokens' conditional probabilities are not included, nor are they counted for token or word counts during length normalization.
$N$ -best rescoring. We follow the log-linear model in Section 3.1 with its hyperparameter  $\lambda$ , i.e., weighted addition of (M)LM scores with sequence-to-sequence scores. When interpolating MLMs with GPT-2 there is also a hyperparameter  $\gamma$  (Appendix D). We do grid search on  $(\lambda, \gamma)$  with increments (0.05, 0.1) for the best weights on the development set for downstream WER or BLEU, then evaluate on the corresponding test set. In the case of ties, we choose the largest  $\lambda, \gamma$ .
A.2 Automatic speech recognition
We use the LibriSpeech corpus (Panayotov et al., 2015) for our experiments. To adapt BERT we use the provided 800M-word text-only data, processed
using Kaldi to match the normalized, downloadable corpus<sup>2</sup> but with sentences in their original order (instead of alphabetically as in Kaldi's recipe), to match the long-context training regime of our language models. Our LibriSpeech-only BERT (base) model was trained on this corpus using GluonNLP's recipe, for 1.5M steps.
We take pre-existing 100-best lists shared via e-mail communication (Shin et al., 2019), which were produced by ESPnet (Watanabe et al., 2018) on LibriSpeech's dev and test sets. The ESPnet model was the sequence-to-sequence BLSTMP model in the librispeech/asrl recipe, except with 5 layers and a beam size of 100.
For speech corpora, to alleviate some of the domain shift from BERT's original written corpora, we appended ". ." at the end of utterances during adaptation, and appended ". ." to all hypotheses before subword tokenization, masking, and token/word counting.
A.3 Neural machine translation
Our pretrained model<sup>3</sup> is the base Transformer on WMT 2014 English-German (Vaswani et al., 2017) trained using GluonNLP's scripts/machine Translation. Evaluation and  $N$ -best rescoring was on the 3003-sentence test set via --full --bleu 13a --beam_size 100.
We consider 5 low-resource directions from the TED Talks dataset (Qi et al., 2018): Arabic (ar), Galician (gl), and Slovak (sk) to English; and English to Arabic, German (de), languages which were considered in Aharoni et al. (2019). We also include a more popular benchmark, English to Vietnamese (vi) from the IWSLT'15 evaluation campaign $^{4}$  (Cettolo et al., 2015). These give a breadth of English-source and English-target pairs and include a right-to-left language; more importantly, the three non-English targets are covered by the 15-language XLMs (Conneau and Lample, 2019).
Our models are also described as baselines in a dedicated work (Nguyen and Salazar, 2019). They are base Transformers with 6 layers, 8 heads, an 8k BPE vocabulary, and dropout of 0.3, except for  $\mathrm{gl}\rightarrow \mathrm{en}$  where we use 4 layers, 4 heads, 3k BPE,
$^{2}$ https://www.openslr.org/resources/11/librispeech-lm-norm.txt.gz
<sup>3</sup>http://apache-mxnet.s3-accelerate.dualstack.amazon.com/glutron/models/transformer_en_de_512_WMT2014-e25287c5.zip
4https://nlp.stanford.edu/projects/nmt/
and a dropout of 0.4 due to its significantly smaller size. We use a warmup of 8k steps and the default hyperparameters (Vaswani et al., 2017). We apply GNMT length normalization (Wu et al., 2016) with  $\alpha = 0.6$  to the sequence-to-sequence log probabilities, and  $\alpha = 1.0$  to the PLLs (motivation is given in Section 4.3), with respect to their chosen tokenization's lengths. We compute tokenized BLEU via multi-bleu.perl from Moses $^5$  to compare with past works on these datasets.
B BERT as a generative model
In their published version (Wang and Cho, 2019), the authors claimed that BERT is a Markov random field language model (MRF-LM) where  $\{\mathbf{w}_t\}_{t=1}^{|W|}$  are categorical random variables (over the vocabulary) in a fully-connected graph  $\mathcal{G}$ . They define a potential over cliques of  $\mathcal{G}$  such that all partial-graph potentials are  $\exp(0) = 1$  and the full-graph potential is  $\exp \sum_{t=1}^{|W|} \log \phi_t(\mathcal{G})$ , where  $\log \phi_t(\mathcal{G})$  is the logit corresponding to  $\log P_{\mathrm{MLM}}(\boldsymbol{w}_t \mid \boldsymbol{W}_{\backslash t})$  (although in their formulation, one could include the softmax into the feature function  $f_\theta$  and take  $\log \phi_t(\mathcal{G}) = \mathrm{PLL}(\mathcal{G})$  exactly).
Abusing notation, we write  $\mathbf{W}$  interchangeably with its graph  $\mathcal{G}$ . An MRF defined this way would give the joint distribution:
$$
P _ {\mathrm {M L M}} (\boldsymbol {W}) = \frac {1}{Z} \prod_ {t = 1} ^ {| \boldsymbol {W} |} \phi_ {t} (\boldsymbol {W}) = \frac {1}{Z} \exp \operatorname {P L L} (\boldsymbol {W}),
$$
where  $Z$  is the partition function
$$
Z = \sum_ {\pmb {W} ^ {\prime} \in \mathbb {S}} \prod_ {t = 1} ^ {| \pmb {W} ^ {\prime} |} \phi_ {t} (\pmb {W} ^ {\prime}) = \sum_ {\pmb {W} ^ {\prime} \in \mathbb {S}} \exp \mathrm {P L L} (\pmb {W} ^ {\prime}),
$$
making this a valid distribution by normalizing over all sequences of the same length  $|\pmb{W}|$ , the set denoted by  $\mathbb{S}$ .
One then hopes to say that  $\log P_{\mathrm{MLM}}(\boldsymbol{w}_t \mid \boldsymbol{W}_{\backslash t})$  is the conditional distribution of this MRF. However, their erratum<sup>6</sup> notes this is not the case, as  $\mathbf{w}_t$  would be affected by other log potentials as well.
In practice, one could instead a priori make the modeling assumption
$$
g (\pmb {W}) = P _ {\mathrm {M L M}} (\pmb {W}) := \frac {1}{Z} \exp \mathrm {P L L} (\pmb {W}),
$$
as done in the work on bi-RNNLMs (Chen et al., 2017). They choose to model the distribution of sentences as a product-of-experts  $\mathbf{w}_t\mid \mathbf{W}_{\backslash t}$ , whose parameters are shared via the underlying bi-RNN.
Suppose one had access to this "normalized MLM probability". In the log-linear setting (Section 3.1), we get
$$
\begin{array}{l} \log P _ {\mathrm {S 2 S}} (\boldsymbol {W} \mid \boldsymbol {X}) + \lambda \log P _ {\mathrm {M L M}} (\boldsymbol {W}) \\ = \dots + \lambda \log \left(\frac {1}{Z} \exp \operatorname {P L L} (\boldsymbol {W})\right) \\ = \dots + \lambda \operatorname {P L L} (\boldsymbol {W}) - \lambda \log Z. \\ \end{array}
$$
For fixed  $\lambda$  and  $Z$  (which is intrinsic to the MLM), we see that  $\lambda \log Z$  does not affect rank-ordering when taking arg max to get the best hypothesis  $\hat{W}$ . Hence, the heuristic interpolation enacted by  $\lambda$  is "the same" for normalized log  $P_{\mathrm{LM}}$ , unnormalized PLL, and our hypothetical log  $P_{\mathrm{MLM}}$ . The remaining issue is whether  $\lambda$  has the same effect for all lengths  $|\mathbf{W}|$ , which one mitigates by applying the correct length penalties to  $f$  and  $g$  (Section 4.3).
C Pseudo-perplexity and rescoring
We briefly examine the relationship between PPPL (Section 2.3) and metrics post-rescoring. We plot negative PLLs versus  $|\pmb{W}|$  and observe linearity, helping justify our simple average over length:
Figure 5: Negative pseudo-log-likelihood scores versus sentence length (in tokens) from BERT, averaged over LibriSpeech's test utterances of each length.
Note that in this section, we consider PPPLs normalized by number of words  $(\mathrm{PPPL_w})$  to improve comparability between different subword vocabularies. We see a good correspondence between  $\mathrm{PPPL_w}$  improvements and post-rescoring WER in Table 9, and post-rescoring BLEU in Table 10.
Thus, one could compute a new pretrained model's word-normalized PPPL on a small target-
5https://statmt.org
6"BERT has a Mouth and must Speak, but it is not an MRF" from https://sites.google.com/site/deepernn/home/blog
Model	test
clean	other
PPPLw	WER	PPPLw	WER
BERT (base, cased)	24.18	5.41	27.47	17.41
RoBERTa (base, cased)	21.85	5.25	24.54	17.18
BERT (large, cased)	17.49	5.25	19.59	16.97
BERT (base, uncased)	17.49	5.14	19.24	16.97
RoBERTa (large, cased)	14.78	5.05	16.23	16.79
BERT (base, Libri. only)	9.86	4.79	10.55	16.50
BERT (base, unc., adapt.)	6.63	4.58	6.56	15.96
Table 9: Word-normalized PPPL vs. WER on LibriSpeech after rescoring, for models with different token vocabularies. WERs are from Table 2 and Table 5.
Model	dev
ar→en	gl→en	sk→en
PPPLw	BLEU	PPPLw	BLEU	PPPLw	BLEU
B-base	13.08	35.71	11.86	20.25	13.20	29.74
B-large	10.17	35.79	9.48	20.21	10.43	29.79
R-base	9.77	35.86	9.36	20.21	9.75	29.79
R-large	6.26	36.02	6.08	20.44	6.29	30.05
Table 10: Word-normalized PPPL vs. BLEU of cased BERT (B) and RoBERTa (R) on English gold sentences in the TED Talks corpus.
domain sample to quickly assess whether rescoring with it could improve on the previous model.
D Combining MLMs and GPT-2
We ask whether scores from a unidirectional LM are complementary with a masked LM for rescoring. When interpolating, we introduce  $\gamma$  such that:
$$
\log g (\boldsymbol {W}) = (1 - \gamma) \log P _ {\mathrm {L M}} (\boldsymbol {W}) + \gamma \operatorname {P L L} (\boldsymbol {W}).
$$
Our results are in Table 11:
Model	test	+ GPT-2
clean	other	clean	other
baseline (100-best)	7.26	20.37	5.30	17.26
BERT (large, cased)	5.25	16.97	5.03	16.80
RoBERTa (large, cased)	5.05	16.79	4.93	16.71
BERT (base, unc., adapt.)	4.58	15.96	4.50	15.92
Table 11: WERs on LibriSpeech after hypothesis rescoring, with and without interpolating with GPT-2 (345M, cased).
As the MLM gets stronger, the improvement from adding scores from GPT-2 goes to zero, suggesting that their roles overlap at the limit. However, unlike recent work (Shin et al., 2019) but like previous work (Chen et al., 2017), we found that interpolating with a unidirectional LM remained optimal, though our models are trained on different datasets and may have an ensembling effect.