SlowGuess/ACL_2020_OCR huggingface dataset:
Julian Salazar\* Davis Liang\* Toan Q. Nguyen\*\* Katrin Kirchhoff\*
$\spadesuit$  Amazon AWS AI, USA
$\diamond$  University of Notre Dame, USA
{julsal,liadavis,katrinki}@amazon.com，tnguye28@nd.edu
Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end-to-end LibriSpeech model's WER by  $30\%$  relative and adds up to  $+1.7$  BLEU on state-of-the-art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL's unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling computation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PP-PLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore translations in multiple languages. We release our library for language model scoring at https://github.com/awslabs/mlm-scoring.
BERT (Devlin et al., 2019) and its improvements to natural language understanding have spurred a rapid succession of contextual language representations (Yang et al., 2019; Liu et al., 2019; inter alia) which use larger datasets and more involved training schemes. Their success is attributed to their use of bidirectional context, often via their masked language model (MLM) objectives. Here, a token  $\boldsymbol{w}_t$  is replaced with [MASK] and predicted using all past and future tokens  $\boldsymbol{W}_{\backslash t} := (\boldsymbol{w}_1, \dots, \boldsymbol{w}_{t-1}, \boldsymbol{w}_{t+1}, \dots, \boldsymbol{w}_{|\boldsymbol{W}|})$ .
Figure 1: To score a sentence, one creates copies with each token masked out. The log probability for each missing token is summed over copies to give the pseudo-log-likelihood score (PLL). One can adapt to the target domain to improve performance, or finetune to score without masks to improve memory usage.
In contrast, conventional language models (LMs) predict  $\boldsymbol{w}_t$  using only past tokens  $\boldsymbol{W}_{< t} := (\boldsymbol{w}_1, \dots, \boldsymbol{w}_{t-1})$ . However, this allows LMs to estimate log probabilities for a sentence  $\boldsymbol{W}$  via the chain rule  $(\log P_{\mathrm{LM}}(\boldsymbol{W}) = \sum_{t=1}^{|W|} \log P_{\mathrm{LM}}(\boldsymbol{w}_t | \boldsymbol{W}_{< t}))$ , which can be used out of the box to rescore hypotheses in end-to-end speech recognition and machine translation (Chan et al., 2016; Gulcehre et al., 2015), and to evaluate sentences for linguistic acceptability (Lau et al., 2017).
Our work studies the corresponding pseudo-log-likelihood scores (PLLs) from MLMs (Wang and Cho, 2019), given by summing the conditional log probabilities  $\log P_{\mathrm{MLM}}(\boldsymbol{w}_t \mid \boldsymbol{W}_{\backslash t})$  of each sentence token (Shin et al., 2019). These are induced in BERT by replacing  $\boldsymbol{w}_t$  with [MASK] (Figure 1).
*Work done during an internship at Amazon AWS AI.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Let  $\Theta$  denote our model's parameters. Our score is
$$
\operatorname {P L L} (\boldsymbol {W}) := \sum_ {t = 1} ^ {| \boldsymbol {W} |} \log P _ {\mathrm {M L M}} (\boldsymbol {w} _ {t} \mid \boldsymbol {W} _ {\backslash t}; \Theta).
$$
PLLs and their corresponding pseudo-perplexities (PPPLs) (Section 2.3) are intrinsic values one can assign to sentences and corpora, allowing us to use MLMs in applications previously restricted to conventional LM scores. Furthermore, we show that one can finetune BERT to compute PLLs in a single, non-recurrent inference pass (Section 2.2).
Existing uses of pretrained MLMs in sequence-to-sequence models for automatic speech recognition (ASR) or neural machine translation (NMT) involve integrating their weights (Clinchant et al., 2019) or representations (Zhu et al., 2020) into the encoder and/or decoder during training. In contrast, we train a sequence model independently, then rescore its  $n$ -best outputs with an existing MLM. For acceptability judgments, one finetunes MLMs for classification using a training set (Warstadt et al., 2019; Devlin et al., 2019); instead, PLLs give unsupervised, relative judgements directly.
In Section 3, we show that scores from BERT compete with or even outperform GPT-2 (Radford et al., 2019), a conventional language model of similar size but trained on more data. Gains scale with dataset and model size: RoBERTa large (Liu et al., 2019) improves an end-to-end ASR model with relativeWER reductions of  $30\%$ $18\%$  on LibriSpeech test-clean, test-other respectively (with further gains from domain adaptation), and improves state-of-the-art NMT baselines by up to  $+1.7$  BLEU on low-resource pairs from standard TED Talks corpora. In the multilingual case, we find that the pretrained 15-language XLM (Conneau and Lample, 2019) can concurrently improve NMT systems in different target languages.
In Section 4, we analyze PLLs and propose them as a basis for other ranking/scoring schemes. Unlike log probabilities, PLL's summands are more uniform across an utterance's length (no left-to-right bias), helping differentiate fluency from likeliness. We use PLLs to perform unsupervised acceptability judgments on the BLiMP minimal pairs set (Warstadt et al., 2020); BERT and RoBERTa models improve the state of the art (GPT-2 probabilities) by up to  $3.9\%$  absolute, with  $+10\%$  on island effects and NPI licensing phenomena. Hence, PLLs can be used to assess the linguistic competence of MLMs in a supervision-free manner.
2.1 Pseudolikelihood estimation
Bidirectional contextual representations like BERT come at the expense of being "true" language models  $P_{\mathrm{LM}}(\mathbf{W})$ , as there may appear no way to generate text (sampling) or produce sentence probabilities (density estimation) from these models. This handicapped their use in generative tasks, where they at best served to bootstrap encoder-decoder models (Clinchant et al., 2019; Zhu et al., 2020) or unidirectional LMs (Wang et al., 2019).
However, BERT's MLM objective can be viewed as stochastic maximum pseudolikelihood estimation (MPLE) (Wang and Cho, 2019; Besag, 1975) on a training set  $\mathbb{W}$ , where  $\{\mathbf{w}_t\}_{t=1}^{|W|}$  are random variables in a fully-connected graph. This approximates conventional MLE, with MLM training asymptotically maximizing the objective:
$$
\mathcal {J} _ {\mathrm {P L}} (\Theta ; \mathbb {W}) = \frac {1}{| \mathbb {W} |} \sum_ {\boldsymbol {W} \in \mathbb {W}} \operatorname {P L L} (\boldsymbol {W}; \Theta).
$$
In this way, MLMs learn an underlying joint distribution whose conditional distributions  $\mathbf{w}_t \mid \mathbf{W}_{\backslash t}$  are modeled by masking at position  $t$ . We include a further discussion in Appendix B.
This enabled text generation with BERT via Gibbs sampling, leading to the proposal (but not evaluation) of a related quantity, the sum of logits, for sentence ranking (Wang and Cho, 2019). More recent work (Shin et al., 2019) extended past research on future-conditional LMs in ASR (Section 5) with deeply-bidirectional self-attentive language models (bi-SANLMs). They trained shallow models from scratch with the [MASK] scoring method, but did not relate their work to pseudolikelihood and fluency, which provide a framework to explain their success and observed behaviors.
Experimentally, we extend both works by evaluating pretrained models, domain adaptation, and usage in NMT and multilingual settings (Section 3), along with acceptability judgements and PLL's intrinsic numerical properties (Section 4).
2.2 [MASK] less scoring
A practical point unaddressed in both works is that computing PLLs from an MLM requires a sentence copy for each position, making the number of inference passes dependent on length (though these can be parallelized). The cost of a softmax is also incurred, which is dependent on vocabulary size
$V$ ; together this gives  $\mathcal{O}(|W| \cdot V)$ . We propose reducing this to  $\mathcal{O}(1)$  by training a network  $q$  with parameters  $\Theta_S$  to match BERT's PLLs without [MASK] tokens:
$$
| \mathrm {P L L} (\pmb {W}) - q (\pmb {W}; \Theta_ {S}) | ^ {2}.
$$
We propose finetuning  $q$  from the pretrained MLM directly (i.e., initializing  $\Theta_S$  with  $\Theta$ ), via regression over the [CLS] token (Figure 2):
Figure 2: We learn a linear map after the [CLS] token, supervised by the PLLs from the pretrained MLM.
More generally, one could use any student model  $q$ , as in knowledge distillation (Hinton et al., 2014). Here, the teacher gives individual token probabilities (  $|\mathbf{W}|$  inference passes) while the student approximates their sum (one inference pass). This is reminiscent of distilling an autoregressive teacher to a parallel student, as in the case of WaveNet (Oord et al., 2018). Other [MASK] less bidirectional models like XLNet (Yang et al., 2019) can also give PLLs; we leave this to future work.
2.3 Pseudo-perplexity
Analogous to conventional LMs, we propose the pseudo-perplexity (PPPL) of an MLM as an intrinsic measure of how well it models a corpus of sentences  $\mathbb{W}$ . Let  $N$  denote the number of tokens in the corpus. Then a model's PPPL on  $\mathbb{W}$  is
$$
\mathrm {P P P L} (\mathbb {W}) := \exp \left(- \frac {1}{N} \sum_ {\boldsymbol {W} \in \mathbb {W}} \mathrm {P L L} (\boldsymbol {W})\right).
$$
Past work (Chen et al., 2017) also computed this quantity with bi-RNNLMs for ASR, although such models are not deeply bidirectional like self-attentive MLMs (see Section 5).
These PPPLs can be used in lieu of perplexities. For example, during domain adaptation, one
can perform early stopping with respect to development PPPL. This is in contrast to MLM accuracy, which is not a continuous loss and is often stochastic (e.g., when performing dynamic masking as in RoBERTa). In Section 4.1, we see that PPPLs naturally separate out sets of acceptable and unacceptable sentences.
Unlike previous works (Chen et al., 2017; Shin et al., 2019) we use pretrained BERTs, which are open-vocabulary (subword) bidirectional LMs. However, PPPLs are only comparable under the same subword vocabulary, which differs between e.g., BERT and RoBERTa. Normalizing with  $N$  as the number of words mitigates this. In Appendix C, we show that word-normalized PPPLs correlate with domain adaptation, and with downstream metrics like ASR and BLEU after rescoring.
3 Sequence-to-sequence rescoring
Let  $\mathbf{X}$  denote audio features or source text tokens, and let  $\mathbf{W} = (\mathbf{w}_1, \dots, \mathbf{w}_{|\mathbf{W}|})$  denote target text tokens. For non-end-to-end ASR and MT systems, having separate acoustic/translation models  $P_{\mathrm{AM / TM}}(\mathbf{X} \mid \mathbf{W})$  and language models  $P_{\mathrm{LM}}(\mathbf{W})$  is motivated by the Bayes rule decomposition used to select the best hypothesis  $\hat{\mathbf{W}}$  (Jelinek et al., 1975; Brown et al., 1993):
$$
\begin{array}{l} \hat {\boldsymbol {W}} = \underset {\boldsymbol {W}} {\arg \max } [ P (\boldsymbol {W} \mid \boldsymbol {X}) ] \\ = \underset {\mathbf {W}} {\arg \max } [ P _ {\mathrm {A M / T M}} (\boldsymbol {X} \mid \boldsymbol {W}) P _ {\mathrm {L M}} (\boldsymbol {W}) ]. \\ \end{array}
$$
3.1 The log-linear model
End-to-end ASR and NMT use encoder-decoder architectures that are trained discriminatively. Though less principled, many still adopt a log-linear model
$$
\begin{array}{l} \hat {\boldsymbol {W}} = \underset {\boldsymbol {W}} {\arg \max } \left[ \log P (\boldsymbol {W} \mid \boldsymbol {X}) \right] \\ \approx \underset {\boldsymbol {W}} {\arg \max } \left[ \log f (\boldsymbol {W}, \boldsymbol {X}) + \lambda \log g (\boldsymbol {W}) \right] \\ \end{array}
$$
with learned functions  $f, g$  and a hyperparameter  $\lambda$ , to good effect (Sutskever et al., 2014; Chan et al., 2016). One often takes  $f = P_{\mathrm{S2S}}(\boldsymbol{W} \mid \boldsymbol{X})$  as the sequence-to-sequence model and  $g = P_{\mathrm{LM}}(\boldsymbol{W})$  as the language model. Since the sequence-level arg max is intractable, one can do fusion, which decomposes  $f = \prod f_t$  and  $g = \prod g_t$  over time (Gulcehre et al., 2015), restricting to the top  $N$  intermediate candidates at each step (beam search). Instead, our work considers  $N$ -best rescoring, which
computes  $f(\boldsymbol{W}, \boldsymbol{X})$  first, still using beam search to maintain the top  $N$  candidates and scores. Then,  $g(\boldsymbol{W})$  is computed for the resulting hypotheses and interpolated with these scores, giving a new top-1 hypothesis. The sequence model is now solely responsible for "capturing" the best hypothesis  $\hat{\boldsymbol{W}}$  in its beam. However, there are two advantages to  $N$ -best rescoring, which motivate PLLs as well as our maskless finetuning approach, respectively:
Decoupling of scale. Fusion requires correspondence between  $f_{t}$  and  $g_{t}$  at every  $t$ . This requires the sequence model and LM to be autoregressive and share tokenizations. In rescoring,  $f = P_{\mathrm{S2S}}$  does not require  $g$  to decompose over time or to be a true probability at all, though  $g$  should scale with  $f$  so that  $\lambda$  remains valid for all lengths  $|\mathbf{W}|$ ; e.g., taking  $g(\mathbf{W})$  to be a "relevance score" between 0 and 1 would not satisfy this property. The choice of log-linear is relevant here (Appendix B).
Length-independent inference. If  $g$  is nonrecurrent, then  $g(W)$  may be computed in a single inference pass. This difference manifests with self-attentive LMs like SANLMs and Transformer-XL (Dai et al., 2019), as recently explored for  $N$ -best rescoring in ASR (Li et al., 2019; Shin et al., 2019).
Further implementation and experimental details can be found in Appendix A and our code release:
LMs. We rescore sequence-to-sequence hypotheses as in Section 3.1. Each hypothesis is assigned its log probability (uni-SANLM, GPT-2) or pseudolog-likelihood score (bi-SANLM, BERT, M-BERT, RoBERTa, XLM). We tune the LM weight  $\lambda$  on the development set to minimize word error rate (WER) for ASR or maximize tokenized BLEU for NMT. We then evaluate on the test set.
ASR. Our 100-best hypotheses are from an end-to-end, 5-layer BLSTMP model (Shin et al., 2019) from ESPnet (Watanabe et al., 2018) on the 960-hour LibriSpeech corpus (Panayotov et al., 2015). Though this baseline is not state-of-the-art, we use their lists to enable direct comparison in Table 5.
NMT. Our 100-best hypotheses are from strong Transformer baselines with BPE subwords. One was pretrained for WMT 2014 English-German (Vaswani et al., 2017); the others are state-of-the-art low-resource models we trained for five pairs from the TED Talks corpus (Qi et al., 2018) and for
IWSLT 2015 English-Vietnamese (Cettolo et al., 2015), which we also describe in a dedicated, concurrent work (Nguyen and Salazar, 2019). For the low-resource models we scored tokenized hypotheses (though with HTML entities unescaped, e.g., &quot;  $\mapsto$  ")). Length normalization (Wu et al., 2016) is applied to NMT ( $\alpha = 0.6$ ) and LM ( $\alpha = 1.0$ ) scores (Section 4.3).
Corpus	Source → target language	# pairs
TED Talks	Galician (gl) → English (en)	10k
TED Talks	Slovakian (sk) → English (en)	61k
IWSLT 2015	English (en) → Vietnamese (vi)	133k
TED Talks	English (en) → German (de)	167k
TED Talks	Arabic (ar) → English (en)	214k
TED Talks	English (en) → Arabic (ar)	214k
WMT 2014	English (en) → German (de)	4.5M
3.3 Out-of-the-box (monolingual)
We consider BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), and RoBERTa (Liu et al., 2019), which are trained on 17GB, 40GB, and 160GB of written text respectively. Each model comes in similarly-sized 6-layer (117M / base) and 12-layer (345M / large) versions. GPT-2 is autoregressive, while BERT and RoBERTa are MLMs. We begin by rescoring ASR outputs in Table 2:
Table 1: Sizes of translation datasets used in this paper.
Model	dev	test
clean	other	clean	other
baseline (100-best)	7.17	19.79	7.26	20.37
GPT-2 (117M, based)	5.39	16.81	5.64	17.60
BERT (base, based)	5.17	16.44	5.41	17.41
RoBERTa (base, based)	5.03	16.16	5.25	17.18
GPT-2 (345M, based)	5.15	16.48	5.30	17.26
BERT (large, based)	4.96	16.26	5.25	16.97
RoBERTa (large, based)	4.75	15.81	5.05	16.79
oracle (100-best)	2.85	12.21	2.81	12.85
Table 2: WERs on LibriSpeech after rescoring. Baseline lists and oracle scores are from Shin et al. (2019).
As GPT-2 is trained on cased, punctuated data while the ASR model is not, we use cased MLMs and append ". ." to hypotheses to compare out-of-the-box performance. BERT outperforms its corresponding GPT-2 models despite being trained on less data. RoBERTa reduces WERs by  $30\%$  relative on LibriSpeech test-clean and  $18\%$  on test-other.
We repeat the same on English-target NMT in Table 3. As 100-best can be worse than 4-best due to the beam search curse (Yang et al., 2018; Murray
and Chiang, 2018), we first decode both beam sizes to ensure no systematic degradation in our models. Hypothesis rescoring with BERT (base) gives up to  $+1.1$  BLEU over our strong 100-best baselines, remaining competitive with GPT-2. Using RoBERTa (large) gives up to  $+1.7$  BLEU over the baseline. Incidentally, we have demonstrated conclusive improvements on Transformers via LM rescoring for the first time, despite only using  $N$ -best lists; the most recent fusion work (Stahlberg et al., 2018) only used LSTM-based models.
Model	TED Talks
gl→en	sk→en	ar→en
Neubig and Hu (2018)	16.2	24.0	-
Aharoni et al. (2019)	-	-	27.84
our baseline (4-best)	18.47	29.37	33.39
our baseline (100-best)	18.55	29.20	33.40
GPT-2 (117M, cased)	19.24	30.38	34.41
BERT (base, cased)	19.09	30.27	34.32
RoBERTa (base, cased)	19.22	30.80	34.45
GPT-2 (345M, cased)	19.16	30.76	34.62
BERT (large, cased)	19.30	30.31	34.47
RoBERTa (large, cased)	19.36	30.87	34.73
We also consider a non-English, higher-resource target by rescoring a pre-existing WMT 2014 English-German system (trained on 4.5M sentence pairs) with German BERT (base) models trained on 16GB of text, similar to English BERT. From 27.3 BLEU we get  $+0.5$ ,  $+0.3$  from uncased, cased; a diminished but present effect that can be improved as in Table 3 with more pretraining, a larger model, or domain adaptation (Section 3.5).
3.4 Out-of-the-box (multilingual)
To assess the limits of our modular approach, we ask whether a shared multilingual MLM can improve translation into different target languages. We use the  $100+$  language M-BERT models, and the 15-language XLM models (Conneau and Lample, 2019) optionally trained with a crosslingual translation LM objective (TLM). Monolingual training was done on Wikipedia, which gives e.g., 6GB of German text; see Table 4.
The 100-language M-BERT models gave no consistent improvement. The 15-language XLMs fared better, giving  $+0.2 - 0.4$  BLEU, perhaps from their use of language tokens and fewer languages. Our
Table 3: Test BLEU scores on English-target language pairs from the TED Talks corpus, after rescoring.
Model	IWSLT'15
en→vi	TED Talks
en→de	en→ar
Wang et al. (2018)	29.09	-	-
Aharoni et al. (2019)	-	23.31	12.95
our baseline (4-best)	31.94	30.50	13.95
our baseline (100-best)	31.84	30.44	13.94
M-BERT (base, uncased)	32.12	30.48	13.98
M-BERT (base, cased)	32.07	30.45	13.94
XLM (base*, uncased)	32.27	30.61	14.13
+ TLM objective	32.26	30.62	14.10
de-BERT (base, uncased)	-	31.27	-
de-BERT (base, cased)	-	31.22	-
German BERT results suggest an out-of-the-box upper bound of  $+0.8$  BLEU, as we found with English BERT on similar resources. We expect that increasing training data and model size will boost XLM performance, as in Section 3.3.
3.5 Domain adaptation
Out-of-the-box rescoring may be hindered by how closely our models match the downstream text. For example, our uncased multilingual models strip accents, exacerbating their domain mismatch with the casing, accented gold translation. We examine this effect in the setting of LibriSpeech, which has its own 4GB text corpus and is fully uncased and unpunctuated, unlike the casing MLMs in Section 3.3. We rescore using in-domain models in Table 5:
Table 4: Test BLEU scores for language pairs with non-English targets, after hypothesis rescoring. Base* uses 1024 hidden dimensions but only 8 heads instead.
Model	dev	test
clean	other	clean	other
baseline (100-best)	7.17	19.79	7.26	20.37
uni-SANLM	6.08	17.32	6.11	18.13
bi-SANLM	5.52	16.61	5.65	17.44
BERT (base, Libri. only)	4.63	15.56	4.79	16.50
BERT (base, cased)	5.17	16.44	5.41	17.41
BERT (base, uncased)	5.02	16.07	5.14	16.97
+ adaptation, 380k steps	4.37	15.17	4.58	15.96
oracle (100-best)	2.85	12.21	2.81	12.85
Table 5: WERs on LibriSpeech after hypothesis rescoring. Baseline, SANLM, and oracle numbers are from Shin et al. (2019).
Using a BERT model trained only on the text corpus outperforms RoBERTa (Table 2) which is trained on far more data, underscoring the tradeoff between in-domain modeling and out-of-the-box integration. Even minor differences like casing gives  $+0.3 - 0.4$  WER at test time. In Section 4.3 we
<https://github.com/dbmdz/german-bert>
see that these domain shifts can be visibly observed from the positionwise scores  $\log P_{\mathrm{MLM}}(\boldsymbol{w}_t \mid \boldsymbol{W}_{\backslash t})$ .
The best results ("adaptation") still come from adapting a pretrained model to the target corpus. We proceed as in BERT, i.e., performing MLM on sequences of concatenated sentences (more details in Appendix A). In contrast, the 3-layer SANLMs (Shin et al., 2019) do per-utterance training, which is slower but may reduce mismatch even further.
Finally, we show in Appendix C that even before evaluating WER or BLEU, one can anticipate improvements in the downstream metric by looking at improvements in word-normalized PPPL on the target corpus. The domain-adapted MLM has lower PPPLs than the pretrained models, and RoBERTa has lower PPPLs than BERT.
3.6 Finetuning without masking
We finetune BERT to produce scores without [MASK] tokens. For LibriSpeech we take the normalized text corpus and keep sentences with length  $|W| \leq 384$ , score them with our adapted BERT (base), then do sentence-level regression (Section 2.2). We train using Adam with a learning rate of  $10^{-5}$  for 10 epochs (Table 6):
Model	dev
clean	other
baseline (100-best)	7.17	19.79
GPT-2 (117M, cased)	5.39	16.81
BERT (base, uncased, adapted)	4.37	15.17
+ no masking	5.79	18.07
+ sentence-level finetuning	4.61	15.53
Table 6: WERs on LibriSpeech upon rescoring, showing the effects of single-copy, maskless scoring.
Sentence-level finetuning degrades performance by  $+0.2 - 0.4$  WER, leaving room for future improvement. This still outperforms GPT-2 (117M, cased), though this gap may be closed by adaptation. For now, maskless finetuning could be reserved for cases where only a masked language model is available, or when latency is essential.
Remarkably, we found that out-of-the-box scoring without [MASK] still significantly improves the baseline. This is likely from the  $20\%$  of the time BERT does not train on [MASK], but instead inputs a random word or the same word (Devlin et al., 2019). Future work could explore finetuning to positionwise distributions, as in word-level knowledge distillation (Kim and Rush, 2016), for which our results are a naive performance bound.
We recall the log-linear model from Section 3.1:
$$
\hat {\boldsymbol {W}} \approx \underset {\boldsymbol {W}} {\arg \max} [ \log f (\boldsymbol {W}, \boldsymbol {X}) + \lambda \log g (\boldsymbol {W}) ]
$$