A Better Way to Do Masked Language Model Scoring
Year: 2023

Abstract:
Estimating the log-likelihood of a given sentence under an autoregressive language model is straightforward: one can simply apply the chain rule and sum the log-likelihood values for each successive token. However, for masked language models (MLMs), there is no direct way to estimate the log-likelihood of a sentence. To address this issue, Salazar et al. (2020) propose to estimate sentence pseudolog-likelihood (PLL) scores, computed by successively masking each sentence token, retrieving its score using the rest of the sentence as context, and summing the resulting values. Here, we demonstrate that the original PLL method yields inflated scores for out-ofvocabulary words and propose an adapted metric, in which we mask not only the target token, but also all within-word tokens to the right of the target. We show that our adapted metric (PLL-word-l2r) outperforms both the original PLL metric and a PLL metric in which all within-word tokens are masked. In particular, it better satisfies theoretical desiderata and better correlates with scores from autoregressive models. Finally, we show that the choice of metric affects even tightly controlled, minimal pair evaluation benchmarks (such as BLiMP), underscoring the importance of selecting an appropriate scoring metric for evaluating MLM properties.1

heading: 1 Introduction
text: Most state-of-the-art transformer-based large language models (LLMs) fall into two classes: unidirectional (or autoregressive) models, where each token is generated based on its left context (e.g., GPT models; Radford et al., 2019), and bidirectional models, where a token is predicted from both left and right context tokens, some of which may be masked (e.g., BERT; Devlin et al., 2018). Often, it is beneficial to compare these models’ performance on controlled sentence generation benchmarks. Whereas unidirectional architectures offer a
1Our results and code are available at https://github. com/carina-kauf/better-mlm-scoring.
natural way of calculating sentence log-likelihood (summing the log-likelihood scores of each sentence token given its left context), there is no direct way of estimating sentence log-likelihood for a bidirectional model.
So far, the best available method to score a sentence under a bidirectional LLM has been the pseudo-log-likelihood (PLL) scoring approach described by Salazar et al. (2020) (and initially used by Shin et al., 2019; Wang and Cho, 2019). The PLL of a sentence is calculated as the sum of PLL scores for each token given all other sentence tokens, thus providing a comparable metric to unidirectional models’ log-likelihood (LL) sentence scoring. The PLL metric is extremely popular; it is used extensively in LLM studies tackling topics as diverse as effects of training data (Sinha et al., 2021; Zhang et al., 2021), model fluency (Laban et al., 2021), syntactic and conceptual knowledge (Sinclair et al., 2022; Bhatia and Richie, 2022), social biases (Nangia et al., 2020), and others. Some of these studies have already accrued dozens of citations.
Here, we show that the metric proposed by Salazar et al. (PLL-original) has important shortcomings that limit its utility. Specifically, PLL-original overestimates the PLL of outof-vocabulary (OOV) words, which LLM tokenizers split into multiple tokens. As a result, PLL-original scores fail on several theoretically
925
desired property tests: a robust inverse relationship between sentence length and sentence PLL (Section 4.1), a robust positive correlation between a word’s frequency and its PLL score (4.2), and a positive correlation between unidirectional and bidirectional model scores for the same sentences (Section 5). To remedy these issues, we propose an adjusted PLL metric, PLL-word-l2r (l2r: leftto-right), which estimates token PLL when future within-word tokens are also masked (Figure 1). We show that the PLL-word-l2r metric outperforms both PLL-original and alternative PLLbased metrics. We therefore recommend to use the PLL-word-l2r metric when estimating sentence PLL under a bidirectional LLM.

heading: 2 Motivation: score inflation for multi-token words
text: The PLL-original metric grossly overestimates the probability of OOV lexical items, such as souvenir (Figure 2). This is because OOV words are tokenized into subword tokens (e.g., so ##uven ##ir), and each subword token is predicted using the token’s bidirectional context, which crucially includes the remaining tokens that make up the OOV word. Thus, even though the OOV word itself may be surprising given the sentence context, the individual parts of the OOV word are not surprising to a bidirectional model given a sentence context that includes all other subtokens of that word (e.g., it is easy to predict so given ##uven ##ir; see Appendix A for additional examples).
To mitigate this bias, we adjust the PLL sentence scoring algorithm such that the model cannot access future within-word tokens (PLL-word-l2r) or any within-word tokens (PLL-whole-word) when predicting the target.
Below, we conduct a rigorous investigation of
our modified metrics to determine whether this intuitive benefit holds quantitatively.

heading: 3 Methods
text: For our analysis, we adapt the scorer module of the minicons library (Misra, 2022), an open-source wrapper library around HuggingFace transformers (Wolf et al., 2020) that enables efficient extraction of word- and sentence-level probabilities from LLMs. The MLM scoring procedure of the minicons library follows the procedure originally proposed by Salazar et al. (2020). For details on sentence preprocessing, see Appendix B.

heading: 3.1 PLL metrics
text: PLL-original. In this metric, each sentence token st of a sentence S with n tokens is consecutively replaced with a [MASK] and is predicted using all past and future tokens, irrespective of whether the context tokens belong to the same or a different word than the target token. Thus, inference is conditioned on the context S\t := (s1, . . . , st−1, st+1, . . . , sn). The final sentence score is obtained as the sum of the log probabilities of each sentence token given its context:
PLLorig(S) := n∑
t=1
log PMLM(st | S\t) (1)
PLL-word-l2r. In this metric, a [MASK] is placed not only over the current target token (now: swt), but also over all future sentence tokens that belong to the same word sw as the target. Inference is then conditioned on a context that includes all preceding sentence tokens (including those belonging to the current word) and all sentence tokens from future words. The final score of a sentence S is obtained as the sum of the log probabilities of each of the |w| tokens in each of the |S| words:
PLLl2r(S) :=
|S|∑
w=1
|w|∑
t=1
log PMLM(swt | S\swt′≥t )
(2) PLL-whole-word. This metric is similar to PLL-word-l2r and differs from it only in that a [MASK] is placed over all sentence tokens that belong to the same word sw as the target (both preceding and future). Inference is then conditioned on a context that includes all sentence tokens except those belonging to the current word. The final score of a sentence S is obtained as the sum of the log probabilities of each of the |w| tokens in each of the |S| words in S given the token’s context:
PLLww(S) :=
|S|∑
w=1
|w|∑
t=1
log PMLM(swt | S\sw)
(3) In Appendix G, we also report results for a PLL metric where not only future within-word tokens, but all sentence tokens to the right of the
target context are masked (PLL-sentence-l2r). Although this method is most similar to autoregressive LL scoring, sentence-l2r masking for BERT is known to produce poor quality generations (Wang and Cho, 2019); we therefore refrain from including this metric in the main text.

heading: 3.2 Models
text: We report results for bert-base-cased (and gpt2-medium for comparison) unless stated otherwise. Results for larger models are provided in Appendices D-F.

heading: 3.3 Datasets
text: For our main analyses, we use the EventsAdapt dataset (Kauf et al., 2022, based on Fedorenko et al., 2020). It contains a curated set of 782 syntactically simple sentence pairs that describe plausible or implausible agent-patient interactions in active or passive voice (e.g., The traveler lost the souvenir). Sentences in this dataset are 5-7 words long (mean: 6.1, std: 1.05), with an average word log frequency of 10.95. We use this dataset because it
contains a high number of OOV words (19.6% for BERT and 40.3% for GPT-2; see also Appendix C). In Appendices D-F, we show that our results generalize to two larger and more diverse corpora: the Brown corpus (Francis and Kucera, 1979) and the reference sentence set from the LibriSpeech corpus (Panayotov et al., 2015). We also apply our PLL metrics to score the sentences in the Benchmark of Linguistic Minimal Pairs (BLiMP) (Warstadt et al., 2020), a challenge set of 67k sentence pairs which target specific aspects of linguistic knowledge.

heading: 4 Evaluating PLL metric properties

heading: 4.1 Effects of sentence length
text: Like Salazar et al. (2020), we expect that models should, on average, assign lower probability to longer sentences. Thus, negative PLL (which reflects model surprisal) should be positively correlated with sentence length. However, the PLL-original metric violates this expectation in our test sentence set, which shows a negative correlation between the number of tokens and negative PLL. In contrast, PLL-word-l2r and PLL-whole-word metrics exhibit a positive correlation between the number of sentence tokens and negative PLL, just as the negative LL scores for a unidirectional model, GPT2-medium (Figure 3A).

heading: 4.2 Effects of word frequency
text: An appropriate (P)LL metric should reflect the fact that LLMs are sensitive to distributional patterns in training text corpora. In particular, we expect more frequent words to have higher (P)LL scores in the absence of contextual effects. This is indeed the case for GPT2-medium; however, the score inflation for multi-token words means
that the PLL-original metric grossly overestimates the scores for low-frequency words (Figure 3B). PLL-word-l2r scores restore this relationship: their correlation with word frequency is much higher than for PLL-original. PLL-whole-word also performs well, although its correlation with word frequency is lower than for PLL-word-l2r, suggesting that it excessively penalizes OOV words.

heading: 5 Correlation with GPT-2 scores
text: We expect that PLL scores for bidirectional models should be at least somewhat consistent with LL scores for unidirectional models: both metrics are designed to serve are a proxy for sentence probability. Here, we show that the GPT-2/BERT score correlation for the PLL-original metric is very low, whereas correlation scores for PLL-word-l2r and PLL-whole-word are much higher (Figure 4), indicating the validity of this metric for cross-model comparison. As in Section 4.2, PLL-word-l2r slightly outperforms PLL-whole-word, likely because it does not penalize OOV words as severely.
See Appendices D-F for evidence that all three trends hold for larger models and for other datasets (although the effects in other datasets are attenuated due to a lower OOV ratio).

heading: 6 Effects on benchmarking
text: Here, we show that the choice of PLL metric affects benchmarking results for a popular, highly controlled, minimal pair linguistic benchmark: BLiMP. Despite the fact that the comparisons are highly controlled, different metrics yield different BLiMP scores. For all four tested models, PLL-word-l2r achieves the best overall BLiMP score (Table 1).
See Appendix H for detailed scores.

heading: 7 Conclusion
text: We have shown that PLL-word-l2r is the preferred metric for evaluating sentence PLL under a masked language model, such as BERT. Although the results from studies using the PLL-original metric can still be informative, they become harder to interpret if the proportion of OOV words in their test set is high. Therefore, we recommend using PLL-word-l2r in future works.
Limitations
The proposed PLL-word-l2r metric has the same practical limitations as previous LL/PLL approaches. Most importantly, these scores can be influenced by many superfluous factors, such as the number of available synonyms (computer vs. laptop; Holtzman et al., 2021). We therefore expect our method to be most useful in highly controlled minimal pair or multiple choice setups.
Even more accurate metrics may emerge in the future. For instance, our approach pre-specifies the number of tokens in a word, thus limiting the space of possible alternatives. Future approaches might investigate a way to normalize the PLL score distribution over words with a varying number of tokens. Further, it would be interesting to attempt to estimate the joint probability of all tokens in a word instead of predicting them left-to-right (as in PLL-word-l2r) or without any other within-word contextual information (as in PLL-whole-word).
Finally, we test our approach on English text corpora; our results might not generalize to agglutinative languages (due to a high number of tokens per word and, therefore, increased uncertainty) and
are of less relevance to isolating languages (where, if enough training data are available, most wordlevel items can be represented as single tokens).
Ethics Statement
In our proposed metric, word tokens are masked from left to right following the writing tradition in English; however, for speakers of languages such as Arabic, a “right to left” notation would be more intuitive. Note, however, that this is primarily a denotational difference that does not affect the score itself (LLMs do not discriminate left and right, only beginning and end). We do not anticipate any specific harms that would be intrinsically associated with the techniques described in this paper.

heading: Acknowledgements
text: We thank Jacob Andreas, Evan Hernandez, and the anonymous ACL reviewers for their insightful feedback. CK was supported by the K. Lisa Yang Integrative Computational Neuroscience (ICoN) Center at MIT. AI was supported by MIT Quest for Intelligence.

heading: A Additional examples of score inflation

heading: B Text preprocessing for (P)LL computation
text: The minicons library borrows the MLM preprocessing algorithm from Salazar et al. (2020): [CLS] and [SEP] tokens are prepended and appended to the text, respectively, and are not masked during PLL computation. For CLMs, we minimally adjust the minicons scorer library default and necessarily prepend the beginning of sentence token, 
, to the text, which enables us to get a probability for the first actual sentence token (see also the lm-zoo library; Gauthier et al., 2020). The (P)LLs of all special tokens are not counted toward the final sentence/word score.
When calculating the (P)LL score of individual words (to estimate word frequency effects), we place them in a neutral context My word is _. To ensure that the same pattern of results holds across multiple neutral contexts, we additionally test the context I opened the dictionary and randomly picked a word. It was _, as well as a nocontext setup. These additional results are reported in Appendix E.1.
Word frequency was operationalized as the log of the number of occurrences of the word in the 2012 Google NGram corpus. Laplace smoothing was applied prior to taking the logarithm.

heading: C Quantification of out-of-vocabulary words per dataset

heading: D Effects of sentence length
text: D.1 Larger LLM versions
D.2 Larger datasets

heading: E Effects of word frequency
text: E.1 Different word contexts
E.2 Different datasets

heading: F Correlation with unidirectional models
text: F.1 Larger LLM versions
F.2 Larger datasets

heading: G Whole-sentence left-to-right token masking
text: Here, we report results for the scoring algorithm that masks the target token, st, and all sentence tokens to its right in a sentence S with n tokens (PLL-sentence-l2r). As in autoregressive language models, target token inference is thus conditioned solely on the token’s leftward context: PMLM(st | S<t). The final sentence score is obtained as the sum of the log probabilities of each sentence token given its context:
PLLsent(S) :=
n∑
t=1
log PMLM(st | S<t) (4)
Overall, the PLL-sentence-l2r metric satisfies the metric desiderata better than the PLL-original metric but worse than PLL-word-l2r. In addition, it is inferior to other metrics on the BLiMP evaluation benchmark (Appendix H), in line with previous reports of subpar generation quality (Wang and Cho, 2019).
Ov era
ll
AN A.
AG R
AR G S
TR .
BIN DI
NG
CT RL
. R AI
S.
DN A
GR
EL LIP
SIS
FIL LE
R G AP
IRR EG
UL AR
ISL AN
D
NP I
QU AN
TIF IER
S
S-V AG
R

heading: H Detailed BLiMP benchmark results
text: Table 3 shows results for each sentence suite within the BLiMP benchmark (in addition to the overall scores reported in the main text). All models shown in Tables 1 and 3 are cased models. PLL-original scores replicate those reported in Salazar et al. (2020).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
7 A2. Did you discuss any potential risks of your work? we do not anticipate specific risks associated with our work
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? all
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? the models are available on huggingface, and the experiments are computationally light
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3 and Appendix A (no hyperparameter search was conducted though)
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? all results figures
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, we recommend using PLL-word-l2r in future works. Limitations
The proposed PLL-word-l2r metric has the same practical limitations as previous LL/PLL approaches. Most importantly, these scores can be influenced by many superfluous factors, such as the number of available synonyms (computer vs. laptop; Holtzman et al. , 2021). We therefore expect our method to be most useful in highly controlled minimal pair or multiple choice setups. Even more accurate metrics may emerge in the future. For instance, our approach pre-specifies the number of tokens in a word, thus limiting the space of possible alternatives. Future approaches might investigate a way to normalize the PLL score distribution over words with a varying number of tokens. Further, it would be interesting to attempt to estimate the joint probability of all tokens in a word instead of predicting them left-to-right (as in PLL-word-l2r) or without any other within-word contextual information (as in PLL-whole-word). Finally, we test our approach on English text corpora; our results might not generalize to agglutinative languages (due to a high number of tokens per word and, therefore, increased uncertainty) and
are of less relevance to isolating languages (where, if enough training data are available, most wordlevel items can be represented as single tokens)