PMI-Masking: Principled masking of correlated spans
Authors: Yoav Levine
Barak Lenz
Opher Lieber
Omri Abend
Kevin Leyton-Brown
Moshe Tennenholtz
Yoav Shoham
Venue: ICLR 2021 Spotlight (2021)
Keywords: Language modeling
BERT
pointwise mutual information

Abstract: Masking tokens uniformly at random constitutes a common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM to minimize its training objective by latching onto shallow local signals, leading to pretraining inefficiency and suboptimal downstream performance. To address this flaw, we propose PMI-Masking, a principled masking strategy based on the concept of Pointwise Mutual Information (PMI), which jointly masks a token n-gram if it exhibits high collocation over the corpus. PMI-Masking motivates, unifies, and improves upon prior more heuristic approaches that attempt to address the drawback of random uniform token masking, such as whole-word masking, entity/phrase masking, and random-span masking. Specifically, we show experimentally that PMI-Masking reaches the performance of prior masking approaches in half the training time, and consistently improves performance at the end of pretraining.

PMI-MASKING:
PRINCIPLED MASKING OF CORRELATED SPANS
Yoav Levine
Barak Lenz
Opher Lieber
Omri Abend
Kevin Leyton-Brown
Moshe Tennenholtz
Yoav Shoham
A121 Labs, Tel Aviv, Israel
bywwl.barkk..ohher..mmi.icc @aiil.com
ABSTRACT
Masking tokens uniformly at random constitutes common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM minimize its training objective by latching onto shal- low local signals, leading to pretraining inefficiency and suboptimal downstream performance. To address this flaw, we propose PMI-Masking, principled mask- ing strategy based on the concept of Pointwise Mutual Information (PMI, which jointly masks a token n-gram it exhibits high collocation over the corpus. PMI- Masking motivates, unifies, and improves upon prior more heuristic approaches that attempt to address the drawback of random uniform token masking, such as whole- word masking, entity/phrase masking, and random-span masking. Specifi- cally, we show experimentally that PMI-Masking reaches the performance of prior masking approaches in half the training time, and consistently improves perfor- mance at the end of training.
INTRODUCTION
In the couple of years since BERT was introduced a seminal paper by Devlin et al. (2019a). Masked I Language Models (MLMs) have rapidly advanced the NLP frontier (Sun et al., 2019 Liu ict al. 2019 Joshi et al. 2020 Raffel et al. 2019). At the heart of the MLM approach is the task of predicting masked subset of the text given the remaining, unmasked text. The text itself is broken up into tokens, each token consisting of a word or part of a word; thus "chair" constitutes a single token, but
words like re-igen-val-ue In BERT, 15% of tokens are chosen to be masked uniformly at random. It is the random choice of single tokens that address in this paper: show that this approach is suboptimal and offer principled alternative.
broken up into several sub- word tokens.
To see why Random-Toke Masking is suboptimal, consider the special case of sub-word tokens. Given the masked sentence "To approximate the matrix, we use the eigenvector corresponding to its largest e-Imask]]vvl-ue",
MLM will quickly learn to predict "igen" based only on the context "e- [mask]-val-ue", rendering the rest of the sentence redundant. The question is whether the network will also learn to relate the broader context to the tokens comprising "eigenvalue". When they are masked together, the network is forced to do so, but such masking occurs with vanishingly small probability. One might hypothesize that the network would nonetheless be able to piece such meaning together from local cues; however, show that it often struggles to do so.
We establish this via controlled experiment, which we reduced the size of the vocabulary, thereby breaking more words into sub-word tokens. We compared the extent to which such vo- cabulary reduction degraded regular BERT relative to so-called Whole- Word Masking BERT (WW- BERT) (Devlin et al. 2019b), version of BERT that jointly masks all sub-word tokens comprising an of vocabulary word during training. We show that vanilla BERT's performance degrades much more rapidly than that of WWBERT the vocabulary size shrinks. The intuitive explanation

---

is that Sondom--Tkert Masking is wasteful; it overtrains on easy sub- word tasks (such as predicting "igen") and undertrains on harder whole-word tasks (predicting "eigenvalue").
The advantage of Whole-Wort Masking over Random- Token Masking is relatively modest for stan- dard vocabularies, because out-of-vocabulary words are rare. However, the tokenization of words is very special case of much broader statistical linguistic phenomenon of collocation: the co- occurrence of series of tokens at levels much greater than would be predicted simply by their in- dividual frequencies in the corpus. There are millions of collocated word r-grams multi-word expressions, phrases, and other common word combinations - whereas there are only tens of thou- sands of words in frequent use. So it is reasonable to hypothesize that Random- Token Masking generates many wastefully easy problems and too few usefully harder problems because of multi- word collocations, and that this affects performance even more than the rarer case of tokenized words; we show that this indeed is the case.
Several prior works have considered the idea of masking across spans longer than single word. Sun et al. (2019) and Guu et al. (2020) proposed Knowledge Masking and Salient Span Masking, respectively, in which tokens comprising entities or phrases, as identified by external parsers, are jointly masked. While extending the scope of Whole Masking, the restriction to specific types of correlated n- n-grams, along with the reliance on imperfect tools for their identification, has lim- ited the gains achievable by these approaches. With similar motivation in mind, SpanBERT of Joshi et al. (2020) introduced Random-Span Masking, which masks word spans of lengths sampled shown
from geometric distribution at random positions in the text. Random-Span Masking to consistently outperform Knowledge Masking, is simple to implement, and inspired prominent MLMs (Raffel et al. 2019). However, while Random-Span Masking increases the chances of mask- ing collocations, with high probability the selected spans break up correlated n-grams, such that the prediction task can often be performed by relying on local cues.
In this paper we offer a principled approach masking spans that consistently provide high signal, unifying the intuitions behind the above approaches while also out- performing them. Our approach, dubbed PMI-Masking, uses Point- wise Mutual Information (PMI)
82
Random--ookkee Token Whole-Word Frequency Random-Span-
80
SQUAD2.0F1 F1 score
78
to identify collocations, which we then mask jointly. At a high level, PMI-Masking consists of two stages. First, given any pretraining corpus, we identify of contigu- ous n-grams that exhibit high co- occurrence probability relative the individual occurrence probabil- ities of their components. We for- malize this notion by proposing extended definition of Pointwise Mutual Information from bi- grams to longer n-grams. Second, we treat these collocated n-grams as single units; the masking strategy selects random both from these units and from standard tokens that do not participate in such units. Figure [ detailed and reinforced by further experiments in section 5 shows that (1) PMI-Masking dramatically accelerates training, matching the end-of-pretraining performance of existing approaches in roughly half of the training time; and (2) PMI-Masking improves upon previous masking approaches at the end of pretraining.
76
74 200K
400K
600K
800K
Pretraining steps
Figure 1: SQuAD2.0 development set F1 scores of BERT BASE models trained with different masking schemes, evaluated ev- ery 200K steps during pretraining.
2 MOTIVATION: MLMS ARE SENSITIVE TO TOKENIZATION
In this section we describe a simple experiment that motivates our PMI-Masking approach. We examined BERT's ability to leam effective representations for words consisting of multiple sub- word tokens, treating this setting problem that truly interests us. Our experiment sought to assess the performance gain obtained from always masking whole words as opposed masking each individual token uniformly random. We
easily controlled analogue for the multi-word collocation
2

---

1.08 tokens per word (30K vocabulary)
1.22 tokens per word (10K vocabulary)
2.06 tokens per word (2K vocabulary)
Random-Token Masking
79.3
77.8
72.8
Whole Word Masking
79.7
79.5
77.6
SQuAD2.0 development set Fl scores of BERT models trained with Random-Toker and Whole-Woor masking schemes and with different vocabulary sizes (30K; 10K; 2K).
Table 1:
compared performance across range of vocabulary sizes, using the same WordPiece Tokenizer that produced the original vocabulary ~ 30K tokens. As we decreased a 30K-token vocabulary to 10K and 2K tokens, the average length of word over the pretraining corpus increased from 1.08 tokens to 1.22 and 2.06 tokens, respectively. Thus, by reducing the vocabulary size, we increased the frequency of multi-token words by large factor.
Table | presents the performance of BERT models trained with these vocabularies, measured as score on the SQuAD2.0 development set (the experimental setup described in section 4 The downstream performance of Random- Token Masking substantially degraded as vocabulary size de- creased and the number of spans of sub-wa word tokens increased. One reason for such degradation might be the model seeing less text as context (512 input tokens cover less text when more words are broken into multiple tokens). This possibly plays a role; however, for models with the same vocabularies trained via Whole-Word Masking, this degradation was significantly attenuated. We therefore conjecture that this degradation occurred primarily because of the random masking strat- egy, which allows the model to use "shortcuts" for minimizing its loss, thus hindering its ability to learn the distribution of the entire multi-token word.
If our conjecture is correct, such shortcuts are just problematic in the case of inter-word colloca- tions. In fact, for the regular 30K-token vocabulary, divided words are rare, so inter-word colloca- tions would pose larger problem than intra- word collocations in the common setting. One possible mitigation might be to expand the vocabulary to include multi-wore collocations. However, there are millions of these, and such vocabulary sizes are currently infeasible. Even if could get around the practical issue of size, this approach may suffer from generalization problems: the frequency of each multi-word collocation representation. An alternative, more practical approach is to leave the vocabulary as is, but jointly mask co-located words, with the intention of cutting off local statistical "shorteuts" and allowing the model to improve further by learning from broader context. This is the approach we take in this paper. what follows we detail such a masking approach and show its advantages experimentally.
be lower than the sample complexity for learning a meaningful
MASKING CORRELATED n-GRAMS
3.1 EXISTING MASKING APPROACHES
We now more formally present the MLM setup as well as existing masking approaches, which we implement as baselines. Given text tokenized into sequence of tokens, Masked Language Models are trained to predict a set fraction of "masked" tokens, where this fraction is called the masking budget and is traditionally set to 15%. The modified input is inserted into the Transformer-based architecture (Vaswani et al. 2017) of BERT, and the pretraining task is to predict the original identity of each chosen token. Several alternatives have been proposed for choosing the set of tokens to mask.
Random-Tofe Token Masking (Devlin et al. 2019a) The original BERT implementation selects to- kens for masking independently at random, where 80% of the 15% chosen tokens are replaced with [MASK], 10% are replaced with a random token, and 10% are kept unchanged.
Whole-Word Masking (Devlin et al., 2019b) The sequence of input tokens is segmented into units corresponding to whole words. Tokens for masking are then chosen by sampling entire units random until the masking budget is met. Following [Delii et al. (2019a), for 80%/10%//10% of the units, all tokens are replaced with [MASK] tokens/ random tokens/ the original tokens, respectively.
3

---

Contiguous random spans are selected iteratively until the 15% masking budget is spent. At each iteration, a span length (in words) is sampled from a geometric distribution & ~ Geo(O.2), and capped at 10 words. Then, the starting point for the span to be masked is randomly selected. Replacement with [MASK], random, or original tokens is done as above, where spans constitute the units.
Random-Span Masking (Joshi et al. 2020)
3.2 PMI: FROM BIGRAMS n-GRAMS
Our aim is to define masking strategy that targets correlated sequences of tokens in principled way. Of course, modeling such correlations in large corpora was widely studied in computational linguistics (Zuidema (2006); Ramisch et al. (2012): inter alia). Particularly relevant to our work is the notion of Pointwise Mutual Information (Fano 1961), which quantifies how often two events occur, compared with what we would expect if they were independent. Define the probability of any n-gram as the number of its occurrences in the corpus divided by the number of all the n-grams in the corpus. PMI leverages these probabilities to give natural measure of collocation of bigrams: how surprising the bigram W1 W2 is, given the unigram probabilities of W1 and Wy. Formally, given two tokens wi and w2, the PMI of the bigram "wi w22 is
PMI(w112) = log p(w W2) p(w1 )p(w2)
Importantly, PMI is qualitatively different from pure frequency: relatively frequent bigram may not have a very high PMT score, and vice versa. For example, the bigram "book is" appears 34772 SKKIPDDA++BOKCORRE
times in the ranking for bi- bi-grams over this corpus, while the bigram "boolean algebra" appears 849 times in the corpus but is ranked around position 16K in the PMI ranking.
dataset but is ranked around position 760K in the PMI
What about contiguous spans of more than two tokens? For a given n- n-gram, we would again like to measure how strongly its components indicate one another. We thus require measure that captures correlations among more than two variables. A standard and direct extension of the PMI measure to more than variables, referred to 'specific correlation' in Van de Cruys] (2011), and 'Naive- PMI, in this paper, is based on the ratio between the n-gram's 's probability and the probabilities of its component unigrams:
pl [1
W.)
(ai wa) = log
(2)
IT (wy)
As in the bivariate case, this measure compares the actual empirical probability of the n-gram in the corpus with the probability it would have if its components occurred independently. However, the above definition suffers from an inherent flaw: an n-gram's 's Naive-PMI, will be high if it contains segment with high PMI, even if that segment is not particularly correlated with the rest of the n-gram. Consider for example the case of trigrams:
Naive-PMIyy (w1 w2ws)) = logf ((wrw2w)))
p(w1u2) (MI(W1u)) log
p(w W2wps)
((w1)p(w2)p((
p( w2)
p(w we (
Where PMI(wruy) is defined in eq. 1 When PMI(W) w2) is high, the Naive-PMI; measure of the
p(w; w (a) trigram "wi uyto"" will start at this high baseline. The added term of log p(we p(uc) (p))(()))) quantifies the actual added information of "wz" to this correlated bigram, i.e., it quantifies how far p(w W2 wy) is from being separable w.r.t. the segmentation into "wi w2" and "wa" " For example, since the PMI of the bigram "Kuala Lumpur" is very high, the NaIve-PMI, of the trigram "Kuala Lumpur is" is misleadingly high, placing at position 43K out of all trigrams in the SEALPDDI+BBOOCORRE dataset. It is in fact placed much higher than obvious collocations such as the trigram "editor in chief", which is ranked position 210K out of all trigrams.
In order to favor n-grams that cannot be easily subdivided into shorter unrelated spans, we propose a measure of distance from separability with respect to all of an n-gram's possible segmentations rather than with respect only to the segmentation into single tokens:
p(wi- W..)
(M ... w,)= min log (E.e
(3)
II. p(s)
4

---

Here, seg(W1... ................................ wn) is the set of all contiguous segmentatiinn of the n-gram "w1 ... " "...,,"""xx"iili (excluding the identity segmentation), where any segmentation o E seg( ... ..,))),,, is composed of sub-spans which together give "w ... Wnn' " Intuitively, this measure effectively discards the contribution of high PMI segments; the minimum in Eq. [ implies that an n-gram's collocation score is given by its weakest link, i.e., by the segmentation that is closest separability. When ranked by the above PMI, measure, the trigram "Kuala Lumpur is" is demoted to position 1.6M, since the segmentation into "Kuala Lumpur" and "is" yields unrelated segments, while the trigram "editor in chief" is upgraded to position 33K since its segmentations yield correlated components. As we will see, this definition is not only conceptually cleaner, but also leads improved performance.
3.2.1 PMI-MASKING
We implement strategy of treating highly collocating n-grams as units for masking by assem- bling list of n-grams as a masking vocabulary in parallel to the 30K-token vocabulary. Specifically, make use of the entire pretraining corpus for compiling list of collocations. We consider word n-grams of lengths 2-5 having over 10 occurrences in the corpus, and include the highest rank- ing collocations over the corpus, as measured via our proposed PMI, measure (Eq, 3 Noticing that the PMI, measure is sensitive to the length of the n-gram, we assemble per-length rankings for each n (2, 4, 5), and integrate these rankings conducting preliminary evaluation of how an n-gram'ss quality as collocation degrades with its PMI, rank (detailed in the appendix), we chose the masking vocabulary size to be 800K, for which approximately half of pretraining corpus tokens were identified as part of some correlated n-gram.
compose the masking vocabulary. After
In order to get some sense of the differences between the attained masking vocabulary and prior approaches, we annotated random sample of 500 bigrams and 500 trigrams from the masking vocabulary with Entity/Not-Entiiy labels. We found that only around 14% of the entries in the bi- grams/trigrams lists were annotated entities, while the rest are other types of collocations. More- over, we found that while named entities are very prevalent at the very top of the list, they are scarce otherwise. By refining the view into highest and lowest ranking PMI bigram groups, 50% of the top 20% are entities while only 1% of the bottom 20% are entities, and similar trends are attained for trigrams. This breakdown can illuminate natural intuition regarding high ranking PMI n-grams representing entities (employed also by previous works (Downey et al., 2007 Korkontze- los al.) 20088)- indeed the top ranking PMI entries are largely entities. But we chose much larger PMI-based masking vocabulary (see appendix I on the process of choosing its size), and the proportion of entities drops to around 1/7, with many of the added entries representing other types of collocations (the annotated lists are given as supplementary material).
get that
After composing the masking vocabulary, we treat its entries as units to be masked together. All input tokens not identified with entries from the masking vocabulary are treated independently as units for masking according to the Whole-Word Masking scheme. If one masking vocabulary entry contains another entry in given input, we treat the larger one the unit for masking, e.g., if the masking vocabulary contains the n-grams "the united states", "air force", and "the united states air force", the latter will be one unit for masking when it appears. In the case of overlapping entries, we choose one at random as a unit for masking and treat the remaining tokens as independent units, e.g., if the input text contains "by the way out" and the masking vocabulary contains the n-grams "by the way" and "the way out", we can choose either "by the way" and "out" or "by" and "the way out" as units for masking.
After we segment the sequence of input tokens into units for masking, we then choose tokens for masking by sampling units uniformly at random until 15% of the tokens (the standard tokens of the 30K-token vocabulary) in the input are selected. As in the prior methods, replacement with [MASK] (80%), random (10%), or original (10%) tokens is done the unit level.
EXPERIMENTAL SETUP
To evaluate the impact of PMI-Masking, we trained Base-sized BERT models (Devlin et al. 20 (0) with each of the masking schemes presented in Section 3. Rather than relying on existing imple- mentations for baseline masking schemes, which vary in training specifics, we reimplemented each scheme within the same framework used to train our PMI-Masked models. For control, we trained within the same framework models with Paive-PMI-Masking and Frequency-Masking, following
5

---

the procedure described above for PMI-Masking, but ranking by the Naive- measure (Eq. and by pure- sure-frequency, respectively. In Section [ we compare our Mas-ing to all internally- trained masking schemes (Table well as with externaly released models (Table 3)
4.1 PRETRAINING
We trained uncased models with a 30K-sized vocabulary that we constructed over WIKIPEDIA +BOOKCORPUU via the WordPiece Tokenizer used in BERT. We omitted the Next Sentence Pre- diction task, as it was shown to be superfluous (Joshi et al., 2020), and trained only on the Masked Language Model task during pretraining. We trained with sequence length of 512 tokens, batch size of 256, and a varying number of steps detailed in Section [ For pretraining, after a warmup of 10, 000 steps used linear learning rate decay, therefore models that ran for different over- all amount of steps are not precisely comparable after a given amount of steps. We set remaining parameters values similar to those used in the original BERT pretraining, detailed in the appendix.
We performed the baseline pretraining over the original corpus used to train BERT: the 16GB BIKIPEDIA-BOOKCORPUU dataset. We show that PMI-Masking achieved even larger perfor- mance gains relative to the baselines when training over more data, by adding the 38GB OPEN- WEBTEXT (Gokaslan Cohen 2019) dataset, an open-source recreation of the WebText corpus described in Radford et al.19). As described in section 3 we compose our PMI, -based masking vocabulary according to the pretraining corpus use.
4.2 EVALUATION
We evaluate our pretrained models on two question answering benchmarks: the Stanford Question Answering Dataset (SQuAD) and the ReAding Comprehension from Examinations (RACE), as well as on the General Language Understanding Evaluation (GLUE) benchmark. Additionally, we report the Single-Token perplexity our pretrained models.
SQuAD (Rajpurkar al. 2016) has served major question answering benchmark for pre- trained models. It provides a paragraph of context and a question, and the task is to answer the question by extracting the relevant span from the context. We focus on the latest more challeng- ing variant, SQuAD2.0 (Rajpurkar et al.) 2018), in which some questions are not answered in the provided context, and the task includes identifying such cases.
RACE LLai et al. 2017) is a large-scale reading comprehension dataset collected from English examinations in China, designed for middle and high school students. Each passage is associated with multiple questions; for each, the task is select one correct answer from four options. RACE has significantly longer context than other popular reading comprehension datasets and the proportion of questions that requires reasoning very large.
GLUE (Wang et al., 2018) collection of 9 datasets for evaluating natural language under- standing systems. Tasks are framed as either single-sentence classification or sentence-pate pair clas- sification tasks. For full details, please see the appendix.
Single-Token perplexity We evaluate an MLM's ability to predict single-tokens by measuring perplexity over held out test set of 1 10K tokens from OPEN TEXT. For each test example, single token for prediction is masked and the remainder of the input tokens are unmasked.
In Tables 2 and for every downstream task we swept 8 different hyperparameter configurations (batch sizes E { [1, 32) and learning rates E {1, 3, 5} 10 10-5). We report the best median devel- opment set score over five random initializations per papee--arameter. When applicable, the model with this score was evaluated on the test set. The development set score of each configuration was attained by fine-tuning the model over 4 epochs (SQuAD2.0 and RACE) or epochs (all GLUE tasks except RTE and STS 10 epochs) and performing early stopping based each task's eval- uation metric on the development set. In the preliminary experiments of Table I and in Figures and m for which evaluate many pretraining checkpoints per model, we report the average of the three middle scores out of 5 random initializations for a single set of parameees (batch size 32 and learning rate 3 10-5).

---

2.4M steps on Wikipedia- (16G)
2.4M steps Wilppdas-BBoCCrrssoooeWWeeeieii (54G)
85
Random-Span Masking
PMI-Masking
85
Random-Span Masking PMI-Maskinn
GQuADo F score
Score
83
83
81
81
79
79
77
75
0.5M
1.5M
2M
2.4M
0.5M
1M
1.5M
2M
2.4M
Pretraining step
Pretraining step
Figure 2: Scores on SQuAD2.0 development set of BERTBASS BASE models trained for 2.4M steps, as done by Joshi et al. 2020)) when proposing Random-Span Masking. Left: PMI-Masking efficiently elicits information from limited data. Right: More data, PMI-Masking continues to improve. See numerical scores in the appendix, along with the same trends on the RACE benchmark.
5 EXPERIMENTAL RESULTS
We evaluated the different masking strategies in two key ways. First, we measured their effect on downstream performance throughout pretraining assess how efficiently they used the pretraining phase. Second, we more exhaustively evaluated downstream performance of different approaches at the end of pretraining. We examine how the advantage of PMI-Masking is affected by the size the pretraining corpus and by amount of examples seen during pretraining (batch size x training steps).
5.1 EVALUATING DOWNSTREAM PERFORMANCE THROUGHOUT PRETRAINING
By examining the model's 's downstream performance after each 200K steps of pretraining, we demon- strate that PMI-Masking speeds up MLM training. Figure I investigates the standard BERT setting of pretraining on the Wikipedia+BBokConre shows that PMI-Masking method clearly outperformed a variety of prior approaches, as well as the baseline pure frequency based masking, on the SQuAD2.0 development set for all examined checkpoints (these patterns are consistent on RACE, see detailed scores in the appendix). PMI- Masking achieved the score of Random-Spp Span Masking, the best of the existing approaches, after roughly half as many steps of pretraining.
dataset for IM training steps with batch size 256. It
We ran second experiment that increased the number of steps from 1M to 2.4M, while main- taining the batch size and the pretraining corpus; this was the setting used by JJshi et al. (2020) when proposing Random-Span Masking. We observed that while PMI-masking learned much more quickly, eventually reached plateau, and Random-Span Masking caught up after enough training steps. Figure (lett) details these results.
details
Finally, we increased the amount of training data by adding the OPEN TEXT corpus (~ 3.5x more data). Figure 2 (right) demonstrates that the plateau we previously observed in PMI- Masking's ' performance was due to limited training data. When training for 2.4M training steps on the AikipediatBookCorpusss
WebText dataset, PMI-masking reached the same score that Random-Span Masking did at the end of training after roughly half of the pretraining, and continued improve. Thus, PMI-Masking definitively outperformed Random-Span masking a scenario where data was not bottleneck, is ideally the case in MIL pretraining (Raffel et al. 2019).
5.2 EVALUATING DOWNSTREAM PERFORMANCE AFTER PRETRAINING
Table 2 shows that after pretraining was complete, PMI-Masking outperformed prior masking ap- proaches in downstream performance on the SQuAD2.0, RACE, and GLUE benchmarks. In agree- ment with Figure 2] for longer pretraining (2.4M training steps) the absolute advantage of PMI- Masking is boosted across all tasks when pretraining over a larger corpus (adding OPEN TEXT). The table also shows that Naive-PMI Masking, based the straightforward extension in eq. 2o the standard bivariate PMI, significantly falls behind our more nuanced definition in eq. and is often on par with Random-S Masking.
7

---

BERT Base with different maskings
SQuAD2.0
RACE
GLUE
EM
F1
Acc.
Avg
1M training steps MIKIPEDIA+BOOKCRRPTT
(6G))
Random-Tokk Masking
76.4/-
79.6/-
67.8/66.2
83.
Random-Span Masking
77.1/-
80.3/-
68.6/66.9
83/
-MMMasinn PMI-Masking
78.2/-
81.3/
69.7/67.8
84.1/
78.5/-
81.4/
70.1/68.4
84.1/-
2.4M training steps on
Random-Span Masking
79.7/80.0
82.7/82.8
71.9/69.5
84.8/79.7
Naive- PM---aaM-
80.3/80.2
83.2/83.2
71.7/69.8
84.5/80.0
PMI- PMI-Masking
80.2/80.9
83.3/ 83.6
72.3/70.9
84.7/80.3
2.4M training steps WIKIPEDIA+
BETEEDDA--OOOOORUUSS
OPPEN
OPEN WEB TETEEX(54GG)
Random-Sper Span Masking
80.1/80.4
83.2/83.3
74.0/72.2
85.1/80.1
Raive-PMI-Masking PMI-Masking
80.4/80.0
83.3/83.0
73.9/71.4
85.6/80.3
80.9/82.0
83.9/84.9
74.8/73.2
86.0/80.8
Table 2: Dev/Test performance on the SQuAD, RACE, and GLUE benchmarks of BERT Base sized We report EM (exact match) and F1 scores
models pretrained and evaluated according to section f for SQuAD2 and accuracy for RACE. For GLUE we report the average scores the development set and the official leaderboard scores on the test set (see the per-task scores in the appendix).
PMI vs Prior BASE MIMS
Corpus size
Batch x Steps Examples
RACE dev/test
PMI vs n-grams in vocabulary
AMBERT (Zhang & Li 2020
47G
1024 x 0.5M 512G
68.97/66.8 88
PMI-Masking
16G
256 x 1M 256M
70.1/68.4
PMI vs Random-Span Masking
SpanBERTBASE BASE (Johh et al. 2020
16G
256 2.4M 614.4M
70.5/68.7
PMI-Masking
16G
256 2.4M 614.4M 72.3/70.9
PMI vs Random-Tokke Token Masking with 3X more data and 6X more training examples
ROBERTaBASE (Liu al. 2019
160G
8K 0.5M 4G
74.9/73
PMI-Masking
54G
256 x 2.4M = 614.4M 74.8/73.2
Table Comparing the RACE scores of our PMI-Masked models with comparable published Base- sized models. The scores of prior MLMs were attained by finetuning released models in the same setup of the PMI-Masked models (Section 4p except for those marked in reported [hhag Li (2020). The number of examples reflects the amounts of text examined during training, as all prior models train over the same sequence length as our PMI-Masked models, namely 512. AMBERT was trained over WIKIPEDIA+OPENWWIE TEXT (47G), SpanBERT over WIKIPEDIA+ CORPS (16G), and RoBERTa over WESEEDD-FBOOOCOOUUURRREEEETTTTS (160G see details in
Liu al. 2019)).
We also compared our PMI-Masking Base-sized models to published Base-sized models (Table and again saw PMI-Masking increase both pretraining efficiency and end-of-training downstream performance. ZZhang& & Li (020) trained their 'AMBERT" model over vocabulary of n-grams in parallel to the regular word/subword level vocabulary, performing the hard task n-gram predic- tion in parallel to the easy Landom-Tooenn level prediction task during pretraining, This approach yielded model with 75% more parameters than the common Base size of our PMI-Masking model. By using the PMI-masking scheme regular BERT architecture and vocabulary, attained significantly higher score on the RACE benchmark. despite training over a corpus 3x smaller and showing the model 2x fewer examples during pretraining.
8

---

Joshi et al. (2020) and Liu et al. (019)) only reported scores for SpanBERT and RoBERTa (respec- tively) for Large-sized models in their original papers, but did release weights for Base-sized models. We fine-tuned these models on the RACE development set via the same fine-tuning procedure employed for our PMI-Masking models (described in Section 4 and evaluated the best performing model on the publicly available RACE test set. A PMI-Masking Base-sized model scored more than 2 points higher than the SpanBERTBASE trained by Random-Span Masking over the same pretrain- ing corpus when shown the same number of examples. Remarkably, PMI-Masking Base-sized model scored slightly higher than RoBERTàBASE trained by Random- Kaneem-Token Masking, even though RoBERTa was given access to a pretraining corpus 3x larger and shown 6x more training examples.
Lastly, we note that the measure of Single-Token perplexity not indicative of downstream performance, when reported for models trained with different masking schemes. Compar- ing the adjacent table with the downstream evaluation of the same models in Table 2 it is clear that the ability to predict single tokens from context is not correlated with performance. This reinforces our observation that by minimizing their train- ing objective, standard MLMs, which mask tokens randomly, train excel on relatively many easy tasks that do not reflect the knowledge required for downstream understanding.
Single-Toker Token Perplexity
Random-Tooke Masking
2.96
Random-Spe Span Masking
4.30
Waive-PMI-Masking PMI-Masking
7.35
21.85
The Single-Token per- plexity of MLMs trained for IM steps over WIKL+BOOKCORPUS.
Table 4:
6 CONCLUSION
Bidirectional language models hold the potential to unlock greater signal from the training data than unidirectional models (such as GPT). BERT-based MLMs are historically the first (and still the most prominent) implementation of inherently bidirectional language models, but they come price. A hint of this price the fact that Single-Toke Token perplexity, which captures the ability to predict single tokens and which has natural probabilistic interpretation in the autoregressive unidirectional case, ceases to correlate with downstream performance across different MIMs (see Table 4). This means that the original MLM task, which is focused on single token prediction, should be reconsidered. This has been the focus of this paper, which points to the inefficiency of random-token masking, and offers PMI-masking as an alternative with several advantages:
It is a principled approach, based on a nuanced extension of binary PMI to the n-ary case. (ii) It leads to better downstream performance, for example it surpasses RoBERTa (which uses vanilla random token masking) on the challenging reading comprehension RACE test with 6x less training over a 3x smaller corpus, and it dominates the more naive, heuristic approach of random span masking at any point during pretraining, matches its end-of-training performance halfway during its own pretraining, and at the end of training improves on it by 1-2 points across a variety of downstream tasks. Perhaps due to their conceptual simplicity, unidirectional models were the first to break the 100B parameter limit with the recent GPT3 (Brown et al. 2020). Bidirectional models will soon follow, and this paper accelerate their development by offering a way to significantly lower their training costs while boosting performance.
REFERENCES
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop recognising textual entailment, pp. 6 64,, 2006.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. /4 165, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Gazpio, and Lucia Specia. Semeval-200 -217 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation (SemEval), pp. 1-14, Vancouver, Canada, 2017.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop. pp. 177-190. Springer, 2005.

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Com- putatiioaa Linguistics (NAACL), 2019a.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Original bert github reposi- tory. https://githubb.cm http:://gillub..oo/ggggg//// oeger//eee.mmmmoooeeeeeerrrreer 2019b.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Doug Downey, Matthew Broadhead, and Oren Etzioni. Locating complex named entities in web text. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI'07, pp- 2733-2739, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.
R.M Fano. Transmission of Information: A Statistical Theory of Communications. Transmission of Information: A Statistical Theory of Communications. M.L.T. Press, 1961. ISBN 9780262060011. URL chttps://booke google co. 11/bookssoss 2id.ooossss..VVSTAAAAAAAAAA
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL PASCAI. textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wing-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint WxXiv:2002.08909 2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 77, 2020.
Ioannis Korkontzelos, Ioannis P Klapaftis, and Suresh Manandhar. Reviewing and evaluating auto- matic term recognition techniques. In International Conference on Natural Language Processing, pp. 248-259. Springer, 2008
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: 2017.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pre- training approach. arxiv preprint arXiv: 19t..//977.1999999 16692, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv preprint, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 100.010833 2019.
Pranay Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.000 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint 180iv: 03822, 2018.
Carlos Ramisch, Vitor De Araujo, and Aline Villavicencio. A broad evaluation of techniques for automatic acquisition of multiword expressions. In Proceedings of ACL 2012 Student Research Workshop, pp. 1-6, 2012.
10

---

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
Yu Stephanie Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xinlun Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.
Tim Van de Cruys. Two multivariate generalizations of pointwise mutual information. In Proceed- ings of the Workshop on Distributional Semantics and Compositionality, pp. 16-20, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NIPS), 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv 2018.
preprint
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Interna- tional Conference on Learning Representations (ICLR), 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 12471, 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. North American Association for Computational Lin- guistics (NAACL), pp. 1112-1122, 2018.
Xinsong Zhang and Hang Li. Ambert: A pre-trained language model with multi-graine tokeniza- tion. arXiv preprint arXiv:2008. 11869, 2020.
Willem Zuidema. What are the productive units of natural language grammar? a dop approach to the automatic identification of constructions. In Proceedings of the Tenth Conference Compu- tational Natural Language Learning (CoNLL-X), pp. 29-36, 2006.
11

---

recall r
complement-recall c
f
cr.c TCC
0.75
0.25
200
400
600
800
1000
1200
1400
1600
1800
2000
n-gram list size (in thousands)
Figure 3: Quality measures of top ranking PMI, n-grams lists increased in increments of 50K. The masking vocabulary size was chosen such that it includes as many n-grams labeled collocation as possible, while not including too many n-grams labeled as not collocation, in internally constructed test set detailed below. set that appear within the given list (recall), is the percent of all negatively labeled examples from the test set that do not appear within the given list (complement-recal)) which both and are high enough, and employ f as a measure for this, finally choosing a list size of 800K.
the percent of all positively labeled examples from the test
We aim for a list size for
A DETERMINING THE MASKING VOCABULARY SIZE
The PMI, measure, defined in eq. provides an n-gram ranking function that is intended to rank an n-gram higher its components are more indicative of one another. However, this measure alone is not enough for composing masking vocabulary: we need to decide on its size M (the masking vocabulary will be composed of top-M ranked n-grams). One could advocate for an ablation study in which M is varied, and models pretrained per M and evaluated. This be done in future work, and perhaps an even stronger result can be shown for PMI-Masking with masking vocabulary size chosen by such optimization.
As a proxy, we determined the masking vocabulary size M via a small scale evaluation of an n- gram's "collocation quality" set composed of 1000 n-grams that we labeled either as collocation or not collocation (avail- able upon request). We did that by choosing random 10 words with frequency above 10000 in WIKIPEDIA+BBOKCCOO
function of its PMI, rank. Specifically,
created an ad hoc test
and for each word sampled 25 n-grams per length E {2,3, 4. 5} that contain it. Finally, we manually labeled each collected n-gram, where the textbook defini- tion of collocation was given to the annotators (the annotator agreement was 80% over 100 shared examples).
Then, we increased list size M in steps of 50K, adding n-grams from the top ranking PMI, downwards. For each -ized list we computed two different scores on the test set. The first the recall of the positive examples in the list, denoted r: the percent of all positively labeled examples from the test set that appear within the given list. The second the recall of the negative examples in the complement of the list. dubbed complement-recall denoted the percent of all negatively labeled examples from the test set that do not appear within the given list. By these definitions, the recall starts low and increases with list size and the complement-recal follows an opposite trend, as can be seen in Figure 3. Our desired masking vocabulary size should yield a list with many n-grams labeled collocation while containing little n-grams labeled not a collocation. we define r+c as a measure for optimization which balances the two requirements, and Figure shows that this measure is highest at sizes of around 700-800, so we set the masking vocabulary size to be 800K.
= 2nc
12

---

B PRETRAINING AND MODEL DETAILS
Table shows the pretraining hyper-parameters we used, as well as the architecture specifies, both follow the standard implementation of BERT.
Number of Layers
12
Hidden Size
768
Sequence Length
512
FFN Inner Hidden Size
3072
Attention Heads
12
Attention Head Size
64
Dropout
0.1
Attention Dropout
0.1
Warmup Steps
10,000
Peak Learning Rate
le-4
Batch Size
256
Weight Decay
0.01
Initializer Range
0.02
Learning Rate Decay
Linear
Adam
le-6
Adam 81
0.9
Adam B2
0.999
Table 5: Hyper-parameters of the architecture and pretraining, complementing the description in Section
C EVALUATION OF DIFFERENT CHECKPOINTS DURING PRETRAINING
Tables 6 and 7 respectively present the development set scores SQuAD2.0 and RACE, attained for models at different checkpoints during pretraining. The SQuAD2.0 scores are depicted in Figures and
pretraining checkpoint:
200
400
600
800
1000
1200
1600
2000
2400
1M training steps WIKIPEDIA+
BoOK CORPUS
Random-Token Masking
74.4
76.7
77.9
78.9
79.3
Whole- Word Masking
74.8
77.9
78.4
79.1
79.6
Frequency-Mase Masking
75.5
78
79.2
79.4
79.7
Random-Span Masking
74.8
77.4
78.9
79.6
80.0
PMI-Masking
77.0
78.8
80.3
81.1
81.3
2.4M training steps WIKIPEDIA+
CORPS
Random-Spal Masking
75.8
78.4
79.8
80.4
80.9
81.8
82.2
82.9
83.1
PMI-Mas MMI-Masking
77.2
79.8
81.0
81.6
81.8
82.4
83.1
83.0
83.3
2.4M training steps
Random-Span Masking
77.1
78.9
80.9
81.0
81.8
82.3
82.7
83.1
83.2
PMI-Masking
78.4
80.7
82.1
82.4
82.9
83.3
83.8
84.0
84.3
Table 6: The Fl score on the SQuAD2.0 development set of models taken at various checkpoints along the pretraining of BERT Base sized models trained with different masking schemes. These scores are depicted in Figures Tan 2 We finetuned on SQuAD2.0 with batch size of 32 and learning rate of 10 over epochs without early stopping. We did this for random initializations of the task's head and the reported score is an average of the three middle scores.
13

---

pretraining checkpoint:
200
400
600
800
1000
1200
1600
2000
2400
1M training steps on WIKIPEDIA+BOOKCOOO CORPUS
Random-Token Masking
61.2
64.3
65.6
66.4
67.1
Whole Word Masking
62.0
64.9
66.0
67.0
67.8
Drequency-Masking Random-Span Masking
63.7
65.7
67.3
68.5
68.8
61.7
64.7
66.8
67.9
68.0
MMI-Masking
63.5
66.8
68.4
68.9
69.7
2.4M training steps on WIKIPEDIA+BOOKCOKEE
Random-Spor Masking
62.3
64.3
65.6
67.8
69.0
68.9
70.3
71.0
71.4
PMI- MMI-Masking
63.6
66.7
67.3
68.5
69.2
70.4
70.5
71.2
72.2
2.4M training steps on TIKIPEDDA+-BOOCORPPUSS
TEXXT
Random-Span Masking
64.6
67.0
69.2
69.9
70.5
71.3
72.9
73.5
73.4
PMI-Masking
66.5
68.6
70.7
71.4
72.4
72.5
73.6
74.1
74.5
Table The aceuracy score on the RACE development of models taken various checkpoints along the pretraining of BERT Base sized models trained with different masking schemes. We finetuned on RACE with batch size 32 and learning rate of 3 10-5 over epochs without early stopping. We did this for 5 random initializations of the task's head and the reported score is an average the three middle scores.
D GLUE TASKS AND DETAILED SCORES
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2019) consists
of sentence-level tasks. Sentence-lever classification tasks: CoLA (Warstadt et 2018 (evalu- ating linguistic acceptability) and SST-2 (Socher et al. 2013) (sentiment classífication). Sentence- pair similarity tasks: MRPC (Dolan & Brockett] 2005) (binary paraphrasing classification task), STS-B CCr et al. 2017 (graded similarity scoring task), and QQF (binary paraphrasing classi- fication task). Natural language inference tasks: MNLI (Williams et al.] [2018). QNLI (Rajpurkar al. 2016), RTE (Dagan et al. (2005: (Ba-Haim al. [2006] Giampiccolo al. 2007) and WNLI [Levesque et al. 2011). Table shows the detailed per-task scores of our examined models.
GLUE
MNLI QNLI
QQP
RTE
SST
MRPC
CoLA
STS
Avg
IM training steps on Wikipedia+ BookCor on dev
Random-Spi Span Masking
84.0/-
91.4
90.8
69.0
92.8
88.5
58.5
88.9
83.0
Siive-PMI-Maskinn PMI- MMl-Masking
85.1/-
91.9
91.0
74.0
93.3
88.2
60.3
89.3
84.1
85.2/-
91.8
91.0
72.2
92.7
89.7
60.6
89.3
84.1
2.4M rraining steps on SEKIPEDIA+BBOKOOREE on
Random- Span Masking
85.7/84.7
92.9
89.4
69.8
93
85.4
56.5
86.6
79.7
Raive-PMI-Masheee PMI- Ma--Mag
85.5/85.3
92.2
89.2
68.9
93.6
85.4
59.4
87.3
80.0
85.3/85.0
92.0
89.2
69.0
94.0
85.6
61.8
86.8
80.3
2.4M training steps on
WEBTEXT; on test
Random Mandom-Span Masking
86.3/85.1
92.2
89.4
71.1
94.6
85.6
56.8
87.2
80.1
Pirre-MM-Maskkinn PMI- MMI-Masking
86/85.4
91.7
89.4
69.2
95.1
87.8
57.5
87.9
80.3
86.6/85.8
93.1
89.5
72.9
94.7
87.7
57.4
87.7
80.8
Table 8: Results on the different tasks of the GLUE benchmark. For all tasks the scores reflect accuracy, except for STS-B (spearman score) and CoLA (Mathews Correlation). For results reported on the development set (1M training steps), the average score is simply the average of reported scores. For results reported the test sets (2.4M training steps), the average score is the official GLUE leaderboard score. The official score includes averaging of Fl scores for QQP and MRPC, well the default majority submission score of 65.1 for WNLII
14


Reviews:
[{"review_id": "ux8v9DFX05N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a variant on the MLM training objective which uses PMI in order to determine which spans to mask. The idea is related to recently-proposed Whole Word Masking and Entity-based masking, but the authors argue the PMI-based approach is more principled. The method is straightforward--it involves computing PMIs for ngrams (in this case, up to length 5) over the training corpus, and then preferring to mask entire collocational phrases rather than single words during training. The intuition is that masking single words allows models to exploit simple collocations, thus optimizing their training objective without learning longer-range dependencies or higher level semantic features of the sentences, and this makes training less efficient than it could be. One contribution of the paper is a variant on the PMI metric that performs better for longer phrases by reducing the scores of phrases that happen to contain high-PMI subphrases, e.g. \"George Washington is\" should not have a high score despite the fact that \"George Washington\" does have a high score.\n\nThe authors compare their method against vanilla BERT with random masking, as well as against recently proposed variants such as SpanBERT and AMBERT, and show consistent improvements in terms of final performance as well as better efficiency during training. By way of analysis, the authors also make an argument that token-level perplexity is not correlated with downstream performance. This is an interesting point to make, though they do not expound upon it in this paper. \n\nStrengths:\n\n* The proposed method is simple and principled\n* The empirical results show consistent improvement on standard benchmark tasks\n* The proposed variation the PMI metric is a nice sub-contribution\n\nWeaknesses:\n\n* A somewhat marginal contribution, its not significantly different from the variants proposed previously (e.g., SpanBERT, entity masking)\n* The evaluation focuses purely on benchmark tasks which are known to have flaws (e.g., the current \"superhuman\" performance on these tasks already makes gains on them suspect). I'd have liked to some more analysis/discussion of the linguistic consequences of this new objective. See more specific comments below.\n\nAdditional Comments/Questions:\n\nI am curious about the more general effect of this training objective on the models linguistic (and particularly syntactic) knowledge. E.g., can you say more about how often the model sees unigrams being masked and how the distribution of these unigrams differs from what would be seen if we did random masking? I ask because I could imagine that this objective has a noticeable effect on the masking of function words (e.g., preposition occurring more often in collocations, pronouns and determines maybe less often) and thus the model might get differing access to these words in isolation. Since function words carry a lot more signal about syntactic structure than do content words and phrases (of the type you are capturing in your PMI metric), I'm very curious if there are some tradeoffs (or, possibly, additional advantages) that comes with your method that are not reflected by the benchmark performance metrics. Squad and GLUE are going to favor knowledge of things like entities and events, and capture very little about more nuanced linguistic reasoning, so reporting performance on some more recently released challenge sets, or using some probing studies, or at least just giving some analysis of win/loss patterns, would be very informative for assessing the contribution of this paper to NLP more generally. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QkgohmxwziL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes an improvement to how tokens are selected for masking in pre-training large masked language models (BERT and family). Specifically, it stipulates that purely random choice of words (or word pieces) makes the MLM task insufficiently hard. It then goes on to propose a data-driven approach for selecting n-grams to mask together. The approach, based on an extension of pointwise mutual information for n-grams, is shown to outperform random token and random spans masking strategies on performance of downstream tasks.\n\n### Strong and weak points\n\nStrong points:\n- The paper is very well written throughout, and easy to follow.\n- The problem is well motivated with empirical evidence. I think Section 2 demonstrates well the case for random masking being too easy.\n- The multivariate version of PMI proposed is simple and well motivated.\n- The evaluation experiments are convincing the results are robust across the tasks shown,\n\nWeak points:\n - The one main drawback in this study is the lack of comparison with entity-based techniques for masking.  In particular [1] has recently defined “salient span masking” based on named entity recognition and dates. Salient span masking has been adopted in [2] where it is shown to boost performance of open-domain question answering by 9+ points (Table1 of [2], models tagged with “+ SSM”).\nI think it would be extremely interesting to compare these techniques (SSM specifically, but entity-based techniques in general) with PMI-Masking. Specifically, it is currently unclear whether the PMI based n-gram masking vocabulary simply ends up rediscovering popular named entity mentions, or whether there are more interesting sub-phrases (e.g., idiomatic sub-phrases) that a NER system would not select. Finally, it would be interesting to empirically test whether these extra non-entity n-grams provide further performance boost over the entity-based “salient span masking”.\n\n[1] REALM: Retrieval-Augmented Language Model Pre-Training (https://arxiv.org/abs/2002.08909).\n[2] How Much Knowledge Can You Pack Into the Parameters of a Language Model? (https://arxiv.org/abs/2002.08910)\n\n### Recommendation\n\nI recommend this paper for acceptance. The analysis and ideas throughout the paper are well executed. I also think the topic should be of high interest to the ICLR and NLP communities, given the importance of MLM pre-training on most state-of-the-art models at the moment. Despite the lack of comparison with entity-based techniques, having a statistically principled alternative, solely based on co-occurrence, without linguistic grounding, seems interesting.\n\n### Questions for authors\n\n1. The main question here related to the entity-based approaches discussed above. I think it would be interesting to address this issue given how closely related it is to this work, and the good performance the cited papers demonstrate using it. I can think of a couple of ways to address this comparison: (1) qualitative analysis of the masking vocabulary to better understand the differences between “PMI ngrams” and entity mentions, (2) experimental analysis incorporating some entity-based masking into the experiments in the paper.\n\n2. Irrespectively of how you choose to address the entity-based comparison, I was interested in some analysis, or sampling, of the PMI-Masking vocabulary to understand what type of n-grams are being selected (entity mentions, idiomatic phrases, noun-phrases, etc.). Would be interesting if you could make this vocab available or add a small sample in the appendix.\n\n3. Throughout the paper there is an assumption that contiguous words are considered for masking. This was not immediately clear in the beginning of the paper (I realized it only in Section 3.2 with - “What about contiguous spans of more than two tokens?”).  But one question came to mind: what about correlated non-contiguous spans? For example “eigenvalue” and “eigenvector” are unlikely to be present in the same n-gram, but have reasonably high chances of showing up together in the same passage. Have you considered extending this work to non-contiguous spans? Is there any expectation that this would help learning, or is it just a bad idea?\n\n4. Was there an attempt to mix masking strategies during pre-training? Although Table 6 is convincing in demonstrating that single-token perplexity is not correlated with performance of downstream tasks, the differences seem curious. One is left wondering if there is any benefit in adding a small number of “easy” masking cases (i.e., random-tokens or random-spans)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "PnB6nHTEU1e", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a masking strategy for training masked language models (MLM). The proposed strategy builds on previous approaches that mask semantically coherent spans of tokens (such as entire words, named entities, or spans) rather than randomly masking individual tokens. Specifically, the proposed method computes the PMI of spans (and the generalization for spans of size >2) over the pretraining corpus, and randomly masks from among the 800K spans (lengths 2-5) with the highest PMI. Masking based on PMI removes the ability for the model to rely on highly local signals to fill in the mask and instead focus on learning higher level semantics. They motivate this hypothesis with an experiment demonstrating that as the size of the WordPiece vocabulary decreases (and words are more frequently split into multiple tokens rather than being their own token), the transfer performance of the resulting MLM decreases. However, using whole-word masking with this same vocabulary size recovers much of the original performance, indicating that allowing the model to rely on these strong local signals harms the transfer quality of the resulting model.\n\nExperiments: The paper evaluates on three standard NLU benchmarks: SQuAD2, GLUE, and RACE. They compare their PMI-based algorithm against random token masking, random span masking, and a naive version of their PMI masking strategy. All baselines are implemented within a single codebase. Their strategy outperforms random-span masking on SQuAD throughout pretraining, though the latter does close the gap on a smaller pretraining corpus (16GB). They show this gap remains if they use a larger pretraining corpus (54GB), as would likely be the case with a large-scale pretraining experiment. The resulting models also consistently outperform all other baselines on all tasks. They do additionally compare to outside models (AMBERT, SpanBERT, RoBERTa) and show their PMI-based models outperform or perform similarly to these models.\nOverall, this paper introduces a solid theoretical basis for existing and new methods for training masked language models. They present robust experiments demonstrating the efficacy of their method under various settings.\n\n\n1. Missing citation for RACE and original SQuAD dataset?\n2. I assume this doesn’t happen often, but is it ever the case that a training example does not have any mask-able spans?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7X_u7N79oXS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach for masked language modeling, where they mask wordpieces together which have high PMI. The idea is relatively simple, has potential for high impact through broad adoption, and the paper is clearly written with extensive experiments.\n\nThe experiments on GLUE, SQuAD, and RACE are very well set up. For example, evaluating multiple learning rates for each downstream task is expensive, but really adds to confidence I have in the results. Reporting the best median dev score over five random initializations per hyperparameter, then evaluating that same model on the test set, definitely improves the reproducibility of the results. In addition, showing how performance is affected by the amount of pretraining data is very useful, and the experiments range from small scale to large scale. The ablations adjusting the vocabulary size (which, in turn, changes the size of wordpiece tokens) is a valuable contribution, and I would have asked to see something like this if it wasn't included. Table 4 is a nice addition -- it's interesting that the MLM loss is not predictive of downstream performance.\n\nI suspect this approach will become widely adopted (or built upon) in future work pretraining language models. I give this paper an 8, only taking off points because the idea is relatively intuitive and doesn't really open a broad new area for future work. I don't see any obvious methodological flaws, which frankly, we can find in most papers.\n\nI would be interested in seeing if this reduced the variance of the fine-tuning process. That might be something the authors could include for the camera ready, maybe in the appendix.\n\nEdit: After reviewing the author response, I will keep my score as it is. I believe the paper should be accepted. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new and experimentally useful way to propose masked spans for MLM pretraining, by masking spans of text that co-occur more often than would be expected given their components - ie that are statistically likely to be non-compositional phrases.

The authors should make some attempt to connect their PMI heuristic with prior methods for statistical phrase-finding and term recognition, eg https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-439.pdf or https://link.springer.com/chapter/10.1007/978-3-540-85287-2_24 in the final paper.
Decision: Accept (Spotlight)