Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces an approach for masked language modeling, where they mask wordpieces together which have high PMI. The idea is relatively simple, has potential for high impact through broad adoption, and the paper is clearly written with extensive experiments.

The experiments on GLUE, SQuAD, and RACE are very well set up. For example, evaluating multiple learning rates for each downstream task is expensive, but really adds to confidence I have in the results. Reporting the best median dev score over five random initializations per hyperparameter, then evaluating that same model on the test set, definitely improves the reproducibility of the results. In addition, showing how performance is affected by the amount of pretraining data is very useful, and the experiments range from small scale to large scale. The ablations adjusting the vocabulary size (which, in turn, changes the size of wordpiece tokens) is a valuable contribution, and I would have asked to see something like this if it wasn't included. Table 4 is a nice addition -- it's interesting that the MLM loss is not predictive of downstream performance.

I suspect this approach will become widely adopted (or built upon) in future work pretraining language models. I give this paper an 8, only taking off points because the idea is relatively intuitive and doesn't really open a broad new area for future work. I don't see any obvious methodological flaws, which frankly, we can find in most papers.

I would be interested in seeing if this reduced the variance of the fine-tuning process. That might be something the authors could include for the camera ready, maybe in the appendix.

Edit: After reviewing the author response, I will keep my score as it is. I believe the paper should be accepted. 
review_title: Clearly written, simple idea, showed through extensive (well set-up) experiments to be very effective

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper proposes a variant on the MLM training objective which uses PMI in order to determine which spans to mask. The idea is related to recently-proposed Whole Word Masking and Entity-based masking, but the authors argue the PMI-based approach is more principled. The method is straightforward--it involves computing PMIs for ngrams (in this case, up to length 5) over the training corpus, and then preferring to mask entire collocational phrases rather than single words during training. The intuition is that masking single words allows models to exploit simple collocations, thus optimizing their training objective without learning longer-range dependencies or higher level semantic features of the sentences, and this makes training less efficient than it could be. One contribution of the paper is a variant on the PMI metric that performs better for longer phrases by reducing the scores of phrases that happen to contain high-PMI subphrases, e.g. "George Washington is" should not have a high score despite the fact that "George Washington" does have a high score.

The authors compare their method against vanilla BERT with random masking, as well as against recently proposed variants such as SpanBERT and AMBERT, and show consistent improvements in terms of final performance as well as better efficiency during training. By way of analysis, the authors also make an argument that token-level perplexity is not correlated with downstream performance. This is an interesting point to make, though they do not expound upon it in this paper. 

Strengths:

* The proposed method is simple and principled
* The empirical results show consistent improvement on standard benchmark tasks
* The proposed variation the PMI metric is a nice sub-contribution

Weaknesses:

* A somewhat marginal contribution, its not significantly different from the variants proposed previously (e.g., SpanBERT, entity masking)
* The evaluation focuses purely on benchmark tasks which are known to have flaws (e.g., the current "superhuman" performance on these tasks already makes gains on them suspect). I'd have liked to some more analysis/discussion of the linguistic consequences of this new objective. See more specific comments below.

Additional Comments/Questions:

I am curious about the more general effect of this training objective on the models linguistic (and particularly syntactic) knowledge. E.g., can you say more about how often the model sees unigrams being masked and how the distribution of these unigrams differs from what would be seen if we did random masking? I ask because I could imagine that this objective has a noticeable effect on the masking of function words (e.g., preposition occurring more often in collocations, pronouns and determines maybe less often) and thus the model might get differing access to these words in isolation. Since function words carry a lot more signal about syntactic structure than do content words and phrases (of the type you are capturing in your PMI metric), I'm very curious if there are some tradeoffs (or, possibly, additional advantages) that comes with your method that are not reflected by the benchmark performance metrics. Squad and GLUE are going to favor knowledge of things like entities and events, and capture very little about more nuanced linguistic reasoning, so reporting performance on some more recently released challenge sets, or using some probing studies, or at least just giving some analysis of win/loss patterns, would be very informative for assessing the contribution of this paper to NLP more generally. 
review_title: Clear empirical gains and well written, but somewhat incremental contribution overall

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper presents a masking strategy for training masked language models (MLM). The proposed strategy builds on previous approaches that mask semantically coherent spans of tokens (such as entire words, named entities, or spans) rather than randomly masking individual tokens. Specifically, the proposed method computes the PMI of spans (and the generalization for spans of size >2) over the pretraining corpus, and randomly masks from among the 800K spans (lengths 2-5) with the highest PMI. Masking based on PMI removes the ability for the model to rely on highly local signals to fill in the mask and instead focus on learning higher level semantics. They motivate this hypothesis with an experiment demonstrating that as the size of the WordPiece vocabulary decreases (and words are more frequently split into multiple tokens rather than being their own token), the transfer performance of the resulting MLM decreases. However, using whole-word masking with this same vocabulary size recovers much of the original performance, indicating that allowing the model to rely on these strong local signals harms the transfer quality of the resulting model.

Experiments: The paper evaluates on three standard NLU benchmarks: SQuAD2, GLUE, and RACE. They compare their PMI-based algorithm against random token masking, random span masking, and a naive version of their PMI masking strategy. All baselines are implemented within a single codebase. Their strategy outperforms random-span masking on SQuAD throughout pretraining, though the latter does close the gap on a smaller pretraining corpus (16GB). They show this gap remains if they use a larger pretraining corpus (54GB), as would likely be the case with a large-scale pretraining experiment. The resulting models also consistently outperform all other baselines on all tasks. They do additionally compare to outside models (AMBERT, SpanBERT, RoBERTa) and show their PMI-based models outperform or perform similarly to these models.
Overall, this paper introduces a solid theoretical basis for existing and new methods for training masked language models. They present robust experiments demonstrating the efficacy of their method under various settings.


1. Missing citation for RACE and original SQuAD dataset?
2. I assume this doesn’t happen often, but is it ever the case that a training example does not have any mask-able spans?

review_title: Solid theoretical and practical contribution, with experiments to back it up

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary

This paper proposes an improvement to how tokens are selected for masking in pre-training large masked language models (BERT and family). Specifically, it stipulates that purely random choice of words (or word pieces) makes the MLM task insufficiently hard. It then goes on to propose a data-driven approach for selecting n-grams to mask together. The approach, based on an extension of pointwise mutual information for n-grams, is shown to outperform random token and random spans masking strategies on performance of downstream tasks.

### Strong and weak points

Strong points:
- The paper is very well written throughout, and easy to follow.
- The problem is well motivated with empirical evidence. I think Section 2 demonstrates well the case for random masking being too easy.
- The multivariate version of PMI proposed is simple and well motivated.
- The evaluation experiments are convincing the results are robust across the tasks shown,

Weak points:
 - The one main drawback in this study is the lack of comparison with entity-based techniques for masking.  In particular [1] has recently defined “salient span masking” based on named entity recognition and dates. Salient span masking has been adopted in [2] where it is shown to boost performance of open-domain question answering by 9+ points (Table1 of [2], models tagged with “+ SSM”).
I think it would be extremely interesting to compare these techniques (SSM specifically, but entity-based techniques in general) with PMI-Masking. Specifically, it is currently unclear whether the PMI based n-gram masking vocabulary simply ends up rediscovering popular named entity mentions, or whether there are more interesting sub-phrases (e.g., idiomatic sub-phrases) that a NER system would not select. Finally, it would be interesting to empirically test whether these extra non-entity n-grams provide further performance boost over the entity-based “salient span masking”.

[1] REALM: Retrieval-Augmented Language Model Pre-Training (https://arxiv.org/abs/2002.08909).
[2] How Much Knowledge Can You Pack Into the Parameters of a Language Model? (https://arxiv.org/abs/2002.08910)

### Recommendation

I recommend this paper for acceptance. The analysis and ideas throughout the paper are well executed. I also think the topic should be of high interest to the ICLR and NLP communities, given the importance of MLM pre-training on most state-of-the-art models at the moment. Despite the lack of comparison with entity-based techniques, having a statistically principled alternative, solely based on co-occurrence, without linguistic grounding, seems interesting.

### Questions for authors

1. The main question here related to the entity-based approaches discussed above. I think it would be interesting to address this issue given how closely related it is to this work, and the good performance the cited papers demonstrate using it. I can think of a couple of ways to address this comparison: (1) qualitative analysis of the masking vocabulary to better understand the differences between “PMI ngrams” and entity mentions, (2) experimental analysis incorporating some entity-based masking into the experiments in the paper.

2. Irrespectively of how you choose to address the entity-based comparison, I was interested in some analysis, or sampling, of the PMI-Masking vocabulary to understand what type of n-grams are being selected (entity mentions, idiomatic phrases, noun-phrases, etc.). Would be interesting if you could make this vocab available or add a small sample in the appendix.

3. Throughout the paper there is an assumption that contiguous words are considered for masking. This was not immediately clear in the beginning of the paper (I realized it only in Section 3.2 with - “What about contiguous spans of more than two tokens?”).  But one question came to mind: what about correlated non-contiguous spans? For example “eigenvalue” and “eigenvector” are unlikely to be present in the same n-gram, but have reasonably high chances of showing up together in the same passage. Have you considered extending this work to non-contiguous spans? Is there any expectation that this would help learning, or is it just a bad idea?

4. Was there an attempt to mix masking strategies during pre-training? Although Table 6 is convincing in demonstrating that single-token perplexity is not correlated with performance of downstream tasks, the differences seem curious. One is left wondering if there is any benefit in adding a small number of “easy” masking cases (i.e., random-tokens or random-spans)?
review_title: Simple and elegant approach to improving MLM showing good results.

Meta-review: This paper describes a new and experimentally useful way to propose masked spans for MLM pretraining, by masking spans of text that co-occur more often than would be expected given their components - ie that are statistically likely to be non-compositional phrases.

The authors should make some attempt to connect their PMI heuristic with prior methods for statistical phrase-finding and term recognition, eg https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-439.pdf or https://link.springer.com/chapter/10.1007/978-3-540-85287-2_24 in the final paper.
Decision: Accept (Spotlight)