Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper identified the false negative issue in the pre-trained language models and proposed two methods to counteract it. The first method replaces the cross-entropy loss in the standard MLM objective with the cosine distance between the representations of predicted token and the ground truth. The second method prunes the gradients from the false positive tokens, where such tokens are identified from synonym alternatives of WordNet. Empirical results on GLUE and SQuAD show that the proposed solution improves ELECTRA-small and ELECTRA-base models for both performance and robustness.
main_review: Strengths:
1) The paper identified the false negatives issue in PrLMs and could possibly inspire other researcher to work on this.
2) The experiments are conducted on two benchmarks and the proposed methods show consistent gains over a strong baseline model.

Weakness:
1) For the soft regularization in equation (6), I am not sure how this loss can be backpropagated since the token prediction step is a discrete argmax function.
2) For the hard correction, it is unclear whether the WordNet will also bring false positive, i.e. a synonym which should not fill in the mask.
3) The experiments only cover one pretrained language model - ELECTRA. Could this method be applied on other MLM PrLMs such as RoBERTa?
4) I also expect some statistics of the hard corrections in the pretraining experiment.

Presentation Issues:
1) In table 2, the test set result of ELECTRA_small on QQP is the best but the number is not bold.
summary_of_the_review: The paper proposed two simple methods for a novel issue in PrLMs. The empirical results are promising in two benchmarks with ELECTRA model. However, more technical description, empirical analysis, and experimental result are still needed to fully support the claim of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper identifies the largely neglected false negative problem in training or pretraining language models. For many cases in training LM, the training objective only uses one single token as the target, while treating other meaningful tokens as equally negative. To tackle this problem, the author proposes two methods: 1) the Soft Regularization method minimizes the embedding distance between the predicted and gold tokens; 2) the Hard Correction employs WordNet and maximizes all the synonyms of the gold token. They demonstrate consistent gains and better robustness by applying this to pretraining ELECTRA, and testing on GLUE and SQuAD downstream tasks. 
main_review: *Strengths:
1. The false negative problem tackled in this paper is critical for pretraining and seems to be ignored by the mainstream. I believe improvement on this problem can lead to better representation models, and can be inspiring to other pretraining objectives.
2. The method brings consistent gains compared to fair baselines on two standard benchmarks.

*Weaknesses:
1. My main concern is about the proposed methods in this paper. First, for the Soft Regularization method, it is not clear how the loss L_reg backpropagates through the model. Is the author only training the two embeddings? It doesn't make sense to me that this can improve the model.
2. For the Hard Correction method, it looks to me like the author is just using a synonym set as the target labels. Despite of the good empirical results, the solution itself lacks novelty. Also, although the WordNet synonyms can mitigate false negative problem, it can also brings new false positives. 
3. The author mainly compares their method with the original pretrained model itself, rather than other methods in solving false positives. I am curious if add label smoothing during training can also bring gains and how it compares with the proposed method in this paper.
summary_of_the_review: This paper studies a critical problem of false negatives in pretraining language models and demonstrates good results. But the proposed solution lacks enough novelty and might have some technical flaws.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The premise of the paper is that true negative examples should be considered in ELECTRA-style token-level discriminative pre-training. The framework uses cosine similarity to find true negatives and softly regularize with semantic distance, or finding true negatives by synonyms and simply ignoring the prediction before feeding to the loss function. Results on GLUE and SQUAD with ELECTRA-based models show that handling true negatives during pre-training improves performance.
main_review: Strengths
- S1 - The idea of handling true negatives for ELECTRA pre-training is an interesting idea.
- S2 - Experimental results show that proposed pre-training objectives can improve the performance of ELECTRA.

Weaknesses and Questions:
- W1 - Quiet unsure about the experimental settings. Seems SR and HC are continually pre-training objectives. Then, are baselines (ELECTRA_small and ELECTRA_base) also continually pre-trained with original ELECTRA pre-training objectives for the same steps of SR and HC? If not, the results are not convincing.
- W2 - Unclear the performance improvement is solely from handling true negatives. what about (1) random (2) false negatives?
- Q1 - It would be great if the paper could provide the correction statistics of HC.
- Q2 - ELECTRA-style token-level discriminative pre-trained models are showing good performance in token-level classification tasks such as NER. It would be great if we could see that performance as well.
summary_of_the_review: Although the paper shows good results compared to the baseline, current writing is not convincing.
It needs more details and extensive analysis to support the claim.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to avoid false negative examples in language model pretraining via modifying the training objective. Two methods, referred to as soft regularization and hard correction, are proposed to reduce the false negatives based on continual training on the ELECTRA model. Experiments on the GLUE benchmark show that the continual trained model with the two methods can both improve the performance on downstream NLU tasks. Experiments on SQuAD show the methods also help with model robustness.
This work provides ways to adjust the pre-training objectives when the model predictions are correct but do not match the ground-truth tokens. This could benefit the performance on downstream task in some cases. But the limited experiments in the paper do not assure that this is a general approach in pretraining that can be widely adopted.
main_review: The motivation of reducing false negative examples is reasonable but still does not convince me that this is necessary in pretraining, given that the main goal of pretraining is to learn good contextualized representation that could be useful for various downstream tasks. The authors claim the false negative examples are harmful to pretraining, but it is not fully supported by the experiments in the paper. The intuition of soft regularization and hard correction is somehow ad-hoc and may also not hold under some circumstances. For example, when the model is not well trained, minimizing the distance between the predicted token and the ground-truth token could be misleading if the prediction is incorrect. The synonyms from the dictionary could also be wrong without considering the given context. In equation (6), it is unclear whether the embedding e_k is token embedding or contextualized representation, and from generator or discriminator.

Some experimental results are problematic and not convincing. 
1)	The proposed methods are only applied to continual training of a well-trained ELECTRA model. This may indicate they are not suitable for early-stage pretraining, as the assumptions may not hold. This is a great limitation as a pretraining strategy, as it is not clear at which training steps one can use it. Moreover, as a fair baseline, ELECTRA should also be continually trained with the same 200k steps as the proposed models. 
2)	The results in Table 2 are inconsistent with Table 1 (ELECTRA_small and ELECTRA_base on GLUE test set). Some important baseline such as RoBERTa, XLNet, DeBERTa are not included either. Moreover, since all the experiments are conducted on ELECTRA small and base model, it is unknown whether the proposed methods work on larger scale models and other types of pretrained models such as BERT.
3)	In Table 6, what is the baseline system, BERT or ELECTRA_small?

summary_of_the_review: The methods proposed in this paper have clear theoretical and empirical limitations. The experiments do not fully support the claims by the authors. My current recommendation to this paper is reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the issue of misconceived false negative predictions and encourage pre-training language models on true negatives or more true negatives to ensure the auto-constructed data in masked language modeling (MLM) is true enough. A motivation behind this is that  false negatives may potentially hurt the pre-training in both efficiency and robustness to a great extent and this vulnerability of pre-trained language has never been studied before. In this paper, the authors propose an enhanced pre-training approach to counteract misconceived negatives.

Based on ElECTRA, the authors present two alternative pre-training objective auxilary to MLM one, which aims to make the model focus on true negatives during MLM. The first one, dubbed as soft regularization (SR), measures the distribution similarity between the predicted token and the original one, to smooth the tough cross-entropy by minimizing the semantic distances. The other, named as hard correction (HC), hinder the gradient propagation of the false negative samples to further avoid training with false negative predictions. 
main_review: Pros:

- As summaried above, this paper presents technically sound methods (i.e., SR and HC) to mitigate false negative problem in masked language models. 
- Besides the main evaluation to beat baseline performance on natural language understanding tasks, this paper also includes more insightful analyses into the proposed methods, from both quantitative and qualitative perspectives. 


Cons and Questions:

- The motivation of this paper is not that convincing to me. I fully agree "it is critical to ensure the auto-constructed data is true enough" but how can we measure the severity of "false negative problem"? Given the 2nd & 3rd examples in Table 1, although both "Ground-truth" and "Prediction" can fit the masked tokens, there are still subtle differences between them especially in long contexts (up to 512/1024 tokens during pretraining vs. a short sentence in the examples). 
- There are also some weaknesses in the proposed pre-training objectives. As for SR, the regularization is only applied to embedding layer. As for HC, Wordnet can only provide a very coarse Symset, which may introduce error propagation. Can author give more empirical studies to demonstrate the successes of these objectives? 
- Why did the author only apply the proposed pre-training objectives on the top of ELECTRA instead of standard MLM, e.g., RoBERT. The latter should be a better testbed for the objectives targeting problems in MLM. 
- Can authors discuss the relation of this work to a recent distillation paradigm, i.e., distill a small LM to large LM for accelerated convergence and promoted performance, which is mainly attributed to label smoothing. 

summary_of_the_review: Technically sound methods but unconvincing motivation and ill-considered methods; More empirical studies needed.
review_title: null

Meta-review: This paper gets decent performance gains (~2% on GLUE) by soft regularization to make negatives closer to positives in contrastive learning and hard correction of too-close negatives to at least avoid synonyms. These are useful ideas which to some extent build on the simple technique of ELECTRA (controlling the size of the generator MLM in Electra encourages the negatives to in general be "close but not too close", right?). As such, the paper is correct and provides potentially useful gains, but it appears a quite small adjustment of existing techniques, and in addition the use of WordNet is fairly brittle (and its similarity calculations do not consider context at all). 

The authors should be commended for the thorough job they did at updating their paper to address particular questions and concerns of reviewers, and useful new information emerged. Relative to the question of whether this method can be applied with other MLMs, the new Appendix A results do show that the answer is Yes, but the gains turn out to be much more modest (~0.5% on GLUE). However, ultimately, while this is all useful information and side experiments, these improvements just can't fix the key problem that all the reviewers felt that this paper does not provide sufficient "Technical Novelty and Significance". As such without bigger new ideas, this improved paper would probably be best as a good workshop paper.

My recommendation is that this paper not be accepted to ICLR 2022 on the basis of its limited technical novelty and significance.
Decision: Reject