Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a pretrained language model that extends Electra. They operate within the generator/discriminator proposed by Electra. The key contributions in the paper are a couple of novel loss functions that allow the discriminator to predict the tokens rather than binary labels and a contrastive loss for the sequence embedding. With a detailed analysis they show that the resulting token level and sentence embedding are much better at discriminating random input. They also demonstrate that this improves the quality on the GLUE and SQUAD datasets. They further demonstrate that their approach results in faster training and improved parameter efficiency.
main_review: Positives

- The results that authors demonstrate are very compelling and reliable.
- The analysis on the token and sentence level embedding is insightful.

Negatives

- Their approach is an extension of Electra rather than something fundamentally new.
- The exposition could be improved.

Detailed comments

- Related works section could be more elaborate.
- It would also be beneficial if the authors included a brief paragraph or two on how this work can be extended to the multilingual setting.
- All-Token MLM is just referred to in line 170. Since this is critical for understanding the paper readers could be eased into it rather than assuming they have read ELECTRA.
- In figure 3 the author shows training time in GPU hours. It is good to plot against examples seen or number of training steps (assuming all three used the same batch size). This would help clarify that there are no inefficiencies in the implementation of Roberta/ELECTRA.
- Line 249, authors mention that the benefit of stop gradients are for stability. It is not clear what they refer to. Removing stop gradients seems to have a very positive effect on the average scores.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a pre-trained technique based on correcting tokens and contrasting sequences of text. The correcting objective is based on ELECTRA with a modification in the formulation of multi-task setup where the one task is to identify the corrupted tokens and the other task is to correct it. The contrasting objective aims to maximize the similarity between similar sentences while pushing dissimilar sentences away in cosine space. With the two modifications, COCO-LM is trained much faster than ELECTRA. The paper also presents a comprehensive evaluation on GLUE and SQUAD 2.0 benchmarks and ablation study of their method.
main_review: - In terms of originality, COCO-LM is built on ELECTRA, thus it lacks a novelty aspect. ELECTRA does not have a sequence-level objective, COCO-LM uses a contrasting objective (SCL) for sequences thus it performs better than ELECTRA on sequence level tasks (i.e. GLUE). As can be seen in table 2, SCL + RTD improves the average score on GLUE significantly compared to RTD (i.e., ELECTRA). Having said that, I think the contribution of the paper is quite marginal, however I think its strong point is the efficiency in training time.

- The paper is well-presented and contains a comprehensive list of experiments showing the effect of each of the modified components. What I find missing is that the paper hasn’t addressed the claim about missing language modelling benefit (line 107-113). Although the authors presented word-similarity (table 3, line 292) results, I think that it’s rather a proxy evaluation but is not convincing enough about the benefit of language modeling capabilities. As the prompt-based approach [1,2,3] is gaining popularity recently, I wonder if COCO-LM can be used in this setup where the model is required to generate (i.e., assessing language modeling capability). This is something MLM such as RoBERTA, BERT are doing quite well.

**post rebuttal**: As authors run additional experiments with prompt tuning and showed good results. I increase my score to 7,

[1] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. 
[2] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. 
[3] Guanghui Qin and Jason Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. 

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles two challenges in current self-supervised learning frameworks, including the pre-training efficiency and the anisotropy of text representations. To address these two challenges, the authors propose the COCO-LM framework that pre-trains language models by correcting and contrasting corrupted text sequences. COCO-LM follows ELECTRA-style pre-training such that an auxiliary language model is employed to corrupt text sequences, upon which two new tasks are constructed to pre-train the main model. The first task is Corrective Language Modeling (CLM) to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second task is Sequence Contrastive Learning (SCL) to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experimental results on GLEU and SQuAD demonstrate that COCO-LM can improve the performance of pre-trained models and also the efficiency of pre-training.
main_review: Strengths
1.	The problems tackled in this paper are critical. While ELECTRA-style improves the pre-training efficiency and achieves improvements on language understanding tasks, the language modeling capability is compromised due to the replaced token detection (RTD) task. This paper incorporates both RTD and LM objectives to fully utilize the self-supervision signals in the text. Besides, representations by BERT-style or ELECTRA-style pre-trained models are often facing the degeneration issue, which may require sophisticated post-adjustment to achieve good performance in fine-tuning.
2.	Experiments on various language understanding tasks demonstrate that COCO-LM can achieve the performance of SOTA approaches with 50% of the pre-training GPU hours. With the same training time, COCO-LM outperforms other approaches with non-trivial margins.


Weaknesses
1.	While the paper claim the importance of language modeling capability of pre-trained models, the authors did not conduct experments on generation tasks that are more likely to require a well-performing language model. Experiments on word similarity and SquAD in section 5.3 cannot really reflect the capability of language modeling. The authors may consider to include tasks like language modeling, machine translation or text sumarization to strenghen this part, as this is one of the main motivations of COCO-LM.
2.	Analysis of SCL in section 5.2 regarding few-shot abaility looks not convincing. The paper claims that a more regularized representation space by SCL may result in better generalization ability in few-shot scenarios. However, results in Figure 7(c) and (d) do not meet our expectation such that COCO-LM achieves much more improvements with less labels and the improvements will gradually disappear with more labels. Besides, the authors may check if COCO-LM brings benefits to sentence retrieval tasks with the learned anisotropy text representations.
3.	The comparison with Megatron is a little overrated. The performance of Megatron and COCO-LM is close to other approaches, for examples, RoBERTa, ELECTRA, and DeBERTa, which are with similar sizes as COCO-LM. If the author claim that COCO-LM is parameter-efficient, the conclusion is also applicable to the above related works. 


Questions for the Authors
1.	In experimental setup, why did the authors switch the types of BPE vocabulary, i.e., uncased and cased. Will the change of BPE cause the variance of performance?
2.	In Table 2, it looks like COCO-LM especially affects the performance on CoLA and RTE hence the final performance. Can the authors provide some explanation on how the proposed pre-training tasks affect the two different GLEU tasks?
3.	In section 5.1, the authors say that the benefits of the stop gradient operation are more on stability. What stability, the training process? If so, are there any learning curves of COCO-LM with and without stop gradient during pre-training to support this claim?
4.	In section 5.2, the term “Data Argumentation” seems wrong. Did the authors mean data augmentation? 


Typos
1.	Check the term “Argumentation” in line 164, 252, and 314.
2.	Line 283, “a unbalanced task”, should be “an unbalanced task”.
3.	Line 326, “contrast pairs”, should be “contrastive pairs” to be consistent throughout the paper?

limitations_and_societal_impact: There is no negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces COCO-LM, a pretraining method. The pretraining has two components: *COrrecting*, and *COntrasting*. The COrrecting objective shares most of the efficiency advantages of ELECTRA, while still allowing some language modelling ability. The COntrasting objective ensures that sentences do not all have high cosine similarity, something that occurs in ELECTRA. 

* The pretraining method achieves higher accuracy more efficiently than counterparts across various tasks (GLUE, SQuAD 2.0) and settings (base, base with more data and large).
* There is a comprehensive ablation study and analysis section that supports the design choices
main_review: This paper introduces COCO-LM, a pretraining method. The pretraining has two components: *COrrecting*, and *COntrasting*.  For correcting, like ELECTRA, the authors first mask some tokens in a sentence. A small auxiliary model is trained to predict those tokens. The main model must then predict which words have been replaced. In contrast with Electra, the main also predicts the original word when it was replaced. For contrasting, the model must predict whether other sentences in a batch are corrupted versions of the sentence (either through cropping or masking) or different sentences altogether. 

The COrrecting objective shares most of the efficiency advantages of ELECTRA, while still allowing some language modelling ability. The COntrasting objective ensures that sentences do not all have high cosine similarity, something that occurs in ELECTRA. Instead, sentences will have high cosine similarity if they are similar, which is useful for many downstream tasks, (particularly for e.g: STSB). 

The authors compare COCO-LM to other pretraining methods on GLUE and SQuAD 2.0. They experiment with three settings, varying model and data size. Overall, the results are strong. On GLUE, COCO-LM improves by about 1 point on average. On SQuAD 2.0, it improves over existing approaches in the base settings but performs on par with them in the large setting. 

In addition, on MNLI the model reaches higher accuracies in less GPU time than other pretraining methods. Given each step is longer, this means that the data efficiency of COCO-LM more than makes up for the overhead the method introduces over e.g ELECTRA.

The ablation study in this paper is very comprehensive and useful. Interestingly, keeping only the ELECTRA setup performs better than the original ELECTRA, likely because of the changes in the auxiliary model setup and due to better optimization (indeed, the authors own implementation of ELECTRA already outperforms the original one). Finally, the authors also convincingly demonstrate the usefulness of their objective design choices.

Overall, this paper makes a compelling case for this new pretraining method. The results are strong, the ablation study comprehensive, the design choices are well motivated and their usefulness is supported by the analysis. Replicating some baselines also shows good scholarship. 
With general pretraining methods, it is easy to want more and ask for more experiments (e.g: SuperGLUE), as many tasks and setups are relevant to show the generality of the method, but I feel like the authors case is strong enough in this instance. 

The biggest missing piece in the paper is the lack of hyperparameter details, because the appendix is not included. The sections do appear in my pdf’s reader table of content, but not the text. There are also some result concerns, which I detail below. 


Pros:
- COCO-LM performs well across various model sizes and tasks and is also more efficient in terms of GPU/h for some level of accuracy.
- The ablation study is comprehensive and compelling. Noteworthy that some design choices show only a small gain (e.g: RTD alone does not perform much worse).
- The analysis of the design choices behind each objective is thorough and convincing.
- The open-sourced model will be useful. 

Cons:
- The appendix is currently missing and with it many of the hyperparameter details required for reproducibility.
- Please share at least one run / set of results on the test set for each setting. This is necessary to ensure that the difference is not just in hyperparameter overfitting during fine-tuning.
- I understand that the claim “matches Megatron-3.9Bn, a model 10x bigger on MNLI” is compelling and makes for a great abstract. However, I think it is misleading and poor - scholarship. Indeed, Megatron-3.9Bn is not a great baseline for a 1Bn+ model. For instance, DeBERTA 1.5B+SIFT achieves 92.0/92.1 on MNLI (see Table 12 in their appendix, though not clear whether this is dev or test), quite a bit higher than Megatron 3.9Bn (and COCO-LM). 
- Minor: Some typos: Throughout the paper (3 times), please replace data argumentation with data augmentation. “The analyses uses base setting” -> “The analysis uses the base setting” or “The analyses use the base setting”, etc. 
- Minor: You can keep only one significant digit for GLUE/SQuAD results. Would make tables easier to read and I am not convinced a difference of <0.1 is significant on those tasks.	


~~~ Update author response:

Thanks for the response. Looking forward to the improvements to the paper. Keeping my score.

limitations_and_societal_impact: This is similar to BERT and has the same potential concerns.
review_title: null

Meta-review: This work improves ELECTRA language pretraining approach by introducing Corrective Language Model task (so the model can generate words) and Sequence Contrastive Learning (so sentence representations are more informative). I agree with the reviewers that the authors have conducted extensive experiments to show that the approach is effective and the analyses are also very thorough and insightful, e.g., analyzing cosine similarities of sentence vectors. There are a few points that the original paper wasn't convincing, e.g., Coco-LM wasn't better than ELECTRA in Large++ setup and only word-similarity is used to demonstrate the main point that Coco-LM has better language modeling capabilities. The new results added for the Large++ and Few-Shot Prompt-Based Fine-Tuning Results convincingly addressed these concerns; hence, I recommend Accept.
Decision: Accept (Poster)