COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
Authors: Yu Meng
Chenyan Xiong
Payal Bajaj
saurabh tiwary
Paul N. Bennett
Jiawei Han
Xia Song
Venue: NeurIPS 2021 Poster (2021)
Keywords: Language Model Pretraining

Abstract: We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

COCO-LM: Correcting and Contrasting Text
Sequences for Language Model Pretraining
Yu Meng'; Chenyan Xiong', Payal Bajaj', Saurabh Tiwary?,
Paul Bennett", Jiawel Han", Xia Song"
University of Illinois at Urbana-Champaign
Microsoft
1 [yumeng5, (hanj)@illll
2 { chenyan xiong, payal. bajaj, satiwary, paul.n. www.ettt,,tttttmmmmmmmmmmmmm...t xiaeeett,xiassee xiaso X @microsoft.ccm com com
Abstract
We present a self-supervised learning framework, COCO-LM, that pretrains Lan- guage Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order better capture token-level semantics. The second sequence-leve task, Sequence Contrastive Learning. is to align text sequences originated from the same source input while suring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-ard pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.
1
Introduction
Pretrained language models (PLMs) have reshaped the way Al systems process natural language 36 39 40). Before task-specific training, it is now common practice to first pretrain the deep neural networks, often Transformers [53], via a self-supervised token-level language modeling task [29 31 40). Whether it is autoregressive 33), permutational [62), or masked language modeling (MLM)
the Transformer networks are pretrained to recover some omitted tokens using the rest of input texts. Then the language semantics captured during pretraining are conveyed to downstream tasks via the pretrained Transformer parameters 5
4:1
Recent research [1] 16, 25 43 observed several challenges in this self-supervised learning frame- work. One challenge its efficiency. After pretrained for a while with the standard token-level language modeling, the networks have already captured the basic language patterns, making large fraction pretraining signals no longer informative. Linear improvement in the model effectiveness often requires exponentially more pretraining compute and parameters [25), which is unsustainable. Another challenge is the anisotropy of text representations from pretrained models. The sequence representations from many pretrained models quite irregular [30 43 and require dedicated
fine-tuning approaches to useful in sequence-level applications [3 60
Clark et al. 7 proposed new pretraining strategy, ELECTRA, that uses an auxiliary language model ("generator") to replace tokens input texts and pretrains the main Transformer ("discriminator") to
Part of this work was done while Yu interning Microsoft.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

detect replaced tokens. This improves the pretraining efficiency and effectiveness, but pretraining via binary classification hinders the model's usage on applications requiring language modeling capability (e.g., prompt-based learning 15 28
46 It could further distort the representation space as the Transformers are pretrained to output the same "non-replacement" label for all actual tokens.
In this paper, we present a new self-supervised learning approach, COCO-LM, that pretrains Lan- guage Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary model to corrupt the input texts, upon which it intro- duces two new pretraining tasks for the main Transformer, one at token level and one at sequence level. The token-level task. corrective language modeling (CLM), pretrains the main Transformer to detect and correct the tokens in the corrupted sequences. It uses a multi-task setup to combine the benefits of replaced token detection and language modeling, The sequence-level task, sequence contrastive learning (SCL., pretrains the model to align text sequences originated from the same source sequence and enforce uniformity of the representation space.
In our experiments on GLUE [54] and SQuAD A1 benchmarks, COCO-LM not only outperforms state-of-the-art pretraining approaches in effectiveness, but also significantly improves the pretraining efficiency. Under the same setting, COCO-LM matches the MNLI accuracy of RoBERTa and ELECTRA with 60% and 50% of their GPU hours in pretraining, respectively. When pretrained with the same number of steps, COCO-LM outperforms the previous best models by 1+ GLUE average points under the standard base/large-sizze sized model evaluations. With 367 million parameters, COCO- Lagett+ reaches the MNLI accuracy of Megatrons, 9B 449, one of the largest BERT-style model with 3.9 billion parameters. Our analyses provide further insights on the advantage of CLM in learning token representations and its effectiveness in prompted- based fine-tuning, as well as the benefit of SCL in ensuring alignment and uniformity in the representation space for better generalizatior!
2 Related Work
Various token-level tasks have been used to pretrain language models. The most classic auto-regressive language modeling to predict a token given all the previous tokens, or all subsequent ones [36] 39] BERT uses masked language modeling (MLM) that recovers randomly masked tokens using the rest input. XLNet proposes permutation language modeling that conducts MLM an autoregressive manner 622. UniLM uses pseudo MLM which unifies autoregressive and MLM tasks 13].
Sequence-level tasks are also explored, which often pretrain the model to predict certain co- occurrences of sequence pairs. For example, next sentence prediction [.] sentence ordering [27] and previous sentence prediction (56) concatenate two sentences (either correlated or random), and train the Transformer to classify the pair.
Empirically, MLM is still among the most effective tasks to pretrain encoders 29
40
RoBERTa (31 found the sentence-level task in BERT not benefitial and discarded it. BART
29 and
T5 44 both observed that MLM is often the most effective task. The empirical advantages of other pretraining tasks are more task-specific, for example, entity related masks for knowledge intensive
applications 20 24 and sequence-level tasks for long form text modeling
42
Instead of randomly altering texts, ELECTRA 77 uses a smaller auxiliary Transformer pretrained by MLM to replace some tokens in the text sequences using its language modeling probability, and pretrains the main Transformer to detect the replaced tokens. ELECTRA achieves state-of-the-as accuracy in many language tasks 77 Later, Clark et el. 6 developed ELECTRIC, which pretrains encoders by contrasting original tokens against negatives sampled from a cloze model. ELECTRIC re-enables the language modeling capability but underperforms ELECTRA in downstream tasks.
Our work is also related to contrastive learning which has shown great success in visual representation learning 4 22 34 Its effectiveness of in language is more observed in the fine-tuning stage, for
example, in sentence representation [16], dense retrieval [60]. and GLUE fine-tuning [19]
3 Method
We present the preliminaries of PLMs, their challenges, and the new COCO-LM framework.
'Code and pretrained models can be found wul.:pps//gttuu..oo///oootttt/tp

---

3.1 Preliminary on Language Model Pretraining
In this work we focus on pretraining BERT-style bidirectional Transformer encoders [ that are widely used in language representation tasks. We first recap the masked language modeling (MLM) task introduced by BERT [TT] and then discuss the pretraining framework of ELECTRA 77.
which to take an input sequence X = orig ...., orig ..... Fn orig with 15% random tokens replaced by [MASK] symbols (e.g., g.g the i-th token), and train the model predict the original tokens at the masked positions:
BERT Pretraining uses the masked language modeling task (MLM)
11
orig
[MAKK]
Transformerr
H
MLM Heal
PM ha)::
where the Transformer generates contextualized representations H (h+ The MLM Head predicts the masked token from the vocabulary V using the hidden representation h. and token em- beddings æ. The pretraining minimizes the MLM loss on the of masked positions M. Specifically,
exp(a h:) PMLM(x)h:) h2)) Eney expla; h.)²
[MLM E
Σ iEM
log PMLM
( (n..)
ELECTRA Pretraining uses two Transformers, "generator" pretrained by MLM, and "discrimi- nator" pretrained using the generator's outputs. We refer them as auxiliary and main Transformers, the former is discarded after pretraining and the latter may be trained by "generative" tasks too.
The auxiliary model outputs a corrupted sequence X-LM by sampling from its predicted probability:
MLM
PMLM (z/h.), if E M
2
MLM
orig T,
else.
(1)
The masked positions are replaced by sampled tokens considered plausible in context by the auxiliary Transformer, which are more deceiving than random replacements. ELECTRA uses a skinnier auxiliary network (e.g., hidden dimension is 1/3 of the main model) to control the signal difficulty.
The main Transformer takes M-M and classifies the replaced tokens:
X M-M Main Transtormer,
RTD Head
[1 = x
PRID
where 1 (.) is the indicator function. The Replaced Token Detection (RTD) head uses a sigmoid linear layer to output the binary probability, and the main Transformer is trained with binary cross entropy loss. The RTD task is trained on all tokens instead of masked ones and improves efficiency.
The two Transformers are pretrained jointly. The auxiliary model gradually generates more realistic replacement tokens and the main model learns to better detect them. This forms a natural learning curriculum and significantly improves ELECTRA's accuracy in downstream tasks 7
3.2 Challenges of ELECTRA--SY Pretraining
Missing Language Modeling Benefits. The classification task in ELECTRA is simpler and more stable [61) but raises two challenges. The first is the lack of language modeling capability which is a necessity in some tasks (6]. For exam-
Dimesty
Exlimated Dumity
Landine
40
candon
eloilaa
slantas
10
20
ple, prompt-based learning requires a language
model to generate labels [15] 33 45 446.. The second is that the binary classification task may not be sufficient to capture certain word-level semantics that are critical for token-level tasks.
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Coeine Similbrity
Creina Similarity
(a) RoBERTa.
(b) ELECTRA.
Figure Cosine similarity distributions of ran- dom/similar sequence pairs using [CLS] embed- dings from pretrained models. Histograms/curves distribution bins/kernel density estimates.
Squeezing Representation Space. Another challenge is that the representations from Transformer-based language models often reside in narrow cone, where two random sentences have high similarity scores (lack of uniformity). and closely related sentences may have more different representations (lack of alignment)
14
30

---

Correcive Modwiing
COCO-LM Pretraining Tasks:
Corrective Language Modeling (CLM)
semplin
sempling
Main Transformer
Sequence Contrastive Learning (SCL)
Auxillary Transformer
Leanring
Ieput
DSACK
Apput
Randam
Masked ALGD_
Main Transformer
Original Sequence: ABCDE
Cropped Sequence: BCD
[PAR]
[PAN]
The overview of COCO-LM. The auxiliary Transformer is pretrained by MLM. Its corrupted text sequence is used as the main Transformer's pretraining input in Corrective Language Modeling and paired with the cropped original sequence for Sequence Contrastive Learning.
Figure 2:
Figure I Dillustrates such behaviors with random sentence pairs (from pretraining corpus) and semanti- cally similar pairs (those annotated with maximum similarity from STS-B (3) With RoBERTa, cosine similarities of most random sentence pairs are near 0.8, bigger than many semantically similar pairs. The representation space from ELECTRA even more squeezed. Nearly all sentence pairs, both random and similar ones, have around 0.9 cosine similarity. This may not be surprising as ELEC- TRA is pretrained to predict the same output ("non-replacement") for tokens these sequences. The irregular representation space raises the risk of degeneration 37 55] and often necessitates sophisticated post-adjustment or fine-tur tue-tuning to improve the sequence representations 16
30 32 60)
3.3 COCO-LM Pretraining
COCO-LM also employs an auxiliary Transformer construct the corrupted text sequence, in Eqn. ( but it introduces two new pretraining tasks upon the corrupted sequences to address the challenges previously described. In the rest of this section, we present these two tasks and then the detailed configurations of COCO-LM. Its framework is illustrated in Figure 2
Corrective Language Modeling (CLM) trains the main Transformer to recover the original tokens, given the corrupted text sequence X ML..
X MLM Main Transformer
H
CLM Head
PCLM (x h.).
The CLM Head uses the hidden representations H to output a language modeling probability, instead of binary classification score. The forward pass of the CL.M Head is the same All-Token MI.M, a variation of ELECTRA [7] that consists of a language modeling layer and a binary classification layer for the copy mechanism:
exp(x. h:)
PiM(zi(hs) (hs) = 1 (x. = L. MLM) Peopy (1)h:) + Peopy (2)) Σ Ev exp(x, h4)
Poopy (yal (ha) = exp(y: w wap h2i)/ (exp( copy he,) +1.
where Weopy is a learnable weight and Peopy (ys (2)) is the copy mechanism (y:: = 1 when the input token is original and can be directly copied to the output; y: O when the input token needs to be corrected to another token from the vocabulary).
In ELECTRA, All-Token MLM performs worse than RTD 7
Language modeling on the corrupted text sequence MLLM hard as the replaced tokens from the auxiliary model are more deceiving than [MASK]. To improve the language model learning, different from All-Token MLM, CLM employs

---

multi-task setup that combines the RTD task to explicitly train the copy mechanism Poopy (-):
T
Coopy = - E
MLM
x; 2
log Peopy (1)1z) + 1 (M. + x log Peopy (0)ku)) (2)
LIM
Σ
log PLM
(2. (n:)
iEM
=-E
Σ IEM
log
( MLM = x2 (1 + Peopy (0)h,)
exp(x: hi) Ex.vv exp(22 hz;)
LCLM = -xcopy Geopy + LIM-
The hyperparameter Acopy balances the weights of the two tasks. The binary cross entropy loss in Eqn. (2) explicitly trains the copy probability. We also stop gradient (sg) decouple the gradient backpropagation to Peopy(:) (.) from the LM task. This way, the main Transformer first learns the easier classification task and then uses it to help learn the harder LM task. The binary classification task
trained on all tokens while the language modeling task is trained only on masked positions.
CLM combines the advantages of MLM and ELECTRA: The main Transformer is trained all tokens with the help of the binary classification task while also being able to predict words, thus enjoying the efficiency benefits of ELECTRA and preserving the language modeling benefits.
Sequence Contrastive Learning (SCL) forms a contrastive learning objective upon the sequence embeddings to learn more robust representations. Broadly, contrastive learning is to align positive pair of instances, often different views of the same information 34). in contrast to unrelated negative instances [22] 66). The different views are often obtained by applying data augmentations on the same input, for example, rotation, cropping, and blurring on visual representations [4, 34] so that the neural networks can learn representations robust to these data alterations.
In COCO-LM, the corrupted sequence X-MIM already provides a form of data augmentation. We pair it with another augmentation, Xemp, randomly cropped contiguous span of X (the length of X is 90% of X so that the major sequence meaning is preserved), to construct the positive pair and to contrast with random negatives.
Specifically, a training batch B in SCL includes random of corrupted and cropped sequences: B [(XMMMM x )..... (XMLM xx contrastive pair (X, x) consists of either (X xxx")) or (X, M.M. (symmetrical contrast). The negative instances all the remaining sequences in the batch B B {(X, (+)). The contrastive loss is formulated as:
xx }}, with X and x-rop originated from X positive
exp(cos(s,s*))/
LSCL = -- log exp(cos(8, 8+)//)+++++))) Ex-en waa-een-xpppooss exp(cos(8, 8-)))
Σ exp (cos(s, 8 )/+) )/+))
=-E cos(s, 8 8/// - log exp(coo((; +//)++
(3)
X-EB-
where 8, 8t, 8 are the representations of X, X X-, respectively, from the main Transformer (i.e., h (cLss)) The similarity metric is cosine similarity (cos) and the temperature is set to 1.
As shown in Wang et al. [55]. the first term in Eqn. 3 (cos(s: (s, s improves alignment of the space. It encourages representations to be robust to the corruptions and the alterations on the original text. The second term Eqn. (3) promotes uniformity. pushes unrelated sequences apart in the representation space and ensures low cosine similarity between random data points. Several studies have observed improved generalization ability from better alignment and uniformity [66] 37 55).
Aligning X MLM with crop requires the main Transformer to produce sequence representations robust to both token-level (i.e., MLM replacements) and sequence-level (i.e., cropping) alterations. The model is thus encouraged to reason more using partially altered sequences to recover the original information.
Overall Training. COCO-LM uses the following loss function:
LCOC0-MM
CAux. +
Mainn MMain LCLM LSCI.
(4)
5

---

The auxiliary Transformer is pretrained by masked language modeling (MLM) and generates cor- rupted sequences. The main Transformer is pretrained correct the corruption (CLM) and contrast the corrupted sequences with the cropped sequences (SCL.. The two Transformers are pretrained jointly with the loss in Eqn. The main Transformer used in downstream applications.
Network Configurations. Similar to ELECTRA, the auxiliary Transformer is smaller than the main model, but we use different configurations in the auxiliary model: (1) We reduce the number of layers to 1/3 or 1/4 (under base or large model setup, respectively) but keep its hidden dimension the same with the main model, instead of shrinking its hidden dimensions; (2) We disable dropout in it when sampling replacement tokens. We find such configurations empirically more effective and use them as the backbone of COCO-LM, The main Transformer follows the standard architecture of BERT/ELECTRA and can be easily adopted by downstream application pipelines with almost no changes.
Experimental Setup
Pretraining Settings. We employ three standard settings, base, base++, and large+ Base is the BERT Base training configuration [Π]: Pretraining on Wikipedia and BookCorpus [6] (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use
the same corpus and 32, 768 uncased BPE vocabulary 47 as with TUPE (2)].
Base++ trains the base size model with larger corpora and/or more training steps. Following recent
research [1 62], we add in OpenWebTex, (18), CC-News (31), and STORIES (52), to total of 160 GB texts, and train for 4 billion (with 2048 batch samples
31. We follow the prepossessing
of UVi [ and 64, 000 cased BPE vocabulary.
Large++ uses the same training corpora as base++ + and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERT ..
Model Architecture. Our baselbase++ model uses the BERT Base architecture 11: 12 layer Trans-
former, 768 hidden size, plus T5 relative position encoding [4]] Our large++ model is the same with Ba 24 layer and 1024 hidden size, plus T5 relative position encoding [40]. Our auxiliary
network uses the same hidden size but shallow 4-layer Transformer in baselbase++ and a 6-layer one in large++. When generating XMLM we disable dropout the auxiliary model.
Downstream Tasks. We use the tasks included in GLUE [54] and SQuAD 2.0 reading compres- sion 4I. Please refer to Appendix for more details about GLUE tasks. Standard hyperparameter search in fine-tuning is performed, and the search space can be found in Appendix [ The fine-tuning
protocols use the open-source implementation of TUPE (26. The reported results are the median of five random seeds on GLUE and SQuAD.
Baselines.
data We compare with various pretrained models in each setting. To reduce the variance in we also pretrain and fine-tune RoBERTa and ELECTRA A under exactly the same setting with COCO-LM, marked with "(Ours)". All numbers unless marked by "(Ours)" from reported results in recent research (more details in Appendix C
Implementation Details. Our implementation builds upon the open-source implementation from MC-BERT [61] and fairseq [35). More implementation details are mentioned in Appendix D
5
Evaluation Results
Three groups of experiments are conducted to evaluate COCO-LM and its two new pretraining tasks.
5.1 Overall Results and Ablations
Overall Results are listed in Table I Under all three settings, COCO-LM outperforms all recent state-of-the-art pretraining models on GLUE average and SQuAD. improves the sfate-of-the-art GLUE score by about one point under all three settings. COCO-LM also enjoys better parameter efficiency. Using less than 10% of Megatron's 's parameters, COCO-LM matches the MNLI accuracy of Megatron; one of the largest pretrained BERT-style encoders.

---

GLUE Stugle Task
SQuAD 2.0
Model
Params
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
STS-B
AVG
EM
Base Setting: BERT Base Size,
Wikipodia Book Corpus (16GB)
BERT |
110M
84.5/-
91.3
91.7
93.2
58.9
68.6
87.3
89.5
83.1
73.7
76.3
RoBERTH 31
125M
84.7/-
92.7
79.7
XLNet (62
110M
85.8/85.4
92.7
78.5
81.3
ELECTRA
110M
86.085.3
90.0
91.9
93.4
64,3
70.8
84,9
89.1
83.7
80.5
83.3
MC-BERT
85.7/85.2
92.3
88.0
83.7
DenIRTaa XX
134M
86.3/86.2
79.3
82.5
26
86.2/86.2
RoBERTa (Ours)
110M
85,8/85,5
91.3
92.0
93.7
60.1
68.2
87.3
88.5
83.3
77.7
80.5
ELECTRA (Ours)
110M
86.9/86.7
91.9
92.6
93.6
66.2
75.1
88.2
89.7
85.5
79.7
82.6
COCO-LMM
110M
88.5/88.3
92.0
93.1
93.2
63.9
84,8
91.4
90.3
87.2
82.4
85.2
Buse++ Setting: BERT Base Size, Bigger Training Data, and/or More Training Steps
110M
94.7
RoBERTa 31
125M
87.6/-
91.9
92.8
94.8
63.6
78.7
90.2
91.2
86.4
80.5
83.7
UniLM V2 1
110M
88.5/-
91.7
95.1
87.1
DeBERTa
134M
88.8/88.5
83.
86.2
CLEAR 599
110M
86.7/
90.0
92.9
94.5
64.3
78.3
89.2
89.8
85.7
COCO-LM
134M
90.2/90.0
92.2
94.2
94.6
67.3
87.4
91.2
91.8
88.6
85.4
88.1
Large++ Setting: BERT Large Size, Bigger Training
Training Steps
XLNot 68
360M
90.8/90.8
92.3
94.9
97.0
69.0
85.9
90.8
92.5
89.2
87.9
90.6
RoBERTH 3
356M
90.29002
92.2
96.4
86.6
90.9
92.4
88.9
86.5
ELECTRA k
335M
90.9/
92.4
95.0
96.9
69.1
88.0
90.8
92.6
89.4
88.0
90.6
DeDERT:
91.1/91.1
92.3
96.8
COCO-LM
367M
91.4/91.6
92.8
95.7
96.9
73.9
91.0
92.2
92.7
90.8
88.2
91.0
Megatron
1.38
90,9/910
92.6
87.1
90.2
Megatron
49
3.98
91,4/91.4
92.7
88.5
91.2
Table Results on GLUE and SQuAD 2.0 development set. All results are single-task, single-model fine-tuning. Results not available in public reports are marked as "_", DeBERTa reported RTE, MRPC and STS-B results by fine-tuning from MNLI checkpoints which are not single-task results. We use Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the rest on GLUE. AVG is the average of the eight tasks on GLUE, All baseline results unless marked by (Ours) are reported by previous research.
Model
Parums
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
STS-B
AVG
Base/Basett Setting: BERT
Base Size
BERT
I 110
84.6/83.4
89.2
90.5
93.5
52.1
66.4
84.8
85.8
80.8
ELECTRABasee I 110M
88.5/88.0
89.5
93.1
96.0
64.6
75.2
88.1
90.2
85.6
COCO-LM Misect
134M
89,8/89.3
89.8
94.2
95.6
68.6
82.3
88.5
90.3
87.4
Large/Large++ Setting: BERT Large Size
BERT 335M
86.7/85.9 90.7/90.2
89.3
92.7
94.9
60.5
70.1
85.4
86.5
83.2
ELECTRALLae 335M
90.4
95.5
96.7
68.1
86.1
89.2
91.7
88.5
COCO-LMunge+ 367M
91.6/91.1
90.5
95.8
96.7
70.5
89.2
88.4
91.8
Table GLUE test set results obtained from the GLUE leaderboard. We perform hyperparameter search for each task with ten random seeds and use the best development set model for test predictions. All results are from vanilla single-task fine-tuning (no ensemble, -sscciccc tricks, etc.).
Table 2 shows GLUE test set results which further confirm the advantages of COCO-LM over previous methods.
Efficiency. In downstream tasks, the efficiency of COCO-LM is the same with BERT. In pretraining, the auxiliary model and SCL introduce extra cost. However, as shown in Figure COCO-LM more efficient in GPU hours. It outperforms RoBERTa & ELECTRA by 1+ points on MNLI with the same GPU hours and reaches their accuracy with around 60% & 50% GPU hours, respectively.
Ablation Studies. Table 3 shows the ablations of COCO-LM under the base setting on GLUE DEV.
Pretraining Task. With only RTD, our backbone model with the shallow auxiliary Transformer is quite effective. CLM and SCL both provide additional improvements on MNLI and GLUE average. Their advantages are better observed on different tasks, for example, CLM MNLI-mm and SCL on RTE and MRPC. Combining the two in COCO-LM provides better overall effectiveness. later experiments, further analyze the benefits of these two tasks.
7

---

Group
Method
MNLI-(m/mmm)
QQr
QNLI
SST-2
CoLA
RTE
MIRCC
STS-B
AVG
COCO-LMase
88. 38.5/
63.9
84.8
Pretruining
RTD Only
88.4/88.2
92.1
93.5
92.7
67.3
80.5
89.0
90.9
86.8
Task
CLM Only
88. 6/88.8
92.0
93.2
93.7
67.4
80.1
90.0
90.4
86.
SCI. RTD
88 88.6/88.2
92.1
93.5
93.8
64.3
82.7
90.2
90.6
86.9
Network
w/o. Rel-Pos
88.2/87.7
92.2
93.4
93.7
68.8
82.7
91.2
90.6
87.6
Setting
w. ELECTRA'S Auxiliary
88.0/87.7
91.9
92.7
93.5
64.3
81.2
89.5
89.7
86.3
Training
Random Replacements
84.9/84.7
41.6
70.0
Signal
Converged Ausiliary
88.3/88.1
92.0
92.8
94.3
64.2
78.3
90.4
90.2
86.3
CLM
All-Toil All-Token LM Only
87 2/87.0
91.8
92.6
93.7
60.6
74.0
88.5
89.7
84.7
Setup
CLM w/o. Copy
88.0/67.9
91.8
93.1
94.4
66.6
76.9
89.5
90.1
86.3
CLM w/o. Stop-grad
88. 38.5/88.2
92.0
92.9
94.3
66.5
80.9
90.0
90.6
86.9
Table Ablations on GLUE Dev. that climinate (w/o.), keep (Only) or switch (w.) one component.
87.5
Dev. Set Acc.
87.5
87.5
GLIE AVG
87.0
85.0
85.0
82.5
RoBERT
DOCO-LM
Dev.
82.5
ROBERTS
COOG-TM
86.5
ELECURA
EILICTIIA
80.0
86.0
10
20
30
10
20
30
70%
90%
100%
Training Time (Hours)
Training Time (Hours))
Xeroy length (w.r.t. X(rik)
(a) MNLI-m
(b) MNLI-mm
Figure 4: The performance of COCO-LMBas when pretrained with different crop fractions. The æ-axis is the fraction of X X be- ing kept (no cropping is 100%).
Figure 3: COCO-LMBase on MNLI Dev. (y-axes) at differ- pretraining hours four DGX-2 nodes (64 V100 GPUs).
The final training hours and accuracy of RoBERTa (Ours) and ELECTRA (Ours) measured in the same settings are marked.
Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers some tasks but significantly hurts MNLI. Using a shallow auxiliary network and keeping the same hidden dimension (768) more effective than ELECTRA'S 12-layer but 256-hidden dimension generator.
Pretraining Signal Construction. Using randomly replaced tokens to corrupt text sequence hurts significantly. Using converged auxiliary network pretrain the main model also hurts. It is better to pretrain the two Transformers together, as the auxiliary model gradually increases the difficulty of the corrupted sequences and provides natural learning curriculum for the main Transformer.
CLM Setup. Disabling the multi-task learning and using All-Token MLM reduces model accuracy. The copy mechanism is effective. The benefits of the stop gradient operation are more on stability (preventing training divergence).
5.2 Analyses of Contrastive Learning with SCL
This group of experiments analyzes the behavior of SCL. All experiments use the base setting.
Ablation on Data Augmentation. Figure 4 shows the effects of the cropping operation when forming positive SCL pairs with the corrupted sequence. Using the original sequence results in worse GLUE accuracy. It is less informative the model no longer needs to learn representations robust to sequence-level alteration. Cropping too much (e.g., only keeping 70% the original sequence), may hurt as it alter the semantics too much. Empirically simple alteration works the best, similar to the observations in recent research 16 22).
Alignment and Uniformity. Figure plots the distribution of cosine similarities between random sequence pairs and similar ones using representations pretrained by COCO-LM. The representation space from COCO-LM is drastically different from those in Figure With COCO-LM, similar pairs are more aligned and random pairs are distributed more uniformly. Many similar pairs have near cosine similarity and are clearly separated from random pairs which center around 0. The t-SNE 9 plot fourthee demonstrates the benefits of SCL., The similar sentence pairs (marked by same shapes) are aligned closer when pretrained with SCL. Their average cosine similarity is 0.925 when

---

Eatinated Density
30
30
random
-30
-30
1.0
005
0.0
0.5
1.0
-0
50 -50
50
Cosine Simnilarity
(a) Without SCL
(b) With SCL
Figure 5: Cosine similarity of se- quence pairs randomly sampled from pretraining corpus and most similar pairs from STS-B using [CLS] J from COCO-LMBas-
Figure 6: The t-SNE of sequence representations learned with without SCL. The points are sampled from the most semantically similar sentences pairs from STS-B (with 5-score labels). The [CLS] embeddings are not fine-tuned. Some randomly selected similar pairs marked by same shapes.
Dev. Stc. Acc.
Cortne Sm.
Nagutice
Costus Sin.
0.5
0.5
Dov. Neastivn
9// 5CL SCT.
82
SCB
1.6
Teaining Steps (xited)
Training Steps (xteet)
Training Sut Sizen (56)
Tradning Swt Size (56)
(a) Without SCL
(b) With SCL
(c) MNLI-M
(d) MNLM
Figure 7: Analyses of SCL. Figs. (a) and (b) show the average cosine similarity between the [CLS] embeddings of positive and negative contrastive pairs during pretraining. Figs. (c) and (d) show the few-shot accuracy MNLI with different fractions of MNLI training set used (z-axes). The error bars mark the max/min and the solid lines are the average of five fine-tuning runs.
pretrained with SCL, while 0.863 without SCL. This better alignment and uniformity is achieved by COCO-LM with SCL via pretraining, without using task-specific data nor supervised labels.
Regularizing the Representation Learning for Better Few-Shot Ability. One would expect any pretrained Transformers to easily align a pair of corrupted sequence and cropped sequence as the two share about 80% tokens. However, shown in Figure 7a that is not the case: Without SCL, the cosine similarity of the positive pairs is even lower than random negatives. SCL necessary to regularize the representation space and reduce the risk of degeneration (Figure 7b
a more regularized representation space results in better generalization ability in scenarios with limited labels. Figure 7c 7d] show the results when COCO-LM are trained (via standard fine-tuning) with only fraction of MNLI labels. The improvements brought by SCL are more significant when fewer fine-tuning labels are available. With 1% MNLI labels, pretraining with SCL improves MNLI-m/mm accuracy by 0.8 0.8/0.5 compared to that without SCL. Using only 10%/20% labels, COCO-LM with SCL reaches similar MNLI accuracy with RoBERTa /ELECTRA (Ours) fine-tuned with all labels, respectively.
Similar to empirical observations and theoretical analyses in recent research
14
16 55
5.3 Analyses of Language Modeling with CLM
The last group of experiments studies the effectiveness and benefits of CLM.
Ablations Training Configurations. Figure 8 illustrates pretraining process with CLM and All-Token MLM. The plots demonstrate the difficulty of language modeling upon corrupted text sequences. It is quite unbalanced task. For the majority of the tokens (Original) the task is simply to copy its input at the same position. For the replaced tokens (7 8% total), however, the model needs detect the abnormality brought by the auxiliary model and recover the original token. Implicitly training the copy mechanism as part of the hard LM task is not effective: The copy accuracy of All-Token MLM is much lower, and thus the LM head may confuse original tokens with replaced ones. As shown in Table 3 and ELECTRA 77.. pretraining with All-Token MLM performs worse than using the RTD task, though the latter is equivalent to only training the copy mechanism. The multi-task learning of CLM is necessary for the main Transformer to stably learn the language modeling task upon the corrupted text sequence.

---

0.30
0.991
CL. Avc.
0.06
CLS Avec.
Oope
0.25
Coppe A8-77iic
0.902
0.
0,009
6.989
0.30
0.02
0.098
10
Traning Strpe (K1ed)
Training Stape (xtel)
Training Steps (xlr6)
Training Stepe (xtel)
(a) Copy Acc. (Replaced)
(b) Copy Acc. (Original)
(c) CLM Acc. (Replaced) (d) CLM Acc. (Original)
Figure 8: The copying accuracy and the language modeling accuracy (y-axes) of CLM and All-Token MLM at different pretraining steps (x-axes, in 10K scale). The accuracy is averaged tokens that are replaced by the auxiliary Transformer (Replaced) or those from the original input text (Original).
Prompt-Base Soaptt-Based Fine-Tuming with CLM. Table in- cludes the prompt-based fine-tuning experiments on MNLI for RoBERTa and COCO-LM under base++ + and large++ sizes, following the same few-shot man- ual prompt fine-tuning with demonstration setup in LM-BFF 15 We use {3e- 6, 4e - 6, 5e-6) for the learning rate search of COCO-LM base+ model, with everything else kept same as described in LM-BFF. With exactly the same pipeline, COCO- LM outperforms RoBERTa under both base++ and large++ sizes by significant margins MNLI- m/mm. Such observations are interesting as COCO- LM's main Transformer does even see any [MASK] tokens during pretraining but still performs well on predicting masked tokens for prompt- based learning. Note that ELECTRA and COCO-LM variants without the CLM task cable: Their main Transformers are not pretrained by language modeling tasks (thus no language modeling capability learned to generate prompt label words). This points out the importance, not necessity. of COCO-LM in the family of ELECTRA-style pretraining models. With the benefits and rapid developments of prompt-base based approaches, the lack of language modeling capability is going to limit the potential of ELECTRA's self-supervised learning framework in many real-world scenarios. COCO-LM not only addresses this limitation but also provides better prompt-based learning results.
Model
MNLI-m
MNLI-mm
RoBERTaBee+*
60.1 (1.5)
61.8 (1.2)
COCO-LMBas COCO-LMBaert MBseet
66.5 (2.1)
68.0 (2.3)
RoBERTa Largo++
70.7 (1.3)
72.0 (1.2)
COCO-LMungee angess
72.5
73.3 (1.1)
Table 4: Few-shot crompt-based fine-tuning using RoBERTa and COCO-LM trained on 16 samples per class. Mean (and standard deviation) accuracy results over different splits on MNLI-m/mm are shown.
appli-
6 Conclusions and Future Work
In this paper, we present COCO-LM, which pretrains language models using Corrective Language Modeling and Sequence Contrastive Learning upon corrupted text sequences. With standard pre- training data and Transformer architectures, COCO-LM improves the accuracy on the GLUE and SQuAD benchmarks, while also being more efficient in utilizing pretraining computing resources and network parameters.
One limitation of this work is that the contrastive pairs are constructed by simple cropping and MLM replacements. Recent studies have shown the effectiveness of advanced data augmentation techniques in fine-tuning language models 16 38 [1] A future research direction is to explore better ways to construct contrastive pairs in language model pretraining.
Despite the empirical advantage this auxiliary- mmain dual model framework, the auxiliary Trans- former training is not influenced by the main Transformer nor learns to generate the optimal pretrain- ing signals for the main model. To better understand and tailor the training of the auxiliary model to the main model is another important future research direction.
Acknowledgments
We sincerely thank Guolin Ke for discussions and advice model implementation. We also thank anonymous reviewers for valuable and insightful feedback, especially the suggestion of adding prompt-based fine-tuning experiments.
10

---

References
[1] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao. Jianfeng Gao. Ming Zhou, and Hsiao- Wuen Hon. UniLMv2: Pseudo-masked language models for unified language model pre-training. In ICML, 2020.
[2] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[3] Daniel Cer, Mona Diab, Encko Agirre, lñigo Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation (SemEval), 2017.
and Lucia Specia. Semeval-2011 task 1:
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffre Hinton. A simple framework for contrastive learning of visual representations, In JCMI, 2020.
[5] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis of BERT's attention. In ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019.
[6] Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. Pre-training transformers energy-based cloze models. In EMNLP, 2020.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA: Pre-training encoders as discriminators rather than generators. ICLR, 2020.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. In IJCAI, 2020.
[9] Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of BERT. In NeurlPS, 2019.
[10] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
[11] Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[12] William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005.
[13] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurlPS, 2019.
[14] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tan Liu. Representation degeneration problem in training natural language generation models. In ICI.R, 2019.
[15] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021.
[16] Tianyu Gao, Xingeheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embed- dings. In EMNLP. 2021.
[17] Danilo Giampiccolo, Bernard Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing, 2007.
[18] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
bttp://SkyyLoonUU..gi.c.
OpenNeebTxxtCorpun 2019.
[19] Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning, In ICLR, 2021.
[20] Kelvin Guu, Kenton Lec, Zora Tung, Panupong Pasupat, and Ming-Wor Wei Chang. REALM: Retrieval- augmented language model pre-training. In ICML, 2020.
[21] R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop Recognising Textual Entailment, 2006.

---

[22] Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[23] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Secoding-enhancee bert with disentangled attention. In ICLRR, 2021.
[24] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. In TACL. 2019.
[25] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alcc Radford, Jeftrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 088361, 2020.
[26] Guolin Ke, Di He, and Tin Liu. Rethinking the positional encoding in language pre-training. In ICLR, 2021.
[27] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self supervied learning of language representations. In ICLR, 2020.
[28]
Teven Le Scao and Alexander Rush. How many data points is a prompt worth? In NAACL-HLT, 2021.
[29] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to -o--uueee---eseeeeeeeeeeee pre-training for natural language generation, translation, and comprehension. In ACL, 2019.
[30] Bohan Li, Hao Zhou, Junxian He,, Mingxuan Wang. Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In EMNLP,, 2020.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetllemoyer and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 190i: 1692, 2019.
[32] Yi Luan, Jacob Eisenstein, Kristina Toutanove, and Michael Collins. Sparse, dense, and attentional representations for text retrieval, In TACL, 2021.
[33] Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text classification using label names only: A language model self-training approach. EMNLP,, 2020.
[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprins 100.::::00000374555 2018
[35] Myle Ott, Sergey Edunov, Alexei Bacvski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. FAIRSEQ: fast, extensible toolkit for sequence modeling. In NAACL-HLT Demonstrations, 2019.
[36] Matthew Peters. Mark Neumann, Mohit lyyer. Matt Gardner. Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018
[37] Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised leamnigg Invariances, augmentations and dataset biases. In NeurlPS, 2020.
[38] Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev. Jiawei Han, and Weizhu Chen. CoDA: Contrast- enhanced and diversity promoting data augmentation for natural language understanding, In ICLR, 2021.
[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188);, 2019.
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2019.
[41] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprchension of text. In EMNLP, 2016.
[42] Anirudh Ravula, Chris Alberti, Joshua Ainslie, Li Yang, Philip Minh Pham, Qifan Wang, Santiago Ontanon, Sumit Kumar Sanghai, Vaclav Cvicek, and Zach Fisher. ETC: Encoding long and structured inputs in transformers. In EMNLP, 2020.
[43] Nils Reimers and Iryna Gurevych. Sentence-BERR Sentence embeddings using siamese BERT- networks. In EMNLP, 2019.
12

---

[44] Adam Roberts. Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In EMNLP, 2020.
[45] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. In EACL, 2021.
[46] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL-IILT, 2021.
[47] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2015.
[48] Iyer Shankar, Dandekar Nikhil, and Csernai Kornél. First Quora dataset release: Question pairs, 2017.
[49] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LLM, Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv: 999900853, 2019.
[50] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
[51] Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. NAACL-ILT, 2021.
[52] Trieu Trinh and Quoc Le. simple method for commonsense reasoning. arXiv preprint urXiv:1806.0284 2018.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018.
[55] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[56] Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. StruetBERT: Incorporating language structures into pre-training for deep language understanding. In ICLR, 2020.
[57] Alex Warstadt, Amanprect Singh, and Samuel R Bowman. Neural network acceptability judgments. In TACL, 2019.
[58] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACI-HILT, 2018.
[59] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. CLEAR: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.1 15466, 2020.
[60] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive lcarning for dense text retrieval. In ICLR, 2021.
[61] Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, and Tie- Yan Liu. MC-BERT: Efficient language pre-training via meta controller. arXiv preprint arXiv:2006.05744 2020.
[62] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
[63] Yukum Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, In ICCV, 2015.
13


Reviews:
[{"review_id": "wnaifbo287w", "reviewer": "Reviewer_SaZ9", "summary": "This paper introduces COCO-LM, a pretraining method. The pretraining has two components: *COrrecting*, and *COntrasting*. The COrrecting objective shares most of the efficiency advantages of ELECTRA, while still allowing some language modelling ability. The COntrasting objective ensures that sentences do not all have high cosine similarity, something that occurs in ELECTRA. \n\n* The pretraining method achieves higher accuracy more efficiently than counterparts across various tasks (GLUE, SQuAD 2.0) and settings (base, base with more data and large).\n* There is a comprehensive ablation study and analysis section that supports the design choices", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces COCO-LM, a pretraining method. The pretraining has two components: *COrrecting*, and *COntrasting*.  For correcting, like ELECTRA, the authors first mask some tokens in a sentence. A small auxiliary model is trained to predict those tokens. The main model must then predict which words have been replaced. In contrast with Electra, the main also predicts the original word when it was replaced. For contrasting, the model must predict whether other sentences in a batch are corrupted versions of the sentence (either through cropping or masking) or different sentences altogether. \n\nThe COrrecting objective shares most of the efficiency advantages of ELECTRA, while still allowing some language modelling ability. The COntrasting objective ensures that sentences do not all have high cosine similarity, something that occurs in ELECTRA. Instead, sentences will have high cosine similarity if they are similar, which is useful for many downstream tasks, (particularly for e.g: STSB). \n\nThe authors compare COCO-LM to other pretraining methods on GLUE and SQuAD 2.0. They experiment with three settings, varying model and data size. Overall, the results are strong. On GLUE, COCO-LM improves by about 1 point on average. On SQuAD 2.0, it improves over existing approaches in the base settings but performs on par with them in the large setting. \n\nIn addition, on MNLI the model reaches higher accuracies in less GPU time than other pretraining methods. Given each step is longer, this means that the data efficiency of COCO-LM more than makes up for the overhead the method introduces over e.g ELECTRA.\n\nThe ablation study in this paper is very comprehensive and useful. Interestingly, keeping only the ELECTRA setup performs better than the original ELECTRA, likely because of the changes in the auxiliary model setup and due to better optimization (indeed, the authors own implementation of ELECTRA already outperforms the original one). Finally, the authors also convincingly demonstrate the usefulness of their objective design choices.\n\nOverall, this paper makes a compelling case for this new pretraining method. The results are strong, the ablation study comprehensive, the design choices are well motivated and their usefulness is supported by the analysis. Replicating some baselines also shows good scholarship. \nWith general pretraining methods, it is easy to want more and ask for more experiments (e.g: SuperGLUE), as many tasks and setups are relevant to show the generality of the method, but I feel like the authors case is strong enough in this instance. \n\nThe biggest missing piece in the paper is the lack of hyperparameter details, because the appendix is not included. The sections do appear in my pdf’s reader table of content, but not the text. There are also some result concerns, which I detail below. \n\n\nPros:\n- COCO-LM performs well across various model sizes and tasks and is also more efficient in terms of GPU/h for some level of accuracy.\n- The ablation study is comprehensive and compelling. Noteworthy that some design choices show only a small gain (e.g: RTD alone does not perform much worse).\n- The analysis of the design choices behind each objective is thorough and convincing.\n- The open-sourced model will be useful. \n\nCons:\n- The appendix is currently missing and with it many of the hyperparameter details required for reproducibility.\n- Please share at least one run / set of results on the test set for each setting. This is necessary to ensure that the difference is not just in hyperparameter overfitting during fine-tuning.\n- I understand that the claim “matches Megatron-3.9Bn, a model 10x bigger on MNLI” is compelling and makes for a great abstract. However, I think it is misleading and poor - scholarship. Indeed, Megatron-3.9Bn is not a great baseline for a 1Bn+ model. For instance, DeBERTA 1.5B+SIFT achieves 92.0/92.1 on MNLI (see Table 12 in their appendix, though not clear whether this is dev or test), quite a bit higher than Megatron 3.9Bn (and COCO-LM). \n- Minor: Some typos: Throughout the paper (3 times), please replace data argumentation with data augmentation. “The analyses uses base setting” -> “The analysis uses the base setting” or “The analyses use the base setting”, etc. \n- Minor: You can keep only one significant digit for GLUE/SQuAD results. Would make tables easier to read and I am not convinced a difference of <0.1 is significant on those tasks.\t\n\n\n~~~ Update author response:\n\nThanks for the response. Looking forward to the improvements to the paper. Keeping my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is similar to BERT and has the same potential concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "f1_6S5fDYrn", "reviewer": "Reviewer_uxzq", "summary": "Authors propose a pretrained language model that extends Electra. They operate within the generator/discriminator proposed by Electra. The key contributions in the paper are a couple of novel loss functions that allow the discriminator to predict the tokens rather than binary labels and a contrastive loss for the sequence embedding. With a detailed analysis they show that the resulting token level and sentence embedding are much better at discriminating random input. They also demonstrate that this improves the quality on the GLUE and SQUAD datasets. They further demonstrate that their approach results in faster training and improved parameter efficiency.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n\n- The results that authors demonstrate are very compelling and reliable.\n- The analysis on the token and sentence level embedding is insightful.\n\nNegatives\n\n- Their approach is an extension of Electra rather than something fundamentally new.\n- The exposition could be improved.\n\nDetailed comments\n\n- Related works section could be more elaborate.\n- It would also be beneficial if the authors included a brief paragraph or two on how this work can be extended to the multilingual setting.\n- All-Token MLM is just referred to in line 170. Since this is critical for understanding the paper readers could be eased into it rather than assuming they have read ELECTRA.\n- In figure 3 the author shows training time in GPU hours. It is good to plot against examples seen or number of training steps (assuming all three used the same batch size). This would help clarify that there are no inefficiencies in the implementation of Roberta/ELECTRA.\n- Line 249, authors mention that the benefit of stop gradients are for stability. It is not clear what they refer to. Removing stop gradients seems to have a very positive effect on the average scores.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "CCw3AyVANk5", "reviewer": "Reviewer_cYxo", "summary": "This paper tackles two challenges in current self-supervised learning frameworks, including the pre-training efficiency and the anisotropy of text representations. To address these two challenges, the authors propose the COCO-LM framework that pre-trains language models by correcting and contrasting corrupted text sequences. COCO-LM follows ELECTRA-style pre-training such that an auxiliary language model is employed to corrupt text sequences, upon which two new tasks are constructed to pre-train the main model. The first task is Corrective Language Modeling (CLM) to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second task is Sequence Contrastive Learning (SCL) to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experimental results on GLEU and SQuAD demonstrate that COCO-LM can improve the performance of pre-trained models and also the efficiency of pre-training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1.\tThe problems tackled in this paper are critical. While ELECTRA-style improves the pre-training efficiency and achieves improvements on language understanding tasks, the language modeling capability is compromised due to the replaced token detection (RTD) task. This paper incorporates both RTD and LM objectives to fully utilize the self-supervision signals in the text. Besides, representations by BERT-style or ELECTRA-style pre-trained models are often facing the degeneration issue, which may require sophisticated post-adjustment to achieve good performance in fine-tuning.\n2.\tExperiments on various language understanding tasks demonstrate that COCO-LM can achieve the performance of SOTA approaches with 50% of the pre-training GPU hours. With the same training time, COCO-LM outperforms other approaches with non-trivial margins.\n\n\nWeaknesses\n1.\tWhile the paper claim the importance of language modeling capability of pre-trained models, the authors did not conduct experments on generation tasks that are more likely to require a well-performing language model. Experiments on word similarity and SquAD in section 5.3 cannot really reflect the capability of language modeling. The authors may consider to include tasks like language modeling, machine translation or text sumarization to strenghen this part, as this is one of the main motivations of COCO-LM.\n2.\tAnalysis of SCL in section 5.2 regarding few-shot abaility looks not convincing. The paper claims that a more regularized representation space by SCL may result in better generalization ability in few-shot scenarios. However, results in Figure 7(c) and (d) do not meet our expectation such that COCO-LM achieves much more improvements with less labels and the improvements will gradually disappear with more labels. Besides, the authors may check if COCO-LM brings benefits to sentence retrieval tasks with the learned anisotropy text representations.\n3.\tThe comparison with Megatron is a little overrated. The performance of Megatron and COCO-LM is close to other approaches, for examples, RoBERTa, ELECTRA, and DeBERTa, which are with similar sizes as COCO-LM. If the author claim that COCO-LM is parameter-efficient, the conclusion is also applicable to the above related works. \n\n\nQuestions for the Authors\n1.\tIn experimental setup, why did the authors switch the types of BPE vocabulary, i.e., uncased and cased. Will the change of BPE cause the variance of performance?\n2.\tIn Table 2, it looks like COCO-LM especially affects the performance on CoLA and RTE hence the final performance. Can the authors provide some explanation on how the proposed pre-training tasks affect the two different GLEU tasks?\n3.\tIn section 5.1, the authors say that the benefits of the stop gradient operation are more on stability. What stability, the training process? If so, are there any learning curves of COCO-LM with and without stop gradient during pre-training to support this claim?\n4.\tIn section 5.2, the term “Data Argumentation” seems wrong. Did the authors mean data augmentation? \n\n\nTypos\n1.\tCheck the term “Argumentation” in line 164, 252, and 314.\n2.\tLine 283, “a unbalanced task”, should be “an unbalanced task”.\n3.\tLine 326, “contrast pairs”, should be “contrastive pairs” to be consistent throughout the paper?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "BLn_SGav0Yz", "reviewer": "Reviewer_gYa8", "summary": "The paper proposes a pre-trained technique based on correcting tokens and contrasting sequences of text. The correcting objective is based on ELECTRA with a modification in the formulation of multi-task setup where the one task is to identify the corrupted tokens and the other task is to correct it. The contrasting objective aims to maximize the similarity between similar sentences while pushing dissimilar sentences away in cosine space. With the two modifications, COCO-LM is trained much faster than ELECTRA. The paper also presents a comprehensive evaluation on GLUE and SQUAD 2.0 benchmarks and ablation study of their method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- In terms of originality, COCO-LM is built on ELECTRA, thus it lacks a novelty aspect. ELECTRA does not have a sequence-level objective, COCO-LM uses a contrasting objective (SCL) for sequences thus it performs better than ELECTRA on sequence level tasks (i.e. GLUE). As can be seen in table 2, SCL + RTD improves the average score on GLUE significantly compared to RTD (i.e., ELECTRA). Having said that, I think the contribution of the paper is quite marginal, however I think its strong point is the efficiency in training time.\n\n- The paper is well-presented and contains a comprehensive list of experiments showing the effect of each of the modified components. What I find missing is that the paper hasn’t addressed the claim about missing language modelling benefit (line 107-113). Although the authors presented word-similarity (table 3, line 292) results, I think that it’s rather a proxy evaluation but is not convincing enough about the benefit of language modeling capabilities. As the prompt-based approach [1,2,3] is gaining popularity recently, I wonder if COCO-LM can be used in this setup where the model is required to generate (i.e., assessing language modeling capability). This is something MLM such as RoBERTA, BERT are doing quite well.\n\n**post rebuttal**: As authors run additional experiments with prompt tuning and showed good results. I increase my score to 7,\n\n[1] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. \n[2] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. \n[3] Guanghui Qin and Jason Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work improves ELECTRA language pretraining approach by introducing Corrective Language Model task (so the model can generate words) and Sequence Contrastive Learning (so sentence representations are more informative). I agree with the reviewers that the authors have conducted extensive experiments to show that the approach is effective and the analyses are also very thorough and insightful, e.g., analyzing cosine similarities of sentence vectors. There are a few points that the original paper wasn't convincing, e.g., Coco-LM wasn't better than ELECTRA in Large++ setup and only word-similarity is used to demonstrate the main point that Coco-LM has better language modeling capabilities. The new results added for the Large++ and Few-Shot Prompt-Based Fine-Tuning Results convincingly addressed these concerns; hence, I recommend Accept.
Decision: Accept (Poster)