Language Model Pre-training on True Negatives
Authors: Zhuosheng Zhang
hai zhao
Masao Utiyama
Eiichiro Sumita
Venue: ICLR 2022 Submitted (2022)
Keywords: Pre-trained Language Models
Masked Language Modeling
False Negatives
Natural Language Understanding

Abstract: Discriminative pre-trained language models (PrLMs) learn to predict original texts from intentionally corrupted ones. Taking the former text as positive and the latter as negative samples, the discriminative PrLM can be trained effectively for contextualized representation. However, though the training of such a type of PrLMs highly relies on the quality of the automatically constructed samples, existing PrLMs simply treat all corrupted texts as equal negative without any examination, which actually lets the resulting model inevitably suffer from the false negative issue where training is carried out on wrong data and leads to less efficiency and less robustness in the resulting PrLMs.
Thus in this work, on the basis of defining the false negative issue in discriminative PrLMs that has been ignored for a long time, we design enhanced pre-training methods to counteract false negative predictions and encourage pre-training language models on true negatives, by correcting the harmful gradient updates subject to false negative predictions. Experimental results on GLUE and SQuAD benchmarks show that our counter-false-negative pre-training methods indeed bring about better performance together with stronger robustness.

Under review as a conference paper at ICLR 2022
LANGUAGE MODEL PRE-TRAINING ON TRUE NEGATIVES
Anonymous authors Paper under double-blind review
ABSTRACT
Discriminative pre-trained language models (PrLMs) learn to predict original texts from intentionally corrupted ones. Taking the former text as positive and the lat- ter as negative samples, the discriminative PrLM can be trained effectively for contextualized representation. However, though the training of such type of PrI.Ms highly relies on the quality of the automatically constructed samples, ex- isting PrLMs simply treat all corrupted texts as equal negative without any ex- amination, which actually lets the resulting model inevitably suffer from the false negative issue where training is carried out on wrong data and leads to less ef- ficiency and less robustness in the resulting PrLMs. Thus in this work, on the basis of defining the false negative issue in discriminative PrLMs that has been ignored for long time, design enhanced pre-training methods counter- act false negative predictions and encourage pre-training language models on true negatives. by correcting the harmful gradient updates subject to false negative predictions. Experimental results on GLUE and SQuAD benchmarks show that our counter-false-nee negative pre-traing pre-training methods indeed bring about better perfor- mance together with stronger robustness.
INTRODUCTION
Large-scale pre-trained language (PrLM) models are playing important role in wide variety of NLP tasks with their impressive empirical performance (Radford et al. 2018 Peters et al.] 2018] Devlin et al. 2019 Yang et al., 2019 Lan et al.] 2019: Clark et al. (019).. So far, there comes two categories Pr..Ms, the generative like GPT (Radford al. 2018) and BART (Lewis et al. 20206), which employ decoder for learning to predict a full sequence, and the discriminative like BERT style of PrLMs which learn to predict the original text from the intentionally corrupted ones. this work, we focus the latter category of PrLMs, typically with denoising objectives (also known as masked language modeling. MLM) (Liu et al.) 2019 Joshi et al. 2020, Sun et al. 2019).
a
denoising objective, certain percentage of tokens in the input sentence masked out, and the model should predict those corrupted tokens during the pre-training (Peters et al., 2018 Sun et al.)
2019 Levine al. 2021 Li & Zhao 2021).
Besides corrupting the texts with masks, some alternatives were proposed for constructing training examples with various arbitrary noising functions motivated by edit operations like insertion, dele- tion, replacement, permutation, and retrieval (Lewis et al. [2000a] Xu & Zhao 2021 Wang et al.)
2019 Guu al. 2020). Auxiliary objectives are also proposed conjunction with MLM, such as next sentence prediction (Devlin et al. 2019), span-boundary objective (Joshi et al. 2020), and sentence-order prediction (Lan et 2019).
Although existing studies have made progress in designing effective masking strategies and auxiliary objectives, there are intrinsic yet critical issues appearing throughout the whole training process that lack attention for a long time. Discriminative PrLM can be regarded as a kind of auto denoising encoder on automatically corrupted texts. Thus, it is critical to ensure the auto-constructed data is true enough. Intuitively, discriminative PrLM learns to distinguish two types of samples, positive (already existing original ones) and negative (the corrupted ones from the auto constructing). Taking MLM example, proportion of tokens in sentences symbols, which would affect the sentence structures, leading to the loss of semantics and increasing the uncertainty of predictions. In extreme cases, such corrupted text may be linguistically correct.
corrupted, e.g., replaced with mask

---

Under review as a conference paper at ICLR 2022
Example
Ground-truth
Prediction
MLM
Mediation
is [MASK] good
very
happy
The cat is [MASK]
cute
smart
X
is a [MASK] [MASK] for discussion
good day
great time
x
Table I: Examples of true negative (the first line) and false negatives (the second and third line).
However, the current PrLMs simply consider all corrupted texts as negative samples, so that the resulting PrLM has to be trained on such false negatives with less efficiency and less robustness, which either waste training time on meaningless data or are vulnerable to adversarial attacks like diversity distraction and synonym substitution (Wang et al.. 2021).
In a general scenario, MLM only calculates label-wise matching between the prediction and the gold tokens in the training process, thus inevitably suffering from the issue of false negatives where the prediction is meaningful but regarded as wrong cases, as examples shown in Table [ The issue is also observed in sequence generation tasks, which is tied to the standard training criterion of maximum likelihood estimation (MLE) that treats all incorrect predictions as being equally incorrect (Wieting al., 2019 Li al. 2020). Instead of measuring negative diversity via the diversity scores between the different incorrect model outputs, our method is dedicated to mediating the training process by detecting the alternative predictions as opposed to the gold one, to steer model training on true negatives, which benefits the resulting language modeling in general.
Though the false negatives may potentially hurt the pre-training in both efficiency and robustness to great extent, it is surprising that this problem is kept out of the research scope of PrLMs until this work to our best knowledge. To address the issue of misconceived false negative predictions and encourage pre-training language models on true negatives or more true negatives, we present an enhanced pre-training approach to counteract misconceived negatives. In detail, we employ two enhanced pre-training objectives: 1) soft regularization by minimizing the semantic distances be- tween the prediction and the original one to smooth the rough cross-entropy and 2) hard correction to shield the gradient propagation of the false negative samples avoid training with false negative predictions. We pre-train our methods on top of the ELECTRA architecture (Clark et al. (0192201)) and fine-tune it on widely-used down- wown-streaming benchmark tasks, including GLUE (Wang et al. 2018) and SQuAD (Rajpurkar et al.) 2016). Experimental results show that our approach boosts the base- line performance by large margin, which verifies the effectiveness of our proposed methods and the importance of training on true negatives. Case studies show that our method keeps the simplicity and also improves the robustness of language model pre-training.
RELATED WORK
Designing effective criteria for language modeling one of the major topics in training pre-trained models, which decides how the model captures knowledge from large-scale unlabeled data. Recent studies have investigated denoising patterns (Raffel et al. 2020 Lewis et al., 2020b), MLM alterna- tives (Yang et al. 2019), and auxiliary objectives (Lan et al. 2019 Joshi et al. (0200) to improve the
power of pre-training. However, studics show that the current models still suffer from fattr-nitting issues, and it remains challenging to find effective and efficient training strategies (Rogers et al. 2020
Denoising Patterns MLM has been widely used as the major objective for pre- pre-training (Devlin et al. 2019 Lan et al.) 2019) Clark et al.) 2019; [Snn et al., 2020), in which the fundamental part is how to construct high-quality masked examples (Raffel et al. 2020). The current studies commonly define specific patterns for mask corruption. For example, some are motivated from the language modeling units, such as subword masking (Devlin et al. 2019), span masking (Joshi et al. 2020). and
n-gram masking (Levine et al. 2021 Li & Zhao 2021 Some employ variety of edit operations
like insertion, deletion, replacement, and retrieval Leewis et al., 2020a Guu et al.- 2020). Others
seck for external knowledge annotations, such named entities (Sun al. 2019). semantics (Zhou ctt al., 2020), and syntax (Zhang et al., 2020b; Xu et al., 202TT. To provide more diversity of mask
tokens, RoBERTa applied dynamic masks in different training iterations (Liu al. 2019). These
2

---

Under review as a conference paper at ICLR 2022
prior studies either employ pre-defined mask construction patterns or improve the diversity of mask tokens to help capture the knowledge from pre-training.
MLM alternatives To alleviate the task mismatch between the pre-training and the fine-tuning for downstream tasks, XLNet (Yang et al., 2219) proposed an autoregressive objective for language modeling through token permutation, which further adopts more complex model architecture, In- stead of corrupting sentences with the mask symbol that never appears in the fine-tuning stage, MacBERT (Cui et al., 2020) propose to use similar words for the masking purpose. Yamaguchi jt al. (202T) also investigates simple pre-training objectives based on token-level classification tasks as replacements of MLM. which are often computationally cheaper and result in comparable per- formance to MLM. In addition, training sequence-lo-seqqeene (Seq2Seq) language models has also
aroused continuous interests (Dong et al. 2019 Lewis al. 2020b Raffel et al. 2020).
Auxiliary objectives Another research line is auxiliary objectives in conjunction with MLM, such as next sentence prediction (Devlin et al. 2019), span-boundary objective (Joshi et al. 2020), and sentence-ordee prediction (Lan al. (2019). Such line of researches emerges hot topics, especially
in bomain--peccife pre-training, such as dialogue-oriented language models, which involve diverse
kinds of interaction entailed in utterances (Zhang et al. 2020a Wu et al., 2020 Zhang Zhao
202T).
As the major difference from the existing studies, our work devotes itself mediating misconceived negatives the essential drawback of MLM during the MLE estimation and aiming to guide lan- guage models to learn from true negatives through our newly proposed regularization and correction methods. The comparison with existing work is illustrated in Figure
Besides the heuristic pre-trained patterns like masking strategies during data construction, stress that there are potential post- post-processing strategies to guide the MLM training: correc- tion and pruning, which are considered to deal with the false negative issue during MLM train- ing, where the model would yield reasonable predictions but discriminated tions because such predictions do not match the single gold token for each training case. For example, many tokens are reasonable but writ- ten in different forms or are the synonyms of the expected gold token. We could correct the training with soft regularization or directly drop view to sentence level, the similarity between the predicted
Positive
Positive
wrong predic-
Negative
True Negative
False Negative
Existing Work
b) Our Study
Figure 1: Overview of our study.
the uncertain predictions. Promoting sentence and the original sentence can also be taken into account to measure the sentence-leva con- fidence that indicates how hard the task is, which would be beneficial provide more fine-grained signals and thus improve the training quality. Based on the rationales above, we are motivated to de- sign the corresponding correction and regularization techniques to mediate misconceived negatives.
In a broader view, our work is also related to knowledge distillation, whose paradigm is training student networks mimic the soft target generated by well-trained teachers (Gou al. [202] Hahn
& Choi 2019). which has been proved a type of label smoothing (Yuan et al.) 2020 Zhang & Sabuneuu 2020) Müller et al.] (2019). Such line of research has supported the hypothesis that regularization of soft targets could accelerate convergence and promote performance. By contrast, our approaches are more efficient without the need to train two models.
3 METHODOLOGY
3.1 PRELIMINARIES
MLM Masked LM (MLM) is denoising language model technique used by BERT (Devlin al. 2019) to take advantage of both the left and right contexts. Given a sentence s = = [wi W2. W,, F. A certain proportion of tokens randomly replaced with special mask symbol. The input is
3

---

Under review as a conference paper at ICLR 2022
fed into the multi-head attention layer to obtain the contextual representations, which is defined as H = FFN(MuliiHadd( V)), where K,Q.V are packed from the input sequence representation Then, the model is trained to predict the masked token based on the context.
Let V E RN denote the set of masked positions using the mask symbol [MM] and Nm is the number of masked tokens. We have wk ) the set of masked tokens, and s' as the masked sentence where the tokens in ) are masked with the mask symbol s. The objective of MLM is to maximize the following objective:
Ne
Cymm Cmimm(uk (Wi: 5')
Σ
log pa(wu s');
(I)
Nm
KEY
where Nm is the total number of masked positions in the input sequence.
ELECTRA MLM only learns from small proportion of masked positions per example, which incur a substantial compute cost. With the goal to improve training efficiency, ELECTRA (Clark al. 2019) proposed, which consists of generator G and a discriminator D. Instead of masking tokens, ELECTRA corrupts the input sequence by replacing them with tokens sampled from a small generator. The discriminator is trained to distinguish whether each replaced one the original or replacement. In the implementation, the generator and discriminator are based on Transformer architecture like BERT (Devlin al., (2019) but mainly differ the model size. The generator is in smaller scale, and the training objective is the same as MLM in Eq. 1 written as:
Co(wx, s')
G log
(wek |s)).
N
During the training iteration, the generator predicts new sequence (denoted as s²) with the pre- dicted token for each corrupted position in the original sequence. The predicted sequence is then fed to the discriminator, which uses binary classification task predict the probability D(wF, to indicate how likely each token wt (t [1, in sa is replaced by generator, whose loss function is:
1 n
Cp(we, s')
L(w so),
L(w], - log (wwf, 89), log(1 - Dwwf, sQ)),
wi = wh w/ +
where n is the length of the input sequence.
The final combined loss of ELECTRA is computed by: Latm = (aww,ss s') + (w,, s'), where A is dyper-parameee to balance the weights of generator and discriminator, and it is set to 50 according to Clark et al. (2019).
3.2 PRE-TRAINING ON TRUE NEGATIVES
An intuitive solution to encourage the language model pre-training true negatives is reduce the "difficulty" or uncertainty of the prediction during pre-training. Therefore, the cloze-style to- ken prediction problem may be simplified between the prediction and ground-truth and smooth measurement of the relevance. Therefore, we motivated employ two techniques counteract the false negative predictions, including 1) soft regularization, which measures the distribution similarity between the predicted token and the original one, to smooth the tough cross- cross-entropy by minimizing the semantic distances (SR); hard correction (HC), which shields the gradient propagation of the false negative samples to further avoid training with false negative predictions.
multi-choice problem to break the rough validation
Soft Regularization Let denote the predicted token from MLM (derived from the generator in this work). For Wh and Pk we fetch their token representations from the model's embedding module denoted as ea and é, respectively. We leverage cosine similarity as the regularization
For ELECTRA, we fetch the embedding from the discriminator. Note that the embeddings of the generator and the discriminator tied following the official implementation (Clark 2019).
4

---

Under review as a conference paper at ICLR 2022
based the intuition that the semantic distance between the prediction and gold tokens should be minimized:
CK Cj |eal ||
Creg
(4)
SR is based on the hypothesis that the predicted tokens should have semantic relationship with the gold ones in the embedding space to some extent, which is supported by various existing studies
(Bordes et al. 2013] Zhang Zhao 202T] Chen et al. 2021 [i et al. 2220)) We choose to apply SR to the embedding layer because the embedding layer is the most fundamental and stable layer. Optimizing the embedding layer would possibly lead a more severe influence of the model training and help the model learn semantics between words better as indicated by Jiang et al. (2020).
Hard Correction The other alternative strategy is to prune the gradient when the model suffers from the confusion of whether the prediction correct not. For each prediction, we check if the token is highly related to the ground-truth token based on a short lookup table V in which each token mapped to a list of alternatives. The lookup table is built by retrieving the synonym alternatives for each word in the model vocabulary, e.g., from WordNet (Miller) 1995) or Vec embedding (Mikolov al. al.] 2013). In this work, we use WordNet synonyms by default (Section 5.2 will compare retrieving synonyms from WordNet and Word2 embedding).
Lcor Σ Ix * log Py(Wk |s'),
(5)
kEY
where I is the identifier indicating whether the k-th prediction should be counted, which is defined by:
I - { ea + ex, EK E Vie'l. otherwise.
For each training iteration, if the gold token is found in the synonym list for the predicted token, then the correction is activated by 4.- Such a prediction will be judged as correct by HC cross- entropy the correction can be applied by simply ignoring this prediction before feeding to the cross-cntropy loss function.
3.3 IMPLEMENTATION VARIANT
According to the motivation and formulation above, the soft regularization and hard correction ap- proaches are supposed to be applied as independent substitutes Therefore, the overall training objective for language modeling is rewritten as L' Laltn Lreg or L' Latm + foo for SR and HC, respectively.
4 EXPERIMENTS
4.1 SETUP
Pre-training In this part, we will introduce the model architecture, hyper-parameter setting, and corpus for pre-training our models. Considering the training efficiency, employ ELECTRA small and base as our backbone models and implement our pre-training objectives on top of them. We follow the model configurations in Clark al. (2019) for fair comparisons. For the batch size is 128 for the base models in our work instead of 256 as in the original setting due to limited resources. The mask ratio is 15%. We set maximum number of tokens as 128 for small models and 512 for base models The small models are pre-trained from scratch for 1000k: steps. To save computation, like previous studies (Dong et al.) 2019), we continue training base models for 200k steps using the pre-trained weights as initialization. The learning rates for small and base
We find that combining the strategies would yield clear advantage over each individual. The possible reason would be the redundancy which may lead to similar effects.
For evaluation on the reading comprehension tasks, we also pre-train the variants with the length of sen- tences each batch up 512 tokens.
5

---

Under review as a conference paper at ICLR 2022
Model
CoLA SST
MRPC
STS
QQP
MNLI
QNLI RTE Average
Single model on dev set
ELECTRA Ammal
56.8
88.3
87.4
86.8
88.3
78.9
87.9
68.5
80.4
ELECTRA Sit 61.1
90.1
89.5
87.0
89.4
80.8
88.8
68.6
81.9
11.5
small
ELECTRA small
62.0
89.8
87.0
86.7
89.0
80.4
88.0
67.9
81.4
11.0
ELECTRA Ahase
68.3
95.3
90.9
91.3
91.7
88.5
93.0
82.3
87.7
ELECTRA
SR
70.4
95.4
90.4
91.2
91.9
89.1
93.4
84.8
88.3
10.6
ELECTRA base
HC
70.9
95.6
91.2
91.3
92.0
88.7
93.6
83.8
88.4
10.7
Single model test
ELECTRA (ECTRASmaa)) smmal
52.3
89.7
84.8
80.5
88.4
79.9
88.0
62.9
78.3
ELECTRA fhaall
ELECTRA SR 58.3
90.6
85.4
81.4
87.9
80.6
88.0
64.3
79.6
11.3
55.3
90.3
84.1
82.0
87.2
80.6
88.4
64.3
79.0
10.7
small
ELECTRA
62.4
95.3
87.3
89.9
89.6
88.6
93.4
78.1
85.6
ELECTRA SIR
65.7
95.7
88.3
90.0
89.9
89.1
93.6
78.8
86.4
10.8
base
ELECTRA hase
HC
67.5
95.8
88.6
89.9
89.7
89.0
93.6
79.1
86.7
11.1
Table 2: Comparisons between proposed methods and the previous strong pre-trained models under small and base setting on the dev and test set of GLUE tasks. STS is reported by Spearman correlation, CoLA is reported by Matthew's correlation, and other tasks are reported by accuracy.
models are 5e-4, and 5e-5, respectively. We use Open WebTxt (Radford et al. 2019) train small models, and Wikipedia and BooksCorpus (Zhu et 2015) for training base models following Clark al. 2019 The baselines are trained the same steps for a fair comparison.
Fine-tuning For evaluation, we fine-tune the pre-trained models GLUE (General Language Understanding Evaluation) (Wang et af., 2018) and SQuAD v1.1 (Rajpurkar et al.) 2016) to eval- uate the performance of the pre-trained models. GLUE include two single-sentence tasks (CoLA (Warstadt et al. 2018), SST-2 (Socher et al. (013)), three similarity and paraphrase tasks (MRPC (Dolan & Brockett 2005). STS-B (Cer et al. 2017). QQP (Chen al., 2018 three inference tasks (MNLI (Nangia et al. 2017). QNLI (Rajpurkar et al. 2016). RTE (Bentivogli et al. 2009). We follow ELECTRA hyper- parameters for single-task fine-tuning. We did not any training strategies like starting from MNLI, I, to avoid extra distractors and focus on the fair comparison the single-model and single-task settings.
4.2 RESULTS
We evaluate the performance of pre- training enhancement compared with the base- lines in small and base sizes GLUE and SQuAD benchmarks in Tables 23 From the results, have the following observations:
Model
Exact Match
F1 Score
ELECTRA Ammal
75.8
83.9
ELECTRA AR
76.0 (10.2)
84.2 (10.3)
small
ELECTRA small
77.7 (11.9)
85.6 (f1.7)
I) The models with our enhanced pre-training objectives outperform the baselines in all the subtasks. With the same configuration and pre- training data, for both the small-size and the base-size, our methods outperform the strong ELECTRA baselines by +1 +1.5(dde))+133(ttss) and +0.7(dev)4+1.1((e))
ELECTRA AaCCee
85.1
91.6
ELECTRA A
85.6 (+0.5)
92.0 (f0.4)
hune
ELECTRA base
85.7 (10.6)
92.1 (10.5)
Table 3: Results on the SQuAD dev set.
on average, respec- tively. The results demonstrate that our pro- posed methods improve the pre-training of ELECTRA substantially and disclose that mediating the training with true negatives is quite beneficial for improving language model pre-training. To verify
*Our codes and models will be publicly available.

---

Under review as a conference paper at ICLR 2022
Model
Params
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE
Avg.
A
BERT
110M
52.1
93.5
84.8
85.8
89.2
84.6
90.5
66.4
80.9
BERT large
335M
60.5
94.9
85.4
86.5
89.3
86.7
92.7
70.1
83.3
SpanBERT lange
large
335M
64.3
94.8
87.9
89.9
89.5
87.7
94.3
79.0
85.9
ELECTRA Aamal
14M
54.6
89.1
83.7
80.3
88.0
79.7
87.7
60.8
78.0
ELECTRA Abase
110M
59.7
93.4
86.7
87.7
89.1
85.8
92.7
73.1
83.5
ELECTRA
SR
14M
58.3
90.6
85.4
81.4
87.9
80.6
88.0
64.3
79.6
11.6
ELECTRA HIGll
14M
55.3
90.3
84.1
82.0
87.2
80.6
88.4
64.3
79.0
11.0
SRAIl
ELECTRA
110M
65.7
95.7
88.3
90.0
89.9
89.1
93.6
78.8
86.4
12.9
bage
ELECTRA
HC
110M
67.5
95.8
88.6
89.9
89.7
89.0
93.6
79.1
86.7
13.2
basse
Table Comparisons with public methods on GLUE test sets. The public results are from BERT
Devlin et al. 2019), SpanBERT (Joshi et al. 2020), and ELECTRA (Clark et al. 2019).
Model
CoLA
SST
MRPC
STS
QQP
MNLI
QNLI
RTE
Average A
ELECTRA small
small
56.8
88.3
87.4
86.8
88.3
78.9
87.9
68.5
80.4
ELECTRA A Word
SR
61.1
90.1
89.5
87.0
89.4
80.8
88.8
68.6
81.9
11.5
ELECTRAS Sent
59.5
89.6
90.0
86.7
89.1
80.4
90.0
68.2
81.6
11.
ELECTRA HC WordNet
62.0
89.8
87.0
86.7
89.0
80.4
88.0
67.9
81.4
11.0
HC ELECTRA Embeediig
59.0
88.5
87.0
86.4
88.8
79.6
87.9
67.1
80.6
10.2
Table 5: Comparative studies of variants on GLUE dev sets based on small models. The first block compare the word-level regularization and sentence-level regularization, respectively. The second block shows the results of HC methods based on WordNet and Word2Vec embedding, respectively.
the generality of methods, we also implement them BERT backbones details presented in Appendix A.2 The results show that our methods achieve consistent gains.
2) Table shows the comparison with public models the GLUE test set. Compared with the public methods, our model not only far exceeds the performance of others under the same model scale, but also outperforms the larger models with much fewer parameters.
3) The performance gains on the small-size models are more obvious than the base-size models. We speculate that is due the learning of the small-size generator more insufficient and suffers from the false negative issue more seriously.
4) Both SR and HC pre-training strategies help the resulting model surpass the baselines obviously. Note that our proposed method is model- so that the convenient usability of its backbone precursor can be kept without architecture modifications. In comparison, SR is more generalizable as it does not require extra resources, while HC has the advantage of interpretation via explicit correction.
5) Our enhanced pre-training objectives show considerable performance improvements on linguistics-related tasks such as CoL.A and MRPC. These tasks are about linguistic acceptability and paraphrase/semannte equivalence relationship. In addition, our methods also achieve obvious gains tasks requiring more complex semantic understanding and reasoning, such as MNLI and SQuAD, showing that they may help capture semanties to some extent.
6) Our methods are lightweight that keep nearly the same parameter size, computation requirement, and training speed as the bascline but with stronger capacity.
ANALYSIS
5.1
WORD-LEVEL REGULARIZATION VS. SENTENCE-LEVE. REGULARIZATION
The soft regularization approach measures the semantic distance between the predicted one and the ground-truth, which may neglect the sentence-level context (though the token representation may

---

Under review as a conference paper at ICLR 2022
have already captured contextualized representation to some extent). We are interested in whether measuring the sentence-level similarity would achieve even better results. To verify the hypothesis, fill the masked sentence s' with the predicted tokens c'g to have the predicted sentence Sp. Then, Sp and 8 are fed to the Transformer encoder to have the contextualized representation Нp and Hs, respectively. To guide the probability distribution of model predictions Hp to match the expected probability distribution Ha, we adopt Kullback-Leibler (KL) divergence:
Lu
(H, H,);
where Cui is applied as the degree to reflect the sentence level semantic mismatch. The loss function
then written as L' Lalm + Lxi-
For clarity, denote the original ELECTRA Amall
SR small method described in Eq. ELECTRAWos
Sit The comparative results are reported in the and the sentence-level variant as ELECTRA Sent' Sent"
first block of Table [] which indicates that using sentence-level regularization (ELECTRAŞH also substantially outperforms the baseline and nearly reaches the performance of word-level SR
one (ELECTRA Word) ELECTRA
average, with slightly better results on MRPC and MNLI. Although SR still keeps the same parameter size with baseline, it leads to more computation re- Sent
sources because requires the extra calculation of the contextualized representation for the pre- dicted token sequence Hp. Therefore, considering the balance between effectiveness and efficiency, SR ELECTRA Word Word can serve as the first preferred choice for practical applications, and ELECTRA A SR Sent
can be employed when computation resources are sufficient.
5.2 RETRIEVING SYNONYMS FROM WORDNET VS. WORD2VEC EMBEDDINGS
For the hard correction approach, the candidate synonyms for detecting false negative predictions can be derived from WordNet (Miller 1995) or Word2 embedding space (Mikolov et al. 2013) as described in Section 3.22 $ To verify the impact of different sources, we compare the results as shown in the second block of Table We see that ELECTRAHC HC WordNet outperforms ELECTRA E.ECcRAAEmbeddinn Embedding by large margin. The most plausible reason would be that the retrieved list of synonyms from HC ELECTRA WordNet WordNer would have higher quality than that from ELECTRA HC Embedding Although the embedding based method may benefit from semantic matching, but would also bring noises as hard to set the threshold ensure the top-ranked words are accurate synonyms. Therefore, ELECTRANC WordNet turns out to be better suitable for our task.
HC
To interpret how our method works, we randomly select some semantic correction examples as shown in Fy taking the baseline as the backbone model. We find that the baseline model pro- duces reasonable predictions such as main, remain, attempt as opposed to the golds ones, primary, stay, effort. Those predictions will be determined as wrong and then harm pre-training. Fortunately, such cases can be easily solved by our proposed method.
5.3 ROBUSTNESS EVALUATION
Intuitively, our method would be helpful for improving the robustness of the pre-trained models because the approaches may indicate lexical semanties and representation diversity during the cor- rection or regularization operations. To verify the hypothesis, we use a robustness evaluation plat- form TextFlint (Wang et al., 202I) on SQuAD, from which two standard transformation methods are adapted: 1) AddSentenceDivere generates distractors with altered questions and fake answer sand 2) SwapSyn WordNer transforms an input by replacing words with synonyms provided by WordNet.
Table shows the robustness evaluation results. We observe that both kinds of attacks induce a significant performance drop of the baseline system, by 54.95% and 6.0% on the EM metrics, re- spectively, indicating that the system sensitive to distractors with similar meanings. In contrast, both of our models can effectively resist those attacks with less performance degradation. Specifi- WorrdNe
cally, the HC method works stably in the SwapSyn WordNet attack. We speculate the reason that the hard correction strategy models the synonym information during pre-training, which would help
SSince the emhedding method returns ranked list by calculating the similarity score with the whole vocab- ulary, only take the top 10 most similar words for each retrieval.
8

---

Under review as a conference paper at ICLR 2022
antibodies are protein components an adaptive immune system whose antigen ##s, or other substances in the body
[MASK] function is bind
(a)
Gold:
primary
[primary_winddnng, 'principall, 'master', elementary" 'chief, pprimary. 'elemental', "basal', primary quill' 'main', primany__eecction" |primary_ feather', "primary_coil coil!]
Pred:
they could not have been in good shape after fighting into the next day in intense moderate heat and having
[MASK]
position overnight, far from helpless and harassed by the infantry
(b)
Gold:
stay
I'continue', 'halt', 'stop', 'chck', "delay". 'quell' 'arrest', 'remain', 'bide', 'detain', persist', 'stick around', 'ride__uu _ut', 'last_out, out', 'stay', 'abide', 'stoppage',
Pred: remain
in 1967, seven ka ##kaa #tipo were transferred to little #itbury island, in an [MASK] establish successful breeding network. ka iitkaa fitppo were last on the island in 1999.
(c)
Gold:
effort
I'campaign', 'elbow grease', : 'endeavor', 'sweat', 'cause', 'tyy. 'exertion', 'exploit', 'drive', 'attempt', 'travail', 'russade, 'feat', 'endeavour', 'movement" eefoort
Pred:
attemppt
Figure 2: Interpretation of the semantic correction process. The orange box contain the input sen- tence, the blue buttoms indicate the gold and predicted tokens, and the green box shows the candidate synonyms from WordNet given the predicted token.
AddSentenceDivvess
Swap SwordSynWrrNNer WordNet
(Ori.-+Tranl
Model
Exact Match
Fl Score
Exact Match
Fl Score
ELECTRA gmaal 80.55-+25.66 (L54.95) 85. +3..44 (L.88677 80.67-474.67 (16.00) 85.38 85.38-480.43 (14.95)
ELECTRA SR emall +8.84->37.20 (.41.64) 80.84 +3.229 (142.55) 78.67-+75.5 (13300) 80.88 +78.51 (12.37)
ELECTRA small 22.59-+34.13 (148.46) -6.788+36.60 (150.18)
(12.66) 28.66563..6 (13.03)
Table Robustness evaluation on the SQuAD dataset. Ori. represents the results of original dataset derived from the SQuAD 1.1 dev set by TextFlint (Wang et al.) (021) while Trans. indicates the transformed one. The assessed models are the small models from Table
capture lexical semantics. The other variant, the soft regularization objective, achieves much bet- ter performance in the AddSentenceDivvess. The most plausible reason might be the advantage of acquiring semantic diversity by regularizing the semantic distance in the SR objective. The results indicate that both methods achieve similar effects of robustness in general but also have some slight emphasis.
CONCLUSIONS
Though discriminative PrLMs may quite straightforwardly suffer from the false negative issue cording to our exploration in this work, it has been completely ignored for a long time and it is bit surprising that maybe this work the first one that formally considers such a big pre-training leak. To counteract the intrinsic and critical issue, we employ extra pre- pre-training objectives to correct or prune the harmful gradient update after detecting the false negative predictions. Experimental results on GLUE and SQuAD benchmarks verify the superiority of our pre-training enhancement. Robustness evaluation shows that our methods can help the resulting PrLM effectively resist various attacks while existing common PrLMs would suffer from significant performance degradation. To our best knowledge, guage model vee-rrainnng at the same time. Our work indicates that mediating false negatives is so important that counter- counter-false-neene nelse-ne pre-training can indeed synchronously improve the effective- ness and robustness of Pr..Ms.
also the first work to consider model effectiveness and robustness of lan-

---

Under review as a conference paper at ICLR 2022
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In ACL-PASCAL, 2009.
Antoine Bordes, Nicolas Usumier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information pro- cessing systems, 26, 2013.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task
I: Semantic textual -aulatttimmllllgualll and cross-lingual focused evaluation. arXiv preprint 1xXi::708.00055: 2017.
Wang Chen, Piji Li, Hou Pong Chan, and Irwin King. Dialogue summarization with supporting utterance flow modelling and fact regularization. Knowledge-Baad Based Systems, 229:107328, 2021.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019. URL
net /pdf r
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: Findings, pp. 657-668, 2020.
Jacob Devlin, Ming-We Bei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 161186611/N11914223 URL https:
7/acl ogy. oww.y//-g/NN111422
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In /WP2005, 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Proceedings of the 33rd International Conference on Neural Information
Processing Systems. pp. 13063-13075, 2019.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6):189--8111 2021.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint MXXi:20022008009 2020.
Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing. In
Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP 2019), pp. 223-430, 2019.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In Proceedings of the 58th Annual Meeting of the Association, for Computa- tional Linguistics, pp. 2177-2190, 2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-training by representing and predicting spans. Trunsactions of the Associ- ation for Computational Linguistics, 8:64-77, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. In Interna- tional Conference on Learning Representations, 2019. URL htttps:// net/ Idffid=Hleas eA
10

---

Under review as a conference paper at ICLR 2022
Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton- Leyton-Brown, Moshe Tennenholtz, and Yoav Shoham. [PMI)-masknn Principled masking of correlated spans. In International Conference Learning Representations, 2021. URLhttps:/ http:///wpneeeeeeeee/eee nermm (d=3Aoft6NWFee
Mike Lewis Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-seuye -equuence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/112000.aclmmaaa org.:/llooo.oo////0//
URL
2020 acl-main. 703
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7887 2020b.
Yian Li and Hai Zhao. Pre-training universal language representation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 5122-5133, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18665/1122221.cc URL
org/
2ttgg/2220 2021 www.aaaclllanl
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. In International Confer- ence on Learning Representations, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0784/ 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems (NIPS 2013), pp. 3111-3119, 2013.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):
39-41, 1995.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in Neural Information Processing Systems, 32:4694 4703, 2019.
Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R Bowman. The repeval 2017 shared task: Multi-genre natural language inference with sentence representations. In RepEval, 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-IILT, pp. 2227-2237, 2018. URL httht::/// http::/wwwwwe.. www. http://.wwwww..... acwebb. www.ee...rgg.aanoooooooo///o
1202.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-wess--. amazonaws. com/openai-
understanding paper. pdf. 2018. URL
https::/s3-us west 2 wwwzonaws com openai www.:aaeteeteeeeeeeeeeeeeeeeeeee www.tteeeeeeereeceeeeeee..eee
pdf
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
11

---

Under review as a conference paper at ICLR 2022
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association Computational Linguistics, 8:842-866, 2020.
Richard Socher, Alex Perclygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Kaitao Song, Xu Tan, Tao Qin. Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre- training for language understanding. arXiv preprint arXiv:2004009297, 2020.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv 2019.
preprint
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 118653//1//18-5444 URL BtLhttps::/ ac antho logy. org/ WW1 8-5446566666
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. Struct- bert: Incorporating language structures into pre-training for deep language understanding. In International Conference on Learning Representations, 2019.
Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, al. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 347-355, 2021.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 0.005./247. /2471, 2018
John Wieting, Taylor Kirkkaiikat Kevin Gimpel, and Graham Neubig. Beyond bleu: Training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
2019.
Chien-Sheng Wu, Steven CH Hoi, Richard Socher, and Caiming Xiong. Tod-bert: Pre-trained natural language understanding for triened dialogue. In Proceedings of the 2020 Conference on Empiricul Methods Natural Language Processing (EMNLP), pp. 917-929, 2020.
Yi Xu and Hai Zhao. Diaeotue pre-training arXiv preprint arXiv: 00420, 2021.
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. Syntax-enhanced pre-trained model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5412-5422, On- line, August 2021. Association for Computational Linguistics. doi: (0.18863/v112022 aacl-ooo...lo-l......o 420. URL htt.htttss:/aaaaathoolleo
org 2021 walliiaallloonn 420
Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, and Nikolaos Aletras. Frustratingly simple pretraining alternatives to masked language modeling. arXiv preprint arXiv:2109.01819 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neu- ral information processing systems, pp. 5754-5764 2019.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
12

---

Under review as a conference paper at ICLR 2022
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. Dialogpt: Large-scale generative pre-traine pre-training for conver- sational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, pp. 270-278, 2020a.
Zhilu Zhang and Mert Sabuncu. Self-distillation as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33, 2020.
Zhuosheng Zhang and Hai Zhao. Structural pre-training for dialogue comprehension. Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5134-5145, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 ael.aacl-oog,,9, URL TRL.ttpp.. wttpt:tp///o...... a antho loyy. www.org//00 2021
0002221 .ac.-lllllll....
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. SG-Net: Syn- tax guided transformer for language representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020b.
Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. LIMIT-BERT: Linguistics in- formed multi-task bert. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: Findings, pp. 4450-44611 2020.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
A APPENDIX
A.1 WOULD THE HARD CORRECTION BRING FALSE POSITIVES?
The hard correction would not bring false positives because it is post-pur pest-prcessing technique. As described in the last paragraph of Section if the predicted token is in the shortlist, the correction will be activated by simply ignoring this prediction before feeding cross-entropy loss function. In our Py Torch implementtaion, for example, the corresponding gold label id will be replaced by -100 (default ignore index in CrossEntropy using PyTorch), which means this token required to predict anymore.
CoLA Mcc
SST
MRPC
STS
QQP
MNLI Acc
QNLI
RTE
Average
Model
Acc
Acc
Spear
Acc
Acc
Acc
BERT base
61.09
93.00
86.76
87.09
90.79
84.72
91.42
67.87
82.84
BERT SR
61.17
93.46
88.97
87.45
90.93
84.83
91.62
68.59
83.38
hage
BERT
62.88
93.23
87.50
87.41
90.92
84.92
91.54
69.31
83.46
BERT Large
61.67
93.69
88.48
90.14
91.30
86.74
92.37
72.92
84.67
BERT Sit
62.26
94.15
89.22
90.12
91.41
87.01
92.82
74.01
85.13
large
BERTHC 62.34
93.35
88.97
90.48
91.46
86.96
92.95
73.65
85.02
large
Table 7: Results of BERT methods under base and large setting on the GLUE dev sets. STS is reported by Spearman correlation, CoLA is reported by Matthew's correlation, and other tasks are reported by accuracy.
A.2 COULD THIS METHOD BE APPLIED ON OTHER MLM PRLMS?
To verify the generality our methods on other PrLMs, we implemented them on BERT base and BERT large backbones (Devlin al. 2019) following the same implementation for ELECTRAbase as described Section 4.1 Specificalll, we train MLM with our methods based on BERT base and BERT large checkpoints for 200h steps the Wikipedia and BooksCorpus, and fine-tune them
13

---

Under review as a conference paper at ICLR 2022
Checkpoint (base)
Iteration
Prediction
Checkpoint (large)
Iteration
Prediction
6.25%
6.90%
1.31%
6.25%
7.46%
1.5%
12.5%
6.96%
1.34%
12.5%
7.58%
1.55%
25.0%
6.97%
1.36%
25.0%
7.31% %
1.49%
50.0%
7.05%
1.36%
50.0%
7.46%
1.56%
80.0%
7.06%
1.40%
80.0%
7.38%
1.57%
100.0%
7.07%
1.41%
100.0%
7.44%
1.60%
Table 8: Statistics of the hard corrections under base and large settings on the wikitext-2-raww-i corpus. Checkpoint means the checkpoint saved at the specific training steps (%).
GLUE tasks. For fair comparison, we train the baseline models based on the same checkpoint in the that our methods achieve consistent gains on BERT methods.
same manner. Results
A.3 STATISTICS OF THE HARD CORRECTIONS
To have intuition about how the hard correction works during pre-training, collect the statistics of the hard corrections in two perspectives: 1) prediction-level: the proportion of corrected predic- tions when they mismatch the gold labels; 2) iteration-level: the proportion of iterations when the correction happens. As the training corpus is relatively large, we use the wikitext-2-raw--' corpus (Merity al. 2016) for efficient validation as suggested by Transformers We use the pre-trained checkpoints with hard correction on the backbones of BERT-base and BERT-large models for the analysis.
Table 8 shows the statistics, from which we have the following observations:
1) The correction ratio is around 1.0%% in token-level and around 6.0%-7.0% iteration- level.
2) As the training goes on, the correction ratio increases, indicating that our method would gradually play more important role when the training goes on, which supports our hypothesis.
3) The correction ratio in larger models would be higher than the base models, which indicates larger models would be more likely to encounter false negatives.
A.4 How CAN WE MEASURE THE SEVERITY OF THE FALSE NEGATIVE PROBLEM?
that our method is gradually playing a more important role when the training goes on. Since the training examples are based on random masking, the PrLMS thus forced to be trained on low-quality samples.
As indicated in the Section
A.3
As the saying goes, "The rotten apple injures its neighbors", training on random low-quality exam- ples would bring training bias from meaningless data, it needs to be corrected with more data and results in more cost of resource and time. Our methods can be regarded as the training correction to help the model train more "true samples", thus, they would improve the training efficiency and help the model to get rid of adversarial attacks like diversity distraction and synonym substitution.
14


Reviews:
[{"review_id": "ueLIztFF2dG", "reviewer": "Reviewer_1R2h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- As summaried above, this paper presents technically sound methods (i.e., SR and HC) to mitigate false negative problem in masked language models. \n- Besides the main evaluation to beat baseline performance on natural language understanding tasks, this paper also includes more insightful analyses into the proposed methods, from both quantitative and qualitative perspectives. \n\n\nCons and Questions:\n\n- The motivation of this paper is not that convincing to me. I fully agree \"it is critical to ensure the auto-constructed data is true enough\" but how can we measure the severity of \"false negative problem\"? Given the 2nd & 3rd examples in Table 1, although both \"Ground-truth\" and \"Prediction\" can fit the masked tokens, there are still subtle differences between them especially in long contexts (up to 512/1024 tokens during pretraining vs. a short sentence in the examples). \n- There are also some weaknesses in the proposed pre-training objectives. As for SR, the regularization is only applied to embedding layer. As for HC, Wordnet can only provide a very coarse Symset, which may introduce error propagation. Can author give more empirical studies to demonstrate the successes of these objectives? \n- Why did the author only apply the proposed pre-training objectives on the top of ELECTRA instead of standard MLM, e.g., RoBERT. The latter should be a better testbed for the objectives targeting problems in MLM. \n- Can authors discuss the relation of this work to a recent distillation paradigm, i.e., distill a small LM to large LM for accelerated convergence and promoted performance, which is mainly attributed to label smoothing. \n", "summary_of_the_paper": "This paper focuses on the issue of misconceived false negative predictions and encourage pre-training language models on true negatives or more true negatives to ensure the auto-constructed data in masked language modeling (MLM) is true enough. A motivation behind this is that  false negatives may potentially hurt the pre-training in both efficiency and robustness to a great extent and this vulnerability of pre-trained language has never been studied before. In this paper, the authors propose an enhanced pre-training approach to counteract misconceived negatives.\n\nBased on ElECTRA, the authors present two alternative pre-training objective auxilary to MLM one, which aims to make the model focus on true negatives during MLM. The first one, dubbed as soft regularization (SR), measures the distribution similarity between the predicted token and the original one, to smooth the tough cross-entropy by minimizing the semantic distances. The other, named as hard correction (HC), hinder the gradient propagation of the false negative samples to further avoid training with false negative predictions. ", "summary_of_the_review": "Technically sound methods but unconvincing motivation and ill-considered methods; More empirical studies needed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "njBSuwAez0-", "reviewer": "Reviewer_dhbH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- S1 - The idea of handling true negatives for ELECTRA pre-training is an interesting idea.\n- S2 - Experimental results show that proposed pre-training objectives can improve the performance of ELECTRA.\n\nWeaknesses and Questions:\n- W1 - Quiet unsure about the experimental settings. Seems SR and HC are continually pre-training objectives. Then, are baselines (ELECTRA_small and ELECTRA_base) also continually pre-trained with original ELECTRA pre-training objectives for the same steps of SR and HC? If not, the results are not convincing.\n- W2 - Unclear the performance improvement is solely from handling true negatives. what about (1) random (2) false negatives?\n- Q1 - It would be great if the paper could provide the correction statistics of HC.\n- Q2 - ELECTRA-style token-level discriminative pre-trained models are showing good performance in token-level classification tasks such as NER. It would be great if we could see that performance as well.", "summary_of_the_paper": "The premise of the paper is that true negative examples should be considered in ELECTRA-style token-level discriminative pre-training. The framework uses cosine similarity to find true negatives and softly regularize with semantic distance, or finding true negatives by synonyms and simply ignoring the prediction before feeding to the loss function. Results on GLUE and SQUAD with ELECTRA-based models show that handling true negatives during pre-training improves performance.", "summary_of_the_review": "Although the paper shows good results compared to the baseline, current writing is not convincing.\nIt needs more details and extensive analysis to support the claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bIY-GHMPSrV", "reviewer": "Reviewer_syob", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The paper identified the false negatives issue in PrLMs and could possibly inspire other researcher to work on this.\n2) The experiments are conducted on two benchmarks and the proposed methods show consistent gains over a strong baseline model.\n\nWeakness:\n1) For the soft regularization in equation (6), I am not sure how this loss can be backpropagated since the token prediction step is a discrete argmax function.\n2) For the hard correction, it is unclear whether the WordNet will also bring false positive, i.e. a synonym which should not fill in the mask.\n3) The experiments only cover one pretrained language model - ELECTRA. Could this method be applied on other MLM PrLMs such as RoBERTa?\n4) I also expect some statistics of the hard corrections in the pretraining experiment.\n\nPresentation Issues:\n1) In table 2, the test set result of ELECTRA_small on QQP is the best but the number is not bold.", "summary_of_the_paper": "The paper identified the false negative issue in the pre-trained language models and proposed two methods to counteract it. The first method replaces the cross-entropy loss in the standard MLM objective with the cosine distance between the representations of predicted token and the ground truth. The second method prunes the gradients from the false positive tokens, where such tokens are identified from synonym alternatives of WordNet. Empirical results on GLUE and SQuAD show that the proposed solution improves ELECTRA-small and ELECTRA-base models for both performance and robustness.", "summary_of_the_review": "The paper proposed two simple methods for a novel issue in PrLMs. The empirical results are promising in two benchmarks with ELECTRA model. However, more technical description, empirical analysis, and experimental result are still needed to fully support the claim of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1OJM0qvr14", "reviewer": "Reviewer_QW2m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of reducing false negative examples is reasonable but still does not convince me that this is necessary in pretraining, given that the main goal of pretraining is to learn good contextualized representation that could be useful for various downstream tasks. The authors claim the false negative examples are harmful to pretraining, but it is not fully supported by the experiments in the paper. The intuition of soft regularization and hard correction is somehow ad-hoc and may also not hold under some circumstances. For example, when the model is not well trained, minimizing the distance between the predicted token and the ground-truth token could be misleading if the prediction is incorrect. The synonyms from the dictionary could also be wrong without considering the given context. In equation (6), it is unclear whether the embedding e_k is token embedding or contextualized representation, and from generator or discriminator.\n\nSome experimental results are problematic and not convincing. \n1)\tThe proposed methods are only applied to continual training of a well-trained ELECTRA model. This may indicate they are not suitable for early-stage pretraining, as the assumptions may not hold. This is a great limitation as a pretraining strategy, as it is not clear at which training steps one can use it. Moreover, as a fair baseline, ELECTRA should also be continually trained with the same 200k steps as the proposed models. \n2)\tThe results in Table 2 are inconsistent with Table 1 (ELECTRA_small and ELECTRA_base on GLUE test set). Some important baseline such as RoBERTa, XLNet, DeBERTa are not included either. Moreover, since all the experiments are conducted on ELECTRA small and base model, it is unknown whether the proposed methods work on larger scale models and other types of pretrained models such as BERT.\n3)\tIn Table 6, what is the baseline system, BERT or ELECTRA_small?\n", "summary_of_the_paper": "This paper proposes to avoid false negative examples in language model pretraining via modifying the training objective. Two methods, referred to as soft regularization and hard correction, are proposed to reduce the false negatives based on continual training on the ELECTRA model. Experiments on the GLUE benchmark show that the continual trained model with the two methods can both improve the performance on downstream NLU tasks. Experiments on SQuAD show the methods also help with model robustness.\nThis work provides ways to adjust the pre-training objectives when the model predictions are correct but do not match the ground-truth tokens. This could benefit the performance on downstream task in some cases. But the limited experiments in the paper do not assure that this is a general approach in pretraining that can be widely adopted.", "summary_of_the_review": "The methods proposed in this paper have clear theoretical and empirical limitations. The experiments do not fully support the claims by the authors. My current recommendation to this paper is reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0E68RgAUaX", "reviewer": "Reviewer_jW6M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:\n1. The false negative problem tackled in this paper is critical for pretraining and seems to be ignored by the mainstream. I believe improvement on this problem can lead to better representation models, and can be inspiring to other pretraining objectives.\n2. The method brings consistent gains compared to fair baselines on two standard benchmarks.\n\n*Weaknesses:\n1. My main concern is about the proposed methods in this paper. First, for the Soft Regularization method, it is not clear how the loss L_reg backpropagates through the model. Is the author only training the two embeddings? It doesn't make sense to me that this can improve the model.\n2. For the Hard Correction method, it looks to me like the author is just using a synonym set as the target labels. Despite of the good empirical results, the solution itself lacks novelty. Also, although the WordNet synonyms can mitigate false negative problem, it can also brings new false positives. \n3. The author mainly compares their method with the original pretrained model itself, rather than other methods in solving false positives. I am curious if add label smoothing during training can also bring gains and how it compares with the proposed method in this paper.", "summary_of_the_paper": "This paper identifies the largely neglected false negative problem in training or pretraining language models. For many cases in training LM, the training objective only uses one single token as the target, while treating other meaningful tokens as equally negative. To tackle this problem, the author proposes two methods: 1) the Soft Regularization method minimizes the embedding distance between the predicted and gold tokens; 2) the Hard Correction employs WordNet and maximizes all the synonyms of the gold token. They demonstrate consistent gains and better robustness by applying this to pretraining ELECTRA, and testing on GLUE and SQuAD downstream tasks. ", "summary_of_the_review": "This paper studies a critical problem of false negatives in pretraining language models and demonstrates good results. But the proposed solution lacks enough novelty and might have some technical flaws.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper gets decent performance gains (~2% on GLUE) by soft regularization to make negatives closer to positives in contrastive learning and hard correction of too-close negatives to at least avoid synonyms. These are useful ideas which to some extent build on the simple technique of ELECTRA (controlling the size of the generator MLM in Electra encourages the negatives to in general be "close but not too close", right?). As such, the paper is correct and provides potentially useful gains, but it appears a quite small adjustment of existing techniques, and in addition the use of WordNet is fairly brittle (and its similarity calculations do not consider context at all). 

The authors should be commended for the thorough job they did at updating their paper to address particular questions and concerns of reviewers, and useful new information emerged. Relative to the question of whether this method can be applied with other MLMs, the new Appendix A results do show that the answer is Yes, but the gains turn out to be much more modest (~0.5% on GLUE). However, ultimately, while this is all useful information and side experiments, these improvements just can't fix the key problem that all the reviewers felt that this paper does not provide sufficient "Technical Novelty and Significance". As such without bigger new ideas, this improved paper would probably be best as a good workshop paper.

My recommendation is that this paper not be accepted to ICLR 2022 on the basis of its limited technical novelty and significance.
Decision: Reject