Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
Authors: Mingzhi Zheng
Dinghan Shen
yelong shen
Weizhu Chen
Lin Xiao
Venue:  (2021)
Keywords: representation learning
natural language processing

Abstract: Masked Language Model (MLM) framework has been widely adopted for self-supervised language pre-training.  In this paper, we argue that randomly sampled masks in MLM would lead to undesirably large gradient variance. Thus, we theoretically quantify the gradient variance via correlating the gradient covariance with the Hamming distance between two different masks (given a certain text sequence). To reduce the variance due to the sampling of masks, we propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments. Thereafter, the tokens within one segment are masked for training. We prove, from a theoretical perspective, that the gradients derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training. We conduct extensive experiments on both continual pre-training and general pre-training from scratch. Empirical results confirm that this new masking strategy can consistently outperform standard random masking. Detailed efficiency analysis and ablation studies further validate the advantages of our fully-explored masking strategy under the MLM framework.

Under review as a conference paper at ICLR 2021
IMPROVING SELF-SUPERVISED PRE-TRAINING via A MASKED LANGUAGE MODEL
Anonymous authors
Paper under double-blind review
ABSTRACT
Masked Language Model (MLM) framework has been widely adopted for self- supervised language pre-training. In this paper, we argue that randomly sampled masks in MLM would lead to undesirably large gradient variance. Thus, we the- oretically quantify the gradient variance via correlating the gradient covariance with the Hamming distance between two different masks (given certain text se- quence). To reduce the variance due to the sampling of masks, we propose a fully-explored masking strategy, where text sequence is divided into certain number of non-overlapping segments. Thereafter, the tokens within one segment are masked for training. We prove, from a theoretical perspective, that the gradi- ents derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training. We conduct extensive experiments on both continual pre-training and general pre-training from scratch. Empirical re- sults confirm that this new masking strategy can consistently outperform standard random masking. Detailed efficiency analysis and ablation studies further validate the advantages of our fully-explore masking strategy under the MLM framework.
INTRODUCTION
Large-scale pre-trained language models have attracted tremendous attention recently due to their impressive empirical performance on a wide variety of NLP tasks. These models typically abstract semantic information from massive unlabeled corpora in sulf-supervised manner. Masked lan- guage model (MLM) has been widely utilized as the objective for pre-training language models. In the MLM setup, certain percentage of words within the input sentence are masked out, and the model Icarns useful semantic information by predicting those missing tokens.
Previous work found that the specific masking strategy employed during pre-training plays a vital role in the effectiveness of the MLM framework (Liu et al., 2019 Joshi et al.) 2019 [Snn et al.) 2019). Specifically. Sun et al. (019 introduce entity-level and phrase-level masking strategies, which incorporate the prior knowledge within sentence into its masking choice. Moreover, Joshi eet al. (2019) propose to mask out random contiguous spans, instead of tokens, since they can serve as more challenging targets for the ML. objective.
Although effective, we identify an issue associated with the random sampling procedure of these masking strategies. Concretely, the difficulty of predicting each masked token varies and is highly dependent on the choice of the masking tokens. For example, predicting stop words such as "the" or "a" tends be easier relative nouns rare words. As result, with the same input sentence, randomly sampling certain input tokens/spans, as a typical masking recipe, will result in undesirable large variance while estimating the gradients. It has been widely demonstrated that large gradi- ent variance typically hurts the training efficiency with stochastic gradient optimization algorithms (Zhang & Xiao| 2019; Xiao & Zhang 2014 Johnson & Zhang, (013). Therefore, we advocate that obtaining gradients with a smaller variance has the potential to enable more sample-efficient learning and thus accelerate the self-supervised learning stage.
In this paper, we start by introducing a theoretical framework to quantify the variance while esti- mating the training gradients. The basic idea is to decompose the total gradient variance into two terms, where the first term is induced by the data sampling process and the second one relates to the sampling procedure of masked tokens. Theoretical analysis on the second variance term demon- strates that it can be minimized by reducing the gradient covariance between two masked sequences.

---

Under review as a conference paper at ICLR 2021
Furthermore, conduct empirical investigation the correlation between the gradient's 's covari- ance while utilizing two masked sequences for training and the Hamming distance between these sequences. We observed that that the gradients' covariance tends to decrease monotonically w.r. the sequences' Hamming distance.
Inspired by the observations above, we propose a efully-explored masking strategy, which maximizes the Hamming distance between any of two sampled masks on a fixed text sequence. First, a text sequence is randomly divided into multiple non-overlapping segments, where each token (e.g. sub- word, word or span) belongs to one of them. While the model processes this input, several different training samples are constructed by masking out one of these segments (and leaving the others as the contexts). In this manner, the gradient w.k.f. this input sequence can be calculated by averaging the gradients across multiple training samples (produced by the same input sequence). We further verify, under our theoretical framework, that the gradients obtained with such a scheme tend to have smaller variance, and thus can improve the efficiency of the pre-training process.
We evaluate the proposed masking strategies on both continued pre-training (Gururangan et al., 2020) and from-scratch pre-training scenarios. Specifically, Computer Science (CS) and News do- main corpus (Gururangan et al. 2020) are leveraged to continually pre-train RoBERTa models, which are then evaluated by fine-tuning on downstream tasks of the corresponding domain. It is demonstrated that the proposed fully-explored masking strategies lead to pre-trained models with stronger generalization ability. Even with only a subset of the pre-training corpus utilized in (Gu- rurangan et al. 2020), our model consistently outperforms reported baselines across four natural language understanding tasks considered. Besides, we also show the effectiveness of our method on the pre-training of language models from scratch. Moreover, the comparison between fully-explored and standard masking strategies in terms of their impacts on the model learning efficiency further validates the advantages of the proposed method. Extensive ablation studies are further conducted to explore the robustness of the proposed masking scheme.
2 RELATED WORK
Self-supervised Language Pre-training Self-supervised learning has been demonstrated as a powerful paradigm for natural language pre-training in recent years. Significant research efforts have been devoted to improve different aspects of the pre-training recipe, including training objec-
tive (Lewis et al., 2019; Clark et al. 2019 Bao et al. 2020 Liu et al. 2019), architecture design Yang et al. 2019 He et al. 2020), the incorporation of external knowledge (Sun et al., 2019 Zhang ltt al. 2019), etc. The idea of self supervied learning has also been extended to generation tasks and achieves great results (Song et al. 2019 Dong et al. 2019). Although impressive empirical perfor- mance has been shown, relatively little attention has been paid to the efficiency of the pre-training stage. ELECTRA (Clark et al.) 2019) introduced a discriminative objective that is defined over all input tokens. Besides, has been showed that incorporating language structures (Wang et al.) 2019) or external knowledge (Sun et al. language models to better abstract useful information from unlabeled samples.
2019 Zhang et al. 2019) into pre-training could also help the
In this work, we approach the training efficiency issue from a different perspective, and argue that the masking strategies, as an essential component within the MLM framework, plays a vital role especially in efficient pre-training. Notably, our fully-explored masking strategies can be easily combined with different model architectures for MLM training. Moreover, the proposed approach can be flexibly integrated with various tokenization choices, such subword, word span (Joshi) jet al., 2019). A concurrent work Chen et al. (2020) also shared similar motivation as this work, although they have a different solution and their method requires additional computation generate the masks, and yet is outperformed by the proposed fully- wally-explored masking (see Table
Domain-specific Continual Pre-training The models mentioned above typically abstract seman- tic information from massive, heterogeneous corpora. Consequently, these models are not tailored to any specific domains, which tends to be suboptimal there is a domain of interest beforehand. Gururangan et al. (2020) showed that continual pre-training (on top of reneral-purpose LMs) with in-domain unlabeled data could bring further gains to downstream tasks (of that particular domain). One challenge inherent in continual pre- preitrain is that in-domain data usually much more lim- corpora. As a result, how to efficiently digest information from unlabeled corpus is especially critical while adapting large pre-trained language models to specific
ited, compared to
2

---

Under review as a conference paper at ICLR 2021
Input
Gotes
Foundation
pledges
miliion
Splitting
Gates
Foundetion
pledges
100
militon
to
combett
new
coron
inua
pledges
[woas]]
X2
Gates
foundation
[MASK]
100
milion
[MASK]
new
[MAAK]
inus
43
Gates
Foundation
pledges
[MAc]]]
100
milton
[MASC]
new
coron
[MAS4]]
24 Gates
Foundation
pledges
[WASK]
miliion
combitt
[MASK]
coron
[MAASK]
Figure 1: Illustration of the proposed fully-explored masking strategy with a specific example. In this case, the input sequence has been divided into 4 exclusive segments, where different colors indicate which segment certain token belongs to.
domains. To this end, we specifically consider the continual pre-training scenario evaluate the effectiveness of our approach.
PROPOSED APPROACH
In this section, we first review the MLM framework that widely employed for natural language pre-training. Motivated by the gradient variance analysis of MLM in section 3.2 we present the fully-explored masking strategy, which serves as simple yet effective solution reduce the gra- dient variance during training. Connections between our method and variance reduction theory are further drawn, which provides theoretical foundation for the effectiveness the proposed strategy. Finally, some specific implementation details are discussed.
3.1 BACKGROUND: THE MI. FRAMEWORK
Let V denote the token vocabulary and (#1. Fx)) denote sentence of n tokens, where Iy E V for i = I,...,n. n. Let m (mi...., mp) denote a binary vector of length n, where m, {0,1), 1]. representing the mask over sentence. Specifically, m; 1 means the token æ; is masked and m, = if x; is not masked. We use m o x to denote masked sentence, that is,
[MASK]
if m, = 1,
omox);
Iz;
if m, = 0.
In addition, let m be the complement of m; in other words, ग-; 0 if m: = and m. = my O. Naturally, m denotes a sentence with the complement mask
For typical language model with parameters 0, its loss function over a sentence x E V and mask E 1)"
((0:x, - log P(m ox mx)) =-
Σ log P(z. m
(1)
m,=1
where P(x,/ 6, m o x) is the probability the model correctly predicting z; given the masked sentence m x. If m, = 0, it always has P(z; 0, m x) = as the ground-trutt is not masked.
We will focus on masks with a fixed length. Let
integer satisfying 0 S S The set of
possible masks of length 7 is defined as M(T),
M(T) m E 13,13" 1 m4 =T
+le r Therefore, the average loss function over which has cardinality [M(T) (*)
sentence with masks of length is,
L(0:x) ())
m.)
mE.((T)
Let's consider Po as the probability distribution of sentence corpus D 1", The overall loss function for training the masked language model over corpus D is
L()) A E L Ex- Umil ((0; (00xx,100 m).
(3)
3

---

Under review as a conference paper at ICLR 2021
During each step of the training process, randomly samples mini-batch of sentences S D. For each x E St, we randomly pick a subset masks Kx(x) C M(T), independently across different x.
Thus, the mini-batch stochastic gradient is
SE ge(0) = S K
Σ XES: mEX,(x)
Σ
x.l(fxx.m)
(4)
where |S,] S and [K](x)] K for all t. Clearly we have gig. (0) = VL(Ø). In the following sections, it first gives the variance of (0) which is an important factor to influence model training efficiency (Xiao & Zhang 2014 Zhang & Xiao, 2019), and then presents proposed fully- explored masking strategy to help reduce the gradient variance of the masked language model.
3.2 ANALYSIS: GRADIENT VARIANCE MLM
According to the law of total variance (Weiss 2005), the variance of the mini-batch stochastic gra- dient Vars, (9t) can be decomposed as follows,
Vars, KC, (g:) = EEs, Varr,, (9:) S.] + Vars, (Ex, [9 (s)]),
(5)
where for simplicity 9t indicates gr (0) as in eqn. the first term captures the variance due to the sampling of masks, and the second term is the variance due to the sampling of mini-batch sentences.
In this work, we focus on the analysis of the first term in eqn. 5 the variance due to the sampling of masks. Denote g(m) Vol(0; x, m) for any fixed sentence x. Consider subset of random masks (mi..... mg) and the K -maaks gradient is defined as the average of them:
g (m m
Σ g(m)).
(6)
K
Theorem 1, The Variance of K-masks gradient:
is
Var(g(m:)) + (11-((1111 1 K
1
Cov(g(m)).g(mm))
(7)
K
where,
(ov(g(m),,g(m)) = (((mm)) 9) (g(g())) -9)
(8)
and
g = (m-
Σ g(m).
mEM(T)
The detailed proof of Theorem is given in Appendix A.I the variance of K K-masks gradient can be reduced by decreasing the gradient covariance between different masks.
the theorem it indicates that
3.3 VARIANCE REDUCTION: FULLY-EXPLONE MASKING
Intuitively, we consider the two random masks m, and m2 are totally overlapped, the gradient covariance between them should be maximized. It motivates consider the correlation between gradient covariance and Hamming distance between these two masks. Thus, we have the following assumption:
The covariance Cov ming distance between my and m2.
Assumption 1.
Cov(g(m) (ov(g(m;) ( is monotone decreasing in term of the Ham-
To verify the Assumption I we sample a small set of CS domain sentences from S2ORC dataset (Gururangan et al.] 2020) as the fixed mini-batch for our analysis, then calculate gradient covari- ance Cov ( g m2)) of mask pairs (m,, m2) with different Hamming distances If (m), m²) using this mini-batch. In Figure 2 the center of gradient covariance distribution is shifting to left (lower value) as Hamming distance increases. In Figure 3 we also observe that the average gradient covariance is decreasing in term od Hamming distance. As shown in Figure holds for both RoBERTa-baa model (Liu et al. (019) and RoBERTa- base model after continually pre-trained on CS domain corpus.
Assumption
4

---

Under review as a conference paper at ICLR 2021
Homming Detance
Homming Dettance
Ararge Gradient Cavartance
Anerage Gradient Devertence
Figure 2: The distributions of gradient covariance Cov(g (m²)) for different Hamming dis- tances H (m1, m2) based on small CS domain corpus. Left: gradient covariance distribution of selected parameters RoBERTa-base model; Right: gradient covariance distribution of selected parameters in RoBERTa-bass model after continually pre-trained on CS domain corpus.
Average Gradient Covariance
Average Gradient Covariance
Jc-48
Jo-4
-1
-1
50 100
150
200 250
100 150
200
250
Hamming Distance I[(my my)
Hamming Distance [((
Figure 3: Empirical analysis of the correlation between gradient covariance Cov (g(m)), ), g (m2)) and Hamming distance H (m,, m²) based on a small CS domain corpus. For sequence of length 512, two masks m1, m2 are randomly sampled with 128 masked tokens, their Hamming distance satisfying 0 < Hm1, m2) I 256. Left: gradient covariance calculated based on RoBERTa- basee model; Right: gradient covariance calculated based on RoBERTa-ben model after continually pre- trained on CS domain corpus.
We propose the fully-explored masking strategy which restricts masks sampled from M(T) to be non-overlapping, denoted MFE(T) for simplicity:
My..... m ~ (MFE(T) (m; mj) = 2r
(10)
With the fully-explored masking strategy, it can be easily approved that expectation of gradient over MFE(T) is an unbiased estimation of the expectation of gradient over .M(T) as in the Lemma 2 In the Lemma B] states that the Theorem is still hold for fully-explored masking strategy, which indicates that the variance of K-masks gradient can be reduced by restricting the masks sampling from MFE
Lemma 2. The expectation of gradient over MFE(T) equals to the expectation of gradient over M(7).
Proof. The joint distributions of (mi...., mx) sampling from MFE(T) is different from the i.i.d. case due the non-overlapping restriction. However, the marginal distributions of m are still the same uniform distribution over M(T). Therefore, we still have E [[(m))] = G. vk=1.... and as consequence E [ggg(m])) (m.......
mx)] ğ.
O
Lemma 3. The derivation of K K-masks gradient variance in Eqm 7 holds for both MFE(T) and M(T).
The detailed proof of Lemma 3 can be seen in Appendix A.2
3.4 IMPLEMENTATION DETAILS
The details fully-explored masking algorithm is illustrated Algorithm T In practice, a text sequence S, is tokenized into subword pieces (Devlin al. 2018) with the maximum sequence
5

---

Under review as a conference paper at ICLR 2021
length set 512 in the experiments. To understand the performance of fully-explored masking strategy at different granularity, the text sequence S; is masked at both subword level (Devlin et al, 2018 Liu et al., 2019) and span level (Joshi et al. 2019; Wang et al. 2019). The details about other hyperparameters, i.e,, masking-ratio and number of splits K will he discussed in experiment section.
Algorithm 1: Fully-explored Masking Language Model
Input:
Language corpus D (Si. ..., Sr), |S, = n;; Masking ratio E; Number of sampling masks K, where K' * I 1; Initial model parameters to; n
Output: model parameters 8*
foreach S; E S do
Sample K split masking vectors (m) mx) from MFF(T) as in Eqn
Calculate the gradient (mm......, mx) as in Eqn. 6
Update model parameters 0,+1- = Optimizer(0;, (MM. mk))
end
return 0" = OT
4 EXPERIMENTS
In this section, we evaluate the proposed fully-explored masking strategy for natural language pre- training two distinct settings: i) continual pre-training, where given pre-trained model is further adapted leveraging domain-specift, unlabeled corpus; ii) pre-training from scratch, where large- scale corpus such as Wikipedia and BookCorpus are employed to pre-train a model from the be- ginning. We also compare the training efficiency of FE-MLM and MLM frameworks to validate our theoretical findings. Ablation studies and analysis are further conducted regarding the proposed approach.
4.1 EXPERIMENTAL SETTINGS
For the continual pre-training scenario, we consider unlabeled corpus from two different domains, i.e., computer science (CS) papers and news text from RealNews, introduced by Gururangan et al. (2020). As to the downstream tasks, ACL-ARC citation intent Jurgens et al. (2018) and SciERC relation classification Luan et al. (2018) are utilized for the CS domain. While for the News domain, HyperPartisan news detection Kiesel et al. (2019) and AGNews Zhang et al. (2055) are employed to facilitate the comparison with Gururangan et al.) (2020).
Following Gururangan et al. 2020) for fair comparison, RoBERTa Liu et al. 2019) is leveraged as the initial model for continual pre-training, where the same training objective is optimized on the domain-specifer corpus. We choose batch size of 48, and the model is trained using Adam Kingma & Ba (2014), with a learning rate of 1 x 10-4, It is worth noting that we observe, in initial experiments, that downsampling only 72k documents from the total of 2. 2.22M used by Gururangan et al. 2020)) can result in similar performance on downstream tasks. This happens in the News domain as well, where randomly sample 623k documents out of 11.90M. The model is continually pre- pre-trained for around 40k and 20k steps on the CS and News domain, respectively. One important hyerparameter under the FE-MLM framework is the number of split the input sequence is divided into, where we use as the default setting. The sensitivity of the proposed algorithm w.r. this hyperparameter is further investigated (see Figure
For the general pre-training experiments, we employ BERT as the bascline model. Wikiepdia and BookCorpus (Zhu al. 2015) are used as the pre-training corpus, with total size of 16G. We adopt the same tokenization (i.e., WordPiece embeddings) as BERT, which consists of 30,522 tokens in the vocabulary. The model is optimized using Adam with the learning rate set as x 14 batch size of 256 is employed, and we train the model for 1M step. The resulting model is evaluated on the GLUE benchmark (Wang et al. 22018), which comprises natural language understanding (NLU)) tasks such as textual entailment (MNLI, RTE), question-answer entailment (QNLI), question paraphrase (QQP), paraphrase (MRPC), sentimnt analysis (SST-2), linguistic acceptability (CoLA) and textual similarity (STS-B). The HuggingFace codebase is used in our implementation for both settings.

---

Under review as a conference paper at ICLR 2021
Model
ACL-ARC
SciERC
HyperPartisan
AGNews
RoBERTa (Gururangan ct al.
2020)
63.0
77.3 + 1.9
86.6 + 0.9
93.9 +.2
DAPT
(Gururangan et al. 2020)
7..4
80.8 + 1.5
88.2 5.9
93.9 +.2
MLM Subword (our implementation)
75.34
81.51
1.05
91.00 2.66
94.05 1: 0.16
FE-MLM Subword
76.24 1.86
82.40-00.86
92.35 + 3.49
94.02 0.09
MLM +
Span (our implementation)
76.63 1 1.65
81.33 1 1.16
91.72 3.26
93.94 1: 0.05
FE-MLM Span
78.06
81.99 1 0.79
93.22.1..11
94.13.10.04
Table 1: The empirical results on continual pre-training setting, where RoBERTa and DAPT (RoBERTa continually pre-trained with the standard MLM objective) is leveraged as our baseline to facilitate comparison with (Gururangan et al. 2020). Specifically, ACL-ARC and SciERC are evaluated with the continually pre-trained model with CS domain corpus, while HyperPartisann and
AGNews are based upon models trained with News domain corpus.
Model
MNL.-m/mm
SST-2
QNLI
QQP
RTF
MRPC
CoLA
STS-B
BERT (MLM)
84.37/84.85
92.78
91.01
91.09
63.54
87.01
59.65
87.89
BERT (FE-MLM)
85.09/84.63
93.23
91.01
91.16
68.59
87.99
61.32
89.51
Table 2:
The results on the dev sets of GLUE benchmarks, where MLM and FE-MLM are compared with the BERT-base model as the testbed.
4.2 EXPERIMENTAL RESULTS
Continual Pre-training Evaluation We applied our fully-explored MLM framework both sub- word and span masking scenarios. The results for the RoEBRTa model continually pre-trained on the CS and News domains are presented in Table It can be observed that the continual pre-training stage can benefit the downstream tasks on both domains with fine-tuning the RoBERTa model directly). Besides, the baseline numbers based on our implementation is on par with or even better than those reported in Gururangan et al. (2020), even though we downsample the original unlabeled corpus (as described in the previous section).
More importantly. in the subword masking casc, our FE-MLM framework consistently exhibits bet- ter empirical results on the downstream tasks. Note that to ensure fair comparison, the same com- putation is taken for both MIM and FE-LMLM training. This indicates that the models pre-trained using the FE-MLM approach have been endowed with stronger generalization ability, relative to standard MLM training. Similar trend is also observed in the span masking experiments, demon- strating that the proposed method can be naturally and flexibly integrated with different masking schemes. Besides, CS domain, whereas the opposite is true as to the News domain. This may be attributed to the different nature of the unlabeled corpus from two domains.
found that subword masking tends work better than span masking in the
General Pre-training Evaluation We also evaluate the FE-MLM framework on the pre- training experiments with general-purpose unla-
Model
Avg. Score
ELMo, (Peters ct al. 2018
71.2
beled corpus. Specifically, we follow the same setting as BERT, except that the proposed fully- explored masking strategy is applied (the same amount of computation is used for the baseline and our method). The corresponding results are shown in Table 2 It can be found that the FE- MLM approach, while fine-tuned on the GLUE benchmark, exhibits better results on 7 out of NLU datasets than the MI. baseline. This demonstrates the wide applicability of the pro- posed FE-MLM framework across different pre-training settings.
GPT, RRadordd el al.
2018)
78.8
BERT-bard baseeaae, Devlnn et al. 2018
82.2
BERT-base (ReEval
82.5
MAP-Net, (Chen (020
82.1
BERT-base (FE-MLM)
83.6
Table 3:
The comparison between the FE-MLM model with several baseline methods, based on the averaged score (on the dev set) across differ- ent tasks from the GLUE benchmark.
We further compare the averaged score over 9 GLUE datasets with other methods, and the numbers are summarized in Table 3 It is worth noting that the BERT-based (ReEval) baseline is obtained by fine-tuning the BERT model released by Devlin et al. (2018) each GLUE datasets, with the results on the dev sets averaged. Another BERT-base number is reported by Clark et al. (2019), which is pretty similar re-evaluation one. MAsk proposal network (MAP-Net) is proposed by Chen al.
7

---

Under review as a conference paper at ICLR 2021
2020), which shares the same motivation of reducing the gradient variance during the masking stage. However, we approach the problem with a distinct strategy based upon extensive theoretical analysis. We found that BERT-base model improved with the FE-MLM training significantly out- perform BERT-base model and Mask Proposal Network, further demonstrating the effectiveness of proposed approach.
4.3
ABLATION STUDIES AND ANALYSIS
2-split
82
76
4-spiit
Accuracy (%)
80
split
8-split
75
78
76
MLM
73
FE-MLM
72
72
70
10
20
30
40
ACL-ARC
SciERC
Training Steps (1e3)
Dataset
Figure 4: Left: the efficiency comparison between the standard MLM and fully-explored MLM approach. Specifically, the RoBERTa-base model and continual pre-training setting (on the CS do- main) are employed. The corresponding models are evaluated on the ACL-ARC dataset (at different training steps). Right: the effect of split number (under the FE-MLM framework) on the generaliza- tion ability of pre-trained models, evaluated on two datasets in the CS domain.
Training Efficiency Although previous results has demonstrated that our model at the end of pre-training process exhibits stronger generalization ability, it is still unclear how the proposed FE- MLM framework influence the training efficiency during training. In this regard, we examine the intermediate models obtained with both MLM and FE-MLM training by fine-tuning and evaluating them the ACL-ARC dataset. Specifically, the RoBERTa-baad setting is used here, which is continually pre-trained on the unlabeled corpus from the CS domain. As shown on the left side of Figure 4 FE-MLM beats MLM at different steps of pre-training. More importantly, the performance of the FE-MLM model improves much faster at the early stage (i.e., less than around 15,000 steps), indicating that the model is able extract useful semantic information (from unlabeled corpus) more efficiently with the proposed masking strategy. This observation further highlights the advantage and importance of reducing gradient variance under the MLM framework.
The Effect of Masking Split Number The number of masking split the input sentence is divided into is a vital hyperparameter for the FE-MLM approach. Therefore, we investigate its impact on the performance of resulting models. Concretely, the setting of continual pre-training on the CS domain is employed, where the RoBERTa model is pre-trained with the FE-MLM objective. Different split number explored. including 2, 4. 6, 8. and 12.5% of all the tokens are masked within each split. The results are visualized on the right side of Figure 4 We found that the downstream task performance (on both ACL-ARC and SciERC datasets) fairly stable w.s.f. different split numbers. This may relate position in the sentence as efficiently as possible, so that the model exhibits strong performance even with only two splits.
non-overlapping sampling strategy, which helps the model explore various
CONCLUSION
In this paper, we identified that under the MLM framework, the procedure of randomly sampling masked tokens will give rise to undesirably large variance while estimating the training gradients. Therefore, we introduced a theoretical framework to quantify the gradient variance, where the con- nection between gradient covariance and the Hamming distance between two different masked se- quences are drawn. Motivated by these observations, we proposed a fully-explored masking strategy, where a text sequence is divided into multiple non-overlapping segments. During training, all tokens in one segment are masked out, and the model is asked to predict them with the other segments as the context. It was demonstrated theoretically that the gradients obtained with such a novel masking efficient pre-training. Extensive experiments
strategy have smaller variance, thus enabling both continual pre-traine vreirraining and general pre- pre-training from scratch showed that the proposed masking strategy consistently outperforms standard random masking.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Karim Ahmed, N. Keskar, and R. Socher. Weighted transformer network for machine translation. ArXiv, abs//711.002122 2017.
Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06433 2015.
Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:/908. 10063, 2019.
Hangbo Bao, Li Dong. Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, al. Unilmv2: basked language models for unified language model pre-training. arXiv preprint arXiv:2002. 12804, 2020.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint 1903. 10676, 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint 11X1::2005.14165. 2020.
Liang Chen, Tianyuan Zhang, Di He, Guolin Ke, Liwei Wang, and Tie- Yan Liu. Variance-reduced language pretraining via mask proposal network. arXiv preprint arXiv:2008.055: 05333, 2020.
Kevin Clark, Minh- Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800, 2018.
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pp. 13063-13075, 2019.
Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. Train evil: Selec- tive masking for task-guided pre-training. arXiv preprint arXiv:2004.09753: 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004110644 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint crXiv:2006.03633 2020.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pp. 315-323, 2013.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint 1rXiv:1907.10529 2019.
David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. Measuring the evolution of scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406, 2018.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Cor- ney, Benno Stein, and Martin Potthast. Semeval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 829-839, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 11/.. 29800 2014.

---

Under review as a conference paper at ICLR 2021
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909. 1et.:1999.11922,,. 2019.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-12) 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising scquence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19XXvv:907.11000 11692, 2019.
Kyle Lo, Lucy Lu Wang. Mark Neumann, Rodney Kinney, and Daniel S Weld. S2ore: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969 4963,4983, 2020.
Yi Luan, Luheng He, M. Ostendorf, and Hannanch Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP, 2018.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint (0Kiv:8802.05.22 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenA/ Blog, 188)99, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
K. Song, X. Tan, T. Qin, Jianfeng Lu, and T. Liu. Mass: Masked sequence sequence pre-training for language generation. In ICML, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine 2014.
learning research,
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0746/ 2018.
Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. Advances in Neural Information Processing Systems, pp. 181-189, 2013.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. Structbert: Incor- porating language structures into pre- training for deep language understanding. arXiv preprint (X00::900.04555 2019.
10

---

Under review as a conference paper at ICLR 2021
Neil A. Weiss. A course in probability. pp. 385-385, 2005.
Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduc- tion, 2014.
Yi Yang, Mark Christopher Siy UY, and Allen Huang. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:200600809, 2020.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le.
Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5754-5764, 2019.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neural Information Processing 2019.
Systems, pp.
Junyu Zhang and Lin Xiao. A stochastic composite gradient method with incremental variance reduction, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. ACI., 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and
Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
A APPENDIX
A.1 PROOF OF THEOREM I
Proof.
Var(g(m,.....
mx) E [lg mx) -
E
Ek o g(m))) - o
K
E
k k= (g(me) 5)
K2
K
E
llg(ma) - gll +
Σ
(g(m=)-9) (g(mi)- (0)mmmm))))) g)
K
Σ
Var (g(m²))
+
Σ
Cor(g(ms).
(11)
K
kyl
where for each pair l 1
(m:)) = [[((m))) - ())((((m)) ( 93)
Since mi...., m i.i.d. samples from the uniform distribution over M(T). we have
Var(g(m:) )) Var(s(mx))
(12)
Cov(g(m)) (m;)) (o ((m))) 11.
(13)
Therefore have the following variance decomposition:
Var Var(g(m,)) + (1 (1-
1 K
Var(g(m mk)) K
Cov(s(m)).g(mm))
(14)
11

---

Under review as a conference paper at ICLR 2021
A.2 PROOF OF LEMMA 3
Proof. The joint distribution of the pairs (m,, m²) sampling from MFF(T) are different from the i.idd. case, it can be shown (by symmetry) that the identity equation 13 also holds. Considering the fact that the derivation in equation[[]] holld for any sampling strategy, we conclude that the variance decomposition in equation 14 still holds.
12


Reviews:
[{"review_id": "etuEUUNuMQr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To reduce the variance due to the sampling of masks, the authors propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments. And they show this technique improves accuracy in downstream tasks.\n\nThis idea is novel and interesting to me, and the derivation and experiment results look encouraging. However, I feel that experiments can be strengthened, and notations can be improved. Below are my major concerns:\n\nIf the major motivation is to reduce gradient variance, can we just use larger mini-batch size? In the experiments, it is not reported that the learning rate or the mini-batch size is well tuned for the baseline. I'd like some confirmation that larger batch size won't get much improvement for the baseline model. For example, in the continual learning, you said \"choose a batch size of 48\", that seems to be small to me?\n\nIn algorithm 1, in each iteration, only data sample (S_i) is used, how is this choice motivated? It'll be good to have some ablation study of the combined effect of using only one data sample in a mini-batch, and the full-explored masking. For example, if for the baseline model, we also only use one data sample and apply different masks, will there be improvement?\n\nIn experiments, mainly accuracy is shown, but since the major motivation to reduce gradient variance, why not show some comparison of gradient variance of MLM and MLM-FE?\n\nIn Sec2, you said \"yet is outperformed by the proposed fully-explored masking (see Table 2).\", do you mean Table 3?\n\nIn addition, I have several notation confusions:\n\nAssumption1: What is the hamming distance m_1 and m_2, when m_i are random variables? Do you mean the expected hamming distance? Or we are assuming x is fixed? Please be more clear.\n\nIn sec3.1, you used S_t for minibatch, but in sec3.4, you use S_i for \"a text sequence\", which is confusing.\n\nIn sec4.1, the authors said \"A batch size of 256 is employed, \", does that mean K=256 in Algorithm 1?", "extra_scores": {}, "extra_text": {}}, {"review_id": "b-kmDuN1cRk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work proposes a fully-explored masking strategy, which maximizes the Hamming distance between any of the two sampled masks on a fixed text sequence. The motivation is to reduce the undesirable large variance of MLM objective, based on the hypothesis that randomly sampled masks in MLM would lead to undesirably large gradient variance, which as a result typically hurts the training efficiency with stochastic gradient optimization algorithms.\n\n---------------------------------------\nStrength:\n\nThe hypothesis from the variance reduction is interesting. The method is sound. Theoretical discussion proves that the gradients derived from the new masking schema have a smaller variance and can lead to more efficient self-supervised training. Experiments on both continual pre-training and general pre-training from scratch show the effectiveness of the proposed method. Case studies show that the method can help improve training efficiency. \n \n---------------------------------------\n\nConcerns & Questions:\n\n1. Regarding the proof, the notations in Section 3 are quite loose, especially for Section 3.1. It might be better to tighten them and put the proofs in the Appendix into the main body of this paper.\n\n2. The authors claim the variance of the K-masks gradient can be reduced by decreasing the gradient co-variance between different masks, but this is just for a given sentence and the problem still exists considering different sentences. \n\n3. The proof in A.2 is not persuasive, I cannot agree that equation 14 can be concluded through previous statements.\n\n4. In terms of the experiments, there are only dev results reported on GLUE (Table 2). It is hard to infer the test gains, given the possibly significant hyperparameter optimization on the dev set. It seems that the authors have the infrastructure for computing single-model test-set results. So why not report the test results?\n\n5. Why are the backbone models (RoBERTa and BERT, respectively) different in Table 1 and Table 2? I'm concerned whether these improvements will hold after optimizing BERT carefully like RoBERTa, or using more advanced backbone methods like ALBERT. Why not using a larger model (e.g., Roberta-large)?\n\n6. There is no comparison with other public masking methods in Table 2, such as whole-word-masking, span masking, etc. Did you use dynamic masking as that was previous used in RoBERTa? How about the benefits compared with the proposed one?\n\n7. Concerning the claim, “the proposed fully-explored masking strategies lead to pre-trained models with stronger generalization ability.”, it is not clear how the proposed method yields stronger generalization ability. \n\n8. The ablations cannot serve the topic of this paper well. For the first ablation, it just gives out the performance of intermediate models on a single task. It might be better to give out the trend of training loss and validation loss. For the second ablation, why do all the larger splits lead to similar performance? Intuitively, doing more mask-then-predict procedures is better for learning language representations. The explanation at the end of Section 4 is not persuasive.\n\n---------------------------------------\nMinor issue: the citation format is not consistent, please check the usage of \\citep{} and \\citet{}.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uee9uPi_YZU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper try to improve Masked Language Model pre-training by reducing the variance using a new masking scheme. \n\nPros:\n1. It is very novel and interesting that the authors try to understand the training of MLM by analyzing the relation gradient variance and masking scheme. \n2. They propose a new masking scheme that reduces the gradient variance. \n3. This paper shows the gradient obtained by the new masking is not biased. (Lemma 2)\n4. The performance improvement under continual pretraining and BERT pretraining looks good to me. \n5. The convergence is faster than the traditional MLM scheme. \n\n\nCons/Questions:\n1. One of my concerns is that in the usual MLM training, each sentence is observed very limited times (e.g., $\\leq 3$). Under such a setting, the overlapping between different maskings won't have too many overlapping. \n2. It would be better to validate the assumption by comparing the gradient variance between MLM training with and w/o the new masking scheme. \n3. It is not clear to me how Theorem 1/Assumption 1 motivates the new masking scheme. What does it mean by \" The covariance $Cov(g(m1),g(m2))$ is monotone decreasing in terms of the Hamming distance\"? Do you mean \"$Cov(g(m1),g(m2) | H(m1,m2)=\\tau)$ is monotone decreasing in term of $\\tau$\"\n4. In Figure 4 (left), it would be better to run the experiments multiple times and plot mean and variance.\n5. If we compare the result of 6-split masking of ACL-ARC (Figure 4 right) and the baseline (DAPT) in Table 1, there is actually no performance gain. I would say selecting an appropriate splitting number is tricky. It would be helpful to present the 1-split performance (i.e., DAPT) for better understanding of the results.\n\nMinor:\n1. The text in Figure 2 can be larger. \n2. Last paragraph on page 4: in terms od ==> of ", "extra_scores": {}, "extra_text": {}}, {"review_id": "A-9B93rYcJT", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n###  Summary\nThis paper theoretically shows that the gradient variance of the standard MLM (masked language modeling) task in BERT-style training depends on the covariance of the gradient covariance between different masks within the mini-batch. This paper then empirically shows that the covariance can be reduced by making the masks less overlapped. A modified version of MLM is proposed, which has been shown with a smaller gradient variance than the standard MLM. The experimental results show that the new masking strategy does lead to some gains on several benchmarks.\n\n\n### Strong Points\n- This paper rigorously shows that the gradient of MLM can be reduced by decreasing the covariance of the gradient covariance between different masks within the mini-batch. This is a very interesting observation and it is nice to see that this can be proved mathematically.\n- This paper is well-written and very easy to follow.\n\n- The experiment is also well done.\n\n### Weak Points\n- This paper is based on several important assumptions, none of which has been proved mathematically (although there are some empirical evidence)\n   + Assumption #1: the model quality can be improved by reducing the variance of the gradient.\n   + Assumption #2: the covariance of gradients can be reduced by increasing the Hamming distance between two different masks.\n- The empirical gain of the proposed method seems to be quite small. Given that the variance of the reported result is so large, it is unclear to me whether those gains are real.\n\n### Other Comments\n- Honestly, I don’t quite see why the proposed method is related to continual pretraining, and why should it be included in the experiment section. Is it because you want to show that under the situation that “in-domain data are usually much more limited” the proposed method is still effective?", "extra_scores": {}, "extra_text": {}}, {"review_id": "28jIK8rqr9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that randomly sampled masks in masked language model can lead to large gradient variance, and hence it proposes a new masking strategy called fully-explored masking with theoretical support to reduce the variance. Experimental results show that fully-explored masking outperforms random masking in most cases.\n\nStrengths:\n1. The proposed method seems theoretically sound.\n2. The new masking strategy is simple and easy to implement.\n\nWeaknesses:\n1. The authors claim that the proposed masking strategy can reduce the gradient variance. However, there are no empirical results to *directly* support the claim.\n\n2. The proposed masking strategy is empirically compared with random masking only. However, there are existing papers which try to improve random masking, e.g. \"ELECTRA\" and \"Ernie: Enhanced representation through knowledge integration\". The authors may need to compare with these papers.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a fully-explored masking strategy, segmenting the input text, which maximizes the Hamming distance between any two sampled masks on a fixed text sequence. The hope is to reduce the large variance of MLM objective, based on the hypothesis that randomly sampled masks in MLM lead to undesirably large gradient variance, which typically hurts training efficiency with stochastic gradient optimization algorithms.

Pro

- A clear and interesting, novel leading theoretical idea. The paper has one good theme that it pursues.
- A mostly well-written paper
- Contains good theoretical discussion
- Experiments support the idea

Con

- The experiments could be better, especially they don't actually measure a reduction in gradient variance only accuracy
- The proofs are at best loose
- Alternative methods of reducing variance like using large mini batches are not considered
- The results might go away with use of stronger (larger) contextual LMs
- There isn't good comparison to other methods of masking like span masking and salient term masking
- Some of the things included seem quite haphazard (the ablations don't seem to the point, it's not really clear what this has to do with continual learning)

Overall, this paper feels to be in a premature state. The idea is interesting, but the idea and the paper needs to be developed more, with stronger results. I think it doesn't deserve to be accepted at this time.

Decision: Reject