On a Benefit of Masked Language Model Pretraining: Robustness to Simplicity Bias
Authors: Ting-Rui Chiang
Venue: Submitted to ICLR 2023 (2023)
Keywords: language model
robustness
pretraining

Abstract: Despite the success of pretrained masked language models (MLM), why MLM pretraining is useful is still a question not fully answered. In this work we theoretically and empirically show that MLM pretraining makes models robust to lexicon-level spurious features, partly answering the question. Our explanation is that MLM pretraining may alleviate problems brought by simplicity bias (Shahet al., 2020), which refers to the phenomenon that a deep model tends to rely excessively on simple features. In NLP tasks, those simple features could be token-level features whose spurious association with the label can be learned easily. We show that MLM pretraining makes learning from the context easier. Thus, pretrained models are less likely to rely excessively on a single token. We also explore the theoretical explanations of MLM’s efficacy in causal settings. Compared with Wei et al. (2021), we achieve similar results with milder assumptions. Finally, we close the gap between our theories and real-world practices by conducting experiments on real-world tasks.

Under review as a conference paper at ICLR 2023
ON A BENEFIT OF MASKED LANGUAGE MODEL PRE- TRAINING: ROBUSTNESS TO SIMPLICITY BIAS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of pretrained masked language models (MLM), why MLM pretraining useful is still question not fully answered. this work we the- oretically and empirically show that ML. pretraining makes models robust to lexicon-level spurious features, partly answering the question. Our explanation is that MLM pretraining may alleviate problems brought by simplicity bias (Shah) et al. 2020), which refers to the phenomenon that a deep model tends to rely exces- sively simple features. In NLP tasks, those simple features could be token-level features whose spurious association with the label can be learned easily. We show that MLM pretraining makes learning from the context easier. Thus, pretrained models are less likely to rely excessively on a single token. We also explore the theoretical explanations of MLM's 's efficacy in causal settings. Compared with Wei et al. (2021). we achieve similar results with milder assumptions. Finally, we close the gap between our theories and real-world practices by conducting experiments on real-world tasks.
INTRODUCTION
The question "why is masked language model (MLM) pre- training (Devlin el al. 22019: Liu el al. (219) useful?" has not been totally answered. In this work, toward the answer, we show and explain that MLM pretrain- ing makes the model robust lexicon-level features that are spuriously associated with the target label. It gives the model better generalization capability under distribution shift.
XXXX
initial step
O
XXXX
0000
Previous studies have empirically shown the robustness of MLM pretrained models. Hao et al. (2019) show that MLM pretraining leads to wider optima and better generalization capability. Henddrycks et al.] (2020) and Tu al. (2020) show that pretrained models are more robust to out-of-distributin data and spurious features. However, remains unanswered why pretrained models are more robust.
XXXX
Figure 1: The pitfall of simplicity bias Shah et al. (2020): The solid line is a simple (linear) decision boundary that utilizes only one dimension, while the dashed line is more complex deci- sion boundary that utilizes two dimen- sions and maximizes the margin.
We conjecture that models trained from scratch suffer from the pitfall of simplicity bias Shah et al. (2020) (Figure I]. Shah et al. (2020) and Kalimeris et al. (201) showed that deep networks tend to converge to a simple decision bound- ary that involves only few features. The networks may not utilize all the features and thus may not maximize the margin, which results in worse robustness. A consequence of this could be that a model may excessively rely on feature that has spurious association with the label and ignore the other features that are more robust. In the studies of Shah et al. (2020) and Kalimeris et al. (2019), they investigated networks with continuous input. Lovering et al. (202I) discovered similar results on synthetic NLP tasks, where the inputs are discrete. We will further explore this discrete setting in this work.
We start the exploration with the following assumptions: Let the sentence, label pair be X, Y. Assumption 1. We assume that from X, we can extract two features X1 and X²-

---

Under review as a conference paper at ICLR 2023
Assumption 2. X1 is spurious f that has strong association with Y. Specifically, it means that, solety relying on X2, one can predict with high accuracy over the data distribution, but cannot be 100% correctly.
Assumption 3. X2 is a robust feature based on which Y can be predicted with 100% accuracy. Namely, there exists deterministic mapping fx--Y that maps X2 to Y.
The assumptions above are realistic in some NLP tasks. In NLP tasks, the input X is a sequence of tokens. Some tasks satisfy Assumption T X can be decomposed into X1 and X2, where X1 is the presence of certain tokens, and X2 is the context of the token. Thus, X2 has much higher dimensionality than 1, As shown by the analysis of Gardner et al. (202T), there are indeed datasets where Assumption 2 is true. However, if Assumption 3 true, we would desire the model to rely on X2, which contains the semantics of the input X.
With these assumptions, in Section 2 we empirically demonstrate that spurious features in discrete inputs can cause problems as in the continuous cases (Shah et al. (2020); Kalimeris et al. (2019). We show that, possibly due to the simplicity bias, deep model is likely to excessively rely on Xi and to rely on X2 less. In Section 3.1 and Section 3.2 we provide a theoretical explanation of how MLM pretraining makes model robust spurious features. Let II; be the conditional probability P(X, X2). We show (1) the relation between the mutual information I(II;; Y) 2 T(X1; Y) and that (2) the convergence rate of learning from IT1 is of the same order learning from X1. That is, when the MI.M model can perfectly model the probability P(X, X2) and thus generate perfect IT, learning from 11, is as easy as learning from X1. As result, the model will be more likely rely on 11. Since 11, is estimated based on X2, higher reliance on 1I; also implies higher reliance on the robust feature X2. This avoids the pitfall of simplicity bias that the model relies excessively on X1-
To relax Assumption 3 we make one step further by considering causal settings in Section 3.3
The above results partly explain why MLM pretrining is useful for NLP. Denote a sequence of tokens as X = (X1, X2, .... Xr.). During the MLM pretraining process, each token is masked randomly at certain probability, and the training objective to predict the masked tokens with the maximum likelihood loss. As result, the model is capable of estimating the conditional probability PXX,|X X2) for all i = 2, ................................ L. Even though which of the tokens is spurious is unknown, as long the spurious token has non-zero probability be masked during pretraining, MLM can estimate its distribution conditioned on the context and thus can reduce the reliance on it.
Finally, close the gap between our theories and reality. One major gap is that, in reality, we do not use the conditional probability for downstream tasks. Instead, we feed the input X without masking any token and fine-tune the model along with shallow layer over its output. Regardless of that, we hypothesize that the robustness brought by MLM pretraining still exists. To prove that, in Section 4 we use the toy example and verify the effect of MLM pretraining when using the common practice for fine-tuning, In Section 5
we validate our theories with two real-world NLP tasks.
In sum, our study leads to new research directions. Firstly, we provide new explanation of MLM pretraing's 's efficacy. Unlike the previous purely theoretical studies Saunshi et al. (2021); Wei et al. (202I), our assumptions are milder and more realistic. Secondly, we study NLP robustness from the perspective of self-supervised model, which has been widely used since Word2vec Mikolov et al. (2013) and thus indispensable to the generalization to unseen data. We reveal the mechanism that leads to its robustness, which may enable us to further reinforce it in the future.
2 A TOY EXAMPLE
To show that spurious association can cause difficulty of convergence, we construct a toy example with random variables X1, X2, Y and experimental variables (d2, v). In our setting, the random variables X X1, Y satisfy the assumptions mentioned above, and the experimental variables d2 control the the strength the spurious association and the difficulty of learning from the robust features. Finally, we measure the difficulty of the task for different (d2, v) by counting the number of updates required for a model to converge.
Specially, we design the relationship between the random variables in the following way. Let the dimension of the random variables X, and X2 be 2 and d2 respectively. Their value x; E X1 (e1,e2 } and 22 E X₂ {ei,".. [dd]], where e is the one-hot vector whose ith element is 1. We
2

---

Under review as a conference paper at ICLR 2023
I layer w/o
2 layers
3 layers
dg
w/o pre
w/ pre
w/o pre
w/ pre
0.04
3680 (189.5)
691 (55.8)
614 (169.1)
302 (47.2)
249 (53.7)
0.10
2664 (121.2)
530 (30.6)
441 (134.9)
242 (27.6)
180 (37.5)
50
0.25
1420 (96.0)
352 (23.8)
300 (62.0)
179 (13.8)
148 (28.7)
0,50
306 (79.8)
141 (40.7)
118 (33,4)
106 (23.1)
89 (24.0)
0.04
5466 (170.1)
945 (57.2)
689 (225.3)
431
275 (72.1)
0.10
3789 (99.2)
677 (32.2)
478 (142.9)
317 (30.3)
208 (44.3)
100
0.25
1952 (64.9)
428 (13.1)
330 (85.0)
214 (16.2)
169 (32.5)
0.50
330 (78.0)
156 (34.0)
133 (41.2)
128 (28.2)
112 (36.1)
0.04
11127 (265.9)
1953 (112.5)
857 (442.6)
792 (69.8)
431 (88.4)
0.10
7912 (169.2)
1279 (67.5)
657 (234.9)
550 (46.7)
402 (97.0)
500
0.25
4321 (152.3)
772 (35.5)
501 (133.5)
399 (42.3)
391 (66.0)
0.50
576 (150.0)
392 (70.2)
407 (81.1)
367 (69.1)
386 (80.0)
Table I: The number of iterations model w/ or w/o pretraining requires to converge. The number is the average of 25 runs with different random seeds, and the number in parentheses the standard deviation.
make X, depends only on X2, so it is not causal feature of Y. We control the strength of the association between X1 and with < 0.5, making Xi = Y with probability - v. Given v, we design the relation between X1 and X2 in way that maximize the number of dimensions in X₂ containing no information about X,. Denote with X2 the middle 2/d2 dimensions of X2, i.c. the [d2/2 vdg th to the [d2/2 22 + vd2. th elements in X2, the random process as follows
X2 = e, i Uniform(1,ds d2)
Y -1 X2 e; for some i < d2/2
+1 otherwise
(I)
X1 = Senii Uniform(1,2) (1,2) if #0 Otherwise
((22)
where f(X2) e Xf = e; for some < d2/2, and f(X2) = é2 otherwise In this way, predicting Y solely based on the spurious feature X, achieve accuracy
We then inspect the effect using different (v, d2), the strength of spurious association between X1 and Y and the difficulty of learning from X2. Since it is a linearly separable problem, choose linear networks as our model. We draw training samples ([X1; X2), Y') pairs from the random process defined in [ We use Adam optimization with learning rate 0.001 and the cross-entropy loss. In addition to single-layer linear networks, we also try werrppramettrrzee 2-layer and 3-layer linear networks. The hidden size is [10, 32]. Since the task is linear separable and our model is also linear, we can check whether the learned weight can lead to 100% accuracy in the defined distribution. We check it every 25 iterations. We say a model has converged if is 100% accurate for consecutive checks. We report the number of the iterations required before it converges for different v and d2.
Even though a lenear-separable convex optimization problem, our results in Table I show that the strength of the spurious association can impact the number of iterations required to converge. We observe that when / < 0.5, the models tend to be trapped by the spurious feature, sticking at accuracy 1 for iterations. When the spurious relation between X; and Y is stronger, i.c. is smaller, the number of iterations required to converge is larger. In addition, the number of iterations is also larger when the d2 is larger. An intuitive explanation is that the learning signal from X₂ is more sparse when d2 is larger.
Uniform(( (a, the uniform distribution over (n)..
3

---

Under review as a conference paper at ICLR 2023
3 A THEORETICAL EXPLANATION OF THE EFFICACY OF MLM PRETRAINING
3.1 ((X)(X2) Is MORE INFORMATIVE THAN X2
The toy example above motivates us to consider the information contained in ((X](Xx) the toy example, when predicting P(Y = 0 x), if we simply output P(X; e (X2), then the accuracy of our prediction of Y will be as high as predicting Y solely based on X1, It motivates us to inspect the reliability of the estimated P(Xi|X (2)
feature for the prediction of Y compared to Xi. Let I1 be a [& -dl-mnnnnsional random variable whose value is P(X1 X2) We can prove that when (xX)(X)) estimated perfectly, II, is at least as informative as X1.
Lemma 1. When X1, X2 are discrete, 111 perfect, namely the value of 11 is exactly P(-)X2), then the mutual information I (11 = ((x1:X2)) (Proof: Appendix A.7
Compared to previous works Hjelm et al. (2019); Belghazi et al. (2018) Oord et al. (0018 Kong et al. (2020) that show some self-supervised training objectives are lower bounds of the mutual information I[x;; we directly show that the output of the MLM, II, maximizes the mutual information, since :( f(x))) I I(x;; X2) for any f. Moreover, instead of explaining the efficacy of pretraining with the infomax principle Linsker 19888 Bell & Sejnowski (1995), our theories below provide different perspective.
Theorem 1. If perfect.
I(I1; YX
Proof. Since IT is perfect, by Lemma] we have
T(X1; 1(x1;II).
By data processing inequality, Equation 3 implies I(X;; II) 0. By Assumption 3 deter-
ministic mapping fx--y from X2 to Y exists. Applying data processing inequality again, we have
X21II 2 I(2,
20,
(4)
which implies X(Y,X; [II) Accordingly,
HY|III) (( SH(IX))
(5)
Theorem 1 shows that II is a more informative feature than X1. However, model does not necessarily rely more on more informative feature if is not easy. We will discuss more in the next section.
3.2 LEARNING FROM II IS EASY
is important that learning from IT is easy. Because of simplicity bias, a neural network model is likely to rely the easy-to-learn features SSath et al. (2020); Kalimeris et al. (2019). We conjecture that a model excessively relies on the spurious feature X when learning from X1 is easier than learning from the robust feature X2. If learning from II is easy, then the model will rely on II more and thus will rely on X1 less. However, features with higher mutual information to Y are not necessarily easy to learn. For instance, although X₂ is more informative, models tend rely on X1 instead of X, at the beginning of the training process. To show that MLM can mitigate the issue brought by simplicity bias, we need show learning from is easy.
Therefore, we have the following theorem that implies learning from II is at least as easy as learning from X,:
Theorem 2. Let (DD)
lix X, y be the classifier trained with MLE loss using data puirs (n) (()) and the converged classifier be h;x,- There exists a learning
(ef),(") (x) (2)), (m)
(2)
"We will omit the subscript of IT, when there is no ambiguity.
4

---

how Under review as a conference paper at ICLR 2023
X1
X1
X1
X2
Z
X2
Y
Z
X2
X3
(c) A case where [[Ils; Y)
2
(a) Causal setting
(b) Anticausal setting
[[I:; is possible.
Figure 2: The causal settings of the (x, Y) pairs.
Ours
Wei et al. 22021
structural assumption linear independence assum. implication
X, Y follow Figure 2b
X, Y follow an HMM.
(P(X) E yT ((P(X] 1(X2;Y)
{P(Xol = h)]4 E H}
(IP(H])X)YY) ((x;Y)
Table 2: Comparison between Theorem in this work and Theorem 3.3 in Wei et al. 2021).
algorithm, which generates Ti(Dw) 11 + y using (II1, y)) ((1)),(2)) ..., (r(m), y(m)), such
that the following three properties are satisfied: (1)
E DKL. (D.)
( )
(6)
11
which is asymptotically at the same rate as E DKL. (D.)
itx,]]
]] (2) Over the distribution of
(X, YY,Y, the expected loss of the converged classifier hij not greater than the expected loss of i x,, (3) big is linear model, whose input is IT. (Proof: Appendix A.2
The remaining question is whether deep learning models used in common practices can perform at least as well the algorithm in Theorem 2 Indeed, without any knowledge of deep learning models, it is impossible to thcoretically prove that a model will necessarily rely on instead of X1- Therefore, in Section 4 and Section we will empirically validate that our theorems applicable in the real world scenarios.
3.3 EXTENDING WITH CAUSAL MODELS
We make step further by relaxing Assumption 3 By treating X, a confounder
MLM pre-training is helpful in the causal and anticausal settings as in Kaushik et al. (2021).
Theorem 3. Even if Assumption 3 is not true, Theorem still holds if X,, X2, Y follow the causal
setting in Figure 2u
Proof. By the structure of X1, X2, Y, inequality 4 hholds even the deterministic mapping fx.-- does not exist.
Theorem 4. Assume that the set of vectors {P Y y)ly E }} is linear independent, and if X1, X2, Y follow the anticausal setting Figure 2b then [[IT; Y) Y) T(X2; Y)..
Proof. The assumption is special case of the one in Lee et al. (2020), similar techniques can be used: According to the structure of X1, Y, we have
P(X)(X2) Σ P(X))))(/X)))
Therefore, if (P(,lY = v)ly E y} is linearly independent, X2)) can be recovered from II = P(X, X2).
5

---

Under review as a conference paper at ICLR 2023
Note that this theorem is very similar to Theorem 3.3 in Wei et al. (021). However, the assumptions required in ours are weaker and more realistic, and the implication is very similar (Table 2 (1) Structure assumption: Wei al. (202T) assumed that X is generated from HMM process with hidden variables Ho, H1. ... which is stronger assumption than our assumption that X1, X2 follow the anticausal setting. (2) Independence assumption: |Wei et al. (202T) assumed that the vectors in {P(X) Ho = fi] / E } need to be linearly independent. In comparison, we require only the independence in { = y)ly ly } Our assumption is more realistic because the number of hidden states [H(] must be very large if X is generated from the HMM model, and [}|] tends to be much smaller than H/. For example. in binary classification cases, our assumption holds as long as P(X)) is not independent of P(Y). (3) Implication: If we further assume that (X2; Y) = T(X; Y), then we reach a similar conclusion that P(Y[X) can be recovered from II P(Xi (22) by applying a linear function.
3.4 LIMITATIONS OF OUR THEOREMS
Our theories do not ensure that 1T is the most informative feature to learn from. Consider tokens in a and let II; be the conditional probability P(X, \ X2). A token with spurious association with the label can locate arbitrary position in the sentence, and its location unknown during pretraining. That is, the pretrained model ís able to generate II; for all i. Without loss of generality, assume X1 is the spurious token. It is possible that there exists some i such that [II;; < I[II,; Y). and that II, is predicted relying on X1. Concretely, here is an example for the causal setting with three features: X3 is independent of X1 and X2 given Y (Figure (c). Using the results in Theorem 4 there is a linear mapping that can recover P(Y X2) from II.. Therefore, is possible that T(TT;; Y) > [[II; Y) if T(X1, Y); > I(I1;; Y) depending on the distribution of the data. We leave the study of (11,; Y' for future work.
sentence X = (X1, x... xx)
Another limitation is that, in practice, NLP practitioners do not use the conditional probability predicted by the pretrained model. Instead, people stack a simple layer over the pretrained model, and fine-tune the whole model on downstream tasks. Regardless of this, conjecture that the representation encoded by an MLM pretrained model still contains the information of {II, )a and thus is robust to spurious lexicon-level feature.
4 ToY EXAMPLE WITH A PRETRAINED MODEL
As the first step to close the gap between our theories and the real world, we repeat the toy experiments with pretraining. Before fitting the model with Y, we first pretrain the first layer to predict X, based on masked X. What want show is that, after pretraining, the representation encoded by the layer will have the equivalent role of II even when the input is not masked.
Specifically, the experimental design is follows: We use the two-layer and three-layer MLP architectures same as in Section 2 When pretraining, we mask X₁ in X by using X' = [0, 0; X2] as inputs. Let the output from the first linear layer as Z = W' The loss function the cross-entropy between X1 and the softmax over [21, 22]. After pretraining, we fine-tune the pretrained model with X2], Y) pairs, and report the average number of iterations required to converge for 25 different random seeds.
We want to eliminate the possibility that the faster convergence of the pretrained model is because of larger initial weights over X,. Therefore, after pretraining, we manually create path from X, to Z. We do so by initializing the weights of the third and fourth row of W with [A: -k; 0, .0] and 1-k, k ... 0, respectively, where & is the average of the absolute value of the weights in the pretrained part, i.c. the weights of the first two rows in W. In this way, the information from X, has the same scale as the pretrained representation Z1, Z2), and thus it can compete with [Z Z2] fairly.
Table shows that pretraining can always reduce the number of iterations required to converge when 0.50. The effect is more significant when de is larger. It could be because of the higher sparsity of the learning signal from X₂ when d2 is larger.
We further inspect how the importance over the inputs changes in the process of training. The importance can be inferred from the product of the linear layers. We observe that if the model is not pretrained, the weights over X, grow faster than the weights over X₂ at the beginning (the

---

Under review as a conference paper at ICLR 2023
2.50.0.44
2-500-0.04
2.50.0.04
2-50D-0.04
15
16
Figure 3: The average weights over the features during training two-layer model. From left to right, (d2, v)) = (50, 0.04), (500, 0.04), (50, 0.04), (500, 0.04). The two figures left/rigtt side are the curves of model without and with pretraining respectively. Blue, green, purple curves represent the average weights over features in X1, X2, and X2 (the middle part of X2) respectively. The orange curve represents the accuracy.
first row Figure 3). The model cannot converge to 100% accuracy until weights on X2, the middle [V x d2] dimensions of X2, become greater than the weights on X1. addition, after the model converges, weights over X1 is still greater than weights over X2. On the other hand, if the model pretrained, weights over X1 stop growing after a few steps (the second row in Figure 3). The above observations are aligned with conjecture that the pretrained representation mitigates the robustness issue brought simplicity bias.
5 EXPERIMENTS
We experiment real world NL.P tasks verify the relation between the capability of modeling the distribution of spurious features IT and robustness. We facilitate datasets with known spurious features. We first pretrain models on the training dataset with different masking policies. One of them does not mask the spurious tokens, leading to the reduced capability of modeling 111 Afterward, we fine-tune the models using the target label and evaluate their robustness on testing set where the spurious association does not exist. Our results show that the models are less robust if spurious tokens are not masked during pretraining. This validates our theories that the capability of modeling the distribution of spurious features is important for the models' robustness.
5.1 DOWSTREAM TASKS
Hate Speech Detection Previous study has shown that hate speech detection datasets tend have lexical bias Dixon et al. (2018). That is, models rely excessively on the presence or the absence of certain words when predicting the label. Here we follow the formulation of lexical bias in hate speech detection proposed by Zhou et al. (202T). We focus on the effect of non-offensive minority identity (NOI) mentions, such as "woman". "gay", "black". Those mentions are often highly associated with hateful instances. However, it is more desirable that a model does not rely on those mentions. Therefore, we can see the presence NOI as spurious feature.
Name Entity Recognition (NER) Lin al. 2020) has shown that name entity recognition (NER) models perform worse when the name entities are not seen in the training data. In this case, we can see the content of the name entities as a spurious feature. Models may learn to memorize the name entities when fitting the training data, while may desire the model to recognize name entities according to the context.
5.1.1 DATASETS
Hate Speech Detection We use portion of the dataset proposed by Founta et al. (2018). In their original dataset, only a small number of hateful instances contain NOL Our preliminary experiments show that the model without pretraining does suffer much from the bias of NOI when training with the full data. Therefore, we create dataset, whose positive (hateful) instances are all the positive samples in the original dataset that contain NOL. As for negative instances, sample them randomly
7

---

Under review as a conference paper at ICLR 2023
NER
Hate Speech Detection
Mask Policy
Origin
Unseen F
All (12893)
NOI (602)
t
Accuracy 1
F1 1
Accuracy
FPR +
scratch
61.5 0.5
38.7 0.6
83.9 1.6
80.3 1.4
74.8 1.5
46.3 7.2
vanilla
74.2 0.4
56.5
83.1 0.8
78.5 0.8
75.8 0.5
25.1 18
unmask random
72.7 0.6
56.5 0.8
83.3 1.1
78.9 1.1
75.8 0.9
25.7 2.3
unmask spurious
72.9 0.5
53.2 o.s
84.1 0.7
79.8 0.6
73.7 1.0
32.5 2.1
remove spurious
69.8 0.5
56.7 0.8
82.4 1.0
77.8 1.0
77.3 0.6
21.7 2.0
Table The performance on downstream tasks. For the hate speech detection task, we also report false positive detection (FPR) on the NOI subset, which is set of instances containing non-offensive minority identity mentions, e.g. "women", "black". The results are the average of runs, and the smaller number is the standard deviation.
from the original training set. We control the number of negative instances so the ratio of positive and negative instances is the same as the original dataset. We create both the training and the validation splits in this way, and use the original full testing set for evaluation. We also evaluate the models NOI subset where all the instances contain NOL. Note that the testing set is from the real world, so the NOIs in the testing set do not spuriously associate with label strong in the training set.
We use the standard NER dataset Conll-2003 Tjong Kim Sang & De Meulder (2003). To create a testing set with name entities unseen in the training set, we replace the name entities in the original validation and testing splits with the entities from WNUT-17 Derczysski et al. (2017). Specifically, we replace the LOC, ORG, PER entities with the corresponding type entities in WNUT-17, while the MISC entities remain untouched.
NER
5.2 MASKING POLICIES
For each sentence with n, spurious tokens, experiment with different masking policies: (1) scratch: We do not pretrain the model before fine-tuning. (2) vanilla: During pretraining. we mask each token with 15% probability, which is same the original implementation in Devlin et al. (2019).
(3) unmask random: This is similar to vanilla MIM, but we uniformly randomly select nı, tokens from the whole sentence and unmask them if they have been masked.
(4) unmask spurious: This is similar to vanilla MLM, but we unmask all the spurious tokens. (5) remove spurious: We replace spurious tokens with a special "[unk]" token, and unmask them. Note that this setting can be seen as an oracle setting, since in most applications the spurious features are unknown.
We will inspect the effect of masking spurious tokens by comparing setting (3), (4). (5). Note that these three setting have the same expected number of masked tokens. Therefore. it rules out the possibility that their downstream performance differs because of the number of masked tokens.
5.3 IMPLEMENTATION DETAILS
For both of the tasks and all the MLM settings, including the scratch setting, we tokenize the input with the bert-bese-uncased tokenizer. We use the bert-base-uncaser architecture and also the pretrained embedding layer, which frozen through the pretraining process. We use the existing token embedding deal with the unseen tokens in the testing set, and this setting is commonly used before contexualized word embeddings were invented. We repeat each experiment times. We include more details in Appendix A.4
5.4 RESULT AND DISCUSSION
Results in Table validate our theorems. For both of the tasks, unmask random performs better than unmask spurious under distribution shift. Specifically, unmask random has higher Fl on the unseen set of the NER task. and unmask random has lower false positive rate (FPR) the NOI set. Also, unmask random performs similarly to vanilla. This implics that modeling the condition distribution of spurious tokens in the original random masking pretraining can reduce models' reliance them.
8

---

Under review as a conference paper at ICLR 2023
Note that unmask random and unmask spurious have similar in-distribution performance, so performance difference is not due to better in nn-istribution generalization suggested by Miller et al. (202I).
the
We also compare unmask random with the oracle setting remove spurious. We notice that even though remove spurious performs as well random, remove spurious hurts the performance in the seen set. indicates that modeling the conditional distribution of spurious tokens has effects beyond simply removing them from the model. On the other hand, remove spurious performs better in the hate speech detection task. A possible explanation is that NOI mentions contain little useful information for the task.
RELATED WORK
Recently, there efforts attempting to explain the effectiveness of massive language modeling pretraining. Theoretically,Saune (Sannshi et al. (2021) explore why auto-regressive language models help solve downstream tasks. However, their explanation is based on the assumption that the downstream tasks are natural tasks, i.e. tasks that can be reformulated as sentence completion tasks. Their explanation also requires the pretrained language model perform well for any sentence completion tasks, which is not likely to be true in the real world. WWi et al. (2021) analyze the effect of fine-tuning pretrained MLM model. Nonetheless, they have stronger assumptions described in Section A.4 Aghajanyan et al. (2220) show that pretrained models have lower intrinsic dimension, providing generalization bound based on Arora et al.] (2018). However, why pretrained models have lower intrinsic dimension is unknown. MMrrrll et al. (2021) show that the parameter norm growth during training makes transformer a saturated model, which can be described in terms of formal languages. Empirically, Zhang & Hashimoto (2021) show that the effectiveness of MLM pretraining cannot be explained by formulating the downstream tasks as sentence completion problems. (Sinha et al. (2021) find evidence supporting the hypothesis that masked language models benefit from modeling high-order word co-occurrence instead of word order. There are also some theories explaining the efficacy of non- non-MLM pretraining [ee et al. (2020); [Saunshi et al. (2019); Zhang & Stratos (2021).
Many of the previous studies on robust NLP focus on supervised learning Wang et al. (2021) Utama al. 2020b ab; Karimi Mahabadi et al. (2020); Chang al. (200)): He al. (2019), Sagawa* et al. (2020) Kennedy et al. (2020); Chiang et al. (202)). However, without self-supervised learning, model can impossibly extrapolate
-isttr-diistiibution data when the domain shifts. Our work also complements previous studies that focus on the bias or robustness of model generated by the pretraining process] Kumar et al.] (2020p; Hawkins et al. (2020); Vargas & Cotterell (2tterel](2020). Liu et al. (2020); Gonen & Goldberg (2019); Kurita et al. (2019); Zhao et al. (2019). In this work we investigate the pretraining process itself.
7 IMPLICATION AND CONCLUSION
Our results provide possible explanations for some common practices found effective empirically. First, it could explain why continuing pretraining on target dataset is useful Gururangan et al. (2020). may be because continuing pretrained models model the distribution of spurious features in the target dataset better. Thus the model can better avoid the simplicity pitfall. Second, it provides reasons for more complex masking policies, such as masking continuous random spans Clark et al. : et al. (2020). It may improve the robustness to spurious features that contain more than one token. Third, if MLM can alleviate the simplicity bias and help the model achieve a greater margin, it may also imply that the model has wider optima, explaining the finding in Hao et al. (2019). On the other hand, a pretrained model's capability of utilizing more features may also suggest that pretraining makes the decision boundary simpler.
In sum, we show a benefit of MLM pretraining, which partly explains its efficacy. We first empirically demonstrate the presence of simplicity bias when the input is discrete. We then theoretically and empirically explain how MLM pretraining can alleviate the problem brought by it. Finally, we close the gap between our theories and real- world practices with experiments real-world NLP tasks. Our theories reveal a desirable mechanism of MLM pretraining, suggesting that reinforcing this mechanism could be promising future research direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiw:2012213255, 2020.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pp. 253-263. PMLR, 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531-540. PMLR, 10-15 Jul 2018. URL https: http::://ppocceeiingg www.iinns........m.
mmr www..ppees///////eeeeeeee http////es/////eeeggcaall..l..11 htm_
Anthony Bell and Terrence Sejnowski. An information-maxiiizzaii approach to blind separation and blind deconvolution. Neural computation, 7(6):1129-1155) 1995.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pp. 1448-1458. PMLR, 2020.
Ting-Rut Bui Chiang, Hao-Tong Ye, and Yun-Nung Chen. An empirical study of content understanding in conversational question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7578-7585, 2020.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003. 10555, 2020.
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140-147, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18633/1/W1174444 URL htttttss///aaatttol.l
org/W17-44 org/817-4418
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-111 11 URL https
Pry.orrgNN 9- 1423
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67-73, 2018.
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media, 2018
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah Smith. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1801-1813, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL htt.a//.pp.///c..atooooooo.ooo/p 2ttp:////22221...mmmmmmmmm.m.
135
Alison L. Gibbs and Francis Edward Su. On choosing and bounding probabiiity metrics. International Statistical Review, 70(3): 2002. doi: http::///ooourr///0...1//1///211
www.:eewww...aa.............ooo x. ott.://tteee/ooonnooaaaaaaaaaaaa
20 2002. 1w..110011..
10

---

Under review as a conference paper at ICLR 2023
Hila Gonen and Yoav Goldberg. Lipstick a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (ong and Short Papers), pp. 609-614, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18665//1/119-1661 URL https
//aclanthology org/N19-106)
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, July 2020, Association for Computational Linguistics. doi: 10. 1/220 aci-main. 740. URL htRphtts::// httptps://aaaantahllooogoooo//// otg/ acl-mmin. 740
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-ICCNLP), pp. 4143 4152, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 18.18/ 1424.
Robert Hawkins, Takateru Yamakoshi, Thomas Griffiths, and Adele Goldberg. Investigating rep- resentations of verb bias in neural language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4653-4663, Online, November URL
2020. Association for Computational Linguistics. doi: 10. 18653/v 10/33/1122200 bettps://aclanthie
org/ 2020. maam---maminn 376
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 132 142, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18663/11/D11--1155
6115
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve dist---iiitiiittionn robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744-2751, Online, July 2020. https
Association for Computational Linguistics. doi: 10. at1.5///11/2222.ac-mmmaaa
7/acl ho 1 ogy- 20200 ac 1 ma in. 244
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL https
//openreiiew net www.:/nnntfootteoe.e forum? fitiuummid-iikiii 3
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trunsactions of the Association for Computational Linguistics, 8(0))64-77, 2020. ISSN 2307-387X. URL https::///// pestpps:/ttrasaaa. /taasac
axq/oja/indezt http://////////aooo/////////a/aa
1853
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32:3496-3506, 2019.
Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Linguistics. doi: 10. 1/66533///222..clllll main.769, URL httph:tt////.. pttps::/acc www.acccnnnnooyo.ooooooooooooooe org/ 2020.ac1-ma 769.
Online, July 2020. Association for Computational
Divyansh Kaushik, Amrith Setlur, Eduard H Hovy, and Zachary Chase Lipton. Explaining the efficacy of counterfactually augmented data. In International Conference Learning Representations, 2021. httl:tttps://opennee
n fett/foreee forum?i ii iQXWsOcV
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th
11

---

Under review as a conference paper at ICLR 2023
Annual Meeting of the Association for Computational Linguistics, pp. 14435-5443 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020..t manee-maiin8.3 URL
https::/acclanthooooyy wttp://lggyyygg//2oooooo
Lingpeng Kong, Cyprien de Masson d' Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama.
A mutual information maximization perspective of language representation learning. In Interna- tional Conference on Learning Representations, 2020. URL
Corumiid-Syx79ett
Vaibhav Kumar, Tenzin Singhay Bhotia, Vaibhav Kumar, and Tanmoy Chakraborty. Nurse is closer to woman than surgeon? mitigating gender-biased proximities in word embeddings. Transactions of the Association for Computational Linguistics, 8:486-500 2020. doi: 10.1162/tac_aa0003
URL
orgp 20c.. 1.32
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pp. 166-172, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/11/W11-33222 URL https: //acl anthology org/W1 9-13-99923
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008. 01064, 2020.
Hongyu Lin, Yaojie Lu, Jialong Tang, Xianpei Han, Le Sun, Zhicheng Wei, and Nicholas Jing Yuan. A rigorous study on named entity recognition: Can fine-tuning pretrained model lead the promised land? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 7291-7300, Online, November 2020. Association for Computational Linguistics. doi:
URL https: http://aaaaattttoooy.e.o org/
2020. wwwweennnnllmmee 592
R. Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988. doi: 10.1 109/136..
Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. Mitigating gender bias for neural dialogue generation with adversarial learning. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 893-903, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.186533w1/2020.coo www.:0000..mmmmmmi.i..m..m main.64. URL https::/aclantholoog wwt.oorg/22000 emnlp-main. 64.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting inductive biases of pre- trained models. In International Conference on Learning Representations, 2021. URLhttps
//openreviewt www.eeiiwww..... iew, net. forum? www.mmiiiieNttttteeea..aa
William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1766-1781, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL httpaptp///ccaaacc...oyy wwtp:/gg..rgg//20// 202g emnlp-main
133
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: the strong correlation between out-of-distribution and in- in-distribution generalization. International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint wwXiv:1807.033 2018.
Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15(6):1191-1253 June 2003, ISSN 0869. doi: 10.11///07776603277777777 URL httpttss////ooorrg 10. 11 62/ 025/0899766633222 780272
Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks, In International Conference on Learning Representations, 2020. URL
https://gpenreviees net www.forrrmeeeoee /too
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5628-5637. PMLR, 09-15 Jun 2019. JRL.ttpss:: http:::://proceeeie ngs. mm r.w.sr.pprssseeee...ee.ee http//////////aa
9a. ht.m
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. International Conference on Learning Representations, 2021. URL
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurIPS, 2020.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arXiv preprint arXiv:2 104. 06644, 2021.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independen named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https://
acl aclanthology 0rrg/0/00/009999
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness spurious corrllaiion using pre- pre-trained language models. Transactions of the Association for Computational Linguistics, 16621-63,, 2020. doi: 10.1 162/tacl a 00335. URL hhttps://acll http:///clllathhh//m l ogy
org/2020. tac -1.40
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8717-8729, Online, July 2020a. Association for Computational Linguistics. doi: 10. 112863//1120222.cl1mmm 770. URL
wwwp:thhhwwloo...g/e222o22e.. 2020:/2220cco aww.am...1mmm....7.... 770
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 7597-7610, Online, November 2020b.
Association for Computational Linguistics. doi: 10. 100.18633/vll h/20200.mmmmpmmain...a main. 613
URL
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mitigation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9902-2913. Online, November 2020. Association for Computational Linguistics. doi: 10.186533/11/200 http:///ee.00mmmminnn.... 2020. wmnlp-manmmm.... maapppmminn. 232
URL hhttps://aclaathollgg..o org
Liwen Wang, Yuanmeng Yan, Keqing He, Yanan Wu, and Weiran Xu. Dynamically disentangling social bias from task-oriented representations with adversarial attack. In Proceedings of the 2021
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1740-3750, Online, June 2021. Association for Computational Linguistics. doi: 10. 18.538 1/2021. Lwwccl main.293. URL https ac htt////aaaattotyy..o.oo
2021 1. naac -maimm.....mmmmmmmmmmmm 293
13

---

Under review as a conference paper at ICLR 2023
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis head and prompt tuning. arXiv preprint wrXiv:2106.09226. 2021.
Tianyi Zhang and Tatsunori B. Hashimoto. On the inductive bias of masked language modeling: From statistical syntactic dependencies. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5131-5146, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 1021.naacl-mae moll-mmai.444.. URL UtL.attps://accaattaoooo org/2//.gg/ nwacl-aainnnnnnaa 404
Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1090-1101, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021 naacl- main.86. URL https
7/acla Praclantho oog.. hrg.oorgg/222 2021 mmaicl-main.. 86
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 629-634, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1..1865//1//11111000 URL https TTaclannholooy..rrg////ooo
Xuhui Zhou, Maarten Sap. Swabha Swayamdipta, Yejin Choi, and Noah Smith. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguisties: Main Vol- ume, pp. 3143-3155, Online, April 2021. Association for Computational Linguistics. URL main.274
http:://eecanttoooo.ooo///2222.2
14


Reviews:
[{"review_id": "q__uwE47Mp", "reviewer": "Reviewer_vBVm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to understand “what makes masked language models (MLMs) useful”. The paper claims that it is because MLM pretraining is specifically effective in reducing the reliance on spurious features, and provides the theoretical view of it. Specifically, it first (1) assumes two random variables from the input, X1 and X2, where X1 is a spurious correlation and X2 is not, and the output Y, (2) shows that the mutual information between P(X1|X2) and Y is larger than the mutual information between X1 and Y, (3) shows that learning from P(X1|X2) is as easy as learning from X1, and (4) claims that this indicates MLM pretraining is effective because it essentially learns from P(X1|X2), since there are instances where non-spurious tokens are masked out, so that the model should predict non-spurious tokens (X2) given the spurious tokens (X1). The paper also empirically shows that, MLM training where spurious tokens are never masked out always give worse results than masking at uniform with the same masking budget, masking a connection between its theory and experiments with real NLP datasets.\n", "summary_of_the_review": "Overall, due to the reasons I wrote above as weaknesses, I think the paper needs more work on making a tighter connection between theoretical concepts and the actual MLM training, as well as the empirical experiments with real NLP datasets.", "strength_and_weaknesses": "### Strengths\nThe paper addresses an important research question on what makes MLMs useful, which I think is a critical research question, given that MLMs have been widely used in a range of NLP problems.\n\n### Weakness\n\n#### Theory part:\n* The summary of the theory part of the paper is “learning from P(X1|X2) is as easy as learning from X1, and is more informative than learning from X2.” The paper then derives the conclusion that MLM training is effective, because it learns from P(X1|X2). I am not convinced with the logical flow here – I don’t think we can say MLM is learning from P(X1|X2) only because in some cases tokens that are masked out belong to X1 and tokens that remain belong to X2.\n* In fact, I am not convinced if we can make a distinction between X1 and X2 during MLM training. In MLM pre-training, there is no Y (while one can consider Y as tokens that are replaced with <mask>, this paper is treating all tokens in the input as either X1 and X2). So the distinction between X1 and X2 cannot be made (because their definition depends on Y).\n* Moreover, it is not clear to me that in this analogy, what the baseline to MLM is, e.g., MLM is more effective than which method? Based on my best attempt, it looks like the paper is comparing learning from P(X1|X2) vs. learning directly from X1 and X2. But in reality, MLMs are trained with MLM objective and then fine-tuned on the downstream data that directly learns from X1 and X2. So, I am not sure if this analogy makes sense.\n#### Empirical part:\n* First of all, because the paper provides almost no detail in training (e.g. pretraining data, the number of pretraining steps, details on fine-tuning), it is very difficult to evaluate the validity of the experiments. However, based on my best attempt – it looks like the MLM pretraining is done on the downstream dataset (e.g. hate speech detection task and NER data) rather than unlabeled text corpus, and it is fundamentally different from how pre-training is typically done. In fact, in typical pretraining, there cannot be a notation of spurious tokens / non-spurious tokens since Y is not pre-defined (related to the point I made earlier.\n* The conclusion that can be made from the experiments is: “masking tokens randomly at random is better than masking tokens that are not the spurious tokens”. It is not clear to me how this is related to the theory provided earlier in the paper or the overall research question on what makes MLM better. In fact, this result is pretty much expected. The tokens that the paper defines as spurious features (and thus never masked out) are those that are critical in downstream tasks, even if it’s not possible to achieve 100% only with these features. For instance, in NER, the paper defines “entity names” as spurious features, and never masks out entity names. Thus, it is very natural that this pretraining will lead to performance drop in NER, and these results itself do not really verify that predicting spurious tokens from non-spurious tokens is the key to the effectiveness of MLM.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BiFaX8I9A6", "reviewer": "Reviewer_RC11", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem that we don't yet fully understood why masked language model (MLM) pretraining is effective for downstream NLP tasks. What the authors present as (part of) the answer is that MLM pretraining makes models robust to lexicon-level spurious features. They show both theoretical and empirical studies to back up this claim.\n\nUnfortunately, I don't think the paper as-is is in an acceptable state. Specifically, I don't think one can learn anything practically relevant from the paper, as I will explain more under \"Weaknesses\" below.", "summary_of_the_review": "While this paper is about an interesting topic, I don't believe the quality is sufficient for it to be published at a top-tier conference. In fact, in its current state it shouldn't be published at all.", "strength_and_weaknesses": "Strengths:\n- A better understanding of MLM pretraining is desirable overall, so the paper's goal is a relevant one.\n- Table 1 is partially interesting.\n\nRegarding weaknesses, it seems to me that the paper doesn't make much sense overall, mainly because the overall setting seems too simplified (i.e., neural models don't necessarily rely on clearly separable features). In addition, as the authors themselves note, \"in practice, NLP practitioners do not use the conditional probability predicted by the pretrained model. Instead, people stack a simple layer over the pretrained model, and fine-tune the whole model on downstream tasks.\" The authors claim that this isn't a problem for their thinking in practice, but I'm not convinced this is true.\n\nIn addition, I see the following weaknesses:\n- The paper is based on the assumption that it's totally unclear why MLM pretraining works (\"In this work, as an initial step toward the answer\"). This isn't true. Specifically, besides what's mentioned in the related work, I was missing a discussion of the distributional hypothesis. \n- The paper is centered around the idea of \"spurious features\". However, I believe the authors are greatly misusing the term: \"spurious\" implies that the features are *not*, in fact, good indicators of the final label. This is true for the empirical experiments that are being conducted, but, in the theoretical part, the authors basically construct an imperfect and easy (as opposed to spurious) feature. \n- I don't understand why Table 1 has no \"with pretraining\" column for 1 layer.\n- \"NER\" = \"name**d**\" entity recognition\"\n- The authors should not use anything pretrained in their from-scratch experiments. I don't understand why they chose to do this.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6zHF7wZNqA", "reviewer": "Reviewer_QRgo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the mechanism by which MLM pre-training may result if fine-tuned models that are more robust to spurious features.\n\nThe paper proposes a setting where there is a spurious feature that can predict a classification decision with a simple decision boundary and high but non-perfect accuracy, and a robust feature that that can predict the decision perfectly but with a more complex decision boundary. In this setting, the training dynamics of a model are such that it learns to use the spurious feature after a small number of training iterations, but requires many more iterations to make use of the robust feature. Given access to a predicted conditional distribution p(spurious feature | robust feature), however, the effect of the spurious feature is mitigated.", "summary_of_the_review": "Overall the paper presents a clearly-described theory for how pre-training to learn conditional distributions between features can help prevent over-reliance on spurious features. This proposal is backed up by theoretical claims, experiments on a toy task, as well as by testing one of the theory's predictions in a real NLP setting with pre-training and fine-tuning. However, beyond proving that this theory *can be* an explanation for why pre-training helps, the body of evidence is not large enough to show that this *is in fact* the dominant factor underlying model robustness.", "strength_and_weaknesses": "Strengths:\n- The paper is effective in presenting the high-level idea and intuition behind why having access to conditional probabilities between spurious and robust features can help converge to more robust solutions.\n- Experiments in a toy setting provide some support for this claim.\n- This explanation gives rise to the falsifiable prediction that pre-training without masking tokens involved in spurious correlations will lead to less robust performance. The paper then conducts experiments targeting this prediction\n\nWeaknesses:\n- For the masking experiments in Table 3, it appears that the \"unmask spurious\" condition for NER involves pre-training where all entity names are defined to be \"spurious\", and never masked. This seems highly suspect, given how central entity names are to the NER task. That said, other tasks need not exhibit this issue to the same degree. For example in NLI datasets, the presense of the word \"not\" can be correlated with a \"contradiction\" label, but simply failing to mask the word \"not\" doesn't undermine the essense of NLI to the same degree that skipping entity names might undermine NER. Hate speech detection might be a similarly good setting, except that in the \"all\" condition training from scratch achieves even higher accuracy than nearly all of the pre-training approaches, making it not a suitable testbed. The overall conception of these experiments makes sense to me, but I find the specific task/dataset selection to be inadequate for demonstrating the intended conclusion. \n- How do the claims in the paper relate to the known effectiveness of unidirectional LMs? Methods like ELMo and GPT might have slightly lower accuracy than bi-directional models at the same model size, but they seem to capture all of the essential characteristics that make pre-training effective. A robustness explanation that applies to MLM but not to unidirectional LMs would seem incomplete and potentially even missing the mark on what actually matters, and the paper would be stronger if it gave consideration to this issue.\n- Some of the claims in the toy setting are perhaps not a strong as the text might suggest. Looking at Figure 3-right, I see many curves where the blue line rises rapidly and does not necessarily exceed the green line after just a few steps. Maybe it's just hard to fully understand the situation with how multiple runs are superimposed on the same plot, but it seems that despite the greater variance given pre-training the relations between the different curves do not necessarily change. For 2-50-0.04, accuracy appears to hit the ceiling at ~600 steps regardless of whether pre-training is used. Results in Table 1 also show that pre-training can sizably increase variance, and increased variance alone could explain any other effects. (cf. how dropout and other methods of adding noise can affect training dynamics)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject