Learning from others' mistakes: Avoiding dataset biases without modeling them
Authors: Victor Sanh
Thomas Wolf
Yonatan Belinkov
Alexander M Rush
Venue: ICLR 2021 Poster (2021)
Keywords: dataset bias
product of experts
natural language processing

Abstract: State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended underlying task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We consider cases where the bias issues may not be explicitly identified, and show a method for training models that learn to ignore these problematic correlations. Our approach relies on the observation that models with limited capacity primarily learn to exploit biases in the dataset. We can leverage the errors of such limited capacity models to train a more robust model in a product of experts, thus bypassing the need to hand-craft a biased model. We show the effectiveness of this method to retain improvements in out-of-distribution settings even if no particular bias is targeted by the biased model.

Published as a conference paper at ICLR 2021
LEARNING FROM OTHERS' MISTAKES: AVOIDING DATASET BIASES WITHOUT MODELING THEM
Victor Sanh', Thomas Wolf', Yonatan Belinkov Alexander M. Rush'
Huggiing Face, 'Technion - Israel Institute of Technology (victor, thomas, sasha h @hugg i face. co bel bel..roovtteehhnio...cc i1
ABSTRACT
State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the in- tended underlying task. Previous work has demonstrated effective methods circumvent these issues when knowledge of the bias is available. We consider cases where the bias issues may not be explicitly identified, and show method for training models that learn to ignore these problematic correlations. Our ap- proach relies on the observation that models with limited capacity primarily learn to exploit biases in the dataset. We can leverage the errors such limited capac- ity models to train a more robust model in a product of experts, thus bypassing the need to hand-craft a biased model. We show the effectiveness of this method to retain improvements in -istr--diittibution settings even no particular bias is targeted by the biased model.
INTRODUCTION
The natural language processing community has made tremendous progress in using pre-trained language models to improve predictive accuracy (Devlin et al., 2019 Raffel et al., 2019). Models have now surpassed human performance on language understanding benchmarks such Super- GLUE (Wang et al. 2019). However, studies have shown that these results are partially driven by these models detecting superficial cues that correlate well with labels but which may not be useful for the intended underlying task (Jia & Liang, 2017, Schwartz et al.) 2017). This brittleness leads overestimating model performance the artificially constructed tasks and poor performance in out-of-distribution or adversarial examples.
A well-studied example of this phenomenon is the natural language inference dataset MNLI (Williams et al., 2018). The generation of this dataset led to spurious surface patterns that corre- late noticeably with the labels. Poliak et al. (2018) highlight that negation words ("not", "no"., etc.) are often associated with the contradiction label. Gururangan et al. (2018). Poliak et al. (2018), and Tsuchiya (2018) show that model trained solely on the hypothesis, completely ignoring the in- tended signal, reaches strong performance. We refer to these surface patterns as dataset biases since the conditional distribution of the labels given such biased features is likely to change in examples outside the training data distribution (as formalized by He et al. (2019)).
major challenge representation learning for NLP is to produce models that are robust to these dataset biases. Previous work (He et al. 2019 Clark et al.) 2019; Mahabadi et al.) (222)) has tar- geted removing dataset biases by explicitly factoring them out of models. These studies explicitly construct a biased model, for instance, hypothesis-only model for NLI experiments, and use to improve the robustness of the main model. The core idea is to encourage the main model to find a different explanation where the biased model is wrong. During training, products--f-eexperss ensembling (Hinton) 2002) is used to factor out the biased model.
While these works show promising results, the assumption of knowledge of the underlying dataset bias is quite restrictive. Finding dataset biases in established datasets is a costly and time-consuning process, and may require access private details about the annotation procedure, while actively re-
"Supported by the Viterbi Fellowship in the Center for Computer Engineering the Technion

---

Published as a conference paper at ICLR 2021
ducing surface correlations in the collection process of new datasets is challenging given the number of potential biases (Zellers et al. 2019; Sakaguchi et al. 2020).
In this work, we explore methods for learning from biased datasets which do not require such an explicit formulation of the dataset biases. We first show how a model with limited capacity, which call weak learner, trained with standard cross-entropy loss learns exploit biases in the dataset. We then investigate the biases on which this weak learner relies and show that they match several previously manually identified biases. Based on this observation, we leverage such limited capacity models in a product of experts ensemble to train a more robust model and evaluate our approach in various settings ranging from toy datasets up to large crowd-sourced benchmarks: con- trolled synthetic bias setup (He et al.) (201:; Clark et al. 2019). natural language inference (McCoy et al. 2019b), extractive question answering (Jia Liang. 2017) and fact verification [Schuster et al. (2019)
Our contributions are the following: (a) we show that weak learners are prone to relying on shallow heuristics and highlight how they rediscover previously human-identified dataset biases; (b) we demonstrate that we do not need to explicitly know or model dataset biases to train more robust models that generalize better to out-of-distribttion examples; (c) we discuss the design choices for weak learners and show trade-offs between higher out of-distribution performance at the expense of the in-distribution performance.
2
RELATED WORK
Many studies have reported dataset biases in various settings. Examples include visual question an- swering (Jabri et al. 2016 ZZhang ZZhang et al., 2016), story completion (Schwartz et al. 2017), and reading comprehension (aushii Lipton 2018 Chen et al. 2016). Towards better evaluation methods, re- searchers have proposed to collect "challenge" datasets that account for surface correlations a model might adopt (Jia & Liang 2017: McCoy et al., 2019b). Standard models without specific robust training methods often drop in performance when evaluated on these challenge sets.
2he8|Chen
While these works have focused on data collection, another approach to develop methods al- lowing models to ignore dataset biases during training. Several active areas of research tackle this challenge by adversarial training (Belinkov et al. [019a|b] [Stacyy et al. 2020), example forgetting
Yaghoobzadeh al., 2019) and dynamic loss ad adjustment (Cadène et al. 2019 Previous work (He et al. [2119] Clark et al. [2019] Mahabadii et al.) (2200) has shown the effectiveness of product of ex- perts train un-biased models. In our work, we show that do not need to explicitly model biases to apply these de-biasing methods and can use more general setup than previously presented.
Orthogonal to these evaluation and optimization efforts, data augmentation has attracted interest as a way to reduce model biases by explicitly modifying the dataset distribution (Min et al. [0200] Bc- linkov & Bisk 2018), either by leveraging human knowledge about dataset biases such as swapping male and female entities (Zhao et al. 2018) or by developing dynamic data collection and bench- marking (Nie et al., 2020). Our work mostly orthogonal these efforts and alleviates the need for co-man-in-the--ee
setup which is common to such aata--ugmmetttioon approaches.
Large pre-trained language models have contributed to improved -fiiiii-ittttuttion generalization (Hendrycks et al. 2020). However, in practice, that remains a challenge in natural language pro- cessing ([inzen] 2020 Yogatama et al. 2019) and our work aims at ouiit--distiibttio robustness without significantly compromising in-distribution performance.
Finally, in parallel the work Utama et al. (020)) presents a related de-biasing method leveraging the mistakes of weakened models without the need to explicitly model dataset biases. Our approach is different in several ways, in particular we advocate for using limited capacity weak learner while Utama et al. (2020) uses the same architecture as the robust model trained on few thousands examples. We investigated the trade-off between learner's capacity and resulting performances as well as the resulting few-shot learning regime in the limit of high capacity weak model.
2

---

Published as a conference paper at ICLR 2021
3 METHOD
3.1
OVERVIEW
Our approach utilizes product of experts (Hinton, 2002) to factor dataset biases out of a learned model. We have access to training set (Ii, yi Vi)iciSN where each example т; has label among K classes, We use two models fw (weak) and fM (main) which produce respective logits vectors w and m E R2 The product of experts ensemble of fw and fas produces logits vector e
A1 j S K, ed = w + +m²
(I)
Equivalently, we have softmax(e) x softmax(w) w) c softmax(m) where C is the element-wise multiplication.
Our training approach can be decomposed in two successive stages: (a) training the weak learner fw with a standard cross-entropy loss (CE) and (b) training a main (robust) model far via product of experts (PoE) to learn from the errors of the weak learner. The core intuition of this method is to encourage the robust model to learn to make predictions that take into account the weak learner's mistakes.
We do not make any assumption on the biases present (or not) in the dataset and rely on letting the weak learner discover them during training. Moreover, in contrast to prior work (Mahabadi ett al. [2020] He et al., 2019 Clark et al. 2019) in which the weak learner had a hand-engineered bias-specifie structure, approach does not make any specific assumption the weak learner such as its architecture, capacity, pre-training, etc. The weak learner fw is trained with standard cross-entropy.
The final goal is producing main model fui After training, the weak model fw is frozen and used only part of the product of experts. Since the weak model is frozen, only the main model fir receives gradient updates during training. This is similar to He et al. (211)) Clark et al. (2019) but differs from Mahabadi al. (2020) who train both weak and main models jointly. For convenience, we refer to the cross-entropy of the prediction e of Equation as the PoE cross-entropy.
3.2 ANALYSIS: THE ROBUST MODEL LEARNS FROM THE ERRORS OF THE WEAK LEARNER
To better explore the impact of PoE training with a weak learner, we consider the special case of binary classification with logistic regression. Here w and m are scalar logits and the softmax becomes sigmoid. The loss of the product of experts for a single positive example is:
Eooaa = - w + log (1 + exp(m + w:)
Logit w is a fixed value since the weak learner is frozen. We also define the entropy of the weak learner as Hi. -ploog(p)) (1 p) log(1 p) where p= aww) as our measure of certainty.
Different values of w from the weak learner induce different gradient updates in the main model. Figure la shows the gradient update of the main model logit m. Each of the three curves corresponds to different value of w the weak model.
Weak Model is Certain Incorrect: the first case (in blue) corresponds to low values of w. The entropy low and the loss of the weak model high. The main model receives gradients when it is classifying the point correctly (== = 5) which encourages to compensate for the weak model's mistake.
Weak Model is Uncertain: the second case (in red) corresponds to which means the weak model's 's entropy is high (uniform probability over all classes). In this case, product of experts is equal the main model, and the gradient is equal the one obtained with cross-entropy.
Weak Model is Certain / Correct: the third case (in green) corresponds to high values of The entropy is low and the loss of the weak model is low. In this case, m'ss gradients are "cropped" early on and the main model receives less gradients on average. When w is extremely high, receives gradient (and the current example is simply ignored).
Put another way, the logit values for which m receives gradients are shifted according the correctness and certainty of the weak model. Figure 1b shows the concentration of training examples of MNLI
3

---

Published as a conference paper at ICLR 2021
Williams et al. 2018) projected the 2D coordinates (correctness, certainty) from trained weak learner (described in Section 4.1). We observe that there are many examples for the 3 cases. More erucially, we verify that the group certain / incorrect is not empty since the examples in this group encourage the model to not rely on the dataset biases.
Uncertain
2.75
230
0.6
2.2
Certain Incorrect
0.4
w-.5.0 H-D.04
Uncertaln w-0.0 H-0.69
175
Certain Correct
Certain Incorrect
Cortain Correct
1.0
W=4.0 H=0.09
125
910
-5
0.5
3.0
1.5
2.0
3.0
4.0
Loss
(a) Gradient update of m for different values of w hinary classification.
(b) 2D projection of MNLI examples from trained weak learner. Colors indicate the concentration and are log scale.
Figure 1: The analysis of the gradients reveals 3 regimes where the gradient is shifted by the certainty and correctness the weak learner. These 3 regions are present in real dataset such as MNLI.
3.3 CONNECTION DISTILLATION
Our product of experts setup bears similarities with knowledge distillation (Hinton al., 2015) where a student network is trained to mimic the behavior of a frozen teacher model.
In our PoE training, encourage the main model fas (analog the student network) to learn from the errors of the weak model fw (analog to the teacher network): instead of mimicking, it learns an orthogonal behavior when the teacher is incorrect. To recognize errors from the weak learner, use the gold labels which alleviates the need to use labeliing or data augmentation as it is commonly done distillation setups (Furlanello et al. 2018 Xie al., 2020).
Similarly to Hinton et al. (2015), our final loss is a linear combination of the original cross-entropy loss (CE) and the PoE cross-entropy. We refer to this multi-loss objective as Poł CE.
EXPERIMENTS
We consider several different experimental settings that explore the use of weak learner to isolate and train against dataset biases. All the experiments are conducted on English datasets, and follow
the standard setup for BERT training. Our main model BERT-base (Devlin al. 2019) with 10M parameters. Except when indicated otherwise, our weak learner is a sigmificantly smaller pre- pre-trained masked language model known as TinyBERT (Ture et al., (209)) with 4M parameters (2 layers, hidden size of 128). The weak learner is fine-tuned on exactly the same data as our main model. For instance, when trained on MNLI, gets a 67% accuracy on the matched development set (compared to 84% for BERT-base).
Part of our discussion relies on natural language inference, which has been widely studied this area. The classification task is to determine whether hypothesis statement is true (entailment), false (contradiction) or undetermined (neural) given a premise statement. MNLI (Williams et al. 2018) is the canonical large-scale English dataset study this problem with 433K labeled exam- ples. For evaluation, it features matched sets (examples from domains encountered in training) and mismatched sets (domains not-seen during training).
4

---

sets and discuss the choice of Published as a conference paper at ICLR 2021
Experiments first examine qualitatively the spurious correlations picked up by the method. We then verify the validity of the method on a synthetic experimental setup. Finally, we verify the impact of method by evaluating robust models several the weak learner.
4.1 WEAK LEARNERS REDISCOVER PREVIOUSLY REPORTED DATASET BIASES
Most approaches for cireumventing dataset bias re- quire modeling the bias explicitly, for example using a model limited to only the hypothesis in NLI (Gu-) rurangan et al. 2018). These approaches are effec- tive, but require isolating specific biases present in dataset. Since this process is costly, time consum- ing and error-prone, it is unrealistic to expect such analysis for all new datasets. On the contrary, we hy- pothesize that weak learners might operate like rapid surface learners (Zellers et al.) 2019), picking up on dataset biases without specific signal or input cura- tion and being rather certain of their biased errors (high certainty on the biased prediction errors).
Table I: Breakdown of the 1,000 top certain incorrect training examples.
Category
(%)
Predicted Contradiction
46
Neg. in the hyp.
43
Predicted Entailment
51
High word overlap prem./hyp.
43
Predicted Neutral
3
We first investigate whether our weak learner re-discovers two well-known dataset biases reported on NLI benchmarks: (a) the presence of negative word the hypothesis is highly correlated with the contradiction label (Poliak et al. 2018 Gururangan et al., 2018). (b) high word overlap between the premise and the hypothesis is highly correlated with the entuilment label (McCoy et al., 20196).
To this aim, we fine-tune a weak learner on MNLI (Williams et al., 2018). Hyper-parameters can be found in Appendix [.I] We extract and manually categorize 1,000 training examples wrongly predicted by the weak learner (with a high loss and a high certainty). Table[]break them down per category. Half of these incorrect examples are wrongly predicted as Contradiction and almost all of these contain a negation in the hypothesis. Another half of the examples are incorrectly predicted as Entailment, a majority of these presenting a high lexical overlap between the premise and the hypothesis (5 more words in common). The weak learner thus appears to predict with high- certainty Contradiction label whenever the hypothesis contains a negative word and with high- certainty an Entailment label whenever there is a strong lexical overlap between premise/hypothesi.. Table in Appendix A.3 presents qualitative examples of dataset biases identified by the fine-tuned weak learner.
This analysis is based on set of biases referenced in the literature and does not exclude the possi- bility of other biases being detected by the weak learner. For instance, during this investigation we notice that the presence of "negative sentiment" words (for instance: dull, boring) in the hypothesis appears to be often indicative of Contradiction prediction. We leave further investigation on such behaviors to future work.
4.2 SYNTHETIC EXPERIMENT: CHEATING FEATURE
We consider a controlled synthetic experiment described in He et al. (2019); Clark et al. (2019) that simulates bias. We modify 20,000 MNLI training examples by injecting a cheating feature which encodes an example's 's label with probability Paheat and a random label selected among the two in- correct labels otherwise. For simplicity, we consider the first 20,000 examples. On the evaluation sets, the cheating feature is random and does not convey any useful information. In the present experiment, the cheating feature takes the form a prefix added to the hypothesis ("O" for Contra- diction, "1" for Entailment, "2" for Neutral). We train the weak and main models these 20,000 examples and evaluate their accuracy on the matched development set We expect a biased model to rely mostly on the cheating feature thereby leading poor evaluation performance.
Figure 2 shows the results. As the proportion of examples containing the bias increases, the eval- uation accuracy of the weak learner quickly decreases to reach 0% when Pcheat 0.9. The weak
We use the following list of negation words: no, not, none, nothing, never, aren't 't isn't, weren't 't, neither, don't 7, didn't i, doesn't it cannot, hasn't ', won 't.
We observe similar trends on the mismatched development set.
5

---

Published as a conference paper at ICLR 2021
learner detects the cheating feature during training and is mainly relying on the synthetic bias which not directly indicative of the gold label.
Both He et al. (2019) and Clark et al. (2019) protect against the reliance on this cheating fea- ture by ensembling the main model with bi- ased model that only uses the hypothesis (or its first token). We instead train the main model in the product of experts setting, relying on the weak learner identify the bias. Figure 2 shows that when a majority of the training ex- amples contain the bias (Pcheat 2 0.6), the
80
70-
Acc.
30-
performance of the model trained with cross- entropy drops faster than the one trained in PoE. PoE training leads to more robust model by encouraging it to learn from the mistakes of the weak learner. As Pcheat comes close to 1, the model's training enters "few-shot regime" where there are very few incorrectly predicted biased examples to learn from (exam- ples where following the biased heuristic lead to wrong answer) and the performance of the model trained with PoE drops as well.
20-
Weak Learner (CE)
10
Baseline (CE)
Main Model (PoE)
0.1
0.2
0.3
0.1
05
06
0.7
0.8
0.9
Cheating Rate
Figure 2: Accuracy on MNLI matched develop-
ment set for models with a cheating feature. The model trained with PoE (Main Model) is less sen- sitive to this synthetic bias.
4.3 ADVERSARIAL DATASETS: NLI AND QA
NLI The HANS adversarial dataset (McCoy al., 2019b) was constructed by writing tem- plates generate examples with high premise/hypothesis word overlap to attack models that rely on this bias. one template the word overlap generates entailed premise/hypothesis pairs (heuristic-entailed examples), whereas in another the examples contradict the heuristic (non- heuristic-entailedi The dataset contains 30K evaluation examples equally split between both.
Table 2 shows that the weak learner cxhibits medium performance the in-distribution sets (MNLI) and that on out-of-distribution evaluation (HANS), it relies heavily on the word overlap heuristic. Product of experts training is effective reducing the reliance biases and leads significant gains on the neuristic-non-entalie examples when compared to a model trained with standard cross- entropy leading an improvement of +24%.
The small degradation on in-distribution data is likely because product of experts training does not specialize for in-distribution performance but focuses the weak model errors (He et al.) 2019). The linear combination of the original cross-entropy loss and the product of experts loss (PoE CE) aims counteracting this effect. This multi-loss objective trades off out-of-distribution gener- alization for in-distribution accuracy. A similar trade-off between accuracy and robustness has been reported in adversarial training (Zhag et al., 2019 Tsipras et al. 2019). In Appendix A.6 we detail the influence of this multi-loss objective.
We also evaluate our method on MNLI's hard test set (Gururangan al. 2018 which is expected to be less biased than MNLI's standard split. These examples are selected such that hypothesis-only model cannot predict the label accurately. Tablee]show the results of this experiment. Our method surpasses the performance of PoE model trained with hypothesis-only biased learner. Results on the mismatched set are given in Appendix A.4
QA Question answering models often rely on heuristics such as type and keyword-matching Weissennorn et al. 2017) that can do well on benchmarks like SQuAD (Rajpurkar et al.) 2016). We evaluate on the Adversarial SQuAD dataset (Jia & Liang 2017) built by appending distractor sentences to the passages in the original SQuAD. Distractors constructed such that they look like a plausible answer to the question while not changing the correct answer or misleading humans.
The weak learner alone has low performance both on in-distribution and adversarial sets. PoE training improves the adversarial performance (+ % AddSent) while sacrificing some in-distribution performance. A multi-loss
Results SQuAD v1.1 and Adversarial SQuAD listed in Table 3

---

Published as a conference paper at ICLR 2021
Table MNLI matched dev accuracies, HANS accuracies and MNLI matched hard test set. We repo the All numbers are averaged on 6 runs (with standard deviations). Detailed results for HANS are given Appendix A.4 Reported results are indicated with Utama al. (2020) is a concurrent work where they use a BERT-base fine-tuned 2000 random examples from MNLI weak learner and "PoE An." refers the annealing mechanism proposed by the authors. / and are respectively referring to in-distribution and -ittrr-diitribttionn sets.
Loss
MNLI (I)
HANS
Hard (0)
Ent (7)
Non-Ent (0)
Clark 2019
PoE
82.97
64.67
71.16
Mahabadi et al. 2020
PoE
84.19
95.99
33.30
76.81
Utama 2020
PoE
80.70
86.13
55.20
Ulama et al.
2020
PoE + An.
81.90
88.40
47.13
BERT-baae TinyBERT- Weak
CE
84.52:10.27
98. 12-20.62
26.74:16.15
76.96-10.38
CE
66.93++0.12
99.80++0.09
0.24
46.65++0448
BERT-base Main
PoE
81.35++0.40
81.13+8.1
56.41+55.91
76.54
BERRThhase Main
PoE + CE
83.32++0.24
94.51+0.82
81.35
77.63++0.49
Loss
SQuAD (Z)
Adversarial QA (0)
AddSent
AddOneSent
Clark et al. (2019)*
CE PoE
80.61
42.54
53,91
BiDAF
78.63
57.64
57.17
BERT-base
88.68
53.98
58.84
TinyBERT Weak
CE
41.08
16.02
18.63
BERT-base Main
PoE
83.11
54.92
58.44
BERT-base Main
PoE CE
86.49
56.80
61.04
Table 3: Fl Scores on SQuAD and Adversarial QA. The AddOneSent set model agnostic while we use the AddSent set obtained using an ensemble of BiDAF models (Seo et al. 2017). are reported results. and O are respectively referring to in-distribution and dusttf-distributtin sets.
optimization closes the gap and even boosts adversarial robustness (+3% on AddSent and +2% on AddOneSent). In contrast to our experiments on MNLI/HANS, multi-loss training thus leads here to better performance learner picks up more useful information and removing it entirely might be non-optimal. Multi-loss in this case allows strike balance between learning from, removing, the weak learner.
nut-of-distributio
well. We hypothesize that in this dataset, the weak
ANALYSIS
5.1 REDUCING BIAS: CORRELATION ANALYSIS
To investigate the behavior of the ensemble of the weak and main learner, compute the Pearson correlation between the element- wise loss of the weak (biased) learner and the loss of the trained models following Mahabadi et al. (2020). A correlation of indicates linear relation between the per-example losses (the two learners make the same mistakes), and O indicates the absence of linear correlation (models" mistakes uncorrelated). Figure 3 shows that models trained with linear combination the PoE cross-entropy and the standard cross-entropy have a higher correlation than when trained solely with PoE. This confirms that PoE training is effective at reducing biases uncovered by the weak learner and re-emphasizes that adding standard cross-entropy leads to trade-off between the two.
5.2
HOW WEAK DO THE WEAK LEARNERS NEED TO BE?
7

---

Published as a conference paper at ICLR 2021
100
80
80
MNLI HANS Ent
Acc.
Acc.
40
HANS Non-Ent
20
20-
MNLI HANS Ent
HANS Non-Ent
ThnGERT (4.4)
MinIBERT (11.2)
SmallERR (28.8)
MediumEERT (42.41
TinyBERT (4.4)
MiniBRRT (11.)
SmalBERT (28
MeduMBERT
[41.]]
Leamer
Learnr
(a) Performance weak learners (CE)
(b) Performance of main models (PoE)
Figure 4: Weaker learners assure good balance between out-of-distribution and in-distribution while stronger learners encourage out-of-distribution generalization at the expense of in-distribution performance. We indicate the number of parameters parenthesis (in millions).
We consider parameter size measure of the capacity "weakness" of the weak learner. We fine-tune different sizes of BERT (Ture et al. 2019) ranging from 4.4 to 41.4 million parameters and shows the accuracies MNLI and HANS of the weak learners and main models trained with various weak learners.
these weak models in PoE setting. Figure showws
Varying the capacity of the weak mod- els affects both in-distribution and out-of- distribution performance. -ustrr-dist performance of the main model increases as the weak model becomes stronger (more parameters) up to certain point while in-distribution performances drop slightly at first and then more strongly. When trained jointly with the larger MediumBERT weak learner (41.4 million parameters), the main model gets 97% accuracy on HANS's heuristic-non-entallee set but very low curacy on the in-distribution examples (28% on MNLI and 3% on the heuristic-entailed examples).
Baseline (CE)
Main Model (PoE)
0.6
Main Model (PoE CE)
Pearson conrelation
0.5
0.4
0.1
0.0
HANS
Figure 3: Pearson correlation between the losses (on evaluation sets) of the biased model and different training methods. The PoE training effective at reducing the correlation with the biased model.
As weak model grows in capacity, it be- comes better learner. The average loss de- creases and the model becomes more confi- dent in its predictions. As result, the group certain correct becomes more populated and the main model receives on average a smaller gra- dient magnitude per input. On the contrary, the certain incorrect group (which generally align with out-of-diitribuutinn samples and induce higher magnitude gradient updates, encouraging gen- eralization at the expense of in-distribution performance) becomes less populated. These results corroborate and complement insights from Yaghoobzadeh et al. (2019). This is also reminiscent of findings from Voorrhalli et al. (2018) and Shrivastava al. (2016): not all training samples tribute equally towards learning and in some cases, a carefully selected subset of the training set is sufficient match (or surpass) the performance the whole
8

---

Published as a conference paper at ICLR 2021
Table 4: Accuracies on MNLI (matched) and HANS when trained on a subset MNLI. Training examples are selected from the whole MNLI training set adversarially as detailed in the text. We use BERT-base as the main model and TinyBERT as the weak leaner. / and O are respectively referring to in-distribution and out-of-distribution sets.
Loss
Training set
MNLI (1)
HANS
Ent (!)
Non-Ent (O)
CE MNLI minus negation/overlap
60.59
91.50
24.19
PoE
MNLI minus negation/overlap
64.63
84.85
50.96
5.3 DE-BIASING STILL EFFECTIVE WHEN DATASET BIASES ARE UNKNOWN OR HARD DETECT
While it is difficult enumerate all sources of bias, focus in this work on superficial cues that correlate with the label in the training set but do not transfer. These superficial cues correlate with what can be captured by weak model. For instance, Conneau al. (2018) suggest that word presence can be detected with very shallow networks (linear classifier on top of FastText bag of words) as they show very high accuracy for Word Content, the probing task of detecting which of the 1 '000 target words is present in given sentence.
To verify that weak model is still effective with unknown or hard to detect biases, we consider an example where the bias is only present in small portion of the training. We remove from the MNLI training set all the examples (192K) that exhibit one of the two biases detailed in Section [4.]] high word overlap between premise and hypothesis with entailment label; and negation in the hypothesis with contradiction label. We are left with 268K training examples.
We apply our de-biasing method with these examples as training set. For comparison, we train a main model with standard cross-entropy on the same subset of selected examples. Our results are shown in Table 4 and confirm HANS that our de- de-biasing method is still effective even when the bias is hard to detect. Note that the accuracies on MNLI can not be directly compared to results in Table 2 the class imbalance in the selected subset of examples lead to harder optimization problem explaining the difference of performance.
We present complementary analyses in Appendix. To further show the effectiveness of our method, we included in Appendix A.. an additional experiment on facts verification (Thorne et al. 2018 Schuster et al.] 2019). In Appendix A.5] evaluate the ability of our method to generalize to other domains that do not share the same annotation artifacts. We highlight the trade-off between in-distribution performance and out-of-distribution robustness by quantifying the influence of the multi-loss objective in Appendix An6 draw connection between our 3 groups of examples and recently introduced Data Maps (Swayamdipta et al. (2020).
CONCLUSION
We have presented an effective method for training models robust to dataset biases. Leveraging a weak learner with limited capacity and a modified product of experts training setup, we show that dataset biases do not need to be explicitly known or modeled to be able to train models that can generalize significantly better to out-of-distribution examples. We discuss the design choices for such weak learner and investigate how using higher-capacity learners leads to higher out-of- distribution performance and trade-off with in-distribution performance. We believe that such approaches capable of automatically identifying and mitigating datasets bias will be essential tools for future bias-discovery and mitigation techniques.
ACKNOWLEDGEMENTS
This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20).

---

Published as a conference paper at ICLR 2021
REFERENCES
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. ArXiv, abs/1711.021177 2018.
Yonatan Belinkov, Adam Poliak, S. Shieber, Benjamin Van Durme, and Alexander M. Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. ArXiv, hts/1907.04380, 2019a.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semanties (*SEM 2019), pp. 256-262, Minneapolis, Minnesota, June 2019b. Association for Computational Linguistics. doi: 10.18653/ #1/S19-1028. URL https: www aclweb. orq /attaaaoooooss--11100
Samuel R. Bowman, Gabor Angeli, Christopher Pout,, and Christopher D. Manning, A large anno- tated corpus for learning natural language inference. ArXiv, abs/1508. 5555000886,,..... 2015.
Rémi Cadène, Corentin Dancette, Hedi Ben-younes, M. Cord, and D. Parikh. Rubi: Reducing unimodal biases in visual question answering. In NeurlPS, 2019.
Danqi Chen, J. Bolton, and Christopher D. Manning. A thorough examination of the cnn/daily mail reading comprehension task. ArXiv, abs/1606.02858, 2016.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. ArXiv, Ne01199990668. 2019.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loie Barrault, and Marco Baroni. What you can cram into a single * vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.). Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018. Melbourne, Australia. July 15-20, 2018, Volume 1: Long Papers, pp. 2126-2136. Association for Computational Linguistics. 2018. doi: 10.18653/v1/PP 1/P18 1198. htLpsts:: /tt::://wwww... www. acl http://www...b..cc.////oooooo///
Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In NAACL-HLT, 2019.
Furlanello, Zachary Chase Lipton, Michael Tschannen, L. Itti, and Anima Anandkumar. Born again neural networks. In ICML, 2018.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL-HLT, 2018.
He He, Sheng Zha, and Haohan Wang. Unleam dataset bias in natural language inference by fitting the residual. In DeepLo@ ReepLo@EMNLP-NN EMNLP-JCCNPP, 2019.
Dan Hendrycks, X. Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and D. Song. Pretrained transformers improve out-of-distribution robustness. ACL, 2020.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14: 14:1001-1800, 2002.
Geoffrey F. Hinton, Oriol Vinyals, Jnd Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
A. Jabri, Armand Joulin, and L. V. D. Maaten. Revisiting visual question answering baselines. In ECCV, 2016.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. ArXiv, abs/1777.003222 2017.
Divyansh Kaushik and Zachary Chase Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. In EMNLP, 2018.
10

---

Published as a conference paper at ICLR 2021
Tushar Khot, A. Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI, 2018.
Tal Linzen. How can accelerate progress towards human-likk linguistic generalization? In ACL, 2020.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and J. Henderson. End-to-end bias mitigation by modelling biases in corpora. In ACL, 2020.
R. T. McCoy, Junghyun Min, and Tal Linzen. Berts of feather do not generalize together: Large variability in generalization across models with similar test set performance. ArXiv, abs/1911.02966 2019a.
R. T. McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. ArXiv, abs/1902.01007, 2019b.
Junghyun Min, R. T. McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. Syntactic data augmenta- tion increases robustness to inference heuristics. In ACL, 2020.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial nli: A new benchmark for natural language understanding, 2020.
Ellie Pavlick and Chris Callison-Burch. Most "babies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In ACI,, 2016.
Ellie Pavlick, T. Wolfe, Pushpendre Rastogi, Chris Callison-Burch Mark Dredze, and Benjamin Van Durme. Framenet+: Fast paraphrastic tripling of framenet. In ACL,, 2015.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. ArXiv, ftss/1805.0142. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10666 2019.
Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The winograd schema challenge. In EMNLP-CONLL, 2012.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. ArXiv, waw/16606055500 2016.
Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Ben- jamin Van Durme. Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3:475 2015. doi: 10. 10.1162/tacl a 001552. URL httph::////// wee..............
acb.
org/antho Q11tthhlogy//15--o1 1034
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In AAAI, 2020.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Barzilay. Towards debiasing fact verification models. 2019.
Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. ArXiv, abs/l702.01 2017.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. ArXiv, abw//611..016. 2017.
Abhinav Shrivastava, A. Gupta, and Ross B. Girshick. Training region-based object detectors with online hard example mining. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 761-769, 2016.

---

Published as a conference paper at ICLR 2021
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. There is strength in numbers: Avoiding the hypothesis-only bias in natural language inference via en- semble adversarial training. ArXiv, abs/2004.07790, 2020.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. 2020.
James Thome, Andreas Vlachos, Christos Christodoullppoulss, and A. Mittal. Fever: a large-scale dataset for fact extraction and verification. NAACL. 2018.
D. Tsipras, Shibani Santurkar, L.. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. arXiv: Machine Learning, 2019.
M. Tsuchiya. Performance impact caused by hidden bias of training data for recognizing textual entailment. ArXiv, abs/1 118808001177 2018.
Iulia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv: Computation and Language, 2019.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases, 2020.
Kailas Vodrahalli, K. Li, and Jitendra Malik. Are all training examples created equal? an empirical study, ArXiv, 11..11811..1,9,,, 2018
Alex Wang, Amanpreet Singh, Julian Michael, F. Hill, Omer Levy. and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv, abs/1804.015 804.07461, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, F. Hill, Omer Levy, and Samuel R. Bowman. Superglue: stickier benchmark for general-purposs language understanding systems. ArXiv, abs/1905.00537, 2019.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. Making neural qa as simple as possible but not simpler. In CoNLL, 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguis- tics, 2018 URL
httpg///nhhooog//////// Bateeeooooy//11111111 1101
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Píerric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: State-of-the-arr natural language processing. ArXiv, abs/1910.03371, 2019.
Qizhe Xie, F. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with noisy student im- proves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2020.
Yadollah Yaghoobzadeh, R. Tachet, Timothy J. Hazen, and Alessandro Sordoni. Robust natural language inference models with example forgetting. ArXiv, abs/1911.03386) 2019.
Dani Yogatama, Cyprien de Masson d' Autume, Connor, Tomás Kociský, M. Chrzanowski, Ling- peng Kong, A. Lazaridou, W. Ling, L. Yu, Chris Dyer, and P. Blunsom. Learning and evaluating general linguistic intelligence. ArXiv, abs/ 1tt/1991..1377 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In ACL, 2019.
Hongyang Zhang, Yaodong Yu, J. Jiao, E. Xing, L. Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML,, 2019.
12

---

Published as a conference paper at ICLR 2021
P. Zhang, Yash Goyal, Douglas Summers Stay, Dhruv Batra, and D. Parikh. Yin and yang: Bal- ancing and answering binary visual questions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5014-5022, 2016.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, V. Ordonez, and Kai-Wei Chang. Gender bias in corefer- ence resolution: Evaluation and debiasing methods. ArXiv, abs/1804 bos//180.000 880406676, 2018.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 EXPERIMENTAL SETUP AND HINE-TUNING PAPEE-PARAMMTERS
Our code is based on the Hugging Face Transformers library (Woff et al. 2019). All of our experi- ments are conducted single 16GB V100 using half-precciion training for speed.
NLI We fine-tuned pre-trained TinyBERT Ture et al. 2019) our weak learner. We use the following hyper-parameters: 3 epochs of training with a learning rate of 3e-5, and batch size of 32. The learning rate is linearly increased for 2000 warming steps and linearly decreased to 0 afterward. We use an Adam optimizer / = (0.9, 0.999), Our robust model is BERT-base uabaaa-nesssede and uses the same hyper-parameters. When we train a robust model with multi-loss objective, we give the standard CE a weight of 0.3 and the PoE cross-entropy weight of 1.0. Because of the high variance on HANS (McCoy et al. 2019a). we average numbers on 6 runs with different seeds.
1c-8 and add a weight decay of 0.1.
SQuAD We fine-tuned pre-trained TinyBERT (Ture et al. 2019) as our weak learner. We use the following paper-parameter: epochs of training with a learning rate of 3e-5, and a batch size of 16. The learning rate is linearly increased for 1500 warming steps and linearly decreased to afterward. We use Adam optimizer 3 (0.9, 0.999), Our robust model is BERT-ba SERR-bbase-ucee sasuneaaaaeeeeeee sed and uses the same hyper-parameters. When we train robust model with multi-loss objective, we give the standard CE a weight of 0.3 and the PoE cross-entropy weight of 1.0.
1e-8 and add weight decay of 0.1.
Fact verification We fine-tuned a pre-trained TinyBERT (Ture et al. 2019) our weak learner. We use the following batch size of 32. The learning rate is linearly increased for 1000 warming steps and linearly decreased to 0 afterward. We use an Adam optimizer 3 = (0.9, 0.999), and uses the same hyper-parameters. parameters.
3 epochs of training with a learning rate of 2c-5, and
1e-8 and add a weight
decay of 0.1. Our robust model is When we train a robust model with a multi-loss objective, we give the standard CE weight of 0.3 and the PoE cross- coos--ntropy weight of 1.0. We average numbers 6 runs with different seeds.
A.2 ADDITIONAL EXPERIMENT: FACT VERIFICATION
Following Mahabadi al. 2020), also experiment fact verification dataset. The FEVER dataset (Thoree et al. 2018) contains claim-evidences pairs generated from Wikipedia. |Schuster ct al. (2019) collected new evaluation set for the FEVER dataset avoid the biases observed in the claims of the benchmark. The authors symmetrically augment the claim-evidences pairs of the FEVER evaluation balance the detected artifacts such that solely relying on statistical cues in claims would lead to a random guess. The collected dataset is challenging, and the performance of the models relying on biases evaluated on this dataset drops significantly.
Table 5: Accuracies on the FEVER dev set Thorre et al. 2018) and symmetric hard test (Schuster al. 2019).
Loss
Dev
Symmetic test
Schuster al. (2019)*
PoE
85.85
57.46
Mahabadi et al.
(2020)
CE
85.99
56.49
Mahabadi al.
2020
PoE
84.46
66.25
BERT-base
CE
85,61
15.1
TinyBERT Weak
CE
69.43:10.2
43.10++0.2
BERT-base Main
PoE
81.97+0.5
59.95-3..3
BERT-base Main
PoE + CE
85.29+00.6
57.86+1.4
Our results are Table 5 Our method is again effective at removing potential biases present in the training set and shows strong improvements the symmetric test set.
14

---

Published as a conference paper at ICLR 2021
A.3 SOME EXAMPLES OF DATASET BIASES DETECTED BY WEAK LEARNER
In Table learner.
we show a few qualitative examples of dataset biases detected by fine-tuned weak
Table 6: Weak learners able to detect previously reported dataset biases without explicitly mod- eling them.
Category
Gold
Pre-
Premise
Hypothesis
dicted Label
Negation in Hypothesis
Entailment Contradiction
What explains the stunning logical inconsistencies and misrepresentations in this book?
Nothing can explain the stunning logical inconsisten- cies this book.
Entailment Contradiction
There were many things that disturbed Jon he stood in silence and observed the scout.
Jon didn't 't say anything.
High word overlap
Neutral En- tailment
Conservatives concede that some safety net may be nec- essary.
Democrats concede that some safety net may be necessary. New Kingston is modern commercial center that has many attractions for tourists.
Contradiction Entailment
New Kingston is the mod- ern commercial center of the capital, but it boasts few at- tractions for visitors.
A.4 DETAILED RESULTS ON HANS AND MNLI MISMATCHED
We report the detailed results per heuristics on HANS in Table and the results on the mismatched hard test set of MNLI in Table 8
Table 7: HANS results per heuristic. All numbers are an average on 6 runs (with standard devia- tions). / and O are respectively referring in-distribution and -uits--disti sets.
Loss
Ent (?)
Non-EEt (O)
Subsey
Const
Lexical
Subseg
Const
BERT-basc TinyBERT Weak
99.50=20.83
99.20:t
99.3620.32
52.43:116..4
10.03+4.001
17.76=1.65
CE
100.0=0.00
100.0=:0.00
99.40=0.28
0.00:10.00
0.00=0.00
1.33
BERT-base- Main
POB
11.89++11.39
85.92++9.38
85.56+44.00
74.60+
35.70+8.422
53.53+5560
BERT-basc- Main
PoB CE
90.17=1194
97.02+1.38
96.32=0.58
66.00=+:0019
17.97+5.53
40.07=4.47
Table 8: Accuracies on the MNLI dev mismatched and HARD test mismatched set. I and O are respectively referring to in-distribution and out-tr-diitributtion sets.
Loss
MNLI (7)
HARD (O)
Mahabadi et 22020)* *
PoE
83.47
76.83
BERT-base
84.87+0.18
76.71:10.48
TinyBERT Weak
CE
68.2440.30
48.5340.49
BERT-base- Main
PoE PoE + CE
81.18::0.65
75.6010.70
BERT-base- Main
83.5440.34
76.39-10.2
A.5 NLI: TRANSFER EXPERIMENTS
As highlighted in Section tings that specifically target dataset biases. We further evaluate how well method improves
4 our method is effective at improving robustness to adversarial set-
15

---

Published as a conference paper at ICLR 2021
generalization to domains that do not share the same annotation artifacts. Mahabadi al. (2020)
highlighted that product of experts is effective at improving generalization to other NLI benchmarks when trained on SNLI (Bowman al., 2013). We follow the same setup: notably, perform sweep on the weight of the cross-entropy in our multi-loss objective and perform model selection on the development diagnostic test (Wang et al. 2018), AddOneRTE (AddOne) (Pavlick & Callison-Burche (201)) Defi- nite Pronoun Resolution (DPR) (Rahman Ng. 2012). FrameNet+ (FN+) (Pavlick et al. (0155) and
of each dataset. We evaluate SciTail (Khot et al. (0018), GI.U benchmark's
Semantic Proto-Roles (SPR) (Reisinger et al. 2015). We also evaluate on the hard SNLI test set (Gururangan et al. 2018), which a set where hypothesis model cannot solve easily.
the results. Without explicitly modeling the bias in the dataset, our method matches or surpasses the generalization performance previously reported, at the exception of GLUU's diagnostic dataset and SNLJ's I hard test set. Moreover, we notice that the multi-loss objective sometimes leads stronger performance, which suggests that some cases, it can be sub-optimal to completely remove the information picked up by the weak learner. We hypothesize that the multi-loss objective balances the emphasis on somain-specific features (favoring in-distribution performance) and their removal through de- de-biasing (benefiting domain transfer performance). This might explain why do not observe improvements on SNLI's hard test set and GLUE's diagnostic set in the PoE setting.
Table 9: Transfer accuracies on NLI benchmarks. We train BERT-base on SNLI and tested on the target test set. * are reported results.
Mahabadi al. (2020)*
Ours PoE
Data
CE
PoE
CE
PoE + CE
AddOne
87.34
87.86
86.82
85.53
87.34
DPR
49.50
50.14
50.29
50.40
50.34
SPR
59.85
62.45
58.27
62.82
60.50
FN+
53.16
53.51
54.35
54.71
54.90
SCITAIL
67.64
71,40
69.71
74.13
73.57
GLUE
54.08
54.71
55.34
49.28
53.26
SNLI Hard
80.53
82.15
81.17
78.10
81.99
A.6 INFLUENCE OF MULTI-LOSS OBJECTIVE
Our best performing setup features a linear combination the PoE cross-entropy and the standard cross-entropy. We fix the weight of the PoE cross-entropy to 1. and modulate the linear coefficient of the standard cross-entropy. Figure 5 shows the influence of this multi-loss objective. As the weight of the standard cross-entropy increases, the in-distribution performance increases while the -it-----iittibttionn performance decreases. This effect is particularly noticeable on MNLI/HANS (see Figure 5ap. Surprisingly, this trade-off's less pronounced on SQuAD/Adversan SQuAD: the Fl development score increases from 85,43 for 0.1 to 88.14 for = 1.9 while decreasing from 56.67 to 55.06 on AddSent.
Our multi-loss objective is similar to the annealing mechanism proposed in (Utama et al. 2020 In fact, the annealing coefficient decreases, the modified probability distribution of the weak model converges to the uniform distribution. As seen in Section 3.2 when the distribution of the weak model is close to the uniform distribution (high-uncertainty)) the gradient of the loss for PoE equivalent to the gradient of the main model trained without PoE (i.e. the standard cross-entropy). In this work, we consider straight-forwart setup where we linearly combine the two losses throughout the training with fixed coefficients.
A.7 CONNECTION TO Data Maps SWAYAMDIPTA ET AL.
2020
We hypothesize that three identified groups (certain / incorrect, certain correct, and un- certain) overlap with the regions identified by data cartographies (Swayamdipta et al.) 2020). The authors project each training example onto 2D coordinates: confidence, variability. The first one the mean of the gold label probabilities predicted for each example across training epochs. The second one is the standard deviation. Confidence is closely related to the loss (intuitively, high-
16

---

Published as a conference paper at ICLR 2021
100
MNLI
75
SQUAD AddSent
HANS Ent
HANS Non-Ent
70-
AddOneSent
60
85
60
0.1
3.5
0.6
0.25
0.50
0.75
1.00
1.25
1:50
1.75
O
Q
(a) MNLI/HANS
(b) SQuAD/Ady SQuAD
Figure 5: The multi-loss objective controls trade-off between the in-distribution performance and out-of-distrne offlo---iitrittion robustness.
1.0
Easy Masy-ttoleen
0.8
Confidence
0.6
Ambiguous
0.4
0.2
Uncertain
Hard-to-lens
Incorrect Certain / Correct
0.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Variability
Variability
Figure 6: Our groups are included in the regions identified Data Maps.
confidence example is "easier" predict). Variability is connected to the uncertainty (the probability of the true class for high-variability examples fluctuates during the training underlying the model's indecisiveness).
Our most interesting group (certain incorrect) bears similarity with the ambiguous region (the model is indecisive about these instances frequently changing its prediction across training epochs) and the hard-to-lead region (which contains a significant proportion of mislabeled examples). The authors observe that the examples in these 2 regions play an important role out-of-disttibutiin generalization. Our findings go in the same direction the weak model encourages the robust model to pay a closer look these certain incorrect examples during training.
To verify this claim, we follow their procedure and log the training dynamics of our weak learner trained on MNLI. Figure 6 (left) shows the Data Map obtained with our weak learner. For each of our 3 groups,
select the 10,000 examples that emphasized the most the characteristic of the group. For instance, for uncertain, we take the 10,000 examples with the highest entropy. In Figure 6 (right) plot these 30,000 examples onto the Data Map. We observe that our certain / correct examples are in the easy-to-learn region, our certain incorrect examples are the hard- to-learn region and uncertain examples are mostly in the ambiguous region.
17

---

Published as a conference paper at ICLR 2021
Conversely, we verify that the examples in the ambiguous region mostly in uncertain group, the examples from the hard-ta word-to-leam are mostly in uncertain and certain / amples from the easy-to-leame are mainly in certain / correct and uncertain.
incorrect, and the ex-
18


Reviews:
[{"review_id": "vr_Y5sIqJFU", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*: This paper proposes a method for training model that are robust to spurious correlations, building upon prior work that uses product-of-experts and a model explicitly trained on a dataset bias (e.g., a hypothesis-only model). Instead of using a model explicitly trained to learn the dataset bias, the authors use a “weak learner” with limited capacity. Then, this model is used in the PoE setting as in past work. The advantage of this method is that a model developer doesn’t need to know that a bias exists, since the hope is that the weak learner will implicitly learn the bias.\n\n*Strengths*: A thorough study of using a limited-capacity auxiliary model to train more robust models, which helps a final model ignore spurious correlations that are easy to learn.\n\n*Weaknesses*: The work is a rather straightforward extension of prior work. Furthermore, the authors only evaluate on 2 textual tasks---I would have liked to see more experiments with spurious correlations in vision (e.g., VQA or the datasets used in https://openreview.net/forum?id=ryxGuJrFvS), and other experiments on text (e.g., the TriviaQA-CP dataset in the Clark paper). As is, it’s hard to glean how broadly applicable this method actually is. I would have also liked to see more of a comparison with methods that use known bias (e.g., Clark et al or He et al)---it seems like some of the comparisons in the table aren’t completely fair.\n\n*Recommendation*: 6 . I think this paper is a potentially-useful extension of a prior method, but I’m still somewhat unconvinced that this method is applicable in settings where the bias is hard to detect, which is what we really care about (since, if the bias is easy to detect, we can use Clark et al and other methods).\n\nComments and Questions:\n\n1. The comparisons to Clark et al aren’t fair comparisons for adversarial SQuAD, since the Clark et al paper uses a different base model for adversarial SQuAD (modifed BIDAF).\n\n2. The weak learner is a rather blunt instrument. It picks up dataset biases, but it also likely picks up features that are actually useful---not all robust features have to be difficult to learn. Is it possible to better quantify what useful information is being learned (and subsequently thrown out) by the weak learner? This would make it easier to determine if using it is worthwhile.\n\n3. While it’s true that the weak model empirically learns to re-learn the same dataset biases targeted in prior work (e.g., negation correlates with contradiction), it’s somewhat unclear to me how well this method would translate to a setting with unknown biases. The MNLI / SQuAD examples are a bit artificial since we already have knowledge of the bias---it’s possible that weak learners can pick up on spurious features that are “easy to learn”, which are the same ones that humans notice. I’d like to see whether this method applies well to tasks where it isn’t immediately obvious that the bias is easy to learn; perhaps a synthetic experiment would be useful here. Is it possible to modulate the learnability of the bias? The synthetic experiments in the paper suggest that for cases the bias is hard to learn, this method isn’t very effective, which makes sense---in how many of the cases in the literature is the bias hard to learn? This is another reason why I think more experiments would be useful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPdOXZwflxg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Reason for score\n\nThe research problem is critical. The solution is appropriate and novel. The claims are validated. The experiments are interesting.\nHowever, the writing in section 3, 4 et 5 should be improved. If so, I would be willing to raise my score.\n\n## My background\n\nMy research is focused on detecting and avoiding data biases (or spurious correlations) learned by deep neural networks. This is the exact scope of this paper. However, my area of expertise is computer vision and multimodal text-image, not natural language processing.\n\n## Summary\n\nContext:\nThe paper focuses on automatically detecting data biases learned by natural language processing models and overcoming them using a learning strategy.\n\nProblem:\nThe authors identify and tackle issues of state-of-the-art methods:\n- they are required to already know about a certain bias to be overcome.\n\nSolution and novelty:\nThe proposed method consists in 1) training a weak model that aims at detecting biases 2) overcoming these biases by training a main model using a product of experts (Hinton, 2002) with the predictions of the fixed weak model.\n\nClaim:\n- A weak model can be used to discover data biases\n- The proposed method produces a main model that generalize better to out-of-distribution examples\n\n## What I liked the most\n\n- meta-problem of automatically detecting and overcoming biases in neural networks is critical\n- well contextualized\n- relevant issues of state of the art have been identified\n- intro and related work are easy to read and understand\n- novel, simple and interesting method to tackle them\n- interesting figures\n- experiments are  interesting and well chosen\n\n## What could be improved\n\n1. Abstract, introduction and 2. Related work\n- Your research problem and solution are general and can be applied to many fields. Is there a specific reason why you decided to focus on NLP only?\n- You could improve the impact of your approach by citing papers that tackle the same problem with similar solutions from different fields. \"Clark et al. 2019 Don’t take the easy way out: Ensemble-based methods for avoiding known dataset biases\" that you already cite ran some experiments in multiple fields (NLP, VQA, etc.). \"Cadene et al. Rubi: Reducing unimodal biases for visual question answering (NeurIPS2019)\" in VQA could also be cited.\n\n3. Proposed Method\n- Next to Eq1: Why an element wise sum is equivalent to an element wise multiplication after softmax? It seems wrong to me.\n- It could be useful to have a general definition of the PoE loss (instead of just an example of binary cross entropy in Eq2)\n- See 4.3, you should define PoE+CE here.\n\n4. Experiments\n- Overall, I think it is important that you improve the writing for this section and reduce jargon. It is really difficult to understand for readers that are not familiar with the datasets on which you perform your study. Also it is really difficult to understand which dataset is \"in-distribution\" or \"out-of-distribution\".\n- You don't define \"development matched accuracy\" before using it.\n4.1\n- You use too many footnotes that could be included in the text.\n4.2\n- You don't define \"CE\" (even in the caption of Figure2).\n- In Table 2, you could reduce jargon by using Weak and Main instead of \"W\" and \"M\".\n- In Table 2, you don't define \"An.\" even in the caption.\n4.3\n- I don't understand why \"PoE+CE\" is better on \"Hard\" \n- I don't like that you propose to use \"PoE+CE\" as your method of choice \"to counteract these effects\" without defining it in section 3. To be clear, I still don't understand what is the learning method that you propose PoE or PoE+CE?\n\n5. Analysis\n5.2\n- Title is on two lines instead of one\n- I don't understand \"When trained jointly with the larger MediumBERT weak learner......\" How many parameters? Don't expect your reader to look at Figure 4 to obtain this information.\n\n6. Conclusion\n- Could you add a discussion about the limitations of your approach. In particular: How to choose the number of parameters of your weak learner? What to choose between PoE and PoE+CE? And most critically, if you don't assess the type of biases and the amount of biases included in the dataset, how to be sure that your method will have a beneficial impact? Then, if you need to assess the type of biases, using another method that specifically targets them could be more efficient.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MemGGWWCt_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper focuses on the known problem that current NLP models tend to solve tasks by exploiting superficial properties of the training data that do not generalize. For example, in the NLI task, models learn that negation words are indicative of the label \"contradiction\" and high word overlap is indicative of the label \"entailment\". There have been many recent solutions proposed for mitigating such behavior, but existing methods have tended to assume knowledge of the specific dataset biases a priori. In this paper, the authors propose a method based on product of experts that doesn't assume particular knowledge of specific dataset biases. The method works by first training a weak model and then training a \"main\" model using a loss that upweights examples on which the weak model performs poorly (namely, predicts the wrong answer with high confidence). The assumption is that weak models will exploit heuristics, and so this method will deincentivize the main model to use those same heuristics. The authors evaluate on a range of tasks, including a simulated bias setting, and NLI setting, and a QA setting, and offer a fair amount of analysis of their results. In particular, the analysis showing that the weak learners do in fact adopt the biases which have been documented elsewhere in the literature is interesting, and the discussion of \"how weak does the weak learner need to be\" is appreciated (a few questions on this below).\n\nStrengths:\n* Straightforward method for addressing an important known problem with neural NLP models\n* Thorough analysis, not just a \"method and results\" paper\n\nWeaknesses:\n* Novelty might be somewhat limited, method is not wildly creative (but I don't necessarily think \"wild creativity\" is a prerequisite for scientific value). The authors do a good job of directly contending with the similar contemporaneous work in their paper\n\nAdditional Comments/Questions:\n\nJust a few thoughts that came up while reading...\n* The weakness-of-weak-learner analysis is interesting. I imagine this is not something that can be understood in absolute terms, i.e., I would not expect there to be some level of weakness that is sufficient for all biases and all datasets. E.g., surely the lexical overlap bias is \"harder\" to learn than a lexical bias like the presence of negation words, since recognizing lexical overlap presupposes recognizing lexical identity. Therefore, I'd imagine knowing how weak the weak learner needs to be requires some intuition about which biases you are trying to remove, which runs counter to the primary thrust of the paper, namely, removing bias without knowing what the bias is. Thoughts?\n* Its interesting that even with this the performance on hans non-entailed is still only 56%, which is better but still not exactly good, and doesn't suggest the model has learned the \"right\" thing so much as its has learned not to use that particular wrong thing. For research questions such as this (\"is the model using the heuristic?\") I always find it unsatisfying to think about performance gains that are in between 0 and 100. E.g., when we talk about human learning, we usually see an abrupt shift when the learner \"gets it\", and our hope in removing the spurious features with methods like yours would be that we'd help the neural models similarly \"get it\" and reach 100% at least on examples that isolate the effect of this spurious feature. I don't expect you to have an answer for this, but just raising to hear your thoughts. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EM6nENB9ut8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThe authors argue that they have proposed a method to train robust models to biases without having prior knowledge of the biases. They argue also to provide analysis on how weak learner capacity impacts the in-domain/out-of-domain performance.\n\nReasons to reject:\n1) The authors argue they have shown the model with limited capacity capture biases. However, this has been shown already in [1] in 2019 and therefore is not a contribution of the authors.\n2) The main method proposed in this paper, is exactly the same method proposed in [2]. Please note that [2] was already available in early July 2020, and on top of existing work, the paper does not provide other contributions. \n3) About the third argued contribution on showing how the performance of the debiasing method change based on the capacity of weak learners, in [1], the authors included the discussion between the choice of weak learners on their impact. Though the method in [1] is different, the discussion in that paper still would apply here as well.  Please refer to table 1-3 and Figure 1 in [1]. \n\nGiven the points above, and since the main method in the paper is proposed in [2], the paper does not provide enough contributions to be suitable for the ICLR venue. \n\n[1] Robust Natural Language Inference Models with Example Forgetting, Yaghoobzadeh et al, https://arxiv.org/pdf/1911.03861.pdf, 2019 \n[2] Towards Debiasing NLU Models from Unknown Biases, Utama et al, 13 July 2020, https://openreview.net/forum?id=UHpxm2K-jHE, EMNLP 2020 ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of learning models for NLP tasks that are less reliant on artifacts and other dataset-specific features that are unlikely to be reliable for new datasets. This is an important problem because these biases limit out-of-distribution generalization. Prior work has considered models that explicitly factor out known biases. This work proposes using an ensemble of weak learners to implicitly identify some of these biases and train a more robust model. The work shows that weak learners can capture some of the same biases that humans identify, and that the resulting trained model is significantly more robust on adversarially designed challenge tasks while sacrificing little accuracy on the test sets of the original data sets.

The paper's method is useful, straightforward, and intuitively appealing. The experiments are generally well conducted. Some of the reviewers raised questions about evaluating on tasks with unknown biases. The authors addressed these concerns in discussion and we encourage them to include this in the final version of the paper using the additional page.
Decision: Accept (Poster)