Learning from others' mistakes: Avoiding dataset biases without modeling them

1 INTRODUCTION . The natural language processing community has made tremendous progress in using pre-trained language models to improve predictive accuracy ( Devlin et al. , 2019 ; Raffel et al. , 2019 ) . Models have now surpassed human performance on language understanding benchmarks such as SuperGLUE ( Wang et al. , 2019 ) . However , studies have shown that these results are partially driven by these models detecting superficial cues that correlate well with labels but which may not be useful for the intended underlying task ( Jia & Liang , 2017 ; Schwartz et al. , 2017 ) . This brittleness leads to overestimating model performance on the artificially constructed tasks and poor performance in out-of-distribution or adversarial examples . A well-studied example of this phenomenon is the natural language inference dataset MNLI ( Williams et al. , 2018 ) . The generation of this dataset led to spurious surface patterns that correlate noticeably with the labels . Poliak et al . ( 2018 ) highlight that negation words ( “ not ” , “ no ” , etc . ) are often associated with the contradiction label . Gururangan et al . ( 2018 ) , Poliak et al . ( 2018 ) , and Tsuchiya ( 2018 ) show that a model trained solely on the hypothesis , completely ignoring the intended signal , reaches strong performance . We refer to these surface patterns as dataset biases since the conditional distribution of the labels given such biased features is likely to change in examples outside the training data distribution ( as formalized by He et al . ( 2019 ) ) . A major challenge in representation learning for NLP is to produce models that are robust to these dataset biases . Previous work ( He et al. , 2019 ; Clark et al. , 2019 ; Mahabadi et al. , 2020 ) has targeted removing dataset biases by explicitly factoring them out of models . These studies explicitly construct a biased model , for instance , a hypothesis-only model for NLI experiments , and use it to improve the robustness of the main model . The core idea is to encourage the main model to find a different explanation where the biased model is wrong . During training , products-of-experts ensembling ( Hinton , 2002 ) is used to factor out the biased model . While these works show promising results , the assumption of knowledge of the underlying dataset bias is quite restrictive . Finding dataset biases in established datasets is a costly and time-consuming process , and may require access to private details about the annotation procedure , while actively re- ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion ducing surface correlations in the collection process of new datasets is challenging given the number of potential biases ( Zellers et al. , 2019 ; Sakaguchi et al. , 2020 ) . In this work , we explore methods for learning from biased datasets which do not require such an explicit formulation of the dataset biases . We first show how a model with limited capacity , which we call a weak learner , trained with a standard cross-entropy loss learns to exploit biases in the dataset . We then investigate the biases on which this weak learner relies and show that they match several previously manually identified biases . Based on this observation , we leverage such limited capacity models in a product of experts ensemble to train a more robust model and evaluate our approach in various settings ranging from toy datasets up to large crowd-sourced benchmarks : controlled synthetic bias setup ( He et al. , 2019 ; Clark et al. , 2019 ) , natural language inference ( McCoy et al. , 2019b ) , extractive question answering ( Jia & Liang , 2017 ) and fact verification Schuster et al . ( 2019 ) . Our contributions are the following : ( a ) we show that weak learners are prone to relying on shallow heuristics and highlight how they rediscover previously human-identified dataset biases ; ( b ) we demonstrate that we do not need to explicitly know or model dataset biases to train more robust models that generalize better to out-of-distribution examples ; ( c ) we discuss the design choices for weak learners and show trade-offs between higher out-of-distribution performance at the expense of the in-distribution performance . 2 RELATED WORK . Many studies have reported dataset biases in various settings . Examples include visual question answering ( Jabri et al. , 2016 ; Zhang et al. , 2016 ) , story completion ( Schwartz et al. , 2017 ) , and reading comprehension ( Kaushik & Lipton , 2018 ; Chen et al. , 2016 ) . Towards better evaluation methods , researchers have proposed to collect “ challenge ” datasets that account for surface correlations a model might adopt ( Jia & Liang , 2017 ; McCoy et al. , 2019b ) . Standard models without specific robust training methods often drop in performance when evaluated on these challenge sets . While these works have focused on data collection , another approach is to develop methods allowing models to ignore dataset biases during training . Several active areas of research tackle this challenge by adversarial training ( Belinkov et al. , 2019a ; b ; Stacey et al. , 2020 ) , example forgetting ( Yaghoobzadeh et al. , 2019 ) and dynamic loss adjustment ( Cadène et al. , 2019 ) . Previous work ( He et al. , 2019 ; Clark et al. , 2019 ; Mahabadi et al. , 2020 ) has shown the effectiveness of product of experts to train un-biased models . In our work , we show that we do not need to explicitly model biases to apply these de-biasing methods and can use a more general setup than previously presented . Orthogonal to these evaluation and optimization efforts , data augmentation has attracted interest as a way to reduce model biases by explicitly modifying the dataset distribution ( Min et al. , 2020 ; Belinkov & Bisk , 2018 ) , either by leveraging human knowledge about dataset biases such as swapping male and female entities ( Zhao et al. , 2018 ) or by developing dynamic data collection and benchmarking ( Nie et al. , 2020 ) . Our work is mostly orthogonal to these efforts and alleviates the need for a human-in-the-loop setup which is common to such data-augmentation approaches . Large pre-trained language models have contributed to improved out-of-distribution generalization ( Hendrycks et al. , 2020 ) . However , in practice , that remains a challenge in natural language processing ( Linzen , 2020 ; Yogatama et al. , 2019 ) and our work aims at out-of-distribution robustness without significantly compromising in-distribution performance . Finally , in parallel the work of Utama et al . ( 2020 ) presents a related de-biasing method leveraging the mistakes of weakened models without the need to explicitly model dataset biases . Our approach is different in several ways , in particular we advocate for using limited capacity weak learner while Utama et al . ( 2020 ) uses the same architecture as the robust model trained on a few thousands examples . We investigated the trade-off between learner ’ s capacity and resulting performances as well as the resulting few-shot learning regime in the limit of a high capacity weak model . 3 METHOD . 3.1 OVERVIEW . Our approach utilizes product of experts ( Hinton , 2002 ) to factor dataset biases out of a learned model . We have access to a training set ( xi , yi ) 1≤i≤N where each example xi has a label yi among K classes . We use two models fW ( weak ) and fM ( main ) which produce respective logits vectors w and m ∈ RK . The product of experts ensemble of fW and fM produces logits vector e ∀1 ≤ j ≤ K , ej = wj +mj ( 1 ) Equivalently , we have softmax ( e ) ∝ softmax ( w ) softmax ( m ) where is the element-wise multiplication . Our training approach can be decomposed in two successive stages : ( a ) training the weak learner fW with a standard cross-entropy loss ( CE ) and ( b ) training a main ( robust ) model fM via product of experts ( PoE ) to learn from the errors of the weak learner . The core intuition of this method is to encourage the robust model to learn to make predictions that take into account the weak learner ’ s mistakes . We do not make any assumption on the biases present ( or not ) in the dataset and rely on letting the weak learner discover them during training . Moreover , in contrast to prior work ( Mahabadi et al. , 2020 ; He et al. , 2019 ; Clark et al. , 2019 ) in which the weak learner had a hand-engineered bias-specific structure , our approach does not make any specific assumption on the weak learner such as its architecture , capacity , pre-training , etc . The weak learner fW is trained with standard cross-entropy . The final goal is producing main model fM . After training , the weak model fW is frozen and used only as part of the product of experts . Since the weak model is frozen , only the main model fM receives gradient updates during training . This is similar to He et al . ( 2019 ) ; Clark et al . ( 2019 ) but differs from Mahabadi et al . ( 2020 ) who train both weak and main models jointly . For convenience , we refer to the cross-entropy of the prediction e of Equation 1 as the PoE cross-entropy . 3.2 ANALYSIS : THE ROBUST MODEL LEARNS FROM THE ERRORS OF THE WEAK LEARNER . To better explore the impact of PoE training with a weak learner , we consider the special case of binary classification with logistic regression . Here w and m are scalar logits and the softmax becomes a sigmoid . The loss of the product of experts for a single positive example is : LPoE , binary = −m− w + log ( 1 + exp ( m+ w ) ) ( 2 ) Logit w is a fixed value since the weak learner is frozen . We also define the entropy of the weak learner asHw = −p log ( p ) − ( 1− p ) log ( 1− p ) where p = σ ( w ) as our measure of certainty . Different values of w from the weak learner induce different gradient updates in the main model . Figure 1a shows the gradient update of the main model logitm . Each of the three curves corresponds to a different value of w the weak model . • Weak Model is Certain / Incorrect : the first case ( in blue ) corresponds to low values of w. The entropy is low and the loss of the weak model is high . The main model receives gradients even when it is classifying the point correctly ( ≈ m = 5 ) which encourages m to compensate for the weak model ’ s mistake . • Weak Model is Uncertain : the second case ( in red ) corresponds tow = 0 which means the weak model ’ s entropy is high ( uniform probability over all classes ) . In this case , product of experts is equal to the main model , and the gradient is equal to the one obtained with cross-entropy . • Weak Model is Certain / Correct : the third case ( in green ) corresponds to high values of w. The entropy is low and the loss of the weak model is low . In this case , m ’ s gradients are “ cropped ” early on and the main model receives less gradients on average . When w is extremely high , m receives no gradient ( and the current example is simply ignored ) . Put another way , the logit values for whichm receives gradients are shifted according the correctness and certainty of the weak model . Figure 1b shows the concentration of training examples of MNLI ( Williams et al. , 2018 ) projected on the 2D coordinates ( correctness , certainty ) from a trained weak learner ( described in Section 4.1 ) . We observe that there are many examples for the 3 cases . More crucially , we verify that the group certain / incorrect is not empty since the examples in this group encourage the model to not rely on the dataset biases . ( a ) Gradient update of m for different values of w on binary classification . ( b ) 2D projection of MNLI examples from a trained weak learner . Colors indicate the concentration and are in log scale . Figure 1 : The analysis of the gradients reveals 3 regimes where the gradient is shifted by the certainty and correctness of the weak learner . These 3 regions are present in real dataset such as MNLI .

Summary: This paper focuses on the known problem that current NLP models tend to solve tasks by exploiting superficial properties of the training data that do not generalize. For example, in the NLI task, models learn that negation words are indicative of the label "contradiction" and high word overlap is indicative of the label "entailment". There have been many recent solutions proposed for mitigating such behavior, but existing methods have tended to assume knowledge of the specific dataset biases a priori. In this paper, the authors propose a method based on product of experts that doesn't assume particular knowledge of specific dataset biases. The method works by first training a weak model and then training a "main" model using a loss that upweights examples on which the weak model performs poorly (namely, predicts the wrong answer with high confidence). The assumption is that weak models will exploit heuristics, and so this method will deincentivize the main model to use those same heuristics. The authors evaluate on a range of tasks, including a simulated bias setting, and NLI setting, and a QA setting, and offer a fair amount of analysis of their results. In particular, the analysis showing that the weak learners do in fact adopt the biases which have been documented elsewhere in the literature is interesting, and the discussion of "how weak does the weak learner need to be" is appreciated (a few questions on this below).