IRM—when it works and when it doesn't: A test case of natural language inference

1 Introduction . Deep learning models show strong performance when tested on data from the same distribution they were trained on , matching or even surpassing humans ( Zhang et al. , 2017 ; Brinker et al. , 2019 ) . However , this performance often stems from relying on spurious correlations rather than human-like reasoning , causing these models to “ break ” in real world scenarios ( Geirhos et al. , 2018 ; McCoy et al. , 2019 ) . A recent method called Invariant Risk Minimization ( IRM ; Arjovsky et al . 2020 ) aims to learn causal features whose correlation with the label is invariant across different distributions , thus leading to better out-of-distribution ( o.o.d ) generalization . IRM splits the training data into different subsets , or environments , across which varying spurious correlations cause distribution shifts . The goal is to learn a representation that yields the same optimal classifier for all environments . Although several studies ( Choe et al. , 2020 ; Ahuja et al. , 2020a ; Kamath et al. , 2021 ) have investigated IRM and its variants , the empirical results so far have mainly focused on synthetic settings and simpler models ( e.g. , shallow multilayer-perceptrons , bag-of-words models ) . This motivates us to investigate IRM in natural settings with complex models . We take natural language inference ( NLI ) as a test case , where the model needs to predict if a hypothesis sentence is entailed by the premise sentence . While theoretically requiring deep language understanding , many NLI datasets contain biases , or spurious correlations between superficial features and labels ( Gururangan et al. , 2018 ; Poliak et al. , 2018 ; Tsuchiya , 2018 ) , such as word overlap correlating with the entailment label . Such heuristics allow models to perform superficially well on the benchmark , but fail catastrophically when these heuristics ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . no longer hold ( Naik et al. , 2018 ; Glockner et al. , 2018 ; McCoy et al. , 2019 ) . Common debiasing methods typically rely on explicit modeling of the biases ( Belinkov et al. , 2019a , b ; Karimi Mahabadi et al. , 2020 ; He et al. , 2019 ) . In contrast , IRM offers interesting possibilities . It is model agnostic , not built for a specific bias and , given appropriate environments , generalizes to distributions where the spurious correlation varies . In this work , we design a series of debiasing experiments , intended to bridge the gap between fully synthetic and naturalistic scenarios . Table 1 shows examples of each setting . We begin with a toy experiment where both the dataset and the bias are synthetic . Next , we add a level of complexity by injecting synthetic bias to a real NLI dataset and use state-of-the-art pre-trained models . Last , we investigate natural NLI datasets with known dataset biases . For the last setting , splitting environments using categorical biased features as done in previous work is not applicable . Therefore , for real-world scenarios where the bias is known but might be a high dimensional feature , we develop a simple way to split data into different environments , described in “ Environment generation ” in Section 3.2 . In each of these settings , we train models with empirical risk minimization ( ERM ) and IRM and evaluate them on o.o.d test sets . Our experiments yield the following results : • In the toy setting , performance follows the theory : ERM performs well on the training set but fails completely on the o.o.d test set , while IRM ignores the bias and thus performs slightly worse on the training set , but perfectly on the o.o.d test set . • On natural NLI datasets and deep models—with either synthetic bias or natural bias—IRM outperforms ERM when evaluated on o.o.d test sets . • However , in these more naturalistic settings , IRM is not able to completely discard the bias , while ERM does not rely solely on the bias . Thus , in practice the advantage of IRM is small . • When used with state-of-the-art models , IRM shows a large variance in performance across random seeds . To better characterize when IRM works better or worse than ERM , we highlight three criteria that govern its success : the prevalence of the bias in the training set ( how many examples are biased ) , the strength of the bias ( how strong is the correlation between a label and a biased feature ) , and the training data size . While bias strength and data size were not empirically investigated in real-world scenarios , bias prevalence was completely overlooked in previous work . We find that when all three criteria are met ( prevalent and strong bias , large training set ) , IRM tends to perform better than ERM . When bias prevalence , strength , or training data size are limited , IRM is less stable and ERM performs better . To conclude , our investigation shows important challenges in assessing the performance of IRM in plausible real-world scenarios , pointing to the need to employ a more naturalistic approach to characterizing the settings in which IRM can perform well . 2 Related Work . IRM IRM was motivated by Peters et al . ( 2016 ) , who connect invariance with causality by establishing a link between the cause of a target variable and the invariant predictor . As a proof-of-concept , Arjovsky et al . experimented only with very simple settings , namely MNIST with varying background colors . Several studies further investigated IRM or proposed variants of IRM . Rosenfeld et al . ( 2020 ) show that in the non-linear setting , there exists a nearly optimal classifier which uses non-invariant features , thus the classifier found by IRM may not generalize to o.o.d . data . However , their result assumes infinite samples and we instead focus on an empirical study in the finite sample case . Kamath et al . ( 2021 ) point out the discrepancies between the original IRM formulation and various instantiations of it . Since in non-toy examples such comparison is non-trivial , we focus on IRMv1 , the practical formulation suggested in the original paper . Both Ahuja et al . ( 2020a ) and Krueger et al . ( 2021 ) propose new objectives and show superior stability , and improvement over IRM when adding covariate shift , respectively , on colored MNIST . Gulrajani and Lopez-Paz ( 2020 ) devise a test suite to fairly compare algorithms for domain generalization , and show that with careful tuning and model selection , ERM outperforms all other algorithms . Choe et al . ( 2020 ) show that IRM outperforms ERM in a sentiment analysis task with synthetic bias , but restrict themselves to a bag-of-words model . While different approaches were suggested , most are evaluated in simple settings . Although Gulrajani and Lopez-Paz targeted more naturalistic scenarios , they focused only on image classification . In addition to focusing on a language task , we show performance gains of IRM over ERM in naturalistic settings with complex models , and perform an analysis of the factors that affect IRM ’ s performance . Bias in NLI NLI is a widely studied task in natural language processing , concerned with identifying the relation between a premise and a hypothesis . Recent studies ( Gururangan et al. , 2018 ; Poliak et al. , 2018 ; McCoy et al. , 2019 ) reveal biases in NLI datasets such as SNLI ( Bowman et al. , 2015 ) and MNLI ( Williams et al. , 2018 ) . Two widely observed biases are hypothesis bias and overlap bias . Overlap bias is characterized by high word overlap between the premise and the hypothesis , shown to be correlated with entailment . Hypothesis bias refers to patterns in the hypothesis correlated with a specific label , allowing for correct prediction without considering the premise ; a common example is negation words correlated with the contradiction label ( Table 1 ) . Consequently , several challenge sets ( Naik et al. , 2018 ; Glockner et al. , 2018 ; McCoy et al. , 2019 ) demonstrated performance degradation when the bias no longer holds . We evaluate the ability of IRM to mitigate reliance on bias and thus improve performance in such cases . 3 Method . 3.1 Background on IRM . Our goal is to investigate the ability of IRM ( Arjovsky et al. , 2020 ) to debias NLI models and improve their o.o.d generalization . IRM attempts to base its prediction on features whose correlation with the target variable is invariant rather than environment-specific . This is translated to finding a data representation such that the optimal classifier on top of it is the same across environments : min Φ : X→H w : H→Y ∑ e∈Etr Re ( w ◦ Φ ) s.t . w ∈ arg min w′ : H→Y Re ( w′ ◦ Φ ) ∀e ∈ Etr . ( 1 ) where Etr is the set of the training environments , Re is the risk of environment e , w is the classifier in the hypothesis spaceH , and Φ is the representation function . This optimization problem is relaxed into a regularized objective function called IRMv1 , and to which we simply refer as IRM : min Φ : X→Y ∑ e∈Etr Re ( Φ ) + λ · ‖∇w|w=1.0R e ( w · Φ ) ‖2 , ( 2 ) where the hypothesis class is now limited to linear classifiers , and our goal is to find a representation function Φ for which the classifier is optimal in all environments . The first term , referred to as the ERM term , promotes low average error over all environments , while the second term , referred to as the IRM term , promotes invariance across environments . The constraint from Eq . 1 is relaxed into a Lagrangian form that penalizes the gradient norm of each environment . The regularizer weight λ controls the trade-off between empirical risk and invariance . 3.2 Problem Setup and Experimental Details . Terminology and notation Let Dtr = { ( xi , yi ) } Ni=1 denote a training dataset of N premise– hypothesis samples x = ( P , H ) with labels y ∈ { entailment , neutral , contradiction } , and an associated dataset bias . A bias is a correlation between some feature of a sample x = ( P , H ) and a label y , that is not guaranteed to hold outside the dataset . We call such a feature a biased feature and denote it as B ( x ) . For instance , B ( x ) might be word overlap in the case of overlap bias or negation words in the case of hypothesis bias . We call a sample xi with label yi bias aligned if p ( y = yi | B ( xi ) ) in the training set is high . Similarly , it is bias misaligned if p ( y = yi | B ( xi ) ) in the training set is low . Finally , unbiased samples , as opposed to biased samples , are such for which the feature B ( x ) is not correlated with any of the labels . Given an environment Etre ( a subset of Dtr ) , we refer to the conditional probability of a label given the biased feature , p ( y = y′ | B ( x ) ) , as the bias strength in the environment and denote it as pe . A related but different quantity is the ratio of biased samples out of all samples in the environment , which we call the bias prevalence and denote as αe . Environment generation The goal of IRM is to find a representation function Φ of the premise– hypothesis pair , for which the classifier is optimal in all environments . In our case , we generate environments by varying the bias strength , i.e. , changing the conditional probability of the label given the biased feature . In the toy and the synthetic bias experiments the bias is synthetically injected into the samples , so generating environments is straightforward . For a bias feature B ( x ) and a label y′ , to create an environment e with p ( y = y′ | B ( x ) ) = pe , we associate the feature with the label with probability pe . This fully controllable setting no longer holds when we proceed to the natural bias case , where the bias can not be decoupled from the samples . In order to generate environments that represent a varying bias strength for some B ( x ) in the natural bias experiments , we first need to quantify p ( y = y′ | B ( x ) ) . We train a model fθ with B ( x ) as its only input , referred to as a biased model . For instance , this might be a model with access only to the hypothesis , in the case of hypothesis bias . The biased model outputs a probability distribution over the labels fθ ( B ( x ) ) ∈ R|Y| , termed a score vector . The model prediction ŷi ( the label receiving the highest probability ) and the score vector are used to assign the sample to one of three subsets—Dun , B ( unbiased ) , Dalign , B ( bias aligned ) or Dmisalign , B ( bias misaligned ) —as follows . First , we categorize samples to unbiased and biased . A sample is considered unbiased if the total variation distance of fθ ( B ) from the uniform distribution is lower than some threshold t1 . Otherwise , it is considered biased . Next , the subsets Dalign , B and Dmisalign , B are constructed from the biased samples such that Dalign , B = { xi | ŷi = yi } and Dmisalign , B = { xi | ŷi 6= yi } . Lastly , to ensure that the biased feature is highly indicative of the label , we filter the subsets Dalign , B , Dmisalign , B by eliminating examples where the top two predictions are very close . Denote by ỹi the second highly predicted label for the i-th example . The resulting sets are Dalign , B = { xi | ŷi = yi ∧ ŷi − ỹi > t2 } , Dmisalign , B = { xi | ŷi 6= yi ∧ ŷi − ỹi > t2 } for some threshold t2 . To generate an environment e with bias strength pe and bias prevalence αe , we sample ( m , n , k ) examples from the subsets ( Dalign , B , Dmisalign , B , Dun , B ) such that pe = mm+n and αe = m+n m+n+k . 2 This process will be used to generate environments with pre-defined bias strength in Section 4 and 2We first choose m and n as large as possible such that required strength holds and resulting environments are of equal size . Then we add the k samples as required by the prevalence . pre-defined bias strength and prevalence in the natural bias experiments in Section 5 . Models are trained on environments Etr generated from Dtr and evaluated on test environments generated from splits of an appropriate test set . First , Eid , B is a test environment with αe = 1.0 and pe similar to what was set in the training environments . Thus , it represents a similar distribution to Etr w.r.t the bias B , and considered in-distribution . Second , Eood , B is a test environment with αe = 1.0 and pe reversed to what was set in the training environments ( i.e. , if pe was high in Etr , it will be low in Eood , B ) . Appropriately , Eood , B represents a ( often significantly ) different distribution w.r.t the bias , and considered o.o.d . Third , Eun , B is a test set with unbiased samples , i.e. , αe = 0.0 . For brevity , we will omit B when it is clear from context . Training details All the experiments feature two training environments , both of the same size , on which the models are trained . In the case of ERM , where we do not have an additional regularization term ( λ = 0 ) , this is equivalent to training on a mixture of data from the two environments . Table 6 in Appendix A describes the environment specification for each experiment in details . According to the practical instantiation of IRM ( Eq . 2 ) , the classifier in all the experiments is a “ dummy ” classifier ( w = 1.0 or a vector of ones for multi-class classification ) , and the representation function Φ is the model used in the experiment . For more experimental details refer to Appendix A . Our code and data can be found at https : //github.com/technion-cs-nlp/irm-for-nli .

Summary: The paper presents an empirical study of IRM on NLI focusing on hypothesis and overlap bias. To do this, the paper defines the notion of bias "strength" p_e (probability of the bias label, given that the instance is indeed biased) and "prevalence" alpha_e (proportion of biased samples) for environment e. In practice this is controlled by training a biased predictor that only uses the (known) bias of the input to predict the label (e.g., for hypothesis bias we can finetune BERT on hypotheses only). Training environments correspond to different values of p_e and alpha_e (e.g., (0.7, 0.82) and (0.9, 0.82)) which is necessary for IRM training. The model is then tested on in-domain/bias aligned (e.g., alpha=1 and p=1), out-of-domain/bias misaligned (alpha=1 and p=0), and unbiased (alpha=0) data. The main finding of the paper is that: (1) IRM can outperform ERM even on natural data if the test set is bias misaligned, (2) the performance of IRM is heavily influenced by p, alpha, and also data size in training.