IRM—when it works and when it doesn't: A test case of natural language inference
Authors: Yana Dranker
He He
Yonatan Belinkov
Venue: NeurIPS 2021 Poster (2021)
Keywords: Causality
Natural Language Processing
Robustness
Natural Language Inference
Out-of-Distribution Generalization
Bias
Debiasing

Abstract: Invariant Risk Minimization (IRM) is a recently proposed framework for out-of-distribution (o.o.d) generalization.  Most of the studies on IRM so far have focused on theoretical results, toy problems, and simple models. In this work, we investigate the applicability of IRM to bias mitigation-a special case of o.o.d generalization-in increasingly naturalistic settings and deep models. Using natural language inference (NLI) as a test case, we start with a setting where both the dataset and the bias are synthetic, continue with a natural dataset and synthetic bias, and end with a fully realistic setting with natural datasets and bias. Our results show that in naturalistic settings, learning complex features in place of the bias proves to be difficult, leading to a rather small improvement over empirical risk minimization. Moreover, we find that in addition to being sensitive to random seeds, the performance of IRM also depends on several critical factors, notably dataset size, bias prevalence, and bias strength, thus limiting IRM's advantage in practical scenarios. Our results  highlight key challenges in applying IRM to real-world scenarios, calling for a more naturalistic characterization of  the problem setup for o.o.d generalization.

IRM-when it works and when it doesn't: A test case of natural language inference
Yana Dranker' yanadr@campus technion ac. il
He He²
Yonatan Belinkoy bel ielinko@ttechion. ac i1
hhe@nyu. edu
Technion Israel Institute of Technology
New York University
Abstract
Invariant Risk Minimization (IRM) is a recently proposed framework for out- of-distribution (o.o.d) generalization. Most of the studies on IRM so far have focused on theoretical results, toy problems, and simple models. In this work, we investigate the applicability of IRM to bias mitigation a special case of o.o.d generalization- increasingly naturalistic settings and deep models. Using natural language inference (NLI) as test case, we start with setting where both the dataset and the bias are synthetic, continue with natural dataset and synthetic bias, and end with a fully realistic setting with natural datasets and bias. Our results show that in naturalistic settings, learning complex features in place of the bias proves to be difficult, leading to a rather small improvement over empirical risk minimization. Moreover, find that in addition to being sensitive random seeds, the performance of IRM also depends on several critical factors, notably dataset size, bias prevalence, and bias strength, thus limiting IRM's advantage in practical scenarios. Our results highlight key challenges in applying IRM to real-world scenarios, calling for more naturalistic characterization of the problem setup for o.o.d generalization.
1 Introduction
Deep learning models show strong performance when tested on data from the same distribution
they were trained on, matching or even surpassing humans (Zhang al., 2017 Brinker et al., (019)). However, this performance often stems from relying on spurious correlations rather than human -ike reasoning, causing these models to "break" in real world scenarios (Geirhos et al.) 2018 McCoy al. 2019). recent method called Invariant Risk Minimization (IRM; Arjovsky et al. 2020) aims to learn causal features whose correlation with the label is invariant across different distributions, thus leading to better nutoof-distributinn (o.o.d) generalization. IRM splits the training data into different subsets, or environments, across which varying spurious correlations cause distribution shifts. The goal is to learn a representation that yields the same optimal classifier for all environments.
Although several studies (Choe et al. 2020 Ahuja et al. 2020a; Kamath et al. 2021) have investigated IRM and its variants, the empirical results so far have mainly focused on synthetic settings and simpler models (e.g., shallow multilayer-perceppron
bag-of-worde models). This motivates us to investigate IRM natural settings with complex models. We take natural language inference (NLI) as a test case, where the model needs to predict if a hypothesis sentence is entailed by the premise sentence. While theoretically requiring deep language understanding. many NI.I datasets contain biases, or spurious correlations between superficial features and labels (Gururangan et al.) 2018 Poliak et al. 2018 Tsuchiya 2018), such as word overlap correlating with the entailment label. Such heuristics allow models perform superficially well on the benchmark, but fail catastrophically when these heuristics
*Supported by the Viterbi Fellowship the Center for Computer Engineering at the Technion.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

no longer hold Naik et al., 2018 Glockner et al., 2018 McCoy et al., 2019). Common debiasing methods typically rely on explicit modeling of the blases (Belinkov al. 2019a Karimi Mahabadi al. 2020; He et al. 2019). In contrast, IRM offers interesting possibilities. is model agnostic, not built for specific bias and, given appropriate environments, generalizes to distributions where the spurious correlation varies.
In this work, we design a series of debiasing experiments, intended bridge the gap between fully synthetic and naturalistic scenarios. Table shows examples of each setting. We begin with a toy experiment where both the dataset and the bias are synthetic. Next, we add a level of complexity by injecting synthetic bias to a real NLI dataset and use state-of-the- investigate natural NLI datasets with known dataset biases. For the last setting, splitting environments using categorical biased features as done in previous work is not applicable. Therefore, for real- scenarios where the bias is known but might be a high dimensional feature, we develop a simple way to split data into different environments, described in "Environment generation" in Section 3.2 In each of these settings, we train models with empirical risk minimization (ERM) and IRM and evaluate them o.o.d test sets. Our experiments yield the following results:
pre- trained models. Last, we
In the toy setting, performance follows the theory: ERM performs well on the training set but fails completely on the o.o.d test set, while IRM ignores the bias and thus performs slightly worse on the training set, but perfectly on the o.o.d test set.
On natural NLI datasets and deep models with either synthetic bias or natural bias IRM outperforms ERM when evaluated on o.o.d test sets.
However, in these more naturalistic settings, IRM is not able to completely discard the bias, while ERM does not rely solely on the bias. Thus, in practice the advantage of IRM is small.
When used with sfate-of-the-ar models, IRM shows large variance in performance across random seeds.
To better characterize when IRM works better or worse than ERM, we highlight three criteria that govern its success: the prevalence of the bias in the training set (how many examples are biased), the strength of the bias (how strong is the correlation between a label and a biased feature), and the training data size. While bias strength and data size were not empirically investigated in real-world scenarios, bias prevalence was completely overlooked in previous work. We find that when all three criteria are met (prevalent and strong bias, large training set). IRM tends to perform better than ERM. When bias prevalence, strength, or training data size are limited, IRM is less stable and ERM performs better.
To conclude, our investigation shows important challenges in assessing the performance of IRM in plausible real-world scenarios, pointing to the need to employ a more naturalistic approach to characterizing the settings in which IRM can perform well.
2 Related Work
IRM IRM was motivated by Peters et al. (2016), who connect invariance with causality by estab- lishing a link between the cause of target variable and the invariant predictor. As a proof-of-concept, Arjovsky al.] experimented only with very simple settings, namely MNIST with varying back- ground colors. Several studies further investigated IRM or proposed variants of IRM. |Rosenfeld al. 2020) show that in the non-linear setting, there exists a nearly optimal classifier which uses non-invariant features, thus the classifier found by IRM may not generalize to o.o.d. data. However, their result assumes infinite samples and we instead focus on an empirical study in the finite sample case. Kamath et al. (2021) point out the discrepancies between the original IRM formulation and various instantiations of Since in non-toy examples such comparison is non-trivial, we focus on IRMv1, the practical formulation suggested in the original paper. Both Ahuja et al. (2000a) and Krueger et al. (202T) propose new objectives and show superior stability, and improvement over IRM when adding covariate shift, respectively, on colored MNIST. Gulrajani and Lopez-Paz (2020) devise a test suite to fairly compare algorithms for domain generalization, and show that with careful tuning and model selection, ERM outperforms all other algorithms. Choe et al. (2020) show that IRM outperforms ERM in a sentiment analysis task with synthetic bias, but restrict themselves to a bag-of- words model.

---

Table 1: Example NLI samples for proposed settings. In the training set, biased features are correlated with the label (same color). In the test set, the correlation does not hold (different colors).
Data
Bias
Training example
Test example
Toy Experiment
y: non-entailment
y: entailment
Synthetic Synthetic B: has "c"
B: has "c"
P: H: bc
P: a
H: ac
Synthetic Bias Experiment
y: contradiction
entailment
B: has "<c>"
B: has "<c>"
Natural Synthetic
P: A man resting street.
P: A car sinking in water.
H: <c> A man jogging on the street.
H: <c> A car is flooding.
Natural Bias Experiment (overlap bias)
entailment
y. non-entailment
B: has word overlap
B: has word overlap
Natural Natural
P: It is most natural.
P: Don't be cynic.
H: It is natural.
H: Be a cynic.
Natural Bias Experiment (hypothesis bias)
y: contradiction
y: entailment
B: has negation
B: has negation
Natural Natural
P: Three bikers stop in town.
P: A barefoot woman reads book [...]
H: The bikers didn't stop in the
H: A woman isn't wearing any shoes.
town.
While different approaches were suggested, most are evaluated in simple settings. Although Guirajani and Lopez-Pat Paz targeted more naturalistic scenarios, they focused only on image classification. In addition to focusing on a language task, we show performance gains of IRM over ERM in naturalistic settings with complex models, and perform an analysis of the factors that affect IRM's performance.
Bias in NLI NLI is widely studied task in natural language processing, concerned with identifying the relation between a premise and a hypothesis. Recent studies (Gururangan et al. 20188] Poliak ett al. 20188 McCoo et al. 2019) reveal biases in NLI datasets such SNLI (Bowman al. 2015)
and MNLI (Williams et al. 2018). Two widely observed biases are hypothesis bias and overlap bias. Overlap bias is characterized by high word overlap between the premise and the hypothesis, shown to be correlated with entailment. Hypothesis bias refers to patterns in the hypothesis correlated with a specific label, allowing for correct prediction without considering the premise; common example is negation words correlated with the contradiction label (Table I). Consequently, several challenge sets (Naik al. 2018 Glockner et al. 2018 McCoy et al.) 2019) demonstrated performance degradation when the bias no longer holds. We evaluate the ability of IRM to mitigate reliance on bias and thus improve performance in such cases.
Method
3.1
Background on IRM
Our goal is to investigate the ability of IRM (Arjovsky et al. 2020) to debias NLJ models and improve their o.o.d generalization. IRM attempts to base its prediction on features whose correlation with This is translated to finding data
the target variable is invariant rather than euvironment-specife representation such that the optimal classifier on top of it the same across environments:
min
Σ eEE.
Rd ))
E arg min R R o b) Ve Etr. w w'
(1)

---

where Eir is the set of the training environments, R is the risk of environment e, w is the classifier in the hypothesis space H. and Φ is the representation function. This optimization problem is relaxed into a regularized objective function called IRMv1, and to which we simply refer as IRM:
min Σ O.
(2) + Vi (2--0wW0) -)];,
(2)
BEE
where the hypothesis class is now limited to linear classifiers, and our goal is find a representation function & for which the classifier is optimal in all environments. The first term, referred to as the ERM term, promotes low average error over all environments, while the second term, referred as the IRM term, promotes invariance across environments. The constraint from Eq. i relaxed into a Lagrangian form that penalizes the gradient norm of each environment. The regularizer weight controls the trade-off between empirical risk and invariance.
3.2 Problem Setup and Experimental Details
Terminology and notation Let Drr- = { y:) denote a training dataset of N premise- hypothesis samples (P, H) with labels y (entailment, neutral, contradiction) ), and associ- ated dataset bias. A bias is a correlation between some feature of a sample x = (P, H) and a label y. that is not guaranteed to hold outside the dataset. We call such feature biased feature and denote it as B(x). For instance, ((x) might be word overlap in the case of overlap bias or negation words in the case of hypothesis bias. We call sample x; with label y; bias aligned if p(y - y; B(xx)) in the training set is high. Similarly, it is bias misaligned if p(y = ys B(x:)) in the training set is low. Finally, unbiased samples, as opposed to biased samples, are such for which the feature Bxx) is not correlated with any of the labels. Given an environment Eir. (a subset of Day),, we refer to the conditional probability of a label given the biased feature, p(y = y B(z)), as the bias strength in the environment and denote it as Pe. A related but different quantity is the ratio of biased samples out of all samples in the environment, which we call the bias prevalence and denote as Oa.
Environment generation The goal of IRM is to find a representation function & of the premise- hypothesis pair, for which the classifier is optimal in all environments. In our case, we generate environments by varying the bias strength, i.e., changing the conditional probability of the label given the biased feature. In the toy and the synthetic bias experiments the bias is synthetically injected into the samples, so generating environments is straightforward. For bias feature B(x:) and label y', to create an environment e with p(y = v/ B(x) Pe- we associate the feature with the label with probability Pe- This fully controllable setting no longer holds when we proceed the natural bias case, where the bias cannot be decoupled from the samples.
In order to generate environments that represent a varying bias strength for some B(x) in the natural bias experiments, we first need to quantify p(y = s/ B(x)). We train a model fe with (xx) as its only input, referred to hypothesis, in the case of hypothesis bias. The biased model outputs a probability distribution over the labels fa(B(x)) E R 2ll termed a score vector. The model prediction d (the label receiving the highest probability) and the score vector are used to assign the sample to one of three subsets-D.... (unbiased), D Palign,B (bias aligned) or D Dmiisalgnn.BB (bias misaligned) as follows.
biased model. For instance, this might be a model with access only to the
First, we categorize samples to unbiased and biased. A sample is considered umbiased if the total variation distance of (B) from the uniform distribution is lower than some threshold &1. Otherwise, it is considered biased. Next, the subsets D Datign. and Dmisaatgn, are constructed from the biased samples such that Datign.B = {xr 0J = yi and D Dinisatign, = {#; 0i # y:). Lastly, to ensure that the biased feature is highly indicative of the label, we filter the subsets Datign.B. D Pmisatign, B by eliminating examples where the top two predictions are very close. Denote by D: the second highly predicted label for the i-th example. The resulting sets are Datign, B {z; | 9: We A Di V/ 12).
D Dmisaatign B = {I: ÿi 7 y/ A 1/- Th > (2) for some threshold 12.
To generate an environment e with bias strength Pe and bias prevalence agg we sample (m. n2, A) m examples from the subsets (Daisgn.B. Pmssaltgn, B, Dun, Bn. such that P. m-i and ag m + This process will be used generate environments with pre-defined bias strength in Section 4 and
We first choose and as large as possible such that required strength holds and resulting environments are of equal size. Then add the k samples required by the prevalence.

---

pre-defined bias strength and prevalence in the natural bias experiments in Section Models are trained on environments Err generated from Drr and evaluated on test environments generated from splits of an appropriate test set. First, &id, is test environment with 0r, = 1.0 and Pe similar to what was set in the training environments. Thus, it represents a similar distribution to &tr w.r.t the bias B, and considered in-distribution. Second, &ood, B is test environment with ag = 1.0 and Pe reversed to what was set in the training environments (i.e., if pe was high in E it will be low in Eood,B) B). Appropriately, B represents (often significantly) different distribution w.r.t the bias, and considered o.o.d. Third, Eun,B B is a test set with unbiased samples, i.c., a = 0.0. For brevity, we will omit B when it is clear from context.
Training details All the experiments feature two training environments, both of the same size, on which the models are trained. In the case of ERM, where we do not have an additional regularization term (A 0), this is equivalent training on a mixture of data from the two environments. Table in Appendix A describes the environment specification for each experiment in details. According to the practical instantiation of IRM (Eq. 2 the classifier in the experiments is "dummy" classifier (w = 1.0 or vector of ones for multi-class classification), and the representation function & is the model used in the experiment. For more experimental details refer to Appendix [ Our code and data
can be found at htthtts:// //gs:://iitie //ggtttt........./ www. conn/technion. cs- ass-nlp/ie irm- for-nil
4 Results
4.1 Toy Experiment
4.1.1 Setup
We design a XOR example as simplified NLI task, and use it simulate hypothesis bias. To construct our synthetic dataset, we adapt the example from Belinkov et al. (0119)). where the premise and hypothesis are each one character long taken from the set {a, 6j. The premise entails the hypothesis (y = 1) if their first character is the same. These samples are randomly partitioned to two disjoint subsets. from which we build environments Etr, (e E {1, 2}) as follows: first add noise to the label (by flipping it with probability ne). then append "c' and "d" to the hypothesis of non-entailment and entailment examples with high probability Pa. respectively, such that the model can take the shortcut to predict by the appended character. By setting pi 0.8, P2 = 0.9, we construct two training environments with a strong yet varying correlation between the label and the bias. This correlation is flipped at test time, by setting Pe- = 0.0 and ye = 0.0. Note that noise is needed when training to make the correlation of the label with the biased feature stronger than its correlation with the causal feature (i.e. equality of the first character) (1 Te Pe). such that ERM will rely on the biased feature. To summarize, we train both ERM and IRM on two training environments EirL, Eiry with 01 02 1.0. p1 = 0.8, and p2 0.9. The models are then evaluated on a test environment Eood with a1 = 02 = 1.0 and p. = 0.0. We use the same model described in Belinkov et al. (2019b) a multi-layered perceptron (MLP) on character embeddings trained with cross-entropy loss. For a detailed description of the model and
see Appendix B
4.1.2 Results
Theoretically, relying on the hypothesis bias (the appended character) to predict entailment should result in approximately P1 85% accuracy on the combined training set (since the classes are balanced), and 0% accuracy on the test set. Relying on the unbiased feature, on the other hand, should result approximately the test set. As 2 shows, our results confirm these expectations. ERM relies on the appended character to predict the label, thus failing completely on the test set. IRM manages to identify the environment-specine correlation and relies on the causal feature, achieving 100% test accuracy. An analysis of the training dynamics of ERM and IRM (Appendix B shows that IRM starts shifting the weight from the biased feature to the causal feature when the constrained phase (high A) begins.
7) 75% accuracy on the training set, and 100% accuracy on
5

---

Table 2: Accuracy in the toy experiment.
Table 3: Accuracy in the synthetic bias experiment.
Train
Test
Ps = 0.8
Pe = 0.33
Ps = 0.0
ERM 85.304 + 0.4
0.0 = 0.0
REF 89.97 + 0.34
89.33 = 0.28
88.93 + 0.55
IRM 75.384+0.69
100.0 + 0.0
ERM
93.49 0.28
85.16 = 0.9
79.16 1.48
IRM 92.32 0.3
87.22 0.45
83.5 = 0.71
4.2 Synthetic Bias Experiment
4.2.1
Setup
In this section we add the complexity of a real world dataset (SNLI) and use a state-of-the-art model. We are interested in how well IRM works with high capacity model, previously shown to rely on bias (McCoy et al., 2019 Poliak et al. 2018). This experiment is inspired by similar experiment suggested in He et al. (2019) and designed to simulate hypothesis bias. We use three bias tokens (<c>, <e>, and <n>), ) correlated with the three labels in SNLI (contradiction, entailment, and neutral, respectively). We prepend the bias token corresponding to the gold label with probability Pe, and randomly sample one of the other two tokens otherwise. We use pre-trained BERT (base- unccaedd as our model, with most of the hyper-parameters as recommended in the original work (Devlin et al.. 2019). For further training details, see Appendix C.I
We train our models on two training environments {E+r. } (e E (1,2]) with dn 02 1.0, pi 0.7 and p2 = 0.9. We additionally include a REF model trained on the original SNLI training set with no bias injected (i.e.. Etr, with a, 0.0), thus considered an unbiased model (w.r.t the synthetic bias). The training environments are of equal size and comprise together the entire SNLI training/validation set. The models are evaluated on three environments generated from the SNLI test Set-E,a with Pe 21+ 0.8, Eood with Pe 0.33 and Eood with p. 0.0, where the latter comes down to bias misaligned set. All test environments are set with a = 1.0.
4.2.2 Results
The results in Table showw that IRM outperforms ERM on o.o.d test environments and degrades on the in-distribution test environment, indicating that it successfully reduces reliance on the biased feature. However, as the test environment moves further away from the training environments (i.e., more bias misaligned samples decreasing Pe), both ERM and IRM display lower accuracy, revealing that IRM is not able to completely ignore the bias when the test environments are sufficiently different. As expected, the unbiased model (REF) performs similarly across all environments. This performance can serve as both a lower bound for the in-distribution degradation and an upper bound for the o.o.d improvement, and indeed we can see that both ERM and IRM performance are higher than REF on Pe = 0.8 and lower on Pe 0.33, 0.0.
Relying solely the biased feature should have resulted in performance close to chance (33%) on the Pe = 0.33 environment, but both ERM and IRM perform significantly better. Comparing the accuracy of an unbiased model (REF model) on test environment Eux, B (~90%) with the expected accuracy of a model relying strictly on the biased feature to predict (~ P1 80%), suggests that unbiased features are more correlated with the label than the biased feature, discouraging ERM from heavily relying on the biased feature. Combined with IRM's inability to completely discard the bias, this leads to IRM showing rather small performance gains over ERM. Finally, in Appendix C.3 we also show the results when injecting the entire test set with the same bias token. This experiment reveals that ERM's performance is more strongly degraded by the neutral bias token, suggesting that biases in the data may be treated differently. contrast, IRM's performance is similar given any of the three bias tokens.
2
'We have experimented with several sets of bias tokens and witnessed similar behavior. Further details in Appendix C22

---

Table 4: Accuracy on SNLI test environments for hypothesis bias.
Table 5: Fl macro on MNLI dev mismatched test environments for overlap bias.
Bias aligned
Bias misaligned
Bias aligned
Bias misaligned
Unbiased
Unbiased
ERM
84.46
97.4
62.63
ERM
85.23
96.67
62.66
(#0.64)
(+0.26)
(+1.19)
(+0.69)
(+0.28)
(+1.95)
IRM
82.64
91.4
65.12
IRM
83.75
95.44
64.12
(+13333
(+22.22)
(+2.28)
(+0.46)
(+1.1)
(+3.86)
4.3
Natural Bias Experiment
4.3.1
Setup
In this experiment our goal is to create environments that vary by known dataset bias. We target two widely observed biases-hypoothess bias and overlap bias (Section -2 two large NLI datasets, SNLI and MNLI. Following previous work (He al. 2019; Karimi Mahabadi al. 2020), we use a BERT model with hypothesis as its only input as a biased model for hypothesis bias and a shallow MLP on top of maanally--essgnedd features as a biased model for overlap bias (Appendix D.2p. Since the overlap feature is not discriminative between neutral and contradiction classes (McCoy al., 2019 Karimi Mahabadi et al.) 2020), we combine neutral and contradiction into a single non-entailment class. As this results in an unbalanced dataset, all models are trained with loss weighted according to class size and we report macro F1 scores instead of accuracy. For further training details, see Appendix [D.]]
For both hypothesis and overlap bias we train our models on two training environments {Er, (e E (1, 2 with pl 0.7 and p2 = 0.9. Similarly to the approach in previous work (Arjovsky et al. 2020 Krueger et al. 202T Choe al. 2020), we only control the bias strength of the environments. The bias prevalence is not fixed, and the resulting average prevalence of the two environments is similar the bias prevalence found in the original dataset without any interventions. For hypothesis bias we have 21 = 0.82 and for overlap bias to
2 = 0.52. We evaluate our models on Edd with a₂ = Pe 1.0 (bias aligned subset) and
in distribution and o.o.d test environments Eood with ae = 1.0 and Pe 0.0 (bias misaligned subset). We also report results on Eun with oe = 0.0, i.e., the unbiased subset. Size of training/validation data is 3305k/ ~5k and ~245k/~6k for hypothesis and overlap bias, respectively.
4.3.2 Results
Hypothesis bias We report results for hypothesis bias on SNLI test environments in Table The performance gap between the environments is significant, indicating that models strongly rely on biased features. Similarly to the observation in the synthetic bias case, IRM improves performance on the bias misaligned subset and degrades on the bias aligned subset compared to ERM, proving to be more robust. Although IRM displays some degradation on the unbiased subset, it is significantly the bias aligned subset. We also note that performance varies more on the bias misaligned subset (larger std from different random sceds), which is consistent with results on o.o.d evaluation (McCoy al.) 2020).
smaller than the
The relatively poor performance (of both IRM and ERM) on the bias misaligned subset may be explained by an inherent difficulty of the samples in this subset. Indeed, we found that the inter- annotator agreement in this subset is much lower compared to the other subsets (Appendix(D2) (D3).. This suggests that the causal features are noisier in the bias misaligned subset and that the upper bound on performance is lower than the two other sets.
Overlap bias Table 5 shows the results for overlap bias MNLI dev mismatched environments Performance discrepancies are as expected, with highest performance on bias aligned subset and
"Since the MNLI test set labels are hidden, and are needed for splitting according to bias, we use the provided dev matched set as validation set, and test models the dev mismatched set.
7

---

lowest on bias misaligned subset. IRM decreases this performance gap between the subsets: It outperforms ERM on the bias misaligned subset, and slightly degrades on the other two. As before, performance varies more on the bias misaligned subset, especially for IRM.
To further understand IRM's improvement on the bias misaligned subset we analyzed the performance per class in Appendix D4 We found that IRM mitigates the bias for the non-entailment class (i.e., improves on no-overlap entailment examples) suggesting that IRM's improvement on the bias misaligned subset comes from mitigating the non-overlap bias; surprisingly, it slightly amplifies bias for the entailment class (i.e., degrades on high overlap non-entailment examples). This suggests that different biases in the data might be unevenly mitigated by IRM.
Evaluation on o.o.d benchmarks We evaluate our models trained in the overlap bias setting and the hypothesis bias setting on HANS (McCoy et al., 2019) and RevisedPremises (RP; Kaushik et al.) (2019) respectively, and report results in Appendix D.5 On RP test set, even though our models are trained on smaller data size, we achieve comparable results to those reported by Kaushik et al. (2019). On HANS, our results are not comparable to those presented in McCoy et al. (2019) for several reasons- the size of the training data smaller our case (due to constraints on environment generation), the bias is stronger by construction, and the labels are collapsed to 2 classes before training. On both benchmarks IRM is not able to outperform ERM, and performs equivalently (on RP) worse (on HANS). We hypothesize that this is due to differences in biases between the o.o.d subsets we construct and the challenge benchmarks. This shows that while IRM might improve on biases targeted by the environments it trained on, it is not able extrapolate to different types of biases.
5 Analysis
In this section, we empirically analyze three factors that influence IRM's performance: bias prevalence, strength, and training data size. In each case, we vary one factor while keeping the other two fixed. Where possible, we use the default values from the main experiments in the previous section. However, to achieve the required bias prevalence and strength in the natural bias setting, we had to reduce the size of training and validation data significantly. As we shall see, the size of the training data is in itself a factor affecting IRM's performance, so this might influence the stability of the results. In addition to the previously observed sensitivity of IRM to random seeds (Ahuja et al.) (2020a), which causes high variation (as shown in the previous section), we also witness another type of failure case with the restricted environments generated for the natural bias analysis. Some of the runs for IRM seem to converge to null representation, displaying near-chance performance on both training and validation. The ERM term (Eq. 2 should have prevented this scenario, however tuning À to mitigate this behavior is an open issue. We therefore report results for all random seeds when necessary.
5.1 Bias Prevalence
To analyze the effect of bias prevalence on IRM, we vary the fraction of biased samples in the training set, while fixing the bias strength (F1 = 0.7, P2 0.9) and the size of training/validation data (>134k/~~2.6 for hypothesis bias, ~87k/~2.33 for overlap bias, and ~5499k/99.. 8k, i.e., the entire SNLI train/val set, for synthetic bias). We evaluate on Eood with a = 1.0 and p. = 0.0 (i.e., the bias misaligned set) for all three settings. [
that as the bias prevalence increases, IRM's performance o.o.d data improves. In the synthetic bias case, IRM surpasses ERM at high bias prevalence. In the natural bias cases, IRM has several failed runs (very low accuracy), probably due to the reduced training set size; however, those that succeeded, surpass ERM. A possible explanation is that as more samples are either bias aligned or bias misaligned (i.e. contain a biased feature), we get more signal in the IRM penalty term indicating the instability of the biased feature across environments, making it casier for IRM to detect and discard it.
5.2 Bias Strength
Similarly to the previous experiment, when analyzing the effect of bias strength we keep the bias
prevalence fixed, (ox = 22 = 1.0 for synthetic bias, a1 02 = 0.82 for hypothesis bias, and
0n = 0g = 0.52 for overlap bias). Size of training/validation data is ~152k//~2.3 for hypothesis bias, ~145k/~3.8 8k for overlap bias, and ~549k~99.88 (the entire SNLI train/val set) for synthetic
8

---

88.0
70.0
ERM
04.0
60.0
F1 macrro
02.0
Accura
55.0
80.0
50.0
76.0
ERM
45.0
55.0
74.0
IRM
IRM
50.0
35.0
0.2
1.0
0.6
0.7
0.8
0.9
blas prevalence
bias prevalence
blas prevalence
synthetic bias.
Hypothesis bias.
Overlap bias.
Figure 1: Performance of ERM and IRM on Food with Pe = 0.0, a 1.0 (bias misaligned subset) when varying bias prevalence (holding size of data and bias strength fixed). Test subsets are generated from SNLI test for synthetic and hypothesis bias, and from MNLI dev mismatched for overlap bias.
bias. As before, we evaluate on good with a = 1.0 and Pe 0.0 for synthetic bias, hypothesis bias, and overlap bias. Figure 2a shows results for the synthetic bias setting, indicating that as bias strength increases ERM degrades while IRM (roughly) maintains its performance. Figures 2b 2c show results for both natural biases. As bias becomes stronger, ERM degrades on o.o.d, while most of the IRM runs significantly outperform ERM. Still, some of the IRM runs crash and perform poorly. We notice that while IRM seems to improve with increased prevalence, it degrades bias strength increases, but just not as much as ERM. We hypothesize that while IRM more robust to bias than ERM, is not able to completely discard it, even in synthetic settings (see Section 4.2.2) let alone natural, more complex settings. Therefore, as bias is stronger it picked up more by both methods and only partially discarded by IRM.
70.0
75.0
85.0
70.0
65.0
Accuracy
Accuracy
65.0
60.0
60.0
75.0
55.0
55.0
70.0
ERM
ERM
50.0
ERM
IRM
50.0
IRM
10.5, 0.7)
10.6, 0a)
10.7. 0.9
10.8. 1.0)
10.6. 0.8)
(e (0.7. 0.9)
10.8. 1.0)
0.7)
10.6. 0.3)
10.7. 0.9)
10.5.
10.8
A)
( g
(a) Synthetic bias.
Hypothesis bias.
Overlap bias.
Figure 2: Performance of ERM and IRM on Eood with Pe = 0.0. 1.0 (bias misaligned subset)
when varying bias strengths (holding size of data and bias prevalence fixed). Test subsets are generated from SNLI test for synthetic and hypothesis bias, and from MNLI dev mismatched for overlap bias.
5.3
Data Size
To isolate the effect of data size on performance, we keep the bias prevalence and strength fixed and equal to the experiments in Section 4.3 (a) 02 1.0, 0.82, 0.52 for synthetic bias, hypothesis bias, and overlap bias, respectively, P1 0.7, P2 = 0.9 for all settings), and only change the size of the environments. We evaluate synthetic bias, Figure Ba shows that IRM performs on par or better than ERM on o.o.d data for all sizes, and loses stability as data size decreases. The gap between ERM's and IRM's performance seems to decrease as data size increases, possibly due to the simplicity of the synthetic bias, and the high correlation between true signal and label, discussed in Section 4.2
Food with Or 1.0 and p. 0.0 for all three settings. For
For both natural biases, IRM is outperformed by ERM in small data regimes (Figures 3b and 3c
For hypothesis bias, IRM outperforms ERM on o.o.d data on several settings, while for overlap bias only in the largest regime is IRM able to outperform ERM. In Appendix D.2 we show that the biased model used for overlap bias generates less confident predictions, compared to that of the hypothesis bias. This might suggest that the overlap bias is less discriminative and negatively affects the quality of environments. This also reflected the lower prevalence the overlap bias setting (0.52) compared to the hypothesis bias setting (0.82). Finally, the natural bias results fall in line with the

---

observation by Ahuja et al. (20206) that, for linear and polynomial models, IRM, contrary to ERM, approaches the desired o.o.d solution in the finite sample regime as data size increases.
80.0
65.0
ERM
65.0
ERM
Accuracy
70.0
Accuracy
60.0
60.0
60.0
55.0
50.0
55.0
ERM
40.0
50.0
(25k: 9.8k)
(Sok, add
(200k, auj
(200k; 8.8k)
(aook, add
(400k 9.8k)
(500k. 3.8k)
(25k: 4ld
(50k. 48)
(200k. 4k)
(250kk 4kg
(200k. 48d
1250k, 48)
(25k. 4.5k1
(50k. 4.5k)
(200k 4.5k
(150k. 4.5k
Dataset size
Dataset size
Dataset size
(a) Synthetic bias.
(b) Hypothesis bias.
(c) Overlap bias.
Figure 3: Performance of ERM and IRM on good test subset when varying size of data (while holding
bias prevalence and strength fixed). The X axis labels show (training data size, validation data size).
6 Conclusion
In this work, we presented an empirical study of IRM across increasingly naturalistic settings. We observed that in all three proposed settings IRM successfully outperforms ERM on o.o.d data. We then pointed out bias prevalence, strength, and data size as factors affecting IRM. We showed that IRM loses its advantage over ERM when decreasing any of these factors. Aside from IRM's known sensitivity to random seeds, we also observe a different failure case of IRM in which the model "collapses" to null representation. As also mentioned in Arjovsky et al. (2000), such degenerate solution should theoretically be discarded by the ERM term in the objective function. However, practically it is not yet clear how properly change the regularization weight during training avoid this. This failure case can be detected and filtered without o.o.d data, by only looking at performance on training and validation environments. However, for the sake of completeness we do not filter these results and report them in the paper. To conclude, IRM and other recent methods taking a causal approach o.o.d generalization offer promising possibilities. Our results highlight the need for more naturalistic characterization of the settings under which these methods are expected to work and extensive empirical experimentation with complex datasets and state-of-the-an models.
Broader impact and ethical considerations
Our work investigates IRM in debiasing framework with naturalistic scenarios and characterizes its strengths and weaknesses. We hope that our results will encourage further research, both theoretical and empirical, characterize and improve IRM's behavior in real-world settings. Such a line of work could contribute to an increased robustness of deep learning models in the presence of biases. While we focused on dataset biases in this study, the approach could also apply to mitigating social biases like gender or racial bias. There are, however, potential harms. One might assume that a debiased model may be used as is in a deployed system, without further safety measures. In practice, debiasing on academic datasets might not lead to safc-to-use models in real applications and could give false impression of safety. Such a scenario somewhat corresponds to unconscious bias, which has been shown to exist in healthcare, criminal justice, and education. Compared with explicit bias, where measures promoting equality can be taken, unconscious bias goes undetected and therefore untreated. For example, Hall et al. (2015) found implicit bias in health care providers against people of colour, which negatively affected treatment decisions, treatment adherence, and patient health outcomes. We believe that carefully analyzing when IRM works and when it fails is important for guiding further research on debiasing natural language processing systems and machine learning systems more broadly.
10

---

Acknowledgements
We would like thank the anonymous reviewers for their helpful comments. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20). HH is partially supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI).
References
Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang. Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun. AlignedRelD: Surpassing human-level performance in person re-identification. arXiv 2017.
preprint
Titus Brinker, Achim Hekler, Alexander Enk, Joachim Klode, Axel Hauschild, Carola Berking, Bastian Schilling, Sebastian Haferkamp. Dirk Schadendorf. Stefan Fröhling, et al. A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in clinical melanoma image classification task. European Journal of Cancer, 1 111:148-154 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pages 3428-3448, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1/P19. 1334. URL.httts://accaantoo.oo wtw.aaattoooooooooo.ggog 1334
the
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. stat, 1050:27, 2020.
Yo Joong Choe, Jiyeon Ham, and Kyubyong Park. An empirical study of invariant risk minimization. arXiv preprint arXiv:2004.05007, 2020.
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pages 145-155. PMLR, 2020a.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? International Conference on Artificial Intelligence and Statistics, pages 4069-4077. PMLR, 2021.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v//N18-2000 URLhttps
//aclanthology www.ooyyyoo////1-/2000
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18853/v1/S188 2023.. URL https //aclanthology Btrg/S18-20 2023
Masatoshi Tsuchiya. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL Sotlhttp::/ htt///aclaattt aclanthology. org/ 2tt//18-12211
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL RtLhhtts::/// https:///ecaaattoooggg..o org/

---

Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650 655, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 118836/1111182222 URL https //aclanthology- org/P18-2100
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 256- 262, Minncapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10.18653/v1/$19-11 1028. URL httpt:://aclaaaaaaattt
org/ 1028
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alcxander Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 877-891, Florence, Italy, July 2019b. Association for Computational Linguisties. doi: 10. 118653/11//19-108. URL https://aclanttollll wrw.l/oyyyorr//1--1005500
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706-87 8716, Online, July 2020. Association for Compu- tational Linguistics. doi: 10. 1118883/11/222 acw.o..aalmmainn700.0 URL httpp:///watt acllaw aclweb. org/
aathollogy/2020.acl.. acl- main. 769
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low- Resource NLP (DeepLo 2019), pages 132-142, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.188533v1/D19..o org/D19-611 6115
URL https: http:///ccaattoooggoooo
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pages 947-1012, 2016.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk mini- mization. In International Conference on Learning Representations, 2020.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distributio generalization via risk extrap- olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/v1/D15-1100 URL httpa Taallantooogg..rrg/// org/D15-
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies. Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/w1/N18-11000 URL htttp:://aclaathologgg wwwphhoooogoooog///.////.. org/N1881///100
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186 2019.
12

---

R. Thomas McCoy, Junghyun Min, and Tal Linzen. BERT' of a feather do not generalize together: Large variability in generalization across models with similar test set performance. Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NL.P, pages 217-227, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653//1/2220.haccooxoppp 21
utthttps::/ htt.hhttss:/accaamholoo org/ 2020. blackboxnlp-1
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a difference with www.rrfacctaall-aameeeeeeee data. In International Conference Learning Representations, 2019.
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empiri- cal invariant risk minimization? sample complexity perspective. In International Conference on Learning Representations, 2020b.
William Hall, Mimi V Chapman, Kent M Lee, Yesenia M Merino, Tainayah W Thomas, B Keith Payne, Eugenia Eng, Steven Day, and Tamera Coyne-Beasley. Implicit racial/ethnic bias among health care professionals and its influence on health care outcomes: systematic review. American journal of public health, 105612):600--76, 2015.
Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. On the value of cuttoo--distribution testing: An example of goodhart's 's law. arXiv preprint arXiv:2005.0924 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Compu- tational Linguistics. URL, https:tpp///. : // www. ac web. org/anthologg/2022 2020 emn em demos. 6
13


Reviews:
[{"review_id": "ve--iRoNULf", "reviewer": "Reviewer_qz54", "summary": "This paper designs natural language inference (NLI) experiments to investigate the properties of the bias mitigation method, Invariant Risk Minimization (IRM). The authors design three out-of-domain setups: synthetic data and synthetic bias, natural data and synthetic bias, and natural data with real bias, to provide detailed analysis on IRM and ERM, under different conditions such as changing bias strength, prevalence, data sizes, and random seeds. The paper advocates the need for more naturalistic settings to study the state-of-the-art bias mitigation models.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper highlights the need for more naturalistic settings to study a state-of-the-art bias mitigation model IRM, and makes a concrete step towards that. The work is well motivated and is well set up in a task (and datasets) where spurious correlation is known to be an issue. The paper carefully constructs the experiments, and performs detailed experiments and analyses that well support the conclusions made on the NLI datasets, contributing to the empirical understanding of IRM in a specific naturalistic setting. The method proposed to construct naturalistic biased datasets is interesting. In general, the paper is clear and easy to follow.\n\nThe paper did not propose new models. The originality in that respect is very limited. As an empirical study (on an existing model), I would like to see experiments on more tasks (instead of just NLI) which would help make the conclusions more convincing. In my opinion, this seriously limits the significance of the study. \n\nI am not fully convinced that the experiment setup represents the synthetic-to-naturalistic continuum for NLI, since the synthetic-data-and-synthetic-bias toy set has little property of “NATURE LANGUAGE” inference, and I am not convinced it is “a simplified NLI task”, e.g., considering the properties of SNLI and MNLI datasets (e.g., these datasets focus a lot on lexical semantic relations, e.g., hypernyms, and has little XOR property.)\n \nI am also concerned with whether the methods used to construct the environments (i.e., environments generation) can be easily adapted to study more complicated bias where manually designing features is hard. I am also concerned if they can be scaled up to consider more diverse biases (e.g., sentence compositional structures can also be the source of bias) and to consider interaction of multiple sources of biases.  \n\nCan the author(s) provide some discussion on this research with regard to the recent work (Kamath et al., 2021)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "mLZNvS54KJT", "reviewer": "Reviewer_64jW", "summary": "The paper performs a case study of applying Invariant Risk Minimization (IRM) training to the task of recognizing Natural Language Inference (NLI). The goal is to test whether IRM can be helpful for training bias-free models on biased data. The considered biases include the hypothesis bias, the word overlap bias and a special synthetic bias whereby a noisy gold label is directly appended to the input. The training environments are constructed to feature different high levels of bias strength, i.e. the degree to which the biased feature is predictive of the label B(x). The test environments are constructed in such a way that the biased feature is a bad predictor of the label. The experiments show that IRM brings a significant but small improvement on top of ERM. The paper also features extensive analysis on the impact of bias strength, bias prevalence and training data size on the performance difference between IRM and ERM. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance: The paper studies a very important research question: how to train bias free models. Perhaps the most interesting result is the one reported in Table 3: IRM training does not help to fully ignore even a very clear and carefully controlled synthetic bias. \n\nClarity: The paper is largely clear. Improvements in the following aspects could help:\n- It would be great to have a table featuring $\\alpha_e$, $p_e$ and training data size for all experiments.\n- It is not 100% clear what ERM models are trained on. Mixture of training data from the two environments?\n- It is not very clear what exactly is the classifier and what is the representation function in the NLI experiments. \n\nQuality: the paper is very well executed. The study appears complete and the conclusions are clearly formulated. \n\nOriginality: I do not know IRM literature well enough to confidently say if experiments with pretrained language models and NLP tasks have never been done before. I’m not aware of such studies, hence I find the paper conclusions novel and original. \n\nAs the paper studies methods to reduce bias, I can’t see negative societal consequences, only positive ones. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the paper studies methods to reduce bias, I can’t see negative societal consequences, only positive ones. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SqFSLRPk55G", "reviewer": "Reviewer_kZ64", "summary": "In this paper, the authors perform an empirical analysis of the effectiveness of Invariant Risk Minimization (IRM). They focus on the natural language inference (NLI) task, and while prior work examining the efficacy of IRM has focused on synthetic benchmarks, results from this paper support the hypothesis that models trained with IRM are generally less sensitive to spurious correlations even in naturalistic settings. The authors conduct a rigorous ablation study with various random seeds, dataset size, strength of spurious correlations, and the presence of spurious correlations and study the influence of each on the efficacy of IRM as applied to NLI. In general the paper presents a good analysis, however, certain generalizations made in the paper may not stand up to scrutiny, as I point below. I think this paper could be a good analysis paper but needs significant improvements to be accepted.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is an analysis paper so to me the novelty lies in the knowledge gained from the analysis. It is particularly exciting to see that IRM works as applied to naturalistic settings for NLI tasks. To the best of my knowledge, this might be the first paper to try and investigate the effectiveness of IRM on naturalistic datasets. It is great to see that the results support prior work that has looked at more synthetic settings.\n- The problem setup is on the right track but needs to be fixed. Your findings are very interesting but you cannot show results on one (or two) out-of-domain evaluation set and claim that one model performs well out-of-domain compared to some other model. You could easily come up with another dataset where the other model performs better by just picking all the examples where the second model was correct and first was not, and that would be an out-of-domain dataset as well. Out-of-domain is not well defined, which is why it is necessary to show your results on a battery of out-of-domain datasets. Showing out-of-domain results on one dataset and claiming out-of-domain generalization is like showing correct prediction on one test example and claiming 100% accuracy iid. There are other ood datasets in NLI that you should also evaluate on, such as SNLI [1], ANLI [2], and counterfactually revised SNLI examples as collected in [3]. \n- Presentation of this paper needs significant improvements. It was not easy to follow your experimental design in particular (both Sections 3 and 4). Additionally, there are other issues such as in the analysis section, it would be nice if you support empirical findings with potential reasons as to why we might be observing something. \n\nAdditional comments:\n- For your synthetic bias experiment, you’re appending certain tokens to various classes to inject synthetic bias. But since you’re using BERT, and it is a contextual model, it would be good to run these experiments with different tokens to weed out any effects a particular token might be having on the contextual representation at-large. Ideally you would expect the same results across all replications as you see right now, but we can’t be sure.\n- I don’t think I quite follow Lines 222-225. Why would the test set have any role in whether a trained model relies on certain correlations or not?\n- It is interesting that in your experiments, IRM showed a large variance in performance across random seeds but ERM did not. This is different from what was observed about ERM in prior work on stress test evaluations (see [4]). Do you have any thoughts on why this might be happening here?\n\n[1] Bowman, Samuel R., Gabor Angeli, Christopher Potts, and Christopher D. Manning. \"A large annotated corpus for learning natural language inference.\" In Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pp. 632-642. Association for Computational Linguistics (ACL), 2015.\n\n[2] Nie, Yixin, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. \"Adversarial NLI: A New Benchmark for Natural Language Understanding.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901. 2020.\n\n[3] Kaushik, Divyansh, Eduard Hovy, and Zachary Lipton. \"Learning The Difference That Makes A Difference With Counterfactually-Augmented Data.\" In International Conference on Learning Representations. 2020.\n\n[4] Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020, November). The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 8215-8228).\n\n\n--------------------------------------------------------------------------------------\nI thank the authors for the time they've taken to address my concerns. After careful deliberation and going back and forth on points raised by other reviewers, I am happy to increase my score to recommend acceptance. I would urge the authors to update their draft to address the concerns raised, including doing a deeper discussion of the negative results that are present in the Appendix.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have done a decent job at scoping out the limitations and societal impacts. I think connecting your point about the assumption of having a \"debiased\" model in production and how it could be harmful without proper understanding or safety measures to prior work in fairness or philosophy could be helpful for the reader.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "Ky0nKG6Scvv", "reviewer": "Reviewer_Ktci", "summary": "The paper presents an empirical study of IRM on NLI focusing on hypothesis and overlap bias. To do this, the paper defines the notion of bias \"strength\" p_e (probability of the bias label, given that the instance is indeed biased) and \"prevalence\" alpha_e (proportion of biased samples) for environment e. In practice this is controlled by training a biased predictor that only uses the (known) bias of the input to predict the label (e.g., for hypothesis bias we can finetune BERT on hypotheses only). Training environments correspond to different values of p_e and alpha_e (e.g., (0.7, 0.82) and (0.9, 0.82)) which is necessary for IRM training. The model is then tested on in-domain/bias aligned (e.g., alpha=1 and p=1), out-of-domain/bias misaligned (alpha=1 and p=0), and unbiased (alpha=0) data. The main finding of the paper is that: (1) IRM can outperform ERM even on natural data if the test set is bias misaligned, (2) the performance of IRM is heavily influenced by p, alpha, and also data size in training.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n- As the paper says, this is one of the few studies on IRM that focus on non-synthetic data. Given the interest on IRM this is a useful resource.\n\n- The paper's experimental design seems sound. NLI (binary formulation) is a good task to consider since it's simple and has known biases, though a flip side is it's sometimes viewed too synthetic to be a \"real\" task in NLP.\n\n- That IRM can outperform ERM on natural data under a right setting is encouraging news.\n\n\nWEAKNESSES \n\n- Inevitably the scope is narrow. At this point we're trying to making something that doesn't work in most settings work mostly because it's theoretically justified and interesting. While the positive finding of the paper is good news, I'm not sure how much it matters outside the scope of IRM/maybe robust learning. \n\n- This also raises some concern on practicality. Basically the finding (Section 5) seems to say that if bias is extremely prevalent in training data (alpha > 0.7) and extremely strong (p_e > 0.7-0.9), and we have plenty of data in each environment, then IRM can possibly improve a little over ERM, but even then the improvement is slight and training is unstable. The training domains are created by controlling bias prevalence and strength, and I'm not sure if it's feasible to satisfy these conditions in real-world data.\n\n\nQUESTION\n\n- The paper (and most existing works, I think) holds a particular type of bias fixed and uses different values of p_e and alpha_e to create IRM-friendly environments. Is this the only way to  use IRM, or is it possible to come up with a more natural setting where we don't assume the knowledge of bias (there may be unknown multiple types of bias) and still create training environments suitable for IRM?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effectiveness of Invariant Risk Minimization (IRM) in order to train unbiased classifiers in a specific Natural Language Inference (NLI) setting. The authors devise three experimental scenarios: synthetic data and synthetic bias, natural data and synthetic bias, natural data and natural bias. In all three settings, the authors study whether IRM is capable of training bias free classifiers by controlling the prevalence, strength of bias and data size. Results seem to show that, apart from a very synthetic setting, IRM seems to only marginally succeed in training robust classifiers, especially when bias prevalence, strength and data sizes are small.

—

All reviewers concur that this paper has extensive experiments and can be a useful resource as is one of the few studies of IRM with non-synthetic data. The paper appears clear although some sentences must be reformulated and made more precise/formal (e.g. "IRM is not able to completely ignore the bias when the test environments are sufficiently different"). Experiments are well-executed. One of the concerns of the reviewers was that the paper is somewhat limited in scope: binary NLI may be a too simplistic task in NLP and thus it is unclear whether the results in the paper can tell something about IRM’s performance in other settings. A reviewer also noticed the absence of performance in more ‘standard’ out-of-domain test sets (e.g. ANLI, counterfactual SNLI, …). We concurred that, in these test sets, it might be difficult to control for factors influencing IRM performance, and it might be reasonable to limit this paper to the careful study of controlled settings. After discussion with reviewers, I suggest the authors to move the HANS results from Appendix C.5 into the main text as this is a well-known evaluation set (even if results might not be directly comparable with reported baselines) and address some of the aforementioned issues. The authors might consider to also test on the Revised Premise dataset from [1]. Finally and most importantly, one source of confusion revolved around the takeaways of the paper. Reviewers and myself agree that this seems like a carefully executed negative results paper, as IRM doesn't seem very successful apart from the synthetic setting. I strongly suggest the authors to take a clearer stance on this by removing emphasis (i.e. from the abstract) on the fact that IRM performs well in a synthetic setting (which is mildly interesting) and emphasize the message in lines 50-51: "However, in these more naturalistic settings, IRM is not able to completely discard the bias, while ERM does not rely solely on the bias. Thus, in practice the advantage of IRM is small."

Overall, this paper is a useful resource as it's one of the few that studies IRM in a controlled and natural, albeit simplistic, task. Provided that the authors incorporate in the final version the suggestions above and reviewers' feedback, I recommend this paper for acceptance.

[1] Kaushik, Divyansh, Eduard Hovy, and Zachary Lipton. "Learning The Difference That Makes A Difference With Counterfactually-Augmented Data." In International Conference on Learning Representations. 2020.
Decision: Accept (Poster)