Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper designs natural language inference (NLI) experiments to investigate the properties of the bias mitigation method, Invariant Risk Minimization (IRM). The authors design three out-of-domain setups: synthetic data and synthetic bias, natural data and synthetic bias, and natural data with real bias, to provide detailed analysis on IRM and ERM, under different conditions such as changing bias strength, prevalence, data sizes, and random seeds. The paper advocates the need for more naturalistic settings to study the state-of-the-art bias mitigation models.
main_review: The paper highlights the need for more naturalistic settings to study a state-of-the-art bias mitigation model IRM, and makes a concrete step towards that. The work is well motivated and is well set up in a task (and datasets) where spurious correlation is known to be an issue. The paper carefully constructs the experiments, and performs detailed experiments and analyses that well support the conclusions made on the NLI datasets, contributing to the empirical understanding of IRM in a specific naturalistic setting. The method proposed to construct naturalistic biased datasets is interesting. In general, the paper is clear and easy to follow.

The paper did not propose new models. The originality in that respect is very limited. As an empirical study (on an existing model), I would like to see experiments on more tasks (instead of just NLI) which would help make the conclusions more convincing. In my opinion, this seriously limits the significance of the study. 

I am not fully convinced that the experiment setup represents the synthetic-to-naturalistic continuum for NLI, since the synthetic-data-and-synthetic-bias toy set has little property of “NATURE LANGUAGE” inference, and I am not convinced it is “a simplified NLI task”, e.g., considering the properties of SNLI and MNLI datasets (e.g., these datasets focus a lot on lexical semantic relations, e.g., hypernyms, and has little XOR property.)
 
I am also concerned with whether the methods used to construct the environments (i.e., environments generation) can be easily adapted to study more complicated bias where manually designing features is hard. I am also concerned if they can be scaled up to consider more diverse biases (e.g., sentence compositional structures can also be the source of bias) and to consider interaction of multiple sources of biases.  

Can the author(s) provide some discussion on this research with regard to the recent work (Kamath et al., 2021)?

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper performs a case study of applying Invariant Risk Minimization (IRM) training to the task of recognizing Natural Language Inference (NLI). The goal is to test whether IRM can be helpful for training bias-free models on biased data. The considered biases include the hypothesis bias, the word overlap bias and a special synthetic bias whereby a noisy gold label is directly appended to the input. The training environments are constructed to feature different high levels of bias strength, i.e. the degree to which the biased feature is predictive of the label B(x). The test environments are constructed in such a way that the biased feature is a bad predictor of the label. The experiments show that IRM brings a significant but small improvement on top of ERM. The paper also features extensive analysis on the impact of bias strength, bias prevalence and training data size on the performance difference between IRM and ERM. 

main_review: Significance: The paper studies a very important research question: how to train bias free models. Perhaps the most interesting result is the one reported in Table 3: IRM training does not help to fully ignore even a very clear and carefully controlled synthetic bias. 

Clarity: The paper is largely clear. Improvements in the following aspects could help:
- It would be great to have a table featuring $\alpha_e$, $p_e$ and training data size for all experiments.
- It is not 100% clear what ERM models are trained on. Mixture of training data from the two environments?
- It is not very clear what exactly is the classifier and what is the representation function in the NLI experiments. 

Quality: the paper is very well executed. The study appears complete and the conclusions are clearly formulated. 

Originality: I do not know IRM literature well enough to confidently say if experiments with pretrained language models and NLP tasks have never been done before. I’m not aware of such studies, hence I find the paper conclusions novel and original. 

As the paper studies methods to reduce bias, I can’t see negative societal consequences, only positive ones. 

limitations_and_societal_impact: As the paper studies methods to reduce bias, I can’t see negative societal consequences, only positive ones.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors perform an empirical analysis of the effectiveness of Invariant Risk Minimization (IRM). They focus on the natural language inference (NLI) task, and while prior work examining the efficacy of IRM has focused on synthetic benchmarks, results from this paper support the hypothesis that models trained with IRM are generally less sensitive to spurious correlations even in naturalistic settings. The authors conduct a rigorous ablation study with various random seeds, dataset size, strength of spurious correlations, and the presence of spurious correlations and study the influence of each on the efficacy of IRM as applied to NLI. In general the paper presents a good analysis, however, certain generalizations made in the paper may not stand up to scrutiny, as I point below. I think this paper could be a good analysis paper but needs significant improvements to be accepted.
main_review: - The paper is an analysis paper so to me the novelty lies in the knowledge gained from the analysis. It is particularly exciting to see that IRM works as applied to naturalistic settings for NLI tasks. To the best of my knowledge, this might be the first paper to try and investigate the effectiveness of IRM on naturalistic datasets. It is great to see that the results support prior work that has looked at more synthetic settings.
- The problem setup is on the right track but needs to be fixed. Your findings are very interesting but you cannot show results on one (or two) out-of-domain evaluation set and claim that one model performs well out-of-domain compared to some other model. You could easily come up with another dataset where the other model performs better by just picking all the examples where the second model was correct and first was not, and that would be an out-of-domain dataset as well. Out-of-domain is not well defined, which is why it is necessary to show your results on a battery of out-of-domain datasets. Showing out-of-domain results on one dataset and claiming out-of-domain generalization is like showing correct prediction on one test example and claiming 100% accuracy iid. There are other ood datasets in NLI that you should also evaluate on, such as SNLI [1], ANLI [2], and counterfactually revised SNLI examples as collected in [3]. 
- Presentation of this paper needs significant improvements. It was not easy to follow your experimental design in particular (both Sections 3 and 4). Additionally, there are other issues such as in the analysis section, it would be nice if you support empirical findings with potential reasons as to why we might be observing something. 

Additional comments:
- For your synthetic bias experiment, you’re appending certain tokens to various classes to inject synthetic bias. But since you’re using BERT, and it is a contextual model, it would be good to run these experiments with different tokens to weed out any effects a particular token might be having on the contextual representation at-large. Ideally you would expect the same results across all replications as you see right now, but we can’t be sure.
- I don’t think I quite follow Lines 222-225. Why would the test set have any role in whether a trained model relies on certain correlations or not?
- It is interesting that in your experiments, IRM showed a large variance in performance across random seeds but ERM did not. This is different from what was observed about ERM in prior work on stress test evaluations (see [4]). Do you have any thoughts on why this might be happening here?

[1] Bowman, Samuel R., Gabor Angeli, Christopher Potts, and Christopher D. Manning. "A large annotated corpus for learning natural language inference." In Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pp. 632-642. Association for Computational Linguistics (ACL), 2015.

[2] Nie, Yixin, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. "Adversarial NLI: A New Benchmark for Natural Language Understanding." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901. 2020.

[3] Kaushik, Divyansh, Eduard Hovy, and Zachary Lipton. "Learning The Difference That Makes A Difference With Counterfactually-Augmented Data." In International Conference on Learning Representations. 2020.

[4] Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020, November). The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 8215-8228).


--------------------------------------------------------------------------------------
I thank the authors for the time they've taken to address my concerns. After careful deliberation and going back and forth on points raised by other reviewers, I am happy to increase my score to recommend acceptance. I would urge the authors to update their draft to address the concerns raised, including doing a deeper discussion of the negative results that are present in the Appendix.
limitations_and_societal_impact: I think the authors have done a decent job at scoping out the limitations and societal impacts. I think connecting your point about the assumption of having a "debiased" model in production and how it could be harmful without proper understanding or safety measures to prior work in fairness or philosophy could be helpful for the reader.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an empirical study of IRM on NLI focusing on hypothesis and overlap bias. To do this, the paper defines the notion of bias "strength" p_e (probability of the bias label, given that the instance is indeed biased) and "prevalence" alpha_e (proportion of biased samples) for environment e. In practice this is controlled by training a biased predictor that only uses the (known) bias of the input to predict the label (e.g., for hypothesis bias we can finetune BERT on hypotheses only). Training environments correspond to different values of p_e and alpha_e (e.g., (0.7, 0.82) and (0.9, 0.82)) which is necessary for IRM training. The model is then tested on in-domain/bias aligned (e.g., alpha=1 and p=1), out-of-domain/bias misaligned (alpha=1 and p=0), and unbiased (alpha=0) data. The main finding of the paper is that: (1) IRM can outperform ERM even on natural data if the test set is bias misaligned, (2) the performance of IRM is heavily influenced by p, alpha, and also data size in training.
main_review: STRENGTHS

- As the paper says, this is one of the few studies on IRM that focus on non-synthetic data. Given the interest on IRM this is a useful resource.

- The paper's experimental design seems sound. NLI (binary formulation) is a good task to consider since it's simple and has known biases, though a flip side is it's sometimes viewed too synthetic to be a "real" task in NLP.

- That IRM can outperform ERM on natural data under a right setting is encouraging news.


WEAKNESSES 

- Inevitably the scope is narrow. At this point we're trying to making something that doesn't work in most settings work mostly because it's theoretically justified and interesting. While the positive finding of the paper is good news, I'm not sure how much it matters outside the scope of IRM/maybe robust learning. 

- This also raises some concern on practicality. Basically the finding (Section 5) seems to say that if bias is extremely prevalent in training data (alpha > 0.7) and extremely strong (p_e > 0.7-0.9), and we have plenty of data in each environment, then IRM can possibly improve a little over ERM, but even then the improvement is slight and training is unstable. The training domains are created by controlling bias prevalence and strength, and I'm not sure if it's feasible to satisfy these conditions in real-world data.


QUESTION

- The paper (and most existing works, I think) holds a particular type of bias fixed and uses different values of p_e and alpha_e to create IRM-friendly environments. Is this the only way to  use IRM, or is it possible to come up with a more natural setting where we don't assume the knowledge of bias (there may be unknown multiple types of bias) and still create training environments suitable for IRM?
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper studies the effectiveness of Invariant Risk Minimization (IRM) in order to train unbiased classifiers in a specific Natural Language Inference (NLI) setting. The authors devise three experimental scenarios: synthetic data and synthetic bias, natural data and synthetic bias, natural data and natural bias. In all three settings, the authors study whether IRM is capable of training bias free classifiers by controlling the prevalence, strength of bias and data size. Results seem to show that, apart from a very synthetic setting, IRM seems to only marginally succeed in training robust classifiers, especially when bias prevalence, strength and data sizes are small.

—

All reviewers concur that this paper has extensive experiments and can be a useful resource as is one of the few studies of IRM with non-synthetic data. The paper appears clear although some sentences must be reformulated and made more precise/formal (e.g. "IRM is not able to completely ignore the bias when the test environments are sufficiently different"). Experiments are well-executed. One of the concerns of the reviewers was that the paper is somewhat limited in scope: binary NLI may be a too simplistic task in NLP and thus it is unclear whether the results in the paper can tell something about IRM’s performance in other settings. A reviewer also noticed the absence of performance in more ‘standard’ out-of-domain test sets (e.g. ANLI, counterfactual SNLI, …). We concurred that, in these test sets, it might be difficult to control for factors influencing IRM performance, and it might be reasonable to limit this paper to the careful study of controlled settings. After discussion with reviewers, I suggest the authors to move the HANS results from Appendix C.5 into the main text as this is a well-known evaluation set (even if results might not be directly comparable with reported baselines) and address some of the aforementioned issues. The authors might consider to also test on the Revised Premise dataset from [1]. Finally and most importantly, one source of confusion revolved around the takeaways of the paper. Reviewers and myself agree that this seems like a carefully executed negative results paper, as IRM doesn't seem very successful apart from the synthetic setting. I strongly suggest the authors to take a clearer stance on this by removing emphasis (i.e. from the abstract) on the fact that IRM performs well in a synthetic setting (which is mildly interesting) and emphasize the message in lines 50-51: "However, in these more naturalistic settings, IRM is not able to completely discard the bias, while ERM does not rely solely on the bias. Thus, in practice the advantage of IRM is small."

Overall, this paper is a useful resource as it's one of the few that studies IRM in a controlled and natural, albeit simplistic, task. Provided that the authors incorporate in the final version the suggestions above and reviewers' feedback, I recommend this paper for acceptance.

[1] Kaushik, Divyansh, Eduard Hovy, and Zachary Lipton. "Learning The Difference That Makes A Difference With Counterfactually-Augmented Data." In International Conference on Learning Representations. 2020.
Decision: Accept (Poster)