Explaining the Efficacy of Counterfactually Augmented Data
Authors: Divyansh Kaushik
Amrith Setlur
Eduard H Hovy
Zachary Chase Lipton
Venue: ICLR 2021 Poster (2021)
Keywords: humans in the loop
annotation artifacts
text classification
sentiment analysis
natural language inference

Abstract: In attempts to produce machine learning models less reliant on spurious patterns in NLP datasets, researchers have recently proposed curating counterfactually augmented data (CAD) via a human-in-the-loop process in which given some documents and their (initial) labels, humans must revise the text to make a counterfactual label applicable. Importantly, edits that are not necessary to flip the applicable label are prohibited. Models trained on the augmented (original and revised) data appear, empirically, to rely less on semantically irrelevant words and to generalize better out of domain. While this work draws loosely on causal thinking, the underlying causal model (even at an abstract level) and the principles underlying the observed out-of-domain improvements remain unclear. In this paper, we introduce a toy analog based on linear Gaussian models, observing interesting relationships between causal models, measurement noise, out-of-domain generalization, and reliance on spurious signals. Our analysis provides some insights that help to explain the efficacy of CAD. Moreover, we develop the hypothesis that while adding noise to causal features should degrade both in-domain and out-of-domain performance, adding noise to non-causal features should lead to relative improvements in out-of-domain performance. This idea inspires a speculative test for determining whether a feature attribution technique has identified the causal spans. If adding noise (e.g., by random word flips) to the highlighted spans degrades both in-domain and out-of-domain performance on a battery of challenge datasets, but adding noise to the complement gives improvements out-of-domain, this suggests we have identified causal spans. Thus, we present a large scale empirical study comparing spans edited to create CAD to those selected by attention and saliency maps. Across numerous challenge domains and models, we find that the hypothesized phenomenon is pronounced for CAD.

Published as a conference paper at ICLR 2021
EXPLAINING THE EFFICACY OF COUNTERFACTUALLY AUGMENTED DATA
Divyansh Kaushik, Amrith Setlur, Eduard Hovy, Zachary C. Lipton Carnegie Mellon University Piusburgh, PA, USA {dkaushik, asetlur, hovy, zlipton] (opptoneemmon @emu.edu
ABSTRACT
In attempts to produce machine learning models less reliant on spurious patterns NLP datasets, researchers have recently proposed curating counterfactually aug- mented data (CAD) via a human-in-the-loww process in which given some docu- ments and their (initial) labels, humans must revise the text to make counterfac- tual label applicable. Importantly, edits that are not necessary to flip the applicable label are prohibited. Models trained on the augmented (original and revised) data appear, empirically, to rely less on semantically irrelevant words and to gener- alize better out of domain. While this work draws loosely causal thinking, the underlying causal model (even at an abstract level) and the principles under- lying the observed out-of-domain improvements remain unclear. In this paper, we introduce toy analog based on linear Gaussian models, observing interesting relationships between causal models, measurement noise, out-of- general- ization, and reliance on spurious signals. Our analysis provides some insights that help to explain the efficacy of CAD. Moreover, we develop the hypothesis that while adding noise to causal features should degrade both in-domain and out-of- domain performance, adding noise to non-causal features should lead to relative improvements in out-of-domain performance. This idea inspires speculative test for determining whether a feature attribution technique has identified the causal spans. If adding noise (e.g., by random word flips) to the highlighted spans de- grades both in-domain and out-of-domait performance on a battery of challenge datasets, but adding noise to the complement gives improvements out-of-domain, this suggests we have identified causal spans. Thus, we present large-scale em- pirical study comparing spans edited create CAD those selected by attention and saliency maps. Across numerous challenge domains and models, we find that the hypothesized phenomenon is pronounced for CAD.
INTRODUCTION
Despite machine learning (ML)'s many practical breakthroughs, formidable obstacles obstruct its deployment in consequential applications. Of particular concern, these models have been shown rely on spurious signals, such as surface-level textures in images (Jo Bengio 2017 Geirhos et al. 2018). and background scenery when the task is to recognize foreground objects (Beery al. 2018). Other studies have uncovered worrisome reliance on gender in models trained for the purpose of recommending jobs [Dastin] 2018),, and on race in prioritizing patients for medical care (Obermeyer et al.) 2019). Moreover, while modern ML performs remarkably well on independent and identically distributed (iid) holdout data, performance often decays catastrophically under both naturally occurring and adversarial distribution shift (Quioner--Cande
al.) 2009 Suggyyma & Kawanabe 2012 Szegedy al. 2014: Ovadia al. 2019 Filos et al. 2020))
These two problems: (i) reliance on semantically irrelevant signals, raising concerns about bias; and (ii) the brittleness of models under distributions shift; might appear unrelated, but share important conceptual features. Concerns about bias stem in part from principles of procedural fairness (Blader & Tyler 2003 Miller 2017 Grgic-Hlaca et al. 2018 Lipton et al. 2018). according to which decisions should be based on qualifications, not on distant proxies that are spuriously associated with the outcome of interest. Arguably key distinction of an actual qualification might be that it

---

Published as a conference paper at ICLR 2021
Ø2
222
2/2
Σ1
y
2
y
T
y
(a) Causal setting
(b) Noisy measurement in (c) Anticausal setting causal setting
(d) Noisy measurements in anticausal setting
Figure 1: Toy causal models with one hidden confounder. In la and Ic the observed covariates are the observed covariates are 21, 22. In all cases, y denotes the label.
F1, F2. In andand Id
actually exerts causal influence on the outcome of interest. In an interesting parallel, one line of work on distribution shift has focused on causal graphical models, addressing settings where some parts of
the model remain stable over time but others do not. One common assumption that the relationship
between the target and its direct causal ancestors remains invariant (Peters et al.) [016)] Ghassami
al. 2017; KKojasCCarulla et al.) 2018 Kuang et al. 2018 Magliacane al.] 2018] Christiansen & Peters 2020; Weichwald & Peters, 2020p. While these papers contribute insight, they focus on toy settings, with few variables related by a known model. However, in complex domains with high-dimensional data, what variables are relevant and what graph relates them is typically unclear.
Recently in NLP, Kaushik et al. (2020) proposed Counterfactually Augmented Data (CAD), inject- ing causal thinking into real world settings by leveraging numan-in-the-Uuee feedback to identify causally relevant features (versus those that merely happen to be predictive due to confounding). Human editors are presented with document-labes pairs and tasked with editing documents to render counterfactual labels applicable. The instructions restrict editors to only make modifications that are necessary to flip the label's applicability. The key result is that many spurious correlations present in the original dataset are absent in the CAD. In case of sentiment analysis, Kaushik et al. (2020) demonstrated that linear classifiers trained to predict the sentiment of movie reviews based on bag- of-words representations assign high- magnitude weights to seemingly irrelevant terms, including "will", "my". "has", "especially". and "script", among others. Notably, "horror" featured among the most negative terms, while "romance" featured among the most positíve, despite both commu- nicating genre, not sentiment. Interestingly, in the revised data, each "horror" review retains the word "horror" (per the instruction not to make unnecessary edits) but is associated with the opposite sentiment label. Models trained on the augmented data (original and revised) perform well on both original and revised data, and assign little weight to the associated but irrelevant terms. Intuitively, one might imagine that the spurious patterns would generalize less reliably out of domain. Most consumer products do not belong movie genres, but words like "excellent" and "awful" continue to connote positive and negative sentiment, respectively. Indeed, Kaushik et al. (2020) demonstrated that models trained on CAD enjoyed out-of-domain performance benefits Tweets, and Amazon and Yelp reviews.
In this paper, we make some initial attempts towards explaining CAD's efficacy. While CAD plainly draws on causal thinking, (invoking interventions and counterfactuals), foundational questions re- main open: What the assumed causal structure underlying settings where CAD might be effec- tive? What are the principles underlying its out-of-domain benefits? Must humans really intervene, or could automatic feature attribution methods, e.g., attention YDeng et al. 2020). or cheaper feedback mechanisms, e.g., feature feedback (Zaidan et al. 2007), produce similar results?
To begin, we consider linear Gaussian models (Figure T: Wright 1934), with the following goals: to (i) gain qualitative insights into when predictor might rely on spurious signals in the first place; and (ii) provide a mechanism action to explain the efficacy of CAD. First, we analyze the causal setting (features cause the label). When the features share common cause and predictor is well- specified (linear). it will assign zero weight (in expectation) to non-causal features. However, when the causal features are subject to observation noise (measurement error). the non-causal features assigned non-zero weight. Conversely, when we inject noise on non-causal features, predictors rely more causal features, which expect to result in better out-of-domain generalization. In the
2

---

Published as a conference paper at ICLR 2021
causal framework, observe that CAD might be usefully formalized as process analogous to intervening on the causal features, thus d-separating the label from the non-causal features (Pearl 1985). Alternatively, we might conceptualize CAD with an anticausal model (Schölkopf et al. 2012). In this setup, the label of interest is one of several latent attributes that directly causes some (but not all features). In this interpretation, we imagine that have intervened on the label and the editor's role is to simulate the counterfactual document that would flow from the alternative label, holding other attributes constant. Note that this too d-separates the label from the spurious correlate. In both cases, any model trained on the resulting data ought to rely only on the causal features.
Our toy abstraction points to useful diagnostic test. If indeed CAD involves interventions on spans that are (in some sense) analogous to the causal features in our toy model, then injecting noise on these words should increase model reliance on the non-causal features and thus (in general) lead to deteriorating performance out-of-domain On the other hand, injecting noise on the non-causal features should lead the model rely more on the causal features, leading to improved performance out of domain. Through series of large-scale empirical experiments addressing sentiment anal- ysis and natural language inference (NLI) tasks, we inject noise on the spans marked as causal vs non-causal. We compare the effects of injecting noise the spans revised by the CAD editors, the spans selected through feature feedback (Zaidan et al., 2007), and to spans selected automatically using feature attribution heuristics such as attention- and gradient-based saliency methods. If indeed the hypotheses that (i) identifying causal features requires human intervention; and (ii) models re- lying on causal features generalize better out of domain; hold, we might expect that (compared to automatic attribution methods) noising human- human-provided provided rationales would deteriorate out-of-domain performance, while noising non-rationales should prove beneficial.
We show that an SVM sentiment analysis model trained on the original 1.7k IMDb reviews from Kaushik et al./ (2020) obtains 87.8% accuracy on the IMDb test set and 79.9% on Yelp reviews but when all rationales are replaced with noise, the classifier experiences = 11% drop on in-sample accuracy and an even bigger drop of Ri 28. 28.7% on Yelp. However, as non-rationales are replaced with noise, in-domain accuracy goes down by R 10% but out-of-domain accuracy increases by 1.5%. Similarly, in NLI, the accuracy of a BERT classifier fine-tuned on a subsample of e-SNLI YDeng et al. 2020) goes down by ~ 20% when rationales are replaced with noise, whereas the out-of- domain accuracy goes down by 21.3-31. 33.5% on various datasets. If non-rationales are replaced with noise, in-sample accuracy goes down by 6.2% but out of domain accuracy drops by only Similar patterns are observed across both tasks, on all datasets and models. However, when using attention masks, the resulting changes in model performance do not appear follow these trends. In another test to probe whether human feedback is indeed necessary to produce datasets with the observed quantitative results of CAD, we experiment with style transfer methods for converting Positive reviews into Negative and vice versa. Compared to an SVM classifier trained on style- transfer-augmented data, training on CAD leads to a gain of 5-16.4% in accuracy on Amazon and 3.7-17.8% on Yelp. Similarly, a BERT classifier fine-tuned on CAD outperforms the same classifier fine-tuned on style-transfer-auggmeett data by 4.9-21.5% on Amazon and 1.9-9.5% on Yelp.
RELATED WORK
NLP papers on spurious associations have addressed social biases (Dixon et al., 2018 Zhao al. 2018 Kiritchenko & Mohammad] [2018] Dinan et al. [2019] May et al. 2019), spurious signals owing to annotation heuristics (Gururangan et al. 2018 Poliak al. 2018). and artifacts from automatic data generation (Chen et al.) 2016 Kaushik & Lipton 2018), Researchers have also demonstrated vulnerabilities to synthetic transformations, such as distractor phrases (Jia Liang 2017 Wallace et al. 2019). document paraphrases (lyyer et al. 2018 Pfeiffer et al. 2019). and synthetic but geeaning-preservve modifications (Ribeiro et al. 2018 Glockner et al.] 2018 Shen et al. 2018).
Researchers have proposed incorporating human feedback solicited through variety of mechanisms including highlighting rationales, spans of text indicative of the label (Zaidan et al. 2007 Zaidan &
Disner 2008; Poulis & Dasgupta, 2017). To combat gender stereotypes, Lu et al. (2018) Zmigrod et al. (2019; Maudslay et al. (2019) describe data augmentation approaches that programmatically alter text. More recently, Kaushik et al. (2020) employed crowd workers to edit text to make an opposite label applicable. Through their experiments they show that classifiers trained on CAD
generalize well out of domain. Teney al. (2020) show the benefits of CAD in computer vision and
3

---

Published as a conference paper at ICLR 2021
NLP, and Srivastava et al. (2020) employ crowdworkers augment their training data capture potential unmeasured variables. growing body of work has also looked at reducing reliance on spurious correlations by exploiting the stability of relationships between the target variable and its (graph) neighbors. |Peters et al. (2016) propose invariant causal prediction to obtain a causal predictor from multiple datasets. Ghassami
al. (2017) discuss similar approach but do not assume that the exogenous noise of the target variable stays fixed among environments. They also demonstrate the benefits of their approach (compared to PPeters al. (2016)) in identifying all direct ancestors of the target variable. Arjovsky et al. (2019) propose invariant risk minimization, with the goal of learning a data representation such that the optimal predictor shared across environments.
ANALYSIS OF A TOY MODEL
We briefly review the OLS estimator for the model Y = = + where Y E R" is the target, X E Rexp the design matrix, 3 E R2 the coefficient vector we want to estimate, and ~ NN, II,)) an iid noise term. The OLS estimate Bols is given by Cov(X, X) Cov(X, (Y Y). Representing Var[X;] as of and Cov(Xi, as o,,,, if we observe only two covariates (p 2). then:
8
02 ow. yy
Jw 02 J22.3
3g
RRL 0 02
(I)
2 2
2 22.22
20 22
Our analysis adopts the structural causal model (SCM) framework (Pearl 2009). formalizing causal
relationships via Directed Acyelic Graphs (DAGss. Each edge of the form A - B E E in a DAG = (VV&) E) indicates that the variable variables X E V in the model are deterministic functions of their corresponding parents Pa(X) C and a set of jointly independent noise terms. For simplicity, we work with linear Gaussian SCMs in the presence of single confounder where each variable is linear function of its parents and the noise terms are assumed to be additive and Gaussian. We look at both causal and anticausal learning settings. In the former, we assume that document causes the applicability of the label (as in annotation, where the document truly causes the label). In the latter interpretation, we assume that the label is one latent variable (among many) that causes features of the document (as when reviewer's s "actual sentiment" influences what they write). For simplicity, we assume that the latent variables are correlated due to confounding but that each latent causes a distinct set of observed features. Without loss of generality, we assume that all variables have zero mean. Both DAGs contain the four random variables 2, 11, 22, y and the anticausal DAG also contains some additional latent variables (Figure [
(potentially) a direct cause of variable B. All measured
The derivations are standard and are included in Appendix A
3.1 THE CAUSAL SETTING
We now focus on the causal setting (Figure la [b] Let the Gaussian SCM be defined as follows where the noise term for variable x is defined as Ugg
Ug ~ N oz,)
z = Ulg.
T1 bz + Uni
Us: N(O0, oir))
T2 = cz + U322
Ux, NNO, (222)
y 0.1 + Uy:
"y o2,
Applying OLS, we obtain gols = a and Sgla = 0. However, consider what happens if we only
observe 2n via noisy proxy कΙ ~ N2 oun, + + (Figure lb Assuming, Ez, 1 (m1, 22, v),
from Eq. we get the estimates gols and Bols (Eq. 3 in the presence of observation noise on Tj.
a(02, (5202 usz + 2022)) + U.1 Uuz
8ols
oy (Doo 20 + o ti, + oz (20g, +
(3)
acho? 2
3g 2 (booz uuz
+
+ o 0% + +o ez
(Co2 + +o
As we can see, gpla x o. This shows us that da increases, | (the magnitude of the coefficient for æ1) decreases and |لووا| (the magnitude of the coefficient for x2) increases. The
4

---

Published as a conference paper at ICLR 2021
asymptotic OLS estimates in the presence of infinite observational noise is limo?, +00 37 O,
whereas Sgls converges a finite non-zero value. On the other hand, observing a noisy version of 22 will not affect our OLS estimates if there is no measurement error on ø1.
These simple graphs provide qualitative insights into when should expect model rely on spurious patterns. In the causal setting, under perfect measurement, the causal variable d-separates the non-causal variable from the label (Figure Tap. However, under observation noise. a predictor will rely on the non-causal variable (Eq. 3 Moreover, when the causal feature is noisily observed, additional observation noise on non-causal features yields models that are more reliant on causal features. We argue that while review text is not noisily observed per se, learning with imperfect feature representations acquired by training deep networks on finite samples has an effect that is analogous to learning with observation noise.
Connection to Counterfactually Augmented Data In the causal setting, intervening on the causal feature, d-separates the label from the non-causal feature 22, and thus models trained on samples from the interventional distribution will rely solely on the causal feature, even when is noisily observed. We argue that in qualitative sense, the process of generating CAD resembles such an intervention, however instead of intervening randomly, we ensure that for each example, we produce two sets of values of 31, one such that the label is applicable and one such that it is not applicable. One is given in the dataset, and the other is produced via the revision.
3.2 AN ANTICAUSAL INTERPRETATION
Alternatively, rather than thinking of features causing the applicable label, we might think of the "causal feature" direct effect of the label (not cause). In this case, so long the relationship truly not deterministic, even absent noisy observation, conditioning on the causal feature does not d-separate the label from the non-causal feature and thus models should be expected to assign weight to both causal and non-causal variables.
As in the causal setting, as we increase observation noise the causal variable, the weight assigned to the non-causal variable should increase. Conversely, as in the causal setting with observation noise on ø1, observation noise on the non-causal feature 22 increases, we expect the learned predictor to rely more on the causal feature. We derive the OLS coefficients (including under the presence of observational noise, Fig. lcb in this setting in Appendix A.2
Connection to Counterfactually Augmented Data In this interpretation, we think of CAD as a process by which we (the designers of the experiment) intervene on the label itself and the human editors, play the role of a simulator that we imagine to be capable of generating a counterfactual example, holding all other latent variables constant. In the sentiment case, we could think of the editors as providing us with the review that would have existed had the sentiment been flipped, holding all other aspects of the review constant. Note that by intervening on the label, d-separate it from the spurious correlate 22 (Figure
3.3 INSIGHTS AND TESTABLE HYPOTHESES
In both the causal and anticausal models, the mechanism underlying the causal relationship that binds 21 to y (regardless of direction) that binding language to semantic concept (such as sen- timent), which we expect to be more stable across settings than the more capricious relationships among the background variables, e.g., those linking genre and production quality.
In that spirit, if spans edited to generate counterfactually revised data (CRD) are analogous to the causal (or anticausal) variables, in the causal (or anticausal) graphs, then we might expect that nois- ing those spans (e.g. by random word replacement) should lead to models that rely more on non- causal features and perform worse on out domain data. On the other hand, we expect that noising unedited spans should have the opposite behavior, leading degraded in-domain performance, but comparatively better oft-o--domain performance. In the remainder of the paper, we investigate these hypotheses, finding evidence that qualitatively confirms the predictions of our theory.
5

---

Published as a conference paper at ICLR 2021
We freely acknowledge the speculative nature of this amalysis and concede that the mapping between the messy unstructured data we wish to model and the neatly disentangled portrait captured by our linear Gaussian models leaves gap to be closed through further iterations of theoretical refinement and scientific experiment. Ultimately, our argument is not that this simple analysis fully accounts for counterfactually augmented data but instead that it is useful abstraction for formalizing two (very different) perspectives on how to conceive of CAD, and for suggesting interesting hypotheses amenable to empirical verification.
4 EMPIRICAL RESULTS
If spans marked as rationales by humans via editing or highlighting are analogous to causal fea- tures, then noising those spans should lead to models that rely more non-causal features and thus perform worse on out-of-domain data, and noising the unmarked spans (analagous to non-causal features) should have the opposite behavior. In this section, we test these hypotheses empirically on real-world datasets. Additionally, we investigate whether the feedback from human workers is yielding anything qualitatively different from what might be seen with spans marked by automated feature attribution methods such as attention and saliency. Along similar, lines we ask whether CAD) in the first place offers qualitative advantages over what might be achieved via automatic sentiment- flipping methods through experiments with text style transfer algorithms.
We conduct experiments on sentiment analysis (Zaidan et al. 2007, Kaushik et al., 2020 and NLI YDung et al. 2020). All datasets are accompanied with human feedback (tokens deemed relevant to the label's ss applicability) which we refer as rationales. For the first set of experiments, we rely on four models: Support Vector Machines (SVMs), Bidirectional Long Short-Term Memory Networks (TM)) with Self-Attention (Graves Schmidhuber 2005), BERT (Devlin et al.) 2019), and Longformer (Beltagy et al. (020). For the second set of experiments, we rely on four state-of-the-are style transfer models representative of different methodologies, each representative of different approach to automatically generate new examples with flipped labels (Hu et al. 2017 Li et al. 2018 Sudhakar et al., 2019, Madaan et al. 2020). To evaluate classifier performance on the resulting augmented data, consider SVMs, Naive Bayes (NB), BiLSTMs with Self Attention, and BERT. We relegate implementation details to Appendix B
For sentiment analysis,
SVM, BiLSTM with Self Attention, BERT, and Longformer mod- els. In each document, we replace a fraction of rationale (or non-rationale) tokens with random tokens sampled from the vocabulary, and train our models, repeating the process 5 times. We per- form similar experiments for NLI using BERT. As an individual premise- rremise-hypothesis pair is often not as long as movie review, many pairs only have one or two words marked as rationales. To observe the effects from gradually injecting noise on rationales or non-rationales, we select only those premise-hypothee
pairs that have a minimum 10 tokens marked as rationales. Since no neu- tral pairs exist with 10 or more rationale tokens, we consider only binary classification setting (entaliment-cootraaciitoon))))) and downsample the majority class to ensure a 50:50 label split.
Figures 2 and 3 show the difference in mean accuracy over 5 runs. For all classifiers, as the noise in rationales increases, in-sample accuracy stays relatively stable compared to of accuracy. An SVM classifier trained on the original 1.7k IMDb reviews from Kaushik et al. (2220) obtains 87.8% accuracy on the IMDb test set and 79.9% on Yelp reviews [ As greater fraction of rationales replaced with random words from the vocabulary, the classifier experiences drop of zs 11% by the time all rationale tokens are replaced with noise. However, it experiences an 28.7% drop in accuracy on Yelp reviews. Similarly, the same datasets, a fine-tuned BERT classifier sees its in-sample accuracy drop by 18.4%, and by 31.4% on Yelp as rationale tokens replaced by noise go more non-rationales are replaced with noise, in-sample accuracy for SVM goes down by s 10% but increases by 1.5% on Yelp. For BERT, in-sample accuracy decreases by only 16.1% 1% and only 13.6% on Yelp (Also see Appendix Table 3 and Appendix Figure 4ap.
from 0 to 100%. However,
We obtain similar results using rationales identified via feature feedback. An SVM classifier trained on reviews from Zaidan al. (2077) sees in- in-sample accuracy drop by 11%, and accuracy on Yelp
'The out-of -fo-o-- evaluation sets in Kaushik et al. (020)) do not have 50:50 label split. We enforce this split to abserve when a classifier approaches random baseline performance. All datasets can be found

---

Published as a conference paper at ICLR 2021
In-sample test
CRD
Amazon
Semeval
Yelp
201
(a) Noising spans marked by humans
06.7700 DMC
2010
M6.0100
(b) Noising spans marked by Attention
2.50
06.5TH 544
D237
200
2E57
(c) Noising spans marked via gradient based feature attribution
Figure 2: Change classifier accuracy as noise is injected on rationales/non--ratioddd for IMDb reviews from Kaushik et al. 22020).
Mit Kilanahi
1980
Lonerreer
8L.THM000 (TMM/04
Longfrrerr
(a) Noising spans marked by humans
Ratonees
Nor Ratoneie
11/0
80.77 554
30.2
OLLTT//54
Longforeer
(b) Noising spans marked by Attention
G4.776 (44
1300
GLSTM/33
(c) Noising spans marked via gradient based feature attribution
Figure 3: Change in classifier accuracy as noise is injected on rationales/non aationaleshonn-nationeeess rationales for IMDb reviews from Zaidan et al. (2007). In both Figures and 3 the vertical dashed line indicates the fraction of median length of non-rationales equal to the median length of rationales.
drop by 16.9% as noise is inserted rationales but goes down by 17.3% and 14.6%, respectively when noise is inserted in non-rational rat-rioinnals. For Longformer, in-sample accuracy drops by 14% and accuracy Yelp goes down by 26.4% compared drop of 17.3% and gain of 3.9%. respectively,

---

Published as a conference paper at ICLR 2021
Table I: Accuracy BERT trained on SNLI (De Young et al. 2020 noise injected on human identified
rationalis.. RP and RH are Revised Premise and Revised Hypothesis test sets in Kaushik et 2220). MNLI-M and MNLI-MM are MNLI (Williams et 2018) dev sets.
Percent noise added train data rationales
Dataset
10
20
30
40
50
60
70
80
90
100
In-sample test
91.6
90.7
90.0
88.9
87.3
86.2
84.4
80.2
78.0
72.2
71.9
RP
72.7
70.7
69.1
67.1
65.7
62.4
61.8
57.7
55.6
53.8
51.4
RH
84.7
80.8
80.4
79.5
77.2
75.7
73.3
67.7
64.0
57.9
53.2
MNLI-M
75.6
74.7
73.9
72.0
70.6
69.1
64.7
59.1
55.8
54.4
53.3
MNLI-MM
77.9
76.7
75.6
73.9
72.3
70.8
65.6
58.4
55.1
53.6
52.5
Percent noise added
train data non-rationales
Dataset
10
20
30
40
50
60
70
80
90
100
In-sample test
91.6
91.4
91.3
90.9
90.8
89.9
89.0
88.7
87.8
86.7
85.4
RP
72.7
73.5
73.2
72.1
71.5
70.7
70.6
70.6
70.6
70.6
70.4
RH
84.7
83.6
82.6
81.9
81.3
81.1
80.5
79.8
79.4
79.4
79.2
MNLI-M
75.6
74.9
74.4
72.6
72.4
71.8
71.3
71.3
70.9
70.9
70.8
MNLI-MM
77.9
76.2
75.8
75.0
74.6
74.3
73.9
73.7
73.3
73.0
72.8
Table 2: Out-of-domain accuracy of models trained on original only, CAD, and original and
sentiment- flipped reviews
Training data
SVM
NB
BiLSTM (SA)
BERT
Accuracy on Amazon Reviews
CAD (3 (3.4k)
79.3
78.6
71.4
83.3
Orig. & Hu et al.
2017
66.4
71.8
62.6
78.4
Orig. &
Li at. 2018)
62.9
65.4
57.6
61.8
Orig. & Sudhakar et al.
(2019)
64.0
69.3
54.7
77.2
Orig. & Madaan et al. (2020)
74.3
73.0
63.8
71.3
Orig. (3.4k)
74.5
74.3
68.9
80.0
Accuracy on Semeval 2017 (Twitter)
CAD (3 4k:
66.8
72.4
58.2
82.8
Orig. & Hu et al.
2017)
60.9
63.4
56.6
79.2
Orig. & Li al. 2018)
57.6
60.8
54.7
62.7
Orig. & Sudhakar et al.
(2019)
59.4
62.6
54.9
72.5
Orig. & Madaan et al.
2020
62.8
63.6
54.6
79.3
Orig. (3. (3.4k)
63.1
63.7
50.7
72.6
Accuracy Yelp Reviews
CAD (3. 4k)
85.6
86.3
73.7
86.6
Orig. & Hu et al.
2017
77.4
80.4
68.8
84.7
Orig. & Li cl al.
2018)
67.8
73.6
63.1
77.1
Orig. & Sudhakar et al.
2009
69.4
75.1
66.2
84.5
Orig. & Madaan et al.
2020
81.3
82.1
68.6
78.8
Orig. (3.4k)
81.9
82.3
72.0
84.3
when noise is inserted in non ratiroationls Similar patterns are observed across datasets and models (see Figure 3al Appendix Table and Appendix Figure 5a
"Wiil similar trends are observed for both feature feedback and CAD, it is less clear how to incorporate feature feedback for training effectively with deep neural networks and pre-trained transformer architectures, whereas training (or fine-tuning) models CAD straightforward.
8

---

Published as a conference paper at ICLR 2021
For NLI, the in-sample accuracy of BERT fine-tuned an SNLI subsample drops by 20% when rationales are replaced with noise, and out- domain accuracy goes down by 21.3-31.553 on various datasets (Table TOP. Whereas, if hon-rationales are replaced with noise, in in-sample accuracy goes down by 6.2% but out-of-domain accuracy drops by only 2.3-5.5%. These results support our hypothesis that spans marked by humans as causing label are analogous to causal variables.
Interestingly, in our NLI experiments, for various models the drops in both in-sample and out-of- domain accuracy are greater in magnitude when noise injected in rationales versus when it is injected in non-rationales. This is opposite to what we observe in sentiment analysis. We conjec- ture that these results are due to the fact that in our experiment design for NLI, we only keep those premise-hypone hypothesis pairs that contain at least 10 tokens marked as rationales so we can observe the difference in accuracy as the amount of noise increases. A consequence of this selection is that many pairs selected have many more tokens marked as rationales than non-rationales, whereas, in senti- ment analysis this is the opposite. Hence, in NLI when some percentage of rationales are replaced by noise, this corresponds to many more edited tokens than when corresponding percentage of non-rationales are noised.
To compare human feedback to automatic feature attribution methods such as attention (Bahdanau jet al. 2015) and gradient based saliency methods (Li et al.) 2016), we conduct the same set of experiments assuming tokens attended to (or not) by an attention based classifier (BiLSTM with Self-Attentoon) or identified as highly influential by a gradient based feature attribution method (salience scores) as rationales (or non-rationales) In this case, unlike our findings with human feedback, we observe markedly different behavior than predicted by our analysis of the toy causal
model (See Figures 2b
2c 3b)
and 3ct and Appendix Tables 5 7 8)d
While we might not expect spurious signals to be as reliable out of domain, that does not mean that they will always fail. For example, while the associations between genre and sentiment learned from a dataset of book reviews might not hold in dataset of kitchen appliances, but nevertheless hold in a dataset of audiobook reviews. In such settings, even though noising non-causal features would lead to models relying more on causal features, this may not result in better out-of -ft-aan performance.
We also look at whether we really need to go through the process of collecting CAD (or human- annotated rationales) at all or if automated methods for generating "counterfactuall" might obtain similar gains in out-of-domain performance, as the former could be an expensive process. We exper- iment with state-of-the-arf style transfer methods to convert Positive reviews into Negative and vice versa. Ideally, we would expect these methods to preserve a document's 's "content" while modifying the attributes that relate sentiment (if they obtain perfect disentanglement in the feature space). Sentiment classifiers trained on original and sentiment-fiiped reviews generated using style transfer methods often give better out-of-domain performance compared to training only original data of same size (Table [). However, models trained on CAD perform even better across all datasets, hinting the value of human feedback.
5 CONCLUSION
While prior work offers promising clues to the benefits of CAD generated through human-in-the- loop mechanisms, previous work lacked formal frameworks for thinking about the technique, or comparisons to plausible alternatives. In this paper, through simple analysis on toy linear Gaussian models followed by a large-scale empirical investigation on sentiment analysis and NLI tasks, we formalize CAD and take some initial steps towards understanding its practical efficacy. Our anal- ysis suggests that data corrupted by adding noise to rationale spans (analogous to adding noise to causal features) will degrade out-oi- performance, while noise added to non-causal features may make models more robust out-of-domall and NLI and our findings remain consistent across datasets and models. Furthermore, the two tasks are subjectively very different as sentiment analysis requires a strong consideration of expressions of opinion than stated facts, whereas NLI is the opposite. We also show that models trained on the augmentation of original data and revised data generated by style transfer methods had better -ft-oa- generalization in some cases compared to models trained on original data alone, but performed worse than models trained on CAD. In future work, will look at how these findings generalize to other domains, including computer vision, and investigate the surprisingly low suscep- tibility of pre-trained transformers spurious associations.
Our empirical study focuses on sentiment analysis

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
The authors are grateful to NVIDIA for providing GPUs to conduct the experiments, Salesforce Research and Facebook AI for their financial support, and Sanket Mehta, Sina Fazelpour and Tejas Khot for our discussions and their valuable feedback.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.00 0.299,,, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In European Con- ference Computer Vision (ECCV), 2018.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
Steven L Blader and Tom R Tyler. What constitutes fairness in work settings? a four-component model of procedural justice. Human Resource Munagement Review, 13(1), 2003.
Danqi Chen, Jason Bolton, and Christopher D Manning. thorough examination of the cnn/daily mail reading comprehension task. In Association for Compulational Linguistics (ACL), 2016.
Rune Christiansen and Jonas Peters. Switching regression models and causal inference in the pres- ence of discrete latent variables. Journal of Machine Learning Research, (41), 2020.
Jeffrey Dastin. Amazon scraps secret ai recruiting tool that showed bias against women. Reuters, 2018.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (NAACL-HITT), 2019.
Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Associ- ation for Computational Linguistics (ACL), 2020.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston.
Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03842. 2019.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. In AAAI/ACM Conference on AI, Ethics, and Society (AIES). ACM, 2018
Angelos Filos, P. Tigas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? International Conference Machine Learning (ICML), 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 1rXiv:1811.12231,, 2018.
AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash, and Kun Zhang. Learning causal structures using regression invariance. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Max Glockner, Vered Shwartz, and Yoav Goldberg, Breaking nli systems with sentences that require simple lexical inferences. In Association for Computational Linguistics (ACI.), 2018.
10

---

Published as a conference paper at ICLR 2021
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6), 2005.
Nina Grgic- Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. Beyond dis- tributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. In AAAI, 2018.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In North American Chap- of the Association for Computational Linguisties: Human Language Technologies (NAACL- HLT), 2018.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning (ICML), 2017.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In North American Chapter the the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP), 2017. 2017
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv: 1711.11561, 2017.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. In Empirical Methods in Natural Language Pro- cessing (EMNLP), 2018
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a differ- ence with wtwprrfacttaally-ameeelte data. In International Conference on Learning Represen- tations (ICLR), 2020. htt.h::ts////tt.... /ttp::::://///evvviiiie eww. net. http//formmmiiiss. gg00FV ONF
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sen- timent analysis systems. In Joint Conference on Lexical and Computational Semantics (*SEM), 2018.
Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018
Jiwei Li, Xinlei Chen, Eduard Hovy and Dan Jurafsky. Visualizing and understanding neural models in NLP. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HITT) 2016. URI. https:/ /ttp://wwwww /ww. ac web. org/
antho ogy/N16-10300 1088
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: simple approach to sentiment and style transfer. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (LA), 2018
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentin sentence embedding. International Conference on Learning Representations (ICLR), 2017.
Zachary Lipton, Julian McAuley, and Alexandra Chouldechova. Does mitigating ml's 's impact dispar- ity require treatment disparity? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
11

---

Published as a conference paper at ICLR 2021
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. arXiv preprint arXiv: 2018.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. Politeness transfer: A tag and generate approach. arXiv preprint arXiv:2004. 14257, 2020.
Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. arXiv preprint arXiv:1909.0000 2019.
Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT), 2019.
David Miller. Justice. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Meta- physics Research Lab, Stanford University, fall 2017 edition, 2017.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 2019. URL
attps://scienn:
sciencemag org/ /trgg//ooneeett3////////
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshninarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances Neural Information Process- ing Systems (NeurIPS), 2019.
Judea Pearl. Bayesian netwerks: A model ef self-activated memory for evidential reasoning. In Proceedings of the 7th Conference of the Cognitive Science Sociery, University of California, Irvine, CA, USA, pp. 15-17, 1985.
Judea Pearl. Causality. Cambridge university press, 2009.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. The Journal of Machine Learning Research (JMLR), 12, 2011.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 5(78):947-0012, 2016.
Jonas Pfeiffer, Aishwarya Kamath, Iryna Gurevych, and Sebastian Ruder. What do deep networks like to read? arXiv preprint arXiv:I909.000
2019.
Adam Poliak, Jason Naradowsky. Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis Only Baselines in Natural Language Inference. In Joint Conference Lexical and Computational Semanties ( (*Sem), 2018.
Stefanos Poulis and Sanjoy Dasgupta. Learning with feature feedback: from theory to practice. In Artificial Intelligence and Statistics (AISTATS), 2017.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Association for Computational Linguistics (ACL), 2018.
Mateo Rojas-Carully Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for
causal transfer learning. The Journal of Machine Learning Research (JMLR), 19(1), 2018.
12

---

Published as a conference paper at ICLR 2021
B Schölkopf, D Janzing, Peters, E Sgouritsa, K Zhang, and J Mooij. On causal and anticausal learning. In 29th International Conference on Machine Learning (ICML 2012). International Machine Learning Society, 2012.
Judy Hanwen Shen, Lauren Fratamico, lyad Rahwan, and Alexander M Rush. Darling or babygirl? investigating stylistic bias in sentiment analysis. 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FATML), 2018.
Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. International Conference on Machine Learning (ICML), 2020.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. Transforming delete, retrieve, generate approach for controlled text style transfer. arXiv preprint 19Xiv:1908,09363 2019.
Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Envirouments: Introduction Covariate Shift Adaptation. MIT Press, 2012.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In International Conference Learning Representations (ICI.R). 2014.
Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. Learning what makes a difference from counterfactual examples and gradient supervision. arXiv preprint rrXiv:2004.09034, 2020.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for nlp. arXiv preprint 1rXivv1908.00
2019.
Sebastian Weichwald and Jonas Peters. Distributional robustness guiding principle for causality in cognitive neuroscience. arXiv preprint arXiv:2002.060660 2020.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL.T), 2018. URL
http::/ o Baapchhoooyy//11--11
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: ofte--of-the-ar natural language processing. ArXiv, pp. arXiv- 2019.
Sewall Wright. The method of path coefficients. The annals of mathematical statistics, 5(3):161- 215, 1934.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using "annotator rationales" to improve machine learning for text categorization. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HITT), 2007.
Omar F Zaidan and Jason Eisner. Modeling annotators: generative approach to learning from annotator rationales. In Empirical Methods in Natural Language Processing (EMNLP), 2008
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018.
Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data aug- mentation for mitigating gender stereotypes in languages with rich morphology. In Association for Computational Linguistics (ACL), 2019.
13

---

Published as a conference paper at ICLR 2021
A OLS ESTIMATION UNDER NOISY MEASUREMENT
A.1 CAUSAL SETTING
Let the Gaussian SCM be defined as follows where the noise term for variable is defined as tgg
Ug:
Us
N(0,
00 uz
T1 bz + M
Uu N(0,
o
(4)
I2 C2 + Uz2
Uxz -N(0,,
y 0.1 + Uy:
"y ~N(00 ouy) ).
a = 03
+ +ou
02 R2 é² 20 ou uga
T #2
bco 02
(5)
0 2 = ab du
+ ao 4x1
22.9 = abeo uz
2
Then if we were to solve the linear regression problem y = Tz + x2 + Bo. then using Eq. 1 we obtain the following values for 58 pppl and Bals.
02 221
(c + to (ab' o + + (.o (bco (abroo
8g
021
(6)
0 02 02 21.00
(b'o, + o (c"03 u, + to 6²c220
(bo0 + o (e'o +o 60000
(602, + +o (c + t 62200
gols 02, 22.8 - 021,22 022022 0222
(boor n. + to (abco? n.
(bco? Mz (ab² 0 + a
uzl
axi
0
2 221.02
(bo02 to (e2or, + ) b2 b2c2ou
(7)
However, if the setting is slightly different, and we observe a noisy version of æ1, given by Zi:
Ii I1 +
Ea ~ o0 fxl,
(8)
Since Cx. 1 (I1, T2, y).
on Varf#1 + Cni c= =bb=bbo
+ 2 usi
+ 02
On. = On,y = E((bz + Uz (ax) + uy)l = ab² - a n21
(10)
2 Oxiix = bco us
(11)
Plugging these values into Eq. 1 we get the OLS estimates Bpla and Bofs in the presence of observa- tion noise on X1:
0²
(2og o
ao2 (nn) (bco 2, (abco )
Bplx
2 02
22
(b'0² + to + +02 (c²o0u to? - b
21.22
a 0b20
202 )nn) + o 02
a oboo uaz
+ e 0
+ 0 Jun t (c o2 + t )2r))
gole
1+>
65) (c202 + to2
A_
22 0b202 Ua2
+ 2002
+ 2 02
Uzl Ux2
Bols
4
0
T T
(b 12
+ 0 un + 02 )abco" a, - (beo (ab²o + ao un
u.
2 22
2 a,
2002 o 2 du
2 (c20 + )
B1
2122
ux
ocho
2 o2
6x1
00
11' 2
+ 2 2
+ 2 2u + 2
(22 )
Uz2
(12)
14

---

Published as a conference paper at ICLR 2021
As we can see xe > and de & on.. This shows us that as o... increases, |89 (magnitude of the coefficient for X1) decreases and | (magnitude of the coefficient for X2) increases.
lim B = O, and lim 3g oo +0
acda
A.2 ANTICAUSAL SETTING
Once again assume that each variable a linear function of its parents Pa(V). The noise terms are assumed to be Gaussian and are jointly independent.
U2 _N(0 o
Ug
Uu NO0 oug
9 = az + Ug.
y bz + tyy
"y
-N(0, o
(13)
Z2 = cq + Ug2
Uri N[0 o
21 dy + U2i
Ug2 NNO,,
2 = Pbr0z
+ +oou + uzi
20²02
+ 202 + t
22 u.
01 abedo az
(14)
ONy db'o oi +do
0.2 = abco u2
If solve the linear regression problem y 31 + 22 Bo then using Eq. 1 we get the OLS estimates 3ols and sa1s:
2 J. Oz Tzy.y
3.
22
02 02 02
(ca2on +co0, + + t (db'o 02, + do (abcdo (abcoz.))
PVo? + too too (ca + coo + (a 2)
aa'"Co' oz + (c20 + to
+ (bb20' + +o²
(dbo' + too + doo o (o + coo + (o + d oz jeaa00
(15)
0 J22N Oz 02
2 02 0 0x1 2
(1 us + ooo dz + 2 Mzl
(abco - (abedo (dDP + +do
(d + doo + to (c2a202 + cc002 + 02 (a2b2c2doo))
abco? 02
(dRo2 +o2 + d ( + 2 + (o2 + d'o "e=a))²aaa" a²aa0
However, if the setting is slightly different, and we observe a noisy version of Ii, given by Z1:
1 σ1 Cx.
G, ~(0.0
(16)
Since Ear IL 222 y, in order to obtain expressions for the OLS estimates gols, Bol in the presence of observation noise, in Eq. 15 we only need to replace o² with o u.g. which is given by:
22
0 Mat
+ cx1
(17)
am
15

---

Published as a conference paper at ICLR 2021
dla of + (6o2, to to (b'0 to2.))
89
(d + th tdo tco - logg do
da'too dn + (C"o + + (B'6 + +o²,))
(PRo + to u2l + +o + d' My
+ tco (d
+ (lo + to + d 0 a
)a
(18)
abco u. uz
(dbb' to + + (o + & (oaz, + 2
abco (oan + +o²a)
us
(dPbo2 + + + to. + d Myy (o +coo ) (loh + + + d Paaoo
(19)
Bols
321
87
Bals
11
(20)
1 + X
1 Xae
2 (c2a20
us + 22002 + +
bz + o + doo (o 822 + +o02 + (oba. + Po 2 uy A (di1022
cay
(21)
)e
where Aot 0 and Not x oe.. Thus, o2 increases, | decreases. The asymptotic OLS
estimates in the presence of infinite observational noise can be seen to be: 0 4000 (d Mo teto2 (o toPoz a
lim
gola 0, where
as lim Bole = Bot (ddPbo2 to t +d
(02g, (ctatog, -ooa +02,2))
Similarly, if we observe a noisy version of X2, given by X2:
T2 = x2 + Ez2i
(2 N(0,02..)
(22)
Since Egg Il 11, y, in order to obtain expressions for the OLS estimates Sole, Bole in the presence of observation noise non-causal features, in Eq. 15 we only need to replace of with of which given by:
uaz
"m
0
2 + +o?
(23)
darooo ot + (c200 + o 10o' + tory)) (P/1oo2 + t - ) + + t02 +
Sqls
)au002 )
da2200 u, of + (c202 + (o + +o () + +
(24)
idbb'r. + to Lo ((o + ).n)) + -co + to + d coaad
abco² 02
agll
(25)
(d'o' + oa + do 1o
+ + (
+ (0 + je
abco us 2
(26)
od + t + tooz (102
+ t
+ 2 20, + to2 + too a
gpls
2 2 (boo + 2z)) 1+ a2c202 02 + (c222 + + (122 + +o2
3gls
Bols-
Bgls
1 + Naa
+ Xa2
(27)
o2 (17300 uu + to + too
x
(d.bot, + to? + &oo + tco + t02 + +do? c
16

---

Published as a conference paper at ICLR 2021
where Aue 0 and Xae oc oe Thus, as de.. increases, | increases.
The asymptotic OLS
estimates in the presence of infinite observational noise can be seen to be: lim
3 0, where
as lim Bp 3gle (b"ol, to ((d"b'o to agl +o 1o to t mas t 6e (a'cta Buy tlc"da to 1b to ba too +d
700
17

---

Published as a conference paper at ICLR 2021
B MODEL IMPLEMENTATION DETAILS FOR SECTION 4
Standard Methods We use
(Pedregosa et al. 2011 implementations of SVMs and Naive Bayes for sentiment analysis. We train these models on TF-IDF bag of words feature representations of the reviews (Jones] 1972). We identify parameters for both classifiers using grid search conducted over the validation set.
BiLSTM We restrict the vocabulary to the most frequent 20k tokens, replacing out-of-vocabular tokens by UNK. We fix the maximum input length at 330 tokens when training on reviews from Kaushik et al. (2020) and 2678 when doing on Zaidan et al. (2007), and pad smaller reviews. Each token is represented by a of bidirectional LSTM (hidden dimension 128) with recurrent dropout (probability 0.5) and self attention following the embedding layer. We use the self attention implementation discussed in Lin et al. (2017) with hyperparameter values d 64 and 64. To generate output, we feed this (fixed-length) representation through a fully-connected hidden layer (hidden dimension 32), and then fully-connected output layer with softmax activation. We train all models for maximum of 20 epochs using Adam (Kingma & Ba (015)) with learning rate of 1e-4 and batch size of 16. We apply early stopping when validation loss does not decrease for epochs.
300-dimensional embedding. Our model consists
Pretrained Transformers We use off- uncased BERT Base and Longformer Base models (Wolf et al. 2019). sine-tuning for each task. We used BERT for experiments on the smaller IMDb dataset used by Kaushik et al. (2020) (with maximum review length of 330 tokens) and Longformer for the dataset presented by Zaidan et al. (2007) (with maximum review length of 2678). To account set the maximum token length is set at 350 for sentiment analysis and 50 for NLI. In case of Longformer, that is 3072 3 We fine-tune BERT up to 20 epochs with same early stopping criteria as for BiLSTM, using the BERT Adam optimizer with batch size of 16 (to fit on 16GB Tesla V-100 GPU). We found learning rates of 5e-5 and 1e-5 to work best for sentiment analysis and NLI respectively. We fine-tune Longformer for 10 epochs with early stopping, using a batch size of 8 (to fit on 64GB of GPU memory).
for BERT's sub-word tokenization,
Style Transfer Methods Hur et al. 2017 Sudhakar et al. (011 and Madaan et al. 2020 found the default hyperparameters used by the authors to work best our task. In case of Li ct al. 20188) we followed the training schedule presented in the paper. However, since the paper does not present results IMDb reviews,
experimented with multiple values of the salience ratio, and used a salience ratio of 5.5 for our downstream task based on transfer accuracy and BLEU scores achieved on the validation set. For all style transfer methods, experimented with multiple sequence lengths, and found that models worked best on sentence level (versus review-level) data, with sequence length of 30, truncating longer sentences the process. For each review, we passed individual sentences through each model and reconstructed whole reviews by joining the resulting sentiment foippeent-fliped sentences.
Loogfoommer is better suited to work on longer texts compared BERT. Maximum length of a review in Zaidan et al. is 2678 tokens whereas in Kaushik et al. is only 330 tokens.
https://githhbb.o http:://hhhh....m//////////s///a homps//wwleeatttcemssatr/a/aacee
https://github http://hhw...ot///////////////ee style transfer
18

---

Published as conference paper at ICLR 2021
FULL RESULTS CORRESPONDING CONNESPONDING TO NOISE INJECTION
C
Table 3: Accuracy of various sentiment analysis classifiers trained on 1.7k original reviews from Kaushik et al. 2020) noise injected rationalee//nnn-rriionee identified via human feedback.
Dataset
Percent noise in rationales
SVM
10
20
30
40
50
60
70
80
90
100
In-sample test
87.8
88.2
85.7
86.9
86.9
84.5
83.3
81.6
80
79.2
76.7
CRD
51.8
47.3
45.7
42.9
39.2
33.5
28.2
25.7
24.1
19.6
17.1
Amazon
73.2
72.2
71.3
69.4
67.3
63.7
63.7
58.2
57
50.1
46.5
Semeval
62.5
62.2
61.9
61.1
60.9
58.3
57.1
55.4
54.5
51.3
50.1
Yelp
79.9
79
77.7
76.7
74.1
71.4
69
65.5
62.4
55.8
51.5
BiLSTM with Self Attention
In-sample test
81.5
78.8
77.6
76.7
75.3
75.2
74.5
72.8
67.3
64.2
63.8
CRD
49.4
49.3
46.3
45.1
39.5
38.1
38.9
38.7
32.6
32.6
29.7
Amazon
65.4
69.1
68.5
66.6
63.2
63.9
58.8
50.6
50.6
47.1
44.2
Semeval
59.3
59.8
57.6
56.4
58.6
56.6
55.3
54.3
54.3
52.3
50
Yelp
71.2
70.8
67.4
65.9
65.3
64.1
63.4
60.1
62.4
49.8
46.4
BERT
In-sample test
87.4
87.4
86.5
85.7
85.3
84.3
83.6
81
76.6
71
69
CRD
82.2
78.1
78.4
75.4
67.6
67.5
65.5
53.9
42.7
36.2
31.8
Amazon
76.2
75.5
75.1
74.2
73.5
73
72.5
70.7
63.4
57.8
56.1
Semeval
76.4
69.7
66.9
69.8
67.8
67.4
66.8
65.5
62.2
54.9
52.6
Yelp
83.7
82.5
82
81.5
80.9
80.2
79.9
75.6
64.3
54.6
52.3
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.8
88.6
89
86.9
85.3
82.4
86.5
83.7
81.6
78
CRD
51.8
55.9
53.5
57.1
58.8
63.7
63.3
65.7
70.2
73.9
74.3
Amazon
73.2
74.9
75.3
77.3
75.8
76.6
76.5
77.4
75.5
75.4
76.9
Semeval
62.5
63.3
62.7
64.3
64.3
65.6
66
65.8
65
66.4
66.4
Yelp
79.9
80.9
80.1
82.2
83.6
84.1
83.5
83.4
82.7
82.1
81.4
BiLSTM with Self Attention
In-sample test
81.5
77.5
77
75.9
75.4
75.2
75.1
73.8
73
72.4
71.7
CRD
49.4
56.25
56.6
57.5
58.4
58.6
60.3
61.5
65.5
66.1
Amazon
65.4
66.5
66.6
66.6
67.6
67.7
68.3
68.6
68.8
68.5
68.4
Semeval
59.3
58.6
58.9
59.3
58.1
57.5
59.2
59.5
59.8
59.5
Yelp
71.2
74.7
72.5
73.3
73.9
73.6
72.2
74.3
73.7
75.6
75.4
BERT
In-sample test
87.4
88.2
87
86.9
87
85.8
83.6
78.9
72.5
72.1
71.3
CRD
82.2
92.8
92.8
92.3
93.1
92.8
89.8
88.6
84.5
81.3
81
Amazon
76.2
78.6
78.9
79.2
75.1
71.7
67.6
65.3
65.2
63.7
61.8
Semeval
76.4
74.6
76.3
75.8
70.9
62.1
64.8
63.3
60.8
58.7
58.7
Yelp
83.7
85.4
85.3
85.1
82.1
78.3
77.2
76.2
74.3
71.6
70.1
19

---

Published as a conference paper at ICLR 2021
Table 4: Accuracy of various sentiment analysis classifiers trained on 1.7k original reviews from
Kaushik et al. (2020) as noise injected on rationalessmon--rationnkk. identified via Attention masks.
Dataset
Percent noise in rationales
SVM
10
20
30
40
50
60
70
80
90
100
In-sample test
87.8
85
85.9
86.3
86.3
85.2
84.6
86.3
83.6
84.2
83.6
CRD
51.8
50.6
51.8
52
51.8
50
50.6
48.6
48.6
47.5
46.1
Amazon
73.2
74.3
73.4
72.8
72.8
72.9
72
72.3
71.1
72
70.3
Semeval
62.5
62.8
62.9
61.8
62.5
61.9
61.4
60.7
61.1
60.6
60.1
Yelp
79.9
80.1
79.3
78.7
78.9
78.5
77.8
77.5
77.8
76.2
75.9
BiLSTM with Self Attention
In-sample test
81.5
78.8
78.6
78.3
78.2
76.2
77.3
76.8
71.8
73.2
74.2
CRD
49.4
53.3
50
53.4
52.4
49.7
49.2
47.4
47.7
47
44.1
Amazon
65.4
66.8
71
64.7
60.7
61.7
65.2
64.6
51.6
57.1
66.4
Semeval
59.3
59.5
60.1
57.4
55.9
57.2
52.2
57.6
51.5
51.8
56.1
Yelp
71.2
72.3
74.2
69.6
70.5
67.3
70.7
72.8
62.8
65
66.2
BERT
In-sample test
87.4
93
90.8
90.3
90.6
91.2
90.3
90.4
90.7
90.6
90.3
CRD
82.2
91.2
92
90.8
90.8
90.9
90.3
90.9
90.2
89.8
90.4
Amazon
76.2
77.3
79.1
78.7
79.8
79.1
79.8
79.5
79.2
78.9
79.3
Semeval
76.4
71.4
73.5
73.2
74.4
76.1
77.6
79.8
78.4
79.2
77.8
Yelp
83.7
83.5
85.4
84.9
86
85.7
85.9
85.6
85.5
85.4
68.9
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.8
85
85.7
84.8
85
84
83.6
84.6
80.7
81.1
77.3
CRD
51.8
50.4
52.2
53.9
50.2
50.8
52.9
54.1
54.1
56.8
56.4
Amazon
73.2
73.5
75.3
74.3
76.2
73.9
73.4
73.6
71
70
67.8
Semeval
62.5
62.6
63.7
63.7
63.1
62.6
63.5
61.5
62.1
62
59.9
Yelp
79.9
79.8
80.9
81.7
80.9
80.5
80
80.1
78.5
77.5
74.4
BiLSTM with Self Attention
In-sample test
81.5
77.6
76
77.1
77.3
75.4
73.7
67.9
68.6
54.2
52.3
CRD
49.4
53.1
52.1
52.1
65
54.1
51.9
53.4
55
52.3
51.6
Amazon
65.4
63.7
65.7
64
58.8
65.5
60.3
58.7
61
58.1
56.2
Semeval
59.3
54.8
58.4
57.3
60.7
56.8
55.2
54
51.2
50
49.9
Yelp
71.2
72
73.6
70.2
61.3
71.5
68.4
64.9
66.3
58.2
55.8
BERT
In-sample test
87.4
86.9
86.7
85.3
84
81.9
80.6
74
74
73
67.2
CRD
82.2
92.3
92.4
92.1
90
86.8
83
73.2
77.7
72.5
68.5
Amazon
76.2
79.5
78.5
77.9
69.2
67.4
58.1
55.9
53.5
55.8
52.6
Semeval
76.4
76.5
75.7
77.1
65.7
61.8
54.6
58.8
51.8
54
50.8
Yelp
83.7
85.8
85
85.5
79.3
78.7
67.8
66.5
59.5
63.2
57.5
20

---

Published as a conference paper at ICLR 2021
Table 5: Accuracy of various sentiment analysis classifiers trained on 1.7k original reviews from Kaushik al. (2020) noise is injected rationales/hon--raiionna identified via Allen NLP Saliency Interpreter.
Dataset
Percent noise in rationales
SVM
0
10
20
30
40
50
60
70
80
90
100
In-sample test
87.8
85.1
85.4
85.1
85.1
83.9
82.5
82.8
81.8
80
77.5
CRD
51.8
51.2
52.5
51.1
51
50.1
49.5
46.6
43.7
42.1
40.5
Amazon
73.2
73.4
73.65
73.2
72.7
72.9
71.8
72.1
70.5
69.6
68.9
Semeval
62.5
62.8
62.5
62.4
61.9
61.2
60.7
60.5
59.6
58.4
57.9
Yelp
79.9
79.8
79.7
79.1
78.7
78.2
78.1
76.6
75.1
74.1
72.2
BiLSTM with Self Attention
In-sample test
81.5
82.3
83.5
80.4
78.2
81.9
80.6
77.8
79.2
76
77
CRD
49.4
48.2
48.6
51.2
48.6
47.3
47.1
46.9
44.3
42.6
37.9
Amazon
65.4
46.6
72.8
66.9
49.7
55.4
53.7
68.5
54.7
49.8
51.8
Semeval
59.3
42.1
49.5
56.2
54.7
52.7
53.7
50.1
51.2
50.2
50
Yelp
71.2
69
73.3
73.2
67.8
69.2
69.5
68.8
54.4
56.9
BERT
In-sample test
87.4
91.1
90.6
90
88
89.1
87.4
86.3
83.6
84.5
81.6
CRD
82.2
93.4
92.3
91.9
90.3
90.2
87.7
83.8
79.3
70
Amazon
76.2
82.4
81.3
79.8
77.2
77.6
77.8
75.6
69.7
69.4
73.6
Semeval
76.4
82.6
82.8
81.3
79.2
78.1
76.7
74.7
67.4
65.8
67.4
Yelp
83.7
88.3
88.8
88.5
87.8
88.1
87
86.2
84.4
83.3
82.7
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.8
85.9
85.7
86.9
83.6
86.9
85.9
83.2
85
81.8
79.1
CRD
51.8
52.3
53.7
53.9
56.8
55.3
53.5
54.3
58
60
61.5
Amazon
73.2
73.9
74.1
71.8
73.6
72.5
73.8
72.6
70.6
70.6
70.8
Semeval
62.5
62.7
62.8
61.3
62.7
62
61.9
63.2
62.3
62.4
63.6
Yelp
79.9
79.8
79.8
81.4
81
80.7
80.5
80.3
79.8
78.6
BiLSTM with Self Attention
In-sample test
81.5
81
81.7
80.8
79.8
78
75.6
73
70.4
51
50
CRD
49.4
49
49.8
47.9
51.6
46.7
53.3
50.2
51.6
48.4
Amazon
65.4
65.3
64.9
62.7
63.3
65.3
67.1
65.3
64
58.3
41.8
Semeval
59.3
55
61.3
50.1
54.6
58.5
55.2
55.7
49.4
49.6
Yelp
71.2
73.8
75.1
71.4
74.1
73.4
74.5
72.5
66.9
55.9
53.6
BERT
In-sample test
87.4
90.5
89.1
88.6
80.6
75.1
70.1
63.7
53.8
54.1
53.1
CRD
82.2
92.1
92.2
91.3
79.9
73.3
67.1
59.2
50.1
49.8
49.6
Amazon
76.2
77.5
79.2
77.3
69
65.9
61.1
61.7
52.9
52.7
51.4
Semeval
76.4
82
83.6
83.1
78.1
77.9
71.1
69.7
55.9
56.5
51.8
Yelp
83.7
88
87.4
87.8
76.8
73
66.9
66.3
55.3
54.4
53.1
21

---

Published as a conference paper at ICLR 2021
Table 6: Accuracy various sentiment analysis classifiers trained on reviews from Zaidan et al.
2007) as noise is injected rationales/mon--rttinnca identified via human feedback.
Dataset
Percent noise in rationales
SVM
10
20
30
40
50
60
70
80
90
100
In-sample test
87.5
86.2
85.5
85
84.5
83.3
82.5
81.1
78.9
77.5
76.5
CRD
46.1
45.6
44.4
43.7
44.1
41.2
38.8
34.4
33.1
30.9
Amazon
68.6
67.1
65.1
64.2
62.2
60.4
57.9
50.5
54.9
53.5
51.8
Semeval
56.7
56.1
55.4
54.8
54.1
53.5
52.7
51.6
50.8
50.4
Yelp
76.2
75
73.5
72
70.2
68.8
66.6
65.1
63.3
61.1
59.3
BiLSTM with Self Attention
In-sample test
80.3
82.1
83.2
81.3
78.4
71.1
78.8
77.4
76.9
77.4
75.5
CRD
49.2
50.6
51
48.8
48
49.6
49.4
48.8
48.8
47.5
48.4
Amazon
50.5
49.4
49.7
49.8
49.7
49.7
49.7
49.6
49.5
49.4
Semeval
50
50
50
50
50
50
50
50
50
50
50
Yelp
50.5
50
53.1
52.1
50.5
50.2
50.1
50
50.2
50.1
Longformer
In-sample test
97.5
96.7
94
90.5
88.3
78.9
81.4
72.6
79.4
78.7
83.5
CRD
93.4
93.6
87.5
85.4
84.2
64.1
61.5
54.2
52.7
50.3
48
Amazon
81.8
77.9
65.3
65.7
64.7
63.6
61.9
62.1
61.3
60.6
57.9
Semeval
80.3
74.9
64
66.9
71.6
61.3
58.4
56.7
58.9
62.1
58.6
Yelp
88.6
85.8
77.7
74.6
72.5
68.4
66.5
64.8
64.3
64.9
62.2
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.5
85.5
83
82
81
80.5
75.5
60
CRD
46.1
46.1
49
49.4
57.1
55.5
58.4
58.4
56.5
56.3
54
Amazon
68.6
67.7
67.2
69.4
69.7
68.9
69.2
64.9
62.3
Semeval
56.7
56.9
57.5
57.4
58.3
57.6
58.8
59.4
59.3
57.4
56.3
Yelp
76.2
76.1
76.9
75.9
77
77.4
75.2
74.1
73.3
68.5
61.6
BiLSTM with Self Attention
In-sample test
80.3
80.8
79.8
75.2
75
62.5
62
57.7
56.7
58.7
57.7
CRD
49.2
50
50.8
52.9
53.9
58.6
58.6
60
60.4
60.8
Amazon
50
50
50.7
50.7
50.9
52.2
52.3
53.2
55
55.1
56.7
Semeval
50
50
50
50
51.8
52.7
53.5
53.8
53.9
Yelp
50.5
50.4
52.7
52.9
52.9
55.2
58
58.9
64.6
64.6
70
Longformer
In-sample test
97.5
97.9
98.1
97.4
94.8
93.4
86.4
82.3
76.3
77.4
80.2
CRD
93.4
94.7
94.1
91.8
91.4
91.8
88
83.4
83.7
83.6
83.4
Amazon
81.8
79
80
81.5
83.2
84.2
84.1
76.3
78.5
79.4
76.9
Semeval
80.3
79.4
77.2
80.6
80.6
84.6
85.3
71.8
79.9
83.7
76.6
Yelp
88.6
85.3
86.4
89
89.5
89.9
89.9
86.2
86.5
86.4
84.7
22

---

Published as a conference paper at ICLR 2021
Table 7: Accuracy various sentiment analysis classifiers trained on reviews from Zaidan et al. 2007 as noise is injected rationales/mon--rttinnadd identified via Attention masks.
Dataset
Percent noise in rationales
SVM
10
20
30
40
50
60
70
80
90
100
In-sample test
87.5
85
84.5
84
82.5
83
81
80
77.5
75.5
75.5
CRD
46.1
51
50.6
52
51.8
52.3
52.3
51.8
50.2
49.8
49.8
Amazon
68.6
68.1
67.1
66.8
66.9
66.5
66.2
65.4
66.1
66.6
65.7
Semeval
56.7
56.6
56.3
56.4
56.2
56.4
56.4
56.2
56.8
56.4
56.4
Yelp
76.2
76.1
76
76.2
76.4
76.5
76.9
76.9
76.7
76.9
76.5
BiLSTM with Self Attention
In-sample test
80.3
78.8
77.9
77.9
78.8
67.3
65.9
63.9
62
65.4
58.7
CRD
49.2
49.4
50.2
50.2
52.1
51
52.1
52.3
56.3
51.8
54.7
Amazon
49.7
49.9
49.9
50.4
50.2
51
51.7
51.1
50.7
50.7
Semeval
50
50
50
50
50
50
50
50.2
50.1
50
50.1
Yelp
50.5
50.1
50.5
50.5
52.1
52.4
56.1
54.9
54.9
52.2
54.9
Longformer
In-sample test
97.5
97.3
97
96.5
88.3
94
93.8
91.2
91.5
87.2
84
CRD
93.4
93.5
93.1
92.8
91.7
91.8
90.7
87.5
83.7
80.8
Amazon
81.8
76.3
69.5
75.4
70.4
64.5
66.3
60.8
64.7
57.3
55.3
Semeval
80.3
73
67.2
75.1
69.6
61.5
67
58.8
67.6
56.4
55.3
Yelp
88.6
85.1
79.3
83.9
79.8
75.4
76.8
69.1
75.4
65.7
61
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.5
87
86.5
87.5
82.5
73
50
50
CRD
46.1
50.4
49.6
48.6
50
46.9
50.6
49.6
50.4
50.2
50.2
Amazon
68.6
66.7
66.8
64.1
65.9
63.2
62.2
57.8
56.2
56.3
Semeval
56.7
56.3
56.8
55.9
56.7
55
54.2
53.8
51.8
51.1
51
Yelp
76.2
74.8
74.2
71.1
71
64.9
59.7
55.2
52.3
51
50
BiLSTM with Self Attention
In-sample test
80.3
79.8
81.3
78.4
63.5
67.3
49.5
49
48.1 1
48.4
48 1
CRD
49.2
51.4
51.4
54.5
49.8
49.4
49.6
49.4
49.4
49.4
49.4
Amazon
50
49.9
50.6
50.4
50.1
49.7
49.6
49.5
49.5
49.5
49.5
Semeval
50
50
50.2
50
50
50
50
50
50
Yelp
50.5
52.3
52.7
56.9
51
50.4
50
50
50
50
50
Longformer
In-sample test
97.5
98.2
97.8
95
90.2
83.3
67.3
62.8
69.3
64.2
52.8
CRD
93.4
93.6
93.5
88.8
83.1
76.5
67.8
69.7
77.6
54.5
51.4
Amazon
81.8
81.6
97.8
95
90.2
83.3
67.3
62.8
79.3
64.2
52.8
Semeval
80.3
74.8
70.3
79.1
79
78.9
69.5
67.9
64
63.3
58.6
Yelp
88.6
83.9
83.1
89.5
90.2
89.7
87.6
83
78.8
62.4
59.4
23

---

Published as a conference paper at ICLR 2021
Table 8: Accuracy of various sentiment analysis classifiers trained on reviews from Zaidan et al. (007) noise is injected on rationaleshoon-ratiinn rationales preter.
rattionnss identified via Allen NLP Saliency inter-
Dataset
Percent rationales tokens replaced by noise
SVM
10
20
30
40
50
60
70
80
90
100
In-sample test
87.5
85
84.5
84
82.5
83
81
77.5
75.5
75.5
CRD
46.1
51
50.6
52
51.8
52.3
52.3
51.8
50.2
49.8
49.8
Amazon
68.6
68.1
67.1
66.8
66.9
66.5
66.2
65.4
66.1
66.6
65.7
Semeval
56.7
56.6
56.3
56.4
56.2
56.4
56.4
56.2
56.8
56.4
56.4
Yelp
76.2
76.1
76
76.2
76.4
76.5
76.9
76.9
76.7
76.9
76.5
BiLSTM with Self Attention
In-sample test
80.3
83.2
78.1
76.9
73.6
80.3
81.7
76.4
76.4
74
74.5
CRD
49.2
49.8
50.6
50.8
50.8
49.2
49.2
49.2
52
49.4
49.8
Amazon
50
49.8
50.5
49.8
50
49.7
49.7
50.1
50
50.3
49.8
Semeval
50
50
50
50
50
50
50
50
50
50
50
Yelp
50.5
50.4
50
50.5
50.8
50.3
50.1
50.9
50.7
50.8
50.5
Longformer
In-sample test
97.5
98
97.5
97.5
97
97
97
96.5
94.3
92.8
CRD
93.4
93.4
93.9
94
92.4
91
92.2
91.7
90.2
86.6
74.5
Amazon
81.8
81
74.2
66.3
74.7
78.3
80.6
76.2
63.2
77.3
55.8
Semeval
80.3
79.9
69.4
64
73.4
77
78
74.6
60.6
78.3
56.4
Yelp
88.6
87.3
84.5
76.6
83.1
86.4
87.6
85.6
72.8
84.1
61.7
Dataset
Percent noise in non-rationales
SVM
In-sample test
87.5
79.7
79.9
79.5
81.1
79.9
80.3
78.9
78.7
79.3
73.4
CRD
46.1
52.7
52
53.1
50
54.3
50.6
54.3
52
57.2
57.2
Amazon
68.6
68.1
66.2
67
65.8
68.8
65.3
64.4
65.3
63.1
61.9
Semeval
56.7
57.4
56.2
56.9
55.9
57.3
55.6
58.1
57
57.9
58
Yelp
76.2
76.4
75.4
76
75.6
75.8
74.2
74.3
73.6
73.7
71.5
BiLSTM with Self Attention
In-sample test
80.3
76.9
80.8
79.3
78.8
77.9
76
76
63.5
73.6
66.8
CRD
49.2
50
50.2
50.4
50.2
50.8
51.4
47.9
48.8
47.5
48.4
Amazon
50
50
49.7
50.1
50.2
50.8
50.2
50
50.3
49.8
47.9
Semeval
50
50
50
50
50
50.1
50
50
50
50
50
Yelp
50.5
50.5
50.3
51.3
54.5
54.9
52.2
51.4
52.7
50.5
54.9
Longformer
In-sample test
97.5
97.8
98
97.8
97.5
98.3
95
92.8
84.5
83.5
74.5
CRD
93.4
94.4
94.1
93.6
93.1
93.3
92.8
91
86.9
70.9
67.6
Amazon
81.8
80.9
75.9
75.8
79.7
68.9
81.4
72.4
71.2
63.5
55.2
Semeval
80.3
78.6
72.7
74.4
79.1
68.9
81.6
73.5
76.2
59.2
55.7
Yelp
88.6
88.1
84.1
84.8
87.5
81.3
89.3
82.2
82.2
70.3
61.5
24

---

Published as a conference paper at ICLR 2021
25
35
8
0212
105
45
Noising spans marked by humans
3
Parae
(b) Noising spans marked by Attention
500
22
30
1500
190 18
1500
00
00
(c) Noising spans marked by AllenNLP Saliency Interpreter
Figure 4: Change in classifier accuracy as noise is injected on rationales (in blue) or non-rationales (in red) for IMDb reviews from Kaushik et al. (2020). The vertical dashed line indicates the fraction of median length of non- non-rationales equal to the median length of rationales.
25

---

Published as a conference paper at ICLR 2021
3 8
Noising spans marked by humans
DN
$
22 1000 8
5
(b) Noising spans marked by Attention
1900
40
150
(c) Noising spans marked by AllenNLP Saliency Interpreter
Figure 5: Change in classifier accuracy as noise injected on rationales (in blue) or non-rationales (in red) for IMDb reviews from Zaidan et al. (2007). The vertical dashed line indicates the fraction of median length of non-rationales equal to the median length of rationales.
26

---

Published as a conference paper at ICLR 2021
pi yer talad
(a) Trained on the original 1.7k IMDb reviews in Kaushik ct al.
2020
(b) 20% noise in rationales
(c) 20% noise in non-rationales
(d) 40% noise in rationales 
(c) 40% noise in non-rationales
(f) 60% noise in rationales
(g) 60% noise in non-rationales
piter aeonalll ip paserrati quteiii Aenooaadd " Mennrnnttttt
(h) 80% noise rationales
(i) 80% noise in non-rationales
(k) 100% noise in non-rationales Cenassaaa
in pn unll pyjice L
ii) 100% noise rationales
Figure 6: Most important features learned by an SVM classifier trained on TF-IDF bag of words. Rationales are identified by humans.
27

---

Published as a conference paper at ICLR 2021
wiirarannna yar talad lell aillaaa
al
(a) Trained on the original 1.7k IMDb reviews in Kaushik ct al. 2020
1 Ae Slllhiee Aaaaaa an loll aeee
(b) 20% noise in rationales
(c) 20% noise in non-rationales
Aaaaiia o marde dedeetnttii ald battll wonteeatttttiieeeeee
40% (d)
in noise
rationales
(c) 40% noise in non-rationales
(f) 60% noise in rationales
(g) 60% noise in non-rationales
rationales in noise 80% (h)
(i) 80% noise non-rationales
pou mp
(i) 100% noise in rationales
(k) 100% noise in non-rationales
Figure 7: Most important features learned by an SVM classifier trained on TF-IDF bag of words. Rationales are identified as tokens attended upon by a BiLSTM with Self Attention model.
28

---

Published as a conference paper at ICLR 2021
di yer talaa
(a) Trained on the original 1.7k IMDb reviews in Kaushik ct al.
2020
(b) 20% noise in rationales
(c) 20% noise in non-rationales
(d) 40% noise in rationales
(c) 40% noise in con-rationales
(f) 60% noise in rationales
(g) 60% noise in non-rationales
de Allooite Aalleaae Mell aitie Siiii las
(h) 80% noise rationales
(i) 80% noise in non-rationales
(i) 100% noise in rationales
(k) 100% noise in non-rationales
Figure 8: Most important features learned by an SVM classifier trained on TF-IDF bag of words. Rationales are identified as tokens marked by the AllenNLP Saliency Interpreter.
29

---

Published as a conference paper at ICLR 2021
(a) Trained on the original dataset Zaidan et al.
2007
(b) 20% noise in rationales
(c) 20% noise in non-rationales
 Aoliinaiiiii 't th of frrre  
(d) 40% noise in rationales
(e) 40% noise in non-rationales
vesiig in vane ih Me waa
(f) 60% noise in rationales
(g) 60% noise in non ratrroatinaee
(h) 80% noise rationales Ko walee
(i) 80% noise in non-rationales
wipppan my te Haryyy
(i) 100% noise in rationales
(k) 100% noise in non-rationales
Figure 9: Most important features learned by an SVM classifier trained on TF-IDF bag of words. All noise inserted on rationales identified by humans.
30

---

Published as a conference paper at ICLR 2021
(a) Trained on the original dataset Zaidan et al.
2007
(b) 20% noise in rationales
(c) 20% noise in non-rationales
B Mhl y yau
2
(d) 40% noise in rationales
(e) 40% noise in non-rationales
(f) 60% noise in rationales
(g) 60% noise in non non-rationn rationales
(h) 80% noise in rationales
(i) 80% noise in non-rationales
W wo P y and Sareeaoeoiiiie in
' " 
(i) 100% noise in rationales
(k) 100% noise in non-rationales
Figure 10: Most important features learned by an SVM classifier trained on TF-IDF bag of words. Rationales are identified as tokens attended upon by a BiLSTMM with Self Attention model.
31

---

Published as a conference paper at ICLR 2021
(a) Trained on the original dataset Zaidan et al.
2007
(b) 20% noise in rationales
(c) 20% noise in non-rationales
(d) 40% noise in rationales
(e) 40% noise in non-rationales
gerer
(f) 60% noise in rationales
(g) 60% noise in non non-rationne rationales
(h) 80% noise rationales
(i) 80% noise in non-rationales
ait e yu
aatr of
(i) 100% noise in rationales
(k) 100% noise in non-rationales
Figure 11: Most important features learned by an SVM classifier trained on TF-IDF bag of words. Rationales are identified as tokens marked by the AllenNLP Saliency Interpreter.
32

---

Published as a conference paper at ICLR 2021
Table Accuracy of various models for sentiment analysis trained with various datasets, O refers to the in-sample test set from Kaushik et al. 2020) whereas R refers to the counterfactually revised counterparts of the same.
Training data
SVM
NB
BiLSTM w/ SA
BERT
O
O
R
Orig. (1.7k)
80.0
51.0
74.9
47.3
78.0
49.4
87.4
82.2
CRD (1.7k)
58.3
91.2
50.9
88.7
63.8
82.0
80.4
90.8
Hu et al.
20177
56.3
68.0
57.1
71.1
55.1
67.0
66.3
74.4
Li al.
2018)
41.3
54.1
37.8
58.0
49.9
49.4
37.9
55.9
Sudhakar et al.
(2019)
47.1
55.9
42.6
58.8
49.9
49.4
43.7
51.8
Madaan et al.
2020)
61.2
77.3
50.2
75.2
59.3
69.7
70.6
81.6
CAD (3.4k)
83.7
87.3
86.1
91.2
80.3
84.8
88.5
95.1
Orig. & Hu et al.
(3.4k)
82.1
66.4
81.5
55.1
76.4
63.9
87.1
89.3
Orig. & Li et al.
(3.4k)
73.3
55.7
77.9
53.3
69.5
53.3
80.3
79.5
Orig. & Sudhakar et (3.4k)
74.1
56.1
79.1
51.4
71.4
55.7
89.1
90.8
Orig. & Madaan et al.
(3.4k)
83.8
65.4
82.1
67.4
75.5
64.1
83.5
81.6
Orig. (3.4k)
85.1
54.3
82.4
48.2
80.1
57.0
90.2
86.1
Table 10: Accuracy of BERT trained on subsample of SNLI Young et al. 2020) (where number of rationale tokens and non rationale tokens are within 30% of one another) as noise is injected on human identified rationales/non- rationales. RP and RH are Revised Premise and Revised Hypoth- esis test sets Kaushik al. (2020). MNLI-M and MNLI-MM are MNLI (Williams et 2018 dev sets.
Percent noise added to train data rationales
Dataset
10
30
40
60
80
90
100
In-sample test
90
87.1
83.5
80.3
80.8
78.9
77.8
77.5
73.5
67.9
69.7
RP
66.9
66.4
60.4
57.9
56.9
54.4
52.3
51.3
51.4
51.2
51.5
RH
79.2
75
69.8
67
66.5
64.2
63.5
65.7
64.9
61.7
61.8
MNLI-M
74.1
66.4
61.9
59.8
59.4
57.4
54.5
56.6
55.7
54.7
54.6
MNLI-MM
76.1
66.5
61.4
59
58.5
56.5
53.6
56
55.6
54.2
54.4
Percent noise added to train data non-rationales
Dataset
10
20
30
40
60
80
90
100
In ia-mamle test
90
88.7
87.5
85.2
85.7
84.2
83.4
82.2
79.3
77.2
74.9
RP
66.9
68.3
66.2
62.7
64.5
63.3
62.6
61.7
61.5
61.5
62.5
RH
79.2
78.4
77.4
75.9
74.6
73.4
72.8
72.2
73
70.6
70.8
MNLI-M
74.1
67.6
66.5
64.4
65.4
62.8
62.6
62.1
61.9
61.8
61.9
MNLI-MM
76.1
68
67.6
65.1
65.1
63.1
63.1
62.3
61.6
61.3
61.1
33


Reviews:
[{"review_id": "u9w9KtpvYXm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after author response:\n\nI appreciate the authors' efforts to address my concerns. I still find the paper's insights lacking in the critical middle ground between theory and practice, but understand that drawing these connections is a long game. I believe the paper leads research in an important direction and so can be published despite these flaws. \n\nSummary:\n\nThe paper investigates the issue of counterfactual data augmentation (CDA), i.e., the process by which humans augment a model's training data by making minimal edits necessary in order to flip the label. For example, in the case of sentiment analysis, humans might edit a positive movie review in order to make it negative, changing as few words as possible. The goal of CDA has been to reduce models' reliance on spurious features (e.g., gender bias, dataset artifacts) and instead encourage the model to base its predictions on features which will generalize outside of the training data. This paper makes two contributions.\n* First, the authors attempt to analyze CDA through the lens of causal inference, looking at a toy setting of Gaussian OLS. Specifically, they consider a simple graph containing a causal feature and a non-causal feature, where these features may be observed with some noise. They walk through a simple derivation for the slopes of the OLS in terms of the variance on these features and show that the slope on the causal feature will decrease to 0 as noise on the causal feature increases, and that the slope on the non-causal feature will increase. The takeaway is that noising non-causal features should lead a model to rely more on causal features (and thus generalize better), whereas noising the causal features should lead the model to rely on spurious features and thus perform worse overall. \n* Second, equipt with the above takeaway, the authors perform a series of empirical studies in which they add noise to allegedly causal vs. non-causal spans of text and evaluate whether models generalize better as a result. Specifically, they focus on some recently released CDA datasets that take the form of e.g., movie reviews in which spans have been marked by humans to denote which words are most important for predicting the label. The authors show that noising these spans leads models to perform worse while noising random other spans causes models to perform better. The experiments are also run using attention (as opposed to human annotation) in order to identify \"causal\" spans, and using automatic style transfer techniques to perform the CDA, though the results of these automatic methods suggest that they perform less well than using the human-identified spans.\n\nOverall, the paper has potential to make an important contribution, but I feel it falls short in its current form. The idea of analyzing CDA in terms of causal graphs is exciting, but its not fully fleshed out here. I don't fault the authors for beginning with a simple/toy setting, but I do think they owe the readers more discussion of the implications and limitations of this setting (see my questions below) in order for the theoretical results to be useful. I also found the tie between the empirical results and the theoretical ones to be tenuous--the empirical results are unsurprising (they can be paraphrased as \"adding noise to important features causes models to perform worse\") and could probably be explained by a number of theoretical frameworks. Thus, its not clear to me why we should accept them as evidence of the causal explanation (I elaborate below). Finally, I think the paper is a bit hard to read and lacking in experimental details (more comments below). For example, the results that use automatic style transfer are a bit of an afterthought, and so its hard to know what to take away from Table 2. \n\nStrengths:\n* Weighs in on a very important open question: how can we reason rigorously about counterfactual data augmentation?\n* Makes efforts toward both a theoretical and an empirical contribution\n\nWeaknesses:\n* Discussion surrounding theoretical contribution is lacking, making application appear quite limited\n* Connection between theory and empirical results is tenuous, and authors should discuss alternative explanations for why we'd see these empirical trends\n* Paper is hard to follow. The empirical results try to do too much, and so no one experiment is properly explained.\n\nDetailed Questions/Comments:\n* Better discussion of theoretical assumptions: You focus on a simple Gaussian OLS model for your theoretical discussion, which you acknowledge is a toy setting and I appreciate that. I don't think there is any harm in these toy settings, but I do think that there should be a more informed discussion about exactly how much we can/should takeaway from it. For example, these causal graphs assume that training amounts to learning a relative weighting on features, when in fact modern NLP models are simultaneously learning which features to extract and how to weight them, which means many things other than noise might mediate the models' reliance on x1 vs. x2. For example this paper (https://arxiv.org/pdf/2004.15012.pdf) suggests that features differ in \"accessibility\" and that might influence which feature is used, regardless of noise. I think the final prediction you make (last paragraph before Section 4) which informs your empirical results is sensitive to these assumptions and thus it requires discussion. Again, I realize you own the speculative nature of these claims and I appreciate that, but even so, that doesn't remove the obligation of connecting the dots. For example, can you comment on: If your assumptions are not met, what does that mean? How should we interpret your empirical results in that case? Are the findings rendered completely irrelevant, or can we take some relaxed interpretation of them? I think a good theoretical contribution requires some discussion along these lines.\n* Alternative explanations for empirical trends: Your primary empirical results show that noising unimportant features hurts models less than noising important features. That result alone seems unsurprising. What is interesting in your paper is that these results are consistent with a prediction that was made using your causal analysis, and thus are interpreted as evidence consistent with the causal model. I think that some additional discussion is warranted, since there are many (arguably simpler) models other than the causal one that may also generate these data. Its possible I missed something in your proofs, so correct me if I am wrong, but my understanding is that the same data could be explained purely in terms of which feature is most strongly correlated with the label during training. I.e., by design, the causal feature is more strongly correlated with the label than the spurious feature, and as you add noise, the model just ends up relying on whichever feature is more strongly correlated. This would be the case regardless of the causal graph. Is that a fair paraphrase? If so, its hard to say that these data substantiate anything about the causal nature of the features in play in CDA. But please correct me if I am wrong, its likely I missed a nuance of your argument.\n* Hard to follow: Overall, I'd recommend making fewer points and spending more time communicating the important ones. I found Tables 1 and 2 hard to follow given how little detail was provided in the text. In Figures 2 and 3 (arguably the main results) its really hard to see what the main takeaway is. You might consider putting rationales on the same plot as nonrationales so that we can clearly see when and by how much they pull away from one another. Other little details are missing as well (e.g., what is lambda_c in eq. 4? this was never defined). I'd recommend asking someone unfamiliar with the work to give it a read through and nit pick for clarity.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qLPwc2WS_NI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Edit after author response\nI thank the authors for their detailed response and for taking into account many of my comments. One issue that remains is the disconnect between toy and natural scenarios. There is some added discussion, which is helpful, but actually sketching out and conducting experiments, including intermediary steps moving from a full toy example to the naturalistic case, could be very helpful here. \n\n\n# Summary\nThis paper studies the impact of counter-factually augmented data on out-of-domain generalisation. The paper starts with a toy example of a structural causal model with Guassian linear models, where noise is added to causal or non-causal features. Increasing noise on casual features affects the least-squares estimates, while increasing noise on non-causal features does not. The paper draws an analogy between this scenario and counter-factual text editing, where edited spans (rationales) are presumed to be the causal features. A hypothesis is put forth that adding noise to rationales (causal features) would cause a model to rely on non-rationales (non-causal features) and result in poorer out-sample performance, while adding noise to non-rationales would lead to worse in-sample performance but better out-sample performance. Experiments on sentiment and natural language inference (NLI) datasets mostly confirm this hypothesis, with some exceptions to be discussed. The experiments include three ways to identify rationales (human edits, human-identified spans, and spans identified by self-attention). Models are trained with or without noise (random tokens replacing real tokens) on rationales or non-rationales. \n\n# Evaluation\n1. The paper studies an important question of how models may rely on causal and non-causal features, and how this impacts out-of-domain generalisation. The toy example is useful for proving the case in a simple setup and getting intuition. The experiments with NLP datasets are comprehensive. \n2. That said, I think the emphasis on counterfactually augmented data is not conducive to the arguments and experiments in the paper. I think the paper may better be pitched as a study of causal vs non-causal features and the impact of them on out-of-sample generalization. The counter-factual data augmentation (text editing) is one way that the paper uses to identify such features, but the paper in fact has other ways to do so. But \n3. There is a potential disconnect between the toy example and the NLP experiments, as acknowledged in the paper. I don't have a good suggestion now on how to bridge this gap, but perhaps there can be intermediary steps moving from a full toy example to the naturalistic case. \n4. There are a lot of experiments and results, but the presentation and discussion of them are hard to follow. There is one long paragraph that discusses the results on page 7. On the one hand, it is full of details and can be split into several paragraphs. On the other hand, there are missing discussions of various aspects, which I discuss below. I would split these paragraphs in several, and discuss each aspect of the experiment in more detail. See more on this below. \n5. Conversely, the introduction is pretty long and perhaps can be shorten to allow for more space. I also give suggestions on what to cut out below. \n\nI'd be willing to reconsider my evaluation if these issues are addressed. \n\n\n# Main comments \n\n## Introduction\n1. \"When the causal features are subject to observation noise, the non-causal features become salient\" - what does \"salient\" mean here? \n2. Counterfactual data augmentation (CAD) is formalized as intervention on causal features (d-separation of label from non-causal features). Alternatively, CAD is formalized in anti-causal framework. \"Here it is us who intervene on the label and the role of the editors is that of the structural equation.\" What does this mean? In general, mode background on the anti-causal framework may be helpful. \n\n\n## Toy example\n3. Why is there noise in x2? (eq 3) Differentiate between the noise in the data generation (eq 3 terms) and the measurement noise mentioned right after. \n4. What is meant by \"intervention\" (for example on x1) in this case? \n\n## Empirical study\n5. There are many aspects that may impact the behaviour: how rationales are identified (human-edited, human-annotated, automatically attributed), whether noise is added to rationales or to non-rationales, the test domain (in-sample / out-sample, but also various out-sample datasets), and which model is used (SVM, LSTM w/SA, BERT, etc.).  It looks like all the results are available, but more careful discussion of different factors would be helpful. For example, how results differ by out-sample dataset is not discussed, and how model choice affects the trends is only briefly hinted at in the conclusion. \n6. In NLI, when noise is added to rationales, out-sample performance drops more than in-sample performance, while when noise is added to non-rationales, in-sample performance drops more than out-sample performance. This is consistent with the sentiment case. However, the drop in in-sample accuracy when adding noise to non-rationales is smaller than the drop when adding noise to rationales. This is different from (some?) cases in the sentiment experiment, where adding noise to non-rationales led to a larger drop in in-sample accuracy compared to adding noise to rationales. Why is there a difference between NLI and sentiment? It seems like the NLI models may not rely as much on non-rationales, so adding noise to them doesn't impact in-sample performance much.  \n7. The discussion of the case of automatically-identified rationales is very brief and I think more can be said there about the results. There is also some potential concern with using attention for feature attribution, and using a gradient-based feature attribution method might be better. \n8. One major difference from the toy example and the hypothesis made in the first part of the paper is that adding noise to non-rationales does not always result in better out-sample performance. Sometimes it results in a more moderate part. Presumably, this can be explained by rationales not being perfectly aligned with causal features, or by models being able to ignore some of the noise, or some other reason. I think it's an important point to discuss. \n9. The style transfer experiments seem disconnected from the main theme of the paper (what are the causal/non-causal features? What is the noise? etc.) They not properly discussed and contrasted with other experiments. It might be better to leave those out completely. \n10. Layout: figures and tables do not appear next to where they are discussed in text, making it hard to navigate the paper. Some tables are mentioned in the main text but appear in the appendix and it's not clear that the reference is to an appendix table. Table 2 is not referred to or discussed from the text at all, as far as I can tell. \n\n## Minor comments\n\n- End of section 3: \"a models\" \n- The clause starting with \"the mapping between\" is missing a verb. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nffkXY8T7e6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a perspective on counterfactually augmented data (CAD) and experiments providing evidence that CAD elicited from humans indeed bears resemblance to interventions on labels or variables that are causally related to the labels.\n\nThis is already the stated intuition behind CAD and why CAD may help with learning a model that generalizes well out of domain. However, as this paper argues, questions remain over the exact mechanism of CAD’s benefits, and whether, for example, existing automated methods (such as attention and style transfer) may be used to accomplish the same goal as human interventions.\n\nThe main contribution of the paper is new indirect evidence in favor of the original intuition behind CAD. This is made possible through an observation provided over a toy model of Gaussian structural causal models. Section 3 shows that noise in the causal variable increases a learned model’s dependence on a confound, whereas noise in a confound does not affect the model in the absence of noise on the causal variable. More weight given to the confound will hurt out-of-domain generalization by the model. All this being the case, we would predict that corrupting causal inputs in training data would hurt out-of-domain performance, while corrupting non-causal inputs would help it (even if it hurts training).\n\nThis provides a framework for the experiments, which show this is indeed the case for certain human-provided annotations of rationales, providing corroborating evidence that human-annotated rationales or counterfactual augmentation do correspond to causal variables. Automated methods such as learned attention or style transfer, on the other hand, do not exhibit this trend, providing evidence that they do not pick out the appropriate causal variables.\n\n### Strengths\n\nThe paper has a fairly clear logical flow and provides clear theoretical and empirical components which are connected well. I also think the motivation is very nicely done. Lots of relevant work is brought together and a good case is made for the importance of the questions addressed by this paper.\n\n### Weaknesses\n\nI don’t have any serious complaints. The contribution is a tad narrow, but it makes progress on some tricky and difficult questions. The experiments also only produce *corroborating* evidence of CAD’s status as implicating causal variables, and we already know by construction that there is a causal aspect to these perturbations, so it’s not exactly an Earth-shaking result. But the experimental framework gives what seems to be pretty good evidence that other automated methods *don’t* implicate causal variables, so that’s nice. The main value seems to be in the theoretical observations about the effect of noise on out-of-domain performance and what that tells us about causality. These observations in the paper seem fairly straightforward, but I’m not aware of any literature making the same point. (However, I am not perfectly familiar with the literature and might have missed it.) Regardless, the application of these observations to experiments is interesting and provides useful conceptual scaffolding for future research about causal variables in models (such as deep NNs) without such explicit notions.\n\n### Recommendation\n\nAccept. The paper seems sound, is well written, and addresses an important problem. The contribution may not be huge but seems to me worth publishing.\n\n### More comments/questions\n\nThe bolded paragraph on P. 5 might be a bit much. It makes a particular claim about “language meaning” which implicitly views meaning as corresponding to the causal connection between input language and output labels. The notion of language “meaning” is nuanced and it’s not clear whether this claim is true for all tasks, where labels may be annotated based on broader (or indeed narrower) inferences regarding the generative process of the text. Since this isn’t purporting to be a paper about language meaning, I would suggest staying away from this.\n\n### Typos, style, etc.\n\n* P. 2: I think you should be able to render (Wright et al., 1934; Figure 1) more naturally by using the [bracketed arguments] in \\citep. ...Not sure how this plays with hyperref.\n* P. 3: period before “and unintentional”\n* P. 6: “use SVM” -> “we use SVM”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1xV6PY5NN3f", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an analysis of the benefits when using counterfactually augmented data (CAD) for classification purposes. The proposed framework is causal inference and DAGs. They focus on the problem of spurious correlations and how they affect out-domain generalization.\n\nThey study a simple scenario with linear models and show that in this scenario, when using CAD,  spurious correlations are no longer used by classifiers. These qualitative insights as well as framing the problem in a causal model are the main contribution of the paper. This is not a minor contribution: up to now the discussion about the benefits of CAD where only supported by empirical findings. Framing the problem, even if considering simplified models, is a critical step.\n\nI appreciate the dual causal model that is studied (causal and \"anticausal\"), not limiting the analysis to the most common interpretation. They show that both models mean to the same conclusions. \n\nThe paper will be more informative with a discussion about the noise model in the case of the noisy proxy (pg. 4). Is this the most probable noise model? Are there alternatives?\n\nThe paper presents a large series of experiments to support the proposed hypothesis. Experiments are well designed and results are convincing. \n\nThe paper opens a full line of research, especially if we consider the generalization of these results to other related domains.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a theoretical characterization of the impact of noise in causal and non-causal features on model generalization, through the lens of counterfactual data augmentations with toy data and models, and demonstrates that the predictions of this characterization bear out in several experiments on language counterfactually-augmented language data with substantial models.

Pros:
- Spurious features and their relationship out-of-domain generalization are a practically issue in modern applied ML, and this work helps to coalesce our understanding of this area.
- Fairly extensive experimental work.

Cons:
- Reviewers didn't find the connection between the theoretical analysis, which focused on a simplified setting, and the experimental work, to be especially clear. In particular, reviewers worried that the predictions that were tested experimentally were fairly intuitive ones that could reasonably be derived from a number of starting assumptions, so it's not clear that they offer strong support for the specific account given here.
- Reviewers found the presentation, especially of the empirical work, confusing.

Overall, this paper makes a legitimate and sound contribution to an important research area. That contribution is small, and somewhat easy to misinterpret, but after some discussion, reviewers agreed that the paper should still be a worthwhile net positive for the field.
 
Decision: Accept (Poster)