Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
Authors: Divyansh Kaushik
Eduard Hovy
Zachary Lipton
Venue:  (2020)
Keywords: humans in the loop
annotation artifacts
text classification
sentiment analysis
natural language inference

Abstract: Despite alarm over the reliance of machine learning systems on so-called spurious patterns, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are due to confounding (e.g., a common cause), but not direct or indirect causal effects. In this paper, we focus on natural language processing, introducing methods and resources  for training models less sensitive to spurious patterns. Given documents and their initial labels, we task humans with revising each document so that it (i) accords with a counterfactual target label; (ii) retains internal coherence;  and (iii) avoids unnecessary changes. Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their  counterfactually-revised counterparts and vice versa. Classifiers trained on combined datasets  perform remarkably well, just shy of those specialized to either domain. While classifiers trained on either original or manipulated data alone  are sensitive to spurious features (e.g., mentions of genre), models trained on the combined data are less sensitive to this signal. Both datasets are publicly available.

Published as a conference paper at ICLR 2020
LEARNING THE DIFFERENCE THAT MAKES A DIFFER- ENCE WITH COUNTRRFACTUALY--UUGEENDD DATA
Divyansh Kaushik, Eduard Hovy, Zachary C. Lipton Careegie Mellon University
Pittsburgh PA, USA {dkaushik, hovy, z1i @eiptonn@cmu...o edd
ABSTRACT
Despite alarm over the reliance of machine learning systems on so-called spuri- ous patterns, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are due to confounding (e.g., common cause), but direct indirect causal effects. In this paper, we focus on natural language processing, introducing methods and resources for training models less sensitive spurious patterns. Given docu- ments and their initial labels, we task humans with revising each document so that it (i) accords with a counterfactual target label; (ii) retains internal coher- ence; and (iii) avoids unnecessary changes. Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their counterfactually-revise counterparts and vice versa. Classifiers trained on combined datasets perform remarkably well, just shy of those specialized to either domain. While classifiers trained on either original or manipulated data alone are sensitive to spurious features (e.g., mentions genre), models trained on the com- bined data are less sensitive to this signal. Both datasets publicly available
INTRODUCTION
What makes a document's 's sentiment positive? What makes a loan applicant creditworthy? What makes a job candidate qualified? When does a photograph truly depict a dolphin? Moreover, what does it mean for a feature to be relevant to such determination?
Statistical learning offers
framework for approaching these questions. First, swap the semantic question for a more readily answerable associative question. For example, instead of asking what conveys document's 's sentiment, recast the question as which documents are likely to be labeled as positive (or negative)? Then, in this associative framing, we interpret as relevant, those features that most predictive of the label. However, despite the rapid adoption and undeniable commercial success of associative learning, this framing seems unsatisfying.
Alongside deep learning's predictive wins, critical questions have piled up concerning spurious patterns, arlifacts, robusiness, and discrimination, that the purely associative perspective appears ill-equipped to answer. For example, in computer vision, researchers have found that deep neural networks rely on surface-level texture (Jo & Bengio 2017;; Geirhos et al.) 2018)) or clues in the image's background to recognize foreground objects even when that seems both unnecessary and somehow wrong: the beach is not what makes a seagull seagull. And yet, researchers struggle to articulate precisely why models should not rely on such patterns.
In natural language processing (NLP), these issues have emerged as central concerns in the literature on annotation artifacts and societal biases. Across myriad tasks, researchers have demonstrated that models tend to rely on spurious associations (Poliak et al.] 2018 Gururangan et al. 2018 Kaushik Lipton 2018; Kiritchenko & Mohammad 2018). Notably, some models for question-answennn tasks may not actually be sensitive to the choice of the question (Kaushik & Lipton 2018), while in Natural Language Inference (NLI), classifiers trained on hypotheses only (vs hypotheses and premises) perform surprisingly well (Poliak et al. 2018: Gururangan et al. 2018). However, papers

---

Published as a conference paper at ICLR 2020
New, counterfactual datapoint
Expert annotator
approved, dalapoint added alongside the original sampled datapoint create a new dataset.
Counterfactually
Original dataset
Sampled
augmented dataset
datapoint
Figure I: Pipeline for collecting and leveraging counterfactually-atteeed data
seldom make clear what, if anything, spuriousness means within the standard supervised learning framework. ML systems are trained to exploit the mutual information between features and label to make accurate predictions. The standard statistical learning toolkit does not offer conceptual distinction between spurious and non-spurious associations.
Causality, however, offers coherent notion of spuriousness. Spurious associations owe to con- founding rather than to a (direct or indirect) causal path. We might consider a factor variation to be spuriously correlated with a label of interest if intervening upon it would not impact the appli- cability of the label or vice versa. While our paper does not call upon the mathematical machinery of causality, we draw inspiration from the underlying philosophy to design new dataset creation procedure in which humans counterfactually revise documents.
Returning to NLP, although we lack automated tools for mapping between raw text and disentangled factors, we nevertheless describe documents terms of these abstract representations. Moreover, it seems natural to speak of manipulating these factors directly (Hovy) 1987). Consider, for exam- ple, the following interventions:
(i) Revise the letter to make it more positive; (ii) Edit the second sentence so that il appears to contradict the first. These edits might be thought of intervening on only those aspects of the text that are necessary to make the counterfactual label applicable.
In this exploratory paper, we design human-in-the-loog system for counterfactually manipulating documents. Our hope is that by intervening only upon the factor of interest, we might disentangle the spurious and non-spurious associations, yielding classifiers that hold up better when spurious associations do not transport out of domain. We employ crowd workers not to label documents, but rather edit them, manipulating the text make targeted (counterfactual) class applicable. For sentiment analysis, we direct the worker to revise this negative movie review to make it posi- tive, without making any gratuilous chunges. We might regard the second part of this directive as a least action principle, ensuring that perturb only those spans necessary to alter the upplicability of the label. For NLI, a 3-class classification task (entailment, contradiction, neutral), we ask the workers to modify the premise while keeping the hypothesis intact, and vice versa, collecting edits corresponding to each of the (two) counterfactual classes. Using this platform, we collect thou- sands of saaiterfactuallymmniuulla
examples for both sentiment analysis and NLI, extending the IMDb MMas et al. [011]) and SNLI (Bowman et al. 2015) datasets, respectively. The result is two new datasets (each extension of standard resource) that enable us to both probe fundamental properties of language and train classifiers less reliant on spurious signal.
We show that classifiers trained on original IMDb reviews fail on coumterfactually-reiii vice versa. We further show that spurious correlations in these datasets are even picked up by linear models. However, augmenting the revised examples breaks up these correlations (e.g., genre ceases to be predictive of sentiment). For a Bidirectional LSTM (Graves & Schmidhuber, (0055 trained on IMDb reviews, classification accuracy goes down from 79.3% to 55.7% when evaluated on original vs revised reviews. The same classifier trained on revised reviews achieves an accuracy of 89.1 % on revised reviews compared to 62.5% on their original counterparts. These numbers go to 81.7% and 92.0% original and revised data, respectively, when the classifier is retrained on the combined dataset. Similar patterns are observed for linear classifiers. We discovered that BERT (Devlin et al. 2019) is more resilient such drops in performance sentiment analysis.
data and
2

---

Published as a conference paper at ICLR 2020
Additionally, SNLI models appear to rely on spurious associations identified by Gururangan al. (018)) Our experiments show that when fine-tuned on original SNLI sentence pairs, BERT fails on pairs with revised premise and vice versa, suffering more than a 30 point drop in accuracy. Fine-tuned on the combined set, BERT's performance improves significantly across all datasets. Similarly, Bi-LSTMM trained on (original) hypotheses alone can accurately classify 69% of pairs correctly but performs worse than the blind classifier when evaluated on the revised dataset. When trained on hypotheses only from the combined dataset, its performance not appreciably better than random guessing.
2 RELATED WORK
Several papers demonstrate cases where NLP systems appear not to learn what humans consider to be the difference that makes the difference. For example, otherwise state-of ootie--f-theaatt models have been shown to be vulnerable to synthetic transformations such as distractor phrases (Jia & Liang 2017
Wallace et al., 2019). misclassify paraphrased task (lyyer et al., 2018 Pfeiffer et al. 2019 and to fail on template-based modifications (Ribeiro et al. 2018). Glockner et al. (2018) demonstrate that simply replacing words by synonyms or hypernyms, which should not alter the applicable label, nevertheless breaks ML-based NLI systems. Gururangan et al. (2018) and Poliak et al. (2018) show that classifiers correctly classified the hypotheses alone in about 69% of SNLI corpus. They further discover that crowd workers adopted specific annotation strategies and heuristics for data genera- tion. Chen et al. (2016) identify similar issues exist with automatiicllyyccnssreece
benchmarks for question- -anttonnnnswering (Hermann et al. 2015). Kaushik & Lipton| (2018) discover that reported num- bers in uuestion-answerrin
benchmarks could often be achieved by the same models when restricted to be blind either to the question or to the passages. Dixon et al. 2018); Zhao et al. (2018) and Kir- itchenko & Mohammad (2018) showed how imbalances in training data lead to unintended bias in the resulting models, and, consequently, potentially unfair applications. (Shen et al. (2018) substitute words to test the behavior of sentiment analysis algorithms in the presence of stylistic variation, finding that similar word pairs produce significant differences in sentiment score.
Several papers explore richer feedback mechanisms for classification. Some ask annotators to high- light rationales, spans of text indicative of the label (Zaidan et al. 2007 Zaidan & Eisner 20088 Poulis & Dasguptal 2017). For each document, Zaidan et al. remove the rationales to generate contrast documents, Tearning classifiers to distinguish original documents from their contrasting counterparts. While this feedback is easier to collect than ours, how to leverage it for training deep NLP models, where features are not neatly separated, remains less clear.
Lu et al. 2018) programmatically alter text to invert gender bias and combined the original and manipulated data yielding gender-balances dataset for learning word embeddings. In the simplest experiments, they swap each gendered word for its other-gendered counterpart. For example, the doctor ran because he is late becomes the doctor ran because she late. However, they do not substitute names even if they co-refer to a gendered pronoun. Building on their work, (Zmigrod et al. (2199) describe a data augmentation approach for mitigating gender stereotypes associated with animate nouns for morphologically--in random field to infer how the sentence must be modified while altering the grammatical gender of particular nouns to preserve morpho-syntactic agreement. In contrast, Maudslay al. (2019) describe a method for probabilistic automatic in-place substitution of gendered words in a corpus. Unlike Lu et al.] they propose an explicit treatment of first names by pre-defining name-pairs for swapping, thus expanding Lu et a list of gendered word pairs significantly.
languages like Spanish and Hebrew. They use Markov
DATA COLLECTION
We use Amazon's 's Mechanical Turk crowdsourcing platform to recruit editors to revise each docu- ment. To ensure high quality of the collected data, we restricted the pool to U.S. residents that had already completed at least 500 HITs and had an over 97% HIT approval rate. For each HIT, we conducted pilot tests to identify appropriate compensation per assignment, receive feedback from workers and revise our instructions accordingly. A total of 713 workers contributed throughout the whole process, of which 518 contributed edits reflected in the final datasets.
3

---

Published as a conference paper at ICLR 2020
Sentiecc b-ich44
Instructions
Dipolar Sentiment
porsage and
Bedtios bulaw, shat
Compp neveew
Label
noem letwor wotdling Jein Connactes no stantig bin lovemur facror criiing byper
(6 etcum end coheriee
Origral
right canglerr blewe way, purtouor tite foom lint eute hepiing
perfectlattd tow nda
accuriidy dactiie revbee
Whev apeni the servee wanching Jahn Comaedda mu stanting we tomome ander the flerro andingg bopiing lec toporin right
encarpt iecatandd aurtouler pectno lim inomile
pmosgee un mormontt
bated fron abou toothiis in through sill thevored cantuu..
checking lover wnitt HIl luttos
Revev'ss
You treccing Suavey Copl felld
Leng torng Merghenees Nover theense glod enoitsss
Negettvv
Mechanioon lcal
bor Mkophonwes Nover have teee losee cnaiing crallorrd
Noat Stop
Podtiee
Figure 2: Annotation platform for collecting counterfactually annotated data for sentiment analysis
Table Percentage of inter-editor agreement for counterfacttaally-reiiidd movie reviews
Number of tokens
Type
0-50
51-100
101-150
151-200
201-250
251-300
301-329
Full
Replacement
35.6
25.7
20.0
17.2
15.0
14.8
11.6
19.3
Insertion
27.7
20.8
14.4
12.2
11.0
11.5
07.6
14.3
Combined
41.6
32.7
26.3
23.4
21.6
20.3
16.2
25.5
Sentiment Analysis The original IMDb dataset consists of 50k reviews divided equally across train and test splits. To keep the task of editing from growing unwieldy, we filter out the longest 20% of reviews, leaving 20k reviews in the train split from which we randomly sample 2.5k reviews, enforcing 50:50 class balance. Following revision by the crowd workers, partition this dataset into muin/validation/test splits containing 1707, 245 and 488 examples, respectively. We present each review two workers, instructing them to revise the review such that (a) the counterfactual label applies; (b) the document remains coherent; and (c) no unecessary modifications are made.
Over four week period, manually inspected each generated review and rejected the ones that were outright wrong (sentiment was still the same or the review was a spam). After review, we rejected roughly 2% of revised reviews. For 60 original reviews, did not approve any among the counterfactually--eeiss counterparts supplied by the workers. To construct the new dataset, we chose one revised review (at random) corresponding each original review. In qualitative analysis, we identified eight common patterns among the edits (Table
By comparing original reviews their counterfactualll-reiiss which aspects are causally relevant. To analyze inter-editor agreement, we mark indices corre- sponding replacements and insertions, representing the edits each original review by binary vector. Using these representations, we compute the Jaccard similarity between the two reviews (Table I. finding be negatively correlated with the length the review.
counterparts we gain insight into
Natural Language Inference Unlike sentiment analysis, SNLI is 3-way classification task, with inputs consisting of two sentences, a premise and a hypothesis and the three possible labels being entailment, contradiction, and neutral. The label is meant describe the relationship between the facts stated in each sentence. We randomly sampled 1750, 250, and 500 pairs from the train, validation, and test sets of SNLI respectively, constraining the new data to have balanced classes. In one HIT, we asked workers to revise the hypothesis while keeping the premise intact, seeking edits corresponding to each of the two counterfactual classes. We refer to this data Revised Hypothesis
4

---

Published as a conference paper at ICLR 2020
Table 2: Most prominent categories of edits performed by humans for sentiment analysis (Origi- nal/Revised, in order). Red spans were replaced by Blue spans.
Types of Revisions
Examples
Recasting fact hoped for
The world of Atlantis, hidden beneath the earth 's core, is fantastic The world of Atlantis, hidden beneath the earth's 's core is supposed to be fantastic
Suggesting sarcasm
thoroughly captivating thriller-drama, taking a deep and real- istic view
thoroughly mind numbing "thriller-rrama,, taking a "deep" and "realistic" (who they kidding?) view
Inserting modifiers
The presentation simply Atlantis' landscape and setting The presentation of Atlantis' predictable landscape and setting
Replacing modifiers
"Election" is highly fascinating and thoroughly captivating thriller-drama
"Election" highly expected and thoroughly mind numbing "thriller-drama"
Inserting phrases
Although there's hardly any action, the ending is still shocking. Although there's hardly any action (or reason to continue watch- ing past 10 minutes), the ending is still shocking.
Diminishing via qualifiers
which, while usually containing some reminder of harshness, be- come more and more intriguing.
which, usually containing some reminder of harshness, became only slightly more intriguing.
Differing perspectives
Granted, not all of the story makes full sense, but the film doesn't feature any amazing new computer-generatel visual effects. Granted, some of the story makes sense, but the film doesn't feature any amazing new computer-generatee visual effects.
Changing ratings
one of the worst ever scenes in sports movie. stars out of 10. one the wildest ever scenes a sports movie. stars out of 10.
(RH), In another HIT, we asked workers to revise the original premise, while leaving the original hypothesis intact, seeking similar edits, calling it Revised Premise (RP).
Following data collection, we employed a different set of workers to verify whether the given label accurately described the relationship between each premise- hhyoiiisss pair. We presented each pair to three workers and performed majority vote. When all three reviewers were in agreement, we approved rejected the pair based on their decision, else, verified the data ourselves. Finally, we only kept poothess pairs for which we had valid revised data in both RP and RH, corresponding to both counterfactual labels. As a result, we discarded x 9% data. RP and RH, each comprised of 3332 pairs in train, 400 in validation, and 800 in test, leading to total of 6664 pairs in train, 800 in validation, and 1600 test in the revised dataset. In qualitative analysis, identified some common patterns among hypothesis and premise edits (Table
We collected all data after IRB approval and measured the time taken to complete each HIT to ensure that all workers were paid more than the federal minimum wage. During our pilot studies, workers spent roughly 5 minutes per revised review, and 4 minutes per revised sentence (for NLI). We paid workers $0.65 per revision, and $0.15 per verification, totalling $10778. 14 for the study.
4 MODELS
Our experiments rely on the following five models: Support Vector Machines (SVMs), Naive Bayes (NB) classifiers, Bidirectional Long Short-Term Memory Networks (Bi-LSTMs; Graves & Schmid- huber 2005), ELMo models with LSTM, and fine-tuned BERT models (Devlin et al. 2019). For brevity, discuss only implementation details necessary for reproducibility.
5

---

Published as a conference paper at ICLR 2020
Table Analysis of edits performed by humans for NLI hypotheses. denotes Premise, OH denotes Original Hypothesis, and NH denotes New Hypothesis.
Types of Revisions
Examples
Modifying/removingg actions
P: A young dark-haired woman crouches on the banks of river while washing dishes.
OH: A woman washes dishes in the river while camping. (Neu- tral)
NH: A woman washes dishes in the river. (Entailment)
Substituting entities
P: Students are inside of lecture hall.
OH: Students are indoors. (Entailment)
NH: Students are on the soccer field. (Contradiction)
Adding details to entities
P: An older man with glasses raises his eyebrows in surprise.
OH: The man has no glasses. (Contradiction)
NH: The man wears bifocals. (Neutral)
Inserting relationships
P: blond woman speaking a brunette woman with her arms crossed.
OH: woman is talking another woman. (Entailment)
NH: A woman is talking to family member. (Neutral)
Numerical modifications
P: Several farmers bent over working the fields while lady
with a baby and four other children accompany them.
OH: The lady has three children. (Contradiction)
NH: The lady has many children. (Entailment)
Using/Removing negation
P: An older man with glasses raises his eyebrows in surprise.
OH: The man has no glasses. (Contradiction)
NH: The man wears glasses. (Entailment)
Unrelated hypothesis
P: A female athlete in crimson top and dark blue shorts is run-
ning on the street.
OH: woman is sitting on a white couch. (Contradiction)
NH: A woman owns a white couch. (Neutral)
Standard Methods We use scikit-lean (Pedregosa et al. 2011) implementations of SVMs and Naive Bayes for sentiment analysis. We train these models on TF-IDF bag of words feature rep- resentations of the reviews. We identify parameters for both classifiers using grid search conducted over the validation set.
Bi-LSTM When training Bi-LSTMs for sentiment analysis, we restrict the vocabulary to the most frequent 20k tokens, replacing out-ol-vocabulary tokens by UNK. We fix the maximum input length at 300 tokens and pad smaller reviews. Each token is represented by randomly-initialian 50-dimensional embedding. Our model consists of bidirectional LSTM (hidden dimension 50) with recurrent dropout (probability 0.5) and global max-pooling following the embedding layer. To generate output, we feed this (fixed-length) representation through a fully-connected hidden layer with ReLU (Nair & Hinton| 2010) activation (hidden dimension 50), and then fully- connected output layer with softmax activation. We train all models for a maximum of 20 epochs using Adam (Kingma & Ba 2015), with learning rate of 1e-3 and batch size of 32. We apply early stopping when validation loss does not decrease for 5 epochs. We also experimented with a larger Bi-LSTM which led to overfitting. We use the architecture due to Poliak et al. (2018) to evaluate hypothesis- only baselines F
We compute contextualized word representations (ELMo) using character-based word representations and bidirectional LSTMs (Peters et al., 2018). The module outputs a 1024- dimensional weighted sum of representations from the 3 Bi-LSTM layers used in ELMo. We rep- resent each word by a 128-dimensional embedding concatenated to the resulting 1024-dimensional EL.Mo representation, leading to a 115-ddimenio hidden representation. Following Batch Nor- malization, this is passed through an LSTM (hidden size 128) with recurrent dropout (probability
ELMo-LSTM

---

Published as a conference paper at ICLR 2020
Table 4: Analysis of edits performed by humans for NLI premises. OP denotes Original Premise, NP denotes New Premise, and H denotes Hypothesis.
Types of Revisions
Examples
Introducing direct evidence
OP: Man walking with tall buildings with reflections behind
him. (Neutral)
NP: Man walking away from his friend, with tall buildings with reflections behind him. (Contradiction) H: The man walking to meet a friend.
Introducing indirect evidence
OP: An Indian man standing on the bank of river. (Neutral)
standing with only camera the bank of a river. (Contradiction)
NP: An Indian
H: He is fishing.
Substituting entities
OP: A young man front of grill laughs while pointing at something to his left. (Entailment)
NP: A young man in front of chair laughs while pointing at something to his left. (Neutral)
H: A man is outside
Numerical modifications
OP: The exhaustion in the woman's 's face while she continues to ride her bicycle in the competition. (Neutral)
NP: The exhaustion in the woman's face while she continues
to ride her bicycle in the competition for people above 7 (Entailment)
H: A tall person on a bike
Reducing evidence
OP: The girl in yellow shorts and white jacket has a tennis ball in her left pocket. (Entailment)
NP: The girl in yellow shorts and white jacket has tennis ball. (Neutral)
H: A girl with tennis ball in her pocket.
Using abstractions
OP: An elderly woman in crowd pushing wheelchair. (En- tailment)
NP: An elderly person in crowd pushing wheelchair, (Neu- tral)
H: There is elderly woman in a crowd.
Substituting evidence
OP: woman cutting something with scissors. (Entail- ment)
A woman is reading something about scissors. (Contra- diction)
NP:
H: A woman uses a tool
0.2). The output from this LSTM is then passed to fully-connected output layer with softmax ac- tivation. We train this model for up to 20 epochs with same early stopping criteria as for Bi-LSTM, using the Adam optimizer with a learning rate of 1e-3 and batch size of 32.
BERT We use off-the-shelf uncased BERT Base model, fine-tuning for each task To account for BERT's sub-word tokenization, we set the maximum token length is set at 350 for sentiment analysis and 50 for NLI. We fine-tune BERT up to 20 epochs with same early stopping criteria as for Bi-LSTM, using the BERT Adam optimizer with a batch size of 16 (to on a Tesla V-100 GPU), We found learning rates of 5e-5 and le-5 to work best for sentiment analysis and NLI respectively.

---

Published as a conference paper at ICLR 2020
50
(a) Trained on the original dataset (b) Trained the revised dataset
(c) Trained on combined datasct
Figure 3: Most important features learned by an SVM classifier trained on TF-IDF bag of words.
5 EXPERIMENTAL RESULTS
Sentiment Analysis We find that for sentiment analysis, linear models trained on the original 1.7k reviews achieve 80% accuracy when evaluated on original reviews but only 51% (level of random guessing) revised reviews (Table 5). Linear models trained on revised reviews achieve 91% accuracy on revised reviews but only 58.3% on the original test set. We see similar pattern for Bi-LSTMs where accuracy drops substantially in both directions. Interestingly, while BERT models suffer drops too, they are less pronounced, perhaps benefit of the exposure to larger dataset where the spurious patterns may not have held. Classifiers trained on combined datasets perform well on both, often within R 3 pts of models trained on the same amount of data taken only from the original distribution. Thus, there may be price to pay for breaking the reliance on spurious associations, but it may not be substantial.
We also conduct experiments to evaluate our sentiment models vis-a-vis their generalization out-of- domain to new domains. We evaluate models Amazon reviews (Ni et al. 2019) on data aggre- gated over six genres: beauty, fashion, appliances, giftcards, maguzines, and software the Twitter sentiment dataset (Rosenthal et al. 2017 and Yelp reviews released as part of the Yelp dataset challenge. We show that in almost all cases, models trained on the auntterfcctually-agmmmmn IMDb dataset perform better than models trained on comparable quantities of original data.
To gain intuition about what is learnable absent the edited spans, we tried training several models on passages where the edited spans have been removed from training set sentences (but not test set). SVM, Naïve Bayes, and Bi-LSTM achieve 57.8%, 59.1%, 60.2% accuracy, respectively, on this task. Notably, these passages are predictive of the (true) label despite being semantially compatible with the counterfactual label. However, BERT performs worse than random guessing.
In one simple demonstration of the benefits of our approach, we note that seemingly irrelevant words such as: romantic, will, my, has, especially, life, works, both, it, its, lives and gives (correlated with positive sentiment), and horror, own, jesus, cannot, even, instead, minutes, your, effort, script, seems and something (correlated with negative sentiment) are picked up as high-weight features by linear models trained on either original or revised reviews as top predictors. However, because humans never edit these during revision owing to their lack of semantic relevance, combining the original and revised datasets breaks these associations and these terms cease to be predictive of sentiment (Fig(4). Models trained on original data but at the same scale combined data are able to perform slightly better on the original test set but still fail on the revised reviews. All models trained on 19k original reviews receive slight boost in accuracy on revised data (except Naïve Bayes), yet their performance significantly worse compared to specialized models. Retraining models on a combination of the original 19% reviews with revised 1.7k reviews leads to significant increases in accuracy for all models on classifying revised reviews, while slightly improving the accuracy on classifying the original reviews. This underscores the importance of including counterfactually- revised examples in training data.
Fine-tuned on 1.67k original sentence pairs, BERT achieves 72.2% accuracy on SNLI dataset but it is only able to accurately classify 39.7% sentence pairs from the RP set (Table 77. Fine-tuning BERT on the full SNLI training set (500k: sentence pairs) results in similar behavior. Fine-tuning it on RP sentence pairs improves its accuracy to 66.3% on RP but causes a drop of roughly 20 pts on SNLI On RH sentence pairs, this results in an accuracy of 67% on RH and 71.9% on SNLI test set but 47. 4% on the RP set. To put these numbers in context, each
Natural Language Inference
WWe use the development set as test data is not public.
8

---

Published as a conference paper at ICLR 2020
Table 5: Accuracy of various models for sentiment analysis trained with various datasets. Orig. denotes original, Rev. denotes revised, and Orig. Edited denotes the original dataset where the edited spans have been removed.
Training data
SVM
NB
ELMo
Bi-LSTM
BERT
O
R
0
R
O
R
R
O
R
Orig. (1.7k)
80.0
51.0
74.9
47.3
81.9
66.7
79.3
55.7
87.4
82.2
Rev. (1.7k)
58.3
91.2
50.9
88.7
63.8
82.0
62.5
89.1
80.4
90.8
Orig. Edited
57.8
59.1
50.3
60.2
49.2
Orig. & Rev. (3.4k)
83.7
87.3
86.1
91.2
85.0
92.0
81.5
92.0
88.5
95.1
Orig. (3.4K)
85.1
54.3
82.4
48.2
82.4
61.1
80.4
59.6
90.2
86.1
Orig. (19%)
87.8
60.9
84.3
42.8
86.5
64.3
86.3
68.0
93.2
88.3
Orig. (19k) Rev.
87.8
76.2
85.2
48.4
88.3
84.6
88.7
79.5
93.2
93.9
Table 6: Accuracy of various sentiment analysis models on out-of-domain data
Training data
SVM
NB
ELMo
Bi-LSTM
BERT
Accuracy on Amazon Reviews
Orig. & Rev. (3.4k)
77.1
82.6
78.4
82.7
85.1
Orig. (3.4k)
74.7
66.9
79.1
65.9
80.0
Accuracy on Semeval 2017 (Twitter)
Orig. & Rev. (3.4k)
66.5
73.9
70.0
68.7
82.9
Orig. (3.4k)
61.2
64.6
69.5
55.3
79.3
Accuracy on Yelp Reviews
Orig. & Rev. (3.4k)
87.6
89.6
87.2
86.2
89.4
Orig. (3.4k)
81.8
77.5
82.0
78.0
85.3
Table 7: Accuracy BERT on NLI with various train and eval sets.
Train/Eval
Original
RP
RH
RP & RH
Original (1.67k)
72.2
39.7
59.5
49.6
Revised Premise (RP; 3.3k)
50.6
66.3
50.1
58.2
Revised Hypothesis (RH; 3.3k)
71.9
47.4
67.0
57.2
RP & RH (6.6k)
64.7
64.6
67.8
66.2
Original w/ RP & RH (8.3k)
73.5
64.6
69.6
67.1
Original (8.3k)
77.8
44.6
66.1
55.4
Original (500k)
90.4
54.3
74.3
64.3
individual hypothesis sentence in RP is associated with two labels, each in the presence of a different premise. model that relies on hypotheses only would at best perform slightly better than choosing the majority class when evaluated this dataset. However, fine-tuning BERT on combination of RP and RH leads to consistent performance on all datasets as the dataset design forces models to look at both premise and hypothesis. Combining original sentences with RP and RH improves these

---

Published as a conference paper at ICLR 2020
Table 8: Accuracy of Bi-LSTM classifier trained on hypotheses only
Train/Test
Original
RP
RH
RP & RH
Majority class
34.7
34.6
34.6
34.6
RP & RH (6.6k)
32.4
35.1
33.4
34.2
Original w/ RP & RH (8.3k)
44.0
25.8
43.2
34,5
Original (8.3k)
60.2
20.5
46.6
33.6
Original (500k)
69.0
15.4
53.2
34.3
Table 9: Accuracy of models trained to differentiate between original and revised data
Model
IMDb
SNLI/RP
SNLI/RH
Majority class
50.0
66.7
66.7
SVM
67.4
46.6
51.0
NB
69.2
66.7
66.6
BERT
77.3
64.8
69.7
numbers even further. We compare this with the performance obtained by fine-tuning it on 8.3k sentence pairs sampled from SNLI training set, and show that while the two perform roughly within 4 pts of each other when evaluated on SNLI, the former outperforms latter on both RP and RH.
To further isolate this effect, Bi-LSTM trained on SNLI hypotheses only achieves 69% accuracy on SNLI test set, which drops to 44% if it is retrained on combination of original, RP and RH data (Table (8.. Note that this combined dataset consists of five variants of each original premise- hypothesis pair. Of these five pairs, three consist of the same hypothesis sentence, each associated with different truth value given the respective premise. Using these hypotheses only would provide conflicting feedback to a classifier during training. thus causing the drop performance. Further, we notice that the gain of the latter over majority class baseline comes primarily from the original data, as the same model retrained only RP and RH data experiences further drop of 11.6% in accuracy, performing worse than just choosing the majority class at all times.
One reasonable concern might be that our models would simply distinguish whether an example were from the original or revised dataset and thereafter treat them differently. The fear might be that models would exhibit hypersensitivity (rather than insensitivity) to domain. To test the potential for this behavior, we train several models to distinguish between original and revised data (Table D). BERT identifies original reviews from revised reviews with 77.3% accuracy. In case of NLI, BERT and Naïve Bayes perform roughly within 3 pts of the majority class baseline (66.7%) whereas SVM performs substantially worse.
6 CONCLUSION
By leveraging humans not only to provide labels but also to intervene upon the data, revising docu- ments to accord with various labels, we can elucidate the difference that makes a difference. More- over, we can leverage the augmented data to train classifiers less dependent on spurious associations. Our study demonstrates the promise of leveraging human-in-the-loow feedback to disentangle the spurious and non-spurious associations, yielding classifiers that hold up better when spurious asso- ciations do not transport out of domain. Our methods appear useful on both sentiment analysis and NLI, two contrasting tasks. In sentiment analysis, expressions of opinion matter more than stated facts, while in NLI this is reversed. SNLI poses another challenge in that it is 3-class classification task using two input sentences. In future work, we will extend these techniques, leveraging humans in the loop to build more robust systems for question answering and summarization.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
The authors are grateful to Amazon AWS and NVIDIA for providing GPUs to conduct the exper- iments, Salesforce Research and Facebook AI for their generous grants that made the data collec- tion possible, Sina Fazelpour, Sivaraman Balakrishnan, Shruti Rijhwani, Shruti Palaskar, Aishwarya Kamath, Michael Collins, Rajesh Ranganath and Sanjoy Dasgupta for their valuable feedback, and Tzu-Hsiang Lin for his generous help in creating the data collection platform. We also thank Abridge AI UPMC, the Center for Machine Learning in Health, and the Al Ethics and Governance Fund for their support of our broader research on robust machine learning.
REFERENCES
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP), 2015.
Danqi Chen, Jason Bolton, and Christopher Manning. thorough examination of the cnn/daily mail reading comprehension task. In Association for Computational Linguistics (ACI.), 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguisfics: Human Language Technologies (NAACI-HI.T), 2019.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. In AAAI/ACM Conference on AI, Ethics, and Society. ACM, 2018
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and biased towards texture; increasing shape bias im-
Wieland Brendel. Imagenet-trained proves accuracy and robustness. arXiv preprint arXiv:1811.1222. 2018.
Max Glockner, Vered Shwartz, and Yoav Goldberg, Breaking nli systems with sentences that require simple lexical inferences. In Association for Computational Linguistics (ACL), 2018.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6), 2005.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In North American Chap- ter of the Association for Computational Linguistics; Human Language Technologies (NAACL- HLT), 2018
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NeurIPS). 2015.
Eduard Hovy. Generating natural language under pragmatic constraints. Journal of Pragmatics, 1987.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)) 2018
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP), 2017. 2017.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv: 1.1 1561, 2017.
Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Empirical Methods in Natural Language Pro- cessing (EMNLP), 2018
11

---

Published as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sen- timent analysis systems. In Joint Conference on Lexical and Computational Semantics (*SEM), 2018.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. arXiv preprini 18X.v 2018.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Association for Computational Linguistics: Human Language Technologies (ACL-IILT), 2011. 2011.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. arXiv preprint
2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML), 2010.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. URL
https: www www.wb.wwwbb.... org/ wwtp:/ttoggaatgggngneeeoooo aatholoooooy wttpoggg//11111111 1018
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research (JMLR), 12, 2011.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HILT), 2018.
Jonas Pfeiffer, Aishwarya Kamath, Iryna Gurevych, and Sebastian Ruder. What do deep networks like to read? arXiv preprint arXiv:1909.044 2019.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis Only Baselines in Natural Language Inference. In Joint Conference on Lexical and Computational Semantics ( (*Sem), 2018.
Stefanos Poulis and Sanjoy Dasgupta. Learning with feature feedback: from theory to practice. In Artificial Intelligence and Statistics (AISTATS), 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Association for Computational Linguistics (ACL), 2018
Sara Rosenthal, Noura Farra, and Preslav Nakov. SemFval-2017 task 4: Sentiment analysis in twitter. In International Workshop Semantic Evaluation (SemEval), 2017. URL https:
7/ wwww. www.aclwwed www.bb.wwb.....................b org/ http://thhhoooyyoeeo S17 2088
Judy Hanwen Shen, Lauren Fratamico, lyad Rahwan, and Alexander M Rush. Darling or babygirl? investigating stylistic bias in sentiment analysis. 5th Workshop on Fairness, Accountabiliry, and Transparency in Machine Learning (FATML), 2018.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for nlp. arXiv preprint arXiv:1908, 07220, 2019.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using annotator rationales improve machine learning for text categorization. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HI.T), 2007.
12

---

Published as a conference paper at ICLR 2020
Omar F Zaidan and Jason Eisner. Modeling annotators: A generative approach to learning from notator rationales. In Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2008
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL-HLT), 2018. 8.
Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data aug- mentation for mitigating gender stereotypes in languages with rich morphology. In Association for Computational Linguistics (ACL), 2019.
13

---

Published as a conference paper at ICLR 2020
APPENDIX
Table 10: Most frequent insertions/deletions by human annotators for sentiment analysis.
Revision
Removed words
Inserted words
Positive to Negative
movie, film, great, like, good, re- ally, would, see, story, love bad, even, worst, waste, nothing, never, much, would, like, little
movie, film, one, like, bad, would, really, even, story, see
Negative to Positive
great, good, best, even, well, amaz- ing, much, many, watch, better
14

---

Published as a conference paper at ICLR 2020
Table 1I: Most frequeet insertions/delctions by human annotators for SNLI.
Revision
Removed words
Inserted words
Revising Premise
Entailment to Neutral
woman, walking, man, blue, sitting, men, girl, standing, looking, running
person, near, child, something, together, people, tall, vehicle, wall, holding
Neutral to Entailment
man, street, black, water, litte, front, young, playing, woman, two
waiting, couple, playing, run- ning, getting, making, tall, game, black, happily
Entailment to Contradiction
blue, people, standing, girl, front, street, red, young, sit- ting, band sitting. man, walking, black, blue, people, red, standing, white, street
sitting, standing. inside, young, women, child, red, men, sits,
Contradiction Entailment
man, sitting, sleeping, woman, sits, eating, playing, park, two, standing
Neutral to Contradiction
man, woman, people, boy, black, red, standing, young, two, water
man, woman, boy, men, alone, sitting, girl, dog, three, one
Contradiction to Neutral
man, sitting, black, blue, walk- ing, red, standing, street, white, street
sitting, woman, peo- ple, person, near, something, something, sits, black
Revising Hypothesis
Entailment to Neutral
man, wearing, white, blue, black, shirt, one, young, peo- ple, woman
people, there, playing, man, person, wearing, outside, two, old, near playing, wearing. man, two, there, woman, people, men, near, person
Neutral Entailment
white, wearing, shirt, black, blue, man, two, standing, young, red
Entailment to Contradiction
man, wearing, white, blue, black, two, shirt, one, young, people wearing, blue, black, man, white, two, red, shirt, young,
people, man, woman, playing, inside, person, two, wear- ing, women
Contradiction to Entailment
people, there, man, two, wear- ing, playing, people, men, woman, outside
one
Neutral to Contradiction
white, man, wearing, shirt, black, blue, two, standing, woman, red
woman, man, there, playing, two, wearing, one, men, girl, no
Contradiction to Neutral
wearing, blue, black, man, white, two, red, sitting, young, standing
people, playing, man, woman, two, wearing, near, tall, men, old
15

---

Published as a conference paper at ICLR 2020
(a) Trained on the original dataset
aioten -o
(b) Trained on the revised dataset
(c) Trained on combined dataset
Figure 4: Thirty most important features learned by an SVM classifier trained on TF-IDF bag of words.
16

---

Published as a conference paper at ICLR 2020
The blue box contains text passage and label. Please edit this text in the textbox below, making small number of changes such that:
(a) the document remains coherent and (b) the new label (colored) accurately describes the revised passage.
Do not change any portions of the passage unnecessarily.
After modifying the passage and checking it over to make sure that coherent and matches the label.
(a) Revising IMDb movie reviews
The upper blue box contains Sentence 1. The lower blue box contains Sentence 2. Given that Sentence I True, Sentence (by implication), must either be (a) definitely True, (b) definitely False, or (c) May be True.
You presented with initial Sentence and Sentence and the correct initial relationship label (True, False, or May be True).
Please edit Sentence in the textboxes, making small number of changes such that:
(a) The new sentences are coherent and (b) The target labels (in red) accurately describe the truthfulness of the modified Sentence 2 given the original Sentence 1.
Do not change any portions of the sentence unnecessarily. After modifying the text and checking over to make sure that it is coherent and matches the target label.
(b) Revising hypothesis in SNLI
The upper blue box contains Sentence 1. The lower blue box contains Sentence 2. Given that Sentence 1 True, Sentence (by implication), must either be (a) definitely True, (b) definitely False, or (c) May be True.
You presented with initial Sentence and Sentence and the correct initial relationship label (True, False, or May be True).
Please edit Sentence in the textboxes, making small number of changes such that:
(a) The new sentences are coherent and
(b) The target labels (in red) accurately describe the truthfulness of the original Sentence 2 given the modified Sentence
Do not change any portions of the sentence unnecessarily. After modifying the text and checking over to make sure that it is coherent and matches the target label.
(c) Revising premise in SNLI
Figure 5: Instructions used on Amazon Mechanical Turk for data collection
17


Reviews:
[{"review_id": "rylen8dl5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new way to augment textual datasets for the task of sentiment analysis, in order to help the learning methods to generalize better by concentrating on learning the different that makes a difference. The main idea of the paper is to augment existing datasets with minimally counteractual versions of them, that change the sentiment of the documents. In this way, all spurious factors will naturally cancel out. The authors use the newly created datasets and show that indeed, the retrained algorithms on the augmented datasets generalize much better.\n\nThe main contribution of the paper is the introduction of the idea of counterfactual datasets for sentiment analysis.\n\nOverall, I find the idea of the paper quite interesting and I’m excited to use the datasets they have created. However, I think the relative novelty of the paper does not meet ICLR standards, and it’s better suited as a whitepaper attached to an open dataset release.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rylDGP7Zjr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n       The authors take two tasks,sentiment analysis and natural language inference, and identify datasets for them which they counterfactually augment it by asking people over the Amazon Mechanical Turk Platform to change either the sentiment (in the case of sentiment analysis) or the nature of relationship in the NLI task by making minimal changes to the text that produce the targeted changes. \n\nAuthors find that popular models trained on either fail on the other dataset while the models trained on both actually generalize much better. This is because the original sample and its counterfactual pair the label changed , has the difference in the text that matters to the change and this pair could reduce spurious correlations that models might find in the data distribution. \n\nPros: \n This is a very interesting experiment and certainly the dataset that will be released would be extremely valuable to the community. The one part (I dont have much NLP background but I do have a causality background) that I like most is that the new text generated are counterfactual in some real sense with respect to a real world generating process - that is people modifying text with changed targets.\n\n A lot of existing work that claim to do counterfactual changes do not specify assumptions about the generating mechanism. For counterfactuals to be valid they have to be intervention on the actual generating mechanism (or an assumed one) acting on a given unit (latent) that produced the current sample. The paper in that respect (even if it does not explicitly specify relationship between counterfactuals and generating mechanisms) tries to be faithful to a \"strict causal notion\" by actually asking people to modify the text. \n\nCons:\n    - I think the authors want to make an explicit connection to counterfactuals as understood in the causality community. Then they shy away from it saying they are inspired by it. May be a formal exposition in the supplement about counterfactuals and generating mechanisms could help readers from other communities (NLP) even it means repeating standard/synthetic examples. Its good to say what exactly in a counterfactual generation process, the \"people\" in amazon turk were substituting. \n\n   -  Is the romantic/ horror flips and their absence the only spurious thing in Figure 4 ? \n  -  In figure 6, it appears that BERT is sensitive to the domain - does it mean that it is bad ? - Authors indicate that ideally it must not be so. Because Table 3 results seem to indicate that BERT performs the best in almost all the cases .\n -  Can the authors highlight the best performances in each case in the Tables by a bold face.  It helps easily eye ball the best performing model.\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgaPkj0YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seeks to separate \"causal\" features from ones with spurious correlations in the context of natural language machine learning tasks. The proposed approach is to ask human annotators to alter examples in a minimal way that changes the label. Thereby the humans separate out the causal features (those changed) from the spurious or irrelevant features (those left unchanged).\n\nExperiments show that classifiers trained on the original data perform poorly on the altered data and vice versa, but (unsurprisingly) training on the union of the two datasets results in a classifier that performs well in both cases. Furthermore, training an SVM on the original results in irrelevant attributes (such as movie genre) being weighted, whereas these weights are largely removed when training on the union of the datasets. This suggests that the augmented training data results in weighting the \"right\" features more. \n\nOverall, I think this paper should be accepted because it makes several interesting contributions: It proposes an interesting approach, shows intriguing experimental results, and produces an interesting dataset (size ~2k) that may be useful for future testing.\n\nThe main limitation of the paper is that the evidence is largely circumstantial. The method has intuitive appeal and the experimental results are suggestive, but the experiments do not conclusively show that the method achieves something that ordinary machine learning does not.\n\nMy suggestion for a further experiment would be to apply the movie review classifiers to, say, book reviews -- something where the task is fundamentally the same but the context is different. If the classifier trained on the union of the original and altered datasets performs better than a classifier trained on only on dataset, then that is strong evidence that this approach yields better extrapolation.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1eNnTeLqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of building models for NLP tasks that are robust against spurious correlations in the data by introducing a human-in-the-loop method: annotators are asked to modify data-points minimally in order to change the label.  They refer to this process as counterfactual augmentation.  The authors apply this method to the IMDB sentiment dataset and to SNLI and show (among other things) that many models cannot generalize from the original dataset to the counterfactually-augmented one.\n\nThis contribution is timely and addresses a very important problem that needs to be addressed in order to build more robust NLP systems.\n\nBecause, however, of a few limitations, I recommend weak acceptance.\n\nMy main hesitation comes from a lack of clarity about the main lesson we have learned.  In particular, if the goal is to use this method to augment the data we use to train NLP systems in order to make them more robust, it seems that the time cost of the process will be prohibitive.  On the other hand, perhaps these methods could be used to identify the kind of spurious correlations that models tend to rely on, which could then be used in a more automated data augmentation process.  If that's the goal, however, a more detailed error analysis would need to be included.\n\nA few small comments:\n\n* There was some analysis of the augmented IMDB dataset, but none of the SNLI dataset.  I would love to see a more detailed investigation of what annotators usually did.  For instance, a reason that hypothesis-only models do well is that certain words are very predictive of certain labels (e.g. \"not\" and contradiction).  Do people leave the negations in when modifying such examples for entailment or neutrality, thus breaking the simple correspondence?  That's a very simple kind of question; more generally, I'd like to see more analysis of the new dataset.\n\n* The BiLSTM they use is very small (embedding and hidden dimension 50).  Given that BERT is most robust against their manipulation, it would be good to see a more powerful recurrent model for comparison.  It would be easy to use ELMo here, if the main question is about Transformers vs recurrent models.\n\n\nSome very minor / typographic comments:\n\n* abstract: \"with revise\" should be \"with revising\"\n* first paragraph page 2: some references to causality literature and definition of spuriousness as common cause\n* page 2, \"We show that...\" I'd break this into two sentences to make it easier to parse.\n* Table 3: I would make two columns for each model with accuracy on original versus revised.  With the current table, one has to compare cells in the top half of the table to those in the bottom half of the table, which is quite difficult to do.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the idea of a counterfactually augmented dataset, in which each example is paired with a manually constructed example with a different label that makes the minimal possible edit to the original example that makes that label correct. The paper justifies the value of these datasets as an aid in both understanding and building classifiers that are robust to spurious features, and releases two small examples.

On my reading, this paper presents a very substantially new idea that is relevant to a major ongoing debate in the applied machine learning literature: How do we build models that learn some intended behavior, where the primary evidence we have of that behavior comes in the form of datasets with spurious correlations/artifacts.

One reviewer argued for rejection on the grounds that dataset papers are not appropriate for publication at a main conference. I don't find that argument compelling, and I'm also not sure that it's accurate to call this paper primarily a dataset paper. We could not reach a complete consensus after further discussion. The other reviews raised some additional concerns about the paper, but the revised manuscript appears to have address them to the extent possible.
Decision: Accept (Spotlight)