SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples
Authors: Deqing Fu
Ameya Godbole
Robin Jia
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering
Counterfactual Generation
Data Augmentation

Abstract: Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models' ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.

SCENE; Self-Labeled Counterfactuals for Extrapolating to Negatíve Examples
Deqing Fu
Ameya Godbole Robin Jia
University of Southern California [deqingfu, ameyagod, robinjiaJ@usc. edu
Abstract
Question
Context
Ground Truth (GT)
highly resstant dermant sructures certain
Certen of G Crmmppo bactono, such as Boeillus, Clostiiuumm Sporehalobacter Arwerobaster and sighly
Detecting negatives (such as non-entailment re- lationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging nega- tive examples can help models detect them, both costly and fomain-specifer In this work, we propose Self-labeled Counterfactuals for Extrapolating Negative Examples (SCENE), automatic method for synthesizing train- ing data that greatly improves models' abil- ity to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels. SCENE synthesize negative exam- ples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with mask infilling model, then determines whether the resulting example is negative based on self-training heuristic. With access only an- swerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, dataset where half of the evaluation examples are unanswerable, compared to model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.
gram pesitive boctert called?
endospores
ca endesporee.
Original Prediction (OP)
Panaphraoe
endorsperss
(1)
[3a]
Perturbed Questions
Parsphras??
Parturbed Predictions (PP)
(4)
Keep Example?
contaglous highly
Self abel new truth.
cortain gram-pasitrv bxecerrie
(6) Wher are highly resistant dormart formations of cortain grom positive bacteria called?
(b) Yes
(b) andospenos
Yes Because PP-OP-EL D..
(c) What highty resoeent dormant sencchrees certain gram-positive disease
Paraphrane disagree
(c)
Figure I: The Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE) pipeline: (1) question perturbation using mask in in-filiing model (BART); (2) paraphrase detection on perturbed ques- tions; (3) QA model prediction for perturbed questions (3a) and the original question (3b); and (4) answer rela- belling or filtering based on (2) and (3) (details in $3.2), black arrows) used for training in the same way the original examples (green arrows).
Accepted new examples (generated
1 Introduction
Many natural language understanding tasks require a model to distinguish claims that are supported by available evidence (i.e., positive instances) from ones that are not (i.e., negative instances). In ques- tion answering, unanswerable questions (negative) can be subtly different from answerable ones (posi- fov instance, inserting an unmentioned entity into answerable question can make it unanswer- able. In recognizing textual entailment (RTE) or fact verification, a hypothesis or claim that is not entailed by a given premise (negative) can be very
similar to one that is entailed (positive). Train- ing models to understand these fine distinctions remains an important open problem (Kim et al., 2021; Asai and Choi, 2021).
Collecting human-writted negative examples, done by datasets like SNLT (Bowman et al., 2015), MNLI (Williams et al., 2018) and SQuAD 2.0 (Ra- jpurkar et al., 2018), 8), can yield training data that

---

helps models detect negatives. However, not only is this process expensive and sime-consuming but asking humans to write negative examples can also introduce dataset biases (Gururangan et al., 2018). Distant supervision (e.g., pairing questions with paragraphs that do not contain the answer to the question) can create negative examples at no addi- tional annotation cost (Karpukhin et al., 2020; Lee al., 2021), but the resulting examples simple for models. Training only on such exam- ples does not prepare models for subtler negative examples that only differ in a small way from pos- itive examples (e.g., see Table 1). Example (a) in Figure provides a subtle unanswerable question by altering the phrase "resistant dormant structure" to "contagious strains." Such an edit keeps the question very related to the context but changes its meaning so that within the same context, the new question is no longer answerable.
1.1 even outperforms a SQuAD 2.0-trained model when evaluated on the out-of-domain test ACE- whQA (Sulem et al., 2021). SCENE can also extr- polate from BoolQ (Clark et al., 2019), a dataset containing only Yes/No questions, to BoolQ-3L (Sulem et al., 2022) which also contains "unan- swerable/IDK" examples, closing 89.3% of the gap with model trained BoolQ-3L. It further ap- plies to RTE (Wang et al., 2018), 1),, where we start with only entailment examples and synthesize non- entailment examples, closing 56. 1% of the gap with a model trained on the full training set. Overall, our results show that automatically generated per- turbations can be useful not only for increasing the amount of training data, but for enabling extrapola- tion to previously unseen types of examples.
often
Problem Setup
We propose new approach named SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples) to automatically generate sub- tly negative examples given a dataset of only pos- itive examples. We use these synthetic examples as training data to help model recognize real neg- ative examples zero-shot, thus avoiding the cost of collecting negative examples manually. Our ap- proach first perturbs existing positive examples ran- domly using mask-denoising model (BART), then uses self-training to dynamically label these per- turbed examples as negative based on the model's current predictions (see Figure 1). To get self- training started, we also include negative examples generated by distant supervision to "warm-start" the model (i.e. train the model to predict the nega- tive class). Unlike previous work (Min al., 2020; Ross et al., 2022: Wu al., 2021: Howard al., 2022) that uses perturbations or counterfactuals for data augmentation, we use augmentation for ex- trapolation: we generate an entirely new class of examples not seen in the original training set.
Extractive QA. We first describe our setup for extractive QA, then modify it for other tasks. Our training data Positive only contains positive exam- ples where each question is answerable, i.e. exam- ples (g. p, where is the answer to question q on passage p. In particular to extractive QA, is a span in p. To extrapolate to the new unanswerable class, aim to synthesize unanswerable examples zero-shot beyond existing baselines that generate simple unanswerable ones (see $3.1). Then, we use our proposed perturbation based self telf-iraining procedure, SCENE, to generate more challenging unanswerable questions (see Figure and $3.2). We use RoBERTagase (Liu al., 2019) as our backbone QA model with parameters O, de- noted as ʃe where for any possible answer y'. fe (v' ()p) = P(y' p, q: 9). Sometimes we drop the parameter θ or shorten to f(q, p) to denote the probability output from QA model.
We adopt the standard procedure in identifying unanswerable questions extractive QA tasks (De- vlin et al., 2019): the model prediction f(a. p) gives the probability for the answer's start and end positions over the entire span of the sequence, and the O-position is always the [CLS] token. If the model has the highest probability in predicting [CLS], treat it as predicting unanswerable.
Synthetic negative examples generated by SCENE teach models to recognize real negative examples. For extractive question answering (QA), we can extrapolate from SQuAD 1.1 (Rajpurkar et al., 2016), positive-only dataset containing no unanswerable questions, to SQuAD 2.0 (Rajpurkar al., 2018), which contains human-written unan- swerable questions: our approach closes 69.6% of the gap with respect to a model trained on the SQuAD 2.0 training set. Our method using SQuAD
Other tasks. For tasks beyond extractive QA, we modify notation and representations slightly. For boolean QA, the unanswerable (IDK) exam- ples are represented as an additional class, on top of the original binary (Yes/No) classes. For

---

RTE, we represent the pmpothhsis-premiiss pair as (h, p) E Poositive with label y = entai Iment. The generated negative examples belong to the new class with label y = not entailment. We also use RoBERTagase as our baseline model.
3.2 Self-Labeled Counterfactuals (SCENE)
The previous methods generate negative examples that are too easy, and thus do not teach the model to recognize more subtle negative examples (see evaluation in $4.2). To generate harder negative amples, introduce Self-labeled Counterfactuals for Extrapolating Negative Examples (SCENE) Given an example (g. P: y) from our positive-only dataset Positive. we use a generator G to synthe- size a perturbed version G(q) of the question q- We then impute label ÿ. which is often the "unan- swerable" label, and train the QA model to output given the input (G(g). p). Prior work (Bartolo et al., 2021; Ross al., 2022) synthesizes questions based on passages but, unlike our approach, cannot introduce a distribution shift to generate examples from a novel unanswerable class.
3 Method
We first present our method for extractive QA, then show how it applies a broader set of tasks in $3.4. Our synthetic data generation process consists of two steps: the first is dedicated to generate easy unanswerable question-passage pairs; and the sec- ond is to generate harder unanswerable examples' through question perturbation and self-labelling.
3.1 Baselines for Generating Negatives
In $3.2, we will use model to self-label per- turbed questions unanswerable. For this work, we first need to introduce the concept of unanswer- able questions to the model. Thus, we present two methods to create simple unanswerable examples.
Inspired by self-training and counterfactual data augmentation (Howard et al., 2022), our method synthesize questions in three steps: Perturb, Filter, and Self-label. Figure illustrates our method, and selected examples are shown in Tables and 9.
Shuffle-Based Generation. The easiest way of synthesizing unanswerable question-passage pairs is through random pairing. To do this ef- ficiently, we randomly pair passages and ques- tions within a batch. Given a batch of question- passage pairs ((1.P1),.-- ... (4m- pm]}, we ran- domly sample an element a from the permu- tation group Sm. and reshuffle the pairs to be ((o(1), P1), (1o(2): P2), (4o(m)) Pm)). For ev- ery ls, we label the pair (go(k)) pk) to be unanswer- able if (kk) A k; otherwise, we discard the exam- ple (since it is already present the original batch). An example is shown Table 8. lStuf denotes the cross-entropy loss on shuffle-generated negatives.
We first randomly mask some to- kens from the question g and use mask-denoising model to fill the masks. Specifically, the proportion of words be masked is randomly drawn from Beta 5) Distribution, and we use LARTeagg (Lewis et al., 2020) as the mask-denoising model.
Perturb.
Before we self-label the perturbed questions and update our model on them, we first filter out perturbations for which we have lot of uncertainty about the correct label. Let öpen (p, q, y) E {0, be the indicator variable for whether we use the perturbed pair (G(g),)) p) or fil- ter it out. Given the original example (9, P. y) and perturbed question G(q), we compute the model prediction á arg max,/ fly/ q, q.P), and the per- turbed prediction ý = arg max, fly,, G(q), order to better determine whether the prediction ū is likely to be correct, we adopt the idea of re- jection sampling by using paraphrase detection model r-a RoBERTaBas model pre-trained QQP from GLUE (Wang et al., 2018)-t0 help determine whether filter out this example.
Filter.
To create harder unanswerable examples, given a question g. we retrieve a passage with high word overlap from the pool P of all passages in the dataset i.e. P= {pl (g, p. 3) E ) Given an ans answerable example (g.P, U), we create unanswerable ex- ample (q, R(g), NoAns) where Rt is the retrieval operator. In particular, R(q) returns the passage from P that does not contain the answer string y with the highest BM25 similarity (Robertson and Zaragoza, 2009) to the question q. EReir denotes the cross-entropy loss on Sttreval--eneraae negatives.
Retrieval-Based Generation.
For synthetic example (G(g)))), dis- card it (i.e., set open(p, q, 0) if one of
the following two cases happen: (1) ambiguity: T(G(g), .) Paraphrase but the perturbation changes the model prediction, i.c. & + g. This suggests that we have contradictory conclusions
"We consider "harder/suble unanswerable examples" as unanswerable questions that have high lexical overlup with original answerahle questions, even though they may not he linguistically harder or syntactically diverse.

---

Question
Context
Answer
What state American Idol contes- tant Chris Daughtry from?
Original
Ten of the fourteen Idol winners have come from the Southern United States
North Carolina
What state is American Idol con- testant Audrina Patridge and Chris Daughtry from?
[.....] including Clay Aiken, Kellic
Perturbed
Pickler, and Chris Daughtry, are all
NoAns
from North Carolina.
What is the downfall of using immu- nization against the pathogens that cause disease?
Immunization against the pathogens that
require targeting certain pathogens for vaccination
Original
diarrheal disease is viable prevention strategy, however does require targeting certain pathogens for vaccination.
What is the long-term goal of using immunization against the pathogens that cause disease?
Perturbed
prevention strategy
Table I: Synthetic examples generated through perturbation and relabelling. The first example shows a perturbation induced unanswerable by inserting an unmentioned entity, and the second shows an induced answerable example but with different answer due to meaning- perturbation.
from the paraphrase detector and the QA model, and cannot self-label it with confidence. (2) bad prediction: if the perturbation doesn't It change the QA model prediction but they're both wrong, i.e. i = 5 # y. We don't self-label with wrong labels. Note that if the QA model prediction stays the same (i.e. 9 - ?)) while the paraphrase detector predicts [(G(g), q) = aphraaaphrase, the perturbed ques- tion still passes the filter because non non-paraphrase questions can still have the same answer.
model parameters Q do not use frozen pre- trained model to statically perform SCENE aug- mentation just once at the beginning. To make sure the predictions á and ý are mostly correct with re- spect their questions, the initial weights for our QA model Se are pre-trained on the positive-only dataset D Dussiivee For warm-starting purposes (see 13.1), ASCENE O for the first few steps (see details in $B.2). The rest of the training follows the objec- tive in Eq 2. For the entire training procedure, our method is not provided with any human-annotater negatives, and therefore methods such early stop- ping, model selection, etc. cannot be used.
Self-Label. If a synthetic example passes the fil- ter, i.e., open (p, q, y) = 1, we trust the QA model's decision on the synthetic labels by self-labeling example (G(g), p) with label ÿ.
3.4 Beyond Extractive Question Answering
The batched objective for SCENE is denoted as follows, where .(),) is the eross-entropy loss.
Our method can also work beyond extractive QA tasks. SCENE can work on boolean QA in a set- ting where we extrapolate from BoolQ (Clark et al., 4019), a collection of question passage pairs (q. p) of label Yes or No, to BoolQ-3L (Sulem et al.. 2022), an extended dataset to BoolQ with addi- tional unanswerable (IDK) examples. We repeat the SCENE pipeline (Perturb, Filter and Self-label), together with Shuffle, and evaluated on BoolQ-3L test set. Because the BoolQ-3L's IDK questions were produced using information retrieval simi- lar to what we did in $3.1, we refrain from using retrieval-based IDK questions during training.
eSCENT( Pk: 8k 1)2=))
(pen Uh, uk) P() (1)
guestion-passage
3.3 Training
Putting everything together, we train on the weighted sum of the normal training objective f and synthetic unanswerable objectives defined in previous sections. We denote the batched version of the weighted training objective as follows:
lownlll (E/pe, Pk, H(
Our method can also work on binary RTE in a setting where we only have access to hypothesis- premise pairs (h, p) E DPositive of label entai mment. For RTE, modify the SCENE pipeline so that we perturb the premise p G(p). Instead of the original self-laheling step, we label G(p) to be ÿ = not not entailment if the paraphrase detector predicts that G(p) is not a paraphrase of
E e(s(ge.pu) wk) Auaaa Luceel (iene(l(gee0 1
+ Astur Cakkl wk))) ARes tRau ((gk. Pe. )))
(2)
where AShuf ARea and ÀSCENE denote weights for their corresponding losses.
Note that we perform SCENE data augmentation for every batch during training using the current

---

P. We note that unlike for QA, SCENE can gen- erate negative examples even without Shuffle or Retrieval-based negatives. The necessity of Shuffle for binary NLI tasks is ablated in experiments (see Table 7). We do not experiment with retrieval for RTE because the dataset contains only small pool of hypotheses use for retrieval.
can force the model to predict unanswerable by setting a probabilistie threshold, i.c., the model pre- diets unanswerable if PÚ p. Q) < Ochreshold- However, one would need a dataset with negative examples first to find the best Othreshold- In our experiments, we compare SCENE with an oracle approach that uses the validation set of the target dataset to choose the best threshold.
4 Experiments
4.1.2 Boolean QA
4.1 Experimental Details
We use BoolQ (Clark et al., 2019) as the training set for boolean QA. a collection of question- massage-answer triples gathered from queries to the Google search engine. Unlike SQuAD, answers in BoolQ are either Yes or No. It is split into 9,427 training examples and 3,270 validation examples. BoolQ-3L (Sulem et al., 2022) is the testing set. It an extended version of BoolQ with additional IDK (unanswerable) examples generated through retrieval. The dataset contains 4,906 validation examples, where 33% of them are IDK examples. When evaluating on BoolQ-3L, the metric of in- terest is the classification accuracy across three categories (Yes/No/IDK).
We define metric, Performance Gap, that mea- sures how much of the gap between training on positives only and the full dataset (with human- written negatives) can be closed with SCENE:
M|SCENE(D (Desstvee Fosstive) ] [Baseline ( Deeaiivvel Gap M [Baseline (Daai
] .M [Baseline Poettive)
(3)
where M any task-specific metric on the full test set D The arguments of M specify the method
and the training set.
4.1.1 Extractive QA
We use SQuAD 1.1 (Rajpurkar et al., 2016) as the training set for extractive QA. It is a collection of question paatnnppassasasaaeeeeeee
triples derived from Wikipedia articles, where the correct answers of questions can be any sequence of tokens in the given text. It is split into 87,599 training examples and 10,570 validation examples.
4.1.3 Recognizing Textual Entailment
RTE (Wang al., 2018) is a collection of hypothesis-premine pairs labeled as either "entail- ment" or "not-entailment." It has 2,490 training examples and 277 validation examples. For our method, we only use the 1,249 entailment subset from the training set our training data Drositie. We still test the full validation split.
SQuAD 2.0 (Rajpurkar et al., 2018) is the testing set. It is an extended version of SQuAD 1.1 with additional unanswerable questions. We evaluate on the development set, which contains 11,873 examples-5,, 5,928 examples with answers and 5,945 unanswerable examples. When evaluat- ing on SQuAD 2.0, the metries of interest are Exact Match (EM) accuracy and Fi scores.
4.2 Extractive QA Results
For comparison as baseline/oracle models, we train normally both on the positive-only training set Positive and on the full training set Drul- We eval- uated on 3 different independent runs of training and report averages to mitigate the effects of ran- domness. For simplicity, we only let A's be zero or one, unless otherwise indicated.
ACE-whQA (Sulem et al., 2021) is the out- of-domain (OOD) testing set for extractive QA, derived from an event extraction corpus, ACE (Walker et al., 2006). Aside from being OOD test set for SQuAD 2.0, ACE-whQA also distin- guishes two types of unanswerable (IDK) ques- tions: (1) competitive, where the passage includes an entity of the same type as the expected answer; and (2) non-competitive, where the passage does not include an entity of the same type. contains 238 answerable examples, 250 competitive IDK examples, and 245 non-competitive IDK examples.
SCENE enables extrapolation to SQuAD 2.0. Table 2 shows results on the SQuAD 2.0 evaluation set. The best SCENE method achieves 71.8 FI, which closes 69.6% of the gap in SQuAD 2.0 ac- curacy between the bascline that trains SQuAD 1.1 (45.7 F1) and the oracle method that trains on SQuAD 2.0 (83.2 F1). Our best method combines retrieval with self-labeled counterfactuals; this is 7.5 Fl points better than only using retrieval. Using nuuffle-bssel negative examples barely improves
A Probabilistic Alternative. Instead of train- ing with
negative examples, one

---

Has Answer
No Answer
Average
Training Set
Method
EM
Pi
HM
RM
Fi
No Augmentation
84.4 (0.2)
91.5 (0.1)
0.0 (0.0)
0.0 (0.0)
42.1 (0.1)
45.7 (0.1)
Best Threshold
39.8 (7.4)
40.2 (7.7)
68.8 (7.5)
68.8 (7.5)
54.4 (0.1)
54.5 (0.1)
Shufte
84.1 (0.3)
91.3 (0.1)
3.4 (0.3)
3.4 (0.3)
43.7 (0.1)
47.3 (0.1)
SQuAD 1.1
Shuf + SCENE
80.1 (0.2)
86.5 (0.1)
40.8 (1.7)
40.8 (1.7)
60.4 (0.8)
63.7 (0.8)
Retrieval
79.3 (0.1)
85.5 (0.2)
43.1 (0.7)
43.1 (0.7)
61.2 (0.4)
(0.4)
Retr + SCENE
75.8 (0.6)
81.4 (0.6)
62.2 (0.9)
62.2 (0.9)
69.0 (0.2)
71.8 (0.1)
No Augmentation
78.2 (0.2)
84.6 (0.3)
81.8 (0.5)
81.8 (0.5)
80.0 (0.2)
83.2 (0.1)
SQuAD 2.0
SCENE
76.3 (0.3)
(0.2)
84.6 (0.4)
84.6 (0.4)
80.4 (0.1)
83.4 (0.1)
Table Evaluation on SQuAD 2.0. Training on positive-only SQuAD I.I, SCENE with Retrieval can close 69.6%
of the gap compared with oracle model trained on SQuAD 2.0. We report the mean and standard deviation (in parentheses) over 3 runs. Best scores among all methods that train on SQuAD 1.1 are highlighted in bold.
Has Answer
No Answer
Average
Training Set
Method
Competitive EMIF
Non-Comp EM/F
E.M
Fi
EM
Fj
No Augmentation
79.1 (0.5)
86.8 (0.8)
1.1 (1.2)
7.9 (7.8)
41.8 (2.1)
45.6 (1.9)
Best Threshold
(3.5)
57.5 (3.7)
68.9 (3.8)
(2.4)
68.7 (0.5)
69.3 (0.5)
Shuffle
69.1 (1.8)
(1.0
46.3 (3.5)
73.7)
65.2 (1.8)
69.7 (1.5)
SQuAD 1.1
Shuf + SCENE
(3.7)
65.1 (4.9)
69.7 (5.0)
87.3 (2.7)
68.7 (0.2)
71.8 (0.6)
Retrieval
59.0 (0.2)
65.4 (0.8)
68.1 (1.8)
86.9 (1.5)
68.2 (0.9)
71.4 (0.6)
Retr + SCENE
55.0
61.7 (4.2)
73.8 (4.4)
95.3 (0.4)
69.8 (0.9)
73.1 (0.9)
No Augmentation SCENE
(0.8)
31.7 (1.1)
60.2 (10.4)
58.4
62.3 (3.2)
SQuAD 2.0
49.2
56.9 (1.8)
75.4 (2.1)
91.3 (1.1)
66.3 (0.7)
70.1 (0.4)
Table OOD Evaluation on ACE-whQA. Trained only on SQuAD I.1, SCENE with Retrieval even outperforms the oracle model trained on SQuAD 2.0. Applying SCENE also improves SQuAD 2.0-trained models oft-of-domain. Best scores among all methods that train on SQuAD 1.1 are highlighted in bold.
over the SQuAD 1.1 baseline; adding self-labeled counterfactuals this improves F1 by 16.4. Com- pared with setting probabilistic thresholds force unanswerable predictions, our best method does not require additional annotated SQuAD 2.0 ex- amples to find thresholds and still achieves higher performance. Finally, adding self-labeled counter- factuals to the SQuAD 2.0 training set does not significantly improve SQuAD 2.0 accuracy since synthetic unanswerable examples are not as strong numan-annotall in-distribution ones. Note that training on either gold-standard SQuAD 2.0 neg- atives or SCENE negatives reduces accuracy on the "HasAns" subset, as introducing the concept of unanswerability inevitably causes models to some- times predict "NoAns" on answerable questions.
model trained SQuAD 2.0 (73.1 FI 62.3 FI). Adding SCENE examples to the SQuAD 2.0 training data improves the ACE-whQA accuracy by 7.8 F1, suggesting that including SCENE improves the overall utility of SQuAD 2.0 as training data.
SCENE can synthesize subtly unanswerable questions in many ways, such as inserting replacing unmen- tioned entities, lexical changes, and tense modifica- tions. Tables and 9 show selected synthetic unanswerable examples generated via SCENE, and ones. In Ta-
Qualitative results and statistics.
Table 10 shows randomly-sampi ble 11, we adopt different categories of negative examples defined by Rajpurkar et al. (2018) and compare their frequency in SCENE with SQuAD 2.0. SCENE tends to create more questions that add an impossible condition, and fewer antonym substitutions or negations since the BART model is not tuned particularly produce these edits.
SCENE improves out-of-domain generalization. Table 3 shows results where we train on SQuAD- derived data and test ACE-whQA. We focus the overall average accuracy, where we average the accuracy on the "HasAns" and "NoAns" subsets of ACE-whQA. SCENE combined with retrieval and the posltive-only SQuAD 1.1 dataset achieves the highest accuracy, even outperforming the oracle
A blation study. We conduct ablation study on our best method, which combines retrieval and self- labeled counterfactuals. We test several simplified versions of our filtering and relabeling pipeline:

---

Method
SQuAD 2.0
ACE-whQA
Shuffic
Retrieval SCENE
FM
Fj
F.
Fi
No Augmentation
42.1 (0.1)
45.7 (0.1)
41.8 (2.1)
45.6 (1.9
43.7 (0.1)
65.2 (1.8)
69.7 (1.5)
61.3 (0.2)
64.3 (0.2)
69.4 (0.7)
72.8 (0.5)
60.4 (0.8)
63.7 (0.8)
68.7 (0.2)
71.8 (0.6)
61.2 (0.4)
64.2 (0.4)
68.2 (0.9)
71.4 (0.6)
69.0 (0.2)
71.8 (0.1)
69.8 (0.9)
73.1 (0.9)
69.7 (0.4)
71.7 (0.2)
63.4 (1.6)
65.9 (1.6)
Table Ablation study on combinations negative example generation methods. All experiments are trained SQuAD 1.1 and evaluated SQuAD 2.0 and ACE-whQA. Best scores are highlighted in bold. Our best model uses the combination of Retrieval and SCENE.
In-Domain Evaluation SQuAD 2.0
as it encourages the model to predict NoAns too much. Counter-intuitively, both "Assume NoAns w/o Retr" and "Only NoAns" reduce accuracy on unanswerable questions in SQuAD 2.0. One possi- ble explanation that if all perturbations are unan- swerable, detecting unanswerables becomes the task of detecting perturbations; also including per- turbed answerable questions reduces this spurious correlation. Finally, removing the paraphrase de- tector decreases accuracy by 1.9 F1 on SQuAD 2.0 and 5.0 F1 on ACE-whQA, showing that our filtering strategy does help reduce noise, but self- training alone without filtering is still effective.
Has Answer
No Answer
Average
Method
EM
Fi
EM
F1
EM
F1
Assume NoAns
66.2
70.2
70.8
70,8
68.5
70.5
w/o Retr
71.3
76.0
58.8
58.8
65.0
67.4
Only NoAns
76.8
82.5
59.0
59.0
67.9
70.7
No Filter
69.9
74.9
65.1
65.1
67.5
69.9
SCENE (ours)
75.8
81.4
62.2
62.2
68.9
71.8
Out-of-Domain Evaluation on ACE-whIA
Has Answer
No Answer
Average
Method
Non Comp
EM
Fi
Comp
EM
Fi
Assume NoAns
23.9
27.5
96.0
99.2
60.8
62.6
w/o Retr
23.9
26.4
91.2
96.2
58.8
60.1
Only NoAns
43.0
49.4
85.3
98.2
67.4
70.6
No Filter
37.1
41.6
90.4
98.9
65.9
68.1
SCENE (ours)
55.0
61.7
73.8
95.3
69.8
73.1
We also conduct ablation study on different com- binations of losses, (Suut, fRer and ESCENE. presented in Table 4. Results suggest that our best model uses the combination of Retrieval and SCENE and performing Shuffle together with Retrieval can be redundant since negatives provided by Retrieval are refined subset of examples provided by Shuffle.
Table 5: Ablation study different filtering and labeling strategies for synthetic samples. The full SCENE pipeline achieves the hest performance both in-domain and out-of- domain.
Assume NoAns: We accept every perturbed ex- ample and relabel everyone as unanswerable, i.e.
4.3 Boolean QA Results
V(p, q, y), (p, 9.3) = 1 and ÿ = NoAns.
2. Assume NoAns w/o Retr: Same as above, but without Retrieval to generate simple negatives.
SCENE enables extrapolation to BoolQ-3L, shown in Table 6. Note that be consistent with our extractive QA experiments, we report the av- erage of the accuracy on negative examples (IDK label) and positive examples (yes and no labels). The best SCENE method achieves 78. accuracy, which closes 89.3% of the gap in BoolQ-3L ac- curacy between the baseline that trains on BoolQ (38.9% accuracy) and the oracle method that trains on BoolQ-3L (82.8% accuracy). Our best method combines shuffle with self-labeled counterfactu- als. Adding self-labelled counterfactuals to the BoolQ-3L training set slightly improves BoolQ-3L accuracy by 0.4 points.
3. Only NoAns: We additionally filter out per- turbed questions for which our imputed label is nor unanswerable, i.e., + NoAns. This tests whether the answerable questions generated by SCENE contribute to the performance.
4. No Filter: We do not filter and we perform self- training with synthetic examples (G(g). p, Ø) Results of these ablations shown in Table 5. Our full approach performs the best both in-domain and out-of-domain. Accepting perturbations as unanswerable surprisingly competitive on SQuAD 2.0, but is much worse on ACE-whQA,

---

Positive
Negative
to annotate new examples for the next batch (stu- dent), and it is also used jointly with a paraphrase detector (as a rejection sampler) to reduce noise. SCENE also differs from standard self-training in that, rather than using a pool of unlabelled exam- plcs, we generate them.
Train Set
Method
Average
Yes
No
IDK
No Augment
83.9 72.0
0.0
38.9
BoolQ
Shuffle
79.9 73.6
78.4
77.5
(Yes/Na)
Shuf + SCENE
77.1 71.7
81.7
78.1
BoolQ-3I.
No Augment
80.9
73.5
88.4
82.8
(Ye/Ne/DDK
SCENE
82.9 70.5
89.7
83.2
Hard Negative Mining, Several NLP datasets have gathered instances of the negative class for their task. Some have relied on human annotation e.g., unsupported claims in fact verification (Aly et al., 2021; Wadden et al., 2020), non-entailed hy- potheses in NLI (Bowman al., 2015), unanswer- able questions (Rajpurkar al., 2018), inter alia. Some have used heuristics and external knowledge sources to automatically mine negative examples (Lee et al., 2021; Wright et al., 2022). Finally, there are hybrid approaches where candidate nega- tive examples are first automatically gathered and then manually verified (Wadden al., 2022). Our baseline approach for generating negatives based on in-batch negatives via shuffling and retrieval is similar motivation mining negatives for neural ranking models (Karpukhin et al., 2020).
Table 6: Evaluation on BoolQ-3L. SCENE with BoolQ alone achieves 89.3% of the performance of oracle model trained on BoolQ-3L. Best scores among all meth- ods that train on BoolQ are highlighted bold.
Training Set
Method
Accuracy
No Augmentation
52.7
Entailment-Onjj
Shuffle
54.2
Suhset (RTFF
SCENE
64.3
Shuf + SCENE
67.9
Full Dataset
No Augmentation
79.8
Table 7: Evaluation on RTE. SCENE with RTEROS alone achieves 56. of the performance of oracle model trained on RTE. Best scores among all methods that train on the RTE are highlighted in bold.
4.4
RTE Results
Counterfactual Data Augmentation. Counter- factual data augmentation (Kaushik et al., 2020) has been studied as an approach to reduce model reliance on spurious correlations (Gardner et al., 2021) by creating minimal pairs that flip the la- bel. Past work includes manual approaches such as Contrast Sets (Gardner al., 2020) which asks expert annotators to minimally perturb exam- ples, or heuristic approaches that rely on synonym- antonym sets to substitute words in the exam- ple (Wang and Culotta, 2021; Chen et al., 2021). Several approaches leveraging language models (LM) for generating counterfactuals have been proposed: Polyjuice (Wu et al., 2021) and Tai- lor (Ross et al., 2022) train LMs to generate mini- mal edit counterfactuals through specified control codes, LIT (Li et al., 2020) automatically gener- ates contrasts sets for NLI tasks based on linguistic rules but lacks flexibility, and NeuroCounterfactu- als (Howard et al., 2022) generates counterfactuals that are relevant but not restricted to be minimal edits by leveraging constrained decoding from task fine-tuned LMs. Despite their success in perturb- ing texts, is unclear how use these methods introduce distribution shift and extrapolate un- seen label sets (in our case, negative classes). Our approach of using masked language models for text in-filling followed by filtering is inspired by recent
SCENE also extends to binary RTE tasks by en- abling extrapolation from the entailment-only sub- set to all of RTE, as shown in Table 7. The best SCENE method achieves 67.9% accuracy, which closes 56. % of the gap in RTE accuracy between the baseline that trains on entailment-only subset (52.7% accuracy) and the oracle method that trains on full RTE (79.8%). Our best method combines shuffle and self-labeled counterfactuals. Using shuffle-based negative examples barely improves over the baseline that trains on the subset, and using self-labeled counterfactuals alone improves over the baseline by 11.6 points. Combining shuffle and selfela counterfactuals improves accuracy by 13.7 and 3.6 points respectively, compared to using each only their own.
5 Discussion and Related Work
Self-Training Self-training is semi-supervise learning approach that utilizes teacher model to assign labels to unlabelled data, which is then used train student model (Yarowsky, 1995; Mc- Closky et al., 2006; Kumar et al., 2020). In our work, we adopt a self-training scheme where the teacher model and the student model are the same. During training, the current model (teacher) is used

---

work on open set classification (Xu et al., 2022).
by paraphrase detector during training, both of which add computational overhead.
Synthetic Question Generation. Many ap- proaches have been proposed synthesize ques- tions given passage and an answer within it (Du et al., 2017; 7; Du and Cardie, 2018; Lewis and Fan, 2019; Alberti et al., 2019; Bartolo et al., 2021; Lewis et al., 2021). These methods are designed to generate answerable question-answer pairs and do not directly apply to our setting where we must syn- thesize challenging unanswerable questions. Pan et al. (2021) generate (topically relevant) unanswer- able questions by conditioning a question gener- ator on phrases from the surrounding context of passages used the QA dataset. SCENE does not assume access to such external text sources (e.g. source documents of QA datasets).
We only validated on extractive QA, boolean QA and RTE. Whether SCENE can be applied to other tasks that require detecting challenging negatives is unknown. SCENE is also limited to extrapolating to one pre-defined new class, negative examples; whether SCENE can used to extrapolate to other types of classes is also unknown.
While Large Language Models (with billions of parameters) have demonstrated the ability to per- form zero/few-shot generalization on unseen tasks, their performance still lags behind fine-tuned, task- specific models. Thus, our method and analysis focus on less expensive smaller language models as they are high-performing while being cheaper to evaluate and fine-tune. Moreover, the pre- tre-training and fine-tuning data for Large Language Models is largely unknown. Additional human feedback is used for training models abstain from answer- ing certain queries (Ouyang et al., 2022). Since SCENE is an approach for training models to de- tect examples from an unseen negative class, we remove the confounding variable of unknown train- ing data by focusing on language models with pub- lic training corpora without human intervention. Finally, it can be an unfair comparison with LLMs open-weh datasets like SQuAD due data con- tamination concerns when LLMs are trained. Data contamination acknowledgements can be found at Section 8 from the PaLM technical report (Chowd- hery et al., 2022) where SQuAD 2.0 is mentioned specifically as contaminated.. Nevertheless, we will consider LLMs for future research and how our proposed method would fit into the LLM cy- cles.
6 Conclusion
In this work, we have developed counterfactual generation pipeline, SCENE, which synthesizes negative examples (as well as some positive exam- ples) from real positive ones. The counterfactual generation is performed through text perturbation, filtering and relabelling. SCENE enables extrapo- lation to negative examples given only real positive examples, closing 69.6% of the performance gap on extractive QA, 89.3% on boolean QA, and 56. % on RTE, compared with models trained on both real positive and negative ones.
In the future, we hope to combine our automatic pipeline with human annotation effort. For exam- ple, we could use adversarial data collection to collect a small number of perturbations that cre- ate challenging negative examples, and use these to guide generation of labeled counterfactu- als. We would also like to explore ways to back- propagate signal from the filtering process into the generator, so the generator learns to generate use- ful perturbations. Overall, we hope that our work can inspire more work on how synthetic data can enable new types of model extrapolation.
Acknowledgements
We thank Ting- Yun Chang, Johnny Wei, Wang Zhu, and the members of USC NLP Group for their valuable feedback. This work was supported in part by an Open Philanthropy research grant and a Google Scholar Research Award.
Limitations
Though our method alleviates data collection cost by human annotators, its computational cost is higher than training with human annotated datasets for multiple reasons (see Appendix B.3). Adding synthetic examples increases the time required for one epoch of training. Moreover, SCENE exam- ples are generated with a BART model and filtered
References
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin. and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6 6168-6173, Flo-

---

rence, Italy, Association for Computational Linguis- tics.
Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brenman Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Rami Aly. Christos Christodoulopoulls, Oana Co- caraseu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos, editors. 2021. Proceedings of the Fourth Workshop on Fact Extruction and VERification (FEVER). As- sociation for Computational Linguistics, Dominican Republic.
Akari Asai and Eunsol Choi. 2021. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1492-1504, Online. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge.
Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. Improving question answering model robustness with synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 8830-8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. Machine learning challenges. evalu- ating predictive uncertainty, visual object clas- sifi- cation, and recognising tectual entailment, pages 177-190. Springer.
Jacob Devlin, Ming-Wae Hing-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009, The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analy- sis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009. NIST.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Xinya Du and Claire Cardie. 2018. Harvest- ing paragraph-Ievel question-answer pairs from Wikipedia. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1907-1917, Mel- bourne, Australia. Association for Computational Linguisties.
Hao Chen, Rui Xia, and Jianfei Yu. 2021. Reinforced counterfactual data augmentation for dual sentiment classification. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 269-278, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 1342-1352, Vancouver, Canada. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hamnanch Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer

---

Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1323, Online. Association for Computational Linguistics.
ference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 5468- 5479. PMLR.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6634-6647, Online. Association for Computational Linguistics.
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts language data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1801-1813, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Mike Lewis and Angela Fan. 2019. Generative question answering: Learning to answer the whole question. In International Conference on Learning Representa- tions.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. 2007. The third PASCAL recog- nizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9. Association for Computational Linguis- tics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-t--sqquene pre-training for natural language generation, tramslation, and com- prehension. In Proceeding of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy. Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018, Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-1 12, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Patrick Lewis, Yuxiang Wu, Linging Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 mil- lion probably-asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics. 9:1098-1115.
Phillip Howard, Gadi Singer, Vasudev Lal, Yejin Choi, and Swabha Swayamdipta. 2022. Neurocounterfac- tuals: Beyond minimal-edit counterfactuals for richer data augmentation.
Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkele Threlkeld. 2020. engguistically-iffrmmeee transformations (LIT): A method for automatically generating contrast sets. In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pages 126-135, Online, Association for Com- putational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min. Patrick Lewis. Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Jimmy Lin, Xueguang Ma. Sheng-Chich Lin, Jheng- Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021). pages 2356-2366 2362.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes differ- ence with counterfactullly-angeeee national Conference on Learning Representations.
data. In Inter-
Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist in- vented the lightbulb? presupposition verification for crestion-answeringg In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3932-3945, Online, Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self- training for parsing. In Proceed- ings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159, New York City, USA. Association for Computational Linguistics.
Ananya Kumar, Tengyu Ma, and Perey Liang. 2020. Understanding self-training for gradual domain adap- tation. In Proceedings of the 37th International Con-

---

Junghyun Min, Thomas McCoy, Dipanjan Das, Emily Pitter, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, Online. Association for Computa- tional Linguisties.
Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do we know what we don't know? studying unanswerable questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 4543-4548, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
pages
Elior Sulem, Jamaal Hay, and Dan Roth. 2022. Yes, or IDK: The challenge of unanswerable yes/no questions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1075-1085 Seattle, United States. Association for Computational Linguistics.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational I Linguistics.
Liangming Pan, Wenhu Chen, Wenhan Xiong, Min- Yen Kan, and William Yang Wang. 2021. Zero-shot fact verification by claim generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 476-483, Online. Association for Computational Linguistics.
David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. Scifact-open: Towards open-domain scientific claim verification.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veti,o, Martin Raison, Alykhan Te- jani. Sasank Chilamkurthy, Benoit Steiner, Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, pigh-performance deep learning library. Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Philadelphia. Linguistic Data Con- sortium.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding, In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisticy (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
Zhao Wang and Aron Culotta. 2021. Robustness to spurious correlations in text classification via auto- matically generated counterfactuals. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16): 1403-1
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A bover-ceverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Pupers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr. (44):333-389.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ane natural language processing.
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew Pe- ters, and Matt Gardner. 2022. Tailor: Generating and perturbing text with semantic controls. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers),, pages 19143223,,11 Dublin, Ireland. Association for Computational Linguistics.

---

Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Dustin Wright, David Wadden, Kyle Lo, Bailey Kuchl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating scientific claims for zero- shot scientific fact checking. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 2448-2460, Dublin, Ireland. Association for Compu- tational Linguistics.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. Polyjuice: Generating counter- factuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 6707-6723, Online. Association for Computa- tional Linguistics.
Jeffrry
Albert Xu, Xiang Ren, and Robin Jia. 2022. CoNAL: Anticipating outliers with large language models. ArXiv, abs/2211.15718.
David Yarowsky. 1995. Unsupervised word sense dis- ambiguation rivaling supervised methods. 33rd Anmual Meeting of the Association for Computa- tional Linguistics, pages 189-196, Cambridge, Mas- sachusetts, USA. Association for Computational Lin- guistics.

---

A Examples and Statisties
A.1 Selected Examples
Original
Shuffle
Information Retrieval
Zinc oxide helieved to he men- tioned in what ancient text?
When did Scoul Semiconductor release the first high DC voltage LED?
Zine oxide is believed to be men- tioned in what ancient text?
Question
The Churaka Sambita, thought have boen written hetween 500 and 500 AD, mentions a metal which, when oxidized. produces push- panjan, thought to be zine uxide. Zine mines Zawar, near Udaipur in India, have been active since Mauryan period. smelt- ing of metallic zine here, however, appears contury AD.
The Charaka Samhitt, thought have writen herween 300 and 500 AD, mentions a metal which, when oxidizzd, produces push- punjan, thought to be zinc uxide. Zinc mines Zawar, near Udaipur in India, have been active since Mauryan period. smelt- ing of metallic zime here. however. appears contury AD.
Roughly one quarter zine culput the United States (2009), is consamed in the form of sine compounds: varity of which are used industrially. Zinc oxide widely used white pignaent in paints, and as catalyst the manufacture rubber.
Context
begun
begun
Answer
Charaka Samhita
No Answer
No Answer
Table 8: Examples of Simple Unanswerable Questions using Shuffle Retrieval.
Question
Context
Self-Labeled Answer
In which video does it show Madonna heing scolded by her boss in Italian?
Original
Madonna's Italian- Matholc backgrund and her relatiomhip with ber parents are reflected in the altum Like Prayer, [....] 'The "Open Your Hear" video soes hoss scolding her the Italian language On the Who's 's That Girl World Tour, she the song "Papa Preach" to Pope John Paul
Open Your Heart
Perturbed
what vidco does it sound like
Madonna being scolded by her mother in Italian?
NoAns
What are highly resistunt dormant structures of certain gram-positive bacteria called?
Original
Certain genera of Gram positive bacteria, such us Bacillus, Clostridiom, Sporohalobucter. Amaerobacter, and Heliobucterium, can form highly resistant, darmant structures called endospores. Endospores have central containing DNA and ribosomes sariounded by cortex layer.
endospores
What hhat are highly contagious strains of certain gram-positive bacteria called?
cytoplasm
Perturbed
NoAns
When did the Everton club board fire Smith?
Original
The Everton board finally Smith and he was sacked March 2002 after an FA Cup exit Middishhrough, with Everton in real danger relegation. Everton qualified 2007-08 and 2008-09 UEFA Cup competitions and they minner-upp the 2009 FA Cup Final.
patience with
March 2002
When will the football club finally fire Smith?
Perturbed
NoAns
revoked the first assent
Original
What did the Sevvenor-Geeerre do with the first assent?
In Australia, tochnical issue with the royal
astent 1976 1976, originating in House of Representatives was mistakenly submitted Gioeeroo--Generall
I....] The Governor- General revoked the first assent, before assenting to the hill which had actullly passed.
Perturbed
What does the Governor General do for the royal assent?
NoAns
Original
What is the Vietnamese word for both blue and green?
langunges, including Chinese, Thai, Japanese, and Vietnamese, the same word can mean either blue green. [....] Viernamese single word for both blue and green, sanh.
xanh
Perturbed
What color are Vietnamese flags, both blue and green?
NoAns
Table 9; Additional Examples of Hard Unanswerable Questions Generated through Perturbation

---

A.2 Kandomly-Sample Examples
Question
Context
Self-Labeled Answer
How long did the amendment extend the trade agreements?
Original
On 10 January 1941, Germany and the Soviet Union signed an agreement settling sevemi ongoing issaes. [....] It also extended trade regulation of the 1940 German-Soviet Commercial Agreement Angust I, 1942, [.....
until August 1, 1942
How long will the amendment ex- tend the trade agreements??
Perurbed
NoAns
Original
Who are Oklahoma's 's US Senators?
112th Congress, Oklahoma's U.S. senators were Republicans Jim Inhofe and Tom Coburn, and U.S. Representatives were John Sullivan
Jim Inhofe and Tom Coburn
(R-OK-1), Dan Boren (D-OK-2), Hrank D. Lucas (R B-OO-- 4), Tom Cole (R-OK 4), and James Lankford (R- (R-OK-5) 5).
Who are the US Senators?
Perturbed
NoAns
Original
How many 60 60-gun frigates did the Russians lose in the Black Sea?
During the siege, the Russians lost four 110- or 120-gun, thres-dacker ships of the line. rwclive 84 gum two deckers and 60 gun frigates in Black Sea, plus a large number of smaller vassels.
four
How many 60-gun warships are in the waters in the Black Sea?
Perturbed
NoAns
Original
The second truth is?
The second truth that the origin of dukkhha known. Within the contexi of the four noble trulls. the origin dukkha commoniy explained as craving (Pali: tanha) conditioned by ignorance (Pali: avijia).
the origin of dukkha can be known
Perturbed
The second question is?
NoAns
Original
Buddhism's 's spread led to what large- scale effort?
Asia, spread Ruddhism led to large-scale ongoing translation efforts spanning well over a thousand years. Tangut Empire efficient in such efforts; exploiting the then newly invented block prinring, and with the fall support of the government [.....]
translation
espocially
What led to what large-scale effort?
Perturbed
the spread of Buddhism
Original
What type of theatrical uniforms did Paul VI eradicate from the Vatican?
Paul remounce many traditional symbols the papacy and the Cahholic Church: some of his reversed by Pape
army
changes to the papal dress Renedict XVI in the carly er century. Refasing Vaticun anmy colourful military uniforms from ccaturics, he pot of them. He became the lirst pope to visit five continents.
What type of theatrical production will Pope Paul visit the Vatican?
Perturbed
colourful military
Along with water vapor, what atmo- spheric substance primarily absorbs the infrared emitted by the Earth?
Original
Earth's is surface and the clouds absarb visible and invisible radiation from the sun and reemit much of the energy as inftured back to atmosphere. Certain substances the atmospheree chiefly cloud droplets and water vapor, but also curbon dioxide, methane, nitrous oxidc, sulfur hexafluoride, and chleoffuorccaroon absorb this infrared, and re-tadiate it in all directions including back Earth.
cloud droplcts
Along with water, what atmospheric substance also absorbs the heat emit- ted by the Earth?
Perturbed
carbon dioxide
Original
When did pragmatism arisc?
19th and carly century several forms of pragmatic philesophy arose. The ideas pragmatism, in its various forms, developed muinly from discussions berween Charles Sanders Peires and William James when both men were Harvard 1870s.
the late 19th and early 20th century
Where did pragmatism arise?
Perturbed
Harvard in the 1870s
Table 10: Randomly Sampled Examples Generated through SCENE

---

A.3 Types of Synthetic Unanswerable Examples
Percentage
Category
Description
SCENE
SQuAD 2.0
Negation
Negation word inserted or removed.
2%
9%
Antonym
Antonym used.
2%
20%
Entity, number, date replaced with other entity, number, date.
Entity Swap
28%
21%
Word or phrase is mutually exclusive with something for which an answer is present.
Mutual Exclusion
15%
15%
Asks for condition that is not satisfied by anything in the paragraph.
Impossible Condition
23%
4%
Other cases where the paragraph does not imply any answer.
Other Neutral
10%
24%
Answerable
Question is answerable (i.e. dataset noise).
5%
7%
Question not well formulated. (i.e. gen- eration noise, only specific SCENE).
Ill-formed questions
15%
0%
Table 11: Types of negative examples generated by SCENE, compared with SQuAD in frequency.
B
Model Details
B.1 Scientific Artifacts
Regarding softwares, we enumerate their license here. PyTorch (Paszke al., 2019) was used the deep learning framework, and is licensed under BSD-3. Transformers library (Wolf et al.. 2020) was used for training and evaluation, and licensed under Apache License Version 2.0. BM25 retrieval was implemented using the Pyserini toolkit (Lin et al., 2021) and is licensed under Apache License 2.0.
Regarding datasets used in this work, we enumerate there license here. SQuAD 1.1 (Rajpurkar et al., 2016) and SQuAD 2.0 (Rajpurkar et al., 2018) are licensed under CC BY-SA 4.0. RTE (Wang et al., 2018; Dagan et al., 2006; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) dataset's 's license is unknown. BoolQ (Clark et al., 2019) is licensed under CC BY-SA 3.0. BoolQ-3L (Sulem et al., 2022) is licensed under CC BY-SA 3.0. ACE-whQA (Sulem al., 2021) is licensed under CC BY-SA 3.0.
B.2 Model Hyperparamters
Models for extractive QA were trained on Nvidia Quadro RTX 6000 GPUs with 24GB GPU Memory, and models for boolean QA and RTE were trained on Nvidia GeForce RTX 2080 GPUs with 1IGB GPU Memory. We report hyperparameters for fine-tuning RoBERTages across all experiments.

---

Task
Hyper- Parameter
SQuAD
BoolQ
RTE
train batch size
32
16
16
max seq length
384
256
128
doc stride
128
128
/
scheduler
Linear
Linear
Linear
optimizer
A W
Adam W
Adam W
adam_ l
0.9
0.9
0.9
0.999
0.98
0.999
addm__epsilon
le-8
le-8
le-8
num epochs
3
10
10
warmup ratio
0.06
0.06
0.06
learning rate
2e-5
le-5
5e-6
weight decay
0.01
0.01
0.1
Table 12: Hyperparamters for training RoBERTagse various tasks.
Warm-starting Details. For warm-starting purposes, ASCENE in Eq 2 for the first 100 steps.
and ASCENE resumes to for the rest of the training procedure.
B.3 Computational Cost
Training Set
Examples (#)
Method
Hours
#/sec
Real
Synthetic
Total
No Augmentation
SQuAD 1.1
87,509
0
87,509
2
12
SCENE (ours)
SQuAD 1.1
87,509
87,509 Retrieval + 87,509 Perturbation
262,527
9
8
No Augmentation
SQuAD 2.0
130,232
0
130,232
3
12
SCENE (ours)
SQuAD 2.0
130.232
130,232 Perturatton
260.464
8
Table 13: Expected Computational Cost in Training Our Method (SCENE) Compared with No-Augmentation Baseline on SQuAD. Because SCENE uses the additional BART mask infiller and QQP pre-trained paraphrase detector, it's expected be slower in training throughput.


Reviews:
[{"review_id": "wIOcAt6NLS", "reviewer": "Reviewer_b1yf", "summary": "", "questions": "3A) How do you measure the quality of the generated counterfactuals? Did you find the need for any sort of filteration?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions. We are glad that the reviewer acknowledged the importance of our main goal: generating subtle negative samples for question answering. We are excited to see that the reviewer found our methods sound, comparisons comprehensive and results justified well. We would also like to respond to the following comments:\n\n**Quality of the Generated Counterfactuals**: We measure the quality of generated counterfactuals by how successfully the perturbation changes the model prediction. We also performed manual inspections of our generated counterfactuals in Appendix A, and concluded that our generated counterfactuals are more subtle and harder to detect unanswerability than both Shuffling and Retrieval baselines. \n\n**Necessity of Filtering**: Please see lines 223-254 for the filtering steps, where we use an off-the-shelf paraphrase detector to filter out bad examples. For ablation on different filtering components, please see our ablation section at Lines 447-488 and Table 4, where we found that removing all filtering processes and assuming all generated counterfactuals NoAns decreased the F1 score by 10.5 points on ACE-whQA, compared to our method SCENE. Therefore, we believe the filleting process is necessary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The methods and comparisons made are sound and exhaustive.\n2) The results are justified by the explanations.\n", "reasons_to_reject": "None, fairly comprehensive work.", "questions_to_authors": "3A) How do you measure the quality of the generated counterfactuals? Did you find the need for any sort of filteration?", "paper_topic_and_main_contributions": "This paper introfuces a new method for generating negative samples for QA mthods. This is an important (and often overlookd problem)", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sGZJmp7UPM", "reviewer": "Reviewer_NjSu", "summary": "", "questions": "1. It has been emphasized many times in the paper that SCENE is able to construct \"harder negative examples\", but the description of \"harder\" is very vague, so can the hardness of the samples be described more accurately? Compared to SCENE, I think the retrieval-based generation method yields syntactically more diverse negative examples, so perhaps SCENE does not produce harder negative examples from the perspective of syntactic diversity.\n2. How the values of the hyperparameters $\\lambda_{\\rm Shuf}$ and $\\lambda_{\\rm Retr}$ vary in the experiment ? Do they change before and after the warm-starting phase?\n3. What is the meaning of the sentence located on lines 392 through 393?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Author Response:\nWe thank the reviewer for their comments and suggestions. We are glad that the reviewer found our task valuable, our method SCENE innovative and effective, and our writing well organized. We would like to respond to the following comments and questions: \n\n**Necessity of Paraphrase Detector**: During our ablation study in lines 447-488, we ablated the Paraphrase Detector and called it “No Filter”. In this ablation set up, a generated counterfactual is acceptable if and only if it changes the model prediction (we will elaborate this more in our revision). Removing the paraphrase detector (“No Filter” case) decreases accuracy by 1.9 F1 on SQuAD 2.0 and  5.0 F1 on ACE-whQA. In this regard, we feel having a paraphrase detector in the loop is necessary. \n\n**Decrease of Performance on “HasAns” subset**: This is due the fact that a model trained on SQuAD 1.1 will never predict \"unanswerable\" whereas any model trained to identify unanswerable questions will sometimes misclassify an answerable question as unanswerable. This happens for all methods of training a model to predict \"unanswerable\", including training on SQuAD 2.0. Table 2 also supports this: the baseline method “No Augmentation” trained on SQuAD 2.0 has worse performance on answerable examples compared with a model trained on just SQuAD 1.1 and evaluated on SQuAD 2.0. Please also see our explanation for this phenomenon in lines 414-419. \n\n**Using LLM in the loop**:  Admittedly LLMs can fit reasonably well in our filtering process by checking for paraphrases. However, if we try to use LLMs directly to check whether a question is unanswerable, we would need to assume that they are already good at the task – which defeats the purpose of creating a training dataset for detecting unanswerability. Moreover, there can be dataset contamination concerns when LLMs were trained (see Section 8 from the PaLM paper [1] where SQuAD 2.0 is mentioned specifically as contaminated). Nevertheless, we sincerely thank the reviewer for suggesting LLMs and we will consider it for future research.\n \n**[Question 1] Comparison of Counterfactuals Quality between SCENE and Retrieval**: We also compared with a Retrieval-based method, where we match each question to a passage with highest word overlap but not containing the answer for the question within the same dataset, and we label these question-passage pairs as unanswerable questions. In Table 2, we found using our method SCENE is better than Retrieval on SQuAD 2.0. \nWe also manually inspected qualities of SCENE-generated counterfactuals and Retrieval-based ones in Appendix A. We found SCENE-generated ones are harder to distinguish answerable from unanswerable than Retrieval-based ones, even though they may not be linguistically “harder” or syntactically diverse. SCENE questions are harder for unanswerable detection tasks because they are minimally edited, and thus they have high lexical overlap with original answerable questions. \n\n**[Question 2 and 3] Hyperparameter Details**: $\\lambda_\\textrm{Shuf}$ and $\\lambda_\\textrm{Retrieval}$ are either 1 or 0 based on different experimental settings (see related ablation in Table 11 at Appendix B.1). They stay the same before and after the warm-starting phase. \nLines 392-393 explain one of our design choices on choosing $\\lambda$’s, that we set them either 1 or 0, unless for warm-starting purposes, where some $\\lambda$’s are temporarily suppressed to 0. For example, $\\lambda_\\textrm{SCENE}$ is set to 0 in Eq 2 for the first 100 steps, and it resumes to 1 for the rest of the training procedure.\n\n**Reference**: [1] Chowdhery, Aakanksha et al. “PaLM: Scaling Language Modeling with Pathways.” ArXiv abs/2204.02311 (2022).  https://arxiv.org/abs/2204.02311", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well organized and contains full of content and detailed experiments, obtaining significant improvement under the different settings.\n2. SCENE is innovative enough that it removes barriers between datasets with different labels in several types of QA and RTE tasks, making it possible to test the migration of future models between datasets or to integrate multiple datasets into the same form.\n3. The problem studied is of high practical value since unanswerable questions are common in the practice of QA, while SCENE can effectively reduce the labeling cost of related tasks", "reasons_to_reject": "1. There are a lot of detailed parts of the paper, but there are still parts that have not been discussed due to the over-complexity of this pipeline. For example, whether a paraphrase detection model in Filter is necessary and how much removing it affects the results\n\n2. Experimental results show that an increase in performance on the new label brings about a decrease in performance on the original label. And in practice, performance labels on the original label (e.g., the F1 or EM performance of the \"Has Answer\" part) tend to be more important. So I don't think creating harder samples in the data processing part is the best solution, maybe going over an additional fact-checker (e.g., an LLM) after generating the answer can keep the performance of the model on the original labels drop less.", "questions_to_authors": "1. It has been emphasized many times in the paper that SCENE is able to construct \"harder negative examples\", but the description of \"harder\" is very vague, so can the hardness of the samples be described more accurately? Compared to SCENE, I think the retrieval-based generation method yields syntactically more diverse negative examples, so perhaps SCENE does not produce harder negative examples from the perspective of syntactic diversity.\n2. How the values of the hyperparameters $\\lambda_{\\rm Shuf}$ and $\\lambda_{\\rm Retr}$ vary in the experiment ? Do they change before and after the warm-starting phase?\n3. What is the meaning of the sentence located on lines 392 through 393?", "paper_topic_and_main_contributions": "This paper proposes a framework for counterfactual sample generation. The core innovation of this paper is to generate new classes of negative samples compared to the previous methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "psdhB5YSAl", "reviewer": "Reviewer_oQE5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions. We are glad that the reviewer found our contribution solid, experiments extensive and our paper well-written. We would like to respond to the following comments: \n\n**Comparison with Few-Shot Performance of LLM**: As we mentioned in the Limitations section, compared to LLMs, smaller models are more efficient. Moreover, it can be an unfair comparison with LLMs on open-web datasets like SQuAD due to data contamination concerns when LLMs are trained. Data contamination acknowledgements can be found at Section 8 (on Page 36) from the PaLM paper [1] where SQuAD 2.0 is mentioned specifically as contaminated. We will add related discussions in the final version. \n\n**Reference**: [1] Chowdhery, Aakanksha et al. “PaLM: Scaling Language Modeling with Pathways.” ArXiv abs/2204.02311 (2022). https://arxiv.org/abs/2204.02311", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Solid contribution: The proposed method saves on data collection costs and leads to improved performance.\n\nGood writing: The paper is clear and well-written. The authors support their major claims with experimental evidence.\n\nExtensive experiments: The authors provide results across a variety of different QA datasets and see consistent trends showing the effectiveness of their approach.", "reasons_to_reject": "While not required, it would have been nice to see few-shot performance from an LLM (such as bloom-3b) on these datasets. The authors mentioned in the Limitations section that they specifically wanted to focus on smaller models, but it would be good to know the performance from an LLM since this task seems relatively straightforward for an LLM with a lot of general knowledge.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel method for data augmentation (or extrapolation) for negative examples in QA. The authors use self-training and rejection sampling via a paraphrase detector to create negative examples that are useful for training a QA model to accurately detect unanswerable questions without any gold negative examples.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel pipeline to automatically create unanswerable questions given QA examples. The pipeline includes four key components: question perturbation, paraphrase detection, answerability prediction, and filter of unanswerable questions. All reviewers appreciated the well-structured content, innovative methods, and comprehensive experiments.
Decision: Accept-Main