A Lightweight Method to Generate Unanswerable Questions in English
Authors: Vagrant Gautam
Miaoran Zhang
Dietrich Klakow
Venue: EMNLP 2023 Findings (2023)
Keywords: extractive question answering
machine reading comprehension
data augmentation
TL;DR: Antonym and entity swaps are a simple but strong baseline for unanswerable question generation.

Abstract: If a question cannot be answered with the available information, robust systems for question answering (QA) should know *not* to answer.
One way to build QA models that do this is with additional training data comprised of unanswerable questions, created either by employing annotators or through automated methods for unanswerable question generation.
To show that the model complexity of existing automated approaches is not justified, we examine a simpler data augmentation method for unanswerable question generation in English: performing antonym and entity swaps on answerable questions.
Compared to the prior state-of-the-art, data generated with our training-free and lightweight strategy results in better models (+1.6 F1 points on SQuAD 2.0 data with BERT-large), and has higher human-judged relatedness and readability.
We quantify the raw benefits of our approach compared to no augmentation across multiple encoder models, using different amounts of generated data, and also on TydiQA-MinSpan data (+9.3 F1 points with BERT-large).
Our results establish swaps as a simple but strong baseline for future work.

A Lightweight Method to Generate Unanswerable Questions in English
Vagrant Gautam
Miaoran Zhang
Dietrich Klakow
Saarland Informatics Campus, Saarland University vgautam@lsv. uni ssaisaaaddd... de
Abstract
Context: The only indigenous mammals of Bermuda are five species of bats, all of which are also foand eastern United Stales....
If a question cannot be answered with the avail- able information, robust systems for question answering (QA) should know nor to answer. One way to build QA models that do this is with additional training data comprised of uman- swerable questions, created either by employ- ing annotators or through automated methods for unanswerable question generation. To show that the model complexity of existing auto- mated approaches is not justified, we examine a simpler data augmentation method for unan- swerable question generation in English: per- forming antonym and entity swaps on answer- able questions. Compared to the prior state-of- the-art, data generated with our training- freee and lightweight strategy results in better mod- els (+1.6 Fl points on SQuAD 2.0 data with BERT-large), and has higher human-judged re- latedness and readability. We quantify the raw benefits of our approach compared to no aug- mentation across multiple encoder models, us- ing different amounts of generated data, and also on TydiQA-MinSpan data (+9.3 Fl points with BaRge- Our results establish swaps simple but strong baseline for future work.
flood in the
Answerable seed question: What are the only native mammuls found in Bermuda?
Unanswerable question generation methods
Human annotators (Rajpurkar et al., 2018):
What is one of five indigenous mammals of Bermuda?
UNANSQ 1M parameters (Zhu al., 2019):
what the only native manmals found bermuda ?
related and readable, but not unanswerable!
CRQDA 593M parameters (Liu al., 2020):
What are the only native manmals found Whht????
related and unanswerable, but not readable!
Antonym swapping 0 parameters:
What are the only foreign mammals found in Bermuda?
* Entity swapping -0 parameters:
What are the only native mammals found in United States?
Figure 1: A context paragraph and an answerable seed question, which can be used to generate unanswerable questions. Examples are shown from human annotators as well as 4 automatic methods with their estimated number training parameters. indicates our methods.
dataset (Clark et al., 2020), both of which contain human-written answerable and unanswerable ex- amples of extractive question answering. As human annotation is slow and costly, various models have been proposed to automate unanswerable question generation using answerable seed questions; most recently, Zhu et al. (2019) proposed training on a pseudo-parallel corpus of answerable and unan- swerable questions, and Liu et al.'s (2020) state-of- the-art model used constrained paraphrasing. Although model-gemerated unanswerable ques- tions give sizeable improvements the SQuAD 2.0 development set, Figure shows that many dif- fer from their answerable counterparts only super- ficially. An estimated 40% of human-written unan- swerable questions also involve minor changes to answerable ones, e.g., swapping words to antonyms or swapping entities (Rajpurkar et al., 2018).
1 Introduction
Question answering datasets in NLP tend to focus on answerable questions (Joshi et al.. 2017; Fisch al., 2019), but unanswerable questions matter too because: (1) real-world queries are unanswer- able surprisingly often e.g., 37% of fact-seeking user questions to Google are unanswerable based on the Wikipedia page in the top 5 search results (K Kwiatkowski et al., 2019); and (2) identifying unanswerable questions is an essential feature of reading comprehension - but conventional extrac- tive QA systems typically guess at plausible an- swers even in these cases (Rajpurkar et al., 2018). To aid in building robust QA systems, more datasets have begun to include unanswerable ques- tions, c.g., the SQuAD 2.0 dataset in English (Ra- jpurkar et al., 2018) and the multilingual TydiQA

---

Motivated by these observations, we present lightweight method for unanswerable question gen- eration: performing antonym and entity swaps on answerable questions. We evaluate it with:
learning (Ji al., 2022), or tackle unanswer- ability and answer extraction separately, by using verifier modules or calibrators (Tan et al., 2018; Hu et al., 2019; Kamath et al., 2020). We use a conventional fine-tuning paradigm and leave to future work to boost performance further by using our high-quality data in other paradigms.
1. metries: development set performance (EM and F1), as well as human-judged unanswer- ability, relatedness, and readability;
2. datasets: SQuAD 2.0 (Rajpurkar et al.,
2018) and TydiQA (Clark et al.. 2020);
3 Our augmentation methods
3. 2 baselines: UNANSQ (Zhu et al.. 2019) and
Inspired by the nowdworker-writeer unanswerable questions in Rajpurkar et al. (2018), we generate types of unanswerable questions by modifying answerable ones with antonym and entity swaps. Our generated data is then filtered based empiri- cal results presented in Appendix B.I. We examine results for each augmentation method separately, but we also experimented with combining them in Appendix B.2. Examples of output from our methods are shown in Figure 1 and Appendix D.
CRQDA (Liu et al., 2020); and
4. 6 encoder models: base and large variants of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and ALBERT (Lan al., 2020).
Swapping significantly outperforms larger and more complex unanswerable question generation models on all metrics. Across models and datasets, our method vastly improves performance over augmentation baseline. These results show that our method has potential for practical applicability and that it is a hard-to-beat baseline for future work.'
We use spaCy (Honnibal et al.. 2020) for part- of-speech tagging and dependency parsing, and AllenNLP's (Gardner et al., 2018) implementation of Peters et al. 's (2017) model for named entity recognition. Using NLTK (Bird et al., 2009), we access WordNet (Fellbaum, 1998) for antonyms and lemmatization.
2 Related work
Unanswerability is not new to QA research, with a rich body of work typically proposing data augmen- tation methods or training paradigm innovations. Papers focusing on data augmentation either gen- erate data for adversarial evaluation (Jia and Liang, 2017; Wang and Bansal, 2018) or for training. Most work on training data generation for QA is limited to generating answerable questions, e.g., Alberti et al. (2019) and Bartolo et al. (2020, 2021), but some generate both answerable and unanswerable questions (Liu et al., 2020) or, like us, just unan- swerable questions (Clark and Gardner, 2018; Zhu al., 2019). Unanswerable questions have been shown be particularly hard for contemporary QA models when they contain false presuppositions (Kim et al., 2023), when they are fluent and related (Zhu et al., 2019), when the context contains a can- didate answer of the expected type (e.g., a date for "When" question; Weissenborn et al., 2017: Sulem et al., 2021), and in datasets beyond SQuAD (Sulem et al., 2021). Our method challenging for models because it generates questions that are fluent, related and unanswerable.
3.1 Antonym augmentation
We antonym-auggent an answerable question by replacing one noun, adjective or verb at a time with its antonym. We replace word when it exactly matches its lemma," with no sense disambiguation. When multiple aitonyy--ugmented versions are generated, we pick the one with the lowest GPT-2 perplexity (Radford al., 2019). Thus, when we augment the question "When did Beyonce start be- coming popular?" we choose "When did Beyonce start becoming unpopular? ?" instead of the clunky "When did Beyonce end becoming popular?".
To avoid creating answerable antonym- augmented questions, we do not augment adjectives a dependency relation with a question word (c.g.. "How big are ostrich eggs?"), and we also skip polar questions (e.g.. "Is botany a narrow science?") and alternative questions (e.g., "Does communication with peers increase or decrease during adolescence?") both of which tend to begin with an AUX part-of-speech tag.
Different training paradigms have been proposed to more effectively use training data, e.g., adver- sarial training (Yang et al., 2019) and contrastive
'Our data and onde are available at https://github. questiinngeeeeattion generation. com/uds-les www.udd-sss/nnnsseraaaaaa lewlll--eiiionn--
FFiglish's 's lack of rich morphology lets us avoid inflection models with little impact on how much data we can generate.

---

3.2 Entity augmentation
Data
Answerable
Unanswerable
We entity-augment an answerable question by re- placing one entity at time with a random entity from the context document that has the same type and does not appear in the question: "How old was Beyoncé when she met LaTavvi Roberson? ?" can be augmented "How old was Beyoncé when she met Kelly Rowland?" but can never be augmented to "How old was Beyoncé when she met Beyoncé?" ?" When we generate multiple sions of a question, we randomly select one. Intuitively, picking an entity of the same type keeps readability high as person entities appear in different contexts (married, died) than, e.g., geopo- litical entities (filmed in, state of). Using entities from the same context ensures high relevance, and leaving everything else unmodified maintains the entity type of the expected answer.
SQuAD 2.0
Training data
86,821
43,498
+ UNANSQ
+
69,090
+ CRQDA
+0
+ 124,085
+ Antonym (ours)
0
34,180
+ Entity (ours)
0
47,624
Development
5,928
5,945
TydiQA-Minnpan (English)
Training data
3,696
4,953
ver-
+ Antonym (ours)
+0
+ 880
+ Entity (ours)
0
+ 2,808
Development data
495
477
Table I: Number of answerable and unanswerable questions with the SQuAD 2.0 and -MinSpann datasets and the available augmentation methods.
5 Comparison with previous SQuAD 2.0 augmentation methods
4 Experimental setup
We compare methods on their BERT lurge perfor- mance using 2 strong baselines for unanswerable question generation: UNANSQ (Zhu et al., 2019) and the oftte-of-the-arr method CRQDA (Liu al., 2020). We use publicly-relcased unanswerable questions for both methods, which only exist for SQuAD 2.0. theory, CRQDA can generate both answerable and unanswerable questions but we only use the latter for an even comparison and be- cause only these are made available.
Task. We evaluate on the downstream task of extractive question answering, i.c., we judge unanswerable question generation method to be better if training with its data improves an extractive QA system's performance compared to other methods. Given a question and some context (a sentence, paragraph or article), the task is to predict the correct answer text span in the passage, or no span if the question is unanswer- able. Performance is measured with exact match (EM) and F1, computed on the answer span strings.
5.1 Main result
Datasets. We use SQuAD 2.0 (Rajpurkar et al., 2018) and the English portion of the TydiQA dataset (Clark et al., 2020) that corresponds to minimal-span extractive question answering. SQuAD 2.0 uses paragraphs as context whereas TydiQA uses full articles. To keep the TydiQA setting similar SQuAD 2.0, we modify the task slightly, discarding yes/no questions and questions for which there is paragraph answer but not a minimal span answer. For both datasets, we train on original or augmented versions of the training set and report performance on the development set. All data statistics are shown in Table 1.
Training Data
EM (f)
F1 (1)
Baseline aug.)
81.220.1 78.0
+ UNANSQ
77.8
81.0
CRQDA
79.1
82.0+0.1
+ Antonym (ours)
79.3+0.2
82.4
Entity (ours)
80.7
83.640.0
Table 2: SQuAD 2.0 development set results (EM/FI) with different data augmentation methods, averaged over random seeds when fine-tuning BERT BaRTaage Coloured cells indicate significant improvements over CRQDA according to Welch's t-test (a = 0.05).
As the results in Table 2 show, our proposed data augmentation methods perform better than other more compute-intensio unanswerable question generation methods. Entity augmentation is more effective than antonym augmentation; anecdotally, some samples of the latter are semantically inco- herent, which models might more easily identify.
Models. We experiment with base and large vari- ants of BERT (Devlin al., 2019), RoBERTa (Liu et al., 2019), and ALBERT (Lan et al., 2020), all trained with HuggingFace Transformers (Wolf et al., 2020); see Appendix A for further details.

---

(a) All questions
(b) Answerable questions
(c) Unanswerable questions
8
entity (ours)
Relative change (F1)
antonym (ours)
croda
unansq
100
500
10K
100
10K
100
500
1K
Number of augmented unanswerable questions
Figure 2: Relative change in BERT large Fl
answerable and unanswerable questions in the SQuAD 2.0 development set, varying the amount of data generated with UNANSQ, CRQDA, antonym and entity augmentation.
Method
Unanswerability (t)
Relatedness (1)
Readability (f)
(Range)
(0.0 1.0)
(0.0 1.0)
(1.0
3.0)
UNANSQ (Zhu et al., 2019)
0.56
0.98
2.46
CRQDA (Liu et al., 2020)
0.91
0.61
1.40
Antonym + entity (ours)
0.78
0.97
2.69
Crowdworkers (Rajpurkar et al., 2018)
0.83
0.95
2.91
Table 3: Results of human evaluation of unanswerable question generation methods with 3 annotators. Inter- annotator agreement (Krippendorff's a.) is 0.67 for unanswerability, 0.60 for relatedness, and 0.74 for readability.
We find our results to be particularly compelling given that method is training-free free and imple- mented in less than 150 lines of Python code, com- pared to, for example, CRQDA, which uses a QA model and transformer-base autoencoder with a total of 593M training parameters (Liu et al., 2020).
able, which might lead the model to abstain more answerable questions at test time.
While our results show the effectiveness of aug- mented unanswerable questions at improving uman- swerability, they also highlight the need to ensure that this boost does not come at the cost of question answering. Using less augmented data might help with this; 5K mttit--agmmente samples vastly im- prove unanswerable question performance at little cost to answerable ones.
5.2 Data-balanced ablation study
As the 4 methods under comparison each generate different number of unanswerable questions, perform data-balanced ablation study by training models with 100, 500, 1K, 5K and 10K randomly- selected samples from all methods.
5.3 Human evaluation
We perform additional human evaluation of the unanswerable question generation methods using the following criteria, based on Zhu al. (2019):
As Figure 2(a) shows, our simpler structural methods perform comparably with or better than more complex methods, even with less data.
1. Unanswerability (0.0-1.0): 0 if the generated question is answerable based on the context, if it is unanswerable;
When split by answerability, all methods show some degradation on answerable questions in the development set, as shown in Figure 2(b); like Ji et (2022), we find that focusing on unanswerable question generation leads to tradeoff between performance on unanswerable and answerable questions. We hypothesize that this tradeoff occurs as result overfitting to unanswerable questions as well as data noise, i.c., generated questions that are labelled unanswerable but are actually answer-
2. Relatedness (0.0 1.0): 0 if the question is unrelated to the context,
it is related,
Readability (1.0 3.0): 1 if the question is incomprchensible, for minor errors that do not obscure meaning, 3 for fluent questions.
"Zhu et al. (2019) evaluate relatedness on 1-3 scale. comparing each question to a context paragraph and un input question. We use a binary scale as we do not have paired input questions for CRQDA and human-written questions.

---

Baseline (no aug.)
Antonym
Entity
Dataset
Model
EM (T)
F1 (1)
(1)
F1 (T)
EM
FI (T)
SQuAD 2.0
BERT hase
72.7
76.020.3
73.9
77.0g0.9
76.0
79.020.5
BERT lurge
78.0
81.220.4
79.3
82.420.3
80.7
83.6400
RoBERTapuse
18.7
81.8
79.2
82.2
79.7
82.6.0.1
RoBERTa Taarge
85.8
88.8 +0.8
85.9
88.9
85.7
88.6+0.1
ALBERT base
79.3+0.1
82,4
79,3+0.2
82.3+0.1
80.0+0.2
82.9+0.1
ALBERT large
82.1
85.2
82.2+0.2
85.2
82.3
85.1
TydiQA-MinSSan (English)
BERT base
48.5
51.620.7
58.7
61.4 61.4
58.921.2
61.2
BaRTe
51.420.8
54.420.7
61.2
63.7
60.7
62.6.1.7
Table 4: Our methods give statistically significant improvements (coloured cells) across multiple encoder models on both SQuAD 2.0 and TydiQA--inSSpn data, compared no augmentation. EM and Fl results averaged over 3 random seeds and significance is measured using a Welch's 's t-test with 0.05.
100 context paragraphs are sampled from SQuAD 2.0 along with questions per paragraph crowdworker-written question from the original dataset, and question from each of the follow- ing automated methods: UNANSQ, CRQDA, and our method (a combination of antonym- and entity- augmented questions). This gives total of 400 questions, evaluated by 3 annotators. Complete annotator instructions provided in Appendix C.
already have strong baseline performance. Using BERT models, the results on TydiQA show very large improvements over the baselines, with and EM improving by 8-10 points on average.
Conclusion and future work
Our lightweight augmentation method outperforms the previous sfate-ofthh-ee art method for English unanswerable question generation on 4 metrics: de- velopment set performance, unanswerability, read- ability and relatedness. We see significant im- provements in SQuAD 2.0 and TydiQA-MinSpan performance (over no-augmentation baseline) across multiple encoder models and using differrnnt amounts of generated data. Overall, find that when it comes to unanswerable question generation, simpler is better. We thus hope that future work justifies its complexity against our strong baseline.
The evaluation results (Table 3) show our method to an all-rounder with high relatedness, near-human unanswerability, and the highest readability of any automatic method. UNANSQ- generated questions are related and readable but a whopping 44% of them are answerable, while CRQDA only shines at unanswerability by gener- ating unrelated gibberish instead of well-formed questions (52% less readable and 36% less related than wrowiworker-written questions, compared to ours 5% less readable but 2% more related). De- spite their higher unanswerability, the CRQDA questions are not as beneficial to training, sug- gesting a compositional effect: unanswerability, relatedness and readability all play a role together and it is important for generation methods to do reasonably well at all of them.
A Although we have shown that entity-based aug- mentation creates data that is useful for models to learn from, it is still unclear why. Several of our ex- amples seem to contain false presuppositions, e.g.. "When did Christopher Columbus begin splitting up the large Bronx high schools?" Kim et al. (2023) term these "questionable assumptions," and find them be challenging for models like MACAW, GPT-3 and Flan-T5, While Sugawara et al. (2022) studies what makes answerable multiple-choice questions hard for models, we still do not know what makes unanswerable questions hard, and how this relates to domain, model type and size.
6 Beyond SQuAD 2.0 and BERT-large
To more robustly evaluate our augmentation meth- ods, we experiment with more models of multiple sizes (ALBERT and RoBERTa) well with an additional dataset (TydiQA-MinSpan)
Beyond unanswerable question generation and even question answering. we hope our work encour- ages NLP researchers consider whether simpler approaches could perform competitively on a task before using sledgehammers to crack nuts.
Table 4 shows that our method benefits SQUAD 2.0 performance across model types, but we note that on RoBERTa and ALBERT, our approach mainly benefits small models, as larger models

---

Limitations
Limited diversity. As we rely on swaps, our generated augmented data is syntactically very close to the original data. We do not evaluate the diversity of our generated questions compared to human-written unanswerable questions, but similar to Rajpurkar et al. (2018), we find a qual- itative gap here, and leave an exploration of this as well as its impact performance to future work.
Heuristic unanswerability. By generating unan- swerable questions with surface-level heuristic swaps instead of deep semantic information, we sometimes end up with answerable questions. Four real examples of our method's failure modes are:
Conjunctions: Given the context 'Edvard Grieg, Nikolai Rimsky-Korsakon and Antonín Dvořák echoed traditional music of their homelands in their compositions' and the seed question 'Edvard Grieg and Antonin Dvorak used what kind of music in their composi- tions?', the entity-augmented 'Edvard Grieg and Nikolai Rimsky used what kind of music in their compositions?" is answerable.
Depending on existing tools. Our methods are limited by the off-the-shelf tools we rely on. We found that POS tagging and dependency parsing were notably worse for questions compared to state- ments, reflecting the under-repressentatin of ques- tions in treebanks and part-of-speech corpora.
To ensure that entities are swapped to com- pletely different entities, we experimented with both coreference analysis tools and substring matching (i.c., assuming that "Beyoncé Giselle Knowles" and "Beyoncé" refer to the same entity). Our substring matching heuristic both faster and more accurate, but unfortunately both approaches struggle with diacritics and cannot identify that "Beyoncé" and "Beyonce" refer to the same person.
Commutative relations: As marriage is commutative, diktoyy--uugmettn the seed question 'Chopin's father married who?' with the context 'Fryderyk's father, Nico- las Chopin, [..]] in 1806 married Justyna Krayzanowska" results in the still-answerable 'Chopin's mother married who?'
Information is elsewhere in the context: With the context 'Twilight Princess was launched North America in November 2006, and in Japan, Europe, and Australia the fol- lowing month' and the seed question, 'When was Twilight Princess launched in North America?', entity augmentation generates the technically answerable 'When was Twilight Princess launched in Japan?' Note that this is not answerable using extractive QA systems.
Other datasets and languages. SQuAD and Ty- diQA are based on Wikipedia data about peo- ple, places and organizations. This lets entity- based augmentation shine, but our methods may work less well on other domains, e.g., ACE-whQA (Sulem et al., 2021), and our conceptualization of unanswerability is specific to extractive QA. Like many methods designed for English, ours relies on simple swaps that fail morpholog- ically more complex languages, c.f., Zmigrod et al. (2019). In German, for instance, we might need to re-inflect some antonyms for case, num- ber and grammatical gender. Even entity swaps may be less straightforward, sometimes requiring different prepositions, e.g., the English sentences "She drives to (Singapore, Switzerland, Central ParkJ' would be "Sie führt [nach Singapur, in die Schweiz, zum Central Park]" in German.
Other forms of polar questions: We do not filter out some less common forms of polar questions, c.g., 'What beverage is consumed by more people in Kathmandu, coffee or tea?' Here, the beverage is consumed by less people in Kath- mandu, coffee or tea?' is still answerable.
version, 'What
Based on our human evaluation (Table 3), we estimate the level of noise of our method at around 20%. Although we cannot provide guarantees on the unanswerability of our generated questions, our goal was show that a lightweight method can outperform more complex methods that also do not provide such guarantees. Thus, we find our ncar-human level of noise acceptable for the task.
Furthermore, our approach for excluding ques- tions for antonym augmentation is syntax-specific in its use of part-of-speech and dependency infor- mation. Though this approach would transfer to a syntactically similar language like German, it would not work on Hindi, where polar questions are indicated by the presence of the particle kya: in almost any position (Bhatt and Dayal, 2020).

---

Ethics statement
synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 9830-884,, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Teaching models to abstain from answering unan- swerable questions improves the robustness and reliability of QA systems. Working unanswer- ability is thus way of directly addressing the pos- sible harms of QA systems giving incorrect results when being used as intended.
Rajesh Bhatt and Veneeta Dayal. 2020. Polar question particles: Hindi-urdu kya:. Natural Language & Linguistic Theory, 38(4):1115-1144
Additionally, our paper presents a more sustain- able approach for unanswerable question genera- tion, heeding Strubell et al.'s (2019) call to use computationally efficient hardware and algorithms. We chose not to employ Amazon Mechanical Turk workers due to its history of exploitative labour practices (Williamson, 2016; Kummerfeld, 2021), and instead employed annotators who are contracted with the authors' institution and paid a fair wage. Our data and annotation tasks posed negligible risk of harm to the annotators.
Steven Bird, Ewan Klein, and Edward LowardLoperr 2009, Nat- ural Language Processing with Python, 1st edition. O' O'Reilly Media, Inc.
Christopher Clark and Matt Gardner. 2018. Simple and effective multi-pane paratiippaggaphhhh reading comprehension. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 845-855, Melbourne, Australia. Association for Computational Linguistics.
Jonathan H. Clark, Eunsol Choi, Michael Collins. Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, 2020. TyDi QA: A benchmark for iiformation-seekinng question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Acknowledgements
The authors are grateful for Fileen Bingert and AriaRay Brown's diligent annotation, to Marius Mosbach for proofreading and mentorship. and to Alexander Koller and our anonymous conference and workshop reviewers for their suggestions to improve this work. We dedicate this paper to China Restaurant Saarbrücken, where this collaboration began with a serendipitous fortune cookie that said: New visions create power and self-confidence. The authors received funding from the BMBF's (German Federal Ministry of Education and Re- search) SLIK project under the grant 01122215CC, and from the DFG (German Research Foundation) under project 232722074, SFB 1102.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Dutabase. The MIT Press.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo. Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China, Association for Computa- tional Linguistics.
References
Malt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6. Melbourne, Australia. Association for Computational Linguistics.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1168-617, Flo- rence, Italy. Association for Computational Linguis- tics.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas- tian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for read- ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Minghao Hu, Furu Wei. Yuxing Peng, Zhen Huang. Nan Yang, and Dongsheng Li. 2019. Read + verify: Machine reading comprehension with unanswerable questions. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01 (:652)6653777
Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel. Pontus Stenetorp, and Douwe Kiela, 2021. Improving question answering model robustness with

---

Yunjie Ji. Liangyu Chen. Chenxiao Dou, Baochang Ma. and Xiangang Li. 2022. To answer or not to an- swer? improving machine reading comprehension model with span-based contrastive learning. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1292-1300, Seattle, United States. Association for Computational Linguistics.
2020. Tell me how to ask again: Question data aug- mentation with controllable rewriting in continuous space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5798-5810, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1167.1
Robin Jia and Perey Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Matthew E. Peters, Waleed Ammar, Chandra Bhaga- vatula, and Russell Power. 2017. Semi-supervin sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
Mandar Joshi, Funsol Choi, Daniel Weld, and Luke Zeulemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguisties.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Amita Kamath, Robin Jia, and Percy Liang. 2020. Se- lective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5684- 5696. Online. Association for Computational Lin- guistics.
Pranav Rajpurkar, Robin Jía, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Mclbourne, Australia, Association for Computational Linguistics.
Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. 2023. (QA)²: Question answering with questionable assumptions. In Proceedings of the 61 Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers). pages Toronto, Canada. Association for Com- putational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NIP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Jonathan K. Kummerfeld. 2021. Quantifying and avoid- ing unfair qualification labour in crowdsoureing. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 343-349, Online. Association for Computational Linguistics.
Saku Sugawara. Nikita Nangia, Alex Warstadt, and Samuel Bowman. 2022. What makes reading com- prehension questions difficult? In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6951-6971, Dublin, Ireland. Association for Compu- tational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 7:452-466.
Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do we know what we don't 't know? studying unanswerable questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguisties: EMNLP P 2021, pages 4543-4548, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Kelcey, Ming- Weig-Wei
Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang. Weifeng Lv, and Ming Zhou. 2018. know there is no answer: Modeling answer validation for ma- chine reading comprehension. In Natural Language Processing and Chinese Computing, pages 85-97, Cham. Springer International Publishing.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Confer- ence on Learning Representations.
Yicheng Wang and Mohit Bansal. 2018. Robust ma- chine comprehension models via adversarial training. In Proceedings of the 2018 Conference of the North
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan. Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou.

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pages 575-581, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Hyperparameter
Value
5c-5 (BERT)
Learning Rate
1.5c-5 (RoBERTa)
3e-5 (ALBERT)
Batch Size
48
Dirk Weissenbom, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the s Conference on Computational Natural Language Learning (CoNLL 2017), pages 271 271- Vancouver, Canada. Associa- tion for Computational Linguistics.
Epochs
2
Max Seq Length
384
Doc Stride
128
Table 5: Model fine-tuning hyperparametess
Vanessa Williamson. 2016. On the ethics of crowd- sourced research. PS: Political Science & Politics, 49(1):77-81.
seeds (42, 31, and 53). Training was conducted on single NVIDIA A100 GPU.
We download augmented datasets from GitHub for UNANSQ 4 and CRQDA" and fine-tune mod- els from scratch with the hyperparameter settings above for a fair comparison. To control for the effect of different codebases and hyperparame- ters, we compare the experimental results from the CRQDA codehase 6 with those of our own in Table 6, showing that our improvements consistent.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- iez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Training Data
EM (t)
F1 (1)
CRQDA codebase
Ziqing Yang, Yiming Cui, Wanxiang Che, Ting Liu, Shijin Wang. and Guoping Hu. 2019. Improving machine reading comprehension via adversarial train- ing.
Baseline (no aug.)
78.2
81.4
CRQDA
78.7
81.5
+ EEtityy (ours)
80.0
82.9
Our codebase
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang. Bing Qin, and Ting Liu. 2019. Learning ask unanswer- able questions for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics. pages 4238- 4248, Florence, Italy, Association for Computational Linguistics.
Basefine (no aug.)
78.3
81.220.4
+ CRQDA
19.
82.0g0.4
+ Entity (ours)
80.7
83.6.0.0
Table 6: Comparing codebases on their SQuAD 2.0 de- velopment set performance when fine-tuning BERT on unaugmented and augmented training data.
Ran Zmigrod, Sabrina Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmenta- tion for mitigating gender stereotypes in languages with rich morphology.
B Filtering and combining augmentation methods
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 1651-1661, Florence, Italy, Asso- ciation for Computational Linguisties.
This appendix presents our ablation experiments with filtering generated data for each augmentation strategy and combining both strategies together.
A Implementation details
For our experiments, we initialize the following models with checkpoints from the Huggingface Transformers Library (Wolf et al., 2020): bert-base- uncased, bert-large-cased. roberta-base roberta- large, alberr- and alreev--arge-v2. We use the SQuAD 2.0 hyperparameters that were sug- gested in the papers for each model (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) and report them in Table 5. We train models with random
B.1 Filtering augmented data
Table 7 shows our experiments with different fil- tering strategies when we generate multiple aug- mented versions of single answerable question.
http://///tttttoooooooo/ggggggga htttps://gi http./occm/aacccho///////nn///// http://ghhh...cmmmaaaeeea///a///
fine tete_ttue_beettwiiit wtwpbbertttttthchphppbee

---

Training Data
(1)
(1)
built-in data validation and we will perform fur- ther tests check for consistent annotation. Task- specific information is provided below.
Baseline (no aug.)
78.0
21.2
Antonym (no filtering)
29.1
g2.
Antonym (random)
79.0g0.2
82.
C.1 Unanswerability
Antonym (ppl)
79.3
82.4g0.3
For each of the questions pertaining para- graph, please annotate unanswerability based on the paragraph on a 0-1 scale as follows:
Entity filtering)
80.1
83.1
Entity (random)
80.7
83.6.0.0
Table 7: Comparing different filtering strategies on their SQuAD 2.0 development set performance (EM/FI)
O: answerable based on the paragraph
when fine-tuning BERT Results averaged over random seeds. Given multiple augmented candidates generated from question, "random" means domly sample one candidate, and "ppl" means we select the candidate with the lowest GPT-2 perplexity.
unanswerable based on the paragraph
Please note that you need to rely exclusively on the paragraph for this annotation, i.e., we are not interested in whether a question is answerable unanswerable in general, but specifically whether the paragraph contains the answer to the question.
For antonym augmentation, we try random sam- pling and perplexity- sampling in addition to using all of the generated data. All strategies improve over the baseline, but random sampling is marginally better than no filtering, and perplexity- based sampling is the best strategy.
Ignore grammatical errors, changes connota- tion, and awkward wording within questions if they do not obscure meaning.
Please pay attention to affixation (e.g.. negation)
that changes the meaning of term.
When negation appears in question, you should use the logical definition of negation, ie., anything in the universe that isn't X counts answering "What isn't X?" However, for this task, the universe restricted to specific answers from the paragraph. As an example:
For entity augmentation, only compare two strategies: random sampling and no filtering. We do not try perplexity-basee sampling as entity changes seem to impact perplexity non-intuitive ways. Again, both strategies improve over the base- line but random sampling is better than no filtering,
Paragraph: Cimnamon and Cumin are going out for lunch. Cinnamon will drive them there.
B.2 Combining augmentation strategies
Table 8 shows the results of combining antonym and entity augmentation. We see statistically signif- icant improvements on 4 out of 6 models. Although these are good results when seen in isolation, we found that they did not show much of an improve- ment over just using entity augmentation. This suggests that it is worth exploring ways to more effectively combine the two strategies.
Question: Who isn't Cinnamon?
=> (answerable) with "Cumin," who can be inferred be another person mentioned in the paragraph who isn't Cinnamon
Question: Where isn inn't lunch?
=> (unanswerable)) because there
didate answers in the paragraph that it would make sense to answer this question with
Annotation instructions
Some more examples:
Together with this annotation protocol, you have received a link to a spreadsheet. The sheet con- tains 2 data columns and 3 task columns. The data columns consist of paragraphs and questions. In the paragraph column, each paragraph is prefaced with its topic. There are 100 paragraphs about ious topics and 4 questions per paragraph, for total of 400 data points. You are asked annotate the questions for the tasks of unanswerability, re- latedness, and readability. Please be precise and consistent in your assignments. The columns have
Paragraph: Cinnamon and Cumin are going out for lunch. Cinnamon will drive them there.
Question: Can Cinnamon drive?
=> (answerable)
Question:
Can Cumin drive?
=> (unanswerable)
Question: cinammon can drive?
=> (answerable), despite the odd syntax and the typo

---

Baseline (no aug.)
Combined
Model
EM (1)
F1 (T)
EM (T)
F1 (T)
BERT
22.7
76.020.3
76.1
78.9
BERT large
78.0
81.220.4
80.6 80.6g0.3
83.5
RoBERTagase
28.7
81.8
49.9
82.8
RoBERTan Eitarge
85.8
88.8
85.7
88.5
ALBERT buse
79.3+0.1
82.4+a.1
79.7+0.1
82.6+0.0
ALBERT large
82.1
85.2
82.2
85 85.0
Table 8: Combining our augmentation methods on SQuAD 2.0 shows significant improvements (coloured cells)
across models according Welch's 1-test (a: 0.05). Results (EM/F1) are averaged over random seeds.
Question:
lunch drive what?
Question: Can Cumin drive?
=> I (unanswerable), because the errors result
=> (related)
in an incomprehensible sentence
Question:
cinammoo can drive?
C.2 Relatedness
=> (related), despite odd syntax and typo
For each of the 4 questions pertaining to para- graph, you need to annotate relatedness to the paragraph 0-1 scale as follows:
Question: lunch drive what?
=> I (related), because "lunch" and "drive"
both appear in the paragraph despite the in- comprehensibility of the question
0: unrelated to the paragraph
What sunscreen do bees use? => 0 (unrelated), because the paragraph has nothing to do with sunscreen or bees
Question:
1: related to the paragraph
For a question to be related to a paragraph, all parts of should be related to what the paragraph discusses. If any parts of the question are unrelated to the contents of the paragraph, please annotate it unrelated.
Question: When was Cumin born?
=> 1 (related), because birth is related to a person's 's existence.
Some more examples of edge cases:
If words question are in the paragraph even if they're 're combined in different ways that poten- tially don't 't make sense, this still counts as related. For instance, mixtures of names created using com- ponents of names in the paragraph count as related, but an entirely new made-up up name would be unre- lated.
What car does Cinnamon use? => (related), because Cinnamon is men- tioned the paragraph and cars are related to driving
Question:
Question: What food will Cinnamon and Cumin eat?
Numbers and dates can be different from the ones mentioned in the question this still counts as related.
=> (related), because Cinnamon and Cumin are mentioned in the paragraph and food is related to their lunch plans
Events that are related to the lives of people or history of companies (e.g., births, deaths, etc.) should be marked as related.
Question: What sunscreen do Cinnamon and Cumin use?
(unrelated), since sunscreen is unrelated driving and cating
Ignore grammatical errors, changes in connota- tion, and awkward wording within questions if they do not obscure meaning. Some examples:
Question: Do bees drive to lunch? => (unrelated), since the paragraph does not discuss bees
Paragraph: Cinnamon and Cumin are going out for lunch. Cinnamon will drive them there.
Question:
Do you want to go out to lunch?
Question: Can Cinnamon drive?
=> (unrelated), because the paragraph is not about you
=> (related)

---

C.3 Readability
For each question, you need to annotate readability and fluency on 1-3 scale as follows:
Context: In 1952. following a referendum, Baden, Württemberg Baden, and Württemberg Hoherzollem merged into Kaden-Württenin Protectorate rejoined the Federal Republic the Saar- land. German reunification in 1990, in which the Ger- man Democratic Republic (East Germany) ascended into the Federal Republic, resulted in the addition of the re- established eastern states of Brandenburg, Mecklenburg- West Pomerania (in German Mecklenburg Mieckenburg-Vorpommeerr Saxony (Sacksen), Saxony-Anhull (Sachsen-Anhalt) and Thuringia (Thäringen),
I: incomprchensible
In 1957. the Suar
2: minor errors that do not obscure the mean- ing of the question (such as typos, agreement enrors, missing words or extra words)
3: fluent questions
well the reunification of West and East Berlin into Berlin and its establishment full and equal state, regional referendum in 1996 merge Berlin with surrounding Brandenburg gerlin-Brandebburg failed reach the necessary majority Brandenburg, while majority of Berlin- ers voted in favour of the merger.
Please focus on how syntactically well-formed a question is without worrying about the meaning making sense. For example, "Do clouds watch television?" is syntactically fluent question even if it does not make sense semantically.
Please ignore extra spaces and capitalization er- rors when they do not change the meaning of the question. Some examples:
Answerable seed question:
Why did regional referendum in 1996 to merge Berlin with surrounding Brandenburg fail?
Entity swapping:
Why did regional referendum in 1996 to merge Berlin with surrounding West Pomerania fail?
Paragraph: Cinnamon and Cumin are going out for lunch. Cinnamon will drive them there.
Answerable seed question:
1957, the Saar Protectorate rejoined the Federal public as which city?
Question: Can Cinnamon drive?
=> 3 (fluent question)
Entity swapping:
1957, the Saar Protecturute rejoined the Hohen- zollern as which city?
Question: can cumin drive?
=> 3 (fluent question), despite the lack of cap- italization
Figure 3: Further examples of entity augmentation of answerable seed questions.
Question:
What sunscreen do bees use?
=> 3 (fluent question)
Question:
cinammon can drive?
=> 2 (minor errors), because of the typo in the
name
Long distance migrants are believed to dis- perse as young birds and form attachments to potential breeding sites and to favourite wintering sites. Once the site attachment is made they show high site-fidelity visiting the same wintering sites year after year.
Context:
Question: Does bees drive?
=> (minor errors), because the question is comprehensible even though it has an agree- ment error between "does" and "bees"
Answerable seed question:
Question: lunch drive what?
When long distance migrants disperse?
=> I (incomprehensible)
Antonym swapping:
When short distance migrants disperse?
Question: Can lunch drive?
Answerable seed question:
=> 3 (fluent question) syntactically, even
What do young birds firdsform attachments to?
though it is semantically nonsensical.
Antonym swapping:
What do old birds form attaclments to?
D More examples of augmented data
We present more examples data generated us- ing our augmentation strategies Figures 3 and 4, along with the context paragraph and the answer- able seed questions from SQuAD 2.0.
Figure 4: Further examples of antonym augmentation of answerable seed questions.


Reviews:
[{"review_id": "NKb5eH3wKE", "reviewer": "Reviewer_1Std", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and gratified that of the 3 issues they flagged in their review as reasons to reject, 2 are merely misunderstandings which we address below and 1 is a request for additional experiments which we have also performed. We hope the reviewer will reconsider their scores in light of our points below.\n\n## “Limited novelty” / “antonyms and entities were introduced in UNANSQ”\n\nThe UNANSQ paper (https://aclanthology.org/P19-1415/) **does not** propose antonym or entity augmentation. Their method is based on aligning human-written answerable and unanswerable questions from the original dataset and training a model on these pairs. Perhaps the reviewer misinterpreted Table 4 from their paper, which is in fact a human analysis of 100 system outputs to see what types of unanswerable questions were being generated by the system (mostly negated versions of the seed questions).\n\n## “It is hard to directly compare the numbers in Table 2 due to different data sizes”\n\nThis is _exactly_ the reason for **our ablation study in Figure 2**, controlling the amounts of data we were using from each method. Our empirical finding (as described in L220) is that our methods perform comparably with or better than more complex methods, even with lower, balanced amounts of data, like 500 samples or 1,000 samples. We will improve the signposting of this ablation study earlier on the paper.\n\n## “Only one evaluation dataset (SQuAD)”\n\n1. To show that our method is better and more lightweight than prior methods in the literature, we had to use the same dataset they do, which is also only SQuAD.\n2. To show the raw benefits of our data augmentation method compared to no augmentation, we have **new results on the English portion of TydiQA** that corresponds to minimum-span extractive QA. We used BERT-large and BERT-base fine-tuned over three runs and both antonym augmentation and entity augmentation had **statistically significant improvements** over the baselines, with F1 and EM improving by 8-10 points on average.\n\n| Methods          | EM        | F1        |\n|------------------|-----------|-----------|\n| BERT-large       | 51.4±0.8  | 54.4±0.7  |\n| + Entity (ours)  | 60.7±1.4* | 62.6±1.7* |\n| + Antonym (ours) | 61.2±0.3* | 63.7±0.4* |\n| BERT-base        | 48.5±0.5  | 51.6±0.7  |\n| + Entity (ours)  | 58.9±1.2* | 61.2±1.4* |\n| + Antonym (ours) | 58.7±0.7* | 61.4±0.7* |\n\nWe thank reviewer iyDW for suggesting TydiQA and hope that the addition of this result to our suite of significance tests across model types on SQuAD convinces all reviewers of the soundness of our claims.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is simple to understand and shown to work well in identifying the unanswerable questions. The author provides a method that does not require training. The paper compared with two baselines and show that their method outperforms baselines in SQUAD 2.0. ", "reasons_to_reject": "- The novelty of the paper is limited. The idea of entity and antonym was introduced previously in UNANSQ paper.\n- According to Table 1, different methods have different augmentations resulting in different number of generated samples. The comparison presentaed in Table 2 is not with common denominator, making it hard to directly compare the numbers.\n- As authors mentioned in Limitations, the QA system considered in the paper is limited to SQAUD 2.0 dataset. Extending the experiments to consider retrieval augmented extractive QA system would show true potential of the method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses the approach of augmenting the question with entity and antonym swap with respect to the context to make it unanswerable. The author proposed the training-free, lightweighted approach to improve the performance of encoder-only models like BERT to identify unanswerable questions.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bhdJQtTTvj", "reviewer": "Reviewer_HhfT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in reviewing our paper and address their comments below.\n\n## “The contribution of the problem is really very simple and small, but has not brought a great value.”\n\nRespectfully, we disagree with this characterization of our work and see plenty of value in our proposed method, which:\n1. Beats the previous state-of-the-art approaches to unanswerable question generation on SQuAD dev set performance\n2. Beats prior methods on generating highly readable and highly related unanswerable questions\n3. Is more data-efficient than prior methods as shown in our ablation study and analysis in Figure 2\n4. Is completely training-free (0 parameters) and transferable compared to prior state-of-the-art method (593M parameters)\n\nWe would appreciate it if the reviewer could clarify precisely what more they would like to see from our contribution.\n\n## “I don’t really see the strength of this method when applied to practical applications.”\n\n- As motivated in our introduction in L31-39, the task of unanswerable question generation is applied practically to build more reliable QA systems.\n- Unanswerable questions do not exist in most QA datasets and are expensive and slow for humans to write, but we expect QA systems built on such datasets to be able to identify them.\n- Our data augmentation method helps to cheaply build more reliable QA systems, and improves performance as mentioned above.\n\n## “Only one evaluation dataset (SQuAD)”\n\n1. To show that our method is better and more lightweight than prior methods in the literature, we had to use the same dataset they do, which is also only SQuAD.\n2. To show the raw benefits of our data augmentation method compared to no augmentation, we have **new results on the English portion of TydiQA** that corresponds to minimum-span extractive QA. We used BERT-large and BERT-base fine-tuned over three runs and both antonym augmentation and entity augmentation had **statistically significant improvements** over the baselines, with F1 and EM improving by 8-10 points on average.\n\n| Methods          | EM        | F1        |\n|------------------|-----------|-----------|\n| BERT-large       | 51.4±0.8  | 54.4±0.7  |\n| + Entity (ours)  | 60.7±1.4* | 62.6±1.7* |\n| + Antonym (ours) | 61.2±0.3* | 63.7±0.4* |\n| BERT-base        | 48.5±0.5  | 51.6±0.7  |\n| + Entity (ours)  | 58.9±1.2* | 61.2±1.4* |\n| + Antonym (ours) | 58.7±0.7* | 61.4±0.7* |\n\nWe thank reviewer iyDW for suggesting TydiQA and hope that the addition of this result to our suite of significance tests across model types on SQuAD convinces all reviewers of the soundness of our claims.\n\n## “What is the method for identifying words and finding antonyms?”\n\nThe types of words to augment (noun, adjective or verb) are identified using part-of-speech tagging with spaCy (L107). Antonyms and lemmas are accessed using WordNet through NLTK (L111). This point struck us more as a question than a reason for rejection from the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study published a simple and lightweight method of generating unanswered questions for experimental implementation and improved performance.", "reasons_to_reject": "The article has a few weak points:\n- In section 3.1, the author talked about finding and replacing antonyms, but he did not describe the method of identifying words and finding antonyms?\n- At experiment and show the results, it would be better if the authors prove this method improves performance on more datasets.\n- I don't really see the strength of this method when applied to practical applications.\n- The contribution of the problem is really very simple and small, but has not brought a great value.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study publishes a data-enhanced approach to generate unanswerable questions to increase the performance of reading comprehension models. And this method has proven to promote the performance of the reading comprehension model when evaluated on the development set of the SQuAD 2.0 dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dwm9JfqiCk", "reviewer": "Reviewer_YiU3", "summary": "", "questions": "- Question A: In Figure 2, as the number of augmented unanswerable questions increases, it creates a tradeoff that impacts the model's performance on answerable questions. Apart from overfitting on unanswerable questions, could you elaborate on the underlying reasons for this effect? Are there distinct characteristics or linguistic patterns present in the generated unanswerable questions that could potentially interfere with the model's accuracy in answering answerable questions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their acknowledgement of our method’s lightweightness, effectiveness across multiple metrics, and generalizability to datasets. Below, we address both issues they raised and answer their question. We hope they find our responses and new results compelling enough to raise their scores, or we request them to clarify which “minor points may need extra support or details.”\n\n## “Overfitting to Training Data”\n\nFirst, we would like to remind the reviewer that as our augmentation method is training-free, **neither training nor overfitting is technically possible** - it can be applied to any dataset.\n\n1. To show that our method is better and more lightweight than prior methods in the literature, we had to _evaluate_ on the same dataset they do, which is also only SQuAD. We discuss the implications of this in our Limitations section in the hope that this provides practical guidance for those trying to use our method on QA domains with different characteristics.\n2. To show the raw benefits of our data augmentation method compared to no augmentation, we have **new results on the English portion of TydiQA** that corresponds to minimum-span extractive QA. We used BERT-large and BERT-base fine-tuned over three runs and both antonym augmentation and entity augmentation had **statistically significant improvements** over the baselines, with F1 and EM improving by 8-10 points on average.\n\n| Methods          | EM        | F1        |\n|------------------|-----------|-----------|\n| BERT-large       | 51.4±0.8  | 54.4±0.7  |\n| + Entity (ours)  | 60.7±1.4* | 62.6±1.7* |\n| + Antonym (ours) | 61.2±0.3* | 63.7±0.4* |\n| BERT-base        | 48.5±0.5  | 51.6±0.7  |\n| + Entity (ours)  | 58.9±1.2* | 61.2±1.4* |\n| + Antonym (ours) | 58.7±0.7* | 61.4±0.7* |\n\nWe thank reviewer iyDW for suggesting TydiQA and hope that the addition of this result to our suite of significance tests across model types on SQuAD convinces all reviewers of the soundness of our claims.\n\n## “Limited diversity and lack of semantic understanding”\n\nWe thank the reviewer for bringing up the limited diversity of our generated examples. We are adding this to our Limitations section as it means that there will always be a qualitative gap between our samples and human-written samples, but want to emphasize that **this does not weaken our claims about outperforming previous automatic systems on SQuAD performance, readability, unanswerability and relevance to the topic**, which are arguably more important in this context.\n\nIt is also true that our surface-level method (using heuristics) lacks semantic understanding, but so do the other automatic methods in this area. We will nevertheless add this point to our Limitations section as well, along with the following examples of failure modes of surface-level heuristics, on the suggestion of this reviewer and another.\n- **Conjunctions**: With a context such as, “Devin and Anneke like milk chocolate.” and a seed question like “What kind of chocolate does Devin like?”, entity augmentation would generate a question like “What kind of chocolate does Anneke like?” which is still answerable.\n- **Commutative relations**: With a context such as “Beyoncé married Jay-Z in 2008.” and a seed question like “When did Beyoncé get married?”, entity augmentation would generate a question like “When did Jay-Z get married?” which is still answerable because marriage is commutative between two people.\n- **Information is elsewhere in the paragraph**: With a context such as “Twilight Princess was launched in North America in November 2006, and in Japan, Europe, and Australia the following month.” and a seed question like “When was Twilight Princess launched in North America?”, entity augmentation would generate a question like “When was Twilight Princess launched in Japan?” because the information is contained in the rest of the sentence.\n- **Other forms of yes/no questions**: Although we do not antonym-augment the most common forms of yes/no questions, we currently do not filter out questions like, “Outdoor cats are a bigger threat to birds than habitat loss—true or false?” and there are some questions of this type in the SQuAD dataset.\n\n-----------------------------------------------\n\n## Question A: What are the underlying reasons for the tradeoff between unanswerable and answerable questions?\n\n- As the reviewer points out in their question, we believe overfitting to be one of the main reasons for this effect.\n- Another potential reason for the tradeoff is noise, i.e., generated questions that are labelled unanswerable but are actually answerable. This could lead the model to abstain more on answerable questions, especially since our swapping means that both types of questions are often syntactically very similar. In column 1 of table 4, we quantify this noise as being about 20%.\n- These are hypotheses but we are interested in investigating them empirically in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Simple, Efficient, and Effective: The method is lightweight and training-free. It is based on simple antonym and entity swaps on answerable questions, reducing the need for costly manual annotation or complex paraphrasing techniques. The strategy outperforms baseline methods and achieves notable results in human-judged unanswerability, relatedness, and readability metrics\n- Potential Generalizability: The approach shows potential for easy extension to datasets beyond SQuAD 2.0, indicating its potential applicability to other question-answering tasks.", "reasons_to_reject": "- Limited Diversity and Lack of Semantic Understanding: The proposed method relies on surface-level manipulations of the input questions (antonym and entity swaps) rather than having a deep understanding of the semantics and context of the questions. While this might yield some unanswerable questions, it could limit the diversity of the generated questions. There could be other linguistic variations and structures that are characteristic of unanswerable questions, which the method may not explore effectively.\n- Overfitting to Training Data: The automatic method is based on SQuAD 2.0, which is a specific dataset that might have its own biases and characteristics. Depending solely on this dataset for training may lead to overfitting, and the generated questions might not generalize well to other domains or datasets.", "questions_to_authors": "- Question A: In Figure 2, as the number of augmented unanswerable questions increases, it creates a tradeoff that impacts the model's performance on answerable questions. Apart from overfitting on unanswerable questions, could you elaborate on the underlying reasons for this effect? Are there distinct characteristics or linguistic patterns present in the generated unanswerable questions that could potentially interfere with the model's accuracy in answering answerable questions?", "paper_topic_and_main_contributions": "A simple yet effective approach is introduced to generate unanswerable questions for extractive question answering systems. Drawing inspiration from human-written unanswerable questions, the method involves performing antonym and entity swaps on answerable questions, thus producing unanswerable variants. The proposed technique surpasses the performance of baseline methods and achieves notable results in human-judged unanswerability, relatedness, and readability metrics. The paper highlights the practical usefulness of the method for building better question answering systems and establishes it as a strong baseline for future research.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LEUfd2spBP", "reviewer": "Reviewer_iyDW", "summary": "", "questions": "\n1. Can the performance of the proposed method be maintained even for more complex multi-hop datasets like HotpotQA?\n\n2. Even when applying the proposed method of antonym and entity swaps, it seems that there is a possibility of still having answerable questions beyond the situations mentioned in the limitations. Have you taken into consideration or been aware of such scenarios?\n\n\n\n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort and address their questions and concerns below. We especially appreciate their recognition of our writing, the simplicity of our methodology, and our rigorous testing across multiple models and runs, along with significance testing.\n\n## “Only one evaluation dataset (SQuAD)”\n\n1. To show that our method is better and more lightweight than prior methods in the literature, we had to use the same dataset they do, which is also only SQuAD.\n2. To show the raw benefits of our data augmentation method compared to no augmentation, we have **new results on the English portion of TydiQA** that corresponds to minimum-span extractive QA. We used BERT-large and BERT-base fine-tuned over three runs and both antonym augmentation and entity augmentation had **statistically significant improvements** over the baselines, with F1 and EM improving by 8-10 points on average.\n\n| Methods          | EM        | F1        |\n|------------------|-----------|-----------|\n| BERT-large       | 51.4±0.8  | 54.4±0.7  |\n| + Entity (ours)  | 60.7±1.4* | 62.6±1.7* |\n| + Antonym (ours) | 61.2±0.3* | 63.7±0.4* |\n| BERT-base        | 48.5±0.5  | 51.6±0.7  |\n| + Entity (ours)  | 58.9±1.2* | 61.2±1.4* |\n| + Antonym (ours) | 58.7±0.7* | 61.4±0.7* |\n\nWe thank the reviewer for suggesting TydiQA and hope that the addition of this result to our suite of significance tests across model types on SQuAD convinces all the reviewers of the soundness of our claims.\n\n## “The methodology is less exciting”\n\nWhile we agree that swaps are a common simple data augmentation method and perhaps not very exciting, our contribution is less about the method itself and more about showing its effectiveness compared to the prior literature which proposes more “exciting” methodology but is ultimately over-engineered for the problem.\n\nPublishing our work would further the field by establishing a simple, strong baseline for unanswerable question generation in the literature, one that future work in this area would have to justify its complexity against. This is a possibility we are excited about.\n\n## “Lower unanswerability of our generated questions vs. higher performance on unanswerable questions”\n\nIt is indeed interesting and seemingly counterintuitive that we outperform CRQDA despite the unanswerability of our generated questions (78%) being lower than theirs (91%). We have two hypotheses for why this is the case:\n1. As Table 4 shows, the human-judged unanswerability of **our method is comparable with human-written unanswerable questions** (78% vs. 83%), suggesting that the setup intrinsically has some level of tolerance to data noise.\n2. **CRQDA questions do not provide a high-quality training signal** because their generated questions are mostly unintelligible (52% less readable than human-written questions, compared to our 7%) and unrelated to the paragraphs (36% less related to human-written questions compared to our equivalent-to-human-written relatedness). This suggests a **compositional effect**; unanswerability, relatedness and readability all play a role _together_ and doing reasonably well at all of them is important.\nAlthough we hint at this in L237-243, we regret that we were unable to address this in more detail in the draft. We thank the reviewer for raising this point and will certainly add this to the camera-ready version.\n\n\n-------------------------------------------------\n\n\n## Question A: Can performance be maintained for complex multi-hop datasets like HotpotQA?\n\nProbably! Although we would have to verify this empirically (and we are excited to pursue future work around this), we hypothesize that yes, our method would work very well for an entity-heavy dataset like HotpotQA, generating highly related and readable unanswerable questions. As an example, take the following question and supporting paragraphs from the dataset:\n\n**Q**: *Risingson is the first single from what album by Massive Attack, that was the first to be produced by Neil Davidge, along with the group?*\n\n**Paragraph A: Risingson**\n*\"Risingson\" is a song by the British trip hop group Massive Attack, released as a single on 7 July 1997. It is the first single from their third album \"Mezzanine\" and the eighth single overall.*\n\n**Paragraph B: Mezzanine (album)**\n*Mezzanine is the third studio album by English trip hop group Massive Attack, released on 20 April 1998 by Circa and Virgin Records. It was the first album to be produced by Neil Davidge, along with the group. The entire album was provided on their website for legal download many months before the physical release was announced, one of the first major uses of the MP3 format by a commercial organisation.*\n\nAntonym-swapping would produce the highly related, readable and unanswerable question:\n\n- *Risingson is the first single from what album by Massive Attack, that was the **last** to be produced by Neil Davidge, along with the group?*\n\nEntity-swapping (according to the NER output on this example) would give us even more options, such as:\n\n- ***Circa** is the first single from what album by Massive Attack, that was the first to be produced by Neil Davidge, along with the group?*\n\n- *Risingson is the first single from what album by **MP3**, that was the first to be produced by Neil Davidge, along with the group?*\n\n## Question B: “Are you aware of scenarios where answerable questions are generated beyond the situations mentioned in the limitations?”\n\nYes, we are aware of a few more error types due to the heuristic of using swaps and explain some of them below. We are adding these to the Limitations section of the paper in a subsection about the limited diversity and lack of deep semantics of our method, on the suggestion of this reviewer and another.\n- **Conjunctions**: With a context such as, “Devin and Anneke like milk chocolate.” and a seed question like “What kind of chocolate does Devin like?”, entity augmentation would generate a question like “What kind of chocolate does Anneke like?” which is still answerable.\n- **Commutative relations**: With a context such as “Beyoncé married Jay-Z in 2008.” and a seed question like “When did Beyoncé get married?”, entity augmentation would generate a question like “When did Jay-Z get married?” which is still answerable because marriage is commutative between two people.\n- **Information is elsewhere in the paragraph**: With a context such as “Twilight Princess was launched in North America in November 2006, and in Japan, Europe, and Australia the following month.” and a seed question like “When was Twilight Princess launched in North America?”, entity augmentation would generate a question like “When was Twilight Princess launched in Japan?” because the information is contained in the rest of the sentence.\n- **Other forms of yes/no questions**: Although we do not antonym-augment the most common forms of yes/no questions, we currently do not filter out questions like, “Outdoor cats are a bigger threat to birds than habitat loss—true or false?” and there are some questions of this type in the SQuAD dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. well-written paper, and easy-to read.\n2. simple, but effective method.\n3. various baseline models and experimental analysis with significant test.\n\n", "reasons_to_reject": "1. Less excitement by utilizing a methodology commonly used in data augmentation.\n2. Experiments on limited dataset (only one SQuAD 2.0 dataset). The authors can use other datasets like NewsQA and TydiQA which contain unanswerable questions.\n3. Despite the lower unanswerability in human evaluation (Table 4), the proposed method achieves high performance on unanswerable questions, as shown in Figure 2.  Analysis of this phenomenon is lacking.\n", "questions_to_authors": "\n1. Can the performance of the proposed method be maintained even for more complex multi-hop datasets like HotpotQA?\n\n2. Even when applying the proposed method of antonym and entity swaps, it seems that there is a possibility of still having answerable questions beyond the situations mentioned in the limitations. Have you taken into consideration or been aware of such scenarios?\n\n\n\n\n", "paper_topic_and_main_contributions": "The authors presented a training-free, simple and effective method through antonym and entity swap to create unanswerable data in the QA dataset. In addition, the proposed method achieved higher readability and relatedness than existing methods and higher final performance.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes augmenting training data for extractive QA by incorporating unanswerable questions, created by swapping antonyms and entities found in answerable questions. Experiments, including ablation studies, demonstrate that the proposed approach enhances performance on SQuAD (as discussed in the paper) and TidyQA (as mentioned in the rebuttal). The paper is well-structured and easy to follow. Its findings, including experimental setting, can be of interest to the QA community.
Decision: Accept-Findings