SlowGuess/NAACL_2022_OCR huggingface dataset:
Elior Sulem, Jamaal Hay and Dan Roth
Department of Computer and Information Science, University of Pennsylvania
eliors, jamaalh, danroth@seas.upenn.edu
The Yes/No QA task (Clark et al., 2019) consists of "Yes" or "No" questions about a given context. However, in realistic scenarios, the information provided in the context is not always sufficient in order to answer the question. For example, given the context "She married a lawyer from New-York," we don't know whether the answer to the question "Did she marry in New York?" is "Yes" or "No". In this paper, we extend the Yes/No QA task, adding questions with an IDK answer, and show its considerable difficulty compared to the original 2-label task. For this purpose, we (i) enrich the BoolQ dataset (Clark et al., 2019) to include unanswerable questions and (ii) create out-of-domain test sets for the Yes/No/IDK QA task. We study the contribution of training on other Natural Language Understanding tasks. We focus in particular on Extractive QA (Rajpurkar et al., 2018) and Recognizing Textual Entailments (RTE, Dagan et al., 2013), analyzing the differences between 2 and 3 labels using the new data.
The ability to know whether a claim is true or false given a context is an important component of language comprehension. One main way to study this ability is the Yes/No Question Answering task, for which a large-scale dataset, BoolQ, has been proposed (Clark et al., 2019). The BoolQ dataset includes paragraphs together with naturally occurring questions whose answer is either "Yes" or "No".
However, in realistic scenarios, the information needed to answer a Yes/No question can be missing. For example, Figure 1 shows Yes/No questions given the context "Jane, who is a native of Los Angeles, married a lawyer from NYC". While the questions in (a) and (b) can be answered respectively by "Yes" and "No" (Jane was born in Los
Context: Jane, who is a native of Los Angeles, married a lawyer from NYC.
(a) Question: Did Jane marry a lawyer? Answer: Yes
(b) Question: Was Jane born in France? Answer: No
(c) Question: Did Jane marry in NYC? Answer: IDK
Figure 1: Examples of a "Yes", "No" and "IDK" Questions in (a), (b) and (c) respectively, given a Context.
Angeles so she was not born in France), the question "Did Jane marry in NYC?" in (c) cannot be answered given the context.
Indeed, the ability to extract information from text only addresses one aspect of the expectations we have from a comprehension system. Another main aspect concerns the ability to identify that a given information is not in the text, a witness of understanding in human comprehension.
In this paper we extend the Yes/No QA task to include IDK questions and show the considerable difficulty of the extended task compared to the two-label setting. For this purpose we first enrich the BoolQ dataset by including unanswerable questions, along with creating two other, out of domain, datasets, to test the ability to answer Yes/No/IDK questions in realistic scenarios (Section 3).
Experimenting with a system based on BERT-LARGE (Devlin et al., 2019), we show that the performance on the dev set drops from 72.88 F1 to 33.64 F1 when moving from a two-label to a three-label setting and observe similar results on out-of-domain test sets (Section 4).
We then explore the contribution of other Natural Language Understanding tasks to the performance, focusing on Extractive QA (Rajpurkar et al., 2016), using the SQuAD 2.0 dataset (Rajpurkar et al., 2018) and Recognizing Textual Entailments (RTE, Dagan et al., 2013), using the MNLI dataset
1All the datasets and the code are available at http:// cogcomp.org/page/publication_view/975.
Human Language Technologies, pages 1075 - 1085
Figure 2: Examples of Yes/No QA (left), Extractive QA (center) and RTE hypotheses (right) for a given context.
(Williams et al., 2018). We obtain that, similar to what has been observed in the two-label setting (Clark et al., 2019), leveraging SQuAD 2.0 does not improve the performance while training on MNLI achieves better performance. As the improvement is limited, we separately analyze the transfer from MNLI in the two-label and three-label settings.
We conclude that current systems that achieve high performance on the BoolQ dataset are not adapted to the task of Yes/No QA where unanswerable questions are involved. Using the RTE data is helpful but not sufficient for a good performance.
In this paper, we provide new datasets that allow a more realistic evaluation of the Yes/No task by (1) addressing unanswerable questions, which appear in real-world scenarios; (2) compiling test sets to evaluate the performance of the systems on domains that are different from the ones seen in training and finetuning. Using the new data, we explore the performance of current systems on this task and show its considerable difficulty compared to its two-label version.
Yes/No Questions In Yes/No QA, the IDK option has been taken into account in the context of FraCaS inference problem (Cooper et al., 1996), which consists of 346 problems targeting specific linguistic phenomena, each containing one or more statements and one yes/no-question. The possible labels are “yes”, “no”, “don’t know” and a “other/complex” label that mainly targets several possible readings. Clark et al. (2019) proposed the large-scale BoolQ dataset in order to address the Yes/No QA task where a question together with a paragraph are given as input. The task consists in classifying the questions into two categories: “Yes” and “No” questions. This dataset does not include an IDK option, a gap we fill by augmenting the original corpus (see Section 3). Our data augmentation method is somewhat similar to the one used
for Extractive QA by Clark and Gardner (2018) who generated negative examples for SQuAD (Rajpurkar et al., 2016) by pairing existing questions with other paragraphs from the same article based on TF-IDF overlap.
Unanswerable Questions Unanswerable questions have been mainly studied in the context of the Extractive QA task. The original Extractive QA task consists in extracting a correct answer to a question from a context paragraph or document. Rajpurkar et al. (2018) enriched the SQuAD 1.1 corpus (Rajpurkar et al., 2016) by including unanswerable questions for the same paragraphs via crowdsourcing, resulting in SQuAD 2.0, and proposed to extend the task so that the predictions will include either a span or an "IDK" answer. Unanswerable questions have also been included in the QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) datasets where questions are asked in the form of a dialog (see Yatskar (2019) for a comparison). The difficulty of unanswerable questions in Extractive QA has been recently explored in the case of out-of-domain scenarios (Sulem et al., 2021) and in questions with a context composed of multiple paragraphs (Asai and Choi, 2021). Although Extractive QA includes IDK questions, the conversion between the Extractive QA format and a Yes/No/IDK QA format will not always preserve the IDK label, as shown in Figure 2. In particular, an IDK instance in Extractive QA (Q2b) can correspond to a "No" answer in Yes/No QA (Q1b).
The selective question answering task in out-of-domain settings (Kamath et al., 2020) is related to the identification of unanswerable questions. However, it targets the ability of a system to refrain from answering in some of the cases in order to avoid errors in out-of-domain settings, independently from the presence of the answer in the context.
Recognizing Textual Entailment (RTE) The RTE task (Dagan et al., 2013) consists of classi
fying a sentence pair composed of a premise  $p$  and a hypothesis  $h$  into three classes, according to the relation between the two sentences: "entailment", "contradiction" and "neutral". In some of the RTE works (Bentivogli et al., 2009; Wang et al., 2018), "contradiction" and "neutral" are unified in a "non-entailed" joint category. Both RTE and Yes/No QA aim to verify whether a given information can be derived from the context. Furthermore, there is a one-to-one mapping between the labels of the two tasks. In particular, if the RTE label is "neutral" then the answer to the corresponding question is IDK (as in Figure 2, Q1a and H3a). However, a main difference between the two tasks is the length of the context. While it is a single sentence in the RTE datasets (e.g., Bowman et al., 2015; Williams et al., 2018, for SNLI and MNLI respectively), it is a paragraph in Yes/No QA. The extension of the Yes/No QA task and data presented in this paper allows their use in multiple applications such as Extractive QA (Sulem et al., 2021), relation extraction (Obamuyide and Vlachos, 2018; Sainz et al., 2021) or event extraction (Lyu et al., 2021; Sainz et al., 2022) in replacement of or jointly with RTE. We explore the use of an RTE dataset for additional training in Section 5. An investigation of the replacement of the RTE task by Yes/No/IDK QA for Extractive QA is presented in Appendix H.
3.1 Enriching BoolQ with IDK Questions
The BoolQ dataset<sup>3</sup>, proposed by Clark et al. (2019), is the largest corpus for Yes/No questions. The training set is composed of 9.4K of Yes/No questions and their answers.  $62.31\%$  of the answers are "yes". The dev set includes 3.2k questions.
General Idea. We aim to generate IDK questions by mapping questions from the original BoolQ to passages that will not contain anymore the information required to answer the question. As a random swapping between passages and questions could generate mostly very simple cases with no relation between the question and the passage, we want to maximize the word overlapping between the two.
Generation. We augment BoolQ with IDK questions automatically by using passages and questions from the original BoolQ dataset. Sampling randomly half of the "yes" questions and half of the "no" questions, we match to each of the extracted questions a passage from BoolQ that has the greatest overlapping with the questions in terms of nouns and verbs, identified using the NLTK PoS tagger (Loper and Bird, 2002). The greatest overlapping is chosen to avoid very simple cases with no relation between the question and the passage. In case there are several passages with the same number of nouns or verbs that appear in the question, we choose one of them randomly. We apply this algorithm (separately) on both the train set and the dev set of BoolQ, obtaining new IDK questions (4.7k for train and 1.6k for dev) that we add to the original sets. We call the new corpus BoolQ<sub>3L</sub> (for BoolQ with three labels). Examples from BoolQ<sub>3L</sub> are shown in Appendix C.
Validation and Analysis. We manually validate the IDK questions we compiled by sampling 100 question-paragraph pairs randomly in each of the sets (the train and the dev sets). The 200 pairs are annotated separately by two of the authors of the paper using the "IDK", "Yes" and "No" labels. In the dev set, we find that  $95\%$  of the instances are correctly labeled with  $100\%$  absolute inter-annotator agreement. In the train set the rates of instances correctly labeled as "IDK" are  $94\%$  and  $93\%$  according to the two annotators, with an absolute inter-annotator agreement of  $97\%$ . Using the same samples we also evaluate difficulty of the questions, based on the relatedness of the paragraph to the question, abstracting away from the word overlapping between the two. Following this procedure, we label  $33\% / 40\%$  ( $85\%$  absolute agreement) and  $43\% / 63\%$  ( $80\%$  absolute agreement) cases as Non-related in the train and dev set respectively. Examples of Non-related IDK questions are presented in Appendix D.
By using questions and answers from the original BoolQ, we ensure that the new IDK subset cannot be identified as having different question or
4We also provide a list of auxiliaries that should not be considered as verbs.
For comparison, in the case of SQuAD 2.0 (Rajpurkar et al., 2018), the authors manually inspected 100 randomly chosen negative examples and found that  $93\%$  of the examples are indeed unanswerable. In the original BoolQ dataset (with "Yes" and "No" labels), 110 randomly chosen examples were annotated, reaching  $90\%$  accuracy relative to the gold labels, corresponding to 6 ambiguous cases and 5 errors.
2Although it is not inherent in the definition of the respective tasks, the available datasets impact the models used by the community.
3https://github.com/  
google-research-datasets/  
boolean-questions
Corpus	Split	# Examples	IDK (%)	# Labels
Existing Corpus
BoolQ	train	9,427	0	2
dev	3,270	0
New Corpora
BoolQ3L	train	14,141	33	3
dev	4,906	33
ACE-YNQA	test	999	52	3
INSTRUCTIONS	test	70	33	3
Table 1: Statistics and properties of the BoolQ corpus (top) and the newly introduced corpora (bottom).
paragraph styles or levels of grammaticality. Also, the IDK questions have paragraphs that also appear in either "Yes" or "No" questions, with the proportion similar to that observed in the entire corpus. Therefore "no-answer" cannot be predicted by only looking at the question.<sup>6</sup>
3.2 Out-of-domain Test Sets
Motivation Addressing unanswerable questions in Yes/No QA is a first step towards a more realistic evaluation of this task. A second step, which we address in this section, consists in the evaluation of the systems on datasets different from the ones they have been trained and finetuned on, using controlled out-of-domain test sets for evaluation, as advocated for example by Linzen (2020). Furthermore, an informative way to evaluate the comprehension of a system is to ask very simple questions whose answers are obvious to humans (Dunietz et al., 2020). We focus here on event-based questions that address simple predicate-argument relations.
ACE-YNQA We leverage the ACE event extraction (Walker et al., 2006) dataset to derive a new test corpus of "Yes", "No" and IDK questions. For this purpose, we first select sentence fragments that include a location or a time mention according to the ACE annotation. For the "Yes" questions we generate questions of the form "Did  $T$  happen at location / time entity", where  $T$  is the event trigger. For the "No" variant we manually created a set of place / time entities and asked "Did T happen at  $r$ ?", where  $r$  is a randomly selected entity. We then checked for grammatical and logical correctness. For the IDK labels, we generated those
questions manually asking context specific questions. An example of IDK question is "has the loan been paid?" given the context "the world bank first offered the loan in 1999". We call the obtained corpus ACE-YNQA. Examples with "Yes", "No" and "IDK" answers are shown in Appendix E.
INSTRUCTIONS We also generate a small test corpus (INSTRUCTIONS; 70 questions) from scratch about instructions. For example, given the context "Change the font color to green", an IDK question is "Is the font size 12?"'. More examples are presented in Appendix F.
A summary of the statistics for the different corpora is presented in Table 1.
4 The Difficulty of Yes/No/IDK QA
We use the BERT-LARGE representation and the BERT TensorFlow implementation for sequence classification. We train on the  $\mathrm{BoolQ}_{3L}$  training set and evaluate on the ACE-YNQA out-of-domain dataset. We also report the average performance on the dev set. We use the BERT-based approach for classification, where the three labels are "Yes", "No" and "IDK"; the final hidden vector corresponding to the first input token([CLS]) is used as the aggregate representation. The fine-tuning details and hyperparameters are presented in Appendix A. For comparison, we also fine-tune BERT-LARGE on the original BoolQ dataset for the Yes/No QA task using the original train and dev sets. We consider the "Yes" and "No" portions of the ACE-YNQA as the out-of-domain test set for the 2-label setting.
The results are presented in the first columns of Table 2 (for Yes/No/IDK) and Table 3 (for Yes/No). We find that the accuracy of the model drops to 33.64 on the  $\mathrm{BoolQ}_{3L}$  dev set while it is 72.88 in the two-level setting (when training and testing on  $\mathrm{BoolQ}$ ). On the out-of-domain ACE-YNQA test set, the performance is 52.02. In this case too the score is higher in the two-label setting (accuracy of 59.53).
<sup>6</sup>This phenomenon has been observed in existing resources in the case of the similar RTE task (Gururangan et al., 2018). For the original BoolQ, the experiments of Clark et al. (2019) suggest that there is little signal in the question by itself, but that some language patterns in the passage correlate with the answer.
$^{7}$ https://catalog.ldc.upenn.edu/LDC2006T06
$^{8}$ https://github.com/google-research/bert
Train → Test ↓	BoolQ3L	MNLI + BoolQ3L	c(MNLI) + BoolQ3L	SQuAD 2.0 +BoolQ3L
BoolQ3L dev	33.64	42.66	43.25	35.27
ACE-YNQA	52.02	52.02	54.94	44.15
Table 2: Accuracy of the different systems, tested on Yes/No/IDK QA with 3 labels. The scores correspond to average across 5 different runs. The columns represent the training strategies. The rows represent the test datasets. In all the cases the trained representation is BERT-LARGE-CASED.
Train → Test ↓	BoolQ	MNLI + BoolQ	c(MNLI) + BoolQ	SQuAD 2.0 + BoolQ
BoolQ dev	72.88	78.24	79.49	62.13
ACE-YNQAy/N	59.53	65.47	68.01	53.81
Table 3: Accuracy of the baseline system as well as the use of Extractive QA for both Yes/No QA with 2 labels on the BoolQ dev set an on the out-of-domain datasets, removing the IDK examples. In all the cases the trained representation is BERT-LARGE-CASED.
5 Leveraging Other Tasks
We leverage the RTE task, using the MNLI corpus (Williams et al., 2018) and the Extractive QA task, using SQuAD  $2.0^{9}$  (Rajpurkar et al., 2018).
Given a language representation  $R$  and two corpora  $C_1$  and  $C_2$  for the tasks  $T_1$  and  $T_2$  respectively,  $C_1 + C_2$  refers to the procedure in which  $R$  is first fine-tuned on  $C_1$  for the task  $T_1$  and then further fine-tuned on  $C_2$  for the task  $T_2$ . This is similar to the STILTS method (Phang et al., 2018). We experiment with the following systems: (i) MNLI + BoolQ3L (ii) c(MNLI) + BoolQ3L where c(MNLI) is a binary version of MNLI, distinguishing between contradictions and non-contradictions (iii) SQuAD 2.0 + BoolQ3L.
We also replicate the above systems in the two-label setting, training and testing on BoolQ.
The results are presented in Tables 2 and 3. Evaluating on  $\mathrm{BoolQ}_{3L}$  dev set, we find that the use of MNLI for intermediary fine-tuning (MNLI +  $\mathrm{BoolQ}_{3L}$ ) improves the overall accuracy, which reaches a score of 42.66. The best performance is achieved when using intermediary fine-tuning on the binary MNLI where the accuracy is 43.25. A similar behavior is observed in the two-label setting although the scores are much higher.
On the ACE-YNQA out-of-domain test set,  $c(\mathrm{MNLI}) + \mathrm{BoolQ}_{3L}$  is the best system. Its 2-level
version is the best system in the case of Yes/No QA. Leveraging SQuAD 2.0 is not helpful in both 2 (as also in Clark et al. (2019)) and 3 label settings.
In order to explore additional types of Yes/No questions, including unanswerable questions, we replicate our experiments on the small INSTRUC-TIONS dataset. As before, we find a gap between the two-label and three-label scores and the usefulness of the MNLI corpus. The full results are presented in Appendix G.
In this paper we aim to allow a more realistic evaluation of the Yes/No QA task. For this purpose, we (i) enrich the BoolQ dataset to include unanswerable questions and (ii) compile out-of-domain test sets. Using the new data, we show the difficulty of current systems to address the task both by training directly on the task-specific data and by leveraging other NLU tasks.
We thank Qing Lyu and the members of the Cognitive Computation Group for their insightful feedback ad well as the reviewers of the paper for their useful suggestions. This work was supported by Contracts FA8750-19-2-1004 and FA8750-19-2-0201 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. This research is also based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program. The views and conclusions contained herein are those of the authors.
$^{9}$ https://rajpurkar.github.io/SquAD-explorer/
10 When training for Extractive QA and then moving to a classification task such as Yes/No QA, we also remove the last layer before training on the classification task.
<sup>11</sup>We chose this binary version for the experiments (the other versions being "entailment"/"non-entailment" and "neutral"/"non-neutral") since it achieved the highest score on the corresponding binary MNLI dev set (92.50 accuracy).
of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This material is also based upon work supported by Google Cloud (TRC Program).
Akari Asai and Eunsol Choi. 2021. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1492-1504, Online. Association for Computational Linguistics.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The sixth pascal recognizing textual entailment challenge. In TAC.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 632-642.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentauu Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184.
Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924—2936.
Robin Cooper, Richard Crouch, Jan van Eijck, Chris Fox, Josef van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, Stephen Pulman, and Espen Vestre. 1996. Describing the approaches. In The FraCaS Consortium.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzoto. 2013. Recognizing Textual Entailment: Models and Applications.