Venue: NAACL 2022

# Yes, No or IDK: The Challenge of Unanswerable Yes/No Questions

Elor Sulem, Jamaal Hay and Dan Roth

Department of Computer and Information Science, University of Pennsylvania

eliors,jamaalh,danroth@seas.upenn.edu

###### Abstract

The Yes/No QA task Clark et al. (2019) consists of "Yes" or "No" questions about a given context. However, in realistic scenarios, the information provided in the context is not always sufficient in order to answer the question. For example, given the context _"She married a lawyer from New-York."_, we don't know whether the answer to the question _"Did she marry in New York?"_ is "Yes" or "No". In this paper, we extend the Yes/No QA task, adding questions with an IDK answer, and show its considerable difficulty compared to the original 2-label task. For this purpose, we (i) enrich the BoolQ dataset Clark et al. (2019) to include unanswerable questions and (ii) create out-of-domain test sets for the Yes/No/IDK QA task. We study the contribution of training on other Natural Language Understanding tasks. We focus in particular on Extractive QA Rajpurkar et al. (2018) and Recognizing Textual Entailments RTE et al. (2013), analyzing the differences between 2 and 3 labels using the new data.1

Footnote 1: All the datasets and the code are available at [http://cogcomp.org/page/publication_view/975](http://cogcomp.org/page/publication_view/975).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1075 - 1085


(Williams et al., 2018). We obtain that, similar to what has been observed in the two-label setting (Clark et al., 2019), leveraging SQuAD 2.0 does not improve the performance while training on MNLI achieves better performance. As the improvement is limited, we separately analyze the transfer from MNLI in the two-label and three-label settings.

We conclude that current systems that achieve high performance on the BoolQ dataset are not adapted to the task of Yes/No QA where unanswerable questions are involved. Using the RTE data is helpful but not sufficient for a good performance.

In this paper, we provide new datasets that allow a more realistic evaluation of the Yes/No task by (1) addressing unanswerable questions, which appear in real-world scenarios; (2) compiling test sets to evaluate the performance of the systems on domains that are different from the ones seen in training and finetuning. Using the new data, we explore the performance of current systems on this task and show its considerable difficulty compared to its two-label version.

## 2 Related Work

Yes/No QuestionsIn Yes/No QA, the IDK option has been taken into account in the context of FraCaS inference problem (Cooper et al., 1996), which consists of 346 problems targeting specific linguistic phenomena, each containing one or more statements and one yes/no-question. The possible labels are "yes", "no", "don't know" and a "other/complex" label that mainly targets several possible readings. Clark et al. (2019) proposed the large-scale BoolQ dataset in order to address the Yes/No QA task where a question together with a paragraph are given as input. The task consists in classifying the questions into two categories: "Yes" and "No" questions. This dataset does not include an IDK option, a gap we fill by augmenting the original corpus (see Section 3). Our data augmentation method is somewhat similar to the one used for Extractive QA by Clark and Gardner (2018) who generated negative examples for SQuAD Rajpurkar et al. (2016) by pairing existing questions with other paragraphs from the same article based on TF-IDF overlap.

Unanswerable QuestionsUnanswerable questions have been mainly studied in the context of the Extractive QA task. The original Extractive QA task consists in extracting a correct answer to a question from a context paragraph or document. Rajpurkar et al. (2018) enriched the SQuAD 1.1 corpus (Rajpurkar et al., 2016) by including unswerable questions for the same paragraphs via crowdsourcing, resulting in SQuAD 2.0, and proposed to extend the task so that the predictions will include either a span or an "IDK" answer. Unanswerable questions have also been included in the QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) datasets where questions are asked in the form of a dialog (see Yatskar (2019) for a comparison). The difficulty of unanswerable questions in Extractive QA has been recently explored in the case of out-of-domain scenarios (Sulem et al., 2021) and in questions with a context composed of multiple paragraphs (Asai and Choi, 2021). Although Extractive QA includes IDK questions, the conversion between the Extractive QA format and a Yes/No/IDK QA format will not always preserve the IDK label, as shown in Figure 2. In particular, an IDK instance in Extracting QA (Q2b) can correspond to a "No" answer in Yes/No QA (Q1b).

The selective question answering task in out-of-domain settings (Kamath et al., 2020) is related to the identification of unanswerable questions. However, it targets the ability of a system to refrain from answering in some of the cases in order to avoid errors in out-of-domain settings, independently from the presence of the answer in the context.

Recognizing Textual Entailment (RTE)The RTE task (Dagan et al., 2013) consists of classis

Figure 2: Examples of Yes/No QA (left), Extractive QA (center) and RTE hypotheses (right) for a given context.

 

fying a sentence pair composed of a premise \(p\) and a hypothesis \(h\) into three classes, according to the relation between the two sentences: "entailment", "contradiction" and "neutral". In some of the RTE works Bentivogli et al. (2009); Wang et al. (2018), "contradiction" and "neutral" are unified in a "non-entailed" joint category. Both RTE and Yes/No QA aim to verify whether a given information can be derived from the context. Furthermore, there is a one-to-one mapping between the labels of the two tasks. In particular, if the RTE label is "neutral" then the answer to the corresponding question is IDK (as in Figure 2, Q1a and H3a). However, a main difference between the two tasks is the length of the context. While it is a single sentence in the RTE datasets (e.g., Bowman et al., 2015; Williams et al., 2018, for SNLI and MNLI respectively), it is a paragraph in Yes/No QA.2 The extension of the Yes/No QA task and data presented in this paper allows their use in multiple applications such as Extractive QA Sulem et al. (2021), relation extraction Obamuyide and Vlachos (2018); Sainz et al. (2021) or event extraction Lyu et al. (2021); Sainz et al. (2022) in replacement of or jointly with RTE. We explore the use of an RTE dataset for additional training in Section 5. An investigation of the replacement of the RTE task by Yes/No/IDK QA for Extractive QA is presented in Appendix H.

Footnote 2: Although it is not inherent in the definition of the respective tasks, the available datasets impact the models used by the community.

## 3 Datasets

### Enriching BoolQ with IDK Questions

The BoolQ dataset3, proposed by Clark et al. (2019), is the largest corpus for Yes/No questions. The training set is composed of 9.4K of Yes/No questions and their answers. 62.31% of the answers are "yes". The dev set includes 3.2k questions.

Footnote 3: [https://github.com/google-research-datasets/boolean-questions](https://github.com/google-research-datasets/boolean-questions)

General Idea.We aim to generate IDK questions by mapping questions from the original BoolQ to passages that will not contain anymore the information required to answer the question. As a random swapping between passages and questions could generate mostly very simple cases with no relation between the question and the passage, we want to maximize the word overlapping between the two.

Generation.We augment BoolQ with IDK questions automatically by using passages and questions from the original BoolQ dataset. Sampling randomly half of the "yes" questions and half of the "no" questions, we match to each of the extracted questions a passage from BoolQ that has the greatest overlapping with the questions in terms of nouns and verbs, identified using the NLTK PoS tagger Loper and Bird (2002).4 The greatest overlapping is chosen to avoid very simple cases with no relation between the question and the passage. In case there are several passages with the same number of nouns or verbs that appear in the question, we choose one of them randomly. We apply this algorithm (separately) on both the train set and the dev set of BoolQ, obtaining new IDK questions (4.7k for train and 1.6k for dev) that we add to the original sets. We call the new corpus BoolQ\({}_{3L}\) (for BoolQ wih three labels). Examples from BoolQ\({}_{3L}\) are shown in Appendix C.

Footnote 4: We also provide a list of auxiliaries that should not be considered as verbs.

Validation and Analysis.We manually validate the IDK questions we compiled by sampling 100 question-paragraph pairs randomly in each of the sets (the train and the dev sets). The 200 pairs are annotated separately by two of the authors of the paper using the "IDK", "Yes" and "No" labels. In the dev set, we find that 95% of the instances are correctly labeled with 100% absolute inter-annotator agreement. In the train set the rates of instances correctly labeled as "IDK" are 94% and 93% according to the two annotators, with an absolute inter-annotator agreement of 97%. 5 Using the same samples we also evaluate difficulty of the questions, based on the relatedness of the paragraph to the question, abstracting away from the word overlapping between the two. Following this procedure, we label 33% / 40% (85% absolute agreement) and 43% / 63 % (80 % absolute agreement) cases as Non-related in the train and dev set respectively. Examples of Non-related IDK questions are presented in Appendix D.

Footnote 5: For comparison, in the case of SQuAD 2.0 Rajpurkar et al. (2018), the authors manually inspected 100 randomly chosen negative examples and found that 93% of the examples are indeed unanswerable. In the original BoolQ dataset (with “Yes” and “No” labels), 110 randomly chosen examples were annotated, reaching 90 % accuracy relative to the gold labels, corresponding to 6 ambiguous cases and 5 errors.

By using questions and answers from the original BoolQ, we ensure that the new IDK subset cannot be identified as having different question or 

paragraph styles or levels of grammaticality. Also, the IDK questions have paragraphs that also appear in either "Yes" or "No" questions, with the proportion similar to that observed in the entire corpus. Therefore "no-answer" cannot be predicted by only looking at the question.6

Footnote 5: This phenomenon has been observed in existing resources in the case of the similar RTE task Gururangan et al. (2018). For the original BoolQ, the experiments of Clark et al. (2019) suggest that there is little signal in the question by itself, but that some language patterns in the passage correlate with the answer.

### Out-of-domain Test Sets

MotivationAddressing unanswerable questions in Yes/No QA is a first step towards a more realistic evaluation of this task. A second step, which we address in this section, consists in the evaluation of the systems on datasets different from the ones they have been trained and finetuned on, using controlled out-of-domain test sets for evaluation, as advocated for example by Linzen (2020). Furthermore, an informative way to evaluate the comprehension of a system is to ask very simple questions whose answers are obvious to humans Dunietz et al. (2020). We focus here on event-based questions that address simple predicate-argument relations.

ACE-YNQAWe leverage the ACE event extraction Walker et al. (2006)7 dataset to derive a new test corpus of "Yes", "No" and IDK questions. For this purpose, we first select sentence fragments that include a location or a time mention according to the ACE annotation. For the "Yes" questions we generate questions of the form "Did \(T\) happen at location / time entity", where \(T\) is the event trigger. For the "No" variant we manually created a set of place / time entities and asked "Did T happen at \(r?\)", where \(r\) is a randomly selected entity. We then checked for grammatical and logical correctness. For the IDK labels, we generated those questions manually asking context specific questions. An example of IDK question is "has the loan been paid?" given the context "the world bank first offered the loan in 1999". We call the obtained corpus ACE-YNQA. Examples with "Yes", "No" and "IDK" answers are shown in Appendix E.

InstructionsWe also generate a small test corpus (INSTRUCTIONS; 70 questions) from scratch about instructions. For example, given the context "Change the font color to green", an IDK question is "Is the font size 12?". More examples are presented in Appendix F.

A summary of the statistics for the different corpora is presented in Table 1.

## 4 The Difficulty of Yes/No/IDK QA

We use the BERT-LARGE representation and the BERT TensorFlow implementation8 for sequence classification. We train on the BoolQ\({}_{3L}\) training set and evaluate on the ACE-YNQA out-of-domain dataset. We also report the average performance on the dev set. We use the BERT-based approach for classification, where the three labels are "Yes", "No" and "IDK"; the final hidden vector corresponding to the first input token([CLS]) is used as the aggregate representation. The fine-tuning details and hyperparameters are presented in Appendix A. For comparison, we also fine-tune BERT-LARGE on the original BoolQ dataset for the Yes/No QA task using the original train and dev sets. We consider the "Yes" and "No" portions of the ACE-YNQA as the out-of-domain test set for the 2-label setting.

Footnote 8: [https://github.com/google-research/bert](https://github.com/google-research/bert)

The results are presented in the first columns of Table 2 (for Yes/No/IDK) and Table 3 (for Yes/No). We find that the accuracy of the model drops to 33.64 on the BoolQ\({}_{3L}\) dev set while it is 72.88 in the two-level setting (when training and testing on BoolQ). On the out-of-domain ACE-YNQA test set, the performance is 52.02. In this case too the score is higher in the two-label setting (accuracy of 59.53).

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Corpus & Split & \# Examples & IDK (\%) & \# Labels \\ \hline \multicolumn{4}{|c|}{**Existing Corpus**} \\ \hline  & train & 9.427 & 0 & 2 \\ \cline{2-4}  & dev & 3.270 & 0 & \\ \hline  & \multicolumn{4}{|c|}{**New Corpora**} \\ \hline  & train & 14,141 & 33 & 3 \\ \hline  & dev & 4,906 & 33 & 3 \\ \hline ACE-YNQA & test & 999 & 52 & 3 \\ \hline INSTRUCTIONS & test & 70 & 33 & 3 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics and properties of the BoolQ corpus (top) and the newly introduced corpora (bottom).

 



## 5 Leveraging Other Tasks

We leverage the RTE task, using the MNLI corpus Williams et al. (2018) and the Extractive QA task, using SQuAD \(2.0^{9}\)Rajpurkar et al. (2018).

Given a language representation \(R\) and two corpora \(C_{1}\) and \(C_{2}\) for the tasks \(T_{1}\) and \(T_{2}\) respectively, \(C_{1}+C_{2}\) refers to the procedure in which \(R\) is first fine-tuned on \(C_{1}\) for the task \(T_{1}\) and then further fine-tuned on \(C_{2}\) for the task \(T_{2}\).10 This is similar to the STILTS method Phang et al. (2018). We experiment with the following systems: (i) **MNLI + BoolQ\({}_{3L}\)** (ii) \(c(\mathrm{MNLI})\) **+ BoolQ\({}_{3L}\)** where c(MNLI) is a binary version of MNLI, distinguishing between contradictions and non-contradictions11 (iii) **SQuAD 2.0 + BoolQ\({}_{3L}\)**.

Footnote 10: When training for Extractive QA and then moving to a classification task such as Yes/No QA, we also remove the last layer before training on the classification task.

Footnote 11: We chose this binary version for the experiments (the other versions being “entailment”/“non-entailment” and “neutral”/"non-neutral”) since it achieved the highest score on the corresponding binary MNLI dev set (92.50 accuracy).

We also replicate the above systems in the two-label setting, training and testing on BoolQ.

The results are presented in Tables 2 and 3. Evaluating on BoolQ\({}_{3L}\) dev set, we find that the use of MNLI for intermediary fine-tuning (MNLI + BoolQ\({}_{3L}\)) improves the overall accuracy, which reaches a score of 42.66. The best performance is achieved when using intermediary fine-tuning on the binary MNLI where the accuracy is 43.25. A similar behavior is observed in the two-label setting although the scores are much higher.

On the ACE-YNQA out-of-domain test set, \(c(\mathrm{MNLI})\) **+ BoolQ\({}_{3L}\)** is the best system. Its 2-level version is the best system in the case of Yes/No QA. Leveraging SQuAD 2.0 is not helpful in both 2 (as also in Clark et al. (2019)) and 3 label settings.

In order to explore additional types of Yes/No questions, including unanswerable questions, we replicate our experiments on the small INSTRUCTIONs dataset. As before, we find a gap between the two-label and three-label scores and the usefulness of the MNLI corpus.The full results are presented in Appendix G.

## 6 Conclusion

In this paper we aim to allow a more realistic evaluation of the Yes/No QA task. For this purpose, we (i) enrich the BoolQ dataset to include unanswerable questions and (ii) compile out-of-domain test sets. Using the new data, we show the difficulty of current systems to address the task both by training directly on the task-specific data and by leveraging other NLU tasks.

### Acknowledgements

We thank Qing Lyu and the members of the Cognitive Computation Group for their insightful feedback ad well as the reviewers of the paper for their useful suggestions. This work was supported by Contracts FA8750-19-2-1004 and FA8750-19-2-0201 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. This research is also based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program. The views and conclusions contained herein are those

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Test\(\downarrow\)Train\(\rightarrow\) & BoolQ\({}_{3L}\) & MNLI + BoolQ\({}_{3L}\) & \(c(\mathrm{MNLI})\) + BoolQ\({}_{3L}\) & SQuAD 2.0 +BoolQ\({}_{3L}\) \\ \hline BooIQ\({}_{3L}\) dev & 33.64 & 42.66 & **43.25** & 35.27 \\ \hline ACE-YNQA & 52.02 & 52.02 & **54.94** & 44.15 \\ \hline \end{tabular}
\end{table}
Table 2: Accuracy of the different systems, **tested on Yes/No/IDK QA with 3 labels.** The scores correspond to average across 5 different runs. The columns represent the training strategies. The rows represent the test datasets. In all the cases the trained representation is BERT-LARGGE-CASED.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Test\(\downarrow\)Train\(\rightarrow\) & BoolQ & MNLI + BoolQ & \(c(\mathrm{MNLI})\) + BoolQ & SQuAD 2.0 + BoolQ \\ \hline BoolQ dev & 72.88 & 78.24 & **79.49** & 62.13 \\ \hline ACE-YNQA\({}_{Y/N}\) & 59.53 & 65.47 & **68.01** & 53.81 \\ \hline \end{tabular}
\end{table}
Table 3: Accuracy of the baseline system as well as the use of Extractive QA for both **Yes/No QA with 2 labels** on the BoolQ dev set an on the out-of-domain datasets, removing the IDK examples. In all the cases the trained representation is BERT-LARGGE-CASED.



of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This material is also based upon work supported by Google Cloud (TRC Program).

## References

* A. Asai and E. Choi (2021)Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 1492-1504. External Links: Link Cited by: SS3.
* L. Bentivogli, P. Clark, Ido Dagan, and D. Giampiccolo (2009)The sixth pascal recognizing textual entailment challenge. In TAC, Cited by: SS1, SS2.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Vol. 632-642. Cited by: SS1, SS2.
* E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. Zettlemoyer (2018)QuAC: question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184. Cited by: SS1, SS2.
* C. Clark and T. Gardner (2018)Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: SS1, SS2.
* C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova (2019)BoolQ: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2924--2936. Cited by: SS1, SS2.
* R. Cooper, R. Crouch, J. van Eijck, C. Fox, J. van Genabith, J. Jaspers, H. Kamp, M. Pinkal, M. Poesio, F. Pulman, and E. Vestre (1996)Describing the approaches. In The FraCaS Consortium, Cited by: SS1, SS2.
* I. Dagan, D. Roth, M. Sammons, and F. Massimo Zanzoto (2013)Recognizing Textual Entailment: Models and Applications. Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1, SS2.
* J. Dunietz, G. Burnham, A. Bharadwaj, O. Rambow, J. Chu-Carroll, and D. Ferrucci (2020)To test machine comprehension, start by defining comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7839-7859. External Links: Link Cited by: SS1, SS2.
* S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith (2018)Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112. Cited by: SS1, SS2.
* A. Kamath, R. Jia, and P. Liang (2020)Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696. Cited by: SS1, SS2.
* T. Linzen (2020)How can we accelerate progress towards human-like linguistic generalization?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5210-5217. External Links: Link Cited by: SS1, SS2.
* E. Loper and S. Bird (2002)NLTK: the Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, Cited by: SS1, SS2.
* Q. Lyu, H. Zhang, E. Sulem, and D. Roth (2021)Zero-shot event extraction via transfer learning: challenges and insights. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1, SS2.
* A. Obamyide and A. Vlachos (2018)Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), Brussels, Belgium, pp. 72-78. External Links: Link Cited by: SS1, SS2.
* J. Phang, T. Feury, and S. R. Bowman (2018) Sentence encoders on STILTs: supplementary training on intermediate labeled-data tasks. Cited by: SS1, SS2.
* P. Rajpurkar, R. Jia, and P. Liang (2018)Know what you don't know: unanswerable questions for squAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789.

 

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392.
* Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. _Transactions of the Association for Computational Linguistics_, 7:249-266.
* Sainz et al. (2022) Oscar Sainz, Itziar Gonzalez-Dios, Oier Lopez de Lacalle, Bonan Min, and Eneko Agirre. 2022. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In _Findings of the Association for Computational Linguistics: NAACL-HLT 2022_, Online and Seattle, Washington. Association for Computational Linguistics.
* Sainz et al. (2021) Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Sulem et al. (2021) Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do We Know What We Don't Know? Studying Unanswerable Questions beyond SQuAD 2.0. In _Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57.
* Wang et al. (2018) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122.
* Yatskar (2019) Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2318-2323, Minneapolis, Minnesota. Association for Computational Linguistics.

## Appendix A Hyperparameter Finetuning

For training on BoolQ\({}_{3L}\) with the BERT-LARGECASED representation, we use a batch size of 24 and a learning rate of 1e-5 and fine tune over the number of epochs (3 or 5). For training on MNLI, we use batch size of 32 and 3 training epochs. We fine tune over three possible learning rate values: 2e-5, 3e-5 and 5e-5. For training on SQuAD 2.0, we use two train epochs and fine-tune for the learning rate (3e-5 and 5e-5) and the batch size (24 and 48). For each of the training settings, we choose the hyperparameter combination that maximizes the accuracy for the target task on the dev set.

Following Clark et al. (2019), we train the system on BoolQ\({}_{3L}\) with 5 different initializations and report the average score on the dev set. We choose the checkpoint with the closest score to the average as a starting point for the following training step and for testing on the out-of-domain datasets (ACE-YNQA and INSTRUCTIONS).

## Appendix B Transfer Analysis

We also evaluate the direct transfer from MNLI to BoolQ and BoolQ\({}_{3L}\) dev set, only training for the RTE task and then experiment with intermediary finetuning on small samples (30 sentences) from the respective training corpus.

While the direct use of MNLI achieves low scores both on the BoolQ\({}_{3L}\) dev (26.73 accuracy) and the BoolQ dev (23.07), the finetuning on the sample considerably improves the performance in the two-label setting (50.43 accuracy), compared to the three-label setting (33.91 F1), showing the difficulty to transfer on the three-label setting. The results are summarized in Table 4.

## Appendix C Examples from BoolQ\({}_{3}l\)

Examples from BoolQ\({}_{3}L\) are shown in Figure 3. For example, an IDK example is created by matching **Question1** (that is associated in BoolQ to **Context1a**), to **Context1b**, which shares with the question the words "Lombardi", "trophy" and "year". While the answer to **Question1** is "Yes" when associated with **Context1a**, it is "IDK" when associated with **Context1b** since the latter does not address the creation of the trophy. Similarly, the answer to **Question2** is "No" when associated with **Context2a** but it is "IDK" when associated with **Context2b** that shares with the question the words "ball", "throw-in" and "goalkeeper" without providing the information required to answer the question 



## Appendix D Non-related IDK Examples

Examples of non-related IDK cases are shown in Figure 4. The word "season" appears both in the question (**Question1**), that is about a TV series and in the paragraph (**Context1**), that is about American football. Similarly, **Question2** and **Context2** that share the word "rate", address different topics.

## Appendix E Examples from ACE-YNQA

Examples from the ACE-YNQA dataset are presented in Figure 5.

## Appendix F Examples from INSTRUCTIONS

Examples from the INSTRUCTIONS dataset are presented in Figure 6.

## Appendix G Results on INSTRUCTIONS

The results for both 2-label and 3-label settings are presented in Section 5.

## Appendix H Yes/No/IDK QA for Extractive QA

In order to test the usefulness of the BoolQ\({}_{3}L\) dataset for other Natural Language Understanding tasks, in particular in cases where IDK answers are required, we replicate the experiment in Sulem et al. (2021), replacing MNLI by BoolQ\({}_{3L}\). In this experiment, Extractive QA systems trained on SQuAD 2.0 are tested on the out-of-domain ACE-whQA test set that includes two types of IDK questions derived from ACE. The results are presented in Table 6. They show that additional training on BoolQ\({}_{3L}\) significantly improves the baseline (where only SQuAD 2.0 is used).



* [105] M. C. Bader, A. Krizhevsky, and S. Levine. Learning to be done by using a deep neural network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-246, 2018.
* [106] J. Dosovitskiy, L. Krahenbuhl, and I. Sutskeverna. Generative adversarial nets for image recognition. In _Advances in Neural Information Processing Systems (NIPS'98)_, volume 1, pages 1-10, 2019.
* [107] R. Guezel, P. Dollar, and Z.-J. Guestrin. Deep residual learning with large margin separable convolutions. In _IEEE International Conference on Acoustics and Speech & Signal Processing (ICASSP)_, pages 1-10, 2019.
* [108] Y. Bengio. The nature of machine translation: A survey. _CoRR_, abs/1412.6050v1, 2014.
* [109] H. Chen, X. Li, and W. Dong. Unsupervised learning from tiny images. In _ECCV_, pages 131-138, 2016.
* [110] T. Deng, F. Ren, and Q. Weinberger. Very accurate representations of natural language. In _ACM SIGGRAPH international conference on Knowledge discovery and data mining_, pages 1-10, 2016.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Test\(\downarrow\)Train & SQuAD 2.0 & MNLI + SQuAD 2.0 & \multicolumn{1}{l|}{\(c(\mathrm{MNLI})\) + SQuAD 2.0} & \multicolumn{1}{l|}{BoolQ\({}_{3L}\) + SQuAD 2.0} \\ \cline{2-5}  & Has answer & 62.39 & 71.68 & 78.13 & 76.90\({}^{*}\) \\ \hline Conpent, IDK & 20.8 & **46.40\({}^{*}\)** & 26.00 & 42.40\({}^{*}\) \\ \hline non-Compet. IDK & 28.46 & **75.61\({}^{*}\)** & 47.15\({}^{*\circ}\) & 70.73\({}^{*}\) \\ \hline \end{tabular}
\end{table}
Table 6: F1 scores of the different systems, **tested on the ACE-whQA out-of-domain test set for the Extractive QA task**. In all the cases the trained representation is BERT-LARGE-CASED. “Compet. IDK” correspond to unanswerable questions with an entity of the same type as the expected answer, while it is not the case in the “non-Compet. IDK” questions. In each line the highest score is presented in bold. The scores significantly higher (using a one-sided t-test, \(p<0.05\)) than the baseline (the first column) appear with a star (\({}^{*}\)). Scores that are significantly higher than the baseline and in the same time, significantly lower than the top system, are presented with a circle (\({}^{\diamond}\)). We note that MNLI + SQuAD 2.0 significantly surpasses \(c(\mathrm{MNLI})\) + SQuAD 2.0 on “IDK” but its score is not significantly higher than that of BoolQ\({}_{3L}\) + SQuAD 2.0.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Test\(\downarrow\)Train \(\rightarrow\) & BooIQ & MNLI + BookQ & \(c(\mathrm{MNLI})\) + BoolQ & SQuAD 2.0 + BoolQ \\ \hline INSTRUCTIONS & 26.56 & **43.75** & 20.31 & 23.44 \\ \hline INSTRUCTIONS\({}_{\mathrm{Y/N}}\) & 65.00 & 65.00 & **70.00** & 65.00 \\ \hline \end{tabular}
\end{table}
Table 5: Accuracy of the baseline system as well as the use of Extractive QA and RTE for both **Yes/No/IDK QA with 3 labels** and **Yes/No QA with 2 labels** on the INSTRUCTIONS test set. In all the cases the trained representation is BERT-LARGE-CASED.





**Question:** Was there an ambush in Latifya?

**Context:** Deputy governor of Disula along with several council members from Ba'qubs were ambushed and killed in Latifys.

**Answer:** Yes

**Question:** Was the bombing in the supermarket?

**Context2:** Ba'aysir, 64, was arrested on preliminary charges of involvement in a series of church bombings on the eve of the 2000 Christmas,

**Answer2:** No

**Question3:** Were 100s of people killed?

**Context3:** Deputy governor of Dialysis along with several council members from Ba'quba were ambushed and killed in Latifysa.

**Answer3:** IDK

Figure 5: Examples of “Yes”, “No” and ‘IDK” examples from the ACE-YNQA test corpus.

Figure 6: Examples of “Yes”, “No” and ‘IDK” examples from the INSTRUCTIONS test corpus.