SlowGuess/NAACL_2022_OCR huggingface dataset:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dunietz, Greg Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and Dave Ferrucci. 2020. To test machine comprehension, start by defining comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839-7859. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112.
Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684-5696.
Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210-5217, Online. Association for Computational Linguistics.
E. Loper and S. Bird. 2002. NLTK: the Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot Event Extraction via Transfer Learning: Challenges and Insights. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL).
Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 72-78, Brussels, Belgium. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Oscar Sainz, Itziar Gonzalez-Dios, Oier Lopez de Lacalle, Bonan Min, and Eneko Agirre. 2022. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Findings of the Association for Computational Linguistics: NAACL-HLT 2022, Online and Seattle, Washington. Association for Computational Linguistics.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do We Know What We Don't Know? Studying Unanswerable Questions beyond SQuAD 2.0. In Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318-2323, Minneapolis, Minnesota. Association for Computational Linguistics.
A Hyperparameter Finetuning
For training on  $\mathrm{BoolQ}_{3L}$  with the BERT-LARGE-CASED representation, we use a batch size of 24 and a learning rate of 1e-5 and fine tune over the number of epochs (3 or 5). For training on MNLI, we use batch size of 32 and 3 training epochs. We fine tune over three possible learning rate values: 2e-5, 3e-5 and 5e-5. For training on SQuAD 2.0, we use two train epochs and fine-tune for the learning rate (3e-5 and 5e-5) and the batch size (24 and 48). For each of the training settings, we choose the hyperparameter combination that maximizes the accuracy for the target task on the dev set.
Following Clark et al. (2019), we train the system on  $\mathrm{BoolQ}_{3L}$  with 5 different initializations and report the average score on the dev set. We choose the checkpoint with the closest score to the average as a starting point for the following training step and for testing on the out-of-domain datasets (ACE-YNQA and INSTRUCTIONS).
B Transfer Analysis
We also evaluate the direct transfer from MNLI to  $\mathrm{BoolQ}$  and  $\mathrm{BoolQ}_{3L}$  dev set, only training for the RTE task and then experiment with intermediary finetuning on small samples (30 sentences) from the respective training corpus.
While the direct use of MNLI achieves low scores both on the  $\mathrm{BoolQ}_{3L}$  dev (26.73 accuracy) and the  $\mathrm{BoolQ}$  dev (23.07), the finetuning on the sample considerably improves the performance in the two-label setting (50.43 accuracy), compared to the three-label setting (33.91 F1), showing the difficulty to transfer on the three-label setting. The results are summarized in Table 4.
C Examples from  $\mathbf{BoolQ}_3L$
Examples from  $\mathrm{BoolQ}_3L$  are shown in Figure 3. For example, an IDK example is created by matching Question1 (that is associated in BoolQ to Context1a), to Context1b, which shares with the question the words "Lombardi", "trophy" and "year". While the answer to Question1 is "Yes" when associated with Context1a, it is "IDK" when associated with Context1b since the latter does not address the creation of the trophy. Similarly, the answer to Question2 is "No" when associated with Context2a but it is "IDK" when associated with Context2b that shares with the question the words "ball", "throw-in" and "goalkeeper" without providing the information required to answer the question.
Train → Test ↓	MNLI	MNLI+s(BoolQ3L)	MNLI+s(BoolQ)
3 labels	26.73	33.91	-
2 labels	23.07	-	50.43
Table 4: Accuracy of the different systems, tested either on  $\mathrm{BoolQ}_{3L}$  (first row) or  $\mathrm{BoolQ}$  with 2 labels (second row). The columns represent the training strategies.
D Non-related IDK Examples
Examples of non-related IDK cases are shown in Figure 4. The word "season" appears both in the question (Question1), that is about a TV series and in the paragraph (Context1), that is about American football. Similarly, Question2 and Context2 that share the word "rate", address different topics.
E Examples from ACE-YNQA
Examples from the ACE-YNQA dataset are presented in Figure 5.
F Examples from INSTRUCTIONS
Examples from the INSTRUCTIONS dataset are presented in Figure 6.
G Results on INSTRUCTIONS
The results for both 2-label and 3-label settings are presented in Section 5.
H Yes/No/IDK QA for Extractive QA
In order to test the usefulness of the  $\mathrm{BoolQ}_3L$  dataset for other Natural Language Understanding tasks, in particular in cases where IDK answers are required, we replicate the experiment in Sulem et al. (2021), replacing MNLI by  $\mathrm{BoolQ}_{3L}$ . In this experiment, Extractive QA systems trained on SQuAD 2.0 are tested on the out-of-domain ACE-whQA test set that includes two types of IDK questions derived from ACE. The results are presented in Table 6. They show that additional training on  $\mathrm{BoolQ}_{3L}$  significantly improves the baseline (where only SQuAD 2.0 is used).
Question1: Do they make a new Lombardi Trophy every year?
Context1a: Unlike trophies such as the Stanley Cup and the Grey Cup, a new Vince Lombardi Trophy is made every year and the winning team maintains permanent possession of that trophy, with one notable exception being Super Bowl V's, won by the then-Baltimore Colts. The city of Baltimore retained that trophy as part of the legal settlement between the team and the city after the Colts' infamous "Midnight Mayflower" move to Indianapolis on March 29, 1984. Since then, both the relocated Colts and their replacement in Baltimore, the Ravens, have won the Super Bowl and earned trophies in their own right.
Answer1a: Yes
Context1b: The Vince Lombardi Trophy is the trophy awarded each year to the winning team of the National Football League's championship game, the Super Bowl. The trophy is named in honor of NFL coach Vince Lombardi.
Answer1b: IDK
Question2: Can a goalkeeper pick up a ball from a throw in?
Context2a: Goalkeepers are normally allowed to handle the ball within their own penalty area, and once they have control of the ball in their hands opposition players may not challenge them for it. However the back-pass rule prohibits goalkeepers from handling the ball after it has been deliberately kicked to them by a team-mate, or after receiving it directly from a throw-in taken by a team-mate. Back-passes with parts of the body other than the foot, such as headers, are not prohibited. Despite the popular name `back-pass rule`, there is no requirement in the laws that the kick or throw-in must be backwards; handling by the goalkeeper is forbidden regardless of the direction the ball travels.
Answer2a: No
Context2b: There is no offside offence if a player receives the ball directly from a goal kick, a corner kick, a throw-in, or a dropped-ball. It is also not an offence if the ball was last deliberately played by an opponent (except for a deliberate save). In this context, according to the IFAB, 'A 'save' is when a player stops, or attempts to stop, a ball which is going into or very close to the goal with any part of the body except the hands/arms (unless the goalkeeper within the penalty area).
Answer2b: IDK
Figure 3: Examples from the  $\mathrm{BoolQ}_{3L}$  corpus. Question1 associated with Context1a is a “Yes” example from the original BoolQ dataset. An “IDK” example is generated by associating Question1 with Context1b. Similarly, an IDK example was generated by associating Context2b with Question2 that appeared in BoolQ in a “No” example (when associated with Context2a). In the case of IDK examples, the content words appearing both in the question and in the context are underlined.
Question1: Are they making a season seven of Once Upon a Time?
Context1: At the end of the 2015-16 season Aston Villa had spent 105 seasons in the top tier of English football; the only club to have spent longer in the top flight are Everton, with 114 seasons, making Aston Villa versus Everton the most-played fixture in English top-flight football. Aston Villa were relegated in from the top tier of English football in 2016, having played in every Premier League season since its establishment in 1992-93. They are seventh in the All-time FA Premier League table, and have the fifth highest total of major honours won by an English club with 21 wins.
Answer1: IDK
Question2: Is there a relationship between molecular weight and diffusion rate?
Context2: There is no sharp limit of development, gestational age, or weight at which a human fetus automatically becomes viable. According to studies between 2003 and 2005, 20 to 35 percent of babies born at 23 weeks of gestation survive, while 50 to 70 percent of babies born at 24 to 25 weeks, and more than 90 percent born at 26 to 27 weeks, survive. It is rare for a baby weighing less than  $500\mathrm{g}$  (17.6 ounces) to survive. A baby's chances for survival increases  $3 - 4\%$  per day between 23 and 24 weeks of gestation and about  $2 - 3\%$  per day between 24 and 26 weeks of gestation. After 26 weeks the rate of survival increases at a much slower rate because survival is high already."
Answer2: IDK
Figure 4: Non-related IDK Examples from the  $\mathrm{BoolQ}_{3L}$  corpus where the question and the context target different topics despite the content word overlap (underlined in the context).
Train → Test ↓	BoolQ	MNLI + BoolQ	c(MNLI) + BoolQ	SQuAD 2.0 + BoolQ
INSTRUCTIONS	26.56	43.75	20.31	23.44
INSTRUCTIONSY/N	65.00	65.00	70.00	65.00
Table 5: Accuracy of the baseline system as well as the use of Extractive QA and RTE for both Yes/No/IDK QA with 3 labels and Yes/No QA with 2 labels on the INSTRUCTIONS test set. In all the cases the trained representation is BERT-LARGE-CASED.
Train → Test ↓	SQuAD 2.0	MNLI + SQuAD 2.0	c(MNLI) + SQuAD 2.0	BoolQ3L + SQuAD 2.0
Has answer	62.39	71.68	78.13	76.90*°
Compet. IDK	20.8	46.40*	26.00	42.40*
non-Compet. IDK	28.46	75.61*	47.15*°	70.73*
Table 6: F1 scores of the different systems, tested on the ACE-whQA out-of-domain test set for the Extractive QA task. In all the cases the trained representation is BERT-LARGE-CASED. "Compet. IDK" correspond to unanswerable questions with an entity of the same type as the expected answer, while it is not the case in the "non-Compet. IDK" questions. In each line the highest score is presented in bold. The scores significantly higher (using a one-sided t-test,  $p < 0.05$ ) than the baseline (the first column) appear with a star (*). Scores that are significantly higher than the baseline and in the same time, significantly lower than the top system, are presented with a circle ( $^\circ$ ). We note that MNLI + SQuAD 2.0 significantly surpasses  $c(\mathrm{MNLI}) + \mathrm{SQuAD}2.0$  on "IDK" but its score is not significantly higher than that of  $\mathrm{BoolQ}_{3L} + \mathrm{SQuAD}2.0$ .
Question1: Was there an ambush in Latifya?
Context1: Deputy governor of Diyala along with several council members from Ba'quba were ambushed and killed in Latifya.
Answer1: Yes
Question2: Was the bombing in the supermarket?
Context2: Ba'asyir, 64, was arrested on preliminary charges of involvement in a series of church bombings on the eve of the 2000 Christmas.
Answer2: No
Question3: Were 100s of people killed?
Context3: Deputy governor of Diyala along with several council members from Ba'quba were ambushed and killed in Latifya.
Answer3: IDK
Figure 5: Examples of "Yes", "No" and "IDK" examples from the ACE-YNQA test corpus.
Question1: Do we need to address a specific cell or column?
Context1: Change the font size in cell C4 to 14.
Answer1: Yes
Question2: Do we need to address a specific cell or column?
Context2: Change the font color to blue.
Answer2: No
Question3: Is the font in column 4 Arial?
Context3: Change the font in column 4.
Answer3: IDK
Figure 6: Examples of "Yes", "No" and "IDK" examples from the INSTRUCTIONS test corpus.