SlowGuess/NAACL_2022_OCR huggingface dataset:
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. Superglue: A stickier benchmark for general-purpose language understanding systems.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv: Artificial Intelligence.
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996-1005, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31-40, Florence, Italy. Association for Computational Linguistics.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 250-255, Minneapolis, Minnesota. Association for Computational Linguistics.
Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. 2021. Exploring transitivity in neural NLI models through veridicality. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 920-934, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2021. Ar-Isat: Investigating analytical reasoning of text.
A Linguistic Phenomena in CURRICULUM
Phenomena	Train Reference	Test Reference
Lexical Phenomena
Lexical Entailment	Schmitt and Schütze 2021	Schmitt and Schütze 2021; Glockner et al. 2018
Hypernymy	Richardson and Sabharwal 2020	Richardson and Sabharwal 2020
Hyponymy	Richardson and Sabharwal 2020	Richardson and Sabharwal 2020
Named Entity	Poliak et al. 2018a	Poliak et al. 2018a
Veridicality and Transitivity	Poliak et al. 2018a; Yanaka et al. 2021	Poliak et al. 2018a; Yanaka et al. 2021
Syntactic Phenomena
VerbNet	Poliak et al. 2018a	Poliak et al. 2018a
VerbCorner	Poliak et al. 2018a	Poliak et al. 2018a
Syntactic Variation	Dolan and Brockett 2005	Dolan and Brockett 2005
Syntactic Alternations	Kann et al. 2019	Kann et al. 2019
Semantic Phenomena
Coreference & Anaphora	Sakaguchi et al. 2019; Wang et al. 2019	Sakaguchi et al. 2019; Wang et al. 2019
Webster et al. 2018	Webster et al. 2018
Sentiment	Poliak et al. 2018a	Poliak et al. 2018a
Relational Knowledge	Poliak et al. 2018a	Poliak et al. 2018a
Puns	Poliak et al. 2018a	Poliak et al. 2018a
Semantic Proto Label	White et al. 2017	White et al. 2017
Context Alignment	White et al. 2017	White et al. 2017; BIG-bench collaboration 2021
Logical Phenomena
Boolean	Richardson et al. 2019	Richardson et al. 2019
Conditional	Richardson et al. 2019	Richardson et al. 2019
Comparative	Richardson et al. 2019	Richardson et al. 2019
Counting	Richardson et al. 2019	Richardson et al. 2019
Quantifier	Richardson et al. 2019	Richardson et al. 2019
Negation	Richardson et al. 2019	Richardson et al. 2019
Monotonicity	Yanaka et al. 2019b	Yanaka et al. 2019a; Richardson et al. 2019
Analytic Phenomena
Entailment Tree	Dalvi et al. 2021	Dalvi et al. 2021
Analytical Reasoning	Zhong et al. 2021	Zhong et al. 2021
Commonsense Phenomena
Physical	Bisk et al. 2019	Bisk et al. 2019
Social	Sap et al. 2019	Sap et al. 2019
HellaSwag	Sap et al. 2018	Sap et al. 2018
Contextual Commonsense Reasoning	Huang et al. 2019	Huang et al. 2019
Comprehension Phenomena
Deductive Reasoning	Liu et al. 2020	Liu et al. 2020
Contextual Reasoning	Liu et al. 2021	Liu et al. 2021
Event Semantic Reasoning	Han et al. 2021	Han et al. 2021
Discrete Reasoning	Dua et al. 2019	Dua et al. 2019
Special Reasoning Phenomena
Defeasible Reasoning	Rudinger et al. 2020	Rudinger et al. 2020
Temporal Reasoning	Weston et al. 2016	Weston et al. 2016
Spatio Reasoning	Weston et al. 2016	Weston et al. 2016
Counterfactual Reasoning	Patil and Baths 2020	Patil and Baths 2020
Table 4: A detailed list of training datasets and test datasets used for each linguistic phenomenon in our benchmark.
B CURRICULUM Dataset Details in CURRICULUM
Name	|Train|	|Dev|	Original task
Lexical Entailment	6398	2964	NLI
Hypernymy	20000	8500	QA
Hyponymy	20000	8500	QA
Named Entity	50000	30000	NLI
Veridicality and Transitivity	20000	8788	NLI
VerbNet	1398	160	NLI
VerbCorner	110898	13894	NLI
Syntactic Variation	3668	408	SC
Syntactic Alternations	19990	8739	SC
Coreference & Anaphora	12135	5799	NLI/SC
Sentiment	4800	600	NLI
Relational Knowledge	21905	761	NLI
Semantic Proto Label	14038	1756	NLI
Puns	14038	1756	NLI
Context Align	14038	1756	NLI
Boolean	3000	1000	NLI
Conditional	3000	1000	NLI
Comparative	3000	1000	NLI
Counting	3000	1000	NLI
Quantifier	3000	1000	NLI
Negation QA	3000	1000	NLI
Monotonicity	35891	5382	NLI
Entailment Tree	1314	340	TG
Analytical Reasoning	3260	922	SC
Physical	10000	1838	QA
Social	6003	6003	QA
HellaSwag	20000	8518	QA
Contextual Commonsense Reasoning	9046	5452	RC
Deductive Reasoning	14752	2604	RC
Contextual Reasoning	6719	1604	RC
Event Semantics Reasoning	2800	662	RC
Discrete Reasoning	20000	13148	RC
Defeasible Reasoning	39036	9860	SC
Temporal Reasoning	4248	1174	NLI
Spatial Reasoning	10000	10000	QA
Counterfactual Reasoning	6062	3364	SC
Table 5: Overview of all the linguistic phenomena datasets in our benchmark. QA is short for Question Answering. NLI is short for Natural Language Inference. SC is short for Sentence Classification. TG is short for Text Generation. RC is short for Reading Comprehension.
C Data Recasting Details
Here we provide more details on the major techniques we used to convert Question Answering (QA) and Reading Comprehension (RC) datasets into recast NLI datasets.
C.1 Entity Swapping
Context: ...The Buccaneers tied it up with a 38-yard field goal by Connor Barth, ... The game's final points came when Mike Williams of Tampa Bay caught a 5-yard pass... Q: Who caught the touchdown for the fewest yard? Answer: Mike Williams
<Recast>
Premise: ...The Buccaneers tied it up with a 38-yard field goal by Connor Barth, ... The game's final points came when Mike Williams of Tampa Bay caught a 5-yard pass... Hypothesis: Mike Williams caught the touchdown for the fewest yard Label: Entailed Hypothesis: Connor Barth caught the touchdown for the fewest yard Label: Not-Entailed
Table 6: Example of converting an RC example from DROP (Dua et al., 2019) to NLI format. The entailed hypothesis is a concatenation of question and answer. The non-entailed hypothesis is created by entity swapping on the entailed one (Mike Williams → Connor Barth).
C.2 Question/Answer Concatenation
Context: The flash in the room that followed was proof of that assumption. The man grabbed his arm again. "Please let go of my arm." He requested, his voice low. "Look."
Q: Why did the man grabbed his arm?
Choice 1: The man wanted to dance with him.
Choice 2: The man wanted to get his attention.
Choice 3: The man wanted to pull him closer so he can cry on this shoulder.
Choice 4: The man was angry with him and wanted to push him outside.
Premise: The flash in the room that followed was proof of that assumption. The man grabbed his arm again. "Please let go of my arm." He requested, his voice low. "Look."
Hypothesis: The man wanted to get his attention.
Label: Entailed
Hypothesis: The man wanted to dance with him.
Label: Not-Entailed
Table 7: Example of converting an QA example from Cosmos QA (Huang et al., 2019) to NLI format. The entailed hypothesis is the correct answer from the given choices. The non-entailed hypothesis is one of the false answers, excluding the choice "None of the above choices".
D Reproducibility
Implementation. Our model training and testing pipeline is modified from the JIANT toolkit. We mainly adapted several components on classes and functions involving task, dataset, reprocessing, tokenization, model version control, and evaluation metrics. All our experiments are implemented with models publicly available from Huggingface Transformers (Wolf et al., 2020)<sup>2</sup>.
Hyper-parameters We mainly follow the practice in (Nie et al., 2020). For all the experiments excluding the zero-shot test in Section 5.1, we use a learning rate of  $1e - 5$  with a batch size of 8. We set the number of warmup updates to be 1000. We set the epoch number to be 3 and 5. We evaluate the model on  $D_{dev}$  every 200 steps for the inoculation and generalization experiments, and 500 steps for the hypothesis-only experiment. For the low-data generalization on ANLI, we evaluate on the full-test set according to the number of training examples listed in Figure ??. We use the AdamW (Loshchilov and Hutter, 2019) as our optimizer.
Infrastructure All experiments are done with one single Geforce RTX 3090 (24GB). A single inoculation or generalization job finishes within 0.5 hours on average. A single hypothesis-only job finishes within 1-2 hours on average. A single job on sequential training and low-data fine-tuning finishes within approximately 1.5 hours on average.
Number of Parameters. RoBERTa-large model contains 355 million parameters. BART-large model contains 139 million parameters. BART-Large model contains 406 million parameters. XLNet-large model contains 340 million parameters.