SlowGuess/NAACL_2022_OCR huggingface dataset:
Hypothesis-only Bias Analysis We conduct analysis on hypothesis-only bias as (1) a sanity check for our converted datasets and also and (2) a verification on whether model's good performance is from leveraging artifacts in the hypotheses. We train a hypothesis-only baseline (Poliak et al., 2018b) for each phenomenon and compare their performance against the best models from the inoculation experiment. We want to ensure that models' improved performance after inoculation is due to their ability to reason about a hypothesis and the given context together. If the hypothesis-only baseline shows good performance, we interpret this as a sign that the datasets contain artifact. If the baseline shows poor performance, it gives evidence that the model is not taking short-cuts.
Cross-Distribution Generalization We conduct the cross-distribution generalization test (Rozen et al., 2019) to verify if the model learns a general reasoning skill from inoculation. The good inoculation performance does not ensure that the model's learned skill is generalizable. The model can likely over-fit the dataset distribution by adopting superficial cues. We evaluate the model's generalization ability by training and testing the model on distributions yielding different difficulty levels within the same dataset. For example, we train the model on the simple part of the dataset (data with high  $\nu$ -information) and test it on the hard part (data with low  $\nu$ -information).
For the zero-shot test, we test a model on each test set without additional fine-tuning. We select NLI models with top performance on NLI benchmarks MNLI and ANLI. We list these models in Table 3. We are interested in evaluating models with both the single-encoder and the text2text architecture. All models are publicly available from Huggingface (Wolf et al., 2019). For inoculation, we fine-tune models on training examples with a size ranging from 10 to 1000 examples per label. For the cross-distribution generalization test, we first create variant data distributions for train and test sets using the  $\nu$ -information-based dataset split method from Section 3.5. We split each dataset into two portions (simple and hard) according to the point-wise  $\nu$  information. Next, we either train and test the model on the same difficulty distribution or train it on one portion and test it on a different portion. In the inoculation, hypothesis-only, and generalization experiments, we all use roberta-anli-mix as our NLI model because its training set covers all the major NLI training datasets: SNLI, MNLI, FEVER (Thorne et al., 2018), and ANLI. We use accuracy as our evaluation metric for all these three experiments. For all the experiments excluding zero-shot test, we run several turns and select the best performance for analysis.
5.1 Zero-shot Linguistic Phenomena Diagnose
First, we report the results on zero-shot diagnostic evaluation for each baseline model. From Figure 2a, we observe that both single-encoder and text2text models trained on MultiNLI show a negative correlation in the majority of linguistic phe
(a) Zero-shot system performance on the CURRICULUM benchmark.
(b) Inoculation by fine-tuning vs. hypothesis-only analysis. The X-axis of the top plot represents training examples per label. Both plots' Y-axis show the accuracy. Models used in these two experiments are both the roberta-anli-mix model, introduced in Section 4.1.
nomena. Meanwhile, anli-mix models (roberta-anli-mix, xlnet-anli-mix) are positively correlated on most  $(77.8\%)$  of the phenomena and they show high correlation  $(>0.50)$  on  $27.8\%$  of the phenomena. On average, models trained on the large dataset mixture show better performance than models trained on MultiNLI alone, suggesting that training on more datasets help models capture more types of linguistic phenomena. However, most of the phenomena captured by the anli-mix models are easier to learn (higher  $\mathcal{V}$  information). On harder phenomena, models did not benefit from the training dataset mixture. For instance, both the anli-mix models have a low correlation on deductive and analytical reasoning. Overall, we find that NLI
datasets from common benchmarks lack examples of a diverse set of reasoning skills.
5.2 Inoculation
Based on Figure 2b, the model can reach high accuracy on about  $64\%$  of the phenomena as the training examples accumulate. Most of these phenomena have higher  $\nu$  information ( $>0.0$ ) that should relatively be easier to learn. We are surprised that for some hard phenomena ( $\leq 0.0$ ) such as commonsense contextual reasoning (cosmo, -0.67), the model's performance improved after inoculation. The improvement shows an gap in the original training data mixture. On  $25\%$  of the phenomena, the model's performance did not improve
Figure 3: Generalization between controlled dataset splits. Here each heat-map shows the generalization performance of the model fine-tuned and evaluated on different distributions within each linguistic phenomenon.
significantly after inoculation, meaning that it fails to learn the reasoning skills for these phenomena. Most of these phenomena are difficult, with a low  $\mathcal{V}$  information, such as monotonicity(mono) and deductive (logi) reasoning. The accuracy is consistently low when training examples accumulate.
We also observe that model struggles to learn phenomena that require complex reasoning, such as phenomena from the comprehension category. This trends show inherent weaknesses in the model or its training strategy that cause its failure to learn complex and hard phenomena. Overall, results from this experiment, combined with the zero-shot evaluation, suggest that many linguistic phenomena are missing from different large-scale NLI datasets but are recoverable through additional training examples. However, the model fails to learn the skills for hard and complex phenomena. In summary, our diagnostic study through inoculation exposes a diverse set of dataset and model weaknesses.
5.3 Hypothesis-only Bias
To determine if models can leverage spurious artifacts in the hypotheses of each phenomenon, we compare full models to hypothesis-only baselines. From Figure 2b, we observe that hypothesis-only baseline performs poorly on a majority of the phenomena. This indicates that our benchmark generally requires the model to learn an inference process between contexts and hypotheses for good performance. We observe that on  $30.6\%$  of the phenomena, the full-model can reach a high accuracy
while the baseline has low accuracy, suggesting the model can learn the phenomenon without relying on hypothesis artifacts. On  $36\%$  of the phenomena, the model does not show a significant performance gain compared to the baseline. Most of these are complex reasoning phenomena like deductive and analytical reasoning. The result validates that the model struggles more with complex linguistic phenomena. On  $33.3\%$  of the phenomena, both the full-model and the baseline achieve high accuracy showing the possibility that the model exploits artifacts from the hypothesis to reach high accuracy.
Also, note that the hypothesis-only baseline performs better for some tasks than the fine-tuned model, which can be interpreted in two ways. When both the baseline and fine-tuned model achieve high accuracy (vbc, syn-alt), higher accuracy on baseline indicates that the hypothesis-only bias is pretty strong in the dataset. When the intervention from the premise is removed (hypothesis-only input), the models can easily exploit the bias to achieve higher accuracy. In contrast, when both the baseline and fine-tuned model achieve low accuracy (hypo, analytic, social, ester), higher accuracy on baseline indicates that the task is very difficult for a model to master successfully. Low baseline accuracy means that the dataset does not contain much bias, so a model must learn the correct reasoning to perform well. However, the fine-tuned model has even worse performance than the baseline, meaning that it fails to learn the skill required for these tasks. Our main finding here is
that good performance on a linguistic phenomenon dataset does not mean the model captured the associated phenomena. The model can learn short-cuts through hypothesis-only bias and artifacts.
5.4 Generalization
As Figure 3 show, the model can adapt between different distributions only on  $22.2\%$  of the phenomena. The model achieves high accuracy consistently for all four categories in the generalization matrix suggesting the learned skills are generalizable. On  $58.3\%$  phenomena, models can not generalize between different difficulty distributions. They show higher accuracy when trained and tested on the same distribution but low accuracy when the test distribution shifted. For example, on relational knowledge reasoning (kg-rel), the model achieves  $83\%$  for simple  $\rightarrow$  simple and  $98\%$  for hard  $\rightarrow$  hard. Nevertheless, the performance drops to  $53\%$  for hard  $\rightarrow$  simple and  $38\%$  for simple  $\rightarrow$  hard.
Notice that model's good performance on inoculation does not align with its generalization ability. For example, the model reaches  $90.9\%$  accuracy on kg-rel, but its generalization performance is poor. This behavior highlights a model weakness: can over-fit to a particular distribution but fail to learn a general reasoning skill for the target phenomenon. We observe an interesting behavior that models struggle to generalize from hard to simple distribution on about  $14\%$  of the phenomena while showing good generalization from simple to hard distribution. We think the possible reason is that the hard distribution contains data with relatively low  $\mathcal{V}$  information. A low amount of usable information makes it hard for the model to learn the phenomena sufficiently for generalization.
In this paper, we introduce a new form of benchmark that can serve as an effective tool for evaluating and analyzing model outcomes. We propose a linguistic-phenomena-driven benchmark that aims to diagnose neural language models to discover types of linguistic skills that remain challenging to models. We compiled a dataset collection covering 36 types of linguistic phenomena ranging from fundamental linguistic properties to complex reasoning skills. In addition, we define an evaluation procedure that can provide an in-depth analysis of model and dataset weaknesses. Using our benchmark, we comprehensively study how well
language models capture specific linguistic skills essential for understanding. Our major findings include:
- Models trained on benchmark NLI datasets fail to reason over a diverse set of linguistic phenomena.
- Good inoculation performance on some phenomena results from the model leveraging superficial artifacts in the hypothesis.
- The model tends to over-fit the dataset distribution without learning a general reasoning skill on a majority of phenomena.
Overall, our benchmark effectively evaluates a model on specific linguistic skills and exposes a list of model and training data weaknesses. We hope that our benchmark and empirical findings can encourage the communicate to rethink dataset construction and model architecture design. In particular, we hope to encourage the development of new datasets that cover richer types of linguistic phenomena and language models that can learn essential linguistic skills that are generalizable. For future work, we plan to add more datasets to cover more phenomena such as psycholinguistics (Laverghetta Jr. et al., 2021). We envision our benchmark to be dynamic, meaning that a higher-quality and more difficult dataset for a phenomenon should replace the current ones in the future. For example, the StepGame benchmark (Shi et al., 2022) provides better data for spatial reasoning, which can replace the current spatial reasoning dataset. We also plan to explore new learning methods to help models overcome the weakness of learning non-generalizable skills, such as calibration through symbolic loss functions.
We thank the anonymous reviewers for their thoughtful and constructive comments. We thank Kyle Richardson from AI2 for his insights and suggestions on improving our camera-ready version. Thanks also to our advisors Laurence S. Moss and Michael Wollowski for their feedback on earlier drafts of this work. Special thanks to the Machine Learning for Language Group at NYU for their wonderful NLP toolkit, JIANT (Phang et al., 2020).
BIG-bench collaboration. 2021. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1078-1088. PMLR.
Zeming Chen and Qiyue Gao. 2021. Probing linguistic information for logical inference in pre-trained language models.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2021. Information-theoretic measures of dataset difficulty.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650-655, Melbourne, Australia. Association for Computational Linguistics.
Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, and Nanyun Peng. 2021. Ester: A machine reading comprehension dataset for event semantic relation reasoning.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41-55, Online. Association for Computational Linguistics.
Giuseppe Jurman, Samantha Riccadonna, and Cesare Furlanello. 2012. A comparison of mcc and cen error measures in multi-class prediction. PLOS ONE, 7(8):1-8.
Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R. Bowman. 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 287-297.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Antonio Laverghetta Jr., Animesh Nighojkar, Jamshidbek Mirzakhalov, and John Licato. 2021. Can transformer language models predict psychometric properties? In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics*, pages 12-25, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2021. Natural language inference in context - investigating contextual reasoning over long texts. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13388-13396. AAAI Press.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.
Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019a. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2171-2179, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Rajaswa Patil and Veeky Baths. 2020. CNRL at SemEval-2020 task 5: Modelling causal reasoning in language with multi-head self-attention weights based counterfactual detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 451-457, Barcelona (online). International Committee for Computational Linguistics.
Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Ian F. Tenney, Phu Mon Htut, Clara Vania, Alex Wang, and Samuel R. Bowman. 2020. jiant 2.0: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018a. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67-81, Brussels, Belgium. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018b. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Inioluwa Deborah Raji, Emily M. Bender, Amanda Lynne Paullada, Emily Denton, and Alex Hanna. 2021. Ai and the everything in the whole wide world benchmark.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad.
Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2019. Probing natural language inference models through semantic fragments.
Kyle Richardson and Ashish Sabharwal. 2020. What does my QA model know? devising controlled probes using expert knowledge. Transactions of the Association for Computational Linguistics, 8:572-588.
Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196-205, Hong Kong, China. Association for Computational Linguistics.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4661-4675, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale.
Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2018. ATOMIC: an atlas of machine commonsense for if then reasoning. CoRR, abs/1811.00146.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China. Association for Computational Linguistics.
Martin Schmitt and Hinrich Schütze. 2021. Language models for lexical inference in context. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1267-1280, Online. Association for Computational Linguistics.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts.
Ishan Tarunesh, Somak Aditya, and Monojit Choudhury. 2021. Lonli: An extensible framework for testing diverse logical reasoning capabilities for nli.