SlowGuess/ACL_2020_OCR huggingface dataset:
Does our dataset exhibit hypothesis bias? Before we consider the question of whether we can model premise-hypothesis relationships, let us first see if a model can learn to predict the entailment label without using the premise, thereby exhibiting an undesirable artifact. We consider three classes of models to study hypothesis bias in INFO TABS. Hypothesis Only (hypo-only): The simplest way to check for hypothesis bias is to train a classifier using only the hypotheses. Without a premise, a classifier should fail to correlate the hypothesis and the label. We represent the hypothesis in two ways: a) using unigrams and bigrams for an SVM, and b) using a single-sentence BERT-class model. The results of the experiments are given in Table 3.
Model	Dev	α1	α2	α3
Majority	33.33	33.33	33.33	33.33
SVM	59.00	60.61	45.89	45.89
BERTB	62.69	63.45	49.65	50.45
RoBERTaB	62.37	62.76	50.65	50.8
RoBERTaL	60.51	60.48	48.26	48.89
Dummy or Swapped Premise: Another approach to evaluate hypothesis bias is to provide an unrelated premise and train a full entailment model. We evaluated two cases, where every premise is changed to a (a) dummy statement (to be or not to be), or (b) a randomly swapped table that is represented as paragraph. In both cases, we trained a RoBERTa $_L$  classifier as described in §6.2. The results for these experiments are presented in Table 4.
Table 3: Accuracy of hypothesis-only baselines on the INFOTABS Dev and test sets
Premise	Dev	α1	α2	α3
dummy	60.02	59.78	48.91	46.37
swapped	62.94	65.11	52.55	50.21
Results and Analysis: Looking at the Dev and  $\alpha_{1}$  columns of Tables 3 and 4, we see that these splits do have hypothesis bias. All the BERT-class models discover such artifacts equally well. However, we also observe that the performance on  $\alpha_{2}$  and  $\alpha_{3}$  data splits is worse since the artifacts in the training data do not occur in these splits. We see a performance gap of  $\sim 12\%$  as compared to Dev and  $\alpha_{1}$  splits in all cases. While there is some hypothesis bias in these splits, it is much less pronounced.
An important conclusion from these results is that the baseline for all future models trained on these splits should be the best premise-free performance. From the results here, these correspond to the swapped setting.
How do trained NLI systems perform on our dataset? Given the high leaderboard accuracies of trained NLI systems, the question of whether these models can infer entailment labels using a linearization of the tables arises. To study this, we trained  $\mathrm{RoBERTa}_L$  models on the SNLI and MultiNLI datasets. The SNLI model achieves an accuracy of  $92.56\%$  on SNLI test set. The
MultiNLI model achieves an accuracy of  $89.0\%$  on matched and  $88.99\%$  on the mismatched MultiNLI test set. We evaluate these models on the WMD-1 and the Para representations of premises.
Table 4: Accuracy with dummy/swapped premises
Premise	Dev	α1	α2	α3
Trained on SNLI
WMD-1	49.44	47.5	49.44	46.44
Para	54.44	53.55	53.66	46.01
Trained on MultiNLI
WMD-1	44.44	44.67	46.88	44.01
Para	55.77	53.83	55.33	47.28
Results and Analysis: In Table 5, all the results point to the fact that pre-trained NLI systems do not perform well when tested on INFOTABS. We observe that full premises slightly improve performance over the WMD-1 ones. This might be due to a) ineffectiveness of WMD to identify the correct premise sentence, and b) multi-row reasoning.
Does training on the paragraph/sentence representation of a premise help? The next set of experiments compares BERT-class models and SVM trained using the paragraph (Para) and sentence (WMD-n) representations. The results for these experiments are presented in Table 6.
Table 5: Accuracy of test splits with structured representation of premises with RoBERTaL trained on SNLI and MultiNLI training data
Premise	Dev	α1	α2	α3
Para	Train with SVM
59.11	59.17	46.44	41.28
Para	Train with BERTB
63.00	63.54	52.57	48.17
Para	Train with RoBERTaB
67.2	66.98	56.87	55.36
WMD-1	Train with RoBERTaL
65.44	65.27	57.11	52.55
WMD-3	72.55	70.38	62.55	61.33
Para	75.55	74.88	65.55	64.94
Table 6: Accuracy of paragraph and sentence premise representation reported on SVM, BERT $_B$ , RoBERT $_A$ , and RoBERT $_L$
Results and Analysis: We find that training with the INFOTABS training set improves model performance significantly over the previous baselines,
except for the simple SVM model which relies on unigrams and bigrams. We see that  $\mathrm{RoBERTa}_L$  outperforms its base variant and  $\mathrm{BERT}_B$  by around  $\sim 9\%$  and  $\sim 14\%$  respectively. Similar to the earlier observation, providing full premise is better than selecting a subset of sentences.
Importantly,  $\alpha_{2}$  and  $\alpha_{3}$  performance is worse than  $\alpha_{1}$ , not only suggesting the difficulty of these data splits, but also showing that models overfit both lexical patterns (based on  $\alpha_{2}$ ) or domain-specific patterns (based on  $\alpha_{3}$ ).
Does training on premise encoded as structure help? Rather than linearizing the tables as sentences, we can try to encode the structure of the tables. We consider two representative approaches for this, TabFact and TabAttn, each associated with a different model as described in §6.2. The results for these experiments are listed in Table 7.
Premise	Dev	α1	α2	α3
Train with BERTB
TabFact	63.67	64.04	53.59	49.05
Train with RoBERTB
TabFact	68.06	66.7	56.87	55.26
Train with RoBERTAL
TabAttn	63.63	62.94	49.37	49.04
TabFact	77.61	75.06	69.02	64.61
Table 7: Accuracy on structured premise representation reported on  $\mathrm{BERT}_B$ , RoBERTa  $B$  and RoBERTa  $L$
Results and Analysis: The idea of using this family of models was to leverage the structural aspects of our data. We find that the TabAttn model, however, does not improve the performance. We assume that this might be due to the bag of words style of representation that the classifier employs. We find, however, that providing premise structure information helps the TabFact model perform better than the RoBERTa  $L + \mathrm{Para}$  model. As before model performance drops for  $\alpha_{2}$  and  $\alpha_{3}$ .
How many types of reasoning does a trained system predict correctly? Using a RoBERTa $_L$ , which was trained on the paragraph (Para) representation, we analyzed the examples in Dev and  $\alpha_3$  data splits that were annotated by experts for their types of reasoning ( $\S 5$ ). Figure 3 shows the summary of this analysis.
Results and Analysis: Figures 3a and 3b show the histogram of reasoning types among correctly
predicted examples. Compared to Figures 2a and 2b, we see a decrease in correct predictions across all reasoning types for both Dev and  $\alpha_{3}$  sets. In particular, in the Dev set, the model performs poorly for the knowledge & common sense, multi-row, coreference, and temporal reasoning categories.
Discussion Our results show that: 1) INFOTABS contains a certain amount of artifacts which transformer-based models learn, but all models have a large gap to human performance; and 2) models accuracies drop on  $\alpha_{2}$  and  $\alpha_{3}$ , suggesting that all three results together should be used to characterize the model, and not any single one of them. All our models are significantly worse than the human performance (84.04%, 83.88% and 79.33% for  $\alpha_{1}$ ,  $\alpha_{2}$  and  $\alpha_{3}$  respectively). With a difference of  $\sim 14\%$  between our best model and the human performance, these results indicate that INFOTABS is a challenging dataset.
NLIDatasets Natural language inference/textual entailment is a well studied text understanding task, and has several datasets of various sizes. The annual PASCAL RTE challenges (Dagan et al., 2005, inter alia) were associated with several thousands of human-annotated entailment pairs. The SNLI dataset (Bowman et al., 2015) is the first large scale entailment dataset that uses image captions as premises, while the MultiNLI (Williams et al., 2018) uses premises from multiple domains. The QNLI and WNLI datasets provide a new perspective by converting the SQuAD question answering data (Rajpurkar et al., 2016) and Winograd Schema Challenge data (Levesque et al., 2012) respectively into inference tasks. More recently, SciTail (Khot et al., 2018) and Adversarial NLI (Nie et al., 2019) have focused on building adversarial datasets; the former uses information retrieval to select adversarial premises, while the latter uses iterative annotation cycles to confuse models.
Reasoning Recently, challenging new datasets have emerged that emphasize complex reasoning. Bhagavatula et al. (2020) pose the task of determining the most plausible inferences based on observation (abductive reasoning). Across NLP, a lot of work has been published around different kinds of reasonings. To name a few, common sense (Talmor et al., 2019), temporal (Zhou et al., 2019), numerical (Naik et al., 2019; Wallace et al., 2019b) and
(a) Number of correct predictions per reasoning type in the Dev set
(b) Number of correct predictions per reasoning type in the  $\alpha_{3}$  test set
Figure 3: Number of correct predictions per reasoning type in the Dev and  $\alpha_{3}$  splits.
multi-hop (Khashabi et al., 2018) reasoning have all garnered immense research interest.
Tables and Semi-structured data Tasks based on semi-structured data in the form of tables, graphs and databases (with entries as text) contain complex reasoning (Dhingra et al., 2019; Chen et al., 2020). Previous work has touched upon semantic parsing and question answering (e.g., Pasupat and Liang, 2015; Khashabi et al., 2016, and references therein), which typically work with tables with many entries that resemble database records.
Our work is most closely related to TabFact (Chen et al., 2020), which considers database-style tables as premises with human-annotated hypotheses to form an inference task. While there are similarities in the task formulation scheme, our work presents an orthogonal perspective: (i) The Wikipedia tables premises of TabFact are homogeneous, i.e., each column in a table has structural redundancy and all entries have the same type. One can look at multiple entries of a column to infer extra information, e.g., all entries of a column are about locations. On the contrary, the premises in our dataset are heterogeneous. (ii) TabFact only considers entailment and contradiction; we argue that inference is non-binary with a third "undetermined" class (neutrals). (iii) Compared to our multi-faceted reasonings, the reasonings of the hypotheses in TabFact are limited and mostly numerical or comparatives. (iv) The  $\alpha_{2}$  and  $\alpha_{3}$  sets help us check for annotation and domain-specific artifacts.
Artifacts Recently, pre-trained transformer-based models (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019, and others) have seemingly outperformed human performance on several NLI tasks (Wang et al., 2019b,a). However, it has
been shown by Poliak et al. (2018); Niven and Kao (2019); Gururangan et al. (2018); Glockner et al. (2018); Naik et al. (2018); Wallace et al. (2019a) that these models exploit spurious patterns (artifacts) in the data to obtain good performance. It is imperative to produce datasets that allow for controlled study of artifacts. A popular strategy today is to use adversarial annotation (Zellers et al., 2018; Nie et al., 2019) and rewriting of the input (Chen et al., 2020). We argue that we can systematically construct test sets that can help study artifacts along specific dimensions.
We presented a new high quality natural language inference dataset, INFOTABS, with heterogeneous semi-structured premises and natural language hypotheses. Our analysis showed that our data encompasses several different kinds of inferences. INFOTABS has multiple test sets that are designed to pose difficulties to models that only learn superficial correlations between inputs and the labels, rather than reasoning about the information. Via extensive experiments, we showed that derivatives of several popular classes of models find this new inference task challenging. We expect that the dataset can serve as a testbed for developing new kinds of models and representations that can handle semistructured information as first class citizens.
We thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. We acknowledge the support of the support of NSF Grants No. 1822877 and 1801446, and a generous gift from Google.
Ron Artstein and Massimo Poesio. 2008. Inter-coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555-596.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. In International Conference on Learning Representations.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. TabFact: A Large-scale Dataset for Table-based Fact Verification. In International Conference on Learning Representations.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling Divergent Reference Texts when Evaluating Table-to-Text Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question Answering via Integer Programming over Semi-structured Knowledge. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A Textual Entailment Dataset from Science Question Answering. In Association for the Advancement of Artificial Intelligence.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From Word Embeddings to Document Distances. In International Conference on Machine Learning.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring Numeracy in Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A New Benchmark for Natural Language Understanding. arXiv preprint arXiv:1910.14599.
Timothy Niven and Hung-Yu Kao. 2019. Probing Neural Network Comprehension of Natural Language Arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI Blog.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019b. Do NLP Models Know Numbers? Probing Numeracy in Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b. GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's Transformers: State-of-the-art Natural Language Processing. ArXiv.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Common-sense Understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
A Examples of Data
Figure 4 shows two additional examples of table premises and their corresponding hypotheses available in the development set of INFOTABS.
Kamloops
Type	Elected city council
Mayor	Ken Christian
Governing body	Kamloops City Council
MP	Cathy McLeod
MLAs	Peter Milobar, Todd Stone
H1: Kamloops has a democracy structure.
H2: If Ken Christian resigns as Mayor of Kamloops then Cathy McLeod will most likely replace him.
H3: Kamloops is ruled by a president.
Jefferson Starship
Origin	San Francisco California
Genres	Rock, hard rock, psychedelic rock, progressive rock, soft rock
Years active	1970 - 1984, 1992 - present
Labels	RCA Grunt Epic
Associated acts	Jefferson Airplane Starship, KBC Band, Hot Tuna
Website	www.jeffersonstarship.net
H1: Jefferson Starship was started on the West Coast of the United States.
H2: Jefferson Starship won many awards for its music.
H3: Jefferson Starship has performed continuously since the 1970s.
Figure 4: Two semi-structured premises (the tables), and three hypotheses (H1: entailment, H2: Neutral, and H3: contradiction) that correspond to each table.
B Reasoning for INFOTABS
Our inventory of reasoning types is based on GLUE diagnostics (Wang et al., 2019b), but is specialized to the problem of reasoning about tables. Consequently, some categories from GLUE diagnostics may not be represented here, or may be merged into one category.
We assume that the table is correct and complete. The former is always true for textual entailment, where we assume that the premise is correct. The latter need not be generally true. However, in our analysis, we assume that the table lists all the relevant information for a field. For example, in a table for a music group as in Figure 4, if there is a row called Labels, we will assume that the labels listed in that row are the only labels associated with the group.
Note that a single premise-hypothesis pair may be associated with multiple types of reasoning. If the same reasoning type is employed multiple times in the same pair, we only mark it once.
Simple lookup This is the simple case where there is no reasoning, and the hypothesis is formed by literally restating information in the table. For example, using the table in Figure 5, *Femme aux Bras Croisés* is privately held. is a simple lookup.
Multi-row reasoning Multiple rows in the table are needed to make an inference. This has the strong requirement that without multiple rows, there is no way to arrive at the conclusion. Exclude instances where multiple rows are used only to identify the type of the entity, which is then used to make an inference. The test for multi-row reasoning is: If a row is removed from the table, then the label for the hypothesis may change.
Entity type Involves ascertaining the type of an entity in question (perhaps using multiple rows from the table), and then using this information to make an inference about the entity.
This is separate from multi-row reasoning even if discovering the entity type might require reading multiple rows in the table. The difference is a practical one: we want to identify how many inferences in the data require multiple rows (both keys and values) separately from the ones that just use information about the entity type. We need to be able to identify an entity and its type separately to decide on this category. In addition, while multi-row reasoning, by definition, needs multiple rows, entity
Femme aux Bras Croisés
Artist	Pablo Picasso
Year	1901-02
Medium	Oil on canvas
Dimensions	81 cm 58 cm (32 in 23 in)
Location	Privately held
Figure 5: An example premise