SlowGuess/ACL_2020_OCR huggingface dataset:
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, Vivek Srikumar
School of Computing, University of Utah
{vgupta,maitrey,pnokhiz,svivek}@cs.utah.edu
In this paper, we observe that semi-structured tabulated text is ubiquitous; understanding them requires not only comprehending the meaning of text fragments, but also implicit relationships between them. We argue that such data can prove as a testing ground for understanding how we reason about information. To study this, we introduce a new dataset called INFOTABS, comprising of human-written textual hypotheses based on premises that are tables extracted from Wikipedia info-boxes. Our analysis shows that the semi-structured, multi-domain and heterogeneous nature of the premises admits complex, multi-faceted reasoning. Experiments reveal that, while human annotators agree on the relationships between a table-hypothesis pair, several standard modeling strategies are unsuccessful at the task, suggesting that reasoning about tables can pose a difficult modeling challenge.
Recent progress in text understanding has been driven by sophisticated neural networks based on contextual embeddings—e.g., BERT (Devlin et al., 2019), and its descendants—trained on massive datasets, such as SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and SQuAD (Rajpurkar et al., 2016). Several such models outperform human baselines on these tasks on the benchmark suites such as GLUE (Wang et al., 2019b). Reasoning about text requires a broad array of skills—making lexical inferences, interpreting the nuances of time and locations, and accounting for world knowledge and common sense. Have we achieved human-parity across such a diverse collection of reasoning skills?
In this paper, we study this question by proposing an extension of the natural language inference (NLI) task (Dagan et al., 2005, and others). In
Dressage
Highest governing body	International Federation for Equestrian Sports (FEI)
Characteristics
Contact	No
Team members	Individual and team at international levels
Mixed gender	Yes
Equipment	Horse, horse tack
Venue	Arena, indoor or outdoor
Presence
Country or region	Worldwide
Olympic	1912
Paralympic	1996
H1: Dressage was introduced in the Olympic games in 1912.
H2: Both men and women compete in the equestrian sport of Dressage.
H3: A dressage athlete can participate in both individual and team events.
H4: FEI governs dressage only in the U.S.
Figure 1: A semi-structured premise (the table). Two hypotheses (H1, H2) are entailed by it, H3 is neither entailed nor contradictory, and H4 is a contradiction.
NLI, which asks whether a premise entails, contradicts or is unrelated to a hypothesis, the premise and the hypothesis are one or more sentences. Understanding the premise requires understanding its linguistic structure and reasoning about it. We seek to separate these two components. Our work stems from the observation that we can make valid inferences about implicit information conveyed by the mere juxtaposition of snippets of text, as shown in the table describing Dressage in Figure 1.
We introduce the INFO TABS dataset to study and model inference with such semi-structured data. Premises in our dataset consist of info-boxes that convey information implicitly, and thus require complex reasoning to ascertain the validity of hypotheses. For example, determining that the hypothesis H2 in Figure 1 entails the premise table requires looking at multiple rows of the table, un
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324
derstanding the meaning of the row labeled Mixed gender, and also that Dressage is a sport.
INFOTABS consists of 23,738 premise-hypothesis pairs, where all premises are info-boxes, and the hypotheses are short sentences. As in the NLI task, the objective is to ascertain whether the premise entails, contradicts or is unrelated to the hypothesis. The dataset has 2,540 unique info-boxes drawn from Wikipedia articles across various categories, and all the hypotheses are written by Amazon's Mechanical Turk workers. Our analysis of the data shows that ascertaining the label typically requires the composing of multiple types of inferences across multiple rows from the tables in the context of world knowledge. Separate verification experiments on subsamples of the data also confirm the high quality of the dataset.
We envision our dataset as a challenging testbed for studying how models can reason about semistructured information. To control for the possibility of models memorizing superficial similarities in the data to achieve high performance, in addition to the standard train/dev/test split, our dataset includes two additional test sets that are constructed by systematically changing the surface forms of the hypothesis and the domains of the tables. We report the results of several families of approaches representing word overlap based models, models that exploit the structural aspect of the premise, and also derivatives of state-of-the-art NLI systems. Our experiments reveal that all these approaches underperform across the three test sets.
1. We propose a new English natural language inference dataset, INFOTABS, to study the problem of reasoning about semi-structured data.
2. To differentiate models' ability to reason about the premises from their memorization of spurious patterns, we created three challenge test sets with controlled differences that employ similar reasoning as the training set.
3. We show that several existing approaches for NLI underperform on our dataset, suggesting the need for new modeling strategies.
The dataset, along with associated scripts, are available at https://infotabs.github.io/.
2 The Case for Reasoning about Semi-structured Data
We often encounter textual information that is neither unstructured (i.e., raw text) nor strictly
structured (e.g., databases). Such data, where a structured scaffolding is populated with free-form text, can range from the highly verbose (e.g., web pages) to the highly terse (e.g. fact sheets, information tables, technical specifications, material safety sheets). Unlike databases, such semi-structured data can be heterogeneous in nature, and not characterized by pre-defined schemas. Moreover, we may not always have accompanying explanatory text that provides context. Yet, we routinely make inferences about such heterogeneous, incomplete information and fill in gaps in the available information using our expectations about relationships between the elements in the data.
Understanding semi-structured information requires a broad spectrum of reasoning capabilities. We need to understand information in an ad hoc layout constructed with elements (cells in a table) that are text snippets, form fields or are themselves sub-structured (e.g., with a list of elements). Querying such data can require various kinds of inferences. At the level of individual cells, these include simple lookup (e.g., knowing that dressage takes place in an arena), to lexical inferences (e.g., understanding that Mixed Gender means both men and women compete), to understanding types of text in the cells (e.g., knowing that the number 1912 is a year). Moreover, we may also need to aggregate information across multiple rows (e.g., knowing that dressage is a non-contact sport that both men and women compete in), or perform complex reasoning that combines temporal information with world knowledge.
We argue that a true test of reasoning should evaluate the ability to handle such semi-structured information. To this end, we define a new task modeled along the lines of NLI, but with tabular premises and textual hypotheses, and introduce a new dataset INFO TABS for this task.
3 The Need for Multi-Faceted Evaluation
Before describing the new dataset, we will characterize our approach for a successful evaluation of automated reasoning.
Recent work has shown that many datasets for NLI contain annotation biases or artifacts (e.g. Poliak et al., 2018). In other words, large models trained on such datasets are prone to learning spurious patterns—they can predict correct labels even with incomplete or noisy inputs. For instance, not and no in a hypothesis are correlated with contra
dictions (Niven and Kao, 2019). Indeed, classifiers trained on the hypotheses only (ignoring the premises completely) report high accuracy; they exhibit hypothesis bias, and achieving a high predictive performance does not need models to discover relationships between the premise and the hypothesis. Other artifacts are also possible. For example, annotators who generate text may use systematic patterns that "leak" information about the label to a model. Or, perhaps models can learn correlations that mimic reasoning, but only for one domain. With millions of parameters, modern neural networks are prone to overfitting to such imperceptible patterns in the data.
From this perspective, if we seek to measure a model's capability to understand and reason about inputs, we cannot rely on a single fixed test set to rank models. Instead, we need multiple test sets (of similar sizes) that have controlled differences from each other to understand how models handle changes along those dimensions. While all the test sets address the same task, they may not all be superficially similar to the training data.
With this objective, we build three test sets, named  $\alpha_{1}$ ,  $\alpha_{2}$  and  $\alpha_{3}$ . Here, we briefly introduce them; §4 goes into specifics. Our first test set  $(\alpha_{1})$  has a similar distribution as the training data in terms of lexical makeup of the hypotheses and the premise domains.
The second, adversarial test set  $(\alpha_{2})$ , consists of examples that are also similar in distribution to the training set, but the hypothesis labels are changed by expert annotators changing as few words in the sentence as possible. For instance, if Album  $X$  was released in the  $21^{st}$  century is an entailment, the sentence Album  $X$  was released before the  $21^{st}$  century is a contradiction, with only one change. Models that merely learn superficial textual artifacts will get confused by the new sentences. For  $\alpha_{2}$ , we rewrite entailments as contradictions and vice versa, while the neutrals are left unaltered.
Our third test set is the cross-domain  $(\alpha_{3})$  set, which uses premises from domains that are not in the training split, but generally, necessitate similar types of reasoning to arrive at the entailment decision. Models that overfit domain-specific artifacts will underperform on  $\alpha_{3}$ .
Note that, in this work, we describe and introduce three different test sets, but we expect that future work can identify additional dimensions along which models overfit their training data and con
struct the corresponding test sets.
4 The INFOTABS Dataset
In this section, we will see the details of the construction of INFOTABS. We adapted the general workflow of previous crowd sourcing approaches for creating NLI tasks (e.g., Bowman et al., 2015) that use Amazon's Mechanical Turk. $^{1}$
Sources of Tables Our dataset is based on 2,540 unique info-boxes from Wikipedia articles across multiple categories (listed in Appendix D). We did not include tables that have fewer than 3 rows, or have non-English cells (e.g., Latin names of plants) and technical information that may require expertise to understand (e.g., astronomical details about exoplanets). We also removed non-textual information from the table, such as images. Finally, we simplified large tables into smaller ones by splitting them at sub-headings. Our tables are isomorphic to key-value pairs, e.g., in Figure 1, the bold entries are the keys, and the corresponding entries in the same row are their respective values.
Sentence generation Annotators were presented with a tabular premise and instructed to write three self-contained grammatical sentences based on the tables: one of which is true given the table, one which is false, and one which may or may not be true. The turker instructions included illustrative examples using a table and also general principles to bear in mind, such as avoiding information that is not widely known, and avoiding using information that is not in the table (including names of people or places). The turkers were encouraged not to restate information in the table, or make trivial changes such as the addition of words like not or changing numerical values. We refer the reader to the project website for a snapshot of the interface used for turking, which includes the details of instructions.
We restricted the turkers to be from English-speaking countries with at least a Master's qualification. We priced each HIT (consisting of one table) at  $50\phi$ . Following the initial turking phase, we removed grammatically bad sentences and rewarded workers whose sentences involved multiple rows in the table with a  $10\%$  bonus. Appendix C gives additional statistics about the turkers.
Data partitions We annotated 2,340 unique tables with nine sentences per table (i.e., three turkers
Appendix A has more examples of tables with hypotheses.
Data split	# tables	# pairs
Train	1740	16538
Dev	200	1800
α1 test	200	1800
α2 test	200	1800
α3 test	200	1800
per table).<sup>2</sup> We partitioned these tables into training, development (Dev),  $\alpha_{1}$  and  $\alpha_{2}$  test sets. To prevent an outsize impact of influential turkers in a split, we ensured that the annotator distributions in the Dev and test splits are similar to that of the training split.
We created the  $\alpha_{2}$  test set from hypotheses similar to those in  $\alpha_{1}$ , but from a separate set of tables, and perturbing them as described in §3. On an average,  $\sim 2.2$  words were changed per sentence to create  $\alpha_{2}$ , with no more than 2 words changing in  $72\%$  of the hypotheses. The provenance of  $\alpha_{2}$  ensures that the kinds of reasoning needed for  $\alpha_{2}$  are similar to those in  $\alpha_{1}$  and the development set. For the  $\alpha_{3}$  test set, we annotated 200 additional tables belonging to domains not seen in the training set (e.g., diseases, festivals). As we will see in §5, hypotheses in these categories involve a set of similar types of reasonings as  $\alpha_{1}$ , but with different distributions.
In total, we collected 23,738 sentences split almost equally among entailments, contradictions, and neutrals. Table 1 shows the number of tables and premise-hypothesis pairs in each split. In all the splits, the average length of the hypotheses is similar. We refer the reader to Appendix D for additional statistics about the data.
Validating Hypothesis Quality We validated the quality of the data using Mechanical Turk. For each premise-hypothesis in the development and the test sets, we asked turkers to predict whether the hypothesis is entailed or contradicted by, or is unrelated to the premise table. We priced this task at  $36\ddagger$  for nine labels.
The inter-annotator agreement statistics are shown in Table 2, with detailed statistics in Appendix F. On all splits, we observed significant
Table 1: Number of tables and premise-hypothesis pairs for each data split
Dataset	Cohen's Kappa	Human Accuracy	Majority Agreement
Dev	0.78	79.78	93.52
α1	0.80	84.04	97.48
α2	0.80	83.88	96.77
α3	0.74	79.33	95.58
Table 2: Inter-annotator agreement statistics
inter-annotator agreement scores with Cohen's Kappa scores (Artstein and Poesio, 2008) between 0.75 and 0.80. In addition, we see a majority agreement (at least 3 out of 5 annotators agree) of range between  $93\%$  and  $97\%$ . Furthermore, the human accuracy agreement between the majority and gold label (i.e., the label intended by the writer of the hypothesis), for all splits is in range  $80\%$  to  $84\%$ , as expected given the difficulty of the task.
5 Reasoning Analysis
To study the nature of reasoning that is involved in deciding the relationship between a table and a hypothesis, we adapted the set of reasoning categories from GLUE (Wang et al., 2019b) to table premises. For brevity, here we will describe the categories that are not in GLUE and defined in this work for table premises. Appendix B gives the full list with definitions and examples. Simple look up refers to cases where there is no reasoning and the hypothesis is formed by literally restating what is in the table as a sentence; multi-row reasoning requires multiple rows to make an inference; and subjective/out-of-table inferences involve value judgments about a proposition or reference to information out of the table that is neither well known or common sense.
All definitions and their boundaries were verified via several rounds of discussions. Following this, three graduate students independently annotated 160 pairs from the Dev and  $\alpha_{3}$  test sets each, and edge cases were adjudicated to arrive at consensus labels. Figures 2a and 2b summarize these annotation efforts. We see that we have a multifaceted complex range of reasoning types across both sets. Importantly, we observe only a small number of simple lookups, simple negations for contradictions, and mere syntactic alternations that can be resolved without complex reasoning. Many instances call for looking up multiple rows, and involve temporal and numerical reasoning. Indeed,
2For tables with ungrammatical sentences, we repeated the HIT. As a result, a few tables in the final data release have more than 9 hypotheses.
as Figures 2c and 2d show, a large number of examples need at least two distinct kinds of reasoning; on an average, sentences in the Dev and  $\alpha_{3}$  sets needed 2.32 and 1.79 different kinds of reasoning, respectively.
We observe that semi-structured premises forced annotators to call upon world knowledge and common sense (KCS);  $48.75\%$  instances in the Dev set require KCS. (In comparison, in the MultiNLI data, KCS is needed in  $25.72\%$  of examples.) We conjecture that this is because information about the entities and their types is not explicitly stated in tables, and have to be inferred. To do so, our annotators relied on their knowledge about the world including information about weather, seasons, and widely known social and cultural norms and facts. An example of such common sense is the hypothesis that "X was born in summer" for a person whose date of birth is in May in New York. We expect that the INFOTABS data can serve as a basis for studying common sense reasoning alongside other recent work such as that of Talmor et al. (2019),
Neutral hypotheses are more inclined to being subjective/out-of-table because almost anything subjective or not mentioned in the table is a neutral statement. Despite this, we found that in all evaluations in Appendix E (except those involving the adversarial  $\alpha_{2}$  test set), our models found neutrals almost as hard as the other two labels, with only an  $\approx 3\%$  gap between the F-scores of the neutral label and the next best label.
The distribution of train, dev,  $\alpha_{1}$  and  $\alpha_{2}$  are similar because the premises are taken from the same categories. However, tables for  $\alpha_{3}$  are from different domains, hence not of the same distribution as the previous splits. This difference is also reflected in Figures 2a and 2b, as we see a different distribution of reasonings for each test set. This is expected; for instance, we cannot expect temporal reasoning from tables in a domain that does not contain temporal quantities.
The goal of our experiments is to study how well different modeling approaches address the INFOTABS data, and also to understand the impact of various artifacts on them. First, we will consider different approaches for representing tables in ways that are amenable to modern neural models.
6.1 Representing Tables
A key aspect of the INFOTABS task that does not apply to the standard NLI task concerns how premise tables are represented. As baselines for future work, let us consider several different approaches.
1. Premise as Paragraph (Para): We convert the premise table into paragraphs using fixed template applied to each row. For a table titled t, a row with key k and value v is written as the sentence The k of t are v. For example, for the table in Figure 1, the row with key Equipment gets mapped to the sentence The equipment of Dressage are horse, horse tack. We have a small number of exceptions: e.g., if the key is born or died, we use the following template: t was k on v.
The sentences from all the rows in the table are concatenated to form the premise paragraph. While this approach does not result in grammatical sentences, it fits the interface for standard sentence encoders.
2. Premise as Sentence (Sent): Since hypotheses are typically short, they may be derived from a small subset of rows. Based on this intuition, we use the word mover distance (Kusner et al., 2015) to select the closest and the three closest sentences to the hypothesis from the paragraph representation (denoted by WMD-1 and WMD-3, respectively).
3. Premise as Structure 1 (TabFact): Following Chen et al. (2020), we represent tables by a sequence of key : value tokens. Rows are separated by a semi-colon and multiple values for the same key are separated by a comma.
4. Premise as Structure 2 (TabAttn): To study an attention based approach, such as that of Parikh et al. (2016), we convert keys and values into a contextually enriched vectors by first converting them into sentences using the Para approach above, and applying a contextual encoder to each sentence. From the token embeddings, we obtain the embeddings corresponding of the keys and values by mean pooling over only those tokens.
6.2 Modeling Table Inferences
Based on the various representations of tables described above, we developed a collection of models for the table inference problem, all based on standard approaches for NLI. Due to space constraints,
(a) Number of examples per reasoning type in the Dev set
(b) Number of examples per reasoning type in the  $\alpha_{3}$  set
(c) Number of reasonings per example in the Dev set
(d) Number of reasonings per example in the  $\alpha_{3}$  set
Figure 2: Distribution of the various kinds of reasoning in the Dev and  $\alpha_{3}$  sets. The labels OOT and KCS are short for out-of-table and Knowledge & Common Sense, respectively.
we give a brief description of the models here and refer the interested reader to the code repository for implementation details.
For experiments where premises are represented as sentences or paragraphs, we evaluated a feature-based baseline using unigrams and bigrams of tokens. For this model (referred to as SVM), we used the LibLinear library (Fan et al., 2008).
For these representations, we also evaluated a collection of BERT-class of models. Following the standard setup, we encoded the premise-hypothesis pair, and used the classification token to train a classifier, specifically a two-layer feedforward network that predicts the label. The hidden layer had half the size of the token embeddings. We compared RoBERTa $_L$  (Large), RoBERTa $_B$  (Base) and  $\mathrm{BERT}_B$  (Base) in our experiments.
We used the above BERT strategy for the TabFact representations as well. For the TabAttn representations, we implemented the popular decomposable attention model (Parikh et al., 2016) using the premise key-value embeddings and hypothesis token embeddings with 512 dimensional attend and compare layers.
We implemented all our models using the Py
Torch with the transformers library (Wolf et al., 2019). We trained our models using Adagrad with a learning rate of  $10^{-4}$ , chosen by preliminary experiments, and using a dropout value of 0.2. All our results in the following sections are averages of models trained from three different random seeds.
Our experiments answer a series of questions.