SlowGuess/ACL_2020_OCR huggingface dataset:
type may be determined by looking at one row. For instance, looking at Figure 5, one can infer that the entity type is a painting by only looking at the row with key value Medium. Lastly, ascertaining the entity type may require knowledge, but if so, then we will not explicitly mark the instance as Knowledge & Common Sense. For example, knowing that SNL is a TV show will be entity type and not Knowledge & Common Sense.
Lexical reasoning Any inference that can be made using words, independent of the context of the words falls. For example, knowing that dogs are animals, and alive contradicts dead would fall into the category of lexical reasoning. This type of reasoning includes substituting words with their synonyms, hybernyms, hyponyms and antonyms. It also includes cases where a semantically equivalent or contradicting word (perhaps belonging to a different root word) is used in the hypothesis., e.g., replacing understand with miscomprehend. Lexical reasoning also includes reasoning about monotonicity of phrases.
Negation Any explicit negation, including morphological negation (e.g., the word affected being mapped to unaffected). Negation changes the morphology without changing the root word, e.g., we have to add an explicit not.
This category includes double negations, which we believe is rare in our data. For example, the introduction of the phrase not impossible would count as a double negation. If the word understand in the premise is replaced with not comprehend, we are changing the root word (understand to comprehend) and introducing a negation. So this change will be marked as both Lexical reasoning and Negation.
Knowledge & Common Sense This category is related to the World Knowledge and Common Sense categories from GLUE. To quote the description from GLUE: "...the entailment rests not only on correct disambiguation of the sentences, but also application of extra knowledge, whether it is
concrete knowledge about world affairs or more common-sense knowledge about word meanings or social or physical dynamics."
While GLUE differentiates between world knowledge and common sense, we found that this distinction is not always clear when reasoning about tables. So we do not make the distinction.
Named Entities This category is identical to the Named Entities category from GLUE. It includes an understanding of the compositional aspect of names (for example, knowing that the University of Hogwarts is the same as Hogwarts). Acronyms and their expansions fall into this category (e.g., the equivalence of New York Stock Exchange as NYSE).
Numerical reasoning Any form of reasoning that involves understanding numbers, counting, ranking, intervals and units falls under this group. This category also includes numerical comparisons and the use of mathematical operators to arrive at the hypothesis.
Temporal reasoning Any inferences that involves reasoning about time fall into this category. There may be an overlap between other categories and this one. Any numerical reasoning about temporal quantities and the use of knowledge about time should be included here. Examples of temporal reasoning:
- 9 AM is in the morning. (Since this is knowledge about time, we will only tag this as Temporal.)
1950 is the  $20^{th}$  century.
- 1950 to 1962 is twelve years.
- Steven Spielberg was born in the winter of 1946. (If the table has the date—18th December, 1946—and the location of birth—Ohio, this sentence will have both knowledge & Common Sense and temporal reasoning. This is because one should be able to tell that the birth location is in the northern hemisphere (knowledge) and December is part of the Winter in the northern hemisphere (temporal reasoning)).
Coreference This category includes cases where expressions refer to the same entity. However, we do not include the standard gamut of coreference phenomena in this category because the premise is
not textual. We specifically include the following phenomena in this category: Pronoun coreference, where the pronoun in a hypothesis refers to a noun phrase either in the hypothesis or the table. E.g., Chris Jericho lives in a different state than he was born in. A noun phrase (not a named entity) in the hypothesis refers to a name of an entity in the table. For example, the table may say that Bob has three children, including John and the hypothesis says that Bob has a son. Here the phrase a son refers to the name John.
If there is a pronoun involved, we should not treat it as entity type or knowledge even though knowledge may be needed to know that, say, Theresa May is a woman and so we should use the pronoun she.
To avoid annotator confusion, when two names refer to each other, we label it only as the Named Entities category. For example, if the table talks about William Henry Gates III and the hypothesis describes Bill Gates, even though the two phrases do refer to each other, we will label this as Named Entities.
Quantification Any reasoning that involves introducing a quantifier such as every, most, many, some, none, at least, at most, etc. in the hypothesis. This category also includes cases where prefixes such as multi- (e.g., multi-ethnic) are used to summarize multiple elements in the table.
To avoid annotator confusion, we decide that the mere use of quantifiers like most and many is quantification. However, if the quantifier is added after comparing two numerical values in the table, the sentence is labeled to have numerical reasoning as well.
Subjective/Out of table Subjective inferences refer to any inferences that involve either value judgment about a proposition or a qualitative analysis of a numerical quantity. Out of table inferences involve hypotheses that use extra knowledge that is neither a well known universal fact nor common sense. Such hypotheses may be written as factive or implicative constructions. Below are some examples of this category:
- Based on a table about Chennai: Chennai is a very good city.
- If the table says that John's height is 6 feet, then the hypothesis that John is a tall person. may be subjective. However, if John's
height is 8 feet tall, then the statement that John is tall. is no longer subjective, but common sense.
- If the table only says that John lived in Madrid and Brussels, and the hypothesis is John lived longer in Madrid than Brussels. This inference involves information that is neither well known nor common sense.
- Based on the table of the movie Jaws, the hypothesis It is known that Spielberg directed Jaws falls in this category. The table may contain the information that Spielberg was the director, but this may or may not be well known. The latter information is out of the table.
Syntactic Alternations This refers to a catch-all category of syntactic changes to phrases. This includes changing the preposition in a PP, active-passive alternations, dative alternations, etc. We expect that this category is rare because the premise is not text. However, since there are some textual elements in the tables, the hypothesis could paraphrase them.
This category is different from reasoning about named entities. If a syntactic alternation is applied to a named entity (e.g., The Baltimore City Police being written as The Police of Baltimore City), we will label it as a Named Entity if, and only if, we consider both phrases as named entities. Otherwise, it is just a syntactic alternation. Below are some examples of this category:
- New Orleans police officer being written as police officer of New Orleans.
- Shakespeare's sonnet being written as sonnet of Shakespeare.
Ellipsis This category is similar in spirit to the category Ellipsis/Implicits in GLUE: "An argument of a verb or another predicate is elided in the text, with the reader filling in the gap." Since in our case, the only well-formed text is in the hypothesis, we expect such gaps only in the hypothesis. (Compared to GLUE, where the description makes it clear that the gaps are in the premises and the hypotheses are constructed by filling in the gaps with either correct or incorrect referents.). For example, in a table about Norway that lists the per capita income as  $74K, the hypothesis that The per capita income is$ 74K. elides the fact that this is about citizens of Norway, and not in general.
C INFOTABS Worker Analysis
Figure 6 shows the number of examples annotated by frequent top- $n$  workers. We can see that the top 40 annotators annotated about  $90\%$  of the data. This observation is concordant with other crowdsourced data annotation projects such as SNLI and MultiNLI (Gururangan et al., 2018).
Figure 6: Number of annotations by frequent annotators
D INFOTABS Dataset Statistics
In this section, we provide some essential statistics that will help in a better understanding of the dataset.
Table 8 shows a split-wise analysis of premises and annotators. The table shows that there is a huge overlap between the train set and the other splits except  $\alpha_{3}$ . This is expected since  $\alpha_{3}$  is from a different domain. Also, we observe that tables in  $\alpha_{3}$  are longer. In the case of annotators, we see that most of our dataset across all splits was annotated by the same set of annotators.
Table 9 presents information on the generated hypotheses. The table lists the average number of words in the hypotheses. This is important because a dissimilar mean value of words would induce the possibility of length bias, i.e., the length of the sentences would be a strong indicator for classification.
Table 10 shows the overlap between hypotheses and premise tables across various splits. Stop words like  $a$ ,  $the$ ,  $it$ ,  $of$ , etc. are removed. We observe that the overlap is almost similar across labels.
Table 11 and 12 show the distribution of table categories in each split. We accumulate all the categories occurring for less than  $3\%$  for every split into the "Other" category.
Split	Train	Dev	α1	α2	α3
Number of Unique Keys	1558	411	466	332	409
Number of Unique Keys Intersection with Train	-	334	312	273	94
Average # of keys per table	8.8	8.7	8.8	8.8	13.1
Number of Distinct Annotators	121	35	37	31	23
Annotator Intersection with Train	-	33	37	30	19
Number of Instances annotated by a Train annotator	-	1794	1800	1797	1647
Table 8: Statistics of the premises and annotators across all discussed train-test splits
Label	Train	Dev	α1	α2	α3
Entail	9.80	9.71	9.90	9.33	10.5
Neutral	9.84	9.89	10.05	9.59	9.84
Contradict	9.37	9.72	9.84	9.40	9.86
Table 9: Mean length of the generated hypothesis sentences across all discussed train-test splits (standard deviation is in range 2.8 to 3.5)
Label	Train	Dev	α1	α2	α3
Entail	0.52	0.47	0.45	0.46	0.48
Neutral	0.46	0.44	0.44	0.49	0.46
Contradict	0.44	0.43	0.45	0.44	0.46
E F1 Score Analysis
The F1 scores per label for two model baselines are in Table 13. We observe that neutral is easier than entailment and contradiction for both baseline, which is expected as neutrals are mostly associated with subjective/out-of-table reasonings which makes them syntactically different and easier to predict correctly. Despite this, we found that in all evaluations in (§6) (except for  $\alpha_{2}$  test set), our models found neutrals almost as hard as the other two labels, with only an  $\sim 3\%$  gap between the F-scores of the neutral label and the next best label. For  $\alpha_{2}$  test set neutral are much easier than entailment and contradiction. This is expected as entailment and contradiction in  $\alpha_{2}$  were adversarily flipped; hence, these predictions become remarkably harder compared to neutrals. Furthermore,  $\alpha_{3}$  is the hardest data split, followed by  $\alpha_{2}$  and  $\alpha_{1}$ .
Table 10: Mean statistic of the hypothesis sentences word overlapped with premises tables across all discussed train-test splits (standard deviation is in range 0.17 to 0.22)
Category	Train	Dev	α1	α2
Person	23.68	27	28.5	35.5
Musician	14.66	19	18.5	22.5
Movie	10.17	10	9	11.5
Album	9.08	7	3.5	4.5
City	8.05	8.5	8	7
Painting	5.98	4.5	4	3.5
Organization	4.14	2	1	0.5
Food / Drinks	4.08	4	4	3
Country	3.74	6	9	3.5
Animal	3.56	4.5	4	4
Sports	4.6	3.5	2.5	0.0
Book	2.18	0.5	3	2.5
Other	6.07	8.00	5.00	2.00
Table 11: Categories for all data splits (excluding  $\alpha_{3}$ ) in percentage (\%). Others (< 3%) include categories such as University, Event, Aircraft, Product, Game, Architecture, Planet, Awards, Wineyard, Airport, Language, Element, Car
Category	α3(%)
Diseases	20.4
Festival	17.41
Bus / Train Lines	14.93
Exams	8.46
Element	4.98
Air Crash	3.98
Bridge	3.98
Disasters	3.48
Smartphone	3.48
Other	18.9
Table 12: Categories for  $\alpha_{3}$  datasplit. Others (<  $3\%$  ) include categories such as Computer, Occupation, Restaurant, Engines, Equilibrium, OS, Cloud, Bus/Train Station, Coffee House, Cars, Bus/Train Provider, Hotel, Math, Flight
Premise as Paragraph
Split	Entailment	Neutral	Contradiction
Dev	76.19	79.02	72.73
α1	74.69	77.85	69.85
α2	57.06	80.36	62.14
α3	65.27	66.06	61.61
Premise as TabFact
Split	Entailment	Neutral	Contradiction
Dev	77.69	79.45	74.77
α1	76.43	80.34	73.07
α2	55.34	80.83	64.44
α3	65.92	67.28	63.57
F Statistics of INFOTABS Verification
Table 14 shows the detailed agreement statistics of verification for the development and the three test splits. For every premise-hypothesis pair, we asked five annotators to verify the label. The table details the verification agreement among the annotators, and also reports how many of these majority labels match the gold label (i.e., the label intended by the author of the hypothesis). We also report individual annotator label agreement by matching the annotator's label with the gold label and majority label for an example. Finally, the table reports the Fleiss Kappa (across all five annotation labels) and the Cohen Kappa (between majority and gold label) for the development and the three test splits.
We see that, on average, about  $84.8\%$  of individual labels match with the majority label across all verified splits. Also, an average of  $75.15\%$  individual annotations also match the gold label across all verified splits.
From Table 14, we can calculate the percentage of examples with at least 3, 4, and 5 label agreements across 5 verifiers for all splits. For all splits, we have very high inter-annotator agreement of  $>95.85\%$  for at-least 3,  $>74.50\%$  for at-least 4 and  $43.91\%$  for at-least 5 annotators. The number of these agreements match with the gold label are:  $>81.76\%$  for at-least 3,  $>67.09\%$  for at-least 4 and  $40.85\%$  for at-least 5 for all splits.
Table 13: F1 Score (%) with various baselines. All models are trained with RoBERTaL
Exact agreement between annotators
Dataset	Number	Gold/Total
Dev	3	350 / 469
4	529 / 601
5	550 / 605
no agreement	116
α1	3	184 / 292
4	459 / 533
5	863 / 922
no agreement	45
α2	3	245 / 348
4	453 / 537
5	812 / 857
no agreement	58
α2	3	273 / 422
4	441 / 524
5	706 / 765
no agreement	79
Individual agreement with gold / majority label
Dataset	Statistics	Agreement (%)
Dev	Gold	71.12
Majority	81.65
α1	Gold	78.52
Majority	87.24
α2	Gold	77.74
Majority	86.32
α3	Gold	73.22
Majority	84.01
Average	Gold	75.15
Majority	84.8
Kappa values across splits
Dataset	Fleiss	Cohen
Dev	0.4601	0.7793
α1	0.6375	0.7930
α2	0.5962	0.8001
α3	0.5421	0.7444
Table 14: Exact, Individual and Kappa values for verification's statistics.