SlowGuess/ACL_2020_OCR huggingface dataset:
William Shakespeare. 1603. The Tragedy of Hamlet, Prince of Denmark.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of LREC.
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. In Transactions of the Association for Computational Linguistics.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. arXiv preprint arXiv:1901.10444.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2017. Mastering the dungeon: Grounded language learning by mechanical turker descent. arXiv preprint arXiv:1711.07950.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL.
A Performance on challenge datasets
Recently, several hard test sets have been made available for revealing the biases NLI models learn from their training datasets (Nie and Bansal, 2017; McCoy et al., 2019; Gururangan et al., 2018; Naik et al., 2018). We examine model performance on two of these: the SNLI-Hard (Gururangan et al., 2018) test set, which consists of examples that hypothesis-only models label incorrectly, and the NLI stress tests (Naik et al., 2018), in which sentences containing antonyms pairs, negations, high word overlap, i.a., are heuristically constructed. We test our models on these stress tests after tuning on each test's respective development set to account for potential domain mismatches. For comparison, we also report results from the original papers: for SNLI-Hard from Gururangan et al.'s implementation of the hierarchical tensor-based Densely Interactive Inference Network (Gong et al., 2018, DIIN) on MNLI, and for the NLI stress tests, Naik et al.'s implementation of InferSent (Conneau et al., 2017) trained on SNLI.
B Further linguistic analysis
We compare the incidence of linguistic phenomena in ANLI with extant popular NLI datasets to get an idea of what our dataset contains. We observe that FEVER and SNLI datasets generally contain many fewer hard linguistic phenomena than MultiNLI and ANLI (see Table 8).
ANLI and MultiNLI have roughly the same percentage of hypotheses that exceeding twenty words in length, and/or contain negation (e.g., 'never', 'no'), tokens of 'or', and modals (e.g., 'must', 'can'). MultiNLI hypotheses generally contains more pronouns, quantifiers (e.g., 'many', 'every'), WH-words (e.g., 'who', 'why'), and tokens of 'and' than do their ANLI counterparts—although A3 reaches nearly the same percentage as MultiNLI for negation, and modals. However, ANLI contains more cardinal numerals and time terms (such as 'before', 'month', and 'tomorrow') than MultiNLI. These differences might be due to the fact that the two datasets are constructed from different genres of text. Since A1 and A2 contexts are constructed from a single Wikipedia data source (i.e., HotPotQA data), and most Wikipedia articles include dates in the first line, annotators appear to prefer constructing hypotheses that highlight numerals and time terms, leading to their high incidence.
Focusing on ANLI more specifically, A1 has
roughly the same incidence of most tags as A2 (i.e., within  $2\%$  of each other), which, again, accords with the fact that we used the same Wikipedia data source for A1 and A2 contexts. A3, however, has the highest incidence of every tag (except for numbers and time) in the ANLI dataset. This could be due to our sampling of A3 contexts from a wider range of genres, which likely affected how annotators chose to construct A3 hypotheses; this idea is supported by the fact that A3 contexts differ in tag percentage from A1 and A2 contexts as well. The higher incidence of all tags in A3 is also interesting, because it could be taken as providing yet another piece of evidence that our HAMLET data collection procedure generates increasingly more difficult data as rounds progress.
C Dataset properties
Table 9 shows the label distribution. Figure 4 shows a histogram of the number of tries per good verified example across for the three different rounds. Figure 5 shows the time taken per good verified example. Figure 6 shows a histogram of the number of tokens for contexts and hypotheses across three rounds. Figure 7 shows the proportion of different types of collected examples across three rounds.
Inter-annotator agreement Table 10 reports the inter-annotator agreement for verifiers on the dev and test sets. For reference, the Fleiss' kappa of FEVER (Thorne et al., 2018) is 0.68 and of SNLI (Bowman et al., 2015) is 0.70. Table 11 shows the percentage of agreement of verifiers with the intended author label.
D Examples
We include more examples of collected data in Table 12.
E User interface
Examples of the user interface are shown in Figures 8, 9 and 10.
Figure 4: Histogram of the number of tries for each good verified example across three rounds.
Figure 5: Histogram of the time spent per good verified example across three rounds.
Figure 6: Histogram of the number of tokens in contexts and hypotheses across three rounds.
Figure 7: Proportion across three rounds. A=Examples that model got right, B1=Examples that model got wrong and the first two verifiers agreed with the writer, B2=Examples that model got wrong and only one of the first two verifiers agreed with the writer and a third verifier also agreed with the writer, C=Examples where two verifiers agreed with each other and overruled the writer, D=Examples for which there is no agreement among verifiers. A and C are added only to training set. B1 and B2 are added to training, dev, or test set. D was discarded.
Figure 8: UI for Creation. (Provide the context to annotator)
Figure 9: Collection UI for Creation. (Give the model feedback to annotator)
Figure 10: UI for Verification Task.
Tag	SNLI	Other Datasets	ANLI
% c	% h	MNLIm	MNLImm	F	A1	A2	A3
% c	% h	% c	% h	% claim	% c	% h	% c	% h	% c	% h
Negation	< 1	1	14	16	12	16	3	2	6	3	10	22	14
‘and’	30	7	41	15	42	18	6	85	12	88	11	75	11
‘or’	1	< 1	7	2	8	2	< 1	6	0	6	< 1	15	1
Numbers	10	4	16	8	15	9	9	72	30	73	27	42	15
Time	12	4	15	7	16	9	6	57	22	56	19	49	11
WH-words	3	1	16	7	18	9	2	28	5	27	5	35	5
Pronouns	11	7	37	20	39	24	2	30	9	28	7	60	13
Quantifiers	5	3	21	16	22	17	3	14	10	17	12	38	12
Modals	< 1	< 1	17	13	18	14	< 1	2	3	3	2	35	14
>20 words	14	< 1	37	2	39	3	< 1	100	5	100	4	98	4
# exs	10k	10k	10k	9999	1k	1k	1200
Table 8: Percentage of development set sentences with tags in several datasets: AdvNLI, SNLI, MultiNLI and FEVER.  $\% c$  refers to percentage in contexts, and  $\% h$  refers to percentage in hypotheses. Bolded values label linguistic phenomena that have higher incidence in adversarially created hypotheses than in hypotheses from other NLI datasets, and italicized values have roughly the same (within  $5\%$ ) incidence.
Round	Entailment / Neutral / Contradiction
Train	Dev	Test
A1	5,371 / 7,052 / 4,523	334 / 333 / 333	334 / 333 / 333
A2	14,448 / 20,959 / 10,053	334 / 333 / 333	334 / 333 / 333
A3	32,292 / 40,778 / 27,389	402 / 402 / 396	402 / 402 / 396
ANLI	52,111 / 68,789 / 41,965	1,070 / 1,068 / 1,062	1,070 / 1,068 /1,062
Table 9: Label distribution in splits across rounds.
Round	Dev + Test	Dev	Test
A1	0.7210	0.7020	0.7400
A2	0.6910	0.7100	0.6720
A3	0.6786	0.6739	0.6832
Table 10: Inter-annotator agreement (Fleiss' kappa) for writers and the first two verifiers.
SNLI	MNLI	A1	A2	A3
85.8	85.2	86.1	84.6	83.9
Table 11: Percentage of agreement of verifiers ("validators" for SNLI and MNLI) with the author label.
4900
Table 12: Extra examples from development sets. 'An' refers to round number, 'orig.' is the original annotator's gold label, 'pred.' is the model prediction, 'valid.' is the validator labels, 'reason' was provided by the original annotator, 'Annotations' is the tags determined by linguist expert annotator.