SlowGuess/ACL_2020_OCR huggingface dataset:
A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4711-4716.
John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648-657.
Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2015, pages 1643-1648.
Zhongyang Li, Tongfei Chen, and Benjamin Van Durme. 2019. Learning to rank for plausible plausibility. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4818-4823.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. The SICK (Sentences Involving Compositional Knowledge) dataset for relatedness and entailment.
Richard Montague. 1973. The proper treatment of quantification in ordinary english. In K. J. J. Hintikka, J. M. E. Moravcsik, and P. Suppes, editors, Approaches to Natural Language: Proceedings of the 1970 Stanford Workshop on Grammar and Semantics, pages 221-242. Springer Netherlands, Dordrecht.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Ellie Pavlick and Chris Callison-Burch. 2016. "babies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1, pages 2164-2173.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Trans. Assoc. Comput. Linguistics, 7:677-694.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191.
8777
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *Logical Formalizations of Commonsense Reasoning*, Papers from the 2011 AAAI Spring Symposium.
Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 731-744.
Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1, pages 208-218.
Amram Shapiro, Louise Firth Campbell, and Rosalind Wright. 2014. Book of Odds: From Lightning Strikes to Love at First Sight, the Odds of Everyday Life. William Morrow Paperbacks.
Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 2, pages 352-357.
Adam R. Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R. Gormley. 2017. Semantic proto-role labeling. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 4459-4466.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation.
Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science, 211(4481):453-458.
Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323.
Ivan Vulic, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. Computational Linguistics, 43(4).
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1112-1122.
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Trans. Assoc. Comput. Linguistics, 5:379-395.
8778
A Annotation
Here we include information about the qualifications used to vet annotators. We also include screenshots of the interface used to collect annotations.
A.1 Qualification Test
Annotators were given a qualification test to ensure non-expert workers were able to give reasonable subjective probability estimates. We first extracted seven statements from Book of Odds (Shapiro et al., 2014), and manually split the statement into a bleached premise and hypothesis. We then wrote three easy premise-hypothesis pairs with definite probabilities like  $(p = "A girl tossed a coin.", h = "The coin comes up a head.", probability: 0.5)$ . We qualify users that meet both criteria: (1) For the three easy pairs, their annotations had to fall within a small error range around the correct label  $y$ , computed as  $\delta = \frac{1}{4} \min \{y, 1 - y\}$ . (2) Their overall annotations have a Pearson  $r > 0.7$  and Spearman  $\rho > 0.4$ . This qualification test led to a pool of 40 trusted annotators, which were employed for the entirety of our dataset creation.
A.2 Annotation Interface
We include screenshots of the instructions and examples shown to crowdsourced workers (Figure 4) as the interface we provided (Figure 3)
B Redundant Annotations
By default, we use two crowdsourced workers to annotate each UNLI sentence-pair. If the two annotations on the raw slider bar  $\{0,\dots ,10000\}$  differ by more than 2000, we then elicit a third annotator.
Table 6 summarizes the statistics of  $u$ -SNLI.
D Additional Training Details
We use the BERT-BASE-UNCASED model, with the Adam optimizer (Kingma and Ba, 2015), an initial learning rate of  $10^{-5}$ , and maximum gradient norm 1.0. Our model is trained for 3 epochs, where the epoch resulting in the highest Pearson  $r$  on the dev set is selected.
Premise: Woman in white in foreground and a man slightly behind walking with a sign 's Pizza and Gyro in the background.
Figure 3: An example of our annotation interface.
Figure 4: Three examples from the instructions.
Figure 5: Our logistic transformation function.
Partition	Breakdown	SNLI	U-SNLI
train	Distinct premises	151k	7,931
ENT hypotheses	183k	7,931
NEU hypotheses	183k	39,655
CON hypotheses	183k	7,931
Total P-H pairs	550k	55,517
dev	Distinct premises	3,319	2,647
ENT hypotheses	3,329	162
NEU hypotheses	3,235	2,764
CON hypotheses	3,278	114
Total P-H pairs	10k	3,040
test	Distinct premises	3,323	2,635
ENT hypotheses	3,368	156
NEU hypotheses	3,219	2,770
CON hypotheses	3,237	114
Total P-H pairs	10k	3,040
Table 6: Statistics of SNLI data re-annotated under UNLI.