SlowGuess/NAACL_2021_OCR huggingface dataset:
Hannah Youngeun An and Aaron Steven White. 2019. The lexical and grammatical sources of neg-raising inferences. In Proceedings of the Society for Computation in Linguistics (SCiL 2020), pages 220-233.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 632-642.
Adrian Calma and Bernhard Sick. 2017. Simulation of annotators for active learning: Uncertain Oracles. In Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017), pages 49-58.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, pages 177-190.
Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The Commitment Bank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171-4186.
William Gantt, Benjamin Kane, and Aaron Steven White. 2020. Natural language inference with mixed effects. In *The Ninth Joint Conference on Lexical and Computational Semantics* (*SEM* 2020).
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019a. Do you know that Florence is packed with visitors? Evaluating state-of-the-art models of speaker commitment. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 4208-4213.
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b. Evaluating BERT for natural language inference: A case study on the Commitmentbank. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP 2019, pages 6085-6090.
Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert
Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: It's complicated! In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pages 1886-1895.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.
Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, IWCS 2009, pages 140-156.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 3428-3448.
Ellie Pavlick and Chris Callison-Burch. 2016. "babies" are "little" and most "problem" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, pages 677-694.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, pages 1532-1543.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, pages 7-14.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. 1992. Query by committee. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992, Pittsburgh, PA, USA, July 27-29, 1992, pages 287-294.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference
on Neural Information Processing Systems 2019, NeurIPS 2019, pages 3261-3275.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pages 1112-1122.
Appendix A Linguistic Rules
Our linguistic rules are inspired by and adapted from Jiang and de Marneffe (2019b) to explicitly include the most discriminating expressions for disagreement items. We utilize three linguistic features which are provided in CB: entailment-canceling environment (negation, modal, question, antecedent of conditional), matrix verb and its subject person.
1. Items under conditional are disagreement.
2. Items under question and with second person are neutral.
3. Items under question and with non-second person are disagreement.
4. Items of the form "I don't know/think/believe" are contradiction (i.e., negRaising structure).
5. Items with factive verbs are entailment.
6. Items under negation and with non-factive verbs are disagreement.
7. Items under modal and with non-third person are entailment.
When this policy is executed, there are two additional auxiliary rules: Items not falling in any group above are assigned a disagreement label as it is the dominant class in CB; For items satisfying more than one rule, the label will be determined by the higher-ranked rule (i.e., a smaller number indicates a higher rank). Note that the rules above also reveal the most discriminating expressions for each class.
Appendix B Impact of  $r$
We experiment with three different  $r$  values, 0.25, 0.4, 0.7. Intuitively,
- 0.25: each module contributes equally;
- 0.4: finer-grained NLI predictor is the main component and three artificial annotators are to complement the main predictor;
- 0.7: over-amplify the role of main predictor and suppress three artificial annotators.
Table A1 shows the results on the dev set in terms of both accuracy and F1 under different  $r$
r	0.25	0.4	0.7
Accuracy	65.33	65.67	65.33
F1	64.86	64.57	65.06
Table A1: Performance of our proposed AAs on the dev set under different  $r$  values.
Train	Test	Overall
CBOW	35	30	65
Vanilla BERT	205	7	212
Ensemble BERT	645	10	655
AAs (ours)	955	9	964
Table A2: The training (10 epochs) and testing time as well as overall running time for each neural network-based model. All numbers are in seconds.
values. We set  $r$  to 0.4 as it achieves the best accuracy on the dev set.
Appendix C Reproducibility Checklist
We set  $r$  in eq. 2 at 0.4 (see Appendix B). For all experiments, we use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1e-5 and finetune up to 10 epochs. The batch size of first three baselines is 8 and that of Ensemble BERT baseline and our AAs is 2. The gradient is clipped when its norm exceeds 5. We select the best model for each method using the accuracy on the dev set, and the average performance on dev set is shown in Table 3 as well. Hyperparameters for underlying BERT (bert-base-uncased)<sup>5</sup> are the default.
C.2 Summary Statistics of Results
For the results reported in Table 3, we run each baseline (except for "Always 0" and Heuristic rules) and AAs 10 times, and report the average. For the results reported in Table 6 and 7, we randomly select 3 trained models for each baseline and AAs, and report the average. We also share the checkpoints of AAs at https://github.com/FrederickXZhang/FgNLI to help reproduce the results given in Section 5.
C.3 Computational Resources
All experiments are conducted using one single GeForce GTX 2080 Ti 12 GB GPU (with significant CPU resources). The overall running time of
basi2es and AAs are listed in Table A2.
C.4 Dataset
The characteristics of the CommitmentBank (CB) are detailed in Section 2. The original version is available at https://github.com/mcdm/ CommitmentBank, and the recast version used in this work is available at https://github. com/FrederickXZhang/FgNLI.