SlowGuess/NAACL_2022_OCR huggingface dataset:
Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Bill MacCartney. 2009. Natural language inference. Stanford University.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586-1596, Brussels, Belgium. Association for Computational Linguistics.
Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4661-4675.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 996-1005.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Dataset	Split	Size
SNLI	train	550,152
valid	10,000
test	10,000
δ-NLI	train	200,694
valid	14,968
test	15,414
δ-SNLI	train	88,676
valid	1,785
test	1,837
Table 4: Split sizes across SNLI,  $\delta$  -NLI,and  $\delta$  -SNLI.
A SNLI and  $\delta$ -NLI Dataset Sizes
To train and tune our neural and lexical models, we utilize the train and validation splits from SNLI and  $\delta$ -NLI. We include train/validation/test split sizes in Table 4. Our  $\delta$ -NLI RoBERTa models were finetuned on examples from all 3 portions of the  $\delta$ -NLI dataset (ATOMIC, SNLI, and SOCIAL-CHEM-101), however for our evaluation set and analysis, we exclusively use the SNLI portion of the  $\delta$ -NLI dataset, abbreviated as  $\delta$ -SNLI. We include the split sizes for  $\delta$ -SNLI in Table 4 as well.
B Model Training Setup
Neural Models. We use the Hugging Face library to train all of our RoBERTa models. We utilize roberta-base, which has 125M trainable parameters. All models were trained on a single NVIDIA 1080-TI GPU. After tuning on the validation set, all models were trained for two epochs with a learning rate of 2e-5 and a batch size of 32. Tables 2 and 3 report best accuracy across five runs.
Lexical Models. We use the fasttext library to implement our bag-of-words models. fast-text is an off-the-shelf text classification library. All lexical models were trained for 5 epochs with 4-grams as the maximum length of word ngrams. We use the default learning rate of 0.1.
C Visualizing Distribution Shifts in SNLI Edited Examples
We are able to visualize confidence distribution shifts for  $\delta$ -NLI before and after editing using a 2D plane with ordered pairs, as in Figure 3, due to the label set containing only two update types—weakener and strengthensener (so, a probability score  $>0.5$  results in the predicted label). Since
the SNLI label set consists of three labels (entailment, neutral, contradiction), we choose to visualize shifts in the confidence distribution before and after editing contexts via ternary plots. For each directional label pair (entailment  $\rightarrow$  {neutral, contradiction}, neutral  $\rightarrow$  {entailment, contradiction}, contradiction  $\rightarrow$  {entailment, neutral}), we plot a heatmap of probabilities, or confidences, in each of the three classes on the simplex with Gaussian smoothing using python-ternary, a ternary plotting package (Harper, 2015). Figures 5, 6, and 7 show these visualizations. We include these plots mainly to help visualize information about the predicted labels of the incorrect examples (Table 2 only reports the accuracy on finer-grained buckets). We observe that for most classes of examples, the SNLI RoBERTa model utilizes the context, and correctly predicts the new induced gold label. However, the {neutral  $\rightarrow$  entailment} class of examples in particular proved difficult for the model, as evidenced by a large chunk of a mass remaining in the neutral corner of the simplex.
D Dataset Limitations
In this work, we choose to explore the role of context with respect to the SNLI and  $\delta$ -NLI datasets. In particular, the proven presence of strong artifacts in SNLI made it an appealing dataset to explore a model's behavior with respect to the utilization of context. We chose to include  $\delta$ -NLI in our analysis, since ultimately, we'd like reasoning systems to operate in complex and dynamic contexts. The ability to be sensitive to shifting contexts and understand when default inferences should be overridden by additional context (i.e more nuanced inference) is both central to our exploration and central to the task of defeasible reasoning itself. Our evaluation set does not include examples sourced from other NLI datasets such as MultiNLI (Williams et al., 2018). It also does not contain datasets across domains, such as MedNLI (Romanov and Shivade, 2018). However, we note that while the datasets may be different, others have shown artifacts present in such datasets (i.e (Herlihy and Rudinger, 2021; Gururangan et al., 2018). Our goal was to utilize datasets containing high amounts of artifacts.
(a) Confidence distribution heatmap for examples with original relation as entailment, edited to induce a neutral relation.
(b) Confidence distribution heatmap for examples with original relation as entailment, edited to induce a contradiction relation.
Figure 5: Confidence distribution heatmaps for SNLI examples before and after editing examples with original relation as entailment in our evaluation set. 5a shows entailment examples edited to induce a neutral relation, and 5b shows examples edited to induce a contradiction relation. Each  $l$  and  $l'$  is shown in the center box. We note that these distributions help visualize the accuracies presented in Table 2.
(a) Confidence distribution heatmap for examples with original relation as neutral, edited to induce an entailment relation. We note that this was the hardest edit category for RoBERTa models to flip, and draw attention to a substantial amount of mass still in the neutral corner of the simplex.
(b) Confidence distribution heatmap for examples with original relation as neutral, edited to induce a contradiction relation.
Figure 6: Confidence distribution heatmaps for SNLI examples before and after editing examples with original relation as neutral in our evaluation set. 6a shows examples edited to induce an entailment relation, and 6b shows examples edited to induce a contradiction relation. Each  $l$  and  $l'$  is shown in the center box. These distributions aid in visualization of the performance metrics reported in Table 2.
(a) Confidence distribution heatmap for examples with original relation as contradiction, edited to induce a neutral relation.
(b) Confidence distribution heatmap for examples with original relation as contradiction, edited to induce an entailment relation. We note that this particular class of edited examples achieved the highest accuracy, reflected in the low density of examples away from the entailment corner of the simplex.
Figure 7: Confidence distribution heatmaps for SNLI examples before and after editing examples with original relation as contradiction in our evaluation set. 7a shows examples edited to induce a neutral relation, and 7b shows examples edited to induce an entailment relation. Each  $l$  and  $l'$  is shown in the center box. These plots help visualize the performance metrics reported in Table 2.