SlowGuess/NAACL_2021_OCR huggingface dataset:
Hady ElSahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016. Neural networks for negation scope detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 495-504, Berlin, Germany. Association for Computational Linguistics.
Federico Fancellu, Adam Lopez, Bonnie Webber, and Hangfeng He. 2017. Detecting negation scope is easy, except when it isn't. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 58-63, Valencia, Spain. Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
Md Mosharaf Hossain, Kathleen Hamilton, Alexis Palmer, and Eduardo Blanco. 2020a. Predicting the focus of negation: Model and error analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8389-8401, Online. Association for Computational Linguistics.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020b. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9106-9118. Association for Computational Linguistics.
Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222-231, Brussels, Belgium. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2019. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.
Hao Li and Wei Lu. 2018. Learning with structured representations for negation scope extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 533-539, Melbourne, Australia. Association for Computational Linguistics.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3428-3448. Association for Computational Linguistics.
Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. CoRR, abs/2004.11999.
Roser Morante and Walter Daelemans. 2009. A meta-learning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, CoNLL 2009, Boulder, Colorado, USA, June 4-5, 2009, pages 21-29. ACL.
Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn Penstein Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In COLING, pages 2340-2353. Association for Computational Linguistics.
Hiroshi Noji and Hiroya Takamura. 2020. An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models. CoRR, abs/2004.02451.
Fabio Petroni, Tim Roktaschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In EMNLP/IJCNLP (1), pages 2463-2473. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. CoRR, abs/2003.07082.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, and AiTi Aw. 2019. Negative focus detection via contextual attention mechanism. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2251-2261. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. olmpics - on what language model pre-training captures. CoRR, abs/1912.13283.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Association for Computational Linguistics.
Alex Warstadt and Samuel R. Bowman. 2019. Grammatical analysis of pretrained sentence encoders with acceptability judgments. CoRR, abs/1901.03438.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6381-6387. Association for Computational Linguistics.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In ICLR. OpenReview.net.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Yiyun Zhao and Steven Bethard. 2020. How does BERT's attention change when you fine-tune? an analysis methodology and a case study in negation scope. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4729-4747, Online. Association for Computational Linguistics.
Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2014. Negation focus identification with contextual discourse information. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pages 522-530. The Association for Computer Linguistics.
Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2015. Unsupervised negation focus identification with word-topic graph model. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1632-1636. The Association for Computational Linguistics.
Here are the hyper-parameters used in our fine-tunings.
Task	Epochs	Batch Size	Learning Rate	Weight Decay
Unlikelihood training	5	32	1e-5	N\A
RTE	50	32	2e-5	N\A
SNLI	3	32	1e-5	0.1
MNLI	3	32	2e-5	N\A
Table 6: Hyper-parameters
Algorithm 1 shows the details of further training the BERT base cased model with the unlikelihood and knowledge distillation objectives.
Algorithm 1: Details of the training procedure of BERTNOT. The unlikelihood loss and knowledge distillation loss are first computed with the <sentence A, sentence B> inputs. These inputs are contradictory for the UL loss, and non-contradictory for knowledge distillation (sec 4.2). We use  $\gamma = 0.4$  in our experiments to sum these losses and compute the gradient  $g_{1}$ . Then, we compute the knowledge distillation loss for inputs sampled from Wikipedia. These inputs do not have our reference based format. The parameters are updated again using the gradient from this knowledge distillation loss  $(g_{2})$ .
B Examples of negated sentences
Here are some examples and details of our syntactic negation method.
Original	Negated	Unlikelihood Token
1	That tournament helped demon-strate the high caliber of play in women's soccer.	That tournament did not help demonstrate the high caliber of play in women's soccer.	tournament
2	The attributes of this vector (length and direction) characterize the ro-tation at that point.	The attributes of this vector (length and direction) do not characterize the rotation at that point.	rotation
3	This was broadcast live on Nor-way's main national TV carrier NRK.	This was not broadcast live on Norway's main national TV carrier NRK.	Norway
4	The latter may occur implicitly through the use of a construct like DEFVAR or DEFPARAMETER.	The latter may not occur implicitly through the use of a construct like DEFVAR or DEFPARAMETER.	latter
5	When Arjuna was fighting Karna, the latter's chariot's wheels sank into the ground.	When Arjuna was fighting Karna, the latter's chariot's wheels did not sank into the ground.	wheels
6	It also prohibits or restricts the use of certain accounts held at financial institutions.	It also does not prohibit or restricts the use of certain accounts held at financial institutions.	use
Table 7: Examples of original and negated sentences with the chosen unlikelihood token. Examples 5 and 6 are incorrect negations since sank in example 5 and restricts in example 6 are incorrect word forms in the negated context.
Rule Name	# of Sentences Matched
simple past	315
simple present	295
Imperative	93
present with auxiliary verb	37
past perfect	35
copula statements	34
present with modal	24
already negated with not	14
NPI words (anywhere, anyone, etc)	5
negative words (no, nobody, etc)	4
other	13
Table 8: Number of matches for each rule in our rule set over 930 sentences used to analyze the syntactic negation.
Original Sentence	Rule	Negated Sentence
Nowhere in his confession did he mention the Monteagle letter.	{ "name": "aux before subj", "pattern": "{\$\};tag:/VB.*}/}=A >/ advmod |cc / {word: /never|nobody | no|nothing |nowhere|neither | Never|Nobody|No|Nothing|Nowhere |Neither}/=npiword >/aux.*/({})= B ++ {}=subject) >/nsubj.*/( }=subject ?>obj {tag:/NN.*}/}= object", "actions": [ { "type": "move", "to_move": "B", "anchor": "A", "position": "before" }, { "type": "replace", "token": "", "to_replace": "npiword" } ]}	in his confession he did mention the Monteagle letter.
Many fonts then made the right leg vertical.	{ "name": "simple past", "pattern": "{\$\};cpos:.*/*Tense=Past.*}/}=A >/nsubj|csubj|=E {}=subject ?>obj {tag:/NN.*}/}=object", "actions": [ { "type": "insert", "token": "did", "rel": "AUX", "anchor": "A", "position": "before" }, { "type": "insert", "token": "not", "rel": "ADV", "anchor": "A", "position": "before" }, { "type": "lemmatize" } }	Many fonts then did not make the right leg vertical.
Table 9: Examples of how the syntactic negation augmentation method works. For the first sentence, the matched rule has two actions, move and replace. The move action has moved the token  $\mathbf{B} =$  did before token  $\mathbf{A} =$  mention. The replace action has replaced npiword  $=$  Nowhere with an empty token, which means removing this token. The token object  $=$  letter is chosen as the unlikelyoid token in this sentence.
In the second sentence, the matched rule has three actions, two inserts and one lemmatize action. The insert actions, add the tokens "did not" before  $\mathbf{A} =$  made, and the token  $\mathbf{A} =$  made is replaced with its lemma by the lemmatize action. The token object  $=$  leg is chosen as the unlikely token in the negated sentence.
C Example rules for transforming a sentence into its negation
D Mixing negation unlikelihood training and knowledge distillation with NLI training
In order to reduce the catastrophic forgetting behavior of the model during NLI training, we added the unlikelihood, knowledge distillation and MLM objectives to the original NLI classification objective and trained the model with the same hyper-parameters for the MNLI task. We also trained one version with only the original NLI classification objective and the MLM objective. As the results in table 10 show, this method did not improve the scores for development split and the new split containing negation from Hossain et al. (2020b) for MNLI.
Model	MNLI
dev	w/neg
BERTNOT + UL + KL + MLM + NLI obj	81.17	60.20
BERTNOT + MLM + NLI obj	81.42	62.00
E Supplementary Results
Table 10: Accuracies on original development split (dev) and new split containing negation from Hossain et al. (2020b) (w/neg) for MNLI (matched genres) task.
Model	lr	SQuAD	ConceptNet	T-REx	Google-RE
BERTNOT without reference setup	1e-5	13.86	15.65	29.54	10.29
BERT-large	1e-5	16.83	19.26	30.76	10.93
BERTNOT-large	1e-5	14.19	19.14	32.09	11.02
BERTNOT-large	5e-5	15.18	16.97	30.71	10.62
BERTNOT-large	1e-4	11.55	13.58	28.41	9.25
Table 11: Mean precision at  $k = 1$  ( $p @ 1$ ) for original LAMA queries (higher is better) of BERT with unlikelihood and distillation objectives without references for sentences, BERT-large, and BERT-large with unlikelihood and distillation objectives with different learning rates.
Model	lr	SQuAD	ConceptNet	T-REx	Google-RE
BERTNOT without reference setup	1e-5	5.96	1.34	21.54	3.73
BERT-large	1e-5	7.95	1.67	22.97	4.13
BERTNOT-large	1e-5	8.28	1.87	23.49	4.22
BERTNOT-large	5e-5	8.28	2.20	24.05	4.09
BERTNOT-large	1e-4	4.97	1.47	20.86	3.60
Table 12: Mean top 1 error rate for negated LAMA queries (lower is better) of BERT with unlikelihood and distillation objectives without references for sentences, BERT-large, and BERT-large with unlikelihood and distillation objectives with different learning rates.
As the results in table 12 show, pre-trained BERT-large performs worse than pre-trained BERT-base on negated LAMA queries. We decreased the batch-size to be able to fine-tune BERT-large. As the scores for negated LAMA queries from table 12 show, fine-tuning BERT-large with our method using the same or slightly larger learning rate does not improve the results. We observe a decrease in the mean top 1 error rates for negated LAMA queries when we use a larger learning rate  $(1e - 5)$ , but this also hinders the performance of the model on the original LAMA queries (table 11). This requires some hyper-parameter tuning and further investigation.