SlowGuess/NAACL_2022_OCR huggingface dataset:
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 7882-7926.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019), pages 5307-5315.
Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020. BERTnesia: Investigating the capture and forgetting of knowledge in BERT. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 174-183.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past Sesame Street? Sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465-4476.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019b. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
A Appendix: Examples
Relation	Negated Query	Wrong Predictions
IsA	Truth isn’t a [MASK].	[“fact”, “statement”, “concept”, “actuality”]
CapableOf	A doctor cannot [MASK] you.	[“care”]
PartOf	England isn’t part of the [MASK].	[“Europe”]
HasA	Apples don’t have [MASK] inside them.	[“stems”, “seeds”]
UsedFor	A map isn’t for [MASK].	[“navigate”, “locating”, “navigating”, “orienteering”, “information”]
MadeOf	Air doesn’t have [MASK].	[“molecules”]
NotDesires	Soldier does want to be [MASK].	[“die”]
Table 10: ConceptNet relations for constructing the MKR-NQ dataset and their corresponding sample data points.
Template	Query	Wrong Predictions
X is a synonym of Y	boy is a synonym of [MASK].	[“sister”, “girl”]
X is an antonym of Y	boy is an antonym of [MASK].	[“boys”, “brat”, “man”, “boy”, “lad”, ... ]
X is another form of Y	learning is another form of [MASK].	[“forgetting”, “teaching”]
X is the opposite of Y	learning is the opposite of [MASK].	[“knowledge”, “erudition”, “eruditeness”, “learning”]
X is a rephrasing of Y	speaker is a rephrasing of [MASK].	[“microphone”, “ listener”, “addressee”]
X is different from Y	speaker is different from [MASK].	[“loudspeaker”, “transducer”, “talker”, “speaker”, ... ]
Table 11: Templates used to construct the MWR dataset and their sample data points.
Word	Definition
abnormal	not normal; not typical or usual or regular or conforming to a norm; out of ordinary; unusual
afebrile	having no fever
barefaced	with no effort to conceal
career	the particular occupation for which you are trained; a job or occupation that a person does for an extended period
cargo	goods carried by a large vehicle
revise	the act of rewriting something; to review, alter and amend, especially of written material
salary	something that remunerates; a determined yearly amount of money paid to an employee by an employer during a job
Table 12: Examples of word-definition pairs that we used for the meaning-matching task.