SlowGuess/NAACL_2021_OCR huggingface dataset:
We train all models with gradient descent using an Adam optimizer, a learning rate of 2e-5, and a batch size of 128. We train for two epochs over the entire training set of examples with a linear warm-up of the learning rate over the first 10,000 iterations. Fine-tuning RoBERTa takes five hours on a single Nvidia V100 32GB GPU. Fine-tuning CONCEPTINJECT takes 12 hours and CONCEPTMAX 24 hours.
4https://pypi.org/project/mlconjug/
5.1 Training Data
We use English Wikipedia to construct the self-supervised training data. As a relatively clean, definitional corpus, plausibility models trained on Wikipedia have been shown to correlate with human judgements better than those trained on similarly sized corpora (Zhang et al., 2019a; Porada et al., 2019).
We parse a dump of English Wikipedia using the Stanford neural dependency parser (Qi et al., 2018). For each sentence with a direct object, no indirect object, and noun arguments (that are not proper nouns), we extract a training example  $(s,v,o)$ : we take  $s$  and  $o$  to be the lemma of the head of the respective relations (nsubj and obj), and  $v$  to be the lemma of the head of the root verb. This results in some false positives such as the sentence "The woman eats a hot dog." being extracted to the triple woman-eat-dog (Table 1).
We filter out triples that occur less than once and those where a word occurred less than 1,000 times in its respective position. We do not extract the same triple more than 1,000 times so as not to over-sample common events. In total, we extract 3,298,396 triples (representing 538,877 unique events).
6 Predicting Human Plausibility Judgements
We evaluate models by their correlation with human plausibility judgements. Each dataset consists of events that have been manually labelled to be plausible or implausible (Table 3). We use AUC (area under the receiver-operating-characteristic curve) as an evaluation metric which intuitively reflects the ability of a model to discriminate a plausible event from an implausible one.
These datasets contain plausible events that are both typical and atypical. While a distributional model should be able to discriminate typical events given that they frequently occur in text, discriminating atypical events (such as dentist-breathe-helium) is more difficult.
6.1 PEP-3K
PEP-3K, the crowdsourced Physical Event Plausibility ratings of Wang et al. (2018), consists of 3,062 events rated as physically plausible or implausible by five crowdsourced workers. Annotators were instructed to ignore possible metaphorical meanings of an event. We divide the dataset
Topic	Question	Answer
cat	Does it lay eggs?	never
carrot	Can you eat it?	always
cocoon	Can it change shape?	sometimes
clock	Can I touch it?	always
Table 2: Example triples from the 20 Questions commonsense dataset. These are those specific examples that contain a simple question with a single s-v-o triple and no modifiers.
PEP-3K	chef-bake-juniper	✓
dog-close-door	✓
fish-throw-elephant	✗
marker-fuse-house	✗
20Q	whale-breathe-air	✓
wolf-wear-collar	✓
cat-hatch-egg	✗
armrest-breathe-air	✗
Table 3: Representative examples taken from the validation splits of the two plausibility evaluation datasets, PEP-3K and 20Q. For simplicity, we present human judgments as plausible (√) or implausible (✗). Details are provided in §6.
equally into a validation and test set following the split of Porada et al. (2019).
To evaluate on this dataset, we make the assumption that all events labeled physically plausible are necessarily more plausible than all those labeled physically implausible.
6.2 20Q
The 20 Questions commonsense dataset<sup>5</sup> is a collection of 20 Questions style games played by crowdsourced workers. We format this dataset as plausibility judgments of s-v-o triples similar to PEP-3K.
In the game 20 Questions, there are two players—one who knows a given topic, and the other who is trying to guess this topic by asking questions that have a discrete answer. The dataset thus consists of triples of topics, questions, and answers where the answer is one of: always, usually, sometimes, rarely, or never (Table 2).
We parse the dataset using the Stanford neural dependency parser (Qi et al., 2018). We then extract questions that contain a simple s-v-o triple
5https://github.com/allenai/ twentyquestions
Model	PEP-3K	20Q	Avg.
n-gram	.51	.52	.52
GloVe+MLP	.55	.52	.53
RoBERTaZero-shot	.56	.57	.56
RoBERTa	.64	.67	.66
CONCEPTINJECT	.64	.66	.65
CONCEPTMAX	.67	.74	.70
with no modifiers where either the subject or object is a third person singular pronoun. We replace this pronoun with the topic, and otherwise replace any occurrence of a personal pronoun with the word "person." We filter out examples where only two of three annotators labelled the likelihood as never. Finally, we take events labelled "never" to be less plausible than all other events. This process results in 5,096 examples equally divided between plausible and implausible. We split examples into equal sized validation and test sets.
6.3 Quantitative Results
Despite making a discrete decision about the right level of abstraction, CONCEPTMAX has higher AUC on both evaluation sets as compared to CONCEPTINJECT and the vanilla RoBERTa baseline (Table 4). The fact that the CONCEPTMAX model aligns with human judgments more than the baselines supports the hypothesis that conceptual consistency improves plausibility estimates.
CONCEPTINJECT performs similarly to the RoBERTa baseline even though this model is aware of the WordNet hierarchy. We hypothesize that the self-supervised learning signal does not incentivize use of this hierarchical information in a way that would increase correlation with plausibility judgements. We do find that CONCEPTINJECT attends to the hypernym chain, however, by qualitatively observing the self-attention weights.
All fine-tuned RoBERTa models correlate better with plausibility judgements than the RoBERTaZero-shot baseline, and the n-gram baseline performs close to random—this is perhaps to be expected, as very few of the evaluation triples occur in our Wikipedia training data.
Table 4: Test set results for predicting human plausibility judgements. Performance is evaluated with AUC with respect to the ground-truth, manually labeled plausibility ratings.
Model	PEP-3K	20Q
CCΔ	LER	CCΔ	LER
n-gram	.06	.50	.07	.50
GloVe+MLP	.03	.61	.03	.49
RoBERTaZero-shot	.13	.70	.12	.65
RoBERTa	.09	.52	.08	.51
CONCEPTINJECT	.08	.52	.07	.51
CONCEPTMAX	.02	.00	.02	.00
Table 5: Evaluation of inconsistency. CC $\Delta$  describes the degree to which sequences of estimates across a hypernym chain diverge from a concave sequence. LER describes how often conceptual abstractions are local extrema with respect to plausibility.
To better understand the performance of these models, we manually inspect 100 examples from each dataset. We find that RoBERTa rarely assigns a high score to a nonsensical event (although this does occur in five cases, such as turtle-climb-wind and person-throw-library). RoBERTa also rarely assigns a low score to a seemingly typical event, although this is somewhat more common (in cases such as kid-use-handbag and basket-hold-clothes, for example). This finding confirms our expectation that discerning the typical and nonsensical should be relatively easy for a distributional model.
Examples not at the extremes of plausibility are harder to categorize; however, one common failure seems to be when the plausibility of an event hinges on the relative size of the subject and object, such as in the case of dog-throw-whale. This finding is similar to the limitations of static word embeddings observed by Wang et al. (2018).
7 Consistency Evaluation
For every event  $e$  in the evaluation sets of human plausibility judgments (§6), we disambiguate  $e$  using BERT-WSD and then calculate models' estimates for the plausibility of every possible abstraction of  $e$  (Figure 4). Based on these estimates, we can analyze the consistency of each model across abstractions.
7.1 Quantitative Results
We use our proposed metrics of consistency (§3.2) to evaluate the extent to which models' estimates are consistent across a hypernym chain (Table 5).
(a) RoBERTa
(b) CONCEPTINJECT
Figure 4: Outputs across conceptual abstractions for the event kid-like-marmalade from the 20Q dataset. This event is taken to be relatively plausible as the ground-truth label was "usually."
(c) CONCEPTMAX
RoBERTaZero-shot, which correlates with plausibility the least of the RoBERTa models, has by far the highest inconsistency.
The fine-tuned RoBERTa and CONCEPTINJECT estimates are also largely inconsistent by our metrics. For these models, half of all estimates are a local extrema in the lexical hierarchy. As shown in Figure 4, the space of plausibility estimates is rigid for these models, and most estimates are a local extremum with respect to the plausibility of the subject or object of the event.
CONCEPTMAX is almost entirely consistent by these metrics, which is to be expected as this model makes use of the same WordNet hierarchy that we are using for evaluation. We also evaluated consistency using the longest rather than the shortest hypernym chain in WordNet, but did not find a significant change in results. This is likely because for the consistency evaluation we are using the hypernym chains that have been filtered as described in §3.1.
7.2 Qualitative Results
We qualitatively evaluate the consistency of models by observing the matrix of plausibility estimates for all abstractions as show in Figure 4.
In agreement with our quantitative metrics, we observe that RoBERTa estimates are often inconsistent in that they vary greatly between two abstractions that have similar plausibility. Surprisingly, however, it is also often the case that RoBERTa estimates are similar or identical between abstractions. In some cases, this may be the result of the model being invariant to the subject or object of a given event.
We also observe the individual examples with the highest CC. In these cases, it does appear that the variance of model estimates is unreasonable. In contrast, LER is sometimes high for an example where the estimates are reasonably consistent. This is a limitation of the LER metric not taking into account the degree of change between estimates.
Finally, we observe that the BERT-WSD sense is often different from what an annotator primed to rate plausibility would assume. For example, in the case of dog-cook-turkey, BERT-WSD takes dog to be a hyponym of person. While this is reasonable in context, it results in a different plausibility than that annotated.
While the state of the art in modeling plausibility has improved in recent years, models still fall short of human ability. We show that model estimates are inconsistent with respect to a lexical hierarchy: they correlate less with human judgments as compared to model estimates that are forced to be consistent, and they do not satisfy our intuitively defined quantitative measures of consistency.
In addition, we show that simply injecting lexical knowledge into a model is not sufficient to correct this limitation. Conceptual consistency appears to require a more discrete, hierarchical bias.
Interesting questions for future work are: 1) can we design a non-monotonic, consistent model of plausibility that better correlates with human judgements? 2) Can we induce a hierarchy of abstractions rather than using a manually created lexical hierarchy?
We would like to thank Ali Emami and Abhilasha Ravichander for useful discussions and comments, as well as the anonymous reviewers for their insightful feedback. This work is supported by funding from Microsoft Research and resources from Compute Canada. The last author is supported by the Canada CIFAR AI Chair program.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In International Conference on Learning Representations.
Stephen Clark and David Weir. 2002. Class-based probability estimation using a semantic hierarchy. Computational Linguistics, 28(2):187-206.
Ido Dagan and Alon Itai. 1990. Automatic processing of large corpora for the resolution of anaphora references. In Proceedings of the 13th Conference on Computational Linguistics - Volume 3, COLING '90, page 330-332, USA. Association for Computational Linguistics.
Pradeep Dasigi, Waleed Ammar, Chris Dyer, and Edward Hovy. 2017. Ontology-aware token embeddings for prepositional phrase attachment. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2089-2098, Vancouver, Canada. Association for Computational Linguistics.
Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723-763.
Martha Walton Evens. 1975. Semantic Representations for Question-Answering Systems. Ph.D. thesis, Northwestern University.
Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third Blackbox NLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163-173, Online. Association for Computational Linguistics.
Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245-288.
Yu Gong, Kaiqi Zhao, and Kenny Q. Zhu. 2016. Representing verbs as argument concepts. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 2615-2621. AAAI Press.
Emily Goodwin, Koustuv Sinha, and Timothy J. O'Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958-1969, Online. Association for Computational Linguistics.
Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmle. 2011. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In SemEval@NAACL-HLT.
Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC '13, pages 25-30, New York, NY, USA. ACM.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Mutian He, Y. Song, Kun Xu, and Y. Dong. 2020. On the role of conceptualization in commonsense knowledge graph construction. ArXiv, abs/2003.03239.
Jerry R. Hobbs. 1978. Resolving pronoun references. *Lingua*, 44(4):311-338.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837-4842, Florence, Italy. Association for Computational Linguistics.
Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 970-976, Denver, Colorado. Association for Computational Linguistics.
Hang Li and Naoki Abe. 1998. Generalizing case frames using a thesaurus and the MDL principle. Computational Linguistics, 24(2):217-244.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):2901-2908.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Diana McCarthy and John Carroll. 2003. Disambiguating nouns, verbs, and adjectives using automatically acquired selectional preferences. Comput. Linguist., 29(4):639-654.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Diarmuid O Seaghdha. 2010. Latent variable models of selectional preference. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 435-444, Uppsala, Sweden. Association for Computational Linguistics.
Diarmuid O Seaghdha and Anna Korhonen. 2012. Modelling selectional preferences in a lexical hierarchy. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 170-179, Montreal, Canada. Association for Computational Linguistics.
Patrick Pantel, Rahul Bhagat, Bonaventura Coppola, Timothy Chklovski, and Eduard Hovy. 2007. ISP: Learning inferential selectional preferences. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 564-571, Rochester, New York. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Ian Porada, Kaheer Suleman, and Jackie Chi Kit Cheung. 2019. Can a gorilla ride a camel? learning semantic plausibility from text. In Proceedings of the
First Workshop on Commonsense Inference in Natural Language Processing, pages 123-129, Hong Kong, China. Association for Computational Linguistics.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160-170, Brussels, Belgium. Association for Computational Linguistics.
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88-102, Barcelona, Spain (Online). Association for Computational Linguistics.
P. Resnik. 1996. **Selectional constraints:** an information-theoretic model and its computational realization. Cognition, 61(1-2):127-159.
Philip Resnik. 1997. Selectional preference and sense disambiguation. In Tagging Text with Lexical Semantics: Why, What, and How?
Philip Stuart Resnik. 1993. Selection and Information: A Class-Based Approach to Lexical Relationships. Ph.D. thesis, University of Pennsylvania.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4661-4675, Online. Association for Computational Linguistics.
Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863-6870, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 26-35, Doha, Qatar. Association for Computational Linguistics.
Benjamin Van Durme, Phillip Michalak, and Lenchart Schubert. 2009. Deriving generalized knowledge from corpora using WordNet abstraction. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 808-816, Athens, Greece. Association for Computational Linguistics.
Su Wang, Greg Durrett, and Katrin Erk. 2018. Modeling semantic plausibility by injecting world knowledge. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 303-308, New Orleans, Louisiana. Association for Computational Linguistics.
Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. Probing neural language models for human tacit assumptions. In Cognitive Science Society.
Yorick Wilks. 1975. A preferential, pattern-seeking, semantics for natural language inference. Artificial Intelligence, 6(1):53-74.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, Remi Louf, Patrick von Platen, Tim Rault, Yacine Jernite, Teven Le Scao, Sylvain Gugger, Julien Plu, Clara Ma, Canwei Shen, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Q. Zhu. 2012. Probase: A probabilistic taxonomy for text understanding. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, page 481-492, New York, NY, USA. Association for Computing Machinery.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31-40, Florence, Italy. Association for Computational Linguistics.
Boon Peng Yap, Andrew Koh, and Eng Siong Chng. 2020. Adapting BERT for word sense disambiguation with gloss selection objective and example sentences. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 41-46, Online. Association for Computational Linguistics.
Fabio Massimo Zanzotto, Marco Pennacchiotti, and Maria Teresa Pazienza. 2006. Discovering asymmetric entailment relations between verbs using selectional preferences. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 849-856, Sydney, Australia. Association for Computational Linguistics.