Venue: NAACL 2018

# Modeling Semantic Plausibility by Injecting World Knowledge

 Su Wang\({}^{1,2}\) Greg Durrett\({}^{3}\) Katrin Erk\({}^{1}\)

\({}^{1}\)Department of Linguistics

\({}^{2}\)Department of Statistics and Data Science

\({}^{3}\)Department of Computer Science

The University of Texas at Austin shrekwang@utexas.edu gdurrett@cs.utexas.eduk katrin.erk@mail.utexas.edu

###### Abstract

Distributional data tells us that a man can swallow candy, but not that a man can swallow a paintball, since this is never attested. However both are physically plausible events. This paper introduces the task of semantic plausibility: recognizing plausible but possibly novel events. We present a new crowdsourced dataset of semantic plausibility judgments of single events such as _man swallow painful_. Simple models based on distributional representations perform poorly on this task, despite doing well on selection preference, but injecting manually elicited knowledge about entity properties provides a substantial performance boost. Our error analysis shows that our new dataset is a great testbed for semantic plausibility models: more sophisticated knowledge representation and propagation could address many of the remaining errors.

## 1 Introduction

Intuitively, a _man_ can _swallow_ a _candy_ or _paintball_ but not a _desk_. Equally so, one cannot plausibly _eat_ a _cake_ and then _hold_ it. What kinds of semantic knowledge are necessary for distinguishing a physically plausible event (or event sequence) from an implausible one? _Semantic plausibility_ stands in stark contrast to the familiar _sielectional preference_[50; 47] which is concerned with the _typicality_ of events (Table 1). For example, _candy_ is a typical entity for _man-swallow-*_ but _paintball_ is not, even though both events are plausible physically. Also, some events are physically plausible but are never stated because humans avoid stating the obvious. Critically, semantic plausibility is sensitive to certain properties such as relative object size that are not explicitly encoded by selectional preferences [6]. Therefore, it is crucial that we learn to model these dimensions in addition to using classical distributional signals.

Semantic plausibility is pertinent and crucial in a multitude of interesting NLP tasks put forth previously, such as narrative schema chambers2013, narrative interpolation bowman2016, story understanding Mostafazadeh et al. (2016), and paragraph reconstruction Li and Jurafsky (2017). Existing methods for these tasks, however, draw predominantly (if not only) on distributional data and produce rather weak performance. Semantic plausibility over _subject-verb-object_ triples, while simpler than these other tasks, is a key building block that requires many of the same signals and encapsulates complex world knowledge in a binary prediction problem.

In this work, we show that world knowledge injection is necessary and effective for the semantic plausibility task, for which we create a robust, high-agreement dataset (details in section 3). Employing methods inspired by the recent work on world knowledge propagation through distributional context Forbes and Choi (2017); Wang et al. (2017), we accomplish the goal with minimal effort in manual annotation. Finally, we perform an indepth error analysis to point to future directions of work on semantic plausibility.

\begin{table}
\begin{tabular}{l c c} \hline \hline _man-swallow-*_ & preferred? & plausible? \\ \hline _-candy_ & ✓ & ✓ \\ _-paintball_ & ✗ & ✓ \\ _-desk_ & ✗ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Distinguishing _semantic plausibility_ from _sélectational preference_. _candy_ is selectionally preferred because it is distributionally common patient in the event _man-swallow-*_, as opposed to the bizarre and rarely seen (if at all) patient _painball_. However both are semantically plausible according to our world knowledge: they are small-sized objects that are swallowable by a man. _desk_ is both distributionally unlikely and implausible (i.e. oversized for swallowing).



Related Work

Simple events (i.e. s-v-o) have seen thorough investigation from the angle of selectional preference. While early works are resource-based [10, 23], later work shows that unsupervised learning with distributional data yields strong performance [5, 47], which has recently been further improved upon with neural approaches [69, 80]. Distribution-only models however, as will be shown, fail on the semantic plausibility task we propose.

Physical world knowledge modeling appears frequently in more closely related work. Bagherinezhad:2016 combine computer vision and text-based information extraction to learn the relative sizes of objects; Forbes:2017 crowdsource physical knowledge along specified dimensions and employ belief propagation to learn relative physical attributes of object pairs. Wang:2017 propose a multimodal LDA to learn the definitional properties (e.g. _animal, four-legged_) of entities. Zhang:2017 study the role of common-sense knowledge in natural language inference, which is inherently between-events rather than single-event focused. Prior work does not specifically handles the (single-event) semantic plausibility task and related efforts do not necessarily adapt well to this task, as we will show, suggesting that new approaches are needed.

## 3 Data

To study the semantic plausibility of s-v-o events, specifically _physical_ semantic plausibility, we create a dataset1 through Amazon Mechanical Turk with the following criteria in mind: (i) _Robustness_: Strong inter-annotator agreement; (ii) _Diversity_: A wide range of typical/atypical, plausible/implausible events; (iii) _Balanced_: Equal number of plausible and implausible events.

Footnote 1: Link: [https://github.com/suwangcompling/Modeling-Semantic-Plausibility-NAACL18/tree/master/data](https://github.com/suwangcompling/Modeling-Semantic-Plausibility-NAACL18/tree/master/data).

In creating physical events, we work with a fixed vocabulary of 150 concrete verbs and 450 concrete nouns from Brysbaert:2014's word list, with a concreteness threshold of 4.95 (scale: 0-5). We take the following steps:

* Have Turkers write down plausible or implausible s-v and v-o selections;
* Randomly generate s-v-o triples from collected s-v and v-o pairs;
* Send resulting s-v-o triples to Turkers to filter for ones with high agreement (by majority vote).

(a) ensures diversity and the cleanness of data (compared with noisy selectional preference data collected unsupervised from free text): the Turkers are instructed (with examples) to (i) consider both typical and atypical selections (e.g. _man-swallow-*_ with _candy_ or _paintball_); (ii) disregard metaphorical uses (e.g. _feel-blue_ or _fish-idea_). 2,000 pairs are collected in the step, balancing typical and atypical pairs. In (b), we manually filter error submissions in triple generation. For (c), 5 Turkers provide labels, and we only keep the ones that have \(\geq 3\) majority votes, resulting with 3,062 triples (of 4,000 annotated triples, plausible-implausible balanced), with **100% \(\geq\) 3 agreement, 95% \(\geq\) 4 agreement, and 90% 5 agreement**.

To empirically show the failure of distribution-only methods, we run Van de Cruys:2014's neural net classifier (hereforth nn), which is one of the strongest models designed for selectional preference (Figure 1, left-box). Let \({\mathbf{x}}\) be the concatenation of the embeddings of the three words in an s-v-o triple. The prediction \(P(y|{\mathbf{x}})\) is computed as follows:

\[P(y=1|{\mathbf{x}})=\sigma_{2}(W_{2}\sigma_{1}(W_{1}{\mathbf{x}})) \tag{1}\]

where \(\sigma\) is a nonlinearity, \(W\) are weights, and we use 300D pretrained GloVe vectors [13]. The model achieves an accuracy of 68% (logistic regression baseline: 64%) after fine-tuning, verifying the intuition that distributional data alone cannot satisfactorily capture the semantics of physical plausibility.

## 4 World Knowledge Features

Recognizing that a distribution-alone method lacks necessary information, we collect a set of world knowledge features. The feature types derive from inspecting the high agreement event triples for knowledge missing in distributional selection (e.g. relative sizes in _man-swallow-paintball/desk_). Previously, Forbes:2017 proposed a three level (3-level) featurization scheme, where an object-pair can take 3 

values for, e.g. relative size: \(\{-1,0,1\}\) (i.e. lesser, similar, greater). This method, however, does not explain many cases we observed. For instance, _man-hug-cat/ant_, _man_ is larger than both _cat_ and _ant_, but the latter event is implausible. 3-level is also inefficient: \(k\) objects incur \(O(k^{2})\) elicitations. We thus propose a binning-by-landmark method, which is sufficiently fine-grained while still being efficient and easy for the annotator: given an entity \(n\), the Turker decides to which of the landmarks \(n\) is closest to. E.g., for size, we have the landmarks {_watch, book, cat, person, jeep, stadium_}, in ascending sizes. If \(n=\textit{dog}\), the Turker may put \(n\) in the bin corresponding to _cat_. The features2 are listed with their landmarks as follows:

Footnote 2: We experimented with numerous feature types, e.g. size, temperature, shape, etc. and kept the subset that contributes most substantially to semantic plausibility classification. More details on the feature types in supplementary material ([https://github.com/suwangcompiling/Modeling-Semantic-Plausibility-NAACL18/tree/master/supplementary](https://github.com/suwangcompiling/Modeling-Semantic-Plausibility-NAACL18/tree/master/supplementary)).

* sentence: _rock, tree, ant, cat, chimp, man_.
* mass-count: _milk, sand, pebbles, car_.
* phase: _smoke, milk, wood_.
* size: _watch, book, cat, person, jeep, stadium_.
* weight: _watch, book, dumbbell, man, jeep, stadium_.
* rigidity: _water, skin, leather, wood, metal_.

5 Turkers provide annotations for all 450 nouns, and we obtained **93% \(\geq\) 3 agreement, 85% \(\geq\) 4 agreement, and 79% 5 agreement**.

Our binning is sufficiently granular, which is crucial for semantic plausibility of an event in many cases. E.g. for _man-hug-cat/ant_, _man, cat_ and _ant_ fall in the \(4^{th},3^{rd}\) and \(1^{st}\) bin, which suffices to explain why _man-hug-cat_ is plausible while _man-hug-cant_ is not. Compared to past work Forbes and Choi (2017), it is efficient. Each entity only needs one assignment in comparison to the landmarks to be located in a "global scale" (e.g. from the smallest to the largest objects), and even for extreme granularity, it only takes \(O(k\log k)\) comparisons. It is also intuitive: differences in bins capture the intuition that one can _hug_ smaller objects as long as those objects are not too small.

## 5 Models

We answer two questions: (i) Does world knowledge improve the accuracy of semantic plausibility classification? (ii) Can we minimize effort in knowledge feature annotation by learning from a small amount of training data?

For question (i), we experiment with various methods to incorporate the features on top of the embedding-only nn (Section 3). Our architecture3 is outlined in Figure 1, where we ensemble the nn (left-box) and another feedforward net for features (wk, right-box) to produce the final prediction. For the feature net, the relative physical attributes of the subject-object pair can be encoded in **3-level** (Section 4) or the bin difference (bin-diff) scheme.4 For bin-diff, given the two entities in an s-v-o event (i.e. s, o) _ant_ and _man_, which are in the bins of the landmark _watch_ (i.e. the \(1^{st}\)) and that of _person_ (i.e. the \(4^{th}\)), the pair _ant-man_ gets a bin-diff value of \(1-4=-3\). Exemplifying the featurization function \(f(s,o)\) with size:

Footnote 3: More configuration details in supplementary material.

Footnote 4: We also tried using _bin numbers_ directly, however it does not produce ideal results (classification accuracy between 3-level and bin-diff). Thus for brevity we drop this setup.

\[f_{\text{3-l}}(\textsc{size}(s),\textsc{size}(o))\in\{-1,0,1\} \tag{2}\] \[f_{\textsc{bin}}(\textsc{size}(s),\textsc{size}(o))=\textsc{ bin}(s)-\textsc{bin}(o) \tag{3}\]

Then, given a featurization scheme, we may feed raw feature values (raw vec, for 3-level, e.g. concatenation of -1, 0 or 1 of all feature types, in that order, and in one-hot format), or feature embeddings (embedding, e.g. concatenation of embeddings looked up with feature values). Fi

Figure 1: Model architecture (example input: _man-swallow-paintball_). Left box: Van de Cruys (2014)’s neural net (nn, embeddings only); Right box: world knowledge feature net wk with different modeling choices (Section 5). Only size, weight, rigidity are shown; the rest receive the same treatment. nw + wk: embedding and world knowledge combined.

 

nally, let \({\mathbf{a}_{\texttt{NN}}},{\mathbf{a}_{\texttt{WK}}}\) be the penultimate-layer vectors of NN and Wk (see Figure 1), we affine transform their concatenation to predict label \(\hat{y}\) with argmax on the final softmax layer:

\[\hat{y}=\underset{y}{\text{argmax~{}softmax}}(\sigma(W[{\mathbf{a}_{\texttt{NN}}};{ \mathbf{a}_{\texttt{wK}}}]+{\mathbf{b}})) \tag{4}\]

where \(\sigma\) is a ReLU nonlinearity. We will only report the results from the best-performing model configuration, which has bin-diff + embedding. The model will be listed below as nn**+ wk-gold** (i.e. with gold, Turker-annotated word knowledge features).

For question (ii), we select a data-efficient feature learning model. Following Forbes and Choi (2017) we evaluate the models with 5% or 20% of training data. We experiment with several previously proposed techniques: (a) _label spreading_; (b) _factor graph_; (c) _multi-LDA_. As a baseline we employ a simple but well-tuned logistic regressor (LR). We also initialize the factor graph with this LR, on account of its unexpectedly strong performance.5 Finally, observing that the feature types are inherently ordinal (e.g. size from small to large), we also run _ordinal logistic regression_Adeleke and Adepoju (2010). For model selection we first evaluate the object-pair attribute data collected by Forbes and Choi (2017), 2.5k pairs labeled in the 3-level scheme. We then compared the the LR and Ordinal-LR (our strongest models6 in this experiment) on 10k randomly generated object-pairs from our annotated nouns. The results are summarized in Table 2, where we see (i) 3-level propagation is much easier; (ii) our object-pairs are more challenging, likely due to sparsity with larger vocabulary size; (iii) ordinality information contributes substantially to performance. The model that uses propagated features (w/ Ordinal-LR) will be listed as nn**+ wk-prop**.

Footnote 5: We verified our setup with the authors and they attributed the higher performance of our LR to hyperparameter choices.

## 6 Semantic Plausibility Results

We evaluate the models on the task of classifying our 3,062 s-v-o triples by semantic plausibility (10-fold CV, taking the average over 20 runs with the same random seed). We compare our three models in the 3-level and bin-diff schemes, with nn**+wk-prop** evaluated in 5% and 20% training conditions. The results are outlined in Table 3. Summarizing our findings: (i) world knowledge undoubtedly leads to a strong performance boost (\(\sim\)8%); (ii) bin-diff scheme works much better than 3-level -- it manages to outperform the latter even with much weaker propagation accuracy; (iii) the accuracy loss with propagated features seems rather mild with 20% labeled training and the best scheme.

## 7 Error Analysis

To understand what challenges remain in this task, we run the models above 200 times (10-fold CV, random shuffle at each run), and inspect the top 200 most frequently misclassified cases. The percentage statistics below are from counting the error cases.

In the cases where nn misclassifies while nn**+ wk-gold correctly classifies, 60% relates to size and weight (e.g. missing _man-hug-ant_

\begin{table}
\begin{tabular}{l c c} \hline \hline Models & 5\% & 20\% \\ \hline Label Spreading Zhu et al. (2004) & 0.56 & 0.59 \\ Factor Graph Forbes and Choi (2017) & 0.69 & 0.71 \\ Multi-LDA Wang et al. (2017) & 0.64 & 0.72 \\ \hline Logistic Regression & 0.72 & 0.83 \\ Factor Graph initialized with our LR & 0.72 & 0.84 \\ Ordinal-LR & **0.76** & **0.88** \\ \hline \hline  & 5\% & 20\% \\ \cline{2-3} models & 3-L & bin 3-L & bin \\ \hline Logistic Regression & 0.61 & 0.21 & 0.68 & 0.26 \\ Ordinal-LR & **0.66** & **0.32** & **0.76** & **0.40** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Feature Propagation. Top-table: results on Forbes and Choi (2017)’s 2.5k object pair data; Bottom-table: results on our 10k object pair data.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline models & \multicolumn{3}{c}{accuracy} \\ \hline Random & \multicolumn{3}{c}{0.50} \\ LR baseline & \multicolumn{3}{c}{0.64} \\ NN Van de Cruys (2014) & \multicolumn{3}{c}{0.68} \\
**N + wk-gold** & \multicolumn{3}{c}{**0.76**} \\ \hline  & \multicolumn{3}{c}{5\%} & \multicolumn{3}{c}{20\%} \\ \cline{2-5}
**nn + wk-prob** & \multicolumn{3}{c}{3-l} & bin 3-l & bin \\ \cline{2-5}  & \multicolumn{3}{c}{0.69} & 0.70 & 0.71 & 0.74 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Semantic Plausibility (binary) Classification. The average of 10-fold CV (splitting on the total 3,062 entries). The neural classifier injected with full annotation of world knowledge (i.e. nn**+ wk-gold**) performs substantially better, and the performance retainment is rather strong with propagated features (by Ordinal-LR) from small fractions of gold annotation (i.e. in nn**+ wk-prop**).

 

(bad) or _dog-pull-paper_ (good)). phase takes up 18% (e.g. missing _monkey-puff-smoke_ (good)). This validates the intuition that distributional contexts do not encode these types of world knowledge.

For cases often misclassified by _all_ the models, we observe two main types of errors: (i) data sparsity; (ii) highly-specific attributes.

**Data sparsity (32%)**_man-choke-ant_, e.g., is a singleton big-object-choke-small-object instance, and there are no distributionally similar verbs that can help (e.g. _sutfocate_); For _sun-heat-water_, because the majority of the actions in the data are limited to solid objects, the models tend to predict implausible for whenever a gas/liquid appears as the object.

**Highly-specific attributes (68%)**. "long-tailed" physical attributes which are absent from our feature set are required. To exemplify a few:7

Footnote 7: Percentages calculated with the 68% as the denominator. Full list in supplementary material.

* _edibility_ (21%). *_fly-egg_ (plausible) and *_fly-cup_ (implausible) are hard to distinguish because _egg_ and _cup_ are similar in size/weight/..., however introducing large free-text data to help learn edibility misguides our model to mind selectional preference, causing misleading of other events.
* _natural vs. artificial_ (18%). Turkers often think creating natural objects like _moon_ or _mountain_ is implausible but creating an equally big (but artificial) object like _skyscraper_ is plausible.
* _hollow objects_ (15%). _plane-contain-shell_ and _purse-contain-scissors_ are plausible, but the hollow-object-can-contain-things attribute is failed to be captured.
* _forefoot dexterity_ (5%). _horse-hus-man_ is implausible but _bear-hus-man_ is plausible; For *_snatch-watch_, _girl_ is a plausible subject, but not _pig_. Obviously the dexterity of the forefoot of the agent matters here.

The analysis shows that the task and the dataset highlights the necessity for more sophisticated knowledge featurization and cleverer learning techniques (e.g. features from computer vision, propagation methods with stronger capacity to generalize) to reduce the cost of manual annotation.

## 8 Conclusion

We present the novel task of _semantic plausibility_, which forms the foundation of various interesting and complex NLP tasks in event semantics Bowman et al. (2016); Mostafazadeh et al. (2016); Li and Jurafsky (2017). We collected a high-quality dedicated dataset, showed empirically that the conventional, distribution data only model fails on the task, and that clever world knowledge injection can help substantially with little annotation cost, which lends initial empirical support for the scalability of our approach in practical applications, i.e. labeling little but propagating well approximates performance with full annotation. Grantted that annotation-based injection method does not cover the full spectrum of leverageable world knowledge information (alternative/complementary sources being images and videos, e.g. Bagherinezhad et al. 2016), it is indeed irreplaceable in some cases (e.g. features such as weight or rigidity are not easily learnable through visual modality), and in other cases presents a low-cost and effective option. Finally, we also discovered the limitation of existing methods through a detailed error analysis, and thereby invite cross-area effort (e.g. multimodal knowledge features) in the future exploration in automated methods for semantic plausibility learning.

This research was supported by NSF grant IIS 1523637. Further, this material is based on research sponsored by DARPA under agreement number FA8750-18- 2-0017. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. We acknowledge the Texas Advanced Computing Center for providing grid resources that contributed to these results. We would also like to thank our reviewers for their insightful comments.

## References

* Adeleke and Adepju (2010) K.A. Adeleke and A.A. Adepjou. 2010. Ordinal Logistic Regression Model: An Application to Pregnancy Outcomes. _Journal of Mathematics and Statistics_ 6(3):279-285.
* Bagherinezhad et al. (2016) Hessam Bagherinezhad, Hannaneh Hjishirzi, Yejin Choi, and Ali Farhadi. 2016. Are Elephants Bigger 

Than Butterflies? Reasoning about Sizes of Objects. In _Proceedings of AAAI_.
* Bowman et al. (2016) Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating Sentences from a Continuous Space. In _Proceedings of CoNLL_.
* Brysbaert et al. (2014) M Brysbaert, A.B. Warriner, and V. Kuperman. 2014. Concreteness Ratings for 40 Thousand Generally Known English Word Lemmas. _Behavior Research Methods_ 46:904-911.
* Chambers (2013) Nathanael Chambers. 2013. Event schema induction with a probabilistic entity-driven model. In _Proceedings of EMNLP_.
* Clark and Weir (2001) Stephen Clark and David Weir. 2001. Class-based probability estimation using a semantic hierarchy. In _Proceedings of NAACL_.
* Erk and Pado (2010) Katrin Erk and Sebastian Pado. 2010. Exemplar-Based Models for Word Meaning in Context. In _Proceedings of ACL_.
* Forbes and Choi (2017) Maxwell Forbes and Yejin Choi. 2017. Verb Physics: Relative Physical Knowledge of Actions and Objects. In _Proceedings of ACL_.
* Li and Jurafsky (2017) Jiwei Li and Daniel Jurafsky. 2017. Neural Net Models of Open-domain Discourse Coherence. In _Proceedings of EMNLP_.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In _Proceedings of NAACL_.
* O'Seaghdha (2010) Diarmiuid O'Seaghdha. 2010. Latent variable models of selectional preference. In _Proceedings of ACL_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Embeddings. In _Proceedings of EMNLP_.
* Resnik (1996) Philip Resnik. 1996. Selectional Constraints: An Information-Theoretic Model and Its Computational Realization. _Cognition_ 61:127-159.
* Tilk et al. (2016) Ottokar Tilk, Vera Demberg, Asad Sayeed, Dietrich Klakow, and Stefan Thater. 2016. Event Participant Modeling with Neural Networks. In _Proceedings of EMNLP_.
* Van de Cruys (2014) Tim Van de Cruys. 2014. A Neural Network Approach to Selection Preference Acquisition. In _Proceedings of EMNLP_.
* Wang et al. (2017) Su Wang, Stephen Roller, and Katrin Erk. 2017. Distributional Modeling on a Diet: Learning Word Properties from Text Only. In _Proceedings of IJCNLP_.
* Zhang et al. (2017) Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal Common-sense Inference. _TACL_ 5:379-395.
* Zhu et al. (2004) Dengyong Zhu, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Scholkopf. 2004. Learning with Local and Global Consistency. In _Proceedings of NIPS_.