Venue: ACL 2020

# Are Natural Language Inference Models ImpPressive?

Learning Implicature and Presupposition

Paloma Jeretic\({}^{1}\), Alex Warstadt\({}^{1}\), Suvrat Bhooshan\({}^{2}\), Adina Williams\({}^{2}\)

Equal Contribution 

###### Abstract

Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer whether a sentence entails another. However, the ability of NLI models to make pragmatic inferences remains understudied. We create an Implicature and Presupposition diagnostic dataset (ImpPres), consisting of \(>\)25k semi-automatically generated sentence pairs illustrating well-studied pragmatic inference types. We use ImpPres to evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI Williams et al. (2018) learn to make pragmatic inferences. Although MultiNLI appears to contain very few pairs illustrating these inference types, we find that BERT learns to draw pragmatic inferences. It reliably treats scalar implicates triggered by "some" as entailments. For some presupposition triggers like _only_, BERT reliably recognizes the presupposition as an entailment, even when the trigger is embedded under an entailment canceling operator like negation. BOW and InferSent show weaker evidence of pragmatic reasoning. We conclude that NLI training encourages models to learn some, but not all, pragmatic inferences.

## 1 Introduction

One of the most foundational semantic discoveries is that systematic rules govern the inferential relationships between pairs of natural language sentences Aristotle (_De Interpretatione_, Ch. 6). In natural language processing, Natural Language Inference (NLI)--a task whereby a system determines whether a pair of sentences instantiates in an entailment, a contradiction, or a neutral relation--has been useful for training and evaluating models on sentential reasoning. However, linguists and philosophers now recognize that there are separate semantic and pragmatic modes of reasoning Grice (1975); Clark (1996); Beaver (1997); Horn and Ward (2004); Potts (2015), and it is not clear which of these modes, if either, NLI models learn. We investigate two pragmatic inference types that are known to differ from classical entailment: scalar implicatures and presuppositions. As shown in Figure 1, implicatures differ from entailments in that they can be denied, and presuppositions differ from entailments in that they are not canceled when placed in entailment-cancelling environments (e.g., negation, questions).

To enable research into the relationship between NLI and pragmatic reasoning, we introduce ImpPres, a fine-grained NLI-style diagnostic test dataset for probing how well NLI models perform implicative and presupposition. Containing 25.5K sentence pairs illustrating key properties of these pragmatic inference types, ImpPres is automatically generated according to linguist-crafted templates, allowing us to create a large, lexically varied, and well controlled dataset targeting specific

Figure 1: Illustration of key properties of classical entailments, implicatures, and presuppositions. Solid arrows indicate valid commonsense entailments, and arrows with X’s indicate lack of entailment. Dashed arrows indicate follow up statements with the addition of _in fact_, which can either be acceptable (marked with ‘✓’) or unacceptable (marked with ’✓’).



instances of both types.

We first investigate whether presuppositions and implicatures are present in NLI models' training data. We take MultiNLI Williams et al. (2018) as a case study, and find it has few instances of pragmatic inference, and almost none that arise from specific lexical triggers (see SS4). Given this, we ask whether training on MultiNLI is sufficient for models to _generalize_ about these largely absent commonsense reasoning types. We find that generalization is possible: the BERT NLI model shows evidence of pragmatic reasoning when tested on the implicature from _some_ to _not all_, and the presuppositions of certain triggers (_only_, cleft existence, possessive existence, questions). We obtain some negative results, that suggest that models like BERT still lack a sophisticated enough understanding of the meanings of the lexical triggers for implicative and presupposition (e.g., BERT treats several word pairs as synonyms, e.g., most notably, _or_ and _and_).

Our contributions are: (i) we provide a new diagnostic test set to probe for pragmatic inferences, complete with linguistic controls, (ii) to our knowledge, we present the first work evaluating deep NLI models on specific pragmatic inferences, and (iii) we show that BERT models can perform some types of pragmatic reasoning very well, even when trained on NLI data containing very few explicit examples of pragmatic reasoning. We publicly release all ImpPres data, models evaluated, annotations of MultiNLI, and the scripts used to process data.1

Footnote 1: github.com/facebookresearch/ImpPres

## 2 Background: Pragmatic Inference

We take pragmatic inference to be a relation between two sentences relying on the utterance context and the conversational goals of interlocutors. Pragmatic inference contrasts with semantic entailment, which instead captures the logical relationship between isolated sentence meanings Grice (1975); Stalnaker (1974). We present **implications** and **presupposition** inferences below.

### Implicature

Broadly speaking, implicatures contrast with entailments in that they are inferences suggested by the speaker's utterance, but not included in its literal Grice (1975). Although there are many types of implicatures we focus here on **scalar implications**. Scalar implicatures are inferences, often optional,2 which can be drawn when one member of a memorized lexical scale (e.g., \(\langle{\textit{some}},{\textit{all}}\rangle\)) is uttered (see SS6.1). For example, when someone utters _Jo ate some of the cake_, they suggest that _Jo didn't eat all of the cake_, (see Figure 1 for more examples). According to Neo-Gricean pragmatic theory Horn (1989); Levinson (2000), the inference _Jo didn't eat all of the cake_ arises because _some_ has a more informative lexical alternative _all_ that could have been uttered instead. We expect the speaker to make the most informative true statement:3 as a result, the listener should infer that a stronger statement, where _some_ is replaced by _all_, is false.

Footnote 2: Implicate computation can depend on the cooperativity of the speakers, or on any aspect of the context of utterance (lexical, syntactic, semantic/pragmatic, discourse). See Degen (2015) for a study of the high variability of implicature computation, and the factors responsible for it.

Footnote 3: This follows if we assume that speakers are cooperative Grice (1975) and knowledgeable Gazdar (1979).

Implicatures differ from entailments (and, as we will see, presuppositions; see Figure 1) in that they are **deniable**, i.e., they can be explicitly negated without resulting in a contradiction. For example, someone can utter _Jo ate some of the cake_, followed by _In fact, Jo ate all of it_. In this case, the implicature (i.e., _Jo didn't eat all the cake_ from above) has been denied. We thus distinguish implicated meaning from literal, or logical, meaning.

### Presupposition

Presuppositions of a sentence are facts that the speaker takes for granted when uttering a sentence Stalnaker (1974); Beaver (1997). Presuppositions are generally associated with the presence of certain expressions, known as **presupposition triggers**. For example, in Figure 1, the definite de

\begin{table}
\begin{tabular}{l l} \hline Type & Example \\ \hline Trigger & Jo’s cat yawned. \\ Presupposition & Jo has a cat. \\ \hline Negated Trigger & Jo’s cat didn’t yawn. \\ Modal Trigger & It’s possible that Jo’s cat yawned. \\ Interrog. Trigger & Did Jo’s cat yawn? \\ Cond. Trigger & If Jo’s cat yawned, it’s OK. \\ \hline Negated Prsp. & Jo doesn’t have a cat. \\ Neutral Prsp. & Amy has a cat. \\ \hline \end{tabular}
\end{table}
Table 1: Sample generated presupposition paradigm. Examples adapted from the ‘change-of-state’ dataset.

 

scription _the cake_ triggers the presupposition that there is a cake Russell1905. Other examples of presupposition triggers are shown in Table 1.

Presuppositions differ from other inference types in that they generally **project** out of operators like questions and negation, meaning that they remain valid inferences even when **embedded under** these operators Karttunen1973. The inference that there is a cake survives even when the presupposition trigger is in a question Did Jordan eat some of the cake?), as shown in Figure 1. However, in questions, classical entailments and implicatures disappear. Table 1 provides examples of triggers projecting out of several **entailment canceling operators**: negation, modals, interrogatives, and conditionals.

It is necessary to clarify in what sense presupposition is a pragmatic inference. There is no consensus on whether presuppositions should be considered part of the semantic content of expressions see stalnaker1974,heim1983, for opposing views). However, presuppositions may come to be inferred via accommodation, a pragmatic process by which a listener infers the truth of some new fact based on its being presupposed by the speaker Lewis1979. For instance, if Jordan tells Harper that the King of Sweden wears glasses, and Harper did not previously know that Sweden has a king, they would learn this fact by accommodation. With respect to NLI, any presupposition in the premise (short of world knowledge) will be new information, and therefore accommodation is necessary to recognize it as entailed.

## 3 Related Work

NLI has been framed as a commonsense reasoning task Dagan et al.2006, Manning2006. One early formulation of NLI defines "entailment" as holding for sentences \(p\) and \(h\) whenever, "typically, a human reading \(p\) would infer that \(h\) is most likely true...[given] common human understanding of language [and] common background knowledge" Dagan et al.2006. Although this sparked debate regarding the terms _inference_ and _entailment_--and whether an adequate notion of "inference" could be defined Zaanen et al.2005, Manning2006, Crouch et al.2006]--in recent work, a commonsense formulation of "inference" is widely adopted Bowman et al.2015, Williams et al.2018 largely because it facilitates untrained annotators' participation in dataset creation.

NLI itself has been steadily gaining in popularity; many datasets for training and/or testing systems are now available including: FraCaS Cooper 

ger a presupposition that its complement clause is true. White et al. (2018) and Jiang and de Marneffe (2019) find that LSTMs trained on a gradient event factuality prediction task on these respective datasets make systematic errors. Turning to implicitures, Degen (2015) introduces a dataset measuring the strength of the implicature from _some_ to _not all_ with crowd-sourced judgments. Schuster et al. (2020) find that an LSTM with supervision on this dataset can predict human judgments well. These resources all differ from ImpPres in two respects: First, their empirical scopes are all somewhat narrower, as all these datasets focus on only a single class of presupposition or implicature triggers. Second, the use of gradient judgments makes it non-trivial to use these datasets to evaluate NLI models, which are trained to make categorical predictions about entailment. Both approaches have advantages, and we leave a direct comparison for future work.

Outside the topic of sentential inference, Rashkin et al. (2018) propose a new task where a model must label actor intents and reactions for particular actions described using text. Cianfloe et al. (2018) create sentence-level adverbial pre-supposition datasets and train a binary classifier to detect contexts in which presupposition triggers (e.g., _too_, _again_) can be used.

## 4 Annotating MultiNLI for Pragmatics

In this section, we present results of an annotation effort that show that MultiNLI contains very little explicit evidence of pragmatic inferences of the type tested by ImpPres. Although Williams et al. (2018) report that 22% of the MultiNLI development set sentence pairs contain lexical triggers (such as _regret_ or _stopped_) in the premise and/or hypothesis, the mere presence of presupposition-triggering lexical items in the data does not show that MultiNLI contains evidence that presuppositions are entailments, since the sentential inference may focus on other types of information. To address this, we randomly selected 200 sentence pairs from the MultiNLI matched development set and presented them to three expert annotators with a combined total of 17 years of training in formal semantics and pragmatics.4 Annotators answered the following questions for each pair: (1) are the sentences \(P\) and \(H\) related by a presupposition/implicature relation (entails/is entailed by, negated or not); (2) what subtype of inference (e.g., existence presupposition, _\(\langle{some}\text{, }all\rangle\)_ implicature); (3) is the presupposition trigger embedded under an entailment-cancelling operator?

Footnote 4: The full annotations are on the ImpPres repository.

Agreement among annotators was low, suggesting that few MultiNLI pairs are paradigmatic cases of implicatures or presuppositions. We found only 8 presupposition pairs and 3 implicature pairs on which two or more annotators agreed. Moreover, we found only one example illustrating a particular inference type tested in ImpPres (the presupposition of possessed definites). All others were tagged as existence presuppositions and conversational implications (i.e. loose inferences dependent on world knowledge). The union of annotations was much larger: 42% of examples were identified by at least one annotator as a presupposition or implicature (51 presuppositions and 42 implicatures, with 10 sentences receiving divergent tags). However, of these, only 23 presuppositions and 19 implicatures could reliably be used to learn pragmatic inference (in 14 cases, the given tag did not match the pragmatic inference, and in 27 cases, computing the inference did not affect the relation type). Again, the large majority of implicatures were conversational, and most presuppositions were existential, and generally not linked to particular lexical triggers (e.g., topic marking).

We conclude that the MultiNLI dataset at best contains some evidence of loose pragmatic reasoning based on world knowledge and discourse structure, but almost no explicit information relevant to lexically triggered pragmatic inference, which is of the type tested in this paper.

## 5 Methods

Data Generation.ImpPres consists of semi-automatically generated pairs of sentences with NLI labels illustrating key properties of implicatures and presuppositions. We generate ImpPres using a codebase developed by Warstadt et al. (2019a) and significantly expanded for the BLiMP dataset (Warstadt et al., 2019b). The codebase, including our scripts and documentation, are publicly available.5 Each sentence type in ImpPres is generated according to a template that specifies the linear order of the constituents in the sentence. The constituents are sampled from a vocabulary of over 3000 lexical items annotated with grammatical features needed to ensure morphological, 

syntactic, and semantic well-formedness. All sentences generated from a given template are structurally analogous up to the specified constituents, but may vary in sub-constituents. For instance, if the template calls for a verb phrase, the generated constituent may include a direct object or complement clause, depending on the argument structure of the sampled verb. See SS6.1 and 7.1 for descriptions of the sentence types in the implicature and presupposition data.

Generating data lets us control the lexical and syntactic content so that we can guarantee that the sentence pairs in ImpPres evaluate the desired phenomenon (see Ettinger et al., 2016, for related discussion). Furthermore, the codebase we use allows for greater lexical and syntactic variety than in many other template datasets (see discussion in Warstadt et al., 2019). One limitation of this methodology is that generated sentences, while generally grammatical, often describe highly unlikely scenarios, or include low frequency combinations of lexical items (e.g., _Sabrina only reveals this pasta_). Another limitation is that generated data is of limited use for training models, since it contains simple regularities that supervised classifiers may learn to exploit. Thus, we create ImpPres solely for the purpose of evaluating NLI models trained on standard datasets like MultiNLI.

Models.Our experiments evaluate NLI models trained on MultiNLI and built on top of three sentence encoding models: a bag of words (BOW) model, InferSent Conneau et al. (2017), and BERT-Large Devlin et al. (2019). The BOW and InferSent models use 300D GloVe embeddings as word representations Pennington et al. (2014). For the BOW baseline, word embeddings for premise and hypothesis are separately summed to create sentence representations, which are concatenated to form a single sentence-pair representation which is fed to a logistic regression softmax classifier. For the InferSent model, GloVe embeddings for the words in premise and hypothesis are respectively fed into a bidirectional LSTM, after which we concatenate the representations for premise and hypothesis, their difference, and their element-wise product Mou et al. (2016). BERT is a multilayer bidirectional transformer pretrained with the masked language modelling and next sequence prediction objectives, and finetuned on the MultiNLI dataset. We concatenate the premise and hypothesis after a special [CLS] token and separated them with the [SEP] token. The BERT representation for the [CLS] token is fed into classifier. We use Huggingface's pre-trained BERT trained on Toronto books Zhu et al. (2015).6

Footnote 6: github.com/huggingface/pytorch-pretrained-BERT/

The BOW and InferSent models have development set accuracies of 49.6% and 67.6%. The development set accuracy for BERT-Large on MultiNLI is 86.6%, similar to the results achieved by Devlin et al. (2019), but somewhat lower than state-of-the-art (currently 90.8% on test from the ensembled RoBERTa model with long pretraining optimization, Liu et al. 2019).

## 6 Experiment 1: Scalar Implicatures

### Scalar Imperlucture Datasets

The scalar implicative portion of ImpPres includes six datasets, each isolating a different scalar implicature trigger from six types of lexical scales (of the type described in SS2): determiners \(\langle{some}\), all\(\rangle\), connectives \(\langle{or}\), and\(\rangle\), modals \(\langle{can}\), have to\(\rangle\), numerals \(\langle{2,3}\rangle\), \(\langle{10,100}\rangle\), scalar adjectives, and

\begin{table}
\begin{tabular}{p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt} p{42.7pt}} \hline \hline
**Premise** & **Hypothesis** & **Relation type** & **Logical label** & **Pragmatic label** & **Item type** \\ \hline _some not all_ & _not all_ & implicature (+ to –) & neutral & entailment & target \\ _not all_ & _some_ & implicature (– to +) & neutral & entailment & target \\ \hline _some all_ & _all_ & negated implicature (+) & neutral & contradiction & target \\ _all_ & _some_ & reverse negated implicature (+) & entailment & contradiction & target \\ _not all_ & _none_ & negated implicature (–) & neutral & contradiction & target \\ _none_ & _not all_ & reverse negated implicature (–) & entailment & contradiction & target \\ \hline _all_ & _none_ & opposite & contradiction & contradiction & control \\ _none_ & _all_ & opposite & contradiction & contradiction & control \\ _some_ & _none_ & negation & contradiction & contradiction & control \\ _none_ & _some_ & negation & contradiction & contradiction & control \\ _all_ & _not all_ & negation & contradiction & contradiction & control \\ _not all_ & _all_ & negation & contradiction & contradiction & control \\ \hline \end{tabular}
\end{table}
Table 2: Paradigm for the scalar implicating datasets, with \(\langle{some}\), all\(\rangle\) as an example.

 

verbs, e.g., \(\langle\)_good, excellent_\(\rangle\), \(\langle\)_run, sprint_\(\rangle\). Examples pairs of each implicature trigger can be found in Table 4 in the Appendix. For each type, we generate 100 paradigms, each consisting of 12 unique sentence pairs, as shown in Table 2.

The six target sentence pairs comprise two main relation types: 'implicative' and 'negated implicature'. Pairs tagged as 'implicature' have a premise that implicates the hypothesis (e.g., _some_ and _not all_). For 'negated implicature', the premise implicates the negation of the hypothesis (e.g., _some_ and _all_), or vice versa (e.g., _all_ and _some_). Six control pairs are logical contradictions, representing either scalar 'opposites' (e.g., _all_ and _none_), or 'negations' (e.g., _not all_ and _all_; _some_ and _none_), probing the models' basic grasp of negation.

As mentioned in SS2.1, implicature computation is variable and dependent on the context of utterance. Thus, we anticipate two possible rational behaviors for a MultiNLI-trained model tested on an implicature: (a) be pragmatic, and compute the implicature, concluding that the premise and hypothesis are in an 'entailment' relation, (b) be logical, i.e., consider only the literal content, and not compute the implicature, concluding they are in a 'neutral' relation. Thus, we measure both possible conclusions, by tagging sentence pairs for scalar implicature with two sets of NLI labels to reflect the behavior expected under "logical" and "pragmatic" modes of inference, as shown in Table 2.

### Implicatures Results & Discussion

We first evaluate model performance on the controls, shown in Figure 2. Success on these controls is a necessary condition for us to conclude that a model has learned the basic function of negation (_not, none_, _neither_) and the scalar relationship between terms like _some_ and _all_. We find that BERT performs at ceiling on control conditions for all implicature types, in contrast with InferSent and BOW, whose performance is very variable. Since only BERT passes all controls, its results on the target items are most interpretable. Full results for all models and target conditions by implicature trigger are in Figures 8-13 in the Appendix.

For connectives, scalar adjectives and verbs, the BERT model results correspond neither to the hypothesized pragmatic nor logical behavior. In fact, for each of these subdatasets, the results are consistent with a treatment of scalemates (e.g., _and_ and _or_; _good_ and _excellent_) as synonyms, e.g. it evaluates the 'negated implicature' sentence pairs as 'entailment' in both directions. This reveals a coarse-grained knowledge of these meanings that lacks information about asymmetric informativity relations between scalemates. Results for modals (_can_ and _have to_) are split between the three labels, not showing any predicted logical or pragmatic pattern. We conclude that BERT has insufficient knowledge of the meaning of these words.

In addition to pragmatic and logical interpretations, numerals can also be interpreted as exact cardinalities. We thus predict three different behaviors: logical "at least \(n\)", pragmatic "at least \(n\)", and "exactly \(n\)". We observe that results are inconsistent: neither the "exactly" nor "at least" interpretations hold across the board.

Figure 3: Results on Target Conditions (Implications)

Figure 2: Results on Controls (Implicatures)



For the determiner dataset (_some-all_), Figure 4 breaks down the results by condition and shows that BERT behaves as though it performs pragmatic and logical reasoning in different conditions. Overall, it predicts a pragmatic relation more frequently (55% vs. 36%), and only 9% of results are consistent with neither mode of reasoning. Furthermore, the proportion of pragmatic reasoning shows consistent effects of sentence order (i.e., whether the implicature trigger is in the premise or the hypothesis), and the presence of negation in one or both sentences. Pragmatic reasoning is consistently higher when the implicature trigger is in the premise, which we can see in the results for negated implicatures: the _some-all_ condition shows more pragmatic behavior compared to the _all-some_ condition (a similar behavior is observed with the _not all_ vs. _none_ conditions).

Generally, the presence of negation lowers rates of pragmatic reasoning. First, the negated implicature conditions can be subdivided into pairs with and without negation. Among the negated ones, pragmatic reasoning is lower than for nonnegated ones. Second, having negation in the premise rather than the hypothesis makes pragmatic reasoning lower: among pairs tagged as direct implicatures (_some_ vs. _not all_), there is higher pragmatic reasoning with non-negated _some_ in the premise than with negated _not all_. Finally, we observe that pragmatic rates are lower for _some_ vs. _not all_ than for _some_ vs. _all_. In this final case, pragmatic reasoning could be facilitated by explicit presentation of the two items on the scale.

In sum, for the datasets besides determiners, we find evidence that BERT fails to learn even the logical relations between scalemates, ruling out the possibility of computing scalar implicatures. It remains possible that BERT could learn these logical relations with explicit supervision (see Richardson et al., 2020), but it is clear that these are not learned from training on MultiNLI. Only the determiner dataset was informative in showing the extent of the NLI BERT model's pragmatic reasoning, since it alone showed a fine-grained enough understanding of the semantic relationship of the scalemates, like _some_ and _all_. In this setting BERT returned impressive results showing a high proportion of pragmatic reasoning compared to logical reasoning, which was affected by sentence order and presence of negation in a predictable way.

## 7 Experiment 2: Presuppositions

### Presupposition Datasets

The presupposition portion of ImpPres includes eight datasets, each isolating a different kind of presupposition trigger. The full set of triggers is shown in Table 5 in the Appendix. For each type, we generate 100 paradigms, with each paradigm consisting of 19 unique sentence pairs. (Examples of the sentence types are in Table 1).

Of the 19 sentence pairs, 15 contain target items. The first target item tests whether the model correctly determines that the presupposition trigger entails its presupposition. The next two alter the presupposition, either negating it, or replacing a constituent, leading to contradiction and neutrality, respectively. The remaining 12 show that the relation between the trigger and the (altered) pre-supposition is not affected by embedding the trigger under various entailment-canceling operators. 4 control items are designed to test the basic effect of entailment-canceling operators--negation, modals, interrogatives, and conditionals. In each control, the premise is a presupposition trigger embedded under an entailment-canceling operator, and the hypothesis is an unembedded sentence containing the trigger. These controls are neces

Figure 4: BERT results for scalar implicatures triggered by determiners \(\langle{\textit{some},\textit{all}}\rangle\), by target condition.

 

sary to establish whether models learn that presuppositions behave differently under these operators than do classical semantic entailments.

### Presupposition Results & Discussion

The results from presupposition controls are in Figure 5. BERT performs well above chance on each control (acc. \(>0.33\)), whereas BOW and InferSent perform at or below chance. In the "negated" condition, BERT correctly identifies that the trigger is contradicted by its negation 100% of the time, e.g., _Jo's cat didn't go_ contradicts _Jo's cat went_. In the other conditions, it correctly identifies the neutral relation the majority of the time, e.g., _Did Jo's cat go?_ is neutral with respect to _Jo's cat went_. This indicates that BERT mostly learns that negation, modals, interrogatives, and conditionals cancel classical entailments, while BOW and InferSent do not capture the ordinary behavior of these common operators.

Next, we test whether models identify presuppositions of the premise as entailments, e.g., that _Jo's cat went_ entails that _Jo has a cat_. Recall from SS2.2 that this is akin to a listener accommodating a presupposition. The results in Figure 6 show that each of the three models accommodates some presuppositions, but this depends on both the nature of the presupposition and the model. For instance, the BOW and InferSent models accommodate presuppositions of nearly all trigger types at well above chance rates (acc. \(\gg 33\%\)). For the uniqueness presupposition of clefts, these models generally correctly predict an entailment (acc. \(>90\%\)), but for most triggers, performance is less reliable. By contrast, BERT's behavior is bimodal. It always accommodates the existence presuppositions of clefts and possessed definites, as well as the presupposition of _only_, but almost never accommodates any presupposition involving numeracy, e.g. _Both flowers that bloomed died_ entails _There are exactly two flowers that bloomed.7_

Footnote 7: The presence of _exactly_ might contribute to poor performance on numercy examples. We suspect MultiNLI annotators may have used it disproportionately for neut. hypotheses.

Finally, we evaluate whether models predict that presuppositions project out of entailment canceling operators (e.g., that _Did Jo's cat go?_ entails that _Jo has a cat_). We can only consider such a prediction as evidence of projection if two conditions hold: (a) the model correctly identifies that the relevant operator cancels entailments in the control from the same paradigm (e.g., _Did Jo's cat go?_ is neutral with respect to _Jo's cat went_), and (b) the model identifies the presupposition as an entailment when the trigger is unembedded in the same paradigm (e.g. _Jo's cat went_ entails _Jo has a cat_). Otherwise, a model might correctly predict entailment essentially by accident if, for instance, it systematically ignores negation. For this reason, we filter out results for the target conditions that do not meet these criteria.

Figure 7 shows results for the target conditions after filtering. While InferSent rarely predicts that presuppositions project, we find strong evidence that the BERT and BOW models do. Specifically, they correctly identify that the premise entails the presupposition (acc. \(\geq 80\%\) for BERT, acc. \(\geq 90\%\) for BOW). Furthermore, BERT is the only model to reliably identify (i.e., over 90% of the time) that the negation of the presupposition is contradicted. These results hold irrespective of the entailment canceling operator. No model reliably performs above chance when the presupposition is altered to be neutral (e.g., _Did Jo's cat go?_ is neu

Figure 5: Results on Controls (Presuppositions).

Figure 6: Results for the unembedded trigger paired with positive presupposition.

 

tral with respect to _Jo has a cat_).

It is surprising that the simple BOW model can learn some of the projective behavior of presuppositions. One explanation for this finding is that many of the key features of presupposition projection are insensitive to word order. If a lexical pre-supposition trigger is present at all in a sentence, a presupposition will generally arise irrespective of its position in the sentence. There are some edge cases where this heuristic is insufficient, but ImpPres is not designed to test such cases.

To summarize, training on NLI is sufficient for all models we evaluate to learn to accommodate presuppositions of a wide variety of unembedded triggers, though BERT rejects presuppositions involving numeracy. Furthermore, BERT and even the BOW model appear to learn the characteristic projective behavior of some presuppositions.

## 8 General Discussion & Conclusion

We observe some encouraging results in SS6-7. We find strong evidence that BERT learns scalar implicatures associated with determiners _some_ and _all_. Pragmatic or logical reasoning was not diagnosable for the other scales, whose meaning was not fully understood by our models (as most scalar pairs were treated as synonymous). In the case of presuppositions, the BERT NLI models, and BOW to some extent, perform well on a number of our subdatasets (_only_, cleft existence, possessive existence, questions). For the other subdatasets, the models did not perform as expected on the basic unembedded presupposition triggers, again suggesting the model's lack of knowledge of the basic meaning of these words. Though their behavior is far from systematic, this is suggestive evidence that some NLI models can perform in ways that correlate with human-like pragmatic behavior.

Given that MultiNLI contains few examples of the type found in ImpPres (see SS4), where might our positive results come from? There are two potential sources of signal for the BERT model: NLI training, and pretraining (either BERT's masked language modeling objective or its input word embeddings). NLI training provides specific examples of valid (or invalid) inferences constituting an incomplete characterization of what commonsense inference is in general. Since presuppositions and scalar implicatures triggered by specific lexical items are largely absent from the MultiNLI data used for NLI training, any positive results on ImpPres would likely use prior knowledge from the pretraining stage to make an inductive leap that pragmatic inferences are valid commonsense inferences. The natural language text used for pretraining certainly contains pragmatic information, since, like any natural language data, it is produced with the assumption that readers are capable of pragmatic reasoning. Maybe this induces patterns in the data that make the nature of those assumptions recoverable from the data itself.

This work is an initial step towards rigorously investigating the extent to which NLI models learn semantic versus pragmatic inference types. We have introduced a new dataset ImpPres for probing this question, which can be reused to evaluate pragmatic performance of any NLI given model.

## Acknowledgments

This material is based upon work supported by the National Science Foundation (NSF) under Grant No. 1850208 awarded to A. Warstadt. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Thanks to the FAIR NLP & Conversational AI Group, the Google AI NLP group, and the NYU ML\({}^{2}\), including Sam Bowman, He He, Phu Mon H tut, Katharina Kann, Haokun Liu, Ethen Perez, Richard Pang, Clara Vania for discussions on the topic, and/or feedback on an earlier draft. Additional thanks to Marco Baroni, Hagen Blix, Emmanuel Chemla, Aaron Steven White, and Luke Zettlemoyer for insightful comments.

Figure 7: Results for presupposition target conditions involving projection.

 



## References

* Aristotle (2017) Aristotle. _De Interpretation_.
* Beaver (1997) David I. Beaver. 1997. Presupposition. In _Handbook of logic and language_, pages 939-1008. Elsevier.
* Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In _Proceedings of the 2020 International Conference on Learning Representations (ICLR)_.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642. Association for Computational Linguistics.
* Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktaschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In _Advances in Neural Information Processing Systems_, pages 9539-9549.
* Cianflone et al. (2018) Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. 2018. Let's do it "again": A first computational approach to detecting adverbial pre-supposition triggers. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2747-2755, Melbourne, Australia. Association for Computational Linguistics.
* Clark (1996) Herbert H Clark. 1996. _Using language_. Cambridge University Press.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485. Association for Computational Linguistics.
* Cooper et al. (1994) Robin Cooper, Richard Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspers, Hans Kamp, Manfreel Pinkal, Massimo Poesio, Stephen Pulman, et al. 1994. FraCaS: A framework for computational semantics. _Deliverable D6_.
* Crouch et al. (2006) Richard Crouch, Lauri Karttunen, and Annie Zaenen. 2006. Circumscribing is not excluding: A reply to Manning. Ms., Palo Alto Research Center.
* Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_, pages 177-190. Springer.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. _Synthesis Lectures on Human Language Technologies_, 6(4):1-220.
* Degen (2015) Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. _Semantics and Pragmatics_, 8:11-1.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ettinger et al. (2016) Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP_, pages 134-139, Berlin, Germany. Association for Computational Linguistics.
* Gazdar (1979) Gerald Gazdar. 1979. _Pragmatics, implicature, presuposition and logical form_. Academic Press, NY.
* Geiger et al. (2018) Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. Stress-testing neural models of natural language inference with multiply-quantified sentences. In _CoRR_.
* Glickman (2006) Oren Glickman. 2006. _Applied textual entailment_. Bar-Ilan University.
* Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 650-655. Association for Computational Linguistics.
* Price (1975) H Paul Price. 1975. Logic and conversation. _1975_, pages 41-58.
* Heim (1983) Irene Heim. 1983. On the projection problem for pre-suppositions. _Formal semantics: The essential readings_, pages 249-260.
* Horn (1989) Laurence Horn. 1989. _A natural history of negation_. University of Chicago Press.
* Horn and Ward (2004) Laurence R Horn and Gregory L Ward. 2004. _The handbook of pragmatics_. Wiley Online Library

* Jiang and de Marneffe (2019) Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Do you know that florce is packed with visitors? evaluating state-of-the-art models of speaker commitment. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4208-4213, Florence, Italy. Association for Computational Linguistics.
* Karttunen (1973) Lauri Karttunen. 1973. Presuppositions of compound sentences. _Linguistic inquiry_, 4(2):169-193.
* Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In _Proceedings of Association for the Advancement of Artificial Intelligence (AAAI)_.
* Kim et al. (2019) Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_, pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
* Lai et al. (2017) Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 100-109, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Levinson (2000) Stephen C Levinson. 2000. _Presumptive meanings: The theory of generalized conversational implications_. MIT press.
* Lewis (1979) David Lewis. 1979. Scorekeeping in a language game. In _Semantics from different points of view_, pages 172-187. Springer.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. _arXiv preprint arXiv:1907.11692_.
* and NLP needs it. Ms., Stanford University.
* Marelli et al. (2014) Marco Marelli, Luisa Bentivogli, Marco Baroni, Rafaellla Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_, pages 1-8, Dublin, Ireland. Association for Computational Linguistics.
* de Marneffe et al. (2019) Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In _Proceedings of Simu and Bedetung_, volume 23, pages 107-124.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
* Mirkin et al. (2009) Shachar Mirkin, Roy Baro-Haim, Ido Dagan, Eyal Shnarch, Asher Stern, Idan Szpektor, and Jonathan Berant. 2009. Addressing discourse and document structure in the RTE search task. In _Textual Analysis Conference_.
* Mou et al. (2016) Lili Mou, Hui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 130-136, Berlin, Germany. Association for Computational Linguistics.
* Naik et al. (2018) Akanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Nie et al. (2019) Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. Analyzing compositionality-sensitivity of NLI models. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 6867-6874.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Poliak et al. (2018) Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 67-81, Brussels, Belgium. Association for Computational Linguistics.
* Potts (2015) Christopher Potts. 2015. Presupposition and implicature. _The handbook of contemporary semantic theory_, 2:168-202.
* Rashkin et al. (2018) Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: 

Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 463-473. External Links: Link Cited by: SS2.
* Kyle Richardson et al. (2020)Probing natural language inference models through semantic fragments. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), Portland Russell, USA, pp. 902-914. External Links: ISBN 978-1-450-4673-4 Cited by: SS2.
* M. Schmitt and H. Schutze (2019)SherLiC: a typed event-focused lexical inference benchmark for evaluating natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Florence, Italy, pp. 902-914. External Links: Link Cited by: SS2.
* S. Schuster, Y. Chen, and J. Degen (2020)Harnessing the richness of the linguistic signal in predicting pragmatic inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. . External Links: Document Cited by: SS2.
* R. Stalnaker (1974)Pragmatic presuppositions. In Semantic Matters, T. K. Munitz and P. K. Unger, editors, Semantics and Philosophy, New York University Press, New York, NY, USA, pp. 135-148. External Links: ISBN 978-1-450-4673-3 Cited by: SS2.
* A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2019)GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: SS2.
* A. Warstadt, X. Parrish, H. Liu, A. Mohananey, W. Peng, S. Wang, and S. R. Bowman (2019)BLiMP: The benchmark of linguistic minimal pairs for English. arXiv preprint arXiv:1912.00582. Cited by: SS2.
* A. Westeradt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S. Wang, and S. R. Bowman (2019)BLEiMP: the benchmark of linguistic minimal pairs for English. arXiv preprint arXiv:1912.00582. Cited by: SS2.
* S. Welleck, J. Weston, C. Szlam, and K. Cho (2019)Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3731-3741. External Links: Link Cited by: SS2.
* A. Stevensh White, P. Rastogi, K. Duh, and B. Van Durme (2017)Inference is everything: recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan, pp. 996-1005. External Links: Link Cited by: SS2.
* A. Stevensh White and K. Rawlins (2018)The role of veridicality and activity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, Amherst, MA, USA, GLSA Publications, pp. . External Links: Link Cited by: SS2.
* A. Stevensh White, R. Rudinger, K. Rawlins, and B. Van Durme (2018)Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 4717-4724. External Links: Link Cited by: SS2.
* A. Williams, N. Nangia, and S. Bowman (2018)A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. External Links: Link Cited by: SS2.
* H. Yanaka, K. Mineshima, D. Bekki, K. Inuiu, S. Sekine, L. Abzianidze, and J. Bos (2019)An neural networks understand monotonicity reasoning?. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, pp. 31-40. External Links: Link Cited by: SS2.
* H. Yanaka, K. Mineshima, D. Bekki, K. Inuiu, S. Sekine, L. Abzianidze, and J. Bos (2019)HELP: a dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), Minneapolis, Minnesota, pp. 250-255. External Links: Link Cited by: SS2.
* A. Zaenen, L. Karttunen, and R. Crouch (2005)Local textual inference: can it be defined or circumscribed?. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, Michigan, pp. 31-36. External Links: Link Cited by: SS2.
* R. Zellers, Y. Bisk, R. Schwartz, and Y. Choi (2018)SWAG: a large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, pp. 93-104. External Links: Link Cited by: SS2.

 

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
* Zhang et al. (2017) Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. _Transactions of the Association for Computational Linguistics_, 5:379-395.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_, pages 19-27.

 



## Appendix

\begin{table}
\begin{tabular}{l l l} \hline \hline Type & Premise & Hypothesis \\ \hline Connectives & These cats or those fish appear. & These cats and those fish don’t both appear. \\ Determiners & Some skateboards tipped over. & Not all skateboards tipped over. \\ Numerals & Ten bananas were scorching. & One hundred bananas weren’t scorching. \\ Modals & Jerry could wake up. & Jerry didn’t need to wake up. \\ Scalar adjectives & Banks are fine. & Banks are not great. \\ Scalar verbs & Dawn went towards the hills. & Dawn did not get to the hills. \\ \hline \hline \end{tabular}
\end{table}
Table 4: The scalar implicature triggers in ImpPres. Examples are automatically generated sentences pairs from each of the six datasets for the scalar implicates experiment. The pairs belong to the “Implicature (+ to –)” condition.

\begin{table}
\begin{tabular}{p{108.2pt} p{95.3pt} p{76.8pt}} \hline \hline Type & Premise (Trigger) & Hypothesis (Presupposition) \\ \hline _All N_ & All six roses that bloomed died. & Exactly six roses bloomed. \\ _Both_ & Both flowers that bloomed died. & Exactly two flowers bloomed. \\ Change of State & The cat escaped. & The cat used to be captive. \\ Cleft Existence & It is Sandra who disliked Veronica. & Someone disliked Veronica. \\ Cleft Uniqueness & It is Sandra who disliked Veronica. & Exactly one person disliked Veronica. \\ _Only_ & Only Lucille went to Spain. & Lucille went to Spain. \\ Possessed Definites & Bill’s handyman won. & Bill has a handyman. \\ Question & Sue learned why Candice testified. & Candice testified. \\ \hline \hline \end{tabular}
\end{table}
Table 5: The presupposition triggers in ImpPres. Examples are automatically generated sentences pairs from each of the eight datasets for the presupposition experiment. The pairs belong to the “Plain Trigger / Presupposition” condition.



Figure 8: Results for the scalar implicatures triggered by adjectives, by target condition.

Figure 9: Results for the scalar implicatures triggered by adjectives, by target condition.

 

Figure 12: Results for the scalar triggered by numerals, by target condition.

Figure 10: Results for the scalar implicatures triggered by determiners, by target condition. Figure 13: Results for the scalar implicates triggered by verbs, by target condition.