Venue: NAACL 2018

# A Broad-Coverage Challenge Corpus for

Sentence Understanding through Inference

Adina Williams\({}^{1}\)

adinawilliams@nyu.edu

Nikita Nangia\({}^{2}\)

nikitanangia@nyu.edu

Samuel R. Bowman\({}^{1,2,3}\)

bowman@nyu.edu

###### Abstract

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. _recognizing textual entailment_), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

## 1 Introduction

Many of the most actively studied problems in NLP, including question answering, translation, and dialog, depend in large part on natural language understanding (NLU) for success. While there has been a great deal of work that uses representation learning techniques to pursue progress on these applied NLU problems directly, in order for a representation learning model to fully succeed at one of these problems, it must simultaneously succeed both at NLU, and at one or more additional hard machine learning problems like structured prediction or memory access. This makes it difficult to accurately judge the degree to which current models extract reasonable representations of language meaning in these settings.

The task of natural language inference (NLI) is well positioned to serve as a benchmark task for research on NLU. In this task, also known as _recognizing textual entailment_Cooper et al. (1996); Fyodorov et al. (2000); Condoravdi et al. (2003); Bos and Markert (2005); Dagan et al. (2006); MacCartney and Manning (2009), a model is presented with a pair of sentences--like one of those in Figure 1--and asked to judge the relationship between their meanings by picking a label from a small set: typically entailment, neutral, and contradiction. Succeeding at NLI does not require a system to solve any difficult machine learning problems except, crucially, that of extracting effective and thorough representations for the meanings of sentences (i.e., their lexical and compositional semantics). In particular, a model must handle phenomena like lexical entailment, quantification, coreference, tense, belief, modality, and lexical and syntactic ambiguity.

As the only large human-annotated corpus for NLI currently available, the Stanford NLI Corpus (SNLI; Bowman et al., 2015) has enabled a good deal of progress on NLU, serving as a major benchmark for machine learning work on sentence understanding and spurring work on core representation learning techniques for NLU, such as attention Wang and Jiang (2016); Parikh et al. (2016), memory Munkhdalai and Yu (2017), and the use of parse structure Mou et al. (2016b); Bowman et al. (2016); Chen et al. (2017). However, SNLI falls short of providing a sufficient testing ground for machine learning models in two ways.



First, the sentences in SNLI are derived from only a single text genre--image captions--and are thus limited to descriptions of concrete visual scenes, rendering the hypothesis sentences used to describe these scenes short and simple, and rendering many important phenomena--like temporal reasoning (e.g., _yesterday_), belief (e.g., _know_), and modality (e.g., _should_)--are enough to be irrelevant to task performance. Second, because of these issues, SNLI is not sufficiently demanding to serve as an effective benchmark for NLU, with the best current model performance falling within a few percentage points of human accuracy and limited room left for fine-grained comparisons between strong models.

This paper introduces a new challenge dataset, the Multi-Genre NLI Corpus (MultiNLI), whose chief purpose is to remedy these limitations by making it possible to run large-scale NLI evaluations that capture more of the complexity of modern English. While its size (433k pairs) and mode of collection are modeled closely on SNLI, unlike that corpus, MultiNLI represents both written and spoken speech in a wide range of styles, degrees of formality, and topics.

Our chief motivation in creating this corpus is to provide a benchmark for ambitious machine learning research on the core problems of NLU, but we are additionally interested in constructing a corpus that facilitates work on domain adaptation and cross-domain transfer learning. These techniques--which use labeled training data for a source domain, and aim to train a model that performs well on test data from a target domain with a different distribution--have resulted in gains across many tasks Daume III and Marcu (2006); Ben-David et al. (2007), including sequence and part-of-speech tagging Blitzer et al. (2006); Peng Dredze (2017). Moreover, in application areas outside NLU, artificial neural network techniques have made it possible to train general-purpose feature extractors that, with no or minimal retraining, can extract useful features for a variety of styles of data Krizhevsky et al. (2012); Zeiler and Fergus (2014); Donahue et al. (2014). However, attempts to bring this kind of general purpose representation learning to NLU have seen only very limited success (see, for example, Mou et al. (2016a). Nearly all successful applications of representation learning to NLU have involved models that are trained on data closely resembling the target evaluation data in both task and style. This fact limits the usefulness of these tools for problems involving styles of language not represented in large annotated training sets.

With this in mind, we construct MultiNLI so as to make it possible to explicitly evaluate models both on the quality of their sentence representations within the training domain and on their ability to derive reasonable representations in unfamiliar domains. The corpus is derived from ten different genres of written and spoken English, which are collectively meant to approximate the full diversity of ways in which modern standard

\begin{table}
\begin{tabular}{p{85.4pt} p{99.6pt} p{142.3pt}} \hline \hline Met my first girlfriend that way. & Face-to-Face **contradiction** & I didn’t meet my first girlfriend until later. \\  & c n C & \\
8 million in relief in the form of emergency housing. & Government **neutral** & The 8 million dollars for emergency housing was still not enough to solve the problem. \\  & n N n N & Letters & All of the children love working in their gardens. \\  & & n N n N & \\ At 8:34, the Boston Center controller received a third transmission from American 11 & 9/11 **entallment** & The Boston Center controller got a third transmission from American 11. \\  & e E e e e & & \\ I am a lacto-vegetarian. & Slate **neutral** & I enjoy eating cheese too much to abstain from dairy. \\  & n n E n & \\ someone else noticed it and i said well i guess that’s true and it was somewhat melodious in other words it wasn’t just you know it was really funny & Telephone **contradiction** & No one noticed and it wasn’t funny at all. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Randomly chosen examples from the development set of our new corpus, shown with their genre labels, their selected gold labels, and the validation labels (abbreviated e, n, c) assigned by individual annotators.

 

American English is used. All of the genres appear in the test and development sets, but only five are included in the training set. Models thus can be evaluated on both the _matched_ test examples, which are derived from the same sources as those in the training set, and on the _mismatched_ examples, which do not closely resemble any of those seen at training time.

## 2 The Corpus

### Data Collection

The data collection methodology for MultiNLI is similar to that of SNLI: We create each sentence pair by selecting a premise sentence from a preexisting text source and asking a human annotator to compose a novel sentence to pair with it as a hypothesis. This section discusses the sources of our premise sentences, our collection method for hypotheses, and our validation (relabeling) strategy.

Premise Text SourcesThe MultiNLI premise sentences are derived from ten sources of freely available text which are meant to be maximally diverse and roughly represent the full range of American English. We selected nine sources from the second release of the Open American National Corpus [1985]; mackel2000, lde2001, lde2006, downloaded 12/20161], balancing the volume of source text roughly evenly across genres, and avoiding genres with content that would be too difficult for untrained annotators.

Footnote 1: [http://www.anc.org/](http://www.anc.org/)

OANC data constitutes the following nine genres: transcriptions from the Charlotte Narrative and Conversation Collection of two-sided, in-person conversations that took place in the early 2000s (Face-to-face); reports, speeches, letters, and press releases from public domain government websites [374]; letters from the Indiana Center for Intercultural Communication of Philanthropic Fundraising Discourse written in the late 1990s-early 2000s [149]; the public report from the National Commission on Terrorist Attacks Upon the United States released on July 22, 20042 (9/11); five non-fiction works on the textile industry and child development published by the Oxford University Press (OUP); popular culture articles from the archives of Slate Magazine (Slate) written between 1996-2000; transcriptions from University of Pennsylvania's Linguistic Data Consortium Switchboard corpus of two-sided, telephone conversations that took place in 1990 or 1991 [149]; travel guides published by Berlitz Publishing in the early 2000s (Travel); and short posts about linguistics for non-specialists from the Verbatim archives written between 1990 and 1996 [374].

For our tenth genre, Fiction, we compile several freely available works of contemporary fiction written between 1912 and 2010, spanning various genres, including mystery (_The Mysterious Affair at Styles_,3 Christie, 1921; _The Secret Adversary_,4 Christie, 1922; _Murder in the Gun Room_,5 Piper, 1953), humor (_Password Incorrect_,6 Name, 2008), western (_Rebel Spurs_,7 Norton, 1962), science fiction (_Seven Swords_,8 Shea, 2008; _Living History_,9 Essex, 2016; _The Sky Is Falling_,10 Del Rey, 1973; _Youth_,11 Asimov, May 1952), and adventure (_Captain Blood_,12 Sabatini, 1922).

Footnote 3: [https://github.com/journals/pien/](https://github.com/journals/pien/)

Footnote 4: [https://github.com/journals/pien/](https://github.com/journals/pien/)

Footnote 5: [https://github.com/journals/pien/](https://github.com/journals/pien/)

We construct premise sentences from these ten source texts with minimal preprocessing; unique the sentences within genres, exclude very short

Figure 1: The main text of a prompt (truncated) that was presented to our annotators. This version is used for the written non-fiction genres.

 

sentences (under eight characters), and manually remove certain types of non-narrative writing, such as mathematical formulae, bibliographic references, and lists.

Although SNLI is collected in largely the same way as MultiNLI, and is also permissively licensed, we do not include SNLI in the MultiNLI corpus distribution. SNLI can be appended and treated as an unusually large additional captions genre, built on image captions from the Flickr30k corpus Young et al. (2014).

Hypothesis CollectionTo collect a sentence pair, we present a crowdworker with a sentence from a source text and ask them to compose three novel sentences (the hypotheses): one which is necessarily true or appropriate whenever the premise is true (paired with the premise and labeled entailment), one which is necessarily false or inappropriate whenever the premise is true (contradiction), and one where neither condition applies (neutral). This method of data collection ensures that the three classes will be represented equally in the raw corpus.

The prompts that surround each premise sentence during hypothesis collection are slightly tailored to fit the genre of that premise sentence. We pilot these prompts prior to data collection to ensure that the instructions are clear and that they yield hypothesis sentences that fit the intended meanings of the three classes. There are five unique prompts in total: one for written non-fiction genres (Slate, OUP, Government, Verbatim, Travel; Figure 1), one for spoken genres (Telephone, Face-to-Face), one for each of the less formal written genres (Fiction, Letters), and a specialized one for 9/11, tailored to fit its potentially emotional content. Each prompt is accompanied by example premises and hypothesis that are specific to each genre.

Below the instructions, we present three text fields--one for each label--followed by a field for reporting issues, and a link to the frequently asked questions (FAQ) page. We provide one FAQ page per prompt. FAQs are modeled on their SNLI counterparts (supplied by the authors of that work) and include additional curated examples, answers to genre-specific questions arising from our pilot phase, and information about logistical concerns like payment.

For both hypothesis collection and validation, we present prompts to annotators using Hybrid (gethybrid.io), a crowdsourcing platform similar to the Amazon Mechanical Turk platform used for SNLI. We used this platform to hire an organized group of workers. 387 annotators contributed through this group, and at no point was any identifying information about them, including demographic information, available to the authors.

ValidationWe perform an additional round of annotation on test and development examples to ensure accurate labelling. The validation phase follows the same procedure used for SICK Marelli et al. (2014) and SNLI: Workers are presented with pairs of sentences and asked to supply a single label (entailment, contradiction, neutral) for the pair. Each pair is relabeled by four workers, yielding a total of five labels per example. Validation instructions are tailored by genre, based on the main data collection prompt (Figure 1); a single FAQ, modeled after the validation FAQ from SNLI, is provided for reference. In order to encourage thoughtful labeling, we manually label one percent of the validation examples and offer a $1 bonus each time a worker selects a label that matches ours.

For each validated sentence pair, we assign a _gold label_ representing a majority vote between the initial label assigned to the pair by the original annotator, and the four additional labels assigned by validation annotators. A small number of examples did not receive a three-vote consensus on any one label. These examples are included in the distributed corpus, but are marked with '-' in the gold label field, and should not be used in standard evaluations. Table 2 shows summary statistics capturing the results of validation, alongside corresponding figures for SNLI. These statistics indicate that the labels included in MultiNLI are about as reliable as those included in SNLI, despite MultiNLI's more diverse text contents.

\begin{table}
\begin{tabular}{l r r} \hline \hline
**Statistic** & **SNLI** & **MultiNLI** \\ \hline Pairs w/ unanimous gold label & 58.3\% & 58.2\% \\ \hline Individual label = gold label & 89.0\% & 88.7\% \\ Individual label = author’s label & 85.8\% & 85.2\% \\ \hline Gold label = author’s label & 91.2\% & 92.6\% \\ Gold label \(\neq\) author’s label & 6.8\% & 5.6\% \\ No gold label (no 3 labels match) & 2.0\% & 1.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Key validation statistics for SNLI (copied from Bowman et al., 2015) and MultiNLI 



### The Resulting Corpus

Table 1 shows randomly chosen development set examples from the collected corpus. Hypotheses tend to be fluent and correctly spelled, though not all are complete sentences. Punctuation is often omitted. Hypotheses can rely heavily on knowledge about the world, and often don't correspond closely with their premises in syntactic structure.

Unlabeled test data is available on Kaggle for both matched and mismatched sets as competitions that will be open indefinitely; Evaluations on a subset of the test set have previously been conducted with different leaderboards through the RepEval 2017 Workshop (Nangia et al., 2017).

The corpus is available in two formats--tab separated text and JSON Lines (jsonl), following SNLI. For each example, premise and hypothesis strings, unique identifiers for the pair and prompt, and the following additional fields are specified:

* gold_label: label used for classification. In examples rejected during the validation process, the value of this field will be '-'.
* sentence{1,2}_parse: Each sentence as parsed by the Stanford PCFG Parset 3.5.2 (Klein and Manning, 2003).
* sentence{1,2}_binary_parse: parses in unlabeled binary-branching format.
* label[1]: The label assigned during the creation of the sentence pair. In rare cases this may be different from gold_label, if a consensus of annotators chose a different label during the validation phase.
* label[2...5]: The four labels assigned during validation by individual annotators to each development and test example. These fields will be empty for training examples.

The current version of the corpus is freely available at nyu.edu/projects/bowman/multinli/ for typical machine learning uses, and may be modified and redistributed. The majority of the corpus is released under the OANC's license, which allows all content to be freely used, modified, and shared under permissive terms. The data in the Fiction section falls under several permissive licenses; _Seven Swords_ is available under a Creative Commons Share-Alike 3.0 Unported License, and with the explicit permission of the author, _Living History_ and _Password Incorrect_ are available under Creative Commons Attribution 3.0 Unported Licenses; the remaining works of fiction are in the public domain in the United States (but may be licensed differently elsewhere).

PartitionThe distributed corpus comes with an explicit train/test/development split. The test and development sets contain 2,000 randomly selected examples each from each of the genres, resulting in a total of 20,000 examples per set. No premise sentence occurs in more than one set.

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & \multicolumn{4}{c}{**\#Examples**} & \multicolumn{2}{c}{**\#Wds.**} & \multicolumn{2}{c}{**‘S’ parses**} & \multicolumn{2}{c}{**Model Acc.**} \\
**Genre** & **Train** & **Dev.** & **Test** & **Prem.** & **Prem.** & **Hyp.** & **Agmt.** & **ESIM** & **CBOW** \\ \hline _SNLI_ & _550,152_ & _10,000_ & _10,000_ & _14.1_ & _74\%_ & _88\%_ & _89.0\%_ & _86.7\%_ & _80.6 \%_ \\ \hline Fiction & 77,348 & 2,000 & 2,000 & 14.4 & 94\% & 97\% & 89.4\% & 73.0\% & 67.5\% \\ Government & 77,350 & 2,000 & 2,000 & 24.4 & 90\% & 97\% & 87.4\% & 74.8\% & 67.5\% \\ Slate & 77,306 & 2,000 & 2,000 & 21.4 & 94\% & 98\% & 87.1\% & 67.9\% & 60.6\% \\ Telephone & 83,348 & 2,000 & 2,000 & 25.9 & 71\% & 97\% & 88.3\% & 72.2\% & 63.7\% \\ Travel & 77,350 & 2,000 & 2,000 & 24.9 & 97\% & 98\% & 89.9\% & 73.7\% & 64.6\% \\ \hline g/11 & 0 & 2,000 & 2,000 & 20.6 & 98\% & 99\% & 90.1\% & 71.9\% & 63.2\% \\ Face-to-face & 0 & 2,000 & 2,000 & 18.1 & 91\% & 96\% & 89.5\% & 71.2\% & 66.3\% \\ Letters & 0 & 2,000 & 2,000 & 20.0 & 95\% & 98\% & 90.1\% & 74.7\% & 68.3\% \\ OUP & 0 & 2,000 & 2,000 & 25.7 & 96\% & 98\% & 88.1\% & 71.7\% & 62.8\% \\ Verbatim & 0 & 2,000 & 2,000 & 28.3 & 93\% & 97\% & 87.3\% & 71.9\% & 62.7\% \\ \hline
**MultiNLI Overall** & **392,702** & **20,000** & **20,000** & **22.3** & **91\%** & **98\%** & **88.7\%** & **72.2\%** & **64.7\%** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Key statistics for the corpus by genre. The first five genres represent the _matched_ section of the development and test sets, and the remaining five represent the _unismatched_ section. The first three statistics provide the number of examples in each genre. _#Ws. Prem._ is the mean token count among premise sentences. ‘_S’ parses_ is the percentage of sentences for which the Stanford Parser produced a parse rooted with an ‘S’ (sentence) node. _Agrnt._ is the percent of individual labels that match the gold label in validated examples. _Model Acc._ gives the test accuracy for ESIM and CBOW models (trained on either SNLI or MultiNLI), as described in Section 3.



StatisticsTable 3 shows some additional statistics. Premise sentences in MultiNLI tend to be longer (max 401 words, mean 22.3 words) than their hypotheses (max 70 words, mean 11.4 words), and much longer, on average, than premises in SNLI (mean 14.1 words); premises in MultiNLI also tend to be parsed as complete sentences at a much higher rate on average (91%) than their SNLI counterparts (74%). We observe that the two spoken genres differ in this--with Face-to-face showing more complete sentences (91%) than Telephone (71%)--and speculate that the lack of visual feedback in a telephone setting may result in a high incidence of interrupted or otherwise incomplete sentences.

Hypothesis sentences in MultiNLI generally cannot be derived from their premise sentences using only trivial editing strategies. While \(2.5\)% of the hypotheses in SNLI differ from their premises by deletion, only \(0.9\)% of those in MultiNLI (170 examples total) are constructed in this way. Similarly, in SNLI, \(1.6\)% of hypotheses differ from their premises by addition, substitution, or shuffling a single word, while in MultiNLI this only happens in \(1.2\)% of examples. The percentage of hypothesis-premise pairs with high token overlap (\(>\!37\)%) was comparable between MultiNLI (30% of pairs) and SNLI (29%). These statistics suggest that MultiNLI's annotations are comparable in quality to those of SNLI.

## 3 Baselines

To test the difficulty of the corpus, we experiment with three neural network models. The first is a simple continuous bag of words (CBOW) model in which each sentence is represented as the sum of the embedding representations of its words. The second computes representations by averaging the states of a bidirectional LSTM RNN BiLSTM Hochreiter and Schmidhuber (1997) over words. For the third, we implement and evaluate Chen et al.'s Enhanced Sequential Inference Model (ESIM), which is roughly tied for the state of the art on SNLI at the time of writing. We use the base ESIM without ensembling with a TreeLSTM (as in the 'HIM' runs in that work).

The first two models produce separate vector representations for each sentence and compute label predictions for pairs of representations. To do this, they concatenate the representations for premise and hypothesis, their difference, and their element-wise product, following Mou et al. (2016b), and pass the result to a single \(\tanh\) layer followed by a three-way softmax classifier.

All models are initialized with 300D reference GloVe vectors (840B token version; Pennington et al. (2014). Out-of-vocabulary (OOV) words are initialized randomly and word embeddings are fine-tuned during training. The models use 300D hidden states, as in most prior work on SNLI. We use Dropout Srivastava et al. (2014) for regularization. For ESIM, we use a dropout rate of 0.5, following the paper. For CBOW and BiLSTM models, we tune Dropout on the SNLI development set and find that a drop rate of 0.1 works well. We use the Adam Kingma and Ba (2015) optimizer with default parameters. Code is available at github.com/nyu-mll/multiNLI/.

We train models on SNLI, MultiNLI, and a mixture; Table 4 shows the results. In the mixed setting, we use the full MultiNLI training set and randomly select 15% of the SNLI training set at each epoch, ensuring that each available genre is seen during training with roughly equal frequency.

We also train a separate CBOW model on each individual genre to establish the degree to which simple models already allow for effective transfer across genres, using a dropout rate of 0.2. When training on SNLI, a single random sample of 15% of the original training set is used. For each genre represented in the training set, the model that performs best on it was trained on that genre; a model trained only on SNLI performs worse on every genre than comparable models trained on any genre from MultiNLI.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & & \multicolumn{3}{c}{**MNLI**} \\

- \\  & BiLSTM & 81.5 &
- \\  & ESIM & **86.7** &
- \\ \hline \multirow{3}{*}{MNLI} & CBOW & 51.5 & 64.8 & 64.5 \\  & BiLSTM & 50.8 & 66.9 & 66.9 \\  & ESIM & 60.7 & **72.3** & **72.1** \\ \hline \multirow{3}{*}{MNLI+ SNLI} & CBOW & 74.7 & 65.2 & 64.6 \\  & BiLSTM & 74.0 & 67.5 & 67.1 \\ \cline{1-1}  & ESIM & 79.7 & **72.4** & **71.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Test set accuracies (%) for all models; _Match._ represents test set performance on the MultiNLI genres that are also represented in the training set, _Mis._ represents test set performance on the remaining ones; _Most freq._ is a trivial ‘most frequent class’ baseline.
 

Models trained on a single genre from MultiNLI perform well on similar genres; for example, the model trained on Telephone attains the best accuracy (63%) on Face-to-Face, which was nearly one point better than it received on itself. Slate seems to be a difficult and relatively unusual genre and performance on it is relatively poor in this setting; when averaging over runs trained on SNLI and all genres in the matched section of the training set, average performance on Slate was only 57.5%. Sentences in Slate cover a wide range of topics and phenomena, making it hard to do well on, but also forcing models trained on it be broadly capable; the model trained on Slate achieves the highest accuracy of any model on 9/11 (55.6%) and Verbatim (57.2%), and relatively high accuracy on Travel (57.4%) and Government (58.3%). We also observe that our models perform similarly on both the matched and mismatched test sets of MultiNLI. We expect genre mismatch issues to become more conspicuous as models are developed that can better fit MultiNLI's training genres.

To evaluate the contribution of sentence length to corpus difficulty, we binned premises and hypotheses by length in 25-word increments for premises and 10-word increments for hypotheses. Using the ESIM model, our strong baseline, we find a small effect (stronger for matched than mismatched) of premise length on model accuracy: accuracy decreases slightly as premise sentences increase in length. We find no effect of hypothesis length on accuracy.

## 4 Discussion and Analysis

### Data Collection

In data collection for NLI, different annotator decisions about the coreference between entities and events across the two sentences in a pair can lead to very different assignments of pairs to labels de Marneffe et al. (2008); Marelli et al. (2014); Bowman et al. (2015). Drawing an example from Bowman et al., the pair _"a boat sank in the Pacific Ocean"_ and _"a boat sank in the Atlantic Ocean"_ can be labeled either contradiction or neutral depending on (among other things) whether the two mentions of boats are assumed to refer to the same entity in the world. This uncertainty can present a serious problem for inter-annotator agreement, since it is not clear that it is possible to define an explicit set of rules around coreference that would be easily intelligible to an untrained annotator (or any non-expert).

Bowman et al. attempt to avoid this problem by using an annotation prompt that is highly dependent on the concreteness of image descriptions; but, as we engage with the much more abstract writing that is found in, for example, government documents, there is no reason to assume _a priori_ that any similar prompt and annotation strategy can work. We are surprised to find that this is not a major issue. Through a relatively straightforward trial-and-error piloting phase, followed by discussion with our annotators, we manage to design prompts for abstract genres that yield high inter-annotator agreement scores nearly identical to those of SNLI (see Table 2). These high scores suggest that our annotators agreed on a single task definition, and were able to apply it consistently across genres.

### Overall Difficulty

As expected, both the increase in the diversity of linguistic phenomena in MultiNLI and its longer average sentence length conspire to make MultiNLI dramatically more difficult than SNLI. Our three baseline models perform better on SNLI than MultiNLI by about 15% when trained on the respective datasets. All three models achieve accuracy above 80% on the SNLI test set when trained only on SNLI. However, when trained on MultiNLI, only ESIM surpasses 70% accuracy on MultiNLI's test sets. When we train models on MultiNLI and downsampled SNLI, we see an expected significant improvement on SNLI, but no significant change in performance on the MultiNLI test sets, suggesting including SNLI in training doesn't drive substantial improvement. These results attest to MultiNLI's difficulty, and with its relatively high inter-annotator agreement, suggest that it presents a problem with substantial headroom for future work.

### Analysis by Linguistic Phenomenon

To better understand the types of language understanding skills that MultiNLI tests, we analyze the collected corpus using a set of annotation tags chosen to reflect linguistic phenomena which are known to be potentially difficult. We use two methods to assign tags to sentences. First, we use the Penn Treebank (PTB; Marcus et al., 1993) part-of-speech tag set (via the included Stanford Parser parses) to automatically isolate sentences 

containing a range of easily-identified phenomena like comparatives. Second, we isolate sentences that contain hand-chosen key words indicative of additional interesting phenomena.

The hand-chosen tag set covers the following phenomena: Quantifiers contains single words with quantifational force (see, for example, Heim and Kratzer, 1998; Szabolcsi, 2010, e.g., _many, all, few, some_); Belief Verbs contains sentence-embedding verbs denoting mental states (e.g., _know, believe, think_), including irregular past tense forms; Time terms contains single words with abstract temporal interpretation, (e.g., _then, today_) and month names and days of the week; Discourse markers contains words that facilitate discourse coherence (e.g., _yet, however, but, thus, despite_); Presupposition Triggers contains words with lexical presuppositions [364, Schlenker, 2016, e.g., _again, too, anymore_13]; Conditionals contains the word _if_. Table 5 presents the frequency of the tags in SNLI and MultiNLI, and model accuracy on MultiNLI (trained only on MultiNLI).

Footnote 13: Because their high frequency in the corpus, extremely common triggers like _the_ were excluded from this tag.

The incidence of tags varies by genre; the percentage of sentence pairs containing a particular annotation tag differs by a maximum over 30% across genres. Sentence pairs containing pronouns are predictably common for all genres, with 93% of Government and Face-to-face pairs including at least one. The Telephone genre has the highest percentage of sentence pairs containing one occurrence of negation, WH-words, _belief_-verbs and time terms, Verbatim has the highest percentage of pairs containing quantifiers and conversational pivots, and Letters has the highest percentage of pairs that contain one or more modals. Pairs containing comparatively and/or superlatives, which is the tag that our baseline models perform worst on, are most common in the Oxford University Press genre. Based on this, we conclude that the genres are sufficiently different, because they are not uniform with respect to the percentages of sentence pairs that contain each of the annotation tags.

The distributions of labels within each tagged subset of the corpus roughly mirrors the balanced overall distribution. The most frequent class overall (in this case, entailment) occurs with a frequency of roughly one third (see Table 4) in most. Only two annotation tags differ from the baseline percentage of the most frequent class in the corpus by at least 5%: sentences containing negation, and sentences exceeding 20 words. Sentences that contain negation are slightly more likely than average to be labeled contradiction, reflecting a similar finding in SNLI, while long sentences are slightly more likely to be labeled entailment.

None of the baseline models perform substantially better on any tagged set than they do on the corpus overall, with average model accuracies on sentences containing specific tags falling within

\begin{table}
\begin{tabular}{l r r r r l c r r} \hline \hline  & \multicolumn{3}{c}{**Dev. Freq.**} & \multicolumn{3}{c}{**Most Frequent Label**} & \multicolumn{3}{c}{**Model Acc.**} \\
**Tag** & **SNLI** & **MultiNLI** & **Diff.** & **Label** & **\%** & **CBOW** & **BiLSTM** & **ESIM** \\ \hline Entire Corpus & 100 & 100 & 0 & entailment & _?35_ & _?65_ & _?67_ & _?72_ \\ \hline Pronouns (PTB) & 34 & 68 & 34 & entailment & 34 & 66 & 68 & 73 \\ Quantifiers & 33 & 63 & 30 & contradiction & 36 & 66 & 68 & 73 \\ Modals (TPB) & \textless{}1 & 28 & 28 & entailment & 35 & 65 & 67 & 72 \\ Negation (PTB) & 5 & 31 & 26 & contradiction & **48** & 67 & 70 & 75 \\ WH terms (PTB) & 5 & 30 & 25 & entailment & 35 & 64 & 65 & 72 \\ Belief Verbs & \textless{}1 & 19 & 18 & entailment & 34 & 64 & 67 & 71 \\ Time Terms & 19 & 36 & 17 & neutral & 35 & 64 & 66 & 71 \\ Discourse Mark. & \textless{}1 & 14 & 14 & neutral & 34 & 62 & 64 & 70 \\ Presup. Triggers & 8 & 22 & 14 & neutral & 34 & 65 & 67 & 73 \\ Compr/Supr.(PTB) & 3 & 17 & 14 & neutral & 39 & 61 & 63 & 69 \\ Conditionally & 4 & 15 & 11 & neutral & 35 & 65 & 68 & 73 \\ Tense Match (PTB) & 62 & 69 & 7 & entailment & 37 & 67 & 68 & 73 \\ Interjections (PTB) & \textless{}1 & 5 & 5 & entailment & 36 & 67 & 70 & 75 \\ \textgreater{}20 words & \textless{}1 & 5 & 5 & entailment & **42** & 65 & 67 & 76 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Dev. Freq. is the percentage of dev. set examples that include each phenomenon, ordered by greatest difference in frequency of occurrence (Diff.) between MultiNLI and SNLI. Most Frequent Label specifies which label is the most frequent for each tag in the MultiNLI dev. set, and % is its incidence. Model Acc. is the dev. set accuracy (%) by annotation tag for each baseline model (trained on MultiNLI only). (PTB) marks a tag as derived from Penn Treebank-style parser output tags [103].

 

about 3 points of overall averages. Using baseline model test accuracy overall as a metric (see Table 4), our baseline models had the most trouble on sentences containing comparatives or superlatives (loging 3-4 points each). Despite the fact that 17% of sentence pairs in the corpus contained at least one instance of comparative or superlative, our baseline models don't utilize the information present in these sentences to predict the correct label for the pair, although presence of a comparative or superlative is slightly more predictive of a neutral label.

Moreover, the baseline models perform below average on discourse markers, such as _despite_ and _however_, losing roughly 2 to 3 points each. Unsurprisingly, the attention-based ESIM model performs better than the other two on sentences with greater than 20 words. Additionally, our baseline models do show slight improvements in accuracy on negation, suggesting that they may be tracking it as a predictor of contradiction.

## 5 Conclusion

Natural language inference makes it easy to judge the degree to which neural network models for sentence understanding capture the full meanings for natural language sentences. Existing NLI datasets like SNLI have facilitated substantial advances in modeling, but have limited headroom and coverage of the full diversity of meanings expressed in English. This paper presents a new dataset that offers dramatically greater linguistic difficulty and diversity, and also serves as a benchmark for cross-genre domain adaptation.

Our new corpus, MultiNLI, improves upon SNLI in its empirical coverage--because it includes a representative sample of text and speech from ten different genres, as opposed to just simple image captions--and its difficulty, containing a much higher percentage of sentences tagged with one or more elements from our tag set of thirteen difficult linguistic phenomena. This greater diversity is reflected in the dramatically lower baseline model performance on MultiNLI than on SNLI (see Table 5) and comparable interannotator agreement, suggesting that MultiNLI has a lot of headroom remaining for future work.

The MultiNLI corpus was first released in draft form in the first half of 2017, and in the time since its initial release, work by others Conneau et al. (2017) has shown that NLI can also be an effective source task for pre-training and transfer learning in the context of sentence-to-vector models, with models trained on SNLI and MultiNLI substantially outperforming all prior models on a suite of established transfer learning benchmarks. We hope that this corpus will continue to serve for many years as a resource for the development and evaluation of methods for sentence understanding.

## Acknowledgments

This work was made possible by a Google Faculty Research Award. SB also gratefully acknowledges support from Tencent Holdings and Samsung Research. We also thank George Dahl, the organizers of the RepEval 2016 and RepEval 2017 workshops, Andrew Drozdov, Angeliki Lazaridou, and our other NYU colleagues for help and advice.

## References

* Asimov (1952) Isaac Asimov. May 1952. _Youth_. Space Science Fiction Magazine, Republic Features Syndicate, Inc.
* Ben-David et al. (2007) Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. In _Advances in neural information processing systems_. pages 137-144.
* Blitzer et al. (2006) John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 120-128. [http://www.aclweb.org/anthology/W06-1615](http://www.aclweb.org/anthology/W06-1615).
* Bos and Markert (2005) Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In _Proceedings of the 2005 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. pages 628-635. [http://www.aclweb.org/anthology/H05-1079](http://www.aclweb.org/anthology/H05-1079).
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 632-642. [https://doi.org/10.18653/v1/D15-1075](https://doi.org/10.18653/v1/D15-1075).
* Bowman et al. (2016) Samuel R. Bowman, Jon Gauthier, Abhinav Rasatogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: 

Long Papers_. Association for Computational Linguistics, pages 1466-1477. [https://doi.org/10.18653/v1/P16-1139](https://doi.org/10.18653/v1/P16-1139).
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 1657-1668. [https://doi.org/10.18653/v1/P17-1152](https://doi.org/10.18653/v1/P17-1152).
* Christie (1921) Agatha Christie. 1921. _Mysterious Affair at Styles_. The Bodley Head.
* Christie (1922) Agatha Christie. 1922. _The Secret Adversary_. Dodd, Mead.
* Condoravdi et al. (2003) Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In _Proceedings of the Human Language Technology-North American Association for Computational Linguistics 2003 Workshop on Text Meaning_.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. _arXiv preprint arXiv:1705.02364_ .
* Cooper et al. (1996) Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Steve Pulman Ted Briscoe Holger Maier Poosio, and Karsten Konrad. 1996. Using the framework. Technical report, Technical Report LTE 62-051 D-16, The FraCaS Consortium.
* Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In _Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising textual entailment_, Springer, pages 177-190.
* Daume III and Marcu (2006) Hal Daume III and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. _Journal of Artificial Intelligence Research_ 26:101-126. [https://doi.org/doi:10.1613/jair.1872](https://doi.org/doi:10.1613/jair.1872).
* de Marneffe et al. (2008) Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In _Proceedings of Association for Computational Linguistics-08: Human Language Technology_. Association for Computational Linguistics, pages 1039-1047. [http://www.aclweb.org/anthology/P08-1118](http://www.aclweb.org/anthology/P08-1118).
* Rey (1973) Lester Del Rey. 1973. _The Sky Is Falling_. Ace Books.
* Donahue et al. (2014) Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A deep convolutional activation feature for generic visual recognition. In _Proceedings of the International Conference on Machine Learning (ICML)_.
* Essex (2016) Ben Essex. 2016. _Living History_. CreateSpace Independent Publishing Platform.
* Fillmore et al. (1998) Charles Fillmore, Nancy Ide, Daniel Jurafsky, and Catherine Maclood. 1998. An American National Corpus: A proposal. In _Proceedings of the First Annual Conference on Language Resources and Evaluation_. pages 965-969.
* Fyodorov et al. (2000) Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In _Proceedings of the 2nd Workshop on Inference in Computational Semantics_.
* Heim and Kratzer (1998) Irene Heim and Angelika Kratzer. 1998. _Semantics in generative grammar_. Blackwell Publishers.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Ide and Macleod (2001) Nancy Ide and Catherine Macleod. 2001. The American National Corpus: A standardized resource of American English. In _Proceedings of Corpus Linguistics_. Lancaster University Centre for Computer Corpus Research on Language, volume 3, pages 1-7.
* Ide and Suderman (2006) Nancy Ide and Keith Suderman. 2006. Integrating linguistic resources: The national corpus model. In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC)_. European Language Resources Association (ELRA). [http://www.aclweb.org/anthology/L06-1138](http://www.aclweb.org/anthology/L06-1138).
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Klein and Manning (2003) Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In _Proc. ACL_. [https://doi.org/10.3115/1075096.1075150](https://doi.org/10.3115/1075096.1075150).
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In _Advances in Neural Information Processing Systems 25_, pages 1097-1105.
* MacCartney and Manning (2009) Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In _Proceedings of the of the Eighth International Conference on Computational Semantics_. pages 140-156.
* Macleod et al. (2000) Catherine Macleod, Nancy Ide, and Ralph Grishman. 2000. The American National Corpus: A standardized resource for American English. In _Conference on Language Resources and Evaluation (LREC)_.
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. _Computational linguistics_ 19(2):313-330 

Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamarelli. 2014a. Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_. Association for Computational Linguistics, pages 1-8. [https://doi.org/10.3115/vl/S14-2001](https://doi.org/10.3115/vl/S14-2001).
* Marelli et al. (2014b) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamarelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In _Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC)_.
* Mou et al. (2016a) Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016a. How transferable are neural networks in NLP applications? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 479-489. [https://doi.org/10.18653/v1/D16-1046](https://doi.org/10.18653/v1/D16-1046).
* Mou et al. (2016b) Lili Mou, Men Rui, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016b. Natural language inference by tree-based convolution and heuristic matching. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, pages 130-136. [https://doi.org/10.18653/v1/P16-2022](https://doi.org/10.18653/v1/P16-2022).
* Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017. Neural semantic encoders. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics, pages 397-407. [http://www.aclweb.org/anthology/E17-1038](http://www.aclweb.org/anthology/E17-1038).
* Name (2008) Nick Name. 2008. _Password Incorrect_. Published online. Author also known as Piotr Kowalczyk.
* Nangia et al. (2017) Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R Bowman. 2017. The repeval 2017 shared task: Multi-genre natural language inference with sentence representations. In _Proceedings of RepEval 2017: The Second Workshop on Evaluating Vector Space Representations for NLP_.
* Norton (1962) Andre Norton. 1962. _The Rebel Spurs_. The World Publishing Company, Cleveland & New York.
* Parikh et al. (2016) Ankur Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 2249-2255. [https://doi.org/10.18653/v1/D16-1244](https://doi.org/10.18653/v1/D16-1244).
* Peng and Dredze (2017) Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_. pages 91-100.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1532-1543. [https://doi.org/10.3115/v1/D14-1162](https://doi.org/10.3115/v1/D14-1162).
* Piper (1953) Henry Beam Piper. 1953. _Murder In the Gun Room_. H.B. Piper, New York.
* Sabatini (1922) Rafael Sabatini. 1922. _Captain Blood_. Houghton Mifflin Company.
* Schlenker (2016) Philippe Schlenker. 2016. _The Cambridge Handbook of Formal Semantics_, Cambridge University Press, chapter The Semantics/Fragmatics Interface, pages 664-727. [https://doi.org/10.1017/CBO9781139236157.023](https://doi.org/10.1017/CBO9781139236157.023).
* Shea (2008) Michael Shea. 2008. _Seven Swords_. Published online.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research (JMLR)_ 15:1929-1958.
* Stalnaker (1974) Robert Stalnaker. 1974. _Semantics and Philosophy_, New York University Press, chapter Pragmatic Presupposition, pages 329-355.
* Szabolcsci (2010) Anna Szabolcsci. 2010. _Quantification_. Cambridge University Press.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 1442-1451. [https://doi.org/10.18653/v1/N16-1170](https://doi.org/10.18653/v1/N16-1170).
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association of Computational Linguistics_ 2:67-78. [http://www.aclweb.org/anthology/Q14-1006](http://www.aclweb.org/anthology/Q14-1006).
* Zeiler and Fergus (2014) Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In _Proceedings of the European Conference on Computer Vision (ECCV)_. pages 818-833.