Venue: ACL 2020

# Do Neural Models Learn Systematic

of Monotonicity Inference in Natural Language?

Hitomi Yanaka\({}^{1}\), Koji Mineshima\({}^{2}\), Daisuke Bekki\({}^{3}\), and Kentaro Inui\({}^{4,1}\)

\({}^{1}\)RIKEN, \({}^{2}\)Keio University, \({}^{3}\)Ochanomizu University, \({}^{4}\)Tohoku University

hitomi.yanaka@riken.jp, minesima@abelard.flet.ketio.ac.jp, bekki@is.ocha.ac.jp, inui@ecei.tohoku.ac.jp

###### Abstract

Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn _systematicity_ of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.

+
Footnote †: July 5 - 10, 2020. ©2020 Association for Computational Linguistics

Systemsticity of inference (or _inferential systematicity_) Fodor and Pylyshyn (1988); Aydedee (1997) in natural language has been intensively studied in the field of formal semantics. From among the various aspects of inferential systematicity, in the context of NLI, we focus on _monotonicity_van Benthem (1983); Icard and Moss (2014) and its _productivity_. Consider the following premise-hypothesis pairs (1)-(3), which have the target label _entailment_:

* [noitemsep]
* \(P\): **Some**[_puppies_\(\uparrow\)] _ran._ H: **Some**_dogs_ran._
* \(P\): **No**[_cats_\(\downarrow\)] _ran._ H: **No**small cats_ran._
* \(P\): **Some**[_puppies which chased **no**[_cats_\(\downarrow\)] _ran._ H: **Some**_dogs which chased **no**small cats_ran._

As in (1), for example, quantifiers such as _some_ exhibit **upward monotone** (shown as [...\(\uparrow\)]), and replacing a phrase in an upward-entailing context in a sentence with a more general phrase (replacing _puppies_ in \(P\) with _dogs_ as in \(H\)) yields a sentence inferable from the original sentence. In contrast, as in (2), quantifiers such as _no_ exhibit **downward monotone** (shown as [...\(\downarrow\)]), and replacing a phrase in a downward-entailing context with a more specific phrase (replacing _cats_ in \(P\) with _small cats_ as in \(H\)) yields a sentence inferable from the original sentence. Such primitive inference patterns combine recursively as in (3). This manner of monotonicity and its productivity produces a potentially infinite number of inferential patterns. Therefore, NLI models must be capable of systematically interpreting such primitive patterns and reasoning over unseen combinations of patterns. Although many studies have addressed this issue by modeling logical reasoning in formal semantics Abzianidze (2015); Mineshima et al. (2015); Hu et al. (2019) and testing DNN-based models on monotonicity inference Yanaka et al. (2019a, b); Richardson et al. (

2020), the ability of DNN-based models to generalize to unseen combinations of patterns is still underexplored.

Given this background, we investigate the systematic generalization ability of DNN-based models on four aspects of monotonicity: (i) systematicity of predicate replacements (i.e., replacements with a more general or specific phrase), (ii) systematicity of embedding quantifiers, (iii) productivity, and (iv) localism (see Section 2.2). To this aim, we introduce a new evaluation protocol where we (i) synthesize training instances from sampled sentences and (ii) systematically control which patterns are shown to the models in the training phase and which are left unseen. The rationale behind this protocol is two-fold. First, patterns of monotonicity inference are highly systematic, so we can create training data with arbitrary combinations of patterns, as in examples (1)-(3). Second, evaluating the performance of the models trained with well-known NLI datasets such as MultiNLI Williams et al. (2018) might severely underestimate the ability of the models because such datasets tend to contain only a limited number of training instances that exhibit the inferential patterns of interest. Furthermore, using such datasets would prevent us from identifying which combinations of patterns the models can infer from which patterns in the training data.

This paper makes two primary contributions. First, we introduce an evaluation protocol1 using the systematic control of the training/test split under various combinations of semantic properties to evaluate whether models learn inferential systematicity in natural language. Second, we apply our evaluation protocol to three NLI models and present evidence suggesting that, while all models generalize to unseen combinations of lexical and logical phenomena, their generalization ability is limited to cases where sentence structures are nearly the same as those in the training set.

Footnote 1: The evaluation code will be publicly available at [https://github.com/verypluming/systematicity](https://github.com/verypluming/systematicity).

## 2 Method

### Basic idea

Figure 1 illustrates the basic idea of our evaluation protocol on monotonicity inference. We use synthesized monotonicity inference datasets, where NLI models should capture both (i) monotonicity directions (upward/downward) of various quantifiers and (ii) the types of various predicate replacements in their arguments. To build such datasets, we first generate a set of premises \(G_{d}^{\mathbf{Q}}\) by a context-free grammar \(G\) with depth \(d\) (i.e., the maximum number of applications of recursive rules), given a set of quantifiers \(\mathbf{Q}\). Then, by applying \(G_{d}^{\mathbf{Q}}\) to elements of a set of functions for predicate replacements (or _replacement functions_ for short) **R** that rephrase a constituent in the input premise and return a hypothesis, we obtain a set \(\mathbf{D}_{d}^{\mathbf{Q,R}}\) of premise-hypothesis pairs defined as

For example, the premise _Some puppies ran_ is generated from the quantifier _some_ in \(\mathbf{Q}\) and

Figure 1: An illustration of the basic idea. For Systematicity and Productivity, we train models on the **Train 1** and **Train 2** sets and test them on the **Test** set. Arrow (****) means entailment relation; lex, adj, and prep mean predicate replacements for lexical relations, adjectives, and prepositional phrases, respectively. In Productivity, we use various quantifiers and predicate replacements in each depth.

 

the production rule \(S\to Q,N,IV\), and thus it is an element of \(G^{\mathbf{Q}}_{1}\). By applying this premise to a replacement function that replaces the word in the premise with its hypernym (e.g., _puppy_\(\sqsubseteq\textit{dog}\)), we provide the premise-hypothesis pair _Some puppies ran_\(\Rightarrow\)_Some dogs ran_ in Fig. 1.

We can control which patterns are shown to the models during training and which are left unseen by systematically splitting \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{d}\) into training and test sets. As shown on the left side of Figure 1, we consider how to test the systematic capacity of models with unseen combinations of quantifiers and predicate replacements. To expose models to primitive patterns regarding **Q** and **R**, we fix an arbitrary element \(q\) from **Q** and feed various predicate replacements into the models from the training set of inferences \(\textbf{D}^{\{q\},\mathbf{R}}_{d}\) generated from combinations of the fixed quantifier and all predicate replacements. Also, we select an arbitrary element \(r\) from **R** and feed various quantifiers into the models from the training set of inferences \(\textbf{D}^{\mathbf{Q},\{r\}}_{d}\) generated from combinations of all quantifiers and the fixed predicate replacement.

We then test the models on the set of inferences generated from unseen combinations of quantifiers and predicate replacements. That is, we test them on the set of inferences \(\textbf{D}^{\overline{\{q\},\{r\}}}_{d}\) generated from the complements \(\overline{\{q\},\overline{\{r\}}}\) of \(\{q\},\{r\}\). If models capture inferential systematicity in combinations of quantifiers and predicate replacements, they can correctly perform all inferences in \(\textbf{D}^{\overline{\{q\},\{r\}}}_{d}\) on an arbitrary split based on \(q,r\).

Similarly, as shown on the right side of Figure 1, we can test the productive capacity of models with unseen depths by changing the training/test split based on \(d\). For example, by training models on \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{d}\) and testing them on \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{d+1}\), we can evaluate whether models generalize to one deeper depth. By testing models with an arbitrary training/test split of \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{d}\) based on semantic properties of monotonicity inference (i.e., quantifiers, predicate replacements, and depths), we can evaluate whether models systematically interpret them.

### Evaluation protocol

To test NLI models from multiple perspectives of inferential systematicity in monotonicity inferences, we focus on four aspects: (i) systematicity of predicate replacements, (ii) systematicity of embedding quantifiers, (iii) productivity, and (iv) localism. For each aspect, we use a set \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{d}\) of premise-hypothesis pairs. Let \(\textbf{Q}=\textbf{Q}^{\uparrow}\cup\textbf{Q}^{\downarrow}\) be the union of a set of selected upward quantifiers \(\textbf{Q}^{\uparrow}\) and a set of selected downward quantifiers \(\textbf{Q}^{\downarrow}\) such that \(|\textbf{Q}^{\uparrow}|=|\textbf{Q}^{\downarrow}|=n\). Let **R** be a set of replacement functions \(\{r_{1},\dots,r_{m}\}\), and \(d\) be the embedding depth, with \(1\leq d\leq s\). (4) is an example of an element of \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{1}\), containing the quantifier _some_ in the subject position and the predicate replacement using the hypernm relation _dogs_\(\sqsubseteq\textit{animals}\) in its upward-entailing context without embedding.

(4)   \(P\): _Some dogs ran_    \(\Rightarrow\)  \(H\): _Some animals ran_

I. Systematicity of predicate replacementsThe following describes how we test the extent to which models generalize to unseen combinations of quantifiers and predicate replacements. Here, we expose models to all primitive patterns of predicate replacements like (4) and (5) and all primitive patterns of quantifiers like (6) and (7). We then test whether the models can systematically capture the difference between upward quantifiers (e.g., _several_) and downward quantifiers (e.g., _no_) as well as the different types of predicate replacements (e.g., the lexical relation _dogs_\(\sqsubseteq\textit{animals}\) and the adjective deletion _small dogs_\(\sqsubseteq\textit{dogs}\)) and correctly interpret unseen combinations of quantifiers and predicate replacements like (8) and (9).

(5)   \(P\): _Some small dogs run_    \(\Rightarrow\)  \(H\): _Some dogs ran_    \(\Rightarrow\)  \(H\): _Several animals ran_    \(P\): _No animals ran_    \(\Rightarrow\)  \(H\): _No dogs ran_    \(\Rightarrow\)  \(H\): _Several dogs run_    \(\Rightarrow\)  \(H\): _No small dogs ran_    \(\Rightarrow\)  __Here, we consider a set of inferences \(\textbf{D}^{\mathbf{Q},\mathbf{R}}_{1}\) whose depth is 1. We move from harder to easier tasks by gradually changing the training/test split according to combinations of quantifiers and predicate replacements. First, we expose models to primitive patterns of **Q** and **R** with the minimum training set. Thus, we define the initial training set \(\textbf{S}_{1}\) and test set \(\textbf{T}_{1}\) as follows:

\[(\textbf{S}_{1},\textbf{T}_{1})=(\textbf{D}^{\{q\},\mathbf{R}}_{1}\cup \textbf{D}^{\mathbf{Q},\{r\}}_{1},\;\textbf{D}^{\overline{\{q\},\{r\}}}_{1})\]

where \(q\) is arbitrarily selected from **Q**, and \(r\) is arbitrarily selected from **R**.

Next, we gradually add the set of inferences generated from combinations of an upward-downward quantifier pair and all predicate replacements to the training set. In the examples above, we add (8) and (9) to the training set to simplify the task. We assume a set \(\textbf{Q}^{\prime}\) of a pair of upward/downward quantifiers, namely, \(\{(q^{\uparrow},q^{\downarrow})~{}|~{}(q^{\uparrow},q^{\downarrow})\subseteq \textbf{Q}^{\uparrow}\times\textbf{Q}^{\downarrow},~{}q^{\uparrow},q^{\downarrow} \neq q\}\). We consider 

a set \({\sf perm}({\bf Q^{\prime}})\) consisting of permutations of \({\bf Q^{\prime}}\). For each \(p\in{\sf perm}({\bf Q^{\prime}})\), we gradually add a set of inferences generated from \(p(i)\) to the training set \({\bf S}_{i}\) with \(1<i\leq n-1\). Then, we provide a test set \({\bf T}_{i}\) generated from the complement \(\overline{{\bf Q}_{i}}\) of \(\underline{{\bf Q}_{i}}=\{x~{}|~{}\exists y(x,y)\in{\bf Q}^{\prime}_{i}\mbox{ or }\exists y(y,x)\in{\bf Q}^{\prime}_{i}\}\) and \(\overline{\{r\}}\) where \({\bf Q}^{\prime}_{i}=\{p(1),\ldots,p(i)\}\). This protocol is summarized as

\[{\bf S}_{i+1} ={\bf S}_{i}\cup{\bf D}_{1}^{\{q^{\uparrow}_{i},q^{\downarrow}_{ i}\},{\bf R}},\] \[{\bf T}_{i} ={\bf D}_{1}^{\overline{{\bf Q}_{i}},\overline{\{r\}}}\quad \text{with $1<i\leq n-1$}\]

where \((q^{\uparrow}_{i},q^{\downarrow}_{i})=p(i)\).

To evaluate the extent to which the generalization ability of models is robust for different syntactic structures, we use an additional test set \({\bf T}^{\prime}_{i}={\bf D}_{1}^{\overline{{\bf Q}_{i}},\overline{\{r\}}}\) generated using three production rules. The first is the case where one adverb is added at the beginning of the sentence, as in example (10).

* \(P_{adv}\): _Slowly,_ **several** _small dogs ran_ \(H_{adv}\): _Slowly,_ **several** _dogs run_

The second is the case where a three-word prepositional phrase is added at the beginning of the sentence, as in example (11).

* \(P_{prep}\): _Near the shore,_ **several** _small dogs ran_ \(H_{prep}\): _Near the shore,_ **several** _dogs ran_

The third is the case where the replacement is performed in the object position, as in example (12).

* \(P_{obj}\): _Some tiger touched_ **several** _small dogs_ \(H_{obj}\): _Some tiger touched_ **several** _dogs_

We train and test models \(|{\sf perm}({\bf Q^{\prime}})|\) times, then take the average accuracy as the final evaluation result.

II. Systematicity of embedding quantifiersTo properly interpret embedding monotonicity, models should detect both (i) the monotonicity direction of each quantifier and (ii) the type of predicate replacements in the embedded argument. The following describes how we test whether models generalize to unseen combinations of embedding quantifiers. We expose models to all primitive combination patterns of quantifiers and predicate replacements like (4)-(9) with a set of non-embedding monotonicity inferences \({\bf D}_{1}^{{\bf Q},{\bf R}}\) and some embedding patterns like (13), where \(\boldsymbol{Q}_{1}\) and \(\boldsymbol{Q}_{2}\) are chosen from a selected set of upward or downward quantifiers such as _some_ or _no_. We then test the models with an inference with an unseen quantifier _several_ in (14) to evaluate whether models can systematically interpret embedding quantifiers.

* \(P\): \(\boldsymbol{Q}_{1}\) _animals that chased_ \(\boldsymbol{Q}_{2}\) _dogs ran_ \(H\): \(\boldsymbol{Q}_{1}\) _animals that chased_ \(\boldsymbol{Q}_{2}\) _animals ran_
* \(P\): **Several** _animals that chased_ **several** _dogs ran_ \(H\): **Several** _animals that chased_ **several** _animals ran_

We move from harder to easier tasks of learning embedding quantifiers by gradually changing the training/test split of a set of inferences \({\bf D}_{2}^{{\bf Q},{\bf R}}\) whose depth is 2, i.e., inferences involving one embedded clause.

We assume a set \({\bf Q^{\prime}}\) of a pair of upward and downward quantifiers as \({\bf Q^{\prime}}\equiv\{(q^{\uparrow},q^{\downarrow})~{}|~{}(q^{\uparrow},q^{ \downarrow})\subseteq{\bf Q^{\dagger}}\times{\bf Q^{\downarrow}}\}\), and consider a set \({\sf perm}({\bf Q^{\prime}})\) consisting of permutations of \({\bf Q^{\prime}}\). For each \(p\in{\sf perm}({\bf Q^{\prime}})\), we gradually add a set of inferences \({\bf D}_{2}\) generated from \(p(i)\) to the training set \({\bf S}_{i}\) with \(1\leq i\leq n-1\).

We test models trained with \({\bf S}_{i}\) on a test set \({\bf T}_{i}\) generated from the complement \(\overline{{\bf Q}_{i}}\) of \({\bf Q}_{i}=\{x~{}|~{}\exists y(x,y)\in{\bf Q}^{\prime}_{i}\mbox{ or }\exists y(y, x)\in{\bf Q}^{\prime}_{i}\}\) where \({\bf Q}^{\prime}_{i}=\{p(1),\ldots,p(i)\}\), summarized as

\[{\bf S}_{0} ={\bf D}_{1}^{{\bf Q},{\bf R}},\] \[{\bf S}_{i} ={\bf S}_{i-1}\cup{\bf D}_{2}^{\{q^{\uparrow}_{i},q^{\downarrow}_{ i}\},{\bf R}},\] \[{\bf T}_{i} ={\bf D}_{2}^{\overline{{\bf Q}_{i}},{\bf R}}\quad\text{with $1\leq i \leq n-1$}\]

where \((q^{\uparrow}_{i},q^{\downarrow}_{i})=p(i)\). We train and test models \(|{\sf perm}({\bf Q^{\prime}})|\) times, then take the average accuracy as the final evaluation result.

III. ProductivityProductivity (or _recursiveness_) is a concept related to systematicity, which refers to the capacity to grasp an indefinite number of natural language sentences or thoughts with generalization on composition. The following describes how we test whether models generalize to unseen deeper depths in embedding monotonicity (see also the right side of Figure 1). For example, we expose models to all primitive non-embedding/single-embedding patterns like (15) and (16) and then test them with deeper embedding patterns like (17).

* \(P\): **Some** _dogs ran_ \(H\): **Some** _animals ran_ \(H\): **Some** _animals which chased_ **some** _dogs ran_ \(H\): **Some** _animals which chased_ **some** _animals ran_ 

* [(17)]
**Some animals which chased some cats which followed some dogs ran H: Some animals which chased some cats which followed some animals ran**

To evaluate models on the set of inferences involving embedded clauses with depths exceeding those in the training set, we train models with \(\bigcup_{d\in\{1,...,i+1\}}\textbf{D}_{d}\), where we refer to \(\textbf{D}^{\textbf{Q},\textbf{R}}_{d}\) as \(\textbf{D}_{d}\) for short, and test the models on \(\bigcup_{d\in\{i+2,...,s\}}\textbf{D}_{d}\) with \(1\leq i\leq s-2\).

IV. LocalismAccording to the principle of compositionality, the meaning of a complex expression derives from the meanings of its constituents and how they are combined. One important concern is how local the composition operations should be [40]. We therefore test whether models trained with inferences involving embedded monotonicity locally perform inferences composed of smaller constituents. Specifically, we train models with examples like (17) and then test the models with examples like (15) and (16). We train models with \(\textbf{D}_{d}\) and test the models on \(\bigcup_{k\in\{1,...,d\}}\textbf{D}_{k}\) with \(3\leq d\leq s\) .

## 3 Experimental Setting

### Data creation

To prepare the datasets shown in Table 1, we first generate premise sentences involving quantifiers from a set of context-free grammar (CFG) rules and lexical entries, shown in Table 6 in the Appendix. We select 10 words from among nouns, intransitive verbs, and transitive verbs as lexical entries. A set of quantifiers **Q** consists of eight elements; we use a set of four downward quantifiers \(\textbf{Q}^{\downarrow}=\)_{no, at most three, less than three, few_}_ and a set of four upward quantifiers \(\textbf{Q}^{\uparrow}=\)_{some, at least three, more than three, a few_}_, which have the same monotonicity directions in the first and second arguments. We thus consider \(n=|\textbf{Q}^{\uparrow}|=|\textbf{Q}^{\downarrow}|=4\) in the protocol in Section 2.2. The ratio of each monotonicity direction (upward/downward) of generated sentences is set to \(1:1\). We then generate hypothesis sentences by applying replacement functions to premise sentences according to the polarities of constituents. The set of replacement functions **R** is composed of the seven types of lexical replacements and phrasal additions in Table 2. We remove unnatural premise-hypothesis pairs in which the same words or phrases appear more than once.

For embedding monotonicity, we consider inferences involving four types of replacement functions in the first argument of the quantifier in Table 2: hyponyms, adjectives, prepositions, and relative clauses. We generate sentences up to the depth \(d=5\). There are various types of embedding monotonicity, including relative clauses, conditionals, and negated clauses. In this paper, we consider three types of embedded clauses: peripheral-embedding clauses and two kinds of center-embedding clauses, shown in Table 6 in the Appendix.

\begin{table}
\begin{tabular}{l l c c c c} \hline
**Depth** & **Pred.** & **Monotone** & **Arg.** & \multicolumn{2}{c}{**Example** (premise, hypothesis, label) & **Avg. Len.**} \\ \hline
1 & CONJ & DOWNARD & SECOND & _Less than three_ & _less_ & 4.6 \\  & & & & _Less than three_ & _less_ & _not_ \\ \hline
2 & PP & UPWARD & FIRST & _Few ions that hurt at most three small dogs walked._ & & 9.0 \\  & & & & _Same elephant no rabbit_ & _wak does hit rushed._ & & \\
3 & Adj & DOWNARD & FIRST & _Some elephant no rabbit_ & _woaked a few_ & _small dogs hit rushed._ & 12.3 \\  & & & & & _Less than three_ & _israch accepted_ & _several rabbits that loved several_ & \\
4 & RC & UPWARD & FIRST & _focus more than three_ & _nonxys cleaned__daudled._ & & \\  & & & & & _Less than three_ & _istershui accepted_ & _several rabbits that loved several foxes more than three monkeys which are thinner cleaned drawded._ & 16.6 \\  & & & & & & ENTALLMENT & \\ \hline \end{tabular}
\end{table}
Table 1: Examples of generated premise–hypothesis pairs. Depth: depth of embedding; Pred..: type of predicate replacements; Monotone: direction of monotonicity; Arg.: argument where the predicate replacement is performed; Avg. Len.: average sentence length.

\begin{table}
\begin{tabular}{l l} \hline
**Function** & **Example** \\ \hline \(r_{1}\): hyponym & _dogs_\(\sqsubseteq\)_animals_ \\ \(r_{2}\): adjective & _small dogs_\(\sqsubseteq\)_dogs_ \\ \(r_{3}\): preposition & _dogs in the park_\(\sqsubseteq\)_dogs_ \\ \(r_{4}\): relative clause & _dogs which are dinner_\(\sqsubseteq\)_dogs_ \\ \(r_{5}\): adverb & _ran quickly_\(\sqsubseteq\)_ran_ \\ \(r_{6}\): disjunction & _ran_\(\sqsubseteq\)_ran or walked_ \\ \(r_{7}\): conjunction & _ran and barked_\(\sqsubseteq\)_ran_ \\ \hline \end{tabular}
\end{table}
Table 2: Examples of replacement functions.

 

The number of generated sentences exponentially increases with the depth of embedded clauses. Thus, we limit the number of inference examples to 320,000, split into 300,000 examples for the training set and 20,000 examples for the test set. We guarantee that all combinations of quantifiers are included in the set of inference examples for each depth. Gold labels for generated premise-hypothesis pairs are automatically determined according to the polarity of the argument position (upward/downward) and the type of predicate replacements (with more general/specific phrases). The ratio of each gold label (entailment/non-entailment) in the training and test sets is set to \(1:1\).

To double-check the gold label, we translate each premise-hypothesis pair into a logical formula (see the Appendix for more details). The logical formulas are obtained by combining lambda terms in accordance with meaning composition rules specified in the CFG rules in the standard way Blackburn and Bos (2005). We prove the entailment relation using the theorem prover Vampire2, checking whether a proof is found in time for each entailment pair. For all pairs, the output of the prover matched with the entailment relation automatically determined by monotonicity calculus.

Footnote 2: [https://github.com/vprover/vampire](https://github.com/vprover/vampire)

### Models

We consider three DNN-based NLI models. The first architecture employs long short-term memory (LSTM) networks Hochreiter and Schmidhuber (1997). We set the number of layers to three with no attention. Each premise and hypothesis is processed as a sequence of words using a recurrent neural network with LSTM cells, and the final hidden state of each serves as its representation.

The second architecture employs multiplicative tree-structured LSTM (TreeLSTM) networks Tran and Cheng (2018), which are expected to be more sensitive to hierarchical syntactic structures. Each premise and hypothesis is processed as a tree structure by bottom-up combinations of constituent nodes using the same shared compositional function, input word information, and between-word relational information. We parse all premise-hypothesis pairs with the dependency parser using the spaCy library3 and obtain tree structures. For each experimental setting, we randomly sample 100 tree structures and check their correctness. In LSTM and TreeLSTM, the dimension of hidden units is 200, and we initialize the word embeddings with 300-dimensional GloVe vectors Pennington et al. (2014). Both models are optimized with Adam Kingma and Ba (2015), and no dropout is applied.

Footnote 3: [https://spacy.io/](https://spacy.io/)

The third architecture is a Bidirectional Encoder Representations from Transformers (BERT) model Devlin et al. (2019). We used the base-uncased model pre-trained on Wikipedia and BookCorpus from the pytorch-pretrained-bert library4, fine-tuned for the NLI task using our dataset. In fine-tuning BERT, no dropout is applied, and we choose hyperparameters that are commonly used for MultiNLI. We train all models over 25 epochs or until convergence, and select the best-performing model based on its performance on the validation set. We perform five runs per model and report the average and standard deviation of their scores.

Footnote 4: [https://github.com/huggingface/pytorch-pretrained-bert](https://github.com/huggingface/pytorch-pretrained-bert)

## 4 Experiments and Discussion

I.Systematicity of predicate replacementsFigure 2 shows the performance on unseen combinations of quantifiers and predicate replacements. In the minimal training set \({\bf S}_{1}\), the accuracy of LSTM and TreeLSTM was almost the same as chance, but that of BERT was around 75%, suggesting that only BERT generalized to unseen combinations of quantifiers and predicate replacements. When we train BERT with the training set \({\bf S}_{2}\), which contains inference examples generated from combinations of one pair of upward/downward quantifiers and all predicate replacements, the accuracy was 100%. This indicates that by being taught two kinds of quantifiers in the training data, BERT could distinguish between upward and downward for the other quantifiers. The accuracy of LSTM and TreeLSTM increased with increasing the training set size, but did not reach 100%. This indicates that LSTM and TreeLSTM also generalize to inferences involving similar quantifiers to some extent, but their generalization ability is imperfect.

When testing models with inferences where adverbs or prepo 

ginning of the sentence, the accuracy of all models significantly decreased. This decrease becomes larger as the syntactic structures of the sentences in the test set become increasingly different from those in the training set. Contrary to our expectations, the models fail to maintain accuracy on test sets whose difference from the training set is the structure with the adverb at the beginning of a sentence. Of course, we could augment datasets involving that structure, but doing so would require feeding all combinations of inference pairs into the models. These results indicate that the models tend to estimate the entailment label from the beginning of a premise-hypothesis sentence pair, and that inferential systematicity to draw inferences involving quantifiers and predicate replacements is not completely generalized at the level of arbitrary constituents.

**II. Systematicity of embedding quantifiers** Figure 3 shows the performance of all models on unseen combinations of embedding quantifiers. Even when adding the training set of inferences involving one embedded clause and two quantifiers step-by-step, no model showed improved performance. The accuracy of BERT slightly exceeded chance, but the accuracy of LSTM and TreeLSTM was nearly the same as or lower than chance. These results suggest that all the models fail to generalize to unseen combinations of embedding quantifiers even when they involve similar upward/downward quantifiers.

**III. Productivity** Table 3 shows the performance on unseen depths of embedded clauses. The accuracy on \({\bf D}_{1}\) and \({\bf D}_{2}\) was nearly 100%, indicating that all models almost completely generalize to inferences containing previously seen depths. When \({\bf D}_{1}+{\bf D}_{2}\) were used as the training set, the accuracy of all models on \({\bf D}_{3}\) exceeded chance. Similarly, when \({\bf D}_{1}+{\bf D}_{2}+{\bf D}_{3}\) were used as the training set, the accuracy of all models on \({\bf D}_{4}\) exceeded chance. This indicates that all models partially generalize to inferences containing embedded clauses one level deeper than the training set.

However, standard deviations of BERT and LSTM were around 10, suggesting that these models did not consistently generalize to inferences containing embedded clauses one level deeper than the training set. While the distribution of monotonicity directions (upward/downward) in the training and test sets was uniform, the accuracy of LSTM and BERT tended to be smaller for downward inferences than for upward inferences. This also indicates that these models fail to properly compute monotonicity directions of constituents from syntactic structures. The standard deviation of TreeLSTM was smaller, indicating that TreeLSTM robustly learns inference patterns containing embedded clauses one level deeper than the training set.

Figure 3: Results for systematicity of embedding quantifiers. \({\bf S}_{n}\) indicates the experimental setting where the training set \({\bf S}_{n}\) is used.

Figure 2: Results for systematicity of predicate replacements. Accuracy on test sets where (a) the replacement is performed in the subject position, (b) one adverb is added at the beginning of the sentence, (c) one three-word prepositional phrase is added at the beginning of the sentence, and (d) the replacement is in the object position. \({\bf S}_{n}\) indicates the experimental setting where the training set \({\bf S}_{n}\) is used.

 

However, the performance of all models trained with \({\textbf{D}_{1}+\textbf{D}_{2}}\) on \({\textbf{D}_{4}}\) and \({\textbf{D}_{5}}\) significantly decreased. Also, performance decreased for all models trained with \({\textbf{D}_{1}+\textbf{D}_{2}+\textbf{D}_{3}}\) on \({\textbf{D}_{5}}\). Specifically, there was significantly decreased performance of all models, including TreeLSTM, on inferences containing embedded clauses two or more levels deeper than those in the training set. These results indicate that all models fail to develop productivity on inferences involving embedding monotonicity.

**IV. Localism** Table 4 shows the performance of all models on localism of embedding monotonicity. When the models were trained with \({\textbf{D}_{3}}\), \({\textbf{D}_{4}}\) or \({\textbf{D}_{5}}\), all performed at around chance on the test set of non-embedding inferences \({\textbf{D}_{1}}\) and the test set of inferences involving one embedded clause \({\textbf{D}_{2}}\). These results indicate that even if models are trained with a set of inferences containing complex syntactic structures, the models fail to locally interpret their constituents.

Performance of data augmentationPrior studies Yanaka et al. (2019b); Richardson et al. (2020) have shown that given BERT initially trained with MultiNLI, further training with synthesized instances of logical inference improves performance on the same types of logical inference while maintaining the initial performance on MultiNLI. To investigate whether the results of our study are transferable to current work on MultiNLI, we trained models with our synthesized dataset mixed with MultiNLI, and checked (i) whether our synthesized dataset degrades the original performance of models on MultiNLI5 and (ii) whether MultiNLI degrades the ability to generalize to unseen depths of embedded clauses.

Footnote 5: Following the previous work Richardson et al. (2020), we used the MultiNLI mismatched development set for MNLI-test.

Table 5 shows that training BERT on our synthetic data \({\textbf{D}_{1}+\textbf{D}_{2}}\) and MultiNLI increases the accuracy on our test sets \({\textbf{D}_{1}}\) (46.9 to 100.0), \({\textbf{D}_{2}}\) (46.2 to 100.0), and \({\textbf{D}_{3}}\) (46.8 to 67.8) while preserving accuracy on MultiNLI (84.6 to 84.4). This indicates that training BERT with our synthetic data does not degrade performance on commonly used corpora like MultiNLI while improving the performance on monotonicity, which suggests that our data-synthesis approach can be combined with naturalistic datasets. For TreeLSTM and LSTM, however, adding our synthetic dataset decreases accuracy on MultiNLI. One possible reason for this is that a pre-training based model like BERT can mitigate catastrophic forgetting in various types of datasets.

Regarding the ability to generalize to unseen depths of embedded claus

\begin{table}
\begin{tabular}{l l c c c} \hline
**Train** & **Dev/Test** & **BERT** & **LSTM** & **TreeLSTM** \\ \hline \multirow{2}{*}{\({\textbf{D}_{3}}\)} & \({\textbf{D}_{1}}\) & 49.6\(\pm\)0.5 & 48.8\(\pm\)13.2 & 49.8\(\pm\)4.1 \\  & \({\textbf{D}_{2}}\) & 49.8\(\pm\)0.6 & 47.3\(\pm\)12.1 & 51.8\(\pm\)1.1 \\  & \({\textbf{D}_{3}}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.2 \\ \hline \multirow{2}{*}{\({\textbf{D}_{4}}\)} & \({\textbf{D}_{1}}\) & 50.3\(\pm\)1.0 & 46.8\(\pm\)6.5 & 49.0\(\pm\)0.4 \\  & \({\textbf{D}_{2}}\) & 49.6\(\pm\)0.8 & 45.4\(\pm\)1.8 & 49.7\(\pm\)0.3 \\  & \({\textbf{D}_{3}}\) & 50.2\(\pm\)0.7 & 45.1\(\pm\)0.6 & 50.5\(\pm\)0.7 \\  & \({\textbf{D}_{4}}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\ \hline \multirow{2}{*}{\({\textbf{D}_{5}}\)} & \({\textbf{D}_{1}}\) & 49.9\(\pm\)0.7 & 43.7\(\pm\)4.4 & 49.1\(\pm\)1.1 \\  & \({\textbf{D}_{2}}\) & 49.1\(\pm\)0.3 & 43.4\(\pm\)3.9 & 51.4\(\pm\)0.6 \\  & \({\textbf{D}_{3}}\) & 50.6\(\pm\)0.2 & 44.3\(\pm\)2.7 & 50.5\(\pm\)0.3 \\  & \({\textbf{D}_{4}}\) & 50.9\(\pm\)0.8 & 44.4\(\pm\)3.4 & 50.3\(\pm\)0.4 \\  & \({\textbf{D}_{5}}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\ \hline \end{tabular}
\end{table}
Table 4: Results for localisation.

\begin{table}
\begin{tabular}{l l l l l} \hline
**Train** & **Dev/Test** & **BERT** & **LSTM** & **TreeLSTM** \\ \hline \multirow{2}{*}{MNLI} & \({\textbf{D}_{1}}\) & 46.9\(\pm\)0.4 & 47.2\(\pm\)1.1 & 43.4\(\pm\)0.3 \\  & \({\textbf{D}_{2}}\) & 46.2\(\pm\)0.6 & 48.3\(\pm\)1.0 & 49.5\(\pm\)0.4 \\  & \({\textbf{D}_{3}}\) & 46.8\(\pm\)0.8 & 48.9\(\pm\)0.7 & 41.0\(\pm\)0.4 \\  & \({\textbf{D}_{4}}\) & 48.5\(\pm\)0.8 & 50.6\(\pm\)0.5 & 48.5\(\pm\)0.2 \\  & \({\textbf{D}_{5}}\) & 48.9\(\pm\)0.6 & 49.3\(\pm\)0.7 & 48.8\(\pm\)0.5 \\  & MNLI-test & 84.6\(\pm\)0.2 & 64.7\(\pm\)0.3 & 70.4\(\pm\)0.1 \\ \hline \multirow{2}{*}{\({\textbf{D}_{1}+\textbf{D}_{2}}\)} & \({\textbf{D}_{1}}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 & 100.0\(\pm\)0.1 \\  & \({\textbf{D}_{2}}\) & 100.0\(\pm\)0.0 & 89.3\(\pm\)9.0 & 99.8\(\pm\)0.1 \\  & \({\textbf{D}_{3}}\) & 67.8\(\pm\)12.5 & 667.1\(\pm\)13.5 & 76.3\(\pm\)4.1 \\  & \({\textbf{D}_{4}}\) & 46.8\(\pm\)3.7 & 47.1\(\pm\)14.6 & 50.7\(\pm\)7.8 \\  & \({\textbf{D}_{5}}\) & 41.2\(\pm\)4.3 & 46.7\(\pm\)11.2 & 47.5\(\pm\)3.7 \\  & MNLI-test & 84.4\(\pm\)0.2 & 39.7\(\pm\)0.5 & 63.0\(\pm\)0.2 \\ \hline \multirow{2}{*}{\({\textbf{D}_{1}+\textbf{D}_{2}+\textbf{D}_{1}}\)} & \({\textbf{D}_{1}}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 \\  & MNLI-test & 84.0\(\pm\)0.0 & 97.1\(\pm\)5.0 & 99.8\(\pm\)0.0 \\ \cline{1-1}  & \({\textbf{D}_{3}}\) & 100.0\(\pm\)0.0 & 89.2\(\pm\)5.1 & 98.3\(\pm\)1.1 \\ \cline{1-1}  & \({\textbf{D}_{4}}\) & 70.9\(\pm\)7.3 & 73.4\(\pm\)10.9 & 76.1\(\pm\)5.6 \\ \cline{1-1}  & \({\textbf{D}_{5}}\) & 42.4\(\pm\)4.2 & 47.8\(\pm\)0.9 & 57.0\(\pm\)4.3 \\ \cline{1-1}  & MNLI-test & 84.4\(\pm\)0.1 & 39.7\(\pm\)0.4 & 62.8\(\pm\)0.2 \\ \hline \end{tabular}
\end{table}
Table 5: Results for productivity where models were trained with our synthesized dataset mixed with MultiNLI (MNLI).

 

models on our synthetic test set containing embedded clauses one level deeper than the training set exceeds chance, but the improvement becomes smaller with the addition of MultiNLI. In particular, with the addition of MultiNLI, the models tend to change wrong predictions in cases where a hypothesis contains a phrase not occurring in a premise but the premise entails the hypothesis. Such inference patterns are contrary to the heuristics in MultiNLI McCoy et al. (2019). This indicates that there may be some trade-offs in terms of performance between inference patterns in the training set and those in the test set.

## 5 Related Work

The question of whether neural networks are capable of processing compositionality has been widely discussed Fodor and Pylyshyn (1988); Marcus (2003). Recent empirical studies illustrate the importance and difficulty of evaluating the capability of neural models. Generation tasks using artificial datasets have been proposed for testing whether models compositionally interpret training data from the underlying grammar of the data Lake and Baroni (2017); Hupkes et al. (2018); Saxton et al. (2019); Loula et al. (2018); Hupkes et al. (2019); Bernardy (2018). However, these conclusions are controversial, and it remains unclear whether the failure of models on these tasks stems from their inability to deal with compositionality.

Previous studies using logical inference tasks have also reported both positive and negative results. Assessment results on propositional logic Evans et al. (2018), first-order logic Mul and Zuidema (2019), and natural logic Bowman et al. (2015) show that neural networks can generalize to unseen words and lengths. In contrast, Geiger et al. (2019) obtained negative results by testing models under fair conditions of natural logic. Our study suggests that these conflicting results come from an absence of perspective on combinations of semantic properties.

Regarding assessment of the behavior of modern language models, Linzen et al. (2016), Tran et al. (2018), and Goldberg (2019) investigated their syntactic capabilities by testing such models on subject-verb agreement tasks. Many studies of NLI tasks Liu et al. (2019); Glockner et al. (2018); Poliak et al. (2018); Tsuchiya (2018); McCoy et al. (2019); Rozen et al. (2019); Ross and Pavlick (2019) have provided evaluation methodologies and found that current NLI models often fail on particular inference types, or that they learn undesired heuristics from the training set. In particular, recent works Yanaka et al. (2019, 2019); Richardson et al. (2020) have evaluated models on monotonicity, but did not focus on the ability to generalize to unseen combinations of patterns. Monotonicity covers various systematic inferential patterns, and thus is an adequate semantic phenomenon for assessing inferential systematicity in natural language. Another benefit of focusing on monotonicity is that it provides hard problem settings against heuristics McCoy et al. (2019), which fail to perform downward-entailing inferences where the hypothesis is longer than the premise.

## 6 Conclusion

We introduced a method for evaluating whether DNN-based models can learn systematicity of monotonicity inference under four aspects. A series of experiments showed that the capability of three models to capture systematicity of predicate replacements was limited to cases where the positions of the constituents were similar between the training and test sets. For embedding monotonicity, no models consistently drew inferences involving embedded clauses whose depths were two levels deeper than those in the training set. This suggests that models fail to capture inferential systematicity of monotonicity and its productivity.

We also found that BERT trained with our synthetic dataset mixed with MultiNLI maintained performance on MultiNLI while improving the performance on monotonicity. This indicates that though current DNN-based models do not systematically interpret monotonicity inference, some models might have sufficient ability to memorize different types of reasoning. We hope that our work will be useful in future research for realizing more advanced models that are capable of appropriately performing arbitrary inferences.

## Acknowledgement

We thank the three anonymous reviewers for their helpful comments and suggestions. We are also grateful to Benjamin Heinzerling and Sosuke Kobayashi for helpful discussions. This work was partially supported by JSPS KAKENHI Grant Numbers JP20K19868 and JP18H03284, Japan 



## References

* Abzianidze (2015) Lasha Abzianidze. 2015. A tableau prover for natural logic and language. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-2015)_, pages 2492-2502.
* Aydede (1997) Murat Aydede. 1997. Language of thought: The connectionist contribution. _Minds and Machines_, 7(1):57-101.
* van Benthem (1983) Johan van Benthem. 1983. Determiners and logic. _Linguistics and Philosophy_, 6(4):447-478.
* Bernardy (2018) Jean-Philippe Bernardy. 2018. Can recurrent neural networks learn nested recursion. _Linguistic Issues in Language Technology_, 16.
* Blackburn and Bos (2005) Patrick Blackburn and Johan Bos. 2005. _Representation and Inference for Natural Language: A First Course in Computational Semantics_. Center for the Study of Language and Information.
* Bowman et al. (2015) Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015. Recursive neural networks can learn logical semantics. In _Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality_, pages 12-21.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. _Recognizing Textual Entailment: Models and Applications_. Morgan & Claypool Publishers.
* Devlin et al. (2019) Jacob Devlin, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2019)_, pages 4171-4186.
* Evans et al. (2018) Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. 2018. Can neural networks understand logical entailment? In _International Conference on Learning Representations (ICLR-2018)_.
* Fodor and Pylyshyn (1988) Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. _Cognition_, 28(1-2):3-71.
* Geiger et al. (2019) Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2019. Posing fair generalization tasks for natural language inference. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP-2019)_, pages 4484-4494.
* Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL-2018)_, pages 650-655.
* Goldberg (2019) Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. _CoRR_, abs/1901.05287.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Hu et al. (2019) Hai Hu, Qi Chen, Kyle Richardson, Arteyee Mukherjee, Lawrence S. Moss, and Sandra Kubler. 2019. Monalog: a lightweight system for natural language inference based on monotonicity. _CoRR_, abs/1910.08772.
* Hupkes et al. (2019) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2019. Compositionality decomposed: how do neural networks generalise? _CoRR_, abs/1908.08351.
* Hurpes et al. (2018) Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. _Journal of Artificial Intelligence Research_, 61:907-926.
* Icard and Moss (2014) Thomas Icard and Lawrence Moss. 2014. Recent progress in monotonicity. _Linguistic Issues in Language Technology_, 9(7):167-194.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR-2015)_.
* Lake and Baroni (2017) Brenden M. Lake and Marco Baroni. 2017. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In _International Conference on Machine Learning (ICML-2017)_.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521-535.
* Liu et al. (2019) Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Incubation by fine-tuning: A method for analyzing challenge datasets. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2019)_, pages 2171-2179.
* Loula et al. (2018) Joao Loula, Marco Baroni, and Brenden Lake. 2018. Rearranging the familiar: Testing compositional generalization in recurrent networks. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 108-114.
* Marcus (2003) Gary Marcus. 2003. _The Algebraic Mind: Integrating Connectionism and Cognitive Science_. MIT Press.
* McCoy et al. (2019) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL-2019)_, pages 3428-3448.
* Mineshima et al. (2015) Koji Mineshima, Pascual Martinez-Gomez, Yusuke Miyao, and Daisuke Bekki. 2015. Higher-order logical inference with compositional semantics. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-2015)_, pages 2055-2061.
* Mul and Zuidema (2019) Mathijs Mul and Willem Zuidema. 2019. Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization. _CoRR_, abs/1908.08351.
* Pagin and Westerstahl (2010) Peter Pagin and Dag Westerstahl. 2010. Compositionality I: Definitions and variants. _Philosophy Compass_, 5(3):250-264.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP-2014)_, pages 1532-1543.
* Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM-2018)_, pages 180-191.
* Richardson et al. (2020) Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In _Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-2020)_.
* Ross and Pavlick (2019) Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP-2019)_, pages 2230-2240.
* Rozen et al. (2019) Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL-2019)_, pages 196-205.
* Saxton et al. (2019) David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In _International Conference on Learning Representations (ICLR-2019)_.
* Tran et al. (2018) Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP-2018)_, pages 4731-4736.
* Tran and Cheng (2018) Nam Khanh Tran and Weiwei Cheng. 2018. Multiplicative tree-structured long short-term memory networks for semantic representations. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM-2018)_, pages 276-286.
* Tsuchiya (2018) Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In _Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC-2018)_.
* Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the International Conference on Learning Representations (ICLR-2019)_.
* Williams et al. (2018) Adina Williams, Nikita Nagia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2018)_, pages 1112-1122.
* Yanaka et al. (2019a) Hitomi Yanaka, Koqi Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abizaniudze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 31-40.
* Yanaka et al. (2019b) Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abizaniudze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_, pages 250-255.

## Appendix A Appendix

### Lexical entries and replacement examples

Table 6 shows a context-free grammar and a set of predicate replacements used to generate inference examples. Regarding the context-free grammar, we consider premise-hypothesis pairs containing the quantifier \(Q\) in the subject position, and the predicate replacement is performed in both the first and second arguments of the quantifier. When generating premise-hypothesis pairs involving embedding monotonicity, we consider inferences involving four types of predicate replacements (hyponyms \(N_{hyp}\), adjectives \(Adj\), prepositions \(PP\), and relative clauses \(RelC\)) in the



first argument of the quantifier. To generate natural sentences consistently, we use the past tense for verbs; for lexical entries and predicate replacements, we select those that do not violate selectional restriction.

To check the gold labels for the generated premise-hypothesis pairs, we translate each sentence to a first-order logic (FOL) formula and test if the entailment relation holds by theorem proving. The FOL formulas are compositionally derived by combining lambda terms assigned to each lexical item in accordance with meaning composition rules specified in the CFG rules in the standard way Blackburn and Bos (2005). Since our purpose is to check the polarity of monotonicity marking, vague quantifiers such as _few_ are represented according to their polarity. For example, we map the quantifier _few_ onto the lambda-term \(\lambda P\lambda Q{\neg}x(\textbf{few}(x)\wedge P(x)\wedge Q(x))\).

### Results on embedding monotonicity

Table 7 shows all results on embedding monotonicity. This indicates that all models partially generalize to inferences containing embedded clauses one level deeper than the training set, but fail to generalize to inferences containing embedded clauses two or more levels deeper.

\begin{table}
\begin{tabular}{l l l} \hline  & \multicolumn{3}{c}{**Context-free grammar for premise sentences**} \\ \(S\) & \(\rightarrow\) & \(NP\ IV_{1}\) \\ \(NP\) & \(\rightarrow\) & \(Q\ N\mid Q\ N\ \overline{S}\) \\ \(\overline{S}\) & \(\rightarrow\) & \(WhNP\ TV\ NP\mid WhNP\ VP\ TV\mid NP\ TV\) \\ \hline \multirow{4}{*}{\(Q\)} & \(\rightarrow\) & \{no, at most three, less than three, few, some, at least three, more than three, a few\} \\ \(N\) & \(\rightarrow\) & \{dog, rabbit, lion, cat, bear, tiger, elephant, fox, monkey, wolf\} \\ \(IV_{1}\) & \(\rightarrow\) & \{ran, walked, came, wallzed, swam, rushed, danced, drawled, escaped, left\} \\ \(IV_{2}\) & \(\rightarrow\) & \{laughed, groaned, roared, screamed, cried\} \\ \(TV\) & \(\rightarrow\) & \{kissed, kicked, hit, cleaned, touched, loved, accepted, hurt, lkoked, followed\} \\ \(WhNP\) & \(\rightarrow\) & \{that, which\} \\ \(N_{hypn}\) & \(\rightarrow\) & \{animal, creature, mammal, beast\} \\ \(Adj\) & \(\rightarrow\) & \{small, large, crazy, polite, wild\} \\ \(PP\) & \(\rightarrow\) & \{in the area, on the ground, at the park, near the shore, around the island\} \\ \(RelC\) & \(\rightarrow\) & \{which ate dinner, that liked flowers, which hated the sun, that stayed up late\} \\ \(Adv\) & \(\rightarrow\) & \{slowly, quickly, seriously, suddenly, lazy\} \\ \hline \end{tabular}
\end{table}
Table 6: A context-free grammar and a set of predicate replacements used to generate inference examples. Predicate replacement is applied to \(N\) or \(IV_{1}\), replacing it with a corresponding phrase.

 



\begin{table}
\begin{tabular}{l c r r l} \hline
**Train** & **Test** & **BERT** & **LSTM** & **TreeLSTM** \\ \hline \(\mathbf{D}_{1}\) & \(\mathbf{D}_{1}\) & 100.0\(\pm\)0.0 & 91.1\(\pm\)5.4 & 100.0\(\pm\)0.0 \\  & \(\mathbf{D}_{2}\) & 44.1\(\pm\)6.4 & 34.1\(\pm\)3.8 & 48.1\(\pm\)1.2 \\  & \(\mathbf{D}_{3}\) & 47.6\(\pm\)3.2 & 45.1\(\pm\)5.1 & 48.5\(\pm\)1.8 \\  & \(\mathbf{D}_{4}\) & 49.6\(\pm\)1.0 & 44.4\(\pm\)6.5 & 50.1\(\pm\)2.1 \\  & \(\mathbf{D}_{5}\) & 49.9\(\pm\)1.1 & 44.1\(\pm\)5.3 & 50.3\(\pm\)1.1 \\ \hline \(\mathbf{D}_{1}\cup\mathbf{D}_{2}\) & \(\mathbf{D}_{1}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\  & \(\mathbf{D}_{2}\) & 100.0\(\pm\)0.0 & 99.8\(\pm\)0.2 & 99.5\(\pm\)0.1 \\  & \(\mathbf{D}_{3}\) & 75.2\(\pm\)10.0 & 75.4\(\pm\)10.8 & 86.4\(\pm\)4.1 \\  & \(\mathbf{D}_{4}\) & 55.0\(\pm\)3.7 & 57.7\(\pm\)8.7 & 58.6\(\mp\)7.8 \\  & \(\mathbf{D}_{5}\) & 49.9\(\pm\)4.4 & 45.8\(\pm\)4.0 & 48.4\(\pm\)3.7 \\ \hline \(\mathbf{D}_{1}\cup\mathbf{D}_{2}\cup\mathbf{D}_{3}\) & \(\mathbf{D}_{1}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 \\  & \(\mathbf{D}_{2}\) & 100.0\(\pm\)0.0 & 95.1\(\pm\)7.8 & 99.6\(\pm\)0.0 \\  & \(\mathbf{D}_{3}\) & 100.0\(\pm\)0.0 & 85.2\(\pm\)8.9 & 97.7\(\pm\)1.1 \\  & \(\mathbf{D}_{4}\) & 77.9\(\pm\)10.8 & 59.7\(\pm\)10.8 & 68\(\pm\)5.6 \\  & \(\mathbf{D}_{5}\) & 53.5\(\pm\)19.6 & 55.1\(\pm\)8.2 & 49.6\(\pm\)4.3 \\ \hline \(\mathbf{D}_{1}\cup\mathbf{D}_{2}\cup\mathbf{D}_{3}\cup\mathbf{D}_{4}\) & \(\mathbf{D}_{1}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\  & \(\mathbf{D}_{2}\) & 100.0\(\pm\)0.0 & 99.4\(\pm\)1.1 & 99.7\(\pm\)0.2 \\  & \(\mathbf{D}_{3}\) & 100.0\(\pm\)0.0 & 91.5\(\pm\)4.0 & 98.9\(\pm\)1.1 \\  & \(\mathbf{D}_{4}\) & 100.0\(\pm\)0.0 & 74.1\(\pm\)4.2 & 94.0\(\pm\)2.3 \\  & \(\mathbf{D}_{5}\) & 89.1\(\pm\)5.4 & 64.2\(\pm\)4.7 & 69.5\(\pm\)4.1 \\ \hline \(\mathbf{D}_{1}\cup\mathbf{D}_{2}\cup\mathbf{D}_{3}\cup\mathbf{D}_{4}\cup \mathbf{D}_{5}\) & \(\mathbf{D}_{1}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\  & \(\mathbf{D}_{2}\) & 100.0\(\pm\)0.0 & 95.8\(\pm\)7.3 & 99.8\(\pm\)0.1 \\  & \(\mathbf{D}_{3}\) & 100.0\(\pm\)0.0 & 90.5\(\pm\)13.1 & 99.1\(\pm\)0.2 \\  & \(\mathbf{D}_{4}\) & 100.0\(\pm\)0.0 & 90.2\(\pm\)6.0 & 94.8\(\pm\)0.1 \\  & \(\mathbf{D}_{5}\) & 100.0\(\pm\)0.0 & 93.6\(\pm\)3.1 & 83.2\(\pm\)12.1 \\ \hline \(\mathbf{D}_{2}\) & \(\mathbf{D}_{1}\) & 36.4\(\pm\)14.4 & 25.3\(\pm\)9.3 & 44.9\(\pm\)4.1 \\  & \(\mathbf{D}_{2}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.2 \\  & \(\mathbf{D}_{3}\) & 47.6\(\pm\)10.3 & 43.9\(\pm\)17.5 & 51.8\(\pm\)1.1 \\  & \(\mathbf{D}_{4}\) & 61.7\(\pm\)7.8 & 57.9\(\pm\)14.7 & 51.7\(\pm\)0.6 \\  & \(\mathbf{D}_{5}\) & 42.6\(\pm\)5.1 & 47.2\(\pm\)2.9 & 50.9\(\pm\)0.4 \\ \hline \(\mathbf{D}_{3}\) & \(\mathbf{D}_{1}\) & 49.6\(\pm\)0.5 & 48.8\(\pm\)13.2 & 49.8\(\pm\)4.1 \\  & \(\mathbf{D}_{2}\) & 49.8\(\pm\)0.6 & 47.3\(\pm\)12.1 & 51.8\(\pm\)1.1 \\  & \(\mathbf{D}_{3}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.2 \\  & \(\mathbf{D}_{4}\) & 49.7\(\pm\)1.0 & 42.0\(\pm\)0.6 & 51.3\(\pm\)0.7 \\  & \(\mathbf{D}_{5}\) & 50.0\(\pm\)0.4 & 38.4\(\pm\)9.6 & 49.8\(\pm\)0.3 \\ \hline \(\mathbf{D}_{4}\) & \(\mathbf{D}_{1}\) & 50.3\(\pm\)1.0 & 46.8\(\pm\)6.5 & 49.0\(\pm\)0.4 \\  & \(\mathbf{D}_{2}\) & 49.6\(\pm\)0.8 & 45.4\(\pm\)1.8 & 49.7\(\pm\)0.3 \\  & \(\mathbf{D}_{3}\) & 50.2\(\pm\)0.7 & 45.1\(\pm\)0.6 & 50.5\(\pm\)0.7 \\  & \(\mathbf{D}_{4}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\  & \(\mathbf{D}_{5}\) & 49.7\(\pm\)0.5 & 45.1\(\pm\)0.9 & 50.5\(\pm\)1.1 \\ \hline \(\mathbf{D}_{5}\) & \(\mathbf{D}_{1}\) & 49.9\(\pm\)0.7 & 43.7\(\pm\)4.4 & 49.1\(\pm\)1.1 \\  & \(\mathbf{D}_{2}\) & 49.1\(\pm\)0.3 & 43.4\(\pm\)3.9 & 51.4\(\pm\)0.6 \\  & \(\mathbf{D}_{3}\) & 50.6\(\pm\)0.2 & 44.3\(\pm\)2.7 & 50.5\(\pm\)0.3 \\  & \(\mathbf{D}_{4}\) & 50.9\(\pm\)0.8 & 44.4\(\pm\)3.4 & 50.3\(\pm\)0.4 \\  & \(\mathbf{D}_{5}\) & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.0 & 100.0\(\pm\)0.1 \\ \hline \end{tabular}
\end{table}
Table 7: All results on embedding monotonicity.