SlowGuess/ACL_2020_OCR huggingface dataset:
Hitomi Yanaka<sup>1</sup>, Koji Mineshima<sup>2</sup>, Daisuke Bekki<sup>3</sup>, and Kentaro Inui<sup>4,1</sup>
$^{1}$ RIKEN,  $^{2}$ Keio University,  $^{3}$ Ochanomizu University,  $^{4}$ Tohoku University hitomi.yanaka@riken.jp, minesima@abelard.flet.keio.ac.jp, bekki@is.ocha.ac.jp, inui@ecei.tohoku.ac.jp
Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.
Natural language inference (NLI), a task whereby a system judges whether given a set of premises  $P$  semantically entails a hypothesis  $H$  (Dagan et al., 2013; Bowman et al., 2015), is a fundamental task for natural language understanding. As with other NLP tasks, recent studies have shown a remarkable impact of deep neural networks in NLI (Williams et al., 2018; Wang et al., 2019; Devlin et al., 2019). However, it remains unclear to what extent DNN-based models are capable of learning the compositional generalization underlying NLI from given labeled training instances.
Systematicity of inference (or inferential systematicity) (Fodor and Pylyshyn, 1988; Aydede, 1997) in natural language has been intensively studied in the field of formal semantics. From among the various aspects of inferential systematicity, in the context of NLI, we focus on monotonicity (van Benthem, 1983; Icard and Moss, 2014) and its productivity. Consider the following premise-hypothesis pairs (1)-(3), which have the target label entailment:
(1)  $P$ : Some [puppies  $\uparrow$ ] ran.
H: Some dogs ran.
(2)  $P\colon \mathbf{No}[\text{cats}\downarrow ]$  ran.
H: No small cats ran.
(3)  $P$ : Some [puppies which chased no  $[\text{cats} \downarrow]$ ] ran.
H: Some dogs which chased no small cats ran.
As in (1), for example, quantifiers such as some exhibit upward monotone (shown as  $[\dots \uparrow ]$ ), and replacing a phrase in an upward-entailing context in a sentence with a more general phrase (replacing puppies in  $P$  with dogs as in  $H$ ) yields a sentence inferable from the original sentence. In contrast, as in (2), quantifiers such as no exhibit downward monotone (shown as  $[\dots \downarrow ]$ ), and replacing a phrase in a downward-entailing context with a more specific phrase (replacing cats in  $P$  with small cats as in  $H$ ) yields a sentence inferable from the original sentence. Such primitive inference patterns combine recursively as in (3). This manner of monotonicity and its productivity produces a potentially infinite number of inferential patterns. Therefore, NLI models must be capable of systematically interpreting such primitive patterns and reasoning over unseen combinations of patterns. Although many studies have addressed this issue by modeling logical reasoning in formal semantics (Abzianidze, 2015; Mineshima et al., 2015; Hu et al., 2019) and testing DNN-based models on monotonicity inference (Yanaka et al., 2019a,b; Richardson et al.,
6105
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6105-6117 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An illustration of the basic idea. For Systematicity and Productivity, we train models on the Train 1 and Train 2 sets and test them on the Test set. Arrow  $(\longrightarrow)$  means entailment relation; LEX, ADJ, and PREP mean predicate replacements for lexical relations, adjectives, and prepositional phrases, respectively. In Productivity, we use various quantifiers and predicate replacements in each depth.
2020), the ability of DNN-based models to generalize to unseen combinations of patterns is still underexplored.
Given this background, we investigate the systematic generalization ability of DNN-based models on four aspects of monotonicity: (i) systematicity of predicate replacements (i.e., replacements with a more general or specific phrase), (ii) systematicity of embedding quantifiers, (iii) productivity, and (iv) localism (see Section 2.2). To this aim, we introduce a new evaluation protocol where we (i) synthesize training instances from sampled sentences and (ii) systematically control which patterns are shown to the models in the training phase and which are left unseen. The rationale behind this protocol is twofold. First, patterns of monotonicity inference are highly systematic, so we can create training data with arbitrary combinations of patterns, as in examples (1)-(3). Second, evaluating the performance of the models trained with well-known NLI datasets such as MultiNLI (Williams et al., 2018) might severely underestimate the ability of the models because such datasets tend to contain only a limited number of training instances that exhibit the inferential patterns of interest. Furthermore, using such datasets would prevent us from identifying which combinations of patterns the models can infer from which patterns in the training data.
This paper makes two primary contributions. First, we introduce an evaluation protocol<sup>1</sup> using
the systematic control of the training/test split under various combinations of semantic properties to evaluate whether models learn inferential systematicity in natural language. Second, we apply our evaluation protocol to three NLI models and present evidence suggesting that, while all models generalize to unseen combinations of lexical and logical phenomena, their generalization ability is limited to cases where sentence structures are nearly the same as those in the training set.
2.1 Basic idea
Figure 1 illustrates the basic idea of our evaluation protocol on monotonicity inference. We use synthesized monotonicity inference datasets, where NLI models should capture both (i) monotonicity directions (upward/downward) of various quantifiers and (ii) the types of various predicate replacements in their arguments. To build such datasets, we first generate a set of premises  $G_{d}^{\mathbf{Q}}$  by a context-free grammar  $G$  with depth  $d$  (i.e., the maximum number of applications of recursive rules), given a set of quantifiers  $\mathbf{Q}$ . Then, by applying  $G_{d}^{\mathbf{Q}}$  to elements of a set of functions for predicate replacements (or replacement functions for short)  $\mathbf{R}$  that rephrase a constituent in the input premise and return a hypothesis, we obtain a set  $\mathbf{D}_{d}^{\mathbf{Q},\mathbf{R}}$  of premise-hypothesis pairs defined as
$$
\mathbf {D} _ {d} ^ {\mathbf {Q}, \mathbf {R}} = \left\{\left(P, H\right) \mid P \in G _ {d} ^ {\mathbf {Q}}, \exists r \in \mathbf {R} (r (P) = H) \right\}.
$$
For example, the premise Some puppies ran is generated from the quantifier some in  $\mathbf{Q}$  and
<sup>1</sup>The evaluation code will be publicly available at https://github.com/verypluming/systematicity.
the production rule  $S \to Q, N, IV$ , and thus it is an element of  $G_1^{\mathbf{Q}}$ . By applying this premise to a replacement function that replaces the word in the premise with its hypernym (e.g., puppy  $\sqsubseteq$  dog), we provide the premise-hypothesis pair Some puppies ran  $\Rightarrow$  Some dogs ran in Fig. 1.
We can control which patterns are shown to the models during training and which are left unseen by systematically splitting  $\mathbf{D}_d^{\mathbf{Q},\mathbf{R}}$  into training and test sets. As shown on the left side of Figure 1, we consider how to test the systematic capacity of models with unseen combinations of quantifiers and predicate replacements. To expose models to primitive patterns regarding  $\mathbf{Q}$  and  $\mathbf{R}$ , we fix an arbitrary element  $q$  from  $\mathbf{Q}$  and feed various predicate replacements into the models from the training set of inferences  $\mathbf{D}_d^{\{q\},\mathbf{R}}$  generated from combinations of the fixed quantifier and all predicate replacements. Also, we select an arbitrary element  $r$  from  $\mathbf{R}$  and feed various quantifiers into the models from the training set of inferences  $\mathbf{D}_d^{\mathbf{Q},\{r\}}$  generated from combinations of all quantifiers and the fixed predicate replacement.
We then test the models on the set of inferences generated from unseen combinations of quantifiers and predicate replacements. That is, we test them on the set of inferences  $\mathbf{D}_d^{\overline{\{q\},\{r\}}}$  generated from the complements  $\overline{\{q\}},\overline{\{r\}}$  of  $\{q\} ,\{r\}$ . If models capture inferential systematicity in combinations of quantifiers and predicate replacements, they can correctly perform all inferences in  $\mathbf{D}_d^{\overline{\{q\},\{r\}}}$  on an arbitrary split based on  $q,r$ .
Similarly, as shown on the right side of Figure 1, we can test the productive capacity of models with unseen depths by changing the training/test split based on  $d$ . For example, by training models on  $\mathbf{D}_d^{\mathbf{Q},\mathbf{R}}$  and testing them on  $\mathbf{D}_{d + 1}^{\mathbf{Q},\mathbf{R}}$ , we can evaluate whether models generalize to one deeper depth. By testing models with an arbitrary training/test split of  $\mathbf{D}_d^{\mathbf{Q},\mathbf{R}}$  based on semantic properties of monotonicity inference (i.e., quantifiers, predicate replacements, and depths), we can evaluate whether models systematically interpret them.
2.2 Evaluation protocol
To test NLI models from multiple perspectives of inferential systematicity in monotonicity inferences, we focus on four aspects: (i) systematicity of predicate replacements, (ii) systematicity of embedding quantifiers, (iii) productivity, and (iv) localism. For each aspect, we use a set  $\mathbf{D}_d^{\mathbf{Q},\mathbf{R}}$
of premise-hypothesis pairs. Let  $\mathbf{Q} = \mathbf{Q}^{\uparrow}\cup \mathbf{Q}^{\downarrow}$  be the union of a set of selected upward quantifiers  $\mathbf{Q}^{\uparrow}$  and a set of selected downward quantifiers  $\mathbf{Q}^{\downarrow}$  such that  $|\mathbf{Q}^{\uparrow}| = |\mathbf{Q}^{\downarrow}| = n$ . Let  $\mathbf{R}$  be a set of replacement functions  $\{r_1,\ldots ,r_m\}$ , and  $d$  be the embedding depth, with  $1\leq d\leq s$ . (4) is an example of an element of  $\mathbf{D}_1^{\mathbf{Q},\overline{\mathbf{R}}}$ , containing the quantifier some in the subject position and the predicate replacement using the hypernym relation dogs  $\sqsubseteq$  animals in its upward-entailing context without embedding.
(4)  $P\colon \underline{\text{Some dogs ran}}\Rightarrow H\colon \underline{\text{Some animals ran}}$
I. Systematicity of predicate replacements The following describes how we test the extent to which models generalize to unseen combinations of quantifiers and predicate replacements. Here, we expose models to all primitive patterns of predicate replacements like (4) and (5) and all primitive patterns of quantifiers like (6) and (7). We then test whether the models can systematically capture the difference between upward quantifiers (e.g., several) and downward quantifiers (e.g., no) as well as the different types of predicate replacements (e.g., the lexical relation dogs  $\sqsubseteq$  animals and the adjective deletion small dogs  $\sqsubseteq$  dogs) and correctly interpret unseen combinations of quantifiers and predicate replacements like (8) and (9).
(5)  $P$ : Some small dogs ran  $\Rightarrow H$ : Some dogs ran
(6)  $P$ : Several dogs ran  $\Rightarrow H$ : Several animals ran
(7)  $P\colon \mathbf{No}$  animals ran  $\Rightarrow H\colon \mathbf{No}$  dogs ran
(8)  $P$ : Several small dogs ran  $\Rightarrow H$ : Several dogs ran
(9)  $P\colon \mathbf{No}$  dogs ran  $\Rightarrow H\colon \mathbf{No}$  small dogs ran
Here, we consider a set of inferences  $\mathbf{D}_1^{\mathbf{Q},\mathbf{R}}$  whose depth is 1. We move from harder to easier tasks by gradually changing the training/test split according to combinations of quantifiers and predicate replacements. First, we expose models to primitive patterns of  $\mathbf{Q}$  and  $\mathbf{R}$  with the minimum training set. Thus, we define the initial training set  $\mathbf{S}_1$  and test set  $\mathbf{T}_1$  as follows:
$$
\left(\mathbf {S} _ {1}, \mathbf {T} _ {1}\right) = \left(\mathbf {D} _ {1} ^ {\{q \}, \mathbf {R}} \cup \mathbf {D} _ {1} ^ {\mathbf {Q}, \{r \}}, \mathbf {D} _ {1} ^ {\overline {{\{q \}}} , \overline {{\{r \}}}}\right)
$$
where  $q$  is arbitrarily selected from  $\mathbf{Q}$ , and  $r$  is arbitrarily selected from  $\mathbf{R}$ .
Next, we gradually add the set of inferences generated from combinations of an upward-downward quantifier pair and all predicate replacements to the training set. In the examples above, we add (8) and (9) to the training set to simplify the task. We assume a set  $\mathbf{Q}'$  of a pair of upward/downward quantifiers, namely,  $\{(q^{\uparrow}, q^{\downarrow}) \mid (q^{\uparrow}, q^{\downarrow}) \subseteq \mathbf{Q}^{\uparrow} \times \mathbf{Q}^{\downarrow}, q^{\uparrow}, q^{\downarrow} \neq q\}$ . We consider
6107
a set  $\operatorname{perm}(\mathbf{Q}')$  consisting of permutations of  $\mathbf{Q}'$ . For each  $p \in \operatorname{perm}(\mathbf{Q}')$ , we gradually add a set of inferences generated from  $p(i)$  to the training set  $\mathbf{S}_i$  with  $1 < i \leq n - 1$ . Then, we provide a test set  $\mathbf{T}_i$  generated from the complement  $\overline{\mathbf{Q}_i}$  of  $\mathbf{Q}_i = \{x \mid \exists y(x,y) \in \mathbf{Q}_i' \text{ or } \exists y(y,x) \in \mathbf{Q}_i'\}$  and  $\{\overline{r}\}$  where  $\mathbf{Q}_i' = \{p(1),\dots,p(i)\}$ . This protocol is summarized as
$$
\mathbf {S} _ {i + 1} = \mathbf {S} _ {i} \cup \mathbf {D} _ {1} ^ {\left\{q _ {i} ^ {\uparrow}, q _ {i} ^ {\downarrow} \right\}, \mathbf {R}},
$$
$$
\mathbf {T} _ {i} = \mathbf {D} _ {1} ^ {\overline {{\mathbf {Q} _ {i}}} \backslash \{r \}} \quad \mathrm {w i t h} 1 <   i \leq n - 1
$$
where  $(q_i^\uparrow, q_i^\downarrow) = p(i)$ .
To evaluate the extent to which the generalization ability of models is robust for different syntactic structures, we use an additional test set  $\mathbf{T}_i' = \mathbf{D}_1^{\overline{\mathbf{Q}_i},\{\overline{r}\}}$  generated using three production rules. The first is the case where one adverb is added at the beginning of the sentence, as in example (10).
(10)  $P_{adv}$ : Slowly, several small dogs ran  
 $H_{adv}$ : Slowly, several dogs ran
The second is the case where a three-word prepositional phrase is added at the beginning of the sentence, as in example (11).
(11)  $P_{prep}$ : Near the shore, several small dogs ran  
 $H_{prep}$ : Near the shore, several dogs ran
The third is the case where the replacement is performed in the object position, as in example (12).
(12)  $P_{obj}$ : Some tiger touched several small dogs  
 $H_{obj}$ : Some tiger touched several dogs
We train and test models  $|\mathrm{perm}(\mathbf{Q}')|$  times, then take the average accuracy as the final evaluation result.
II. Systematicity of embedding quantifiers To properly interpret embedding monotonicity, models should detect both (i) the monotonicity direction of each quantifier and (ii) the type of predicate replacements in the embedded argument. The following describes how we test whether models generalize to unseen combinations of embedding quantifiers. We expose models to all primitive combination patterns of quantifiers and predicate replacements like (4)-(9) with a set of non-embedding monotonicity inferences  $\mathbf{D}_1^{\mathbf{Q},\mathbf{R}}$  and some embedding patterns like (13), where  $Q_{1}$  and  $Q_{2}$  are chosen from a selected set of upward or downward quantifiers such as some or no. We then
test the models with an inference with an unseen quantifier several in (14) to evaluate whether models can systematically interpret embedding quantifiers.
(13)  $P: \mathbf{Q}_1$  animals that chased  $\mathbf{Q}_2$  dogs ran  
 $H: \mathbf{Q}_1$  animals that chased  $\mathbf{Q}_2$  animals ran
(14)  $P$ : Several animals that chased several dogs ran  
 $H$ : Several animals that chased several animals ran
We move from harder to easier tasks of learning embedding quantifiers by gradually changing the training/test split of a set of inferences  $\mathbf{D}_2^{\mathbf{Q},\overline{\mathbf{R}}}$  whose depth is 2, i.e., inferences involving one embedded clause.
We assume a set  $\mathbf{Q}'$  of a pair of upward and downward quantifiers as  $\mathbf{Q}' \equiv \{(q^{\uparrow}, q^{\downarrow}) \mid (q^{\uparrow}, q^{\downarrow}) \subseteq \mathbf{Q}^{\uparrow} \times \mathbf{Q}^{\downarrow}\}$ , and consider a set  $\operatorname{perm}(\mathbf{Q}')$  consisting of permutations of  $\mathbf{Q}'$ . For each  $p \in \operatorname{perm}(\mathbf{Q}')$ , we gradually add a set of inferences  $\mathbf{D}_2$  generated from  $p(i)$  to the training set  $\mathbf{S}_i$  with  $1 \leq i \leq n - 1$ .
We test models trained with  $\mathbf{S}_i$  on a test set  $\mathbf{T}_i$  generated from the complement  $\overline{\mathbf{Q}_i}$  of  $\mathbf{Q}_i = \{x\mid$ $\exists y(x,y)\in \mathbf{Q}_i'$  or  $\exists y(y,x)\in \mathbf{Q}_i'\}$  where  $\mathbf{Q}_i^{\prime} =$ $\{p(1),\dots ,p(i)\}$  , summarized as
$$
\mathbf {S} _ {0} = \mathbf {D} _ {1} ^ {\mathbf {Q}, \mathbf {R}},
$$
$$
\mathbf {S} _ {i} = \mathbf {S} _ {i - 1} \cup \mathbf {D} _ {2} ^ {\{q _ {i} ^ {\uparrow}, q _ {i} ^ {\downarrow} \}, \mathbf {R}},
$$
$$
\mathbf {T} _ {i} = \mathbf {D} _ {2} ^ {\overline {{\mathbf {Q}}} _ {i}; \mathbf {R}} \quad \text {w i t h} 1 \leq i \leq n - 1
$$
where  $(q_i^\uparrow ,q_i^\downarrow) = p(i)$ . We train and test models  $|\mathrm{perm}(\mathbf{Q}')|$  times, then take the average accuracy as the final evaluation result.
III. Productivity Productivity (or recursiveness) is a concept related to systematicity, which refers to the capacity to grasp an indefinite number of natural language sentences or thoughts with generalization on composition. The following describes how we test whether models generalize to unseen deeper depths in embedding monotonicity (see also the right side of Figure 1). For example, we expose models to all primitive non-embedding/single-embedding patterns like (15) and (16) and then test them with deeper embedding patterns like (17).
(15)  $P$ : Some dogs ran
H: Some animals ran
(16)  $P$ : Some animals which chased some dogs ran
H: Some animals which chased some animals ran
6108
Depth	Pred.	Monotone	Arg.	Example (premise, hypothesis, label)	Avg. Len.
1	CONJ	DOWNWARD	SECOND	Less than three lions left.
Less than three lions left and cried. ENTAILMENT	4.6
2	PP	UPWARD	FIRST	Few lions that hurt at most three small dogs walked.
Few lions that hurt at most three dogs walked. ENTAILMENT	9.0
3	AdJ	DOWNWARD	FIRST	Some elephant no rabbit which touched a few dogs hit rushed.
Some elephant no rabbit which touched a few small dogs hit rushed.
ENTAILMENT	12.3
4	RC	UPWARD	FIRST	Less than three tigers which accepted several rabbits that loved several foxes more than three monkeys cleaned dawdled.
Less than three tigers which accepted several rabbits that loved several foxes more than three monkeys which ate dinner cleaned dawdled.
ENTAILMENT	16.6
(17)  $P$ : Some animals which chased some cats which followed some dogs ran
H: Some animals which chased some cats which followed some animals ran
To evaluate models on the set of inferences involving embedded clauses with depths exceeding those in the training set, we train models with  $\bigcup_{d\in \{1,\ldots ,i + 1\}}\mathbf{D}_d$  where we refer to  $\mathbf{D}_d^{\mathbf{Q},\mathbf{R}}$  as  $\mathbf{D}_d$  for short, and test the models on  $\bigcup_{d\in \{i + 2,\dots,s\}}\mathbf{D}_d$  with  $1\leq i\leq s - 2$
IV. Localism According to the principle of compositionality, the meaning of a complex expression derives from the meanings of its constituents and how they are combined. One important concern is how local the composition operations should be (Pagin and Westerstahl, 2010). We therefore test whether models trained with inferences involving embedded monotonicity locally perform inferences composed of smaller constituents. Specifically, we train models with examples like (17) and then test the models with examples like (15) and (16). We train models with  $\mathbf{D}_d$  and test the models on  $\bigcup_{k\in \{1,\dots,d\}}\mathbf{D}_k$  with  $3\leq d\leq s$ .
3.1 Data creation
To prepare the datasets shown in Table 1, we first generate premise sentences involving quantifiers from a set of context-free grammar (CFG) rules and lexical entries, shown in Table 6 in the Appendix. We select 10 words from among nouns, intransitive verbs, and transitive verbs as lexical entries. A set of quantifiers  $\mathbf{Q}$  consists of eight elements; we use a set of four downward quantifiers  $\mathbf{Q}^{\downarrow} = \{\text{no, at most three, less than three, few}\}$  and a set of four upward quantifiers  $\mathbf{Q}^{\uparrow} = \{\text{some, at}\}$ .
Table 1: Examples of generated premise-hypothesis pairs. Depth: depth of embedding; Pred.: type of predicate replacements; Monotone: direction of monotonicity; Arg.: argument where the predicate replacement is performed; Avg. Len.: average sentence length.
Function	Example
r1: hyponym	dogs ⊆ animals
r2: adjective	small dogs ⊆ dogs
r3: preposition	dogs in the park ⊆ dogs
r4: relative clause	dogs which ate dinner ⊆ dogs
r5: adverb	ran quickly ⊆ ran
r6: disjunction	ran ⊆ ran or walked
r7: conjunction	ran and barked ⊆ ran
Table 2: Examples of replacement functions.
least three, more than three, a few}, which have the same monotonicity directions in the first and second arguments. We thus consider  $n = |\mathbf{Q}^{\uparrow}| = |\mathbf{Q}^{\downarrow}| = 4$  in the protocol in Section 2.2. The ratio of each monotonicity direction (upward/downward) of generated sentences is set to  $1:1$ . We then generate hypothesis sentences by applying replacement functions to premise sentences according to the polarities of constituents. The set of replacement functions  $\mathbf{R}$  is composed of the seven types of lexical replacements and phrasal additions in Table 2. We remove unnatural premise-hypothesis pairs in which the same words or phrases appear more than once.
For embedding monotonicity, we consider inferences involving four types of replacement functions in the first argument of the quantifier in Table 2: hyponyms, adjectives, prepositions, and relative clauses. We generate sentences up to the depth  $d = 5$ . There are various types of embedding monotonicity, including relative clauses, conditionals, and negated clauses. In this paper, we consider three types of embedded clauses: peripheral-embedding clauses and two kinds of center-embedding clauses, shown in Table 6 in the Appendix.
6109
The number of generated sentences exponentially increases with the depth of embedded clauses. Thus, we limit the number of inference examples to 320,000, split into 300,000 examples for the training set and 20,000 examples for the test set. We guarantee that all combinations of quantifiers are included in the set of inference examples for each depth. Gold labels for generated premise-hypothesis pairs are automatically determined according to the polarity of the argument position (upward/downward) and the type of predicate replacements (with more general/specific phrases). The ratio of each gold label (entailment/non-entailment) in the training and test sets is set to  $1:1$ .