SlowGuess/ACL_2020_OCR huggingface dataset:
To double-check the gold label, we translate each premise-hypothesis pair into a logical formula (see the Appendix for more details). The logical formulas are obtained by combining lambda terms in accordance with meaning composition rules specified in the CFG rules in the standard way (Blackburn and Bos, 2005). We prove the entailment relation using the theorem prover  $\mathsf{V}$  ampire², checking whether a proof is found in time for each entailment pair. For all pairs, the output of the prover matched with the entailment relation automatically determined by monotonicity calculus.
We consider three DNN-based NLI models. The first architecture employs long short-term memory (LSTM) networks (Hochreiter and Schmidhuber, 1997). We set the number of layers to three with no attention. Each premise and hypothesis is processed as a sequence of words using a recurrent neural network with LSTM cells, and the final hidden state of each serves as its representation.
The second architecture employs multiplicative tree-structured LSTM (TreeLSTM) networks (Tran and Cheng, 2018), which are expected to be more sensitive to hierarchical syntactic structures. Each premise and hypothesis is processed as a tree structure by bottom-up combinations of constituent nodes using the same shared compositional function, input word information, and between-word relational information. We parse all premise-hypothesis pairs with the dependency parser using the spaCy li
brary $^3$  and obtain tree structures. For each experimental setting, we randomly sample 100 tree structures and check their correctness. In LSTM and TreeLSTM, the dimension of hidden units is 200, and we initialize the word embeddings with 300-dimensional GloVe vectors (Pennington et al., 2014). Both models are optimized with Adam (Kingma and Ba, 2015), and no dropout is applied.
The third architecture is a Bidirectional Encoder Representations from Transformers (BERT) model (Devlin et al., 2019). We used the base-uncased model pre-trained on Wikipedia and BookCorpus from the pytorch-pretrained-bert library, fine-tuned for the NLI task using our dataset. In fine-tuning BERT, no dropout is applied, and we choose hyperparameters that are commonly used for MultiNLI. We train all models over 25 epochs or until convergence, and select the best-performing model based on its performance on the validation set. We perform five runs per model and report the average and standard deviation of their scores.
I. Systematicity of predicate replacements Figure 2 shows the performance on unseen combinations of quantifiers and predicate replacements. In the minimal training set  $\mathbf{S}_1$ , the accuracy of LSTM and TreeLSTM was almost the same as chance, but that of BERT was around  $75\%$ , suggesting that only BERT generalized to unseen combinations of quantifiers and predicate replacements. When we train BERT with the training set  $\mathbf{S}_2$ , which contains inference examples generated from combinations of one pair of upward/downward quantifiers and all predicate replacements, the accuracy was  $100\%$ . This indicates that by being taught two kinds of quantifiers in the training data, BERT could distinguish between upward and downward for the other quantifiers. The accuracy of LSTM and TreeLSTM increased with increasing the training set size, but did not reach  $100\%$ . This indicates that LSTM and TreeLSTM also generalize to inferences involving similar quantifiers to some extent, but their generalization ability is imperfect.
When testing models with inferences where adverbs or prepositional phrases are added to the be
3https://spacy.io/
<sup>4</sup>https://github.com/huggingface/pytorch-pretrained-bert
$^{2}$ https://github.com/vprover/vampire
6110
(a) Subject nouns
(b) +Adverbs
(c) +Prepositional phrases
Figure 2: Results for systematicity of predicate replacements. Accuracy on test sets where (a) the replacement is performed in the subject position, (b) one adverb is added at the beginning of the sentence, (c) one three-word prepositional phrase is added at the beginning of the sentence, and (d) the replacement is in the object position.  $\mathbf{S}_n$  indicates the experimental setting where the training set  $\mathbf{S}_n$  is used.
Beginning of the sentence, the accuracy of all models significantly decreased. This decrease becomes larger as the syntactic structures of the sentences in the test set become increasingly different from those in the training set. Contrary to our expectations, the models fail to maintain accuracy on test sets whose difference from the training set is the structure with the adverb at the beginning of a sentence. Of course, we could augment datasets involving that structure, but doing so would require feeding all combinations of inference pairs into the models. These results indicate that the models tend to estimate the entailment label from the beginning of a premise-hypothesis sentence pair, and that inferential systematicity to draw inferences involving quantifiers and predicate replacements is not completely generalized at the level of arbitrary constituents.
II. Systematicity of embedding quantifiers Figure 3 shows the performance of all models on unseen combinations of embedding quantifiers. Even when adding the training set of inferences involving one embedded clause and two quantifiers step-by-step, no model showed improved performance. The accuracy of BERT slightly exceeded chance, but the accuracy of LSTM and TreeLSTM was nearly the same as or lower than chance. These results suggest that all the models fail to generalize to unseen combinations of embedding quantifiers even when they involve similar upward/downward quantifiers.
III. Productivity Table 3 shows the performance on unseen depths of embedded clauses. The accuracy on  $\mathbf{D}_1$  and  $\mathbf{D}_2$  was nearly  $100\%$ , indicating that all models almost completely generalize to inferences containing previously seen depths. When
Figure 3: Results for systematicity of embedding quantifiers.  $\mathbf{S}_n$  indicates the experimental setting where the training set  $\mathbf{S}_n$  is used.
$\mathbf{D}_1 + \mathbf{D}_2$  were used as the training set, the accuracy of all models on  $\mathbf{D}_3$  exceeded chance. Similarly, when  $\mathbf{D}_1 + \mathbf{D}_2 + \mathbf{D}_3$  were used as the training set, the accuracy of all models on  $\mathbf{D}_4$  exceeded chance. This indicates that all models partially generalize to inferences containing embedded clauses one level deeper than the training set.
However, standard deviations of BERT and LSTM were around 10, suggesting that these models did not consistently generalize to inferences containing embedded clauses one level deeper than the training set. While the distribution of monotonicity directions (upward/downward) in the training and test sets was uniform, the accuracy of LSTM and BERT tended to be smaller for downward inferences than for upward inferences. This also indicates that these models fail to properly compute monotonicity directions of constituents from syntactic structures. The standard deviation of TreeLSTM was smaller, indicating that TreeLSTM robustly learns inference patterns containing embedded clauses one level deeper than the training set.
6111
Train	Dev/Test	BERT	LSTM	TreeLSTM
D1 + D2	D1	100.0±0.0	100.0±0.0	100.0±0.1
D2	100.0±0.0	99.8±0.2	99.5±0.1
D3	75.2±10.0	75.4±10.8	86.4±4.1
D4	55.0±3.7	57.7±8.7	58.6±7.8
D5	49.9±4.4	45.8±4.0	48.4±3.7
D3 (down)	71.2±4.0	70.4±4.0	86.4±4.1
D3 (up)	80.5±7.5	84.7±4.9	86.4±4.1
D1 + D2 + D3	D1	100.0±0.0	100.0±0.0	100.0±0.0
D2	100.0±0.0	95.1±7.8	99.6±0.0
D3	100.0±0.0	85.2±8.9	97.7±1.1
D4	77.9±10.8	59.7±10.8	68.0±5.6
D5	53.5±19.6	55.1±8.2	49.6±4.3
D4 (down)	85.8±10.5	76.9±6.6	68.0±5.6
D4 (up)	86.8±1.8	81.1±5.6	68.0±5.6
Table 3: Results for productivity.  $\mathbf{D}_d$  indicates the set of inferences where the embedding depth is  $d$ .
Train	Dev/Test	BERT	LSTM	TreeLSTM
D3	D1	49.6±0.5	48.8±13.2	49.8±4.1
D2	49.8±0.6	47.3±12.1	51.8±1.1
D3	100.0±0.0	100.0±0.0	100.0±0.2
D4	D1	50.3±1.0	46.8±6.5	49.0±0.4
D2	49.6±0.8	45.4±1.8	49.7±0.3
D3	50.2±0.7	45.1±0.6	50.5±0.7
D4	100.0±0.0	100.0±0.0	100.0±0.1
D5	D1	49.9±0.7	43.7±4.4	49.1±1.1
D2	49.1±0.3	43.4±3.9	51.4±0.6
D3	50.6±0.2	44.3±2.7	50.5±0.3
D4	50.9±0.8	44.4±3.4	50.3±0.4
D5	100.0±0.0	100.0±0.0	100.0±0.1
However, the performance of all models trained with  $\mathbf{D}_1 + \mathbf{D}_2$  on  $\mathbf{D}_4$  and  $\mathbf{D}_5$  significantly decreased. Also, performance decreased for all models trained with  $\mathbf{D}_1 + \mathbf{D}_2 + \mathbf{D}_3$  on  $\mathbf{D}_5$ . Specifically, there was significantly decreased performance of all models, including TreeLSTM, on inferences containing embedded clauses two or more levels deeper than those in the training set. These results indicate that all models fail to develop productivity on inferences involving embedding monotonicity.
IV. Localism Table 4 shows the performance of all models on localism of embedding monotonicity. When the models were trained with  $\mathbf{D}_3$ ,  $\mathbf{D}_4$  or  $\mathbf{D}_5$ , all performed at around chance on the test set of non-embedding inferences  $\mathbf{D}_1$  and the test set of inferences involving one embedded clause  $\mathbf{D}_2$ . These results indicate that even if models are trained with a set of inferences containing complex syntactic structures, the models fail to locally interpret their constituents.
Performance of data augmentation Prior studies (Yanaka et al., 2019b; Richardson et al., 2020) have shown that given BERT initially trained with
Table 4: Results for localism.
Train	Dev/Test	BERT	LSTM	TreeLSTM
MNLI	D1	46.9±0.4	47.2±1.1	43.4±0.3
D2	46.2±0.6	48.3±1.0	49.5±0.4
D3	46.8±0.8	48.9±0.7	41.0±0.4
D4	48.5±0.8	50.6±0.5	48.5±0.2
D5	48.9±0.6	49.3±0.7	48.8±0.5
MNLI-test	84.6±0.2	64.7±0.3	70.4±0.1
D1 + D2	D1	100.0±0.0	100.0±0.1	100.0±0.1
+MNLI	D2	100.0±0.0	89.3±9.0	99.8±0.1
D3	67.8±12.5	66.7±13.5	76.3±4.1
D4	46.8±3.7	47.1±14.6	50.7±7.8
D5	41.2±4.3	46.7±11.2	47.5±3.7
MNLI-test	84.4±0.2	39.7±0.5	63.0±0.2
D1 + D2 + D3	D1	100.0±0.0	100.0±0.0	100.0±0.0
+MNLI	D2	100.0±0.0	97.1±5.0	99.8±0.0
D3	100.0±0.0	89.2±5.1	98.3±1.1
D4	70.9±7.9	73.4±10.9	76.1±5.6
D5	42.4±4.2	47.8±3.9	57.0±4.3
MNLI-test	84.0±0.1	39.7±0.4	62.8±0.2
Table 5: Results for productivity where models were trained with our synthesized dataset mixed with MultiNLI (MNLI).
MultiNLI, further training with synthesized instances of logical inference improves performance on the same types of logical inference while maintaining the initial performance on MultiNLI. To investigate whether the results of our study are transferable to current work on MultiNLI, we trained models with our synthesized dataset mixed with MultiNLI, and checked (i) whether our synthesized dataset degrades the original performance of models on MultiNLI $^5$  and (ii) whether MultiNLI degrades the ability to generalize to unseen depths of embedded clauses.
Table 5 shows that training BERT on our synthetic data  $\mathbf{D}_1 + \mathbf{D}_2$  and MultiNLI increases the accuracy on our test sets  $\mathbf{D}_1$  (46.9 to 100.0),  $\mathbf{D}_2$  (46.2 to 100.0), and  $\mathbf{D}_3$  (46.8 to 67.8) while preserving accuracy on MultiNLI (84.6 to 84.4). This indicates that training BERT with our synthetic data does not degrade performance on commonly used corpora like MultiNLI while improving the performance on monotonicity, which suggests that our data-synthesis approach can be combined with naturalistic datasets. For TreeLSTM and LSTM, however, adding our synthetic dataset decreases accuracy on MultiNLI. One possible reason for this is that a pre-training based model like BERT can mitigate catastrophic forgetting in various types of datasets.
Regarding the ability to generalize to unseen depths of embedded clauses, the accuracy of all
5Following the previous work (Richardson et al., 2020), we used the MultiNLI mismatched development set for MNLI-test.
6112
models on our synthetic test set containing embedded clauses one level deeper than the training set exceeds chance, but the improvement becomes smaller with the addition of MultiNLI. In particular, with the addition of MultiNLI, the models tend to change wrong predictions in cases where a hypothesis contains a phrase not occurring in a premise but the premise entails the hypothesis. Such inference patterns are contrary to the heuristics in MultiNLI (McCoy et al., 2019). This indicates that there may be some trade-offs in terms of performance between inference patterns in the training set and those in the test set.
The question of whether neural networks are capable of processing compositionality has been widely discussed (Fodor and Pylyshyn, 1988; Marcus, 2003). Recent empirical studies illustrate the importance and difficulty of evaluating the capability of neural models. Generation tasks using artificial datasets have been proposed for testing whether models compositionally interpret training data from the underlying grammar of the data (Lake and Baroni, 2017; Hupkes et al., 2018; Saxton et al., 2019; Loula et al., 2018; Hupkes et al., 2019; Bernardy, 2018). However, these conclusions are controversial, and it remains unclear whether the failure of models on these tasks stems from their inability to deal with compositionality.
Previous studies using logical inference tasks have also reported both positive and negative results. Assessment results on propositional logic (Evans et al., 2018), first-order logic (Mul and Zuidema, 2019), and natural logic (Bowman et al., 2015) show that neural networks can generalize to unseen words and lengths. In contrast, Geiger et al. (2019) obtained negative results by testing models under fair conditions of natural logic. Our study suggests that these conflicting results come from an absence of perspective on combinations of semantic properties.
Regarding assessment of the behavior of modern language models, Linzen et al. (2016), Tran et al. (2018), and Goldberg (2019) investigated their syntactic capabilities by testing such models on subject-verb agreement tasks. Many studies of NLI tasks (Liu et al., 2019; Glockner et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; McCoy et al., 2019; Rozen et al.,
2019; Ross and Pavlick, 2019) have provided evaluation methodologies and found that current NLI models often fail on particular inference types, or that they learn undesired heuristics from the training set. In particular, recent works (Yanaka et al., 2019a,b; Richardson et al., 2020) have evaluated models on monotonicity, but did not focus on the ability to generalize to unseen combinations of patterns. Monotonicity covers various systematic inferential patterns, and thus is an adequate semantic phenomenon for assessing inferential systematicity in natural language. Another benefit of focusing on monotonicity is that it provides hard problem settings against heuristics (McCoy et al., 2019), which fail to perform downward-entailing inferences where the hypothesis is longer than the premise.
We introduced a method for evaluating whether DNN-based models can learn systematicity of monotonicity inference under four aspects. A series of experiments showed that the capability of three models to capture systematicity of predicate replacements was limited to cases where the positions of the constituents were similar between the training and test sets. For embedding monotonicity, no models consistently drew inferences involving embedded clauses whose depths were two levels deeper than those in the training set. This suggests that models fail to capture inferential systematicity of monotonicity and its productivity.
We also found that BERT trained with our synthetic dataset mixed with MultiNLI maintained performance on MultiNLI while improving the performance on monotonicity. This indicates that though current DNN-based models do not systematically interpret monotonicity inference, some models might have sufficient ability to memorize different types of reasoning. We hope that our work will be useful in future research for realizing more advanced models that are capable of appropriately performing arbitrary inferences.
We thank the three anonymous reviewers for their helpful comments and suggestions. We are also grateful to Benjamin Heinzerling and Sosuke Kobayashi for helpful discussions. This work was partially supported by JSPS KAKENHI Grant Numbers JP20K19868 and JP18H03284, Japan.
6113
Lasha Abzianidze. 2015. A tableau prover for natural logic and language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-2015), pages 2492-2502.
Murat Aydede. 1997. Language of thought: The connectionist contribution. *Minds and Machines*, 7(1):57-101.
Johan van Benthem. 1983. Determininers and logic. Linguistics and Philosophy, 6(4):447-478.
Jean-Philippe Bernardy. 2018. Can recurrent neural networks learn nested recursion. Linguistic Issues in Language Technology, 16.
Patrick Blackburn and Johan Bos. 2005. Representation and Inference for Natural Language: A First Course in Computational Semantics. Center for the Study of Language and Information.
Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015. Recursive neural networks can learn logical semantics. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 12-21.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Morgan & Claypool Publishers.
Jacob Devlin, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2019), pages 4171-4186.
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. 2018. Can neural networks understand logical entailment? In International Conference on Learning Representations (ICLR-2018).
Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3-71.
Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2019. Posing fair generalization tasks for natural language inference. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP-2019), pages 4484-4494.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL-2018), pages 650–655.
Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. CoRR, abs/1901.05287.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S. Moss, and Sandra Kübler. 2019. Monalog: a lightweight system for natural language inference based on monotonicity. CoRR, abs/1910.08772.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2019. Compositionality decomposed: how do neural networks generalise? CoRR, abs/1908.08351.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
Thomas Icard and Lawrence Moss. 2014. Recent progress in monotonicity. Linguistic Issues in Language Technology, 9(7):167-194.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR-2015).
Brenden M. Lake and Marco Baroni. 2017. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning (ICML-2017).
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2019), pages 2171-2179.
João Loula, Marco Baroni, and Brenden Lake. 2018. Rearranging the familiar: Testing compositional generalization in recurrent networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 108-114.
Gary Marcus. 2003. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In
6114
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL-2019), pages 3428-3448.
Koji Mineshima, Pascual Martínez-Gómez, Yusuke Miyao, and Daisuke Bekki. 2015. Higher-order logical inference with compositional semantics. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-2015), pages 2055–2061.
Mathijs Mul and Willem Zuidema. 2019. Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization. CoRR, abs/1908.08351.
Peter Pagin and Dag Westerstahl. 2010. Compositionality I: Definitions and variants. Philosophy Compass, 5(3):250-264.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP-2014), pages 1532-1543.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM-2018), pages 180–191.
Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-2020).
Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP-2019), pages 2230-2240.
Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL-2019), pages 196-205.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations (ICLR-2019).
Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP-2018), pages 4731-4736.