SlowGuess/ACL_2020_OCR huggingface dataset:
We test whether the models treat the meaning of closed-class words systematically by perturbing correctly classified jabberwocky sentence pairs with a closed-class word. More precisely, for a pair of closed-class words  $w$  and  $w'$ , we consider test items which can be formed by substitution of  $w$  by  $w'$  in a correctly classified test item. We allow both  $w$  and  $w'$  to be any of the closed-class items, including quantifiers, negation, nominal post-modifiers, or the empty string  $\epsilon$  (thus modeling insertions and deletions of these known, closed-class items). Suppose that Example 1 was correctly classified. Substituting some for all in the premise of yields Example 2, and changes the relation from entailment  $(\square)$  to reverse entailment  $(\square)$ .
(1) All bickets wug.  
All blockets wug.
(2) Some bickets wug.
All blockets wug.
There are two critical features of this probe. First, because we start from a correctly-classified jabberwocky pair, we can conclude that the novel words (e.g., wug and blickets above) were assigned appropriate meanings.
Second, since the perturbation only involves closed-class items which do not vary in meaning and have been highly trained, the perturbation should not affect the models ability to correctly classify the resulting sentence pair. If the model does misclassify the resulting pair, it can only be because a perturbed closed-class word (e.g., some) interacts with the open-class items (e.g., wug), in a way that is different from the pre-perturbation closed-class item (i.e., all). This is non-systematic behavior.
In order to rule out trivially correct behavior where the model simply ignores the perturbation, we consider only perturbations which result in a change of class (e.g.,  $\square \mapsto \square$ ) for the sentence pair. In addition to accuracy on these perturbed items, we also examine the variance of model accuracy on probes across different blocks. If a model's accuracy varies depending only on the novel open-class items in a particular block, this provides further evidence that it does not treat word meaning systematically.
7.2 Identical Open-class Words Probe
Some sentence pairs are classifiable without any knowledge of the novel words' meaning; for example, pairs where premise and hypothesis have identical open-class words. An instance is shown in Example 3: the two sentences must stand in contradiction, regardless of the meaning of blicket or wug.
(3) All bickets wug.
Some bickets don't wug.
The closed-class items and compositional structure of the language is sufficient for a learner to deduce the relationships between such sentences, even with unfamiliar nouns and verbs. Our second probe, the identical open-class words probe, tests the models' ability to correctly classify such pairs.
7.3 Consistency Probe
Consider Examples 4 and 5, which present the same two sentences in opposite orders.
(4) All bickets wug.  
All red bickets wug.
(5) All red bickets wug.  
All bickets wug.
In Example 4, the two sentences stand in an entailment  $(\square)$  relation. In Example 5, by contrast, the two sentences stand in a reverse entailment  $(\square)$  relation. This is a logically necessary consequence of the way the relations are defined. Reversing the order of sentences has predictable effects for all seven natural logic relations: in particular, such reversals map  $\square \mapsto \square$  and  $\square \mapsto \square$ , leaving all other relations intact. Based on this observation, we develop a consistency probe of systematicity. We ask for each correctly classified jabberwocky block test item, whether the corresponding reversed item is also correctly classified. The intuition behind this probe is that whatever meaning a model assumes for the novel open-class words, it should assume the same meaning when the sentence order is reversed. If the reverse is not correctly classified, then this is strong evidence of contextual dependence in meaning.
8 Results
In this section, we report the results of two control analyses, and that of our three systematicity probes described above.
8.1 Analysis I: Holdout Evaluations
We first establish that the models perform well on novel configurations of known words. Table 3 reports accuracy on heldout sentence pairs, described in §5.5. The table reports average accuracies across training blocks together with the standard deviations of these statistics. As can be seen in the table, all models perform quite well on holdout forms across training blocks, with very little variance. Because these items use the same sampling scheme and vocabulary as the trained blocks, these simulations serve as a kind of upper bound on the performance and a lower bound on the variance that we can expect from the more challenging jabberwocky-block-based evaluations below.
8.2 Analysis II: Distribution of Novel Words
Our three systematicity probes employ jabberwocky-type sentences—novel open-class words in sentential frames built from known closed-class words. Since models are not
Condition	BGRU mean (sd)	CONV mean (sd)	SATT mean (sd)	INFS mean (sd)
small	95.1 ±0.21	95.43 ±0.12	93.14 ±0.94	96.02 ±0.51
large	95.09 ±1.03	95.22 ±0.55	94.89 ±1.09	96.17 ±0.74
Table 3: Accuracy on holdout evaluations (training conditions and holdout evaluation are explained in §5.5)
Figure 1: Visualization of trained and novel open-class word embeddings.
trained on these novel words, it is important to establish that they are from the same distribution as the trained words and, thus, that the models' performance is not driven by some pathological feature of the novel word embeddings.
Trained word embeddings were initialized randomly from  $\mathcal{N}(0,1)$  and then updated during training. Novel word embeddings were simply drawn from  $\mathcal{N}(0,1)$  and never updated. Figure 1 plots visualizations of the trained and novel open-class word embeddings in two dimensions, using t-SNE parameters computed over all open-class words (Maaten and Hinton, 2008). Trained words are plotted as  $+$ , novel words as  $\bullet$ . Color indicates the proportion of test items containing that word that were classified correctly. As the plot shows, the two sets of embeddings overlap considerably. Moreover, there does not appear to be a systematic relationship between rates of correct classification for items containing novel words and their proximity to trained words. We also performed a resampling analysis, determining that novel vectors did not differ significantly in length from trained vectors ( $p = 0.85$ ). Finally, we observed mean and standard deviation of the pairwise cosine similarity between trained and novel words to be 0.999 and 0.058 respectively, confirming that there is little evidence the distributions are different.
8.3 Analysis III: Known Word Perturbation Probe
Recall from §7.1 that the known word perturbation probe involves insertion, deletion, or substitution
Figure 2: Performance on the known word perturbation probe, small and large training conditions (see §5.5).
of a trained closed-class word in a correctly classified jabberwocky-type sentence pair. Figure 2 plots the results of this probe. Each point represents a perturbation type—a group of perturbed test items that share their before/after target perturbed closed-class words and before/after relation pairs. The upper plot displays the mean accuracy of all perturbations, averaged across blocks, and the lower plot displays the standard deviations across blocks.
All models perform substantially worse than the holdout-evaluation on at least some of the perturbations. In addition, the standard deviation of accuracy between blocks is higher than the holdout tests. As discussed in §7.1, low accuracy on this probe indicates that closed-class words do not maintain a consistent interpretation when paired with different open-class words. Variance across blocks shows that under all models the behavior of closed-class words is highly sensitive to the novel words they appear with.
Performance is also susceptible to interference from sentence-level features. For example, consider the perturbation which deletes a post-modifier from a sentence pair in negation, yielding a pair in cover relation. The self-attentive encoder performs perfectly when this perturbation is applied to a premise  $(100\% \pm 0.00\%)$ , but not when applied to a hypothesis  $(86.60\% \pm 18.08\%)$ . Similarly, deleting the adjective red from the hypothesis of a forward-entailing pair results in an unrelated sen
tence pair  $(84.79\% \pm 7.50\%)$  or another forward- entailing pair  $(92.32\%, \pm 3.60\%)$  or an equality pair  $(100\% \pm 0.00\%)$ . All the possible perturbations we studied exhibit similarly inconsistent performance.
8.4 Analysis IV: Identical Open-Class Words Probe
Recall that the identical open-class words probe consist of sentence pairs where all open-class lexical items were identical. Table 4 shows the accuracies for these probes, trained on the small language. Average accuracies across jabberwocky blocks are reported together with standard deviations.
Relation	BGRU mean (sd)	CONV mean (sd)	SATT mean (sd)	INFS mean (sd)
#	100 ± 0	100 ± 0	99.94 ± 0.26	99.67 ± 0.98
^	55.68 ± 20.29	73.29 ± 10.8	23.71 ± 11.45	90.67 ± 10.98
□	90.78 ± 4.99	82.84 ± 6.51	75.22 ± 5.98	95.53 ± 2.64
≡	90.43 ± 17.1	38.12 ± 15.56	71.94 ± 24.1	95.93 ± 6.5
□	90.34 ± 4.18	77.11 ± 5.9	81.4 ± 6.67	93.81 ± 2.96
|	93.08 ± 3.58	85.34 ± 5.47	74.05 ± 8.03	92.23 ± 4.6
\(\smile\)	88.01 ± 3.55	71.5 ± 7.32	78.4 ± 7.91	95.22 ± 3.58
Table 4: Identical open-class words probe performance, trained on the small language condition (trained on 51,743 sentence pairs, see §5.5)
Accuracy on the probe pairs fails to reach the holdout test levels for most models and most relations besides  $\#$ , and variance between blocks is much higher than in the holdout evaluation. Of special interest is negation ( $\wedge$ ), for which accuracy is dramatically lower and variance dramatically higher than the holdout evaluation.
The results are similar for the large language condition, shown in Table 5. Although model accuracies improve somewhat, variance remains higher than the heldout level and accuracy lower. Recall that these probe-items can be classified while ignoring the specific identity of their open-class words. Thus, the models inability to leverage this fact, and high variance across different sets novel open-class words, illustrates their sensitivity to context.
8.5 Analysis V: Consistency Probe
The consistency probe tests abstract knowledge of relationships between logical relations, such as the fact that two sentences that stand in a contradiction still stand in a contradiction after reversing their order. Results of this probe in the small-language condition are in Table 6: For each type of relation, we show the average percentage of correctly-labeled
Relation	BGRU mean (sd)	CONV mean (sd)	SATT mean (sd)	INFS mean (sd)
#	99.82 ±0.45	99.57 ±0.73	98.67 ±1.81	100 ±0
^	84.18 ±12.29	73.73 ±18.31	79.97 ±16.58	85.54 ±14.11
□	96.13 ±2.59	93.88 ±2.67	97.3 ±2.36	97.02 ±2.39
≡	89.33 ±12.5	77.84 ±12.08	94.44 ±11.23	94.59 ±7.02
⊃	95.4 ±2.48	94.55 ±2.04	98.05 ±1.51	97.6 ±2.08
|	89.97 ±6.73	92.36 ±6.29	84.52 ±7.07	98.72 ±2.08
←	90.78 ±6.33	93.18 ±2.95	87.85 ±6.46	97.48 ±2.56
sentence pairs that, when presented in reverse order, were also correctly labeled.
The best-performing model on negation reversal is SATT, which correctly labeled reversed items  $66.92\%$  of the time. Although performance on negation is notably more difficult than the other relations, every model, on every relation, exhibited inter-block variance higher than that of the hold-out evaluations.
Table 5: Identical open-class words probe performance when trained on the large language training condition (trained on 478,649 sentence pairs, see §5.5)
Relation	BGRU mean (sd)	CONV mean (sd)	SATT mean (sd)	INFS mean (sd)
#	97.4 ±0.86	97.8 ±0.93	98.58 ±0.74	97.03 ±0.87
^	63.03 ±36.19	63.42 ±35.91	66.92 ±31.45	57.16 ±38.24
□	92.45 ±6.26	88.1 ±8.16	93.16 ±5.42	90.64 ±6.76
≡	100 ±0	100 ±0	100 ±0	100 ±0
□	91.37 ±6.23	94.73 ±6.51	96.42 ±3.22	87.02 ±9.61
|	96.02 ±2.6	96.29 ±2.51	96.95 ±2.14	94.2 ±3.48
\(\smile\)	93.57 ±3.56	95 ±2.97	96.4 ±2.83	93.1 ±3.77
Furthermore, as can be seen in Table 7, the large language condition yields little improvement. Negation pairs are still well below the hold-out test threshold, still with a high degree of variation. Variation remains high for many relations, which is surprising because the means report accuracy on test items that were chosen specifically because the same item, in a reverse order, was already correctly labeled. Reversing the order of sentences causes the model to misclassify the resulting pair, more often for some blocks than others.
9 Discussion and Conclusion
Systematicity refers to the property of natural language representations whereby words (and other units or grammatical operations) have consistent meanings across different contexts. Our probes test whether deep learning systems learn to represent linguistic units systematically in the natural lan
Table 6: Consistency probe performance, trained on the small language condition (51,743 sentence pairs, see §5.5).
Relation	BGRU mean (sd)	CONV mean (sd)	SATT mean (sd)	INFS mean (sd)
#	98.45 ±0.65	98.69 ±0.54	98.83 ±0.6	98.38 ±0.74
^	70.46 ±33.72	77.82 ±26	84.27 ±23.89	65.64 ±35.13
□	96.02 ±2.96	96.6 ±3.26	96.78 ±4.23	95.01 ±5.38
=	100 ±0	100 ±0	100 ±0	100 ±0
□	93.5 ±4.51	95.76 ±4.23	94.23 ±5.86	90.11 ±8.5
|	96.31 ±2.73	97.25 ±2.05	97.17 ±2.23	94.46 ±4.24
\(\smile\)	96.25 ±2.49	96.98 ±2.66	97.18 ±2.17	93.88 ±4.78
Table 7: Consistency probe performance, trained on the large language condition (478, 649 sentence pairs).
guage inference task. Our results indicate that despite their high overall performance, these models tend to generalize in ways that allow the meanings of individual words to vary in different contexts, even in an artificial language where a totally systematic solution is available. This suggests the networks lack a sufficient inductive bias to learn systematic representations of words like quantifiers, which even in natural language exhibit very little meaning variation.
Our analyses contain two ideas that may be useful for future studies of systematicity. First, two of our probes (known word perturbation and consistency) are based on the idea of starting from a test item that is classified correctly, and applying a transformation that should result in a classifiable item (for a model that represents word meaning systematically). Second, our analyses made critical use of differential sensitivity (i.e., variance) of the models across test blocks with different novel words but otherwise identical information content. We believe these are a novel ideas that can be employed in future studies.
We thank Brendan Lake, Marco Baroni, Adina Williams, Dima Bahdanau, Sam Gershman, Ishita Dasgupta, Alessandro Sordoni, Will Hamilton, Leon Bergen, the Montreal Computational and Quantitative Linguistics, and Reasoning and Learning Labs at McGill University for feedback on the manuscript. We are grateful to Facebook AI Research for providing extensive compute and other support. We also gratefully acknowledge the support of the Natural Sciences and Engineering Research Council of Canada, the Fonds de Recherche du Québec, Société et Culture, and the Canada CI-FAR AI Chairs Program.
Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in English past tenses: A computational/experimental study. Cognition, 90(2):119-161.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2019. Systematic generalization: What is required and can it be learned? arXiv, abs/1811.12889(arXiv:1811.12889v2 [cs.CL]).
Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, and Roberto Zamparelli. 2016. Sick through the semeval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. Language Resources and Evaluation, 50(1):95-124.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
Samuel R Bowman, Christopher Potts, and Christopher D Manning. 2014. Recursive neural networks can learn logical semantics. arXiv preprint arXiv:1406.1827.
Rudolf Camap. 1947. Meaning and necessity: A study in semantics and modal logic.
Brian Cantwell Smith. 1996. On the Origins of Objects. MIT Press, Cambridge, Massachusetts.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680.
Ishita Dasgupta, Demi Guo, Samuel J. Gershman, and Noah D. Goodman. 2019. Analyzing machine-learned representations: A natural language case study.
Ishita Dasgupta, Demi Guo, Andreas Stuhlmuller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.
Catarina Dutilh Novaes. 2012. Formal Languages in Logic: A Philosophical and Cognitive Analysis. Cambridge University Press, Cambridge, England.
Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790-1801, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. 2018. Can neural networks understand logical entailment? arXiv preprint arXiv:1802.08535.
Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3-71.
Samuel Gershman and Joshua B Tenenbaum. 2015. Phrase similarity in humans and machines. In CogSci.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. arXiv preprint arXiv:1808.08079.
Irene Heim and Angelika Kratzer. 2000. Semantics in Generative Grammar. Blackwell Publishing, Malden, MA.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Wilfrid Hodges. 2012. Formalizing the relationship between meaning and syntax. In *The Oxford handbook of compositionality*. Oxford Handbooks in Linguistic.
Drew A. Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning.
Theo MV Janssen et al. 2012. Compositionality: its historic context. The Oxford handbook of compositionality, pages 19-46.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2019. Measuring compositional generalization: A comprehensive method on realistic data.
Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018. Dynamic meta-embeddings for improved sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1466-1477.
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2879-2888.
Brenden Lake, Tal Linzen, and Marco Baroni. 2017a. Human few-shot learning of compositional instructions. In Ashok Goel, Colleen Seifert, and Christian Freksa, editors, Proceedings of the 41st Annual Conference of the Cognitive Science Society, pages 611-616. Cognitive Science Society, Montreal, Canada.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017b. Building machines that learn and think like people. *Behavioral and Brain Sciences*, 40.
George Lakoff. 1970. Linguistics and natural logic. Synthese, 22(1-2):151-271.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eighth International Conference on Computational Semantics, IWCS-8 '09, pages 140-156, Stroudsburg, PA, USA. Association for Computational Linguistics.
Bill MacCartney and Christopher D. Manning. 2014. Natural logic and natural language inference. In Computing Meaning: Volume 4, pages 129-147, Dordrecht. Springer Netherlands.