Venue: ACL 2020

# Probing Linguistic Systematicity

Emily Goodwin,\({}^{4,5}\) Koustuv Sinha,\({}^{2,3,5}\) Timothy J. O'Donnell\({}^{1,4,5}\)

\({}^{1}\)Department of Linguistics, \({}^{2}\)School of Computer Science, McGill University, Canada

\({}^{3}\)Facebook AI Research (FAIR), Montreal \({}^{4}\)Canada CIFAR AI Chair, Mila

\({}^{5}\)Quebec Artificial Intelligence Institute (Mila)

{emily.goodwin, koustuv.sinha}@mail.mcgill.ca timothy.odonnell@mcgill.ca

Corresponding author

###### Abstract

Recently, there has been much interest in the question of whether deep natural language understanding models exhibit _systematicity_--generalizing such that units like words make consistent contributions to the meaning of the sentences in which they appear. There is accumulating evidence that neural models often generalize non-systematically. We examined the notion of systematicity from a linguistic perspective, defining a set of probes and a set of metrics to measure systematic behaviour. We also identified ways in which network architectures can generalize non-systematically, and discuss why such forms of generalization may be unsatisfying. As a case study, we performed a series of experiments in the setting of natural language inference (NLI), demonstrating that some NLU systems achieve high overall performance despite being non-systematic.

## 1 Introduction

Language allows us to express and comprehend a vast variety of novel thoughts and ideas. This creativity is made possible by compositionality--the linguistic system builds utterances by combining an inventory of primitive units such as morphemes, words, or idioms (the lexicon), using a small set of structure-building operations (the grammar; Camap, 1947; Fodor and Pylyshyn, 1988; Hodges, 2012; Janssen et al., 2012; Lake et al., 2017b; Szabo, 2012; Zadrozny, 1994; Lake et al., 2017a).

One property of compositional systems, widely studied in the cognitive sciences, is the phenomenon of systematicity. Systematicity refers to the fact that lexical units such as words make consistent contributions to the meaning of the sentences in which they appear. Fodor and Pylyshyn (1988) provided a famous example: If a competent speaker of English knows the meaning of the sentence _John loves the girl_, they also know the meaning of _The girl loves John_. This is because for speakers of English knowing the meaning of the first sentence implies knowing the meaning of the individual words _the_, _loves_, _girl_, and _John_ as well as grammatical principles such as how transitive verbs take their arguments. But knowing these words and principles of grammar implies knowing how to compose the meaning of the second sentence.

Deep learning systems now regularly exhibit very high performance on a large variety of natural language tasks, including machine translation Wu et al. (2016); Vaswani et al. (2017), question answering Wang et al. (2018); Henaff et al. (2016), visual question answering Hudson and Manning (2018), and natural language inference Devlin et al. (2018); Storks et al. (2019). Recently, however, researchers have asked whether such systems generalize systematically (see SS4).

Systematicity is the property whereby words have consistent contributions to composed meaning; the alternative is the situation where words have a high degree of contextually conditioned meaning variation. In such cases, generalization may be based on local heuristics McCoy et al. (2019); Niven and Kao (2019), variegated similarity Albright and Hayes (2003), or local approximations Veldhoen and Zuidema (2017), where the contribution of individual units to the meaning of the sentence can vary greatly across sentences, interacting with other units in highly inconsistent and complex ways.

This paper introduces several novel probes for testing systematic generalization. We employ an artificial language to have control over systematicy and contextual meaning variation. Applying our probes to this language in an NLI setting reveals

that some deep learning systems which achieve very high accuracy on standard holdout evaluations do so in ways which are non-systematic: the networks do not consistently capture the basic notion that certain classes of words have meanings which are consistent across the contexts in which they appear.

The rest of the paper is organized as follows. SS2 discusses degrees of systematicity and contextually conditioned variation; SS3 introduces the distinction between open- and closed-class words, which we use in our probes. SS5 introduces the NLI task and describes the artificial language we use; SS6 discusses the models that we tested and the details of our training setup; SS7 introduces our probes of systematicity and results are presented in SSS8.1

Footnote 1: Code for datasets and models can be found here: [https://github.com/emilygoodwin/systematicity](https://github.com/emilygoodwin/systematicity)

## 2 Systematicity and Contextual Conditioning

Compositionality is often stated as the principle that the meaning of an utterance is determined by the meanings of its parts and the way those parts are combined (see, e.g., Heim and Kratzer, 2000).

Systematicity, the property that words mean the same thing in different contexts, is closely related to compositionality; nevertheless, compositional systems can vary in their degree of systematicity. At one end of the spectrum are systems in which primitive units contribute exactly one identical meaning across all contexts. This high degree of systematicity is approached by artificial formal systems including programming languages and logics, though even these systems don't fully achieve this ideal Cantwell Smith (1996); Dutilh Novaes (2012).

The opposite of systematicity is the phenomenon of contextually conditioned variation in meaning where the contribution of individual words varies according to the sentential contexts in which they appear. Natural languages exhibit such context dependence in phenomena like homophony, polysemy, multi-word idioms, and co-compositionality. Nevertheless, there are many words in natural language--especially closed-class words like quantifiers (see below)--which exhibit very little variability in meaning across sentences.

At the other end of the spectrum from programming languages and logics are systems where many or most meanings are highly context dependent. The logical extreme--a system where each word has a different and unrelated meaning every time it occurs--is clearly of limited usefulness since it would make generalization impossible. Nevertheless, learners with sufficient memory capacity and flexibility of representation, such as deep learning models, can learn systems with very high degrees of contextual conditioning--in particular, higher than human language learners. An important goal for building systems that learn and generalize like people is to engineer systems with inductive biases for the right degree of systematicity. In SS8, we give evidence that some neural systems are likely too biased toward allowing contextually conditioned meaning variability for words, such as quantifiers, which do not vary greatly in natural language.

## 3 Compositional Structure in Natural Language

Natural language distinguishes between content or open-class lexical units and function or closed-class lexical units. The former refers to categories, such a nouns and verbs, which carry the majority of contentiful meaning in a sentence and which permit new coinages. Closed-class units, by contrast, carry most of the grammatical structure of the sentence and consist of things like inflectional morphemes (like pluralizing _-s_ in English) and words like determiners, quantifiers, and negation (e.g., _all, some, the_ in English). These are mostly fixed; adult speakers do not coin new quantifiers, for example, the way that they coin new nouns.

Leveraging this distinction gives rise to the possibility of constructing probes based on jabberwocky-type sentences. This term references the poem _Jabberwocky_ by Lewis Carroll, which combines nonsense open-class words with familiar closed-class words in a way that allows speakers to recognize the expression as well formed. For example, English speakers identify a contradiction in the sentence _All Jabberwocks flug, but some Jabberwocks don't flug_, without a meaning for _jabberwock_ and _flug_. This is possible because we expect the words _all, some_, _but_, and _don't_ to contribute the same meaning as they do when combined with familiar words, like _All pigs sleep, but some pigs don't sleep_.

Using jabberwocky-type sentences, we tested the generalizability of certain closed-class word representations learned by neural networks. Giving the networks many examples of each construction 

with a large variety of different content words--that is, large amounts of highly varied evidence about the meaning of the closed-class words--we asked during the test phase how fragile this knowledge is when transferred to new open-class words. That is, our probes combine novel open-class words with familiar closed-class words, to test whether the closed-class words are treated systematically by the network. For example, we might train the networks to identify contradictions in pairs like _All pigs sleep; some pigs don't sleep_, and test whether the network can identify the contradiction in a pair like _All Jabberwocks flug; some Jabberwocks don't flug_. A systematic learner would reliably identify the contradiction, whereas a non-systematic learner may allow the closed-class words (_all, some, don't_) to take on contextually conditioned meanings that depend on the novel context words.

## 4 Related Work

There has been much interest in the problem of systematic generalization in recent years Bahdanau et al. (2019); Bentivogli et al. (2016); Lake et al. (2017a, b); Gershman and Tenenbaum (2015); McCoy et al. (2019); Veldhoen and Zuidema (2017); Soulos et al. (2019); Prasad et al. (2019); Richardson et al. (2019); Johnson et al. (2017), inter alia).

In contrast to our approach (testing novel words in familiar combinations), many of these studies probe systematicity by testing familiar words in novel combinations. Lake and Baroni Lake and Baroni (2018) adopt this approach in semantic parsing with an artificial language known as SCAN. Dasgupta et al. Dasgupta et al. (2018, 2019) introduce a naturalistic NLI dataset, with test items that shuffle the argument structure of natural language utterances. In the in the inductive logic programming domain, Sinha et al. Sinha et al. (2019) introduced the CLUTTR relational-reasoning benchmark. The novel-combinations-of-familiar-words approach was formalized in the CFQ dataset and associated distribution metric of Keysers et al. Kayser et al. (2019). Ettinger et al. Ettinger et al. (2018) introduced a semantic-role-labeling and negation-scope labeling dataset, which tests compositional generalization with novel combinations of familiar words and makes use of syntactic constructions like relative clauses. Finally, Kim et al. Kim et al. (2019) explore pre-training schemes' abilities to learn prepositions and wh-words with syntactic transformations (two kinds of closed-class words which our work does not address).

A different type of systematicity analysis directly investigates learned representations, rather than developing probes of model behavior. This is done either through visualization Veldhoen and Zuidema (2017), training a second network to approximate learned representations using a symbolic structure Soulos et al. (2019) or as a diagnostic classifier Giulianelli et al. (2018), or reconstructing the semantic space through similarity measurements over representations Prasad et al. (2019).

## 5 Study Setup

### Natural Language Inference

We make use of the Natural language inference (NLI) task to study the question of systematicity. The NLI task is to infer the relation between two sentences (the premise and the hypothesis). Sentence pairs must be classified into one of a set of predefined logical relations such as entailment or contradiction. For example, the sentence _All mammals growl_ entails the sentence _All pigs growl_. A rapidly growing number of studies have shown that deep learning models can achieve very high performance in this setting Evans et al. (2018); Conneau et al. (2017); Bowman et al. (2014); Yoon et al. (2018); Kiela et al. (2018); Munkhdalai and Yu (2017); Rocktaschel et al. (2015); Peters et al. (2018); Parikh et al. (2016); Zhang et al. (2018); Radford et al. (2018); Devlin et al. (2018); Storks et al. (2019).

### Natural Logic

We adopt the formulation of NLI known as natural logic MacCartney and Manning (2014, 2009); Lakoff (1970). Natural logic makes use of seven logical relations between pairs of sentences. These are shown in Table 1. These relations can be interpreted as the set theoretic relationship between the extensions of the two expressions. For instance, if the expressions are the simple nouns _warthlog_ and _pig_, then the entailment relation (\(\sqsubset\)) holds between these extensions (_warthog_\(\sqsubset\mathit{pig}\)) since every warthog is a kind of pig.

For higher-order operators such as quantifiers, relations can be defined between sets of possible worlds. For instance, the set of possible worlds consistent with the expression _All blickets wug_ is a subset of the set of possible worlds consistent with the logically weaker expression _All red blocks vug_. Critically, the relationship between composed expressions such as _All X Y_ and _All P Q_ is determined entirely by the relations between X/Y and P/Q, respectively. Thus, natural logic allows 

us to compute the relation between the whole expressions using the relations between parts. We define an artificial language in which such alignments are easy to compute, and use this language to probe deep learning systems' ability to generalize systematically.

### The Artificial Language

In our artificial language, sentences are generated according to the six-position template shown in Table 2, and include a quantifier (position 1), noun (position 3), and verb (position 6), with optional pre- and post-modifiers (position 2 and 4) and optional negation (position 5). For readability, all examples in this paper use real English words; however, simulations can use uniquely identified abstract symbols (i.e., generated by gensym).

We compute the relation between position-aligned pairs of sentences in our language using the natural logic system (described in SS5.2). Quantifiers and negation have their usual natural-language semantics in our artificial language; pre- and post-modifiers are treated intersectively. Open-class items (nouns and verbs) are organized into linear hierarchical taxonomies, where each open-class word is the sub- or super-set of exactly one other open-class item in the same taxonomy. For example, since _dogs_ are all _mammals_, and all mammals _animals_, they form the entailment hierarchy _dogs_\(\sqsubset\)_mammals_\(\sqsubset\)_animals_. We vary the number of distinct noun and verb taxonomies according to an approach we refer to as block structure, described in the next section.

### Block Structure

In natural language, most open-class words do not appear with equal probability with every other word. Instead, their distribution is biased and clumpy, with words in similar topics occurring together. To mimic such topic structure, we group nouns and verbs into blocks. Each block consists of six nouns and six verbs, which form taxonomic hierarchies (e.g., _lizards/animals_, _run/move_). Nouns and verbs from different blocks have no taxonomic relationship (e.g., _lizards_ and _screwdrivers_ or _run_ and _read_) and do not co-occur in the same sentence pair. Because each block includes a six verbs and six nouns in a linear taxonomic hierarchy, no single block is intrinsically harder to learn than any other block.

The same set of closed-class words appear with all blocks of open-class words, and their meanings are systematic regardless of the open-class words (nouns and verbs) they are combined with. For example, the quantifier _some_ has a consistent meaning when it is applied to _some screwdrivers_ or _som animals_. Because closed-class words are shared across blocks, models are trained on extensive and varied evidence of their behaviour. We present closed-class words in a wide variety of sentential contexts, with a wide variety of different open-class words, to provide maximal pressure against overfitting and maximal evidence of their consistent meaning.

### Test and Train Structure

We now describe the structure of our training blocks, holdout test set, and jabberwocky blocks. We also discuss our two test conditions, and several other issues that arise in the construction of our dataset.

Training set:For each training block, we sampled (without replacement) one sentence pair for every possible combination of open-class words, that is, every combination of nouns and verbs \(\langle{\tt noun}_{1},{\tt noun}_{2},{\tt verb}_{1},{\tt verb}_{2}\rangle\). Closed-class words were sampled uniformly to fill each remaining positions in the sentence (see Table 2). A random subset of 20% of training items were reserved for validation (early stopping) and not used during training.

Holdout test set:For each training block, we sampled a holdout set of forms using the same nouns and verbs, but disjoint from the training set just described. The sampling procedure was identical to that for the training blocks. These holdout items allow us to test the generalization of the models with known words in novel configurations (see SS8.1).

Jabberwocky test set:Each jabberwocky block consisted of novel open-class items (i.e., nouns and verbs) that did not appear in training blocks. For each jabberwocky block, we began by following a

\begin{table}
\begin{tabular}{c l l l} \hline \hline Symbol & Name & Example & Set-theoretic definition \\ \hline \(x=y\) & equivalence & _pig_\(\equiv pig\) & \(z=y\) \\ \(x\preceq y\) & forward entailment & _pg_\(\subseteq mammal\) & \(x\leq y\) \\ \(x^{\prime}\succcuit\) & reverse entailment & _animal_\(?\Rightarrow\) & \(x\geq y\) \\ \(x^{\prime}\succcuit\) & negation & _pq_\(\mathrel{\wedge}not~{}orig\) & \(z\sqcap\bar{y}=h\wedge z\cup y=U\) \\ \(x\succcuit\) & alternation & _kg_\(|cat|\) & \(x\sqcap\bar{y}=b\wedge x\sqcup v\not\in U\) \\ \(x\succ y\) & cover & _animal_\(\to~{}neq~{}rig\) & \(xz\sqcap\bar{y}+b\wedge x\sqcup v=U\) \\ \(x\not y\) & independence & _humpy_\(\not\) & _working_ & (all other cases) \\ \hline \hline \end{tabular}
\end{table}
Table 1: MacCartney and Manning [2009]’s implementation of natural logic relations 

sampling procedure identical to that for the training/holdout sets with these new words. Several of our systematicy probes are based on the behavior of neighboring pairs of test sentences (see SS7). To ensure that all such necessary pairs were in the jabberwocky test set, we extended the initial sample with any missing test items.

Training conditions:Since a single set of closed-class words is used across all blocks, adding more blocks increases evidence of the meaning of these words without encouraging overfitting. To study the effect of increasing evidence in this manner, we use two training conditions: small with 20 training blocks and large with 185 training blocks. Both conditions contained 20 jobberwocky blocks. The small condition consisted of \(51,743\) training, \(10,399\) validation, and \(3,694,005\) test (holdout and jobberwocky) pairs. The large condition consisted of \(478,649\) training, \(96,005\) validation, and \(3,694,455\) test items.

Balancing:One consequence of the sampling method is that logical relations will not be equally represented in training. In fact, it is impossible to simultaneously balance the distributions of syntactic constructions, logical relations, and instances of words. In this trade-off, we chose to balance the distribution of open-class words in the vocabulary, as we are focused primarily on the ability of neural networks to generalize closed-class word meaning. Balancing instances of open-class words provided the greatest variety of learning contexts for the meanings of the closed-class items.

## 6 Simulations

### Models

We analyze performance on four simple baseline models known to perform well on standard NLI tasks, such as the Stanford Natural Language Inference datasets, Bowman et al. (2015). Following Conneau et al. (2017), the hypothesis \(u\) and premise \(v\) are individually encoded by neural sequence encoders such as a long short-term memory (LSTM; Hochreiter and Schmidhuber (1997) or gated recurrent unit GRU; Cho et al. (2014). These vectors, together with their element-wise product \(u*v\) and element-wise difference \(u-v\) are fed into a fully connected multilayer perceptron layer to predict the relation. The encodings \(u\) and \(v\) are produced from an input sentence of \(M\) words, \(w_{1},\dots,w_{M}\), using a recurrent neural network, which produces a set of a set of \(M\) hidden representations \(h_{1},\dots,h_{t}\), where \(h_{t}=f(w_{1},\dots,w_{M})\). The sequence encoding is represented by its last hidden vector \(h_{T}\).

The simplest of four models sets \(f\) to be a bidirectional gated recurrent unit (BGRU). This model concatenates the last hidden state of a GRU run forwards over the sequence and the last hidden state of GRU run backwards over the sequence, for example, \(u=[\overleftarrow{h_{M}},\overrightarrow{h_{M}^{\prime}}]\).

Our second embedding system is the Infersent model reported by Conneau et al. (2017), a bidirectional LSTM with max pooling (INFS). This is a model where \(f\) is an LSTM. Each word is represented by the concatenation of a forward and backward representation: \(h_{t}=[\overleftarrow{h_{t}},\overrightarrow{h_{t}}]\). We constructed a fixed vector representation of the sequence \(h_{t}\) by selecting the maximum value over each dimension of the hidden units of the words in the sentence.

Our third model is a self-attentive sentence encoder (SATT) which uses an attention mechanism over the hidden states of a BiLSTM to generate the sentence representation Lin et al. (2017). This attention mechanism is a weighted linear combination of the word representations, denoted by \(u=\sum_{M}\alpha_{i}h_{i}\), where the weights are calculated as follows:

\[\begin{split}&\bar{h_{i}}=\text{tanh}(Wh_{i}+b_{w})\\ &\alpha_{i}=\frac{e^{\bar{h_{i}}\top u_{w}}}{\sum_{i}e^{\bar{h_{ i}}\top u_{w}}}\end{split}\]

\begin{table}
\begin{tabular}{|l||c c c l l l|} \hline
**Position** & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**Category** & quantifier & nominal premodifier & noun & nominal postmodifier & negation & verb \\
**Status** & Obligatory & Optional & Obligatory & Optional & Optional & Obligatory \\
**Class** & Closed & Closed & Open & Closed & Closed & Open \\
**Example** & _All_ & _brown_ & _dogs_ & _that bark_ & _don’t_ & _run_ \\ \hline \end{tabular}
\end{table}
Table 2: A template for sentences in the artificial language. Each sentence fills the obligatory positions 1, 3, and 6 with a word: a quantifier, noun, and verb. Optional positions (2, 4 and 5) are filled by either a word (adjective, postmodifier or negation) or by the empty string. Closed-class categories (Quantifiers, adjectives, post modifiers and negation) do not include novel words, while open-class categories (nouns and verbs) includes novel words that are only exposed in the test set.

 

where, \(u_{w}\) is a learned context query vector and \((W,b_{w})\) are the weights of an affine transformation. This self-attentive network also has multiple views of the sentence, so the model can attend to multiple parts of the given sentence at the same time.

Finally, we test the Hierarchical Convolution-alNetwork (CONV) architecture from Conneau et al. (2017) which is itself inspired from the model _AdaSent_Zhao et al. (2015). This model has four convolution layers; at each layer the intermediate representation \(u_{i}\) is computed by a max-pooling operation over feature maps. The final representation is a concatenation \(u=[u_{1},...,u_{l}]\) where \(l\) is the number of layers.

## 7 Probing Systematicity

In this section, we study the systematicity of the models described in SS6.1. Recall that systematicity refers to the degree to which words have consistent meaning across different contexts, and is contrasted with contextually conditioned variation in meaning. We describe three novel probes of systematicity which we call the known word perturbation probe, the identical open-class words probe, and the consistency probe.

All probes take advantage of the distinction between closed-class and open-class words reflected in the design of our artificial language, and are performed on sentence pairs with novel open-class words (jabberwocky-type sentences; see SS5.5 ). We now describe the logic of each probe.

### Known Word Perturbation Probe

We test whether the models treat the meaning of closed-class words systematically by perturbing correctly classified jabberwocky sentence pairs with a closed-class word. More precisely, for a pair of closed-class words \(w\) and \(w^{\prime}\), we consider test items which can be formed by substitution of \(w\) by \(w^{\prime}\) in a correctly classified test item. We allow both \(w\) and \(w^{\prime}\) to be any of the closed-class items, including quantifiers, negation, nominal post-modifiers, or the the empty string \(\epsilon\) (thus modeling insertions and deletions of these known, closed-class items). Suppose that Example 1 was correctly classified. Substituting _some_ for _all_ in the premise of yields Example 2, and changes the relation from entailment (\(\sqsubseteq\)) to reverse entailment (\(\sqsupseteq\)).

* **All** blacksets wug. All blocks wug.
* **Some** blackets wug. All blockets wug.

There are two critical features of this probe. First, because we start from a correctly-classified jabberwocky pair, we can conclude that the novel words (e.g., _wing_ and _blackets_ above) were assigned appropriate meanings.

Second, since the perturbation only involves closed-class items which do not vary in meaning and have been highly trained, the perturbation should not affect the models ability to correctly classify the resulting sentence pair. If the model does misclassify the resulting pair, it can only be because a perturbed closed-class word (e.g., _some_) interacts with the open-class items (e.g., _wug_), in a way that is different from the pre-perturbation closed-class item (i.e., _all_). This is non-systematic behavior.

In order to rule out trivially correct behavior where the model simply ignores the perturbation, we consider only perturbations which result in a change of class (e.g., \(\sqsubset\rightarrow\sqsupset\)) for the sentence pair. In addition to accuracy on these perturbed items, we also examine the variance of model accuracy on probes across different blocks. If a model's accuracy varies depending only on the novel open-class items in a particular block, this provides further evidence that it does not treat word meaning systematically.

### Identical Open-class Words Probe

Some sentence pairs are classifiable without any knowledge of the novel words' meaning; for example, pairs where premise and hypothesis have identical open-class words. An instance is shown in Example 3: the two sentences must stand in contradiction, regardless of the meaning of _blicket_ or _wug_.

* All blickets wug. Some blackets don't wug.

The closed-class items and compositional structure of the language is sufficient for a learner to deduce the relationships between such sentences, even with unfamiliar nouns and verbs. Our second probe, the identical open-class words probe, tests the models' ability to correctly classify such pairs.

### Consistency Probe

Consider Examples 4 and 5, which present the same two sentences in opposite orders 

4. All blickets wug. All red blickets wug.
5. All red blickets wug. All blickets wug. In Example 4, the two sentences stand in an entailment (\(\sqsubset\)) relation. In Example 5, by contrast, the two sentences stand in a reverse entailment (\(\sqsupset\)) relation. This is a logically necessary consequence of the way the relations are defined. Reversing the order of sentences has predictable effects for all seven natural logic relations: in particular, such reversals map \(\sqsubset\mapsto\sqsupset\) and \(\sqsupset\mapsto\sqsubset\), leaving all other relations intact. Based on this observation, we develop a consistency probe of systematicity. We ask for each correctly classified jabberwocky block test item, whether the corresponding reversed item is also correctly classified. The intuition behind this probe is that whatever meaning a model assumes for the novel open-class words, it should assume the same meaning when the sentence order is reversed. If the reverse is not correctly classified, then this is strong evidence of contextual dependence in meaning.

## 8 Results

In this section, we report the results of two control analyses, and that of our three systematicity probes described above.

### Analysis I: Holdout Evaluations

We first establish that the models perform well on novel configurations of known words. Table 3 reports accuracy on heldout sentence pairs, described in SS5.5. The table reports average accuracies across training blocks together with the standard deviations of these statistics. As can be seen in the table, all models perform quite well on holdout forms across training blocks, with very little variance. Because these items use the same sampling scheme and vocabulary as the trained blocks, these simulations serve as a kind of upper bound on the performance and a lower bound on the variance that we can expect from the more challenging jabberwockyblock-based evaluations below.

### Analysis II: Distribution of Novel Words

Our three systematicity probes employ jabberwocky-type sentences--novel open-class words in sentential frames built from known closed-class words. Since models are not trained on these novel words, it is important to establish that they are from the same distribution as the trained words and, thus, that the models' performance is not driven by some pathological feature of the novel word embeddings.

Trained word embeddings were initialized randomly from \({\cal N}(0,1)\) and then updated during training. Novel word embeddings were simply drawn from \({\cal N}(0,1)\) and never updated. Figure 1 plots visualizations of the trained and novel open-class word embeddings in two dimensions, using t-SNE parameters computed over all open-class words Maaten and Hinton (2008). Trained words are plotted as \(+\), novel words as \(\bullet\). Color indicates the proportion of test items containing that word that were classified correctly. As the plot shows, the two sets of embeddings overlap considerably. Moreover, there does not appear to be a systematic relationship between rates of correct classification for items containing novel words and their proximity to trained words. We also performed a resampling analysis, determining that novel vectors did not differ significantly in length from trained vectors (\(p=0.85\)). Finally, we observed mean and standard deviation of the pairwise cosine similarity between trained and novel words to be \(0.999\) and \(0.058\) respectively, confirming that there is little evidence the distributions are different.

### Analysis III: Known Word Perturbation Probe

Recall from SS7.1 that the known word perturbation probe involves insertion, deletion, or substitution

Figure 1: Visualization of trained and novel open-class word embeddings.

 

of a trained closed-class word in a correctly classified jabberwocky-type sentence pair. Figure 2 plots the results of this probe. Each point represents a perturbation type--a group of perturbed test items that share their before/after target perturbed closed-class words and before/after relation pairs. The upper plot displays the mean accuracy of all perturbations, averaged across blocks, and the lower plot displays the standard deviations across blocks.

All models perform substantially worse than the holdout-evaluation on at least some of the perturbations. In addition, the standard deviation of accuracy between blocks is higher than the holdout tests. As discussed in SS7.1, low accuracy on this probe indicates that closed-class words do not maintain a consistent interpretation when paired with different open-class words. Variance across blocks shows that under all models the behavior of closed-class words is highly sensitive to the novel words they appear with.

Performance is also susceptible to interference from sentence-level features. For example, consider the perturbation which deletes a post-modifier from a sentence pair in negation, yielding a pair in cover relation. The self-attentive encoder performs perfectly when this perturbation is applied to a premise (\(100\%\pm 0.00\%\)), but not when applied to a hypothesis (\(86.60\%\pm 18.08\%\)). Similarly, deleting the adjective _red_ from the hypothesis of a forward-entailing pair results in an unrelated sentence pair (\(84.79\%\pm 7.50\%\)) or another forward-entailing pair (\(92.32\%,\pm 3.60\%\)) or an equality pair (\(100\%\pm 0.00\%\)). All the possible perturbations we studied exhibit similarly inconsistent performance.

### Analysis IV: Identical Open-Class Words Probe

Recall that the identical open-class words probe consist of sentence pairs where all open-class lexical items were identical. Table 4 shows the accuracies for these probes, trained on the small language. Average accuracies across jabberwocky blocks are reported together with standard deviations.

Accuracy on the probe pairs fails to reach the holdout test levels for most models and most relations besides \(\#\), and variance between blocks is much higher than in the holdout evaluation. Of special interest is negation (\({}^{\wedge}\)), for which accuracy is dramatically lower and variance dramatically higher than the holdout evaluation.

The results are similar for the large language condition, shown in Table 5. Although model accuracies improve somewhat, variance remains higher than the heldout level and accuracy lower. Recall that these probe-items can be classified while ignoring the specific identity of their open-class words. Thus, the models inability to leverage this fact, and high variance across different sets novel open-class words, illustrates their sensitivity to context.

### Analysis V: Consistency Probe

The consistency probe tests abstract knowledge of relationships between logical relations, such as the fact that two sentences that stand in a contradiction still stand in a contradiction after reversing their order. Results of this probe in the small-language condition are in Table 6: For each type of relation, we show the average percentage of correctly-labeled

\begin{table}
\begin{tabular}{l l c c c} \hline \hline Relation & BGRU & CONV & SATT & INFS \\  & mean (sdB) & mean (sdB) & mean (sdB) & mean (sdB) \\ \hline \(\#\) & 100.0 \(\pm\)0 & 100.0 \(\pm\)0 & 99.94 \(\pm\)0.26 & 99.67 \(\pm\)0.98 \\ \(\land\) & 55.68 \(\pm\)20.29 & 73.29 \(\pm\)10.8 & 23.71 \(\pm\)11.45 & 96.67 \(\pm\)10.98 \\ \(\sqcap\) & 90.78 \(\pm\)9.99 & 82.84 \(\pm\)6.51 & 75.22 \(\pm\)5.98 & 95.53 \(\pm\)2.64 \\ \(=\) & 90.43 \(\pm\)17.1 & 38.12 \(\pm\)15.56 & 71.94 \(\pm\)21.41 & 95.93 \(\pm\)6.5 \\ \(\sqcap\) & 90.34 \(\pm\)18.7 & 77.11 \(\pm\)5.9 & 81.44 \(\pm\)6.67 & 93.81 \(\pm\)2.96 \\ \(|\) & 93.08 \(\pm\)3.58 & 85.34 \(\pm\)5.47 & 74.05 \(\pm\)8.03 & 92.23 \(\pm\)4.6 \\ \(>\) & 88.01 \(\pm\)3.55 & 71.5 \(\pm\)7.32 & 78.4 \(\pm\)7.91 & 95.22 \(\pm\)3.58 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Identical open-class words probe performance, trained on the small language condition (trained on \(51,743\) sentence pairs, see §5.5)

Figure 2: Performance on the known word perturbation probe, small and large training conditions (see §5.5).

 

sentence pairs that, when presented in reverse order, were also correctly labeled.

The best-performing model on negation reversal is SATT, which correctly labeled reversed items \(66.92\%\) of the time. Although performance on negation is notably more difficult than the other relations, every model, on every relation, exhibited inter-block variance higher than that of the hold-out evaluations.

Furthermore, as can be seen in Table 7, the large language condition yields little improvement. Negation pairs are still well below the hold-out test threshold, still with a high degree of variation. Variation remains high for many relations, which is surprising because the means report accuracy on test items that were chosen specifically because the same item, in a reverse order, was already correctly labeled. Reversing the order of sentences causes the model to misclassify the resulting pair, more often for some blocks than others.

## 9 Discussion and Conclusion

Systematicity refers to the property of natural language representations whereby words (and other units or grammatical operations) have consistent meanings across different contexts. Our probes test whether deep learning systems learn to represent linguistic units systematically in the natural language inference task. Our results indicate that despite their high overall performance, these models tend to generalize in ways that allow the meanings of individual words to vary in different contexts, even in an artificial language where a totally systematic solution is available. This suggests the networks lack a sufficient inductive bias to learn systematic representations of words like quantifiers, which even in natural language exhibit very little meaning variation.

Our analyses contain two ideas that may be useful for future studies of systematicity. First, two of our probes (known word perturbation and consistency) are based on the idea of starting from a test item that is classified correctly, and applying a transformation that should result in a classifiable item (for a model that represents word meaning systematically). Second, our analyses made critical use of differential sensitivity (i.e., variance) of the models across test blocks with different novel words but otherwise identical information content. We believe these are a novel ideas that can be employed in future studies.

## Acknowledgements

We thank Brendan Lake, Marco Baroni, Adina Williams, Dima Bahdanau, Sam Gershman, Ishita Dasgupta, Alessandro Sordoni, Will Hamilton, Leon Bergen, the Montreal Computational and Quantitative Linguistics, and Reasoning and Learning Labs at McGill University for feedback on the manuscript. We are grateful to Facebook AI Research for providing extensive compute and other support. We also gratefully acknowledge the support of the Natural Sciences and Engineering Research Council of Canada, the Fonds de Recherche du Quebec, Societe et Culture, and the Canada CIFAR AI Chairs Program.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Relation & BGRU & CONV & SATT & INFS \\  & mean (sd) & mean (sd) & mean (sd) & mean (sd) \\ \hline \# & 97.40\(\pm\)0.86 & 97.8\(\pm\)0.93 & 98.58\(\pm\)0.74 & 97.03\(\pm\)0.87 \\ \(\wedge\) & 63.03\(\pm\)16.9 & 63.42\(\pm\)35.91 & 66.92\(\pm\)31.45 & 57.16\(\pm\)38.21 \\ \(\square\) & 92.45\(\pm\)6.26 & 88.1\(\pm\)8.16 & 93.16\(\pm\)5.42 & 90.64\(\pm\)7.76 \\ \(\blacksquare\) & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 \\ \(\Box\) & 91.37\(\pm\)6.23 & 94.73\(\pm\)6.51 & 96.42\(\pm\)32.28 & 87.02\(\pm\)9.61 \\ \(\mid\) & 96.02 \(\pm\)2.6 & 96.29\(\pm\)2.51 & 96.95\(\pm\)2.14 & 94.2\(\pm\)3.48 \\ \(\neg\) & 93.57\(\pm\)3.56 & 95 \(\pm\)29.7 & 96.4\(\pm\)28.3 & 93.1\(\pm\)3.77 \\ \hline \end{tabular}
\end{table}
Table 6: Consistency probe performance, trained on the small language condition (\(51,743\) sentence pairs, see §5.5).

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Relation & BGRU & CONV & SATT & INFS \\  & mean (sd) & mean (sd) & mean (sd) & mean (sd) \\ \hline \# & 98.95\(\pm\)0.65 & 98.96\(\pm\)0.54 & 98.83\(\pm\)0.6 & 98.38\(\pm\)0.74 \\ \(\wedge\) & 70.46 \(\pm\)33.72 & **77.25**\(\pm\) 26 & 84.27\(\pm\)23.89 & 95.64\(\pm\)35.13 \\ \(\square\) & 96.02 \(\pm\)2.96 & 96.68\(\pm\)32.6 & 96.78\(\pm\)4.23 & 95.01\(\pm\)5.38 \\ \(\blacksquare\) & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 \\ \(\supset\) & 93.53 \(\pm\)4.51 & 95.76\(\pm\)4.23 & 94.23\(\pm\)5.86 & 90.11\(\pm\)8.5 \\ \(\mid\) & 96.31 \(\pm\)2.73 & 97.25\(\pm\)2.05 & 97.17\(\pm\)2.23 & 94.46\(\pm\)2.42 \\ \(\sim\) & 96.25 \(\pm\)2.49 & 96.98\(\pm\)2.66 & 97.18\(\pm\)2.17 & 93.88\(\pm\)4.78 \\ \hline \end{tabular}
\end{table}
Table 7: Consistency probe performance, trained on the large langauge condition (\(478,649\) sentence pairs).

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Relation & BGRU & CONV & SATT & INFS \\  & mean (sd) & mean (sd) & mean (sd) & mean (sd) \\ \hline \# & 99.82\(\pm\)0.65 & 98.96\(\pm\)0.54 & 98.83\(\pm\)0.6 & 98.38\(\pm\)0.74 \\ \(\wedge\) & 70.46 \(\pm\)33.72 & **77.25**\(\pm\) 26 & 84.27\(\pm\)23.89 & 95.64\(\pm\)35.13 \\ \(\square\) & 96.02 \(\pm\)2.96 & 96.68\(\pm\)32.6 & 96.78\(\pm\)4.23 & 95.01\(\pm\)5.38 \\ \(\blacksquare\) & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 & 100 \(\pm\)0 \\ \(\supset\) & 93.53 \(\pm\)4.51 & 95.76\(\pm\)4.23 & 94.23\(\pm\)5.86 & 90.11\(\pm\)8.5 \\ \(\mid\) & 96.31 \(\pm\)2.73 & 97.25\(\pm\)2.05 & 97.17\(\pm\)2.23 & 94.46\(\pm\)2.42 \\ \(\sim\) & 96.25 \(\pm\)2.49 & 96.98\(\pm\)2.66 & 97.18\(\pm\)2.17 & 93.88\(\pm\)4.78 \\ \hline \end{tabular}
\end{table}
Table 7: Consistency probe performance, trained on the large langauge condition (\(478,649\) sentence pairs).

 



## References

* Albright and Hayes (2003) Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in English past tenses: A computational/experimental study. _Cognition_, 90(2):119-161.
* Bahdanau et al. (2019) Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2019. Systematic generalization: What is required and can it be learned? _arXiv_, abs/1811.12889(arXiv:1811.12889v2 [cs.CL]).
* Bentivogli et al. (2016) Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, and Roberto Zamparelli. 2016. Sick through the semeval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. _Language Resources and Evaluation_, 50(1):95-124.
* Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. _arXiv preprint arXiv:1508.05326_.
* Bowman et al. (2014) Samuel R Bowman, Christopher Potts, and Christopher D Manning. 2014. Recursive neural networks can learn logical semantics. _arXiv preprint arXiv:1406.1827_.
* Camap (1947) Rudolf Camap. 1947. Meaning and necessity: A study in semantics and modal logic.
* Smith (1996) Brian Cantwell Smith. 1996. _On the Origins of Objects_. MIT Press, Cambridge, Massachusetts.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_, pages 103-111.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680.
* Dasgupta et al. (2019) Ishita Dasgupta, Demi Guo, Samuel J. Gershman, and Noah D. Goodman. 2019. Analyzing machine-learned representations: A natural language case study.
* Dasgupta et al. (2018) Ishita Dasgupta, Demi Guo, Andreas Stuhlmuller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. _arXiv preprint arXiv:1802.04302_.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.
* Novaes (2012) Catarina Dutilh Novaes. 2012. _Formal Languages in Logic: A Philosophical and Cognitive Analysis_. Cambridge University Press, Cambridge, England.
* Ettinger et al. (2018) Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1790-1801, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Evans et al. (2018) Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. 2018. Can neural networks understand logical entailment? _arXiv preprint arXiv:1802.08535_.
* Fodor and Pylyshyn (1988) Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. _Cognition_, 28(1):3-71.
* Gershman and Tenenbaum (2015) Samuel Gershman and Joshua B Tenenbaum. 2015. Phrase similarity in humans and machines. In _CogSci_.
* Giulianelli et al. (2018) Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. _arXiv preprint arXiv:1808.08079_.
* Heim and Kratzer (2000) Irene Heim and Angelika Kratzer. 2000. _Semantics in Generative Grammar_. Blackwell Publishing, Malden, MA.
* Henaff et al. (2016) Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hodges (2012) Wilfrid Hodges. 2012. Formalizing the relationship between meaning and syntax. In _The Oxford handbook of compositionality_. Oxford Handbooks in Linguistic.
* Hudson and Manning (2018) Drew A. Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning.
* Janssen et al. (2012) Theo MV Janssen et al. 2012. Compositionality: its historic context. _The Oxford handbook of compositionality_, pages 19-46.
* Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2901-2910

Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Safiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2019. Measuring compositional generalization: A comprehensive method on realistic data.
* Kiela et al. (2018) Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018. Dynamic meta-embeddings for improved sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1466-1477.
* Kim et al. (2019) Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexi Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_, pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
* Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In _International Conference on Machine Learning_, pages 2879-2888.
* Lake et al. (2017a) Brenden Lake, Tal Linzen, and Marco Baroni. 2017a. Human few-shot learning of compositional instructions. In Ashok Goel, Colleen Seifert, and Christian Freksa, editors, _Proceedings of the 41st Annual Conference of the Cognitive Science Society_, pages 611-616. Cognitive Science Society, Montreal, Canada.
* Lake et al. (2017b) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017b. Building machines that learn and think like people. _Behavioral and Brain Sciences_, 40.
* Lakoff (1970) George Lakoff. 1970. Linguistics and natural logic. _Synthese_, 22(1-2):151-271.
* Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. _arXiv preprint arXiv:1703.03130_.
* Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(Nov):2579-2605.
* MacCartney and Manning (2009) Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In _Proceedings of the Eighth International Conference on Computational Semantics_, IWCS-8 '09, pages 140-156, Stroudsburg, PA, USA. Association for Computational Linguistics.
* MacCartney and Manning (2014) Bill MacCartney and Christopher D. Manning. 2014. Natural logic and natural language inference. In _Computing Meaning: Volume 4_, pages 129-147, Dordrecht. Springer Netherlands.
* McCoy et al. (2019a) R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. Berts of a feather do not generalize together: Large variability in generalization across models with similar test performance. _arXiv preprint arXiv:1911.02969_.
* McCoy et al. (2019b) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, Florence, Italy.
* Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017. Neural semantic encoders. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_, volume 1, page 397. NIH Public Access.
* Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments.
* Parikh et al. (2016) Ankur P Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. _arXiv preprint arXiv:1606.01933_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237.
* Prasad et al. (2019) Grusha Prasad, Marten van Schijandel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. _URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf_.
* Richardson et al. (2019) Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2019. Probing natural language inference models through semantic fragments.
* Rocktaschel et al. (2015) Tim Rocktaschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. _arXiv preprint arXiv:1509.06664_.
* Sinha et al. (2019) Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. Clutrr: A diagnostic benchmark for inductive reasoning from text.
* Soulos et al. (2019) Paul Soulos, Tom McCoy, Tal Linzen, and Paul Smolensky. 2019. Discovering the compositional structure of vector representations with role learning networks.

 

Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. _arXiv preprint arXiv:1904.01172_.
* Szabo (2012) Zoltan Szabo. 2012. The case for compositionality. _The Oxford Handbook of Compositionality_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
* Veldhoen and Zuidema (2017) Sara Veldhoen and Willem Zuidema. 2017. Can neural networks learn logical reasoning? _Proceedings of the Conference on Logic and Machine Learning in Natural Language_.
* Wang et al. (2018) Wei Wang, Ming Yan, and Chen Wu. 2018. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. _preprint arXiv:1811.11934_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Viynals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation.
* Yoon et al. (2018) Deunsol Yoon, Dongbok Lee, and SangKeun Lee. 2018. Dynamic self-attention: Computing attention over words dynamically for sentence embedding. _arXiv preprint arXiv:1808.07383_.
* Zadrozny (1994) Wlodek Zadrozny. 1994. From compositional to systematic semantics. _Linguistics and philosophy_, 17(4):329-342.
* Zhang et al. (2018) Zhuosheng Zhang, Yuwei Wu, Zuchao Li, Shexia He, Hai Zhao, Xi Zhou, and Xiang Zhou. 2018. I know what you want: Semantic learning for text comprehension. _arXiv preprint arXiv:1809.02794_.
* Zhao et al. (2015) Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_