SlowGuess/ACL_2020_OCR huggingface dataset:
Emily Goodwin,\*1,5 Koustuv Sinha,\*2,3,5 Timothy J. O'Donnell\*1,4,5
$^{1}$ Department of Linguistics,  $^{2}$ School of Computer Science, McGill University, Canada
$^{3}$ Facebook AI Research (FAIR), Montreal  $^{4}$ Canada CIFAR AI Chair, Mila
$^{5}$ Quebec Artificial Intelligence Institute (Mila)
{emily.goodwin, koustuv.sinha}@mail.mcgill.ca
timothy.odonnell@mcgill.ca
Recently, there has been much interest in the question of whether deep natural language understanding models exhibit systematicity—generalizing such that units like words make consistent contributions to the meaning of the sentences in which they appear. There is accumulating evidence that neural models often generalize non-systematically. We examined the notion of systematicity from a linguistic perspective, defining a set of probes and a set of metrics to measure systematic behaviour. We also identified ways in which network architectures can generalize non-systematically, and discuss why such forms of generalization may be unsatisfying. As a case study, we performed a series of experiments in the setting of natural language inference (NLI), demonstrating that some NLU systems achieve high overall performance despite being non-systematic.
Language allows us to express and comprehend a vast variety of novel thoughts and ideas. This creativity is made possible by compositionality—the linguistic system builds utterances by combining an inventory of primitive units such as morphemes, words, or idioms (the lexicon), using a small set of structure-building operations (the grammar; Camap, 1947; Fodor and Pylyshyn, 1988; Hodges, 2012; Janssen et al., 2012; Lake et al., 2017b; Szabó, 2012; Zadrozny, 1994; Lake et al., 2017a).
One property of compositional systems, widely studied in the cognitive sciences, is the phenomenon of systematicity. Systematicity refers to the fact that lexical units such as words make consistent contributions to the meaning of the sentences in which they appear. Fodor and Pylyshyn
(1988) provided a famous example: If a competent speaker of English knows the meaning of the sentence John loves the girl, they also know the meaning of The girl loves John. This is because for speakers of English knowing the meaning of the first sentence implies knowing the meaning of the individual words the, loves, girl, and John as well as grammatical principles such as how transitive verbs take their arguments. But knowing these words and principles of grammar implies knowing how to compose the meaning of the second sentence.
Deep learning systems now regularly exhibit very high performance on a large variety of natural language tasks, including machine translation (Wu et al., 2016; Vaswani et al., 2017), question answering (Wang et al., 2018; Henaff et al., 2016), visual question answering (Hudson and Manning, 2018), and natural language inference (Devlin et al., 2018; Storks et al., 2019). Recently, however, researchers have asked whether such systems generalize systematically (see §4).
Systematicity is the property whereby words have consistent contributions to composed meaning; the alternative is the situation where words have a high degree of contextually conditioned meaning variation. In such cases, generalization may be based on local heuristics (McCoy et al., 2019b; Niven and Kao, 2019), variegated similarity (Albright and Hayes, 2003), or local approximations (Veldhoen and Zuidema, 2017), where the contribution of individual units to the meaning of the sentence can vary greatly across sentences, interacting with other units in highly inconsistent and complex ways.
This paper introduces several novel probes for testing systematic generalization. We employ an artificial language to have control over systematicity and contextual meaning variation. Applying our probes to this language in an NLI setting reveals
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958-1969 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
that some deep learning systems which achieve very high accuracy on standard holdout evaluations do so in ways which are non-systematic: the networks do not consistently capture the basic notion that certain classes of words have meanings which are consistent across the contexts in which they appear.
The rest of the paper is organized as follows. §2 discusses degrees of systematicity and contextually conditioned variation; §3 introduces the distinction between open- and closed-class words, which we use in our probes. §5 introduces the NLI task and describes the artificial language we use; §6 discusses the models that we tested and the details of our training setup; §7 introduces our probes of systematicity and results are presented in §8.1
2 Systematicity and Contextual Conditioning
Compositionality is often stated as the principle that the meaning of an utterance is determined by the meanings of its parts and the way those parts are combined (see, e.g., Heim and Kratzer, 2000).
Systematicity, the property that words mean the same thing in different contexts, is closely related to compositionality; nevertheless, compositional systems can vary in their degree of systematicity. At one end of the spectrum are systems in which primitive units contribute exactly one identical meaning across all contexts. This high degree of systematicity is approached by artificial formal systems including programming languages and logics, though even these systems don't fully achieve this ideal (Cantwell Smith, 1996; Dutilh Novaes, 2012).
The opposite of systematicity is the phenomenon of contextually conditioned variation in meaning where the contribution of individual words varies according to the sentential contexts in which they appear. Natural languages exhibit such context dependence in phenomena like homophony, polysemy, multi-word idioms, and co-compositionality. Nevertheless, there are many words in natural language—especially closed-class words like quantifiers (see below)—which exhibit very little variability in meaning across sentences.
At the other end of the spectrum from programming languages and logics are systems where many or most meanings are highly context dependent.
The logical extreme—a system where each word has a different and unrelated meaning every time it occurs—is clearly of limited usefulness since it would make generalization impossible. Nevertheless, learners with sufficient memory capacity and flexibility of representation, such as deep learning models, can learn systems with very high degrees of contextual conditioning—in particular, higher than human language learners. An important goal for building systems that learn and generalize like people is to engineer systems with inductive biases for the right degree of systematicity. In §8, we give evidence that some neural systems are likely too biased toward allowing contextually conditioned meaning variability for words, such as quantifiers, which do not vary greatly in natural language.
3 Compositional Structure in Natural Language
Natural language distinguishes between content or open-class lexical units and function or closed-class lexical units. The former refers to categories, such a nouns and verbs, which carry the majority of contentful meaning in a sentence and which permit new coinages. Closed-class units, by contrast, carry most of the grammatical structure of the sentence and consist of things like inflectional morphemes (like pluralizing -s in English) and words like determiners, quantifiers, and negation (e.g., all, some, the in English). These are mostly fixed; adult speakers do not coin new quantifiers, for example, the way that they coin new nouns.
Leveraging this distinction gives rise to the possibility of constructing probes based on jabberwocky-type sentences. This term references the poem Jabberwocky by Lewis Carroll, which combines nonsense open-class words with familiar closed-class words in a way that allows speakers to recognize the expression as well formed. For example, English speakers identify a contradiction in the sentence All Jabberwocks flug, but some Jabberwocks don't flug, without a meaning for jabberwock and flug. This is possible because we expect the words all, some, but, and don't to contribute the same meaning as they do when combined with familiar words, like All pigs sleep, but some pigs don't sleep.
Using jabberwocky-type sentences, we tested the generalizability of certain closed-class word representations learned by neural networks. Giving the networks many examples of each construction
$^{1}$ Code for datasets and models can be found here: https://github.com/emilygoodwin/systematicity
with a large variety of different content words—that is, large amounts of highly varied evidence about the meaning of the closed-class words—we asked during the test phase how fragile this knowledge is when transferred to new open-class words. That is, our probes combine novel open-class words with familiar closed-class words, to test whether the closed-class words are treated systematically by the network. For example, we might train the networks to identify contradictions in pairs like All pigs sleep; some pigs don't sleep, and test whether the network can identify the contradiction in a pair like All Jabberwocks flug; some Jabberwocks don't flug. A systematic learner would reliably identify the contradiction, whereas a non-systematic learner may allow the closed-class words (all, some, don't) to take on contextually conditioned meanings that depend on the novel context words.
There has been much interest in the problem of systematic generalization in recent years (Bahdanau et al., 2019; Bentivogli et al., 2016; Lake et al., 2017a,b; Gershman and Tenenbaum, 2015; McCoy et al., 2019a; Veldhoen and Zuidema, 2017; Soulos et al., 2019; Prasad et al., 2019; Richardson et al., 2019; Johnson et al., 2017, inter alia).
In contrast to our approach (testing novel words in familiar combinations), many of these studies probe systematicity by testing familiar words in novel combinations. Lake and Baroni (2018) adopt this approach in semantic parsing with an artificial language known as SCAN. Dasgupta et al. (2018, 2019) introduce a naturalistic NLI dataset, with test items that shuffle the argument structure of natural language utterances. In the in the inductive logic programming domain, Sinha et al. (2019) introduced the CLUTTR relational-reasoning benchmark. The novel-combinations-of-familiar-words approach was formalized in the CFQ dataset and associated distribution metric of Keysers et al. (2019). Ettinger et al. (2018) introduced a semantic-role-labeling and negation-scope labeling dataset, which tests compositional generalization with novel combinations of familiar words and makes use of syntactic constructions like relative clauses. Finally, Kim et al. (2019) explore pre-training schemes' abilities to learn prepositions and wh-words with syntactic transformations (two kinds of closed-class words which our work does not address).
A different type of systematicity analysis directly
investigates learned representations, rather than developing probes of model behavior. This is done either through visualization (Veldhoen and Zuidema, 2017), training a second network to approximate learned representations using a symbolic structure (Soulos et al., 2019) or as a diagnostic classifier (Giulianelli et al., 2018), or reconstructing the semantic space through similarity measurements over representations (Prasad et al., 2019).
5 Study Setup
5.1 Natural Language Inference
We make use of the Natural language inference (NLI) task to study the question of systematicity. The NLI task is to infer the relation between two sentences (the premise and the hypothesis). Sentence pairs must be classified into one of a set of predefined logical relations such as entailment or contradiction. For example, the sentence All mammals growl entails the sentence All pigs growl. A rapidly growing number of studies have shown that deep learning models can achieve very high performance in this setting (Evans et al., 2018; Conneau et al., 2017; Bowman et al., 2014; Yoon et al., 2018; Kiela et al., 2018; Munkhdalai and Yu, 2017; Rocktäschel et al., 2015; Peters et al., 2018; Parikh et al., 2016; Zhang et al., 2018; Radford et al., 2018; Devlin et al., 2018; Storks et al., 2019).
5.2 Natural Logic
We adopt the formulation of NLI known as natural logic (MacCartney and Manning, 2014, 2009; Lakoff, 1970). Natural logic makes use of seven logical relations between pairs of sentences. These are shown in Table 1. These relations can be interpreted as the set theoretic relationship between the extensions of the two expressions. For instance, if the expressions are the simple nouns  $\text{warthog}$  and  $\text{pig}$ , then the entailment relation  $(\square)$  holds between these extensions  $(\text{warthog} \square \text{pig})$  since every  $\text{warthog}$  is a kind of  $\text{pig}$ .
For higher-order operators such as quantifiers, relations can be defined between sets of possible worlds. For instance, the set of possible worlds consistent with the expression All bickets wug is a subset of the set of possible worlds consistent with the logically weaker expression All red bickets wug. Critically, the relationship between composed expressions such as All  $X Y$  and All  $P Q$  is determined entirely by the relations between  $X / Y$  and  $P / Q$ , respectively. Thus, natural logic allows
us to compute the relation between the whole expressions using the relations between parts. We define an artificial language in which such alignments are easy to compute, and use this language to probe deep learning systems' ability to generalize systematically.
Symbol	Name	Example	Set-theoretic definition
x ≡ y	equivalence	pig ≡ pig	x = y
x □ y	forward entailment	pig □ mammal	x ⊆ y
x ▷ y	reverse entailment	mammal □ pig	x ⊥ y
x ∧ y	negation	pig ∧ not pig	x ∩ y = ∅ ∧ x ∪ y = U
x | y	alternation	pig | cat	x ∩ y = ∅ ∧ x ∪ y ≠ U
x ∪ y	cover	mammal ∪ not pig	x ∩ y ≠ ∅ ∧ x ∪ y = U
x#y	independence	hungry # warthog	(all other cases)
Table 1: MacCartney and Manning (2009)'s implementation of natural logic relations
5.3 The Artificial Language
In our artificial language, sentences are generated according to the six-position template shown in Table 2, and include a quantifier (position 1), noun (position 3), and verb (position 6), with optional pre- and post-modifiers (position 2 and 4) and optional negation (position 5). For readability, all examples in this paper use real English words; however, simulations can use uniquely identified abstract symbols (i.e., generated bygensym).
We compute the relation between position-aligned pairs of sentences in our language using the natural logic system (described in §5.2). Quantifiers and negation have their usual natural-language semantics in our artificial language; pre- and post-modifiers are treated intersectively. Open-class items (nouns and verbs) are organized into linear hierarchical taxonomies, where each open-class word is the sub- or super-set of exactly one other open-class item in the same taxonomy. For example, since dogs are all mammals, and all mammals animals, they form the entailment hierarchy dogs  $\square$  mammals  $\square$  animals. We vary the number of distinct noun and verb taxonomies according to an approach we refer to as block structure, described in the next section.
5.4 Block Structure
In natural language, most open-class words do not appear with equal probability with every other word. Instead, their distribution is biased and clumpy, with words in similar topics occurring together. To mimic such topic structure, we group nouns and verbs into blocks. Each block consists of six nouns and six verbs, which form taxonomic hi
erarchies (e.g., lizards/animals, run/move). Nouns and verbs from different blocks have no taxonomic relationship (e.g., lizards and screwdrivers or run and read) and do not co-occur in the same sentence pair. Because each block includes a six verbs and six nouns in a linear taxonomic hierarchy, no single block is intrinsically harder to learn than any other block.
The same set of closed-class words appear with all blocks of open-class words, and their meanings are systematic regardless of the open-class words (nouns and verbs) they are combined with. For example, the quantifier some has a consistent meaning when it is applied to some screwdrivers or some animals. Because closed-class words are shared across blocks, models are trained on extensive and varied evidence of their behaviour. We present closed-class words in a wide variety of sentential contexts, with a wide variety of different open-class words, to provide maximal pressure against overfitting and maximal evidence of their consistent meaning.
5.5 Test and Train Structure
We now describe the structure of our training blocks, holdout test set, and jabberwocky blocks. We also discuss our two test conditions, and several other issues that arise in the construction of our dataset.
Training set: For each training block, we sampled (without replacement) one sentence pair for every possible combination of open-class words, that is, every combination of nouns and verbs  $\langle \text{noun}_1, \text{noun}_2, \text{verb}_1, \text{verb}_2 \rangle$ . Closed-class words were sampled uniformly to fill each remaining positions in the sentence (see Table 2). A random subset of  $20\%$  of training items were reserved for validation (early stopping) and not used during training.
Holdout test set: For each training block, we sampled a holdout set of forms using the same nouns and verbs, but disjoint from the training set just described. The sampling procedure was identical to that for the training blocks. These holdout items allow us to test the generalization of the models with known words in novel configurations (see §8.1).
Jabberwocky test set: Each jabberwocky block consisted of novel open-class items (i.e., nouns and verbs) that did not appear in training blocks. For each jabberwocky block, we began by following a
Position	1	2	3	4	5	6
Category	quantifier	nominal premodifier	noun	nominal postmodifier	negation	verb
Status	Obligatory	Optional	Obligatory	Optional	Optional	Obligatory
Class	Closed	Closed	Open	Closed	Closed	Open
Example	All	brown	dogs	that bark	don’t	run
Table 2: A template for sentences in the artificial language. Each sentence fills the obligatory positions 1, 3, and 6 with a word: a quantifier, noun, and verb. Optional positions (2, 4 and 5) are filled by either a word (adjective, postmodifier or negation) or by the empty string. Closed-class categories (Quantifiers, adjectives, post modifiers and negation) do not include novel words, while open-class categories (nouns and verbs) includes novel words that are only exposed in the test set.
sampling procedure identical to that for the training/holdout sets with these new words. Several of our systematicity probes are based on the behavior of neighboring pairs of test sentences (see §7). To ensure that all such necessary pairs were in the jabberwocky test set, we extended the initial sample with any missing test items.
Training conditions: Since a single set of closed-class words is used across all blocks, adding more blocks increases evidence of the meaning of these words without encouraging overfitting. To study the effect of increasing evidence in this manner, we use two training conditions: small with 20 training blocks and large with 185 training blocks. Both conditions contained 20 jabberwocky blocks. The small condition consisted of 51,743 training, 10,399 validation, and 3,694,005 test (holdout and jabberwocky) pairs. The large condition consisted of 478,649 training, 96,005 validation, and 3,694,455 test items.
Balancing: One consequence of the sampling method is that logical relations will not be equally represented in training. In fact, it is impossible to simultaneously balance the distributions of syntactic constructions, logical relations, and instances of words. In this trade-off, we chose to balance the distribution of open-class words in the vocabulary, as we are focused primarily on the ability of neural networks to generalize closed-class word meaning. Balancing instances of open-class words provided the greatest variety of learning contexts for the meanings of the closed-class items.
6.1 Models
We analyze performance on four simple baseline models known to perform well on standard NLITasks, such as the Stanford Natural Language Inference datasets, (Bowman et al., 2015). Following Conneau et al. (2017), the hypothesis  $u$  and
premise  $v$  are individually encoded by neural sequence encoders such as a long short-term memory (LSTM; Hochreiter and Schmidhuber, 1997) or gated recurrent unit (GRU; Cho et al., 2014). These vectors, together with their element-wise product  $u * v$  and element-wise difference  $u - v$  are fed into a fully connected multilayer perceptron layer to predict the relation. The encodings  $u$  and  $v$  are produced from an input sentence of  $M$  words,  $w_{1},\ldots ,w_{M}$ , using a recurrent neural network, which produces a set of a set of  $M$  hidden representations  $h_1,\dots ,h_t$ , where  $h_t = f(w_1,\dots ,w_M)$ . The sequence encoding is represented by its last hidden vector  $h_T$ .
The simplest of four models sets  $f$  to be a bidirectional gated recurrent unit (BGRU). This model concatenates the last hidden state of a GRU run forwards over the sequence and the last hidden state of GRU run backwards over the sequence, for example,  $u = [\overrightarrow{h_M},\overrightarrow{h_M}]$ .
Our second embedding system is the Inferent model reported by Conneau et al. (2017), a bidirectional LSTM with max pooling (INFS). This is a model where  $f$  is an LSTM. Each word is represented by the concatenation of a forward and backward representation:  $h_t = \overleftarrow{h_t}, \overrightarrow{h_t}$ . We constructed a fixed vector representation of the sequence  $h_t$  by selecting the maximum value over each dimension of the hidden units of the words in the sentence.
Our third model is a self-attentive sentence encoder (SATT) which uses an attention mechanism over the hidden states of a BiLSTM to generate the sentence representation (Lin et al., 2017). This attention mechanism is a weighted linear combination of the word representations, denoted by  $u = \sum_{M} \alpha_{i} h_{i}$ , where the weights are calculated as follows:
$$
\begin{array}{l} \bar {h} _ {i} = \tanh  \left(W h _ {i} + b _ {w}\right) \\ \alpha_ {i} = \frac {e ^ {\bar {h} _ {i} ^ {\top} u _ {w}}}{\sum_ {i} e ^ {\bar {h} _ {i} ^ {\top} u _ {w}}} \\ \end{array}
$$
where,  $u_{w}$  is a learned context query vector and  $(W, b_{w})$  are the weights of an affine transformation. This self-attentive network also has multiple views of the sentence, so the model can attend to multiple parts of the given sentence at the same time.
Finally, we test the Hierarchical ConvolutionalNetwork (CONV) architecture from (Conneau et al., 2017) which is itself inspired from the model AdaSent (Zhao et al., 2015). This model has four convolution layers; at each layer the intermediate representation  $u_{i}$  is computed by a max-pooling operation over feature maps. The final representation is a concatenation  $u = [u_{1},\dots,u_{l}]$  where  $l$  is the number of layers.
7 Probing Systematicity
In this section, we study the systematicity of the models described in §6.1. Recall that systematicity refers to the degree to which words have consistent meaning across different contexts, and is contrasted with contextually conditioned variation in meaning. We describe three novel probes of systematicity which we call the known word perturbation probe, the identical open-class words probe, and the consistency probe.
All probes take advantage of the distinction between closed-class and open-class words reflected in the design of our artificial language, and are performed on sentence pairs with novel open-class words (jabberwocky-type sentences; see §5.5). We now describe the logic of each probe.
7.1 Known Word Perturbation Probe