SlowGuess/ACL_2020_OCR huggingface dataset:
James Henderson  
Idiap Research Institute, Switzerland  
james.henderson@idiap.ch
In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the challenges facing research in deep learning architectures for natural language understanding.
When neural networks first started being applied to natural language in the 1980s and 90s, they represented a radical departure from standard practice in computational linguistics. Connectionists had vector representations and learning algorithms, and they didn't see any need for anything else. Everything was a point in a vector space, and everything about the nature of language could be learned from data. On the other hand, most computational linguists had linguistic theories and the poverty-of-the-stimulus argument. Obviously some things were learned from data, but all the interesting things about the nature of language had to be innate.
A quarter century later, we can say two things with certainty: they were both wrong. Vector-space representations and machine learning algorithms are much more powerful than was thought. Much of the linguistic knowledge which computational linguists assumed needed to be innate can in fact be learned from data. But the unbounded discrete structured representations they used have not been replaced by vector-space representations. Instead, the successful uses of neural networks in computational linguistics have replaced specific pieces of computational-linguistic models with new neural
network architectures which bring together continuous vector spaces with structured representations in ways which are novel for both machine learning and computational linguistics.
Thus, the great progress which we have made through the application of neural networks to natural language processing should not be viewed as a conquest, but as a compromise. As well as the unquestionable impact of machine learning research on NLP, the nature of language has had a profound impact on progress in machine learning. In this paper we trace this impact, and speculate on future progress and its limits.
We start with a sketch of the insights from grammar formalisms about the nature of language, with their multiple levels, structured representations and rules. The rules were soon learned with statistical methods, followed by the use of neural networks to replace symbols with induced vectors, but the most effective models still kept structured representations, such as syntactic trees. More recently, attention-based models have replaced hand-coded structures with induced structures. The resulting models represent language with multiple levels of structured representations, much as has always been done. Given this perspective, we identify remaining challenges in learning language from data, and its possible limitations.
2 Grammar Formalisms versus Connectionism
2.1 Grammar Formalisms
Our modern understanding of the computational properties of language started with the introduction of grammar formalisms. Context Free Grammars (Chomsky, 1959) illustrated how a formal system could model the infinite generative capacity of language with a bounded grammar. This formalism soon proved inadequate to account for the diversity
6294
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6294-6306 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
of phenomena in human languages, and a number of linguistically-motivated grammar formalisms were proposed (e.g HPSG (Pollard and Sag, 1987), TAG (Joshi, 1987), CCG (Steedman, 2000)).
All these grammar formalisms shared certain properties, motivated by the understanding of the nature of languages in Linguistics. They all postulate representations which decompose an utterances into a set of sub-parts, with labels of the parts and a structure of inter-dependence between them. And they all assume that this decomposition happens at multiple levels of representation. For example that spoken utterances can be decomposed into sentences, sentences can be decomposed into words, words can be decomposed into morphemes, and morphemes can be decomposed into phonemes, before we reach the observable sound signal. In the interests of uniformity, we will refer to the subparts in each level of representation as its entities, their labels as their properties, and their structure of inter-dependence as their relations. The structure of inter-dependence between entities at different levels will also be referred to as relations.
In addition to these representations, grammar formalisms include specifications of the allowable structures. These may take the form of hard constraints or soft objectives, or of deterministic rules or stochastic processes. In all cases, the purpose of these specifications is to account for the regularities found in natural languages. In the interests of uniformity, we will refer to all these different kinds of specifications of allowable structures as rules. These rules may apply within or between levels of representation.
In addition to explicit rules, computational linguistic formalisms implicitly make claims about the regularities found in natural languages through their expressive power. Certain types of rules simply cannot be specified, thus claiming that such rules are not necessary to capture the regularities found in any natural language. These claims differ across formalisms, but the study of the expressive power of grammar formalisms have identified certain key principles (Joshi et al., 1990). Firstly, that the set of rules in a given grammar is bounded. This in turn implies that the set of properties and relations in a given grammar is also bounded.
But language is unbounded<sup>1</sup> in nature, since sentences and texts can be arbitrarily long. Grammar
formalisms capture this unboundedness by allowing an unbounded number of entities in a representation, and thus an unbounded number of rule applications. It is generally accepted that the number of entities grows linearly with the length of the sentence (Joshi et al., 1990), so each level can have at most a number of entities which is linear in the number of entities at the level(s) below.
Computational linguistic grammar formalisms also typically assume that the properties and relations are discrete, called symbolic representations. These may be atomic categories, as in CFGs, TAGs, CCG and dependency grammar, or they may be feature structures, as in HPSG.
2.2 Connectionism
Other researchers who were more interested in the computational properties of neurological systems found this reliance on discrete categorical representations untenable. Processing in the brain used real-valued representations distributed across many neurons. Based on successes following the development of multi-layered perceptrons (MLPs) (Rumelhart et al., 1986b), an approach to modelling cognitive phenomena was developed called connectionism. Connectionism uses vector-space representations to reflect the distributed continuous nature of representations in the brain. Similarly, their rules are specified with vectors of continuous parameters. MLPs are so powerful that they are arbitrary function approximators (Hornik et al., 1989). And thanks to backpropagation learning (Rumelhart et al., 1986a) in neural network models, such as MLPs and Simple Recurrent Networks (SRNs) (Elman, 1990), these vector-space representations and rules could be learned from data.
The ability to learn powerful vector-space representations from data led many connectionist to argue that the complex discrete structured representations of computational linguistics were neither necessary nor desirable (e.g. Smolensky (1988, 1990); Elman (1991); Miikkulainen (1993); Seidenberg (2007)). Distributed vector-space representations were thought to be so powerful that there was no need for anything else. Learning from data made linguistic theories irrelevant. (See also (Collobert and Weston, 2008; Collobert et al., 2011; Sutskever et al., 2014) for more recent incarnations.)
The idea that vector-space representations are adequate for natural language and other cognitive phenomena was questioned from several directions.
<sup>1</sup>A set of things (e.g. the sentences of a language) have unbounded size if for any finite size there is always some element in the set which is larger than that.
6295
From neuroscience, researchers questioned how a simple vector could encode features of more than one thing at a time. If we see a red square together with a blue triangle, how do we represent the difference between that and a red triangle with a blue square, since the vector elements for red, blue, square and triangle would all be active at the same time? This is known as the variable binding problem, so called because variables are used to do this binding in symbolic representations, as in  $red(x) \wedge triangle(x) \wedge blue(y) \wedge square(y)$ . One proposal has been that the precise timing of neuron activation spikes could be used to encode variable binding, called Temporal Synchrony Variable Binding (von der Malsburg, 1981; Shastri and Ajjanagadde, 1993). Neural spike trains have both a phase and a period, so the phase could be used to encode variable binding while still allowing the period to be used for sequential computation. This work indicated how entities could be represented in a neurally-inspired computational architecture.
The adequacy of vector-space representations was also questioned based on the regularities found in natural language. In particular, Fodor and Pylyshyn (1988) argued that connectionist architectures were not adequate to account for regularities which they characterised as systematicity (see also (Smolensky, 1990; Fodor and McLaughlin, 1990)). In essence, systematicity requires that learned rules generalise in a way that respects structured representations. Here again the issue is representing multiple entities at the same time, but with the additional requirement of representing the structural relationships between these entities. Only rules which are parameterised in terms of such representations can generalise in a way which accounts for the generalisations found in language.
Early work on neural networks for natural language recognised the significance of variable binding for solving the issues with systematicity (Henderson, 1996, 2000). Henderson (1994, 2000) argued that extending neural networks with temporal synchrony variable binding made them powerful enough to account for the regularities found in language. Using time to encode variable bindings means that learning could generalise in a linguistically appropriate way (Henderson, 1996), since rules (neuronal synapses) learned for one variable (time) would systematically generalise to other variables. Although relations were not stored explicitly, it was claimed that for language understanding it is
adequate to recover them from the features of the entities (Henderson, 1994, 2000). But these arguments were largely theoretical, and it was not clear how they could be incorporated in learning-based architectures.
2.3 Statistical Models
Although researchers in computational linguistics did not want to abandon their representations, they did recognise the importance of learning from data. The first successes in this direction came from learning rules with statistical methods, such as part-of-speech tagging with hidden Markov models. For syntactic parsing, the development of the Penn Treebank led to many statistical models which learned the rules of grammar (Collins, 1997, 1999; Charniak, 1997; Ratnaparkhi, 1999).
These statistical models were very successful at learning from the distributions of linguistic representations which had been annotated in the corpus they were trained on. But they still required linguistically-motivated designs to work well. In particular, feature engineering is necessary to make sure that these statistical machine-learning method can search a space of rules which is sufficiently broad to include good models but sufficiently narrow to allow learning from limited data.
3 Inducing Features of Entities
Early work on neural networks for natural language recognised the potential of neural networks for learning the features as well, replacing feature engineering. But empirically successful neural network models for NLP were only achieved with approaches where the neural network was used to model one component within an otherwise traditional symbolic NLP model.
The first work to achieve empirical success in comparison to non-neural statistical models was work on language modelling. Bengio et al. (2001, 2003) used an MLP to estimate the parameters of an n-gram language model, and showed improvements when interpolated with a statistical n-gram language model. A crucial innovation of this model was the introduction of word embeddings. The idea that the properties of a word could be represented by a vector reflecting the distribution of the word in text was introduced earlier in non-neural statistical models (e.g. Deerwester et al., 1990; Schütze, 1993; Burgess, 1998; Padó and Lapata, 2007; Erk, 2010)). This work showed that similarity in the
6296
PTB Constituents
model	LP	LR	F1
Costa et al. (2001)	PoS	57.8	64.9
Henderson (2003)	PoS	83.3	84.3
Henderson (2003)	88.8	89.5	89.1
Henderson (2004)	89.8	90.4	90.1
Vinyals et al. (2015) seq2seq	<70
Vinyals et al. (2015) attn	88.3
Vinyals et al. (2015) seq2seq semisup	90.5
CoNLL09 Dependencies
model (transition-based)	UAS	LAS
Titov and Henderson (2007a)*	91.44	88.65
Chen and Manning (2014)*	89.17	86.49
Yazdani and Henderson (2015)	90.75	88.14
Stanford Dependencies
model (transition-based)	UAS	LAS
Chen and Manning (2014)	91.80	89.60
Dyer et al. (2015)	93.10	90.90
Andor et al. (2016)	94.61	92.79
Kiperwasser and Goldberg (2016)	93.9	91.9
Mohammadshahi and Henderson (2019) BERT	95.63	93.81
Table 1: Some neural network parsing results on Penn Treebank WSJ. LP/LR/F1: labelled constituent precision/recall/F-measure. UAS/LAS: unlabeled/labelled dependency accuracy. *results reported in (Yazdani and Henderson, 2015).
resulting vector space is correlated with semantic similarity. Learning vector-space representations of words with neural networks (rather than SVD) have showed similar effects (e.g. (Turian et al., 2010; Mikolov et al., 2013; Levy et al., 2015; Pennington et al., 2014)), resulting in impressive improvements for many NLP tasks.
More recent work has used neural network language models to learn context-dependent embeddings of words. We will refer to such context-dependent embeddings as token embeddings. For example, Peters et al. (2018) train a stacked BiLSTM language model, and these token embeddings have proved effective in many tasks. More such models will be discussed below.
For syntactic parsing, early connectionist approaches (Jain, 1991; Miikkulainen, 1993; Ho and Chan, 1999; Costa et al., 2001) had limited success. The first neural network models to achieve empirical success used a recurrent neural network to model the derivation structure of a traditional syntactic constituency parser (Henderson, 2003, 2004). The recurrent neural network learns to model the sequence of parser actions, estimating the probability of the next parser action given the history of previous parser actions. This allows the decoding algorithm from the traditional parsing model to be used to efficiently search the space of possi
ble parses. These models have also been applied to syntactic dependency parsing (Titov and Henderson, 2007b; Yazdani and Henderson, 2015) and joint syntactic-semantic dependency parsing (Henderson et al., 2013).
Crucially, these neural networks do not model the sequence of parser decisions as a flat sequence, but instead model the derivation structure it specifies. A derivation structure includes relationships for the inter-dependencies between nodes in the parse tree. The pattern of interconnections between hidden layers of the recurrent neural network (henceforth referred to as the model structure) is designed to follow locality in this derivation structure, thereby giving the neural network a linguistically appropriate inductive bias. More recently, Dyer et al. (2015) provide a more direct relationship between the derivation structure and the model structure with their StackLSTM parsing model.
In all these models, the use of recurrent neural networks allows arbitrarily large parse structures to be modelled without making any hard independence assumptions, in contrast to non-neural statistical models. Feed-forward neural networks have also been applied to modelling the derivation structure (Chen and Manning, 2014), but the accuracy is worse than using recurrent models (see Table 1), presumably because such models suffer from the need to make hard independence assumptions.
Representing the parse tree as a derivation sequence, rather than a derivation structure, makes it possible to define syntactic parsing as a sequence-to-sequence problem, mapping the sentence to its parse sequence. If a neural network architecture for modelling sequences (called seq2seq models) can perform well at this task, then maybe the structured linguistic representations of natural language are not necessary (contrary to Fodor and Pylyshyn (1988)), not even to predict those structures. Vinyals et al. (2015) report very poor results for seq2seq models when trained on the standard dataset, but good results when trained on very large automatically-parsed corpora (see Table 1 semisup). They only achieve good results with the limited standard dataset by adding attention, which we will argue below makes the model no longer a seq2seq model. This indicates that structured representations really do capture important generalisations about language. $^{2}$
See (Collobert and Weston, 2008; Collobert et al., 2011) for an earlier related line of work.
6297
In contrast to seq2seq models, there have also been neural network models of parsing which directly represent linguistic structure, rather than just derivation structure, giving them induced vector representations which map one-to-one with the entities in the linguistic representation. Typically, a recursive neural network is used to compute embeddings of syntactic constituents bottom-up. Dyer et al. (2015) showed improvements by adding these representations to a model of the derivation structure. Socher et al. (2013a) only modelled the linguistic structure, making it difficult to do decoding efficiently. But the resulting induced constituent embeddings have a clear linguistic interpretation, making it easier to use them within other tasks, such as sentiment analysis (Socher et al., 2013b). Similarly, models based on Graph Convolutional Networks have induced embeddings with clear linguistic interpretations within pre-defined model structures (e.g. (Marcheggiani and Titov, 2017; Marcheggiani et al., 2018)).
All these results demonstrate the incredible effectiveness of inducing vector-space representations with neural networks, relieving us from the need to do feature engineering. But neural networks do not relieve us of the need to understand the nature of language when designing our models. Instead of feature engineering, these results show that the best accuracy is achieved by engineering the inductive bias of deep learning models through their model structure. By designing a hand-coded model structure which reflects the linguistic structure, locality in the model structure can reflect locality in the linguistic structure. The neural network then induces features of the entities in this model structure.
4 Inducing Relations between Entities
With the introduction of attention-based models, the model structure can now be learned. By choosing the nodes to be linguistically-motivated entities, learning the model structure in effect learns the statistical inter-dependencies between entities, which is what we have been referring to as relations.
4.1 Attention-Based Models and Variable Binding
The first proposal of an attention-based neural model learned a soft alignment between the target and source words in neural machine translation (NMT) (Bahdanau et al., 2015). The model structure of the source sentence encoder and the model
structure of the target sentence decoder are both flat sequences, but when each target word is generated, it computes attention weights over all source words. These attention weights directly express how target words are correlated with source words, and in this sense can be seen as a soft version of the alignment structure. In traditional statistical machine translation, this alignment structure is determined with a separate alignment algorithm, and then frozen while training the model. In contrast, the attention-based NMT model learns the alignment structure jointly with learning the encoder and decoder, inside the deep learning architecture (Bahdanau et al., 2015).
This attention-based approach to NMT was also applied to mapping a sentence to its syntactic parse (Vinyals et al., 2015). The attention function learns the structure of the relationship between the sentence and its syntactic derivation sequence, but does not have any representation of the structure of the syntactic derivation itself. Empirical results are much better than their seq2seq model (Vinyals et al., 2015), but not as good as models which explicitly model both structures (see Table 1).
The change from the sequential LSTM decoders of previous NMT models to LSTM decoders with attention seems like a simple addition, but it fundamentally changes the kinds of generalisations which the model is able to learn. At each step in decoding, the state of a sequential LSTM model is a single vector, whereas adding attention means that the state needs to include the unboundedly large set of vectors being attended to. This use of an unbounded state is more similar to the above models with predefined model structure, where an unboundedly large stack is needed to specify the parser state. This change in representation leads to a profound change in the generalisations which can be learned. Parameterised rules which are learned when paying attention to one of these vectors (in the set or in the stack) automatically generalise to the other vectors. In other words, attention-based models have variable binding, which sequential LSTMs do not. Each vector represents the features for one entity, multiple entities can be kept in memory at the same time, and rules generalise across these entities. In this sense it is wrong to refer to attention-based models as sequence models; they are in fact induced-structure models. We will expand on this perspective in the rest of this section.
6298
4.2 Transformer and Systematicity
The generality of attention as a structure-induction method soon became apparent, culminated in the development of the Transformer architecture (Vaswani et al., 2017). Transformer has multiple stacked layers of self-attention (attention to the other words in the same sequence), interleaved with nonlinear functions applied to individual vectors. Each attention layer has multiple attention heads, allowing each head to learn a different type of relation. A Transformer-encoder has one column of stacked vectors for each position in the input sequence, and the model parameters are shared across positions. A Transformer-decoder adds attention over an encoded text, and predicts words one at a time after encoding the prefix of previously generated words.