Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

1 INTRODUCTION . Natural language has a sequential overt form as spoken and written , but the underlying structure of language is not strictly sequential . This structure is usually tree-like . Linguists agree on a set of rules , or syntax , that determine this structure ( Chomsky , 1956 ; 1965 ; Sandra & Taft , 2014 ) and dictate how single words compose to form meaningful larger units , also called “ constituents ” ( Koopman et al. , 2013 ) . The human brain can also implicitly acquire the latent structure of language ( Dehaene et al. , 2015 ) : during language acquisition , children are not given annotated parse trees . This observation brings more interest in latent structure induction with artificial neural network approaches , which are inspired by information processing and communication patterns in biological nervous systems . From a practical point of view , integrating a tree structure into a neural network language model may be important for multiple reasons : ( i ) to obtain a hierarchical representation with increasing levels of abstraction , a key feature of deep neural networks ( Bengio et al. , 2009 ; LeCun et al. , 2015 ; Schmidhuber , 2015 ) ; ( ii ) to model the compositional effects of language ( Koopman et al. , 2013 ; Socher et al. , 2013 ) and help with the long-term dependency problem ( Bengio et al. , 2009 ; Tai et al. , 2015 ) by providing shortcuts for gradient backpropagation ( Chung et al. , 2016 ) ; ( iii ) to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data . The study of deep neural network techniques that can infer and use tree structures to form better representations of natural language sentences has received a great deal of attention in recent ∗Equal contribution . { yi-kang.shen , jing.shan.shawn.tan } @ umontreal.ca . 1The code can be found at https : //github.com/yikangshen/Ordered-Neurons . years ( Bowman et al. , 2016 ; Yogatama et al. , 2016 ; Shen et al. , 2017 ; Jacob et al. , 2018 ; Choi et al. , 2018 ; Williams et al. , 2018 ; Shi et al. , 2018 ) . Given a sentence , one straightforward way of predicting the corresponding latent tree structure is through a supervised syntactic parser . Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics ( Socher et al. , 2013 ; Bowman et al. , 2015 ) , or even to help next word prediction given previous words ( Wu et al. , 2017 ) . However , supervised parsers are limiting for several reasons : i ) few languages have comprehensive annotated data for supervised parser training ; ii ) in some domains , syntax rules tend to be broken ( e.g . in tweets ) ; and iii ) languages change over time with use , so syntax rules may evolve . On the other hand , grammar induction , defined as the task of learning the syntactic structure from raw corpora without access to expert-labeled data , remains an open problem . Many such recent attempts suffer from inducing a trivial structure ( e.g. , a left-branching or right-branching tree ( Williams et al. , 2018 ) ) , or encounter difficulties in training caused by learning branching policies with Reinforcement Learning ( RL ) ( Yogatama et al. , 2016 ) . Furthermore , some methods are relatively complex to implement and train , like the PRPN model proposed in Shen et al . ( 2017 ) . Recurrent neural networks ( RNNs ) have proven highly effective at the task of language modeling ( Merity et al. , 2017 ; Melis et al. , 2017 ) . RNNs explicitly impose a chain structure on the data . This assumption may seem at odds with the latent non-sequential structure of language and may pose several difficulties for the processing of natural language data with deep learning methods , giving rise to problems such as capturing long-term dependencies ( Bengio et al. , 2009 ) , achieving good generalization ( Bowman et al. , 2015 ) , handling negation ( Socher et al. , 2013 ) , etc . Meanwhile , some evidence exists that LSTMs with sufficient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly , as shown by Gulordava et al . ( 2018 ) ; Kuncoro et al . ( 2018 ) and very recently by Lakretz et al . ( 2019 ) . We believe that the following question remains : Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures ? In this work , we introduce ordered neurons , a new inductive bias for recurrent neural networks . This inductive bias promotes differentiation of the life cycle of information stored inside each neuron : high-ranking neurons will store long-term information which is kept for a large number of steps , while low-ranking neurons will store short-term information that can be rapidly forgotten . To avoid a strict division between high-ranking and low-ranking neurons , we propose a new activation function , the cumulative softmax , or cumax ( ) , to actively allocate neurons to store long/short-term information . We use the cumax ( ) function to produce a vector of master input and forget gates ensuring that when a given neuron is updated ( erased ) , all of the neurons that follow it in the ordering are also updated ( erased ) . Based on the cumax ( ) and the LSTM architecture , we have designed a new model , ON-LSTM , that is biased towards performing tree-like composition operations . Our model achieves good performance on four tasks : language modeling , unsupervised constituency parsing , targeted syntactic evaluation ( Marvin & Linzen , 2018 ) and logical inference ( Bowman et al. , 2015 ) . The result on unsupervised constituency parsing suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts better than previously proposed models . The experiments also show that ON-LSTM performs better than standard LSTM models in tasks requiring capturing long-term dependencies and achieves better generalization to longer sequences . 2 RELATED WORK . There has been prior work leveraging tree structures for natural language tasks in the literature . Socher et al . ( 2010 ) ; Alvarez-Melis & Jaakkola ( 2016 ) ; Zhou et al . ( 2017 ) ; Zhang et al . ( 2015 ) use supervised learning on expert-labeled treebanks for predicting parse trees . Socher et al . ( 2013 ) and Tai et al . ( 2015 ) explicitly model the tree-structure using parsing information from an external parser . Later , Bowman et al . ( 2016 ) exploited guidance from a supervised parser ( Klein & Manning , 2003 ) in order to train a stack-augmented neural network . Theoretically , RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars ( Gers & Schmidhuber , 2001 ) . However , recent results suggest that introducing structure information into LSTMs is beneficial . Kuncoro et al . ( 2018 ) showed that RNNGs ( Dyer et al. , 2016 ) , which have an explicit bias to model the syntactic structures , outperform LSTMs on the subject-verb agreement task ( Linzen et al. , 2016 ) . In our paper , we run a more extensive suite of grammatical tests recently provided by Marvin & Linzen ( 2018 ) . Bowman et al . ( 2014 ; 2015 ) also demonstrate that tree-structured models are more effective for downstream tasks whose data was generated by recursive programs . Interestingly , Shi et al . ( 2018 ) suggests that while the prescribed grammar tree may not be ideal , some sort of hierarchical structure , perhaps task dependent , might help . However , the problem of efficiently inferring such structures from observed data remains an open question . The task of learning the underlying grammar from data is known as grammar induction ( Chen , 1995 ; Cohen et al. , 2011 ) . Early work incorporated syntactic structure in the context of language modeling ( Roark , 2001 ; Charniak , 2001 ; Chelba & Jelinek , 2000 ) . More recently , there have been attempts at incorporating some structure for downstream tasks using neural models ( Grefenstette et al. , 2015 ; Sun et al. , 2017 ; Joulin & Mikolov , 2015 ) . Generally , these works augment a main recurrent model with a stack and focus on solving algorithmic tasks . Yogatama et al . ( 2018 ) focus on language modeling and syntactic evaluation tasks ( Linzen et al. , 2016 ) but they do not show the extent to which the structure learnt by the model align with gold-standard parse trees . Shen et al . ( 2017 ) introduced the Parsing-Reading-Predict Networks ( PRPN ) model , which attempts to perform parsing by solving a language modeling task . The model uses self-attention to compose previous states , where the range of attention is controlled by a learnt “ syntactic distance ” . The authors show that this value corresponds to the depth of the parse tree . However , the added complexity in using the PRPN model makes it unwieldy in practice . Another possible solution is to develop models with varying time-scales of recurrence as a way of capturing this hierarchy . El Hihi & Bengio ( 1996 ) ; Schmidhuber ( 1991 ) ; Lin et al . ( 1998 ) describe models that capture hierarchies at pre-determined time-scales . More recently , Koutnik et al . ( 2014 ) proposed Clockwork RNN , which segments the hidden state of a RNN by updating at different time-scales . These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data . Chung et al . ( 2016 ) proposed a method that , unlike the Clockwork RNN , would learn a multi-scale hierarchical recurrence . However , the model still has a pre-determined depth to the hierarchy , depending on the number of layers . Our work is more closely related to Rippel et al . ( 2014 ) , which propose to induce a hierarchy in the representation units by applying “ nested ” dropout masks : units are not dropped independently at random but whenever a unit is dropped , all the units that follow in the ordering are also dropped . Our work can be seen as a soft relaxation of the dropout by means of the proposed cumax ( ) activation . Moreover , we propose to condition the update masks on the particular input and apply our overall model to sequential data . Therefore , our model can adapt the structure to the observed data , while both Clockwork RNN and nested dropout impose a predefined hierarchy to hidden representations . 3 ORDERED NEURONS . Given a sequence of tokens S = ( x1 , . . . , xT ) and its corresponding constituency tree ( Figure 2 ( a ) ) , our goal is to infer the unobserved tree structure while processing the observed sequence , i.e . while computing the hidden state ht for each time step t. At each time step , ht would ideally contain a information about all the nodes on the path between the current leaf node xt and the root S. In Figure 2 ( c ) , we illustrate how ht would contain information about all the constituents that include the current token xt even if those are only partially observed . This intuition suggests that each node in the tree can be represented by a set of neurons in the hidden states . However , while the dimensionality of the hidden state is fixed in advance , the length of the path connecting the leaf to the root of the tree may be different across different time steps and sentences . Therefore , a desiderata for the model is to dynamically reallocate the dimensions of the hidden state to each node . Given these requirements , we introduce ordered neurons , an inductive bias that forces neurons to represent information at different time-scales . In our model , high-ranking neurons contain long-term or global information that will last anywhere from several time steps to the entire sentence , representing nodes near the root of the tree . Low-ranking neurons encode short-term or local information that only last one or a few time steps , representing smaller constituents , as shown in Figure 2 ( b ) . The differentiation between high-ranking and low-ranking neurons is learnt in a completely data-driven fashion by controlling the update frequency of single neurons : to erase ( or update ) high-ranking neurons , the model should first erase ( or update ) all lower-ranking neurons . In other words , some neurons always update more ( or less ) frequently than the others , and that order is pre-determined as part of the model architecture .

Summary: Language is hierarchically structured: smaller units (e.g., noun phrases) are nested within larger units (e.g., clauses). This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the different units of an LSTM can learn to track information at different time scales, the standard architecture does not impose this sort of strict hierarchy. This paper proposes to add this constraint to the system by ordering the units; a vector of "master" input and forget gates ensures that when a given unit is reset all of the units that follow it in the ordering are also reset.