Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

1 INTRODUCTION . Natural language has a sequential overt form as spoken and written , but the underlying structure of language is not strictly sequential . This structure is usually tree-like . Linguists agree on a set of rules , or syntax , that determine this structure ( Chomsky , 1956 ; 1965 ; Sandra & Taft , 2014 ) and dictate how single words compose to form meaningful larger units , also called “ constituents ” ( Koopman et al. , 2013 ) . The human brain can also implicitly acquire the latent structure of language ( Dehaene et al. , 2015 ) : during language acquisition , children are not given annotated parse trees . This observation brings more interest in latent structure induction with artificial neural network approaches , which are inspired by information processing and communication patterns in biological nervous systems . From a practical point of view , integrating a tree structure into a neural network language model may be important for multiple reasons : ( i ) to obtain a hierarchical representation with increasing levels of abstraction , a key feature of deep neural networks ( Bengio et al. , 2009 ; LeCun et al. , 2015 ; Schmidhuber , 2015 ) ; ( ii ) to model the compositional effects of language ( Koopman et al. , 2013 ; Socher et al. , 2013 ) and help with the long-term dependency problem ( Bengio et al. , 2009 ; Tai et al. , 2015 ) by providing shortcuts for gradient backpropagation ( Chung et al. , 2016 ) ; ( iii ) to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data . The study of deep neural network techniques that can infer and use tree structures to form better representations of natural language sentences has received a great deal of attention in recent ∗Equal contribution . { yi-kang.shen , jing.shan.shawn.tan } @ umontreal.ca . 1The code can be found at https : //github.com/yikangshen/Ordered-Neurons . years ( Bowman et al. , 2016 ; Yogatama et al. , 2016 ; Shen et al. , 2017 ; Jacob et al. , 2018 ; Choi et al. , 2018 ; Williams et al. , 2018 ; Shi et al. , 2018 ) . Given a sentence , one straightforward way of predicting the corresponding latent tree structure is through a supervised syntactic parser . Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics ( Socher et al. , 2013 ; Bowman et al. , 2015 ) , or even to help next word prediction given previous words ( Wu et al. , 2017 ) . However , supervised parsers are limiting for several reasons : i ) few languages have comprehensive annotated data for supervised parser training ; ii ) in some domains , syntax rules tend to be broken ( e.g . in tweets ) ; and iii ) languages change over time with use , so syntax rules may evolve . On the other hand , grammar induction , defined as the task of learning the syntactic structure from raw corpora without access to expert-labeled data , remains an open problem . Many such recent attempts suffer from inducing a trivial structure ( e.g. , a left-branching or right-branching tree ( Williams et al. , 2018 ) ) , or encounter difficulties in training caused by learning branching policies with Reinforcement Learning ( RL ) ( Yogatama et al. , 2016 ) . Furthermore , some methods are relatively complex to implement and train , like the PRPN model proposed in Shen et al . ( 2017 ) . Recurrent neural networks ( RNNs ) have proven highly effective at the task of language modeling ( Merity et al. , 2017 ; Melis et al. , 2017 ) . RNNs explicitly impose a chain structure on the data . This assumption may seem at odds with the latent non-sequential structure of language and may pose several difficulties for the processing of natural language data with deep learning methods , giving rise to problems such as capturing long-term dependencies ( Bengio et al. , 2009 ) , achieving good generalization ( Bowman et al. , 2015 ) , handling negation ( Socher et al. , 2013 ) , etc . Meanwhile , some evidence exists that LSTMs with sufficient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly , as shown by Gulordava et al . ( 2018 ) ; Kuncoro et al . ( 2018 ) and very recently by Lakretz et al . ( 2019 ) . We believe that the following question remains : Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures ? In this work , we introduce ordered neurons , a new inductive bias for recurrent neural networks . This inductive bias promotes differentiation of the life cycle of information stored inside each neuron : high-ranking neurons will store long-term information which is kept for a large number of steps , while low-ranking neurons will store short-term information that can be rapidly forgotten . To avoid a strict division between high-ranking and low-ranking neurons , we propose a new activation function , the cumulative softmax , or cumax ( ) , to actively allocate neurons to store long/short-term information . We use the cumax ( ) function to produce a vector of master input and forget gates ensuring that when a given neuron is updated ( erased ) , all of the neurons that follow it in the ordering are also updated ( erased ) . Based on the cumax ( ) and the LSTM architecture , we have designed a new model , ON-LSTM , that is biased towards performing tree-like composition operations . Our model achieves good performance on four tasks : language modeling , unsupervised constituency parsing , targeted syntactic evaluation ( Marvin & Linzen , 2018 ) and logical inference ( Bowman et al. , 2015 ) . The result on unsupervised constituency parsing suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts better than previously proposed models . The experiments also show that ON-LSTM performs better than standard LSTM models in tasks requiring capturing long-term dependencies and achieves better generalization to longer sequences . 2 RELATED WORK . There has been prior work leveraging tree structures for natural language tasks in the literature . Socher et al . ( 2010 ) ; Alvarez-Melis & Jaakkola ( 2016 ) ; Zhou et al . ( 2017 ) ; Zhang et al . ( 2015 ) use supervised learning on expert-labeled treebanks for predicting parse trees . Socher et al . ( 2013 ) and Tai et al . ( 2015 ) explicitly model the tree-structure using parsing information from an external parser . Later , Bowman et al . ( 2016 ) exploited guidance from a supervised parser ( Klein & Manning , 2003 ) in order to train a stack-augmented neural network . Theoretically , RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars ( Gers & Schmidhuber , 2001 ) . However , recent results suggest that introducing structure information into LSTMs is beneficial . Kuncoro et al . ( 2018 ) showed that RNNGs ( Dyer et al. , 2016 ) , which have an explicit bias to model the syntactic structures , outperform LSTMs on the subject-verb agreement task ( Linzen et al. , 2016 ) . In our paper , we run a more extensive suite of grammatical tests recently provided by Marvin & Linzen ( 2018 ) . Bowman et al . ( 2014 ; 2015 ) also demonstrate that tree-structured models are more effective for downstream tasks whose data was generated by recursive programs . Interestingly , Shi et al . ( 2018 ) suggests that while the prescribed grammar tree may not be ideal , some sort of hierarchical structure , perhaps task dependent , might help . However , the problem of efficiently inferring such structures from observed data remains an open question . The task of learning the underlying grammar from data is known as grammar induction ( Chen , 1995 ; Cohen et al. , 2011 ) . Early work incorporated syntactic structure in the context of language modeling ( Roark , 2001 ; Charniak , 2001 ; Chelba & Jelinek , 2000 ) . More recently , there have been attempts at incorporating some structure for downstream tasks using neural models ( Grefenstette et al. , 2015 ; Sun et al. , 2017 ; Joulin & Mikolov , 2015 ) . Generally , these works augment a main recurrent model with a stack and focus on solving algorithmic tasks . Yogatama et al . ( 2018 ) focus on language modeling and syntactic evaluation tasks ( Linzen et al. , 2016 ) but they do not show the extent to which the structure learnt by the model align with gold-standard parse trees . Shen et al . ( 2017 ) introduced the Parsing-Reading-Predict Networks ( PRPN ) model , which attempts to perform parsing by solving a language modeling task . The model uses self-attention to compose previous states , where the range of attention is controlled by a learnt “ syntactic distance ” . The authors show that this value corresponds to the depth of the parse tree . However , the added complexity in using the PRPN model makes it unwieldy in practice . Another possible solution is to develop models with varying time-scales of recurrence as a way of capturing this hierarchy . El Hihi & Bengio ( 1996 ) ; Schmidhuber ( 1991 ) ; Lin et al . ( 1998 ) describe models that capture hierarchies at pre-determined time-scales . More recently , Koutnik et al . ( 2014 ) proposed Clockwork RNN , which segments the hidden state of a RNN by updating at different time-scales . These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data . Chung et al . ( 2016 ) proposed a method that , unlike the Clockwork RNN , would learn a multi-scale hierarchical recurrence . However , the model still has a pre-determined depth to the hierarchy , depending on the number of layers . Our work is more closely related to Rippel et al . ( 2014 ) , which propose to induce a hierarchy in the representation units by applying “ nested ” dropout masks : units are not dropped independently at random but whenever a unit is dropped , all the units that follow in the ordering are also dropped . Our work can be seen as a soft relaxation of the dropout by means of the proposed cumax ( ) activation . Moreover , we propose to condition the update masks on the particular input and apply our overall model to sequential data . Therefore , our model can adapt the structure to the observed data , while both Clockwork RNN and nested dropout impose a predefined hierarchy to hidden representations . 3 ORDERED NEURONS . Given a sequence of tokens S = ( x1 , . . . , xT ) and its corresponding constituency tree ( Figure 2 ( a ) ) , our goal is to infer the unobserved tree structure while processing the observed sequence , i.e . while computing the hidden state ht for each time step t. At each time step , ht would ideally contain a information about all the nodes on the path between the current leaf node xt and the root S. In Figure 2 ( c ) , we illustrate how ht would contain information about all the constituents that include the current token xt even if those are only partially observed . This intuition suggests that each node in the tree can be represented by a set of neurons in the hidden states . However , while the dimensionality of the hidden state is fixed in advance , the length of the path connecting the leaf to the root of the tree may be different across different time steps and sentences . Therefore , a desiderata for the model is to dynamically reallocate the dimensions of the hidden state to each node . Given these requirements , we introduce ordered neurons , an inductive bias that forces neurons to represent information at different time-scales . In our model , high-ranking neurons contain long-term or global information that will last anywhere from several time steps to the entire sentence , representing nodes near the root of the tree . Low-ranking neurons encode short-term or local information that only last one or a few time steps , representing smaller constituents , as shown in Figure 2 ( b ) . The differentiation between high-ranking and low-ranking neurons is learnt in a completely data-driven fashion by controlling the update frequency of single neurons : to erase ( or update ) high-ranking neurons , the model should first erase ( or update ) all lower-ranking neurons . In other words , some neurons always update more ( or less ) frequently than the others , and that order is pre-determined as part of the model architecture .

Summary: The paper proposes a new RNN unit: ON-LSTM. The idea is to explicitly integrates the latent tree structure into recurrent models. Experiments are conducted to evaluate performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. Good results on unsupervised parsing show that the model learns something close to human judgments of the sentence parses.