Metagross: Meta Gated Recursive Controller Units for Sequence Modeling

1 INTRODUCTION . Sequences are fundamentally native to the world we live in , i.e. , language , logic , music and time are all well expressed in sequential form . To this end , the design of effective and powerful sequential inductive biases has far-reaching benefits across many applications . Across many of these domains , e.g. , natural language processing or speech , the sequence encoder lives at the heart of many powerful state-of-the-art model architectures . Models based on the notion of recurrence have enjoyed pervasive impact across many applications . In particular , the best recurrent models operate with gating functions that not only ameliorate vanishing gradient issues but also enjoy fine-grain control over temporal compositionality ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ) . Specifically , these gating functions are typically static and trained via an alternate transformation over the original input . In this paper , we propose a new sequence model that recursively parameterizes the recurrent unit . More concretely , the gating functions of our model are now parameterized repeatedly by instances of itself which imbues our model with the ability to reason deeply1 and recursively about certain inputs . To achieve the latter , we propose a soft dynamic recursion mechanism , which softly learns the depth of recursive parameterization at a per-token basis . Our formulation can be interpreted as a form of meta-gating since temporal compositionality is now being meta-controlled at various levels of abstractions . Our proposed method , Meta Gated Recursive Controller Units ( METAGROSS ) , marries the benefits of recursive reasoning with recurrent models . Notably , we postulate that this formulation brings about benefits pertaining to modeling data that is instrinsically hierarchical ( recursive ) in nature , e.g. , natural language , music and logic , an increasingly prosperous and emerging area of research ( Shen et al. , 2018 ; Wang et al. , 2019 ; Choi et al. , 2018 ) . While the notion of recursive neural networks is not new , our work is neither concerned with syntax-guided composition ( Tai et al. , 2015 ; Socher et al. , 2013 ; Dyer et al. , 2016 ) nor unsupervised grammar induction ( Shen et al. , 2017 ; Choi et al. , 2018 ; Havrylov et al. , 2019 ; Yogatama et al. , 2016 ) . Instead , our work is a propulsion on a different frontier , i.e. , learning recursively parameterized models which bears a totally different meaning . 1This is meant literally . Overall , the key contributions of this work are as follows : • We propose a new sequence model . Our model is distinctly characterized by recursive parameterization of recurrent gates , i.e. , compositional flow is controlled by instances of itself , á la repeatedly and recursively . We propose a soft dynamic recursion mechanism that dynamically and softly learns the recursive depth of the model at a token-level . • We propose a non-autoregressive parallel variation of METAGROSS , that when equipped with the standard Transformer model ( Vaswani et al. , 2017 ) , leads to gains in performance . • We evaluate our proposed method on a potpourri of sequence modeling tasks , i.e. , logical recursive tasks ( sorting , tree traversal , logical inference ) , pixel-wise sequential image classification , semantic parsing , neural machine translation and polyphonic music modeling . METAGROSS achieves state-of-the-art performance ( or close ) on all tasks . 2 META GATED RECURSIVE CONTROLLERS ( METAGROSS ) . This section introduces our proposed model . METAGROSS is fundamentally a recurrent model . Our proposed model accepts a sequence of vectors X ∈ R ` ×d as input . The main unit of the Metagross unit ht = Metagrossn ( xt , ht−1 ) is defined as follows : fnt = σs ( αt Metagrossn+1 ( xt , ht−1 ) + ( 1− αt ) FFn ( xt , ht−1 ) ) ont = σs ( βt Metagrossn+1 ( xt , h n t−1 ) + ( 1− βt ) FOn ( xt , ht−1 ) ) znt = σr ( F Z n ( xt , ht ) ) cnt = ( 1− ft ) ht−1 + ( ft ) zt hnt = ot ct hnt = ht + xt where σr is a nonlinear activation such as tanh . σs is the sigmoid activation function . In a nutshell , the Metagross unit recursively calls itself until a max depth L is hit . When n = L , ft and ot are parameterized by : fLt = σs ( F F L ( xt , ht−1 ) ) oLt = σs ( F O L ( xt , ht−1 ) ) where fLt , o L t is the forget and output gate of METAGROSS at time step t while at the maximum depth L. We also include an optional residual connection hnt = ht + xt to facilitate gradient flow down the recursive parameterization of METAGROSS . 2.1 SOFT DYNAMIC RECURSION . We propose learning the depth of recursion in a data-driven fashion . To learn αt , βt , we use the following : αt = Fα ( xt ) and βt = Fβ ( xt ) where F∗ ( xt ) = Wxt + b is a simple linear transformation layer applied to sequence X across the temporal dimension . Intuitively , α , β control the extent of recursion , enabling a soft depth pertaining to the hierarchical parameterization . Alternatively , we may also consider a static variation where : αt = Fα ( ∑̀ t=0 xt ) and βt = Fβ ( ∑̀ t=0 xt ) where the same value of α , β is computed based on global information from the entire sequence . Note that this strictly can not be used for autoregressive decoding . Finally , we note that it is also possible to assign α ∈ R , β ∈ R to be trainable scalar parameters . 2.2 LEVEL-WISE PARAMETERIZATION . Intuitively , F ∗n∀∗ ∈ F , O , Z are level-wise parameters of METAGROSS . We parameterize Fn with either level-wise RNN units or simple linear transformations . F ∗n ( xt ) = RNN ∗ n ( xt , h n t−1 ) or F ∗ n ( xt ) =W ∗ nxt + b ∗ n Overall , METAGROSS is agnostic to the choice of F ∗n ( xt ) and even the RNN unit . Note that for RNN , the hidden states are initialized with zero and each level uses a new initial hidden state . 2.3 PARALLEL NON-AUTOREGRESSIVE VARIATION . We postulate that METAGROSS can also be useful as a non-autoregressive parallel model . This can be interpreted as a form of recursive feed-forward layer that is used in place of recurrent METAGROSS for speed benefits . In early experiments , we find this a useful enhancement to state-of-theart Transformer ( Vaswani et al. , 2017 ) models . The non-autoregressive variant of METAGROSS is written as follows : fnt = σs ( αt Metagrossn+1 ( xt ) + ( 1− αt ) FFn ( xt ) ) ont = σs ( βt Metagrossn+1 ( xt ) + ( 1− βt ) FOn ( xt ) ) znt = σr ( F Z n ( xt ) ) hnt = ( f n t xt ) + ( ot znt ) hnt = ht + xt More concretely , we dispense with the reliance on the previous hidden state . This can be used in place of any position-wise feed-forward layer . In this case , note that F ∗n ( xt ) are typically positionwise functions as well . 3 EXPERIMENTS . We conduct experiments on a suite of diagnostic synthetic tasks and real world tasks . 3.1 LOGIC AND RECURSIVE TASKS . We evaluate our model on three diagnostic logical tasks as follows : • Task 1 ( SORT SEQUENCES ) - The input to the model is a sequence of integers . The correct output is the sorted sequence of integers . Since mapping sorted inputs to outputs can be implemented in a recursive fashion , we evaluate our model ’ s ability to better model recursively structured sequence data . Example input output pair would be 9 , 1 , 10 , 5 , 3→ 1 , 3 , 5 , 9 , 10 . • Task 2 ( TREE TRAVERSAL ) - We construct a binary tree of maximum depth N . The goal is to generate the postorder tree traversal given the inorder and preorder traversal of the tree . Note that this is known to arrive at only one unique solution . The constructed trees have random sparsity where we assign a probability p of growing the tree up to maximum depth N . Hence , the trees can be of varying depths2 . This requires inferring hierarchical structure and long-term reasoning across sequences . We concatenate the postorder and inorder sequences , delimiteted by a special token . We evaluate on n ∈ { 3 , 4 , 5 , 8 , 10 } . For n = { 5 , 8 } , we ensure that each tree traversal has at least 10 tokens . For n = 10 , we ensure that each path has at least 15 tokens . Example input output pair would be 13 , 15 , 4 , 7 , 5 , X , 13 , 4 , 15 , 5 , 7→ 7 , 15 , 13 , 4 , 5 . • Task 3 ( LOGICAL INFERENCE ) - We use the standard logical inference dataset3 proposed in ( Bowman et al. , 2014 ) . This is a classification task in which the goal is to determine the semantic equivalence of two statements expressed with logic operators such as not , and , and or . The language vocabulary is of six words and three logic operators . As per prior work ( Shen et al. , 2018 ) , the model is trained on sequences with 6 or less operations and evaluated on sequences of 6 to 12 operations . For Task 1 and Task 2 , we frame these tasks as a Seq2Seq ( Sutskever et al. , 2014 ) task and evaluate models on exact match accuracy and perplexity ( P ) metrics . We use a standard encoder-decoder architecture with attention ( Bahdanau et al. , 2014 ) . We vary the encoder module with BiLSTMs , Stacked BiLSTMs ( 3 layers ) and Ordered Neuron LSTMs ( Shen et al. , 2018 ) . For Task 3 ( logical inference ) , we use the common setting in other published works . Results on Sorting and Tree Traversal Table 1 reports our results on the Sorting and Tree Traversal task . All models solve the task with n = 3 . However , the task gets increasingly harder with a greater maximum possible length and largely still remains a challenge for neural models today . The relative performance of METAGROSS is on a whole better than any of the baselines , especially pertaining to perplexity . We also found that S-BiLSTMs are always better than LSTMs on this task and Ordered LSTMs are slightly worst than vanilla BiLSTMs . However , on sorting , ON-LSTMs are much better than standard BiLSTMs and S-BiLSTMs . # Operations Model 7 8 9 10 11 12 Tree-LSTM† ( Tai et al. , 2015 ) 93 90 87 89 86 87 LSTM ( Bowman et al. , 2014 ) 88 85 80 78 71 69 RRNet ( Jacob et al. , 2018 ) 84 81 78 74 72 71 ON-LSTM ( Shen et al. , 2018 ) 91 87 86 81 78 76 METAGROSS 97 95 93 92 90 88 Table 2 : Experimental results on Logical Inference task . † denotes models with access to ground truth syntax . Results reported from ( Shen et al. , 2018 ) . METAGROSS achieves state-of-the-art performance . Results on Logical Inference Table 2 reports our results on logical inference task . We compare with mainly other published work . METAGROSS is a strong and competitive model on this task , outperforming ON-LSTM by a wide margin ( +12 % on the longest nunber of operations ) . Performance of our model also exceeds Tree-LSTM , which has access to ground truth syntax . Our model achieves state-of-the-art performance on this dataset even when considering models with access to syntactic information . 2Note that all our models solves the problem entirely when the tree is fixed and full . Hence , random trees provide a necessary challenge . 33https : //github.com/sleepinyourhat/vector-entailment .

Summary: The authors propose a variant of gating functions for recurrent neural networks and feed-forward layers of Transformer and apply it to variety of tasks including toy tasks such as sorting, tree traversal and more realistic tasks such as machine translation. The gating function is applied recursively for N number of steps and depth of recursion is learned softly in data-driven function. Authors show similar or slightly better performance of their approach when applied to LSTM and Transformer compared to vanilla LSTM and Transformer.