Metagross: Meta Gated Recursive Controller Units for Sequence Modeling
Authors: Yi Tay
Yikang Shen
Alvin Chan
Yew Soon Ong
Venue:  (2020)
Keywords: Deep Learning
Natural Language Processing
Recurrent Neural Networks

Abstract: This paper proposes Metagross (Meta Gated Recursive Controller), a new neural sequence modeling unit. Our proposed unit is characterized by recursive parameterization of its gating functions, i.e., gating mechanisms of Metagross are controlled by instances of itself, which are repeatedly called in a recursive fashion. This can be interpreted as a form of meta-gating and recursively parameterizing a recurrent model. We postulate that our proposed inductive bias provides modeling benefits pertaining to learning with inherently hierarchically-structured sequence data (e.g., language, logical or music tasks). To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), sequential pixel-by-pixel classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving state-of-the-art (or close) performance on all tasks.

Under review as a conference paper at ICLR 2020
METAGROSS: META GATED RECURSIVE CONTROLLER UNITS FOR SEQUENCE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes METAGROSS (Meta Gated Recursive Controller), new neu- ral sequence modeling unit. Our proposed unit characterized by recursive pa- rameterization of its gating functions, i.e., gating mechanisms of METAGROSS are controlled by instances of itself, which are repeatedly called in a recursive fashion. This can be interpreted as form of meta-gating and recursively parameterizing a recurrent model. We postulate that our proposed inductive bias provides modeling sequence data (c.g., language, logical or music tasks). To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), se- quential pixel-by-pixx. classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving state-of-the-art (or close) performance all tasks.
benefits pertaining to learning with inherently hicrarchically-sttuttii
INTRODUCTION
Sequences are fundamentally native to the world we live in, i.e., language, logic, music and time are all well expressed in sequential form. this end, the design of effective and powerful sequential inductive biases has far-reaching benefits across many applications. Across many of these domains, e.g., natural language processing or speech, the sequence encoder lives at the heart of many powerful tfate-of-the-ar model architectures.
Models based on the notion of recurrence have enjoyed pervasive impact across many applications. In particular, the best recurrent models operate with gating functions that not only ameliorate van- ishing gradient issues but also enjoy fine-grain control over temporal compositionality (Hochreiter & Schmidhuber, 1997; Cho et al., 2014). Specifically, these gating functions typically static and trained via an alternate transformation over the original input.
sequence model that recursively parameterizes the recurrent unit. More concretely, the gating functions of our model are now parameterized repeatedly by instances of itself which imbues our model with the ability reason deeply' and recursively about certain inputs. To achieve the latter, we propose a soft dynamic recursion mechanism, which softly learns the depth of recursive parameterization a per-token basis. Our formulation can interpreted as a form of meta-gatinn meta-gating since temporal compositionality is now being meta-controlled at various levels of abstractions.
In this paper, propose
Our proposed method, Meta Gated Recursive Controller Units (METAGROSS), marries the benefits of recursive reasoning with recurrent models. Notably, we postulate that this formulation brings about benefits pertaining to modeling data that is instrinsically hierarchical (recursive) in nature, e.g., natural language, music and logic, an increasingly prosperous and emerging area of research (Shen et al., 2018; Wang et al., 2019; Choi et al., 2018). While the notion of recursive neural networks not new, our work is neither concerned with syntax- yynax-guiddd composition (Tai et al., 2015; Socher et al., 2013; Dyer al., 2016) nor unsupervised grammar induction (Shen et al., 2017; Choi et al., 2018; Havrylov et al., 2019; Yogatama et al., 2016). Instead, our work is a propulsion on a different frontier, i.e., learning recursively parameterized models which bears totally different meaning.
This is meant literally.

---

Under review as a conference paper at ICLR 2020
Overall, the key contributions of this work are as follows:
We propose sequence model. Our model is distinctly characterized by recursive parameterization of recurrent gates, i.e., compositional flow is controlled by instances of itself, á la repeatedly and recursively. We propose soft dynamic recursion mechanism that dynamically and softly learns the recursive depth of the model at a token-level.
We propose non-autoregressive parallel variation of METAGROSS, that when equipped with the standard Transformer model (Vaswani et al., 2017), leads gains in performance.
We evaluate our proposed method on a potpourri of sequence modeling tasks, i.c., logical recursive tasks (sorting. tree traversal, logical inference), pixel-wise sequential image clas- sification, semantic parsing, neural machine translation and polyphonic music modeling. METAGROSS achieves state-of-the-art performance (or close) on all tasks.
META GATED RECURSIVE CONTROLLERS (METAGROSS)
This section introduces our proposed model. METAGROSS fundamentally a recurrent model. Our proposed model accepts sequence of vectors X fxxd as input. The main unit of the Metagross unit hr Metagross, (Iz, 11-1) is defined as follows:
ft = a. (at Metagross,++14n ( 11 + (1 10--ay az) FR (0t, hz-1))
0r (3r Metagross, (oe, M1_1) + (1 Be) F (I h1-1))
2 (F (2t, ha))
& = (1 - -(t @ 14-1 + (ft) 2z
h 0r O
Ar = he + It
where o, is a nonlinear activation such as tanh. 0, the sigmoid activation function. In nutshell, the Metagross unit recursively calls itself until a max depth L is hit. When L, ft and or are parameterized by:
ft = o (Ie, he-1)) of- = o. FF (x1,h)-1)
where ft. of is the forget and output gate of METAGROSS at time step 1 while at the maximum depth T. We also include optional residual connection hü h4 sy to facilitate gradient flow down the recursive parameterization of METAGROSS.
2.1 SOFT DYNAMIC RECURSION
We propose learning the depth of recursion in a data-driven fashion. To learn o,, Sr we use the following:
(z)) and B (2t)
where F.(1)) Wx, + is simple linear transformation layer applied sequence across the temporal dimension. Intuitively, a, control the extent of recursion, enabling soft depth pertaining the hierarchical parameterization. Alternatively, we may also consider static variation where:
02
F
3i and B, FB
r4)
where the same value of a, 3 is computed based global information from the entire sequence. Note that this strictly cannot used for autoregressive decoding. Finally, we note that it is also possible to assign a E R, R to be trainable scalar parameters.
2

---

Under review as a conference paper at ICLR 2020
2.2 LEVEL-WISE PARAMETERIZATION
Z are level-wise parameters of METAGROSS. We parameterize P, with either level-wise RNN units simple linear transformations.
Intuitively, V E F, 0,
(ax) RNN" (zt,
(m:) W b
Overall, METAGROSS is agnostic to the choice of Fz (zz) and the RNN unit Note that for RNN, the hidden states are initialized with zero and each level uses a new initial hidden state.
Timestop
Timestep
Timestep b+2
Timestep ++
Gate Concrellers
Gute Controllers
Dynsmic Rocurive Depth
Controllass
Figure 1: Architecture of Proposed METAGROSS network with maximum depth N = 3. The extent of greyness illustrates effect from soft dynamic recursion.
2.3 PARALLEL NON-AUTOREGRESSIVE VARIATION
We postulate that METAGROSS can also be useful as a non-autoregressive parallel model. This can be interpreted as form of recursive feed-forward layer that is used in place of recurrent META- GROSS for speed benefits. In early experiments, we find this a useful enhancement to state-of-the- art Transformer (Vaswani et al., 2017) models. The non-autoregressive variant of METAGROSS is written as follows:
J = o. (ar Melagross,+ + (1 0z) Ff (xt)
0 = o, (8, Metagross, (z,) + (1 - 3.) (x,)
2 or (F)():)
h ( @ 2z) + (or 0 2i"
b, h4 +
More coneretely, dispense with the reliance the previous hidden state. This be used in place any position-wis wise feod-forward layer. this case, note that F (zx) are typically position- wise functions well.
EXPERIMENTS
We conduct experiments suite of diagnostic synthetic tasks and real world tasks.
3.1 LOGIC AND RECURSIVE TASKS
We evaluate model three diagnostic logical tasks follows:
Task 1 (SORT SEQUENCES) The input to the model is sequence of integers. The correct output is the sorted sequence of integers. Since mapping sorted inputs to outputs can im- plemented in recursive fashion, evaluate model's ability to better model recursively structured sequence data. Example input output pair would 9, 1, 10, 5.3 + 1,3, 5, 9, 10.
Task 2 (TREE TRAVERSAL)- We construct a binary tree of maximum depth N. The goal is to generate the postorder tree traversal given the inorerr and preorder traversal of the tree. Note that this is known to arrive at only one unique solution. The constructed trees have random sparsity where we assign probability p of growing the tree up maximum
3

---

Under review as a conference paper at ICLR 2020
depth N. Hence, the trees be of varying depths". This requires inferring hierarchi- cal structure and long-ten Long-term reasoning across sequences. We concatenate the postorder and inorder sequences, delimiteted by special token. We evaluate on {3, 4, 5, 10). For n. = (5.8), we ensure that each tree traversal has at least 10 tokens. For n = 10, ensure that each path has least 15 tokens. Example input output pair would be 13, 15, 4, 7, 5, X, 13, 4, 15, 5, 7 + 7, 15, 1,,
Task 3 (LOGICAL INFERENCE) We use the standard logical inference dataset proposed in (Bowman et al., 2014). This classification task in which the goal determine the semantic equivalence of two statements expressed with logic operators such as not, and, and or. The language vocabulary is of six words and three logic operators. As per prior work (Shen et al., 2018), the model is trained on sequences with 6 or less operations and evaluated on sequences of to 12 operations.
For Task and Task we frame these tasks as a Seq2Seq (Sutskever al., 2014) task and evaluate models on exact match accuracy and perplexity (P) metrics. We use a standard encoder-decoder architecture with attention (Bahdanau et al., 2014). We vary the encoder module with BTM,SMs, Stacked BiLSTMS (3 layers) and Ordered Neuron LSTMss (Shen et al., 2018). For Task 3 (logical inference), we use the common setting in other published works.
Results on Sorting and Tree Traversal Table reports our results on the Sorting and Tree Traver- sal task. All models solve the task with 3. However, the task gets increasingly harder with greater maximum possible length and largely still remains a challenge for neural models today. The relative performance of METAGROSS taining to perplexity. We also found that S-BiLSTMs are always better than LSTMs on this task and Ordered LSTMs are slightly worst than vanilla BiLSTMs. However, on sorting, ON-LSTMs are much better than standard BiLSTMs and S-BiLSTMs.
whole better than any of the baselines, especially per-
TREE TRAVERSAL
SORT
TL
5
T2
8
YL
10
5
N
10
Model
EM
P
EM
P
EM
EM
P
EM
P
EM
EM
P
BiLSTM
100
1.0
96.9
2.4
60.3
30.6
132
79.9
78.9
1.2
S-BiLSTM
100
1.0
98.0
1.0
63.4
2.5
5.9
99.9
2.8
225
83.4
1.2
88.0
1.1
ON-LSTM
100
1.0
81.0
1.4
55.7
2.8
5.5
52.3
2.7
173
90.8
1.1
87.4
1.1
METAGROSS
100
1.0
98.4
1.0
63.4
1.8
5.6
20.4
2.8
119
92.2
90.6
1.1
Table 1: Experimental Results on Tree Traversal and Sorting.
Results on Logical Inference Table 2 reports results on logical infer- ence task. We compare with mainly other published work. METAGROSS is a strong and competitive model on this task, outperforming ON-LSTM by wide margin (+12% on the longest nun- ber of operations). Performance of our model also exceeds Tree-LSTM, which has access to ground truth syntax. Our model achieves state-of-the-art perfor- mance on this dataset even when con- sidering models with access to syntactic information.
Operations
Model
7
8
11
Tree L.TM (Tai
al., 2015)
93
90
87
86
87
LSTM (Howman et al., 2014)
88
85
80
78
71
69
RRNet (Jacob al., 2018)
84
78
72
ON-LSTM (Shen al., 2018)
91
87
86
81
78
76
METAGROSS
97
95
93
92
90
88
Table 2: Experimental results Logical Inference task. t denotes models with access to ground truth syntax. Results reported from (Shen et al., 2018). METAGROSS achieves statc-of-the-e performance.
Note that our models solves the problem entirely when the tree is fixed full. Hence, random trees provide necessary challenge. "3htpps: / com eepi beepppooorattteettoooooooe ment..
4

---

Under review as a conference paper at ICLR 2020
3.2 PIXEL-WISE SEQUENTIAL IMAGE CLASSIFICATION
We evaluate our model on its ability to model and capture long-range dependencies. More specif- ically, the sequential pixel-wise image classification problem treats pixels in images sequences. We use the well-established pixel-wise MNIST and CIFAR-10 datasets. We use 3 layered META- GROSS of 128 hidden units each.
Results on Pixel-wise Image Classification Table 3 reports the results of METAGROSS
DilatedGRU (Chang al., 2017)
99.00
IndRNN (Li ct al., 2018a)
99.00
against other published works. Our method achieves state-of-the-art performance on the CIFAR-10 dataset, outperforming the recent
1-LSTM (Trinh
Trunsformer (Trinh et al., 2018)
98.90
62.20
TrellisNet (Bai al., 8h)
99.20
TrellisNet (Our run)
8.0M
97.59
55.83
Trellis Network (Bai et al., 2018b). On the other hand, results MNIST are reasonable, outper- forming a wide range of other published works. On top of that, method has 8 times less pa- rameters than Trellis network (Bai et al., 2018b) while achieving similar better performance. This ascertains that METAGROSS is a reason- ably competitive long-range sequence encoder.
METAGROSS
0.9M
99.04
73.01
METAGROSS*
0.9M
99.09
73.95
Table Experimental results (accuracy) on Pixel-wise Sequential Image Classification. We also trained the recent R-Adam optimizer (Liiu et al., 2019) which we found to have improved performance (results denoted with t).
3.3 SEMANTIC PARSING/ CODE GENERATION
We evaluate METAGROSS on semantic parsing (GEO, ATIS, JOBS) and code generation (DJANGO). We run our experiments the publicly released code of (Yin & Neubig, 2018), replacing the recurrent decoder with our METAGROSS decoder. Hyperparameter details followed the codebase of (Yin & Neubig, 2018) quite strictly.
Results on Semantic Parsing and Code Generation Table reports our experimental results on Semantic Pars- ing (GEO, ATIS, JOBS) and Code Gen- eration (DJANGO). We observe that TranX + METAGROSS outperforms all competitor approaches, achieving state- af-the-art performance. More impor- tantly, the performance gain over the base TranX method allows us to observe the ablative benefits of METAGROSS.
Sey2 (Dong & Lapula, 2016)
87.1
84.6
31.5
LPN (l.ngg al., 2016)
62.3
NMT (Neubig, 2015)
45.1
YN17 (Yin & Ncubig, 2017)
71.6
(Rabinovich et 2017)
85.7
ASN-Au (Rabinovich el al., 2017)
87.1
85.9
TrinX (Yin & Neabig, 2018)
88.2
86.2
72.7
TranX (Code reported)
88.6
87.7
90.0
77.2
TranX (Our Rus)
87.5
87.5
90.0
76.7
METAGROSS
88.6
90.7
Table 4: Experimental results on Semantic Pars-
ing and Code Generation.
3.4 NEURAL MACHINE TRANSLATION
We conduct experiments two IWSLT datasets which are collections derived from TED talks. Specifically, we compare on the IWSLT 2014 German-English and IWSLT 2015 English- Vietnamese datasets. We compare against suite of published results and strong baselines. For our method, we replaced the multi-head aggregation layer in the Transformer networks (Vaswani al., 2017) with a parallel autaaorrerressvvee adaptation of METAGROSS. The base models all linear layers. For our experiments, we use the standard implementation and hyperparameters in Ten- sor2Tensor (Vaswani al., 2018), using the small (S) and base (B) setting for Transformers. Model averaging is used and beam size of 8/4 and length penalty of 0.6 is adopted for De-En and En- Vi respectively. For our model, max depth is tuned amongst {1, 21. We also ensure to compare, in an ablative fashion, our own reported runs of the base Transformer models.
'https: thub. hwtp:hcc/eeeeeorfoooooooooeoeoem
5

---

Under review as a conference paper at ICLR 2020
Model
BLEU
MIXER (Ranzato et al., 2015)
21.83
AC+LL (Bahdanau et al., 2016)
28.53
Model
BLEU
NPMT (Huang al., 2017)
28.96
(Luong & Manning, 2015)
23.30
Dual Transfer (Wang al., 2018)
32.35
Att-Seq2Seq (Bahdanau al., 2014)
26.10
Transformer (Vaswani et al., 2017)
32.86
NPMT (Huang et al., 2017)
27.69
Layer-wise (Hc ct al., 2018)
35.07
NPMT LM (Huang al., 2017)
28.07
Transformer S (Our run)
34.68
Transformer B (Vaswani et al., 2017)
28.43
Transformer (Our run)
36.30
Transformer B METAGROSS
30.81
Transformer METAGROSS
35.15
Transformer + METAGROSS
37.09
Table 6: Experimental results on Neural Ma-
chine Translation IWSLT 2015 En-Vi.
Table 5: Experimental results on Neural Ma- chine Translation on IWSLT 2014 De-En.
Results on Neural Machine Translation Table 5 reports results on IWSLT 2014 de-en task. Our proposed model performs very competitively (37.09 BLEU), outperforming many well-established baselines. Our results also show that equipping Transformer models with METAGROSS can also lead to improvements
performance. Notably there +0.69 BLEU improvement on Trans- former Base and +0.42 BLEU improvement for Transformer Small. On the other hand, our method achieves 30.81 BLEU on the IWSLT 2015 En- Vi dataset, with +2338 improvement BLEU from the standard Transformer Base model.
3.5 POLYPHONIC MUSIC MODELING
We evaluate METAGROSS the polyphonic music modeling. We use three well-established datasets, namely Nottingham, JSB Chorales and Piano Midi BBoulanger-Lewandover The input the model are 88-bit sequences, each corresponding the 88 keys of the piano. The task evaluated on the Negative Log-likelihood (NLL). We compare with a wide range of published works (Chung al., 2014; Bai et al., 201 8a; Song et al., 2019)
et al., 2012).
Model
Nott
JSB
Piano
GRU (Chung et al.)
3.13
8.54
8.82
LSTM (Song et al.)
3.25
8.61
7.99
Results Music Modeling Table reports our scores on this task. METAGROSS achieves state- of-the-art performance the Nottingham and Pi- ano midi datasets, outperforming wide range of competitive models such as Gumbel Gate LSTMs (Li et al., 201 20)
G2-LSTM (Li al.)
3.21
8.67
8.18
B-LSTM (Song et al.)
3.16
8.30
7.55
TCN (Bai et al.)
3.07
8.10
TCN (our run)
2.95
8.13
7.53
METAGROSS
2.88
8.12
7.49
Table 7: Experimental Results (NLL) Poly- phonic Music Modeling.
3.6 ANALYSIS AND DISCUSSION
This section reports some analysis and discussion regarding the proposed model.
3.6.
EFFECT OF MAXIMUM DEPTH AND BASE UNIT
Unit
Mox
Base Model
DIANGO
Tree Traversal
2
Recurrent
Sorting
2
Recurrent
2
Lincar
88,40
77.36
Logical Inference
3
Recureett
4
3
Linear
Lineur
87.80
88.21
76,84
77.62
Pixel-wise Classilictiion
2
Recurrent
LSTM
86.61
78.33
Semantic Parsing
Lincar
LSTM
77.39
Code Generation
Recurrent
Machine Translation
Linear
Polyphonic Music
Linear
Table 8: Ablation studies on Semantic Parsing and Code Generation.
Table 9: Optimal Maximum Depth and base unit for different tasks.

---

Under review as a conference paper at ICLR 2020
Table 8 reports some ablation studies on the semantic parsing and code generation tasks. We observe that the base unit and optimal maximum depth used is task dependent. For ATIS dataset, using the linear transform the base unit performs the best. Conversely, the linear base unit performs worse than the recurrent base unit (LSTTM) on the DJANGO dataset.
On whole, we also observed this across other tasks, i.e., the base unit and maximum depth of METAGROSS a critical choice for most tasks. Table 9 reports the optimal max depth N and best base unit for each task.
3.6.2 ANALYSIS OF SOFT DYNAMIC RECURSION
Figure illustrates the depth gate values on CIFAR and MNIST datasets. These values reflect the and 3 values METAGROSS, signifying how the parameter tree being constructed during training. This is reflected I. and R in the figures representing left and right gates. Firstly, we observe that our model indeed builds data-specific parameterization of the network. This is denoted by how METAGROSS builds different trees for CIFAR and MNIST.
Gan
Maus
Figure 2: Depth Gates Initial (CIFAR)
Figure 3: Depth Gates at epoch 10 (CIFAR)
Figure
Depth Gates at Initial (MNIST)
Figure 5: Depth Gates at epoch 10 (MNIST)
Figure 6: Depth Gate Visualization on CIFAR and MNIST.
Secondly, we analyze the dynamic recursion depth with respect to time steps. The key ob- servation that all datasets have very diverse construction of recursive parameters. The re- cursive gates fluctuate aggressively on CI- FAR while remaining more stable Mu- sic modeling. Moreover, we found that the recursive gates remain totally constant MNIST. This demonstrates that our model has the ability to adjust the dynamic con- struction adaptively and can revert to static recursion over time if necessary. We find that compelling.
Figure 7: Dynamic Recursion CIFAR.
Figure 9: Dynamic Recursion MNIST.
Figure Dynamic Recursion on Music.
Though not depicted, we also found that the prohability of each node expanding to children has low vari- across batches in the same dataset.

---

Under review as a conference paper at ICLR 2020
The adaptive recursive depth is made more intriguing by observing how the recursive parameteriza- tion alters on CIFAR and Music datasets. From Figure 8 we observe that the structure of the network changes in rhythmic fashion, in line with intuition of musical data. When dealing with pixel information, the tree structure changes adaptively according to the more complex information pro- cessed by the network.
4 RELATED WORK
The study of effective inductive biases for sequential representation learning has been a prosperous research direction. This has spurred on research across multiple fronts, starting from gated recurrent models (Hochreiter & Schmidhuber. 1997; Cho et al., 2014), convolution (Bai et al., 2018a) 8a) to the recently popular self-attention based models (Vaswani et al., 2017).
The intrinsic hierarchical structure native to many forms of sequences have long fascinated and in- spired many researchers (Socher et al., 2013; Bowman et al., 2014; 2016; Dyer al., 2016). The study of recursive networks, popularized by (Socher et al., 2013) has provided foundation for learning syntax- guuided composition in language processing research. Along the same vein, (Tai et al., 2015) proposed Tree-LSTMS which guide LSTM composition with grammar. Recent at- tempts have been made to learn this process without guidance nor syntax-based supervision (Choi et al., 2018; Shen et al., 2017: Havrylov et al., 2019; Yogatama et al., 2016). Ordered Neuron LSTMs (Shen et al., 2018) proposed structured gating mechanisms, imbuing the recurrent unit with rre-structured inductive bias. (Tran al., 2018) shows that recurrence is important for model- ing hierarchical structure. Notably, learning hierachical representations across multiple time-scales (EI Hihi & Bengio, 1996; Schmidhuber, 1992; Koutnik al., 2014; Chung al., 2016; Hafner al., 2017) have also demonstrated reasonable success.
Learning an abstraction and controller over a base recurrent unit also another compelling direc- tion. First proposed by Fast Weights (Schmidhuber, 1992). several recent works explore this notion. HyperNetworks (Ha et al., 2016) learns to generate weights for another recurrent unit, i.e., form of relaxed weight sharing. On the other hand, RCRN (Tay et al., 2018) explicitly parameterizes the gates of RNN unit with other RNN units. Recent attempts speed up the recurrent unit are also reminiscent of this particular notion (Bradbury al., 2016; Lei et al., 2018).
The marriage of recursive and recurrent architectures is also notable. This direction is probably the closest relevance to our proposed method, although with vast differences. (Liu et al., 2014) proposed Recursive Recurrent Networks for machine translation which are concerned with the more traditional syntactic supervision concept of vanilla recursive nets. (Jacob et al., 2018) proposed RR- Net, which learns hierarchical structures on the fly. RR-Net proposes to learn to split or merge nodes at each time step, which makes it reminiscent of (Choi et al., 2018; Shen et al., 2018). (Alvarez- Melis & Jaakkola, 2016) proposed doubly recurrent decoders for tree-structured decoding. The core of their method is depth and breath-wise recurrence which is similar to our model. However, METAGROSS is concerned with learning gating controllers which is different from the objective of decoding trees.
Our work combines the idea of external meta-controllers (Schmidhuber, 1992; Ha et al., 2016; Tay al., 2018) with recursive architectures. In particular, our recursive parameterization is also form of dynamic memory which gives our model improved expressiveness in similar spirit to memory- augmented recurrent models (Santoro et al., 2018; Graves et al., 2014; Tran al., 2016).
5 CONCLUSION
We proposed Meta Gated Recursive Controller Units (METAGROSS) sequence model characterized by recursive parameterization of gating functions. Our proposed method achieves very promising and competitive results on a spectrum of benchmarks across multiple modalities (e.g.. language, logic, music). We propose a non-autoreeresssvve variation of METAGROSS, which allows simple drop-in enhancement to state-of-ther thr Transformers. We study and visualise network as it learns a dynamic recursive parameterization, shedding light on the expressiveness and ficxibility to lear dynamic parameter structures depending the data.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Alvarez-Melis and Tommi S Jaakkola. Tree-structured decoding with doubly-r neural networks. 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0444 2014.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086, 2016.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 1rXIvv1803.0122 2018a.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv: 118..00..0622 2018b.
Nicolas Koulanger-Lewandowwe Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. arXiv preprint arXiv: 6392, 2012.
Samuel R Bowman, Christopher Potts, and Christopher Manning. Recursive neural networks can learn logical semantics. arXiv preprint arXiv:1400. /827, 2014.
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. arXiv preprint arXiv:1603.06021, 2016.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural net- works. arXiv preprint 161.0 2016.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances Neural Information Processing Systems, pp. 77-87, 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Jihun Choi, Kang Min Yoo, and Sang-gon goo Lee. Learning to compose task-specific tree structures. In Thirly-Second AAAI Conference on Artificial Intelligence, 2018.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Junyoung Chung. Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural net- works. arXiv preprint 11X.v.1609.01704 2016.
Li Dong and Mirella Lapata. Language to logical form with neural attention. arXiv preprint 1601.0260. 2016.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. Recurrent neural network grammars. arXiv preprint arXiv: /o020007007766 2016.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependen- cies. In Advances in neural information processing systems, pp. 493-499, 1996.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.5401, 2014.
arXiv preprint
David Ha, Andrew Dai, and Quoe V Le. Hypemetworks.. arXiv preprint arXiv: 1609.09100 2016.
Danijar Hafner, Alexander Irpan, James Davidson, and Nicolas Heess. Learning hierarchical in- formation flow with recurrent neural modules. In Advances in Neural Information Processing Systems, pp. 6724-6733, 2017.

---

Under review as a conference paper at ICLR 2020
Serhii Havrylov, Germán Kruszewski, and Armand Joulin. Cooperative learning of disjoint syntax and semantics. arXiv preprint arXiv:1902.09393, 2019.
Tianyu He, Xu Tan, Yingee Xia, Di He, Tao Qin, Zhibo Chen, and Tin-Yan Liu. Layer-wise co- ordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems, pp. 7944-79544 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. Towards neural phrase- based machine translation. arXiv preprint arXiv:/706.05565, 2017.
Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, and Yoshua Bengio. Learning hierarchical structures on-the-fly with recurrent-recuree model for sequences. In Proceedings of The Third Workshop on Representation Learning for NLP, pp. 154-158, 2018.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork mn. arXiv preprint arXiv:1402.3511 2014.
Tao Lei, Yu Zhang, Sida Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly par- allelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4470-4481 2018
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn): Building a longer and deeper rnn. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5457-5466, 2018a.
Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-Yan Liu. Towards binary- valued gates for robust Istm training. arXiv preprint arXiv: 02988, 2018b.
Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Andrew Senior, Fumin Wang, and Phil Blunsom. Latent predictor networks for code generation. arXiv preprint arXiv:1603.0674. 2016.
Liyuan Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint /rXiv: 03265, 2019.
Shujie Liu, Nan Yang, Mu Li, and Ming Zhou, A recursive recurrent neural network for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics ( (Volume 1: Long Papers), pp. 1491-1500, 2014.
Minh- Minh-Thang Luong and Christopher D Manning. Stanford neural machine translation systems for spoken language domains. 2015.
Graham Neubig. lamtram: A toolkit for language and translation modeling using neural networks, 2015.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing, arXiv preprint arXiv:1704.07535 2017.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv:1511.067322 2015.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rac, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 7299-7310, 2018.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history com- pression. Neural Computation, 4(2):234-442. 1992.
Yikang Shen, Zhouhan Lin, Chin- Wei Huang, and Aaron Courville, Neural language modeling by jointly learning syntax and lexicon. arXiv preprint arXiv:1711.2001: 2017.
10

---

Under review as a conference paper at ICLR 2020
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:110000953,, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Kyungwoo Song. JoonHo Jang, Il-Chul Moon, al. Bivariate beta Istm. 10055,,,0 2019.
arXiv preprint
Hya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3102-3112, 2014.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.0000 2015.
Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. Recurrently controlled recurrent networks. In Advances in Neural Information Processing Systems, pp. 4731 1731-4743, 2018.
Ke Tran, Arianna Bisazza, and Christof Monz. Recurrent memory networks for language modeling. arXiv preprint arXiv: 160100127, 2016.
Ke Tran, Arianna Bisazza, and Christof Monz. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv:1803.03558 2018.
Trieu Trinh, Andrew M Dai, Minh-Thang Luong, and Quoe Le. Learning longer-term depen- dencies in mns with auxiliary losses. arXiv preprint arXiv: 1803.00/44, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. -998-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/1803.0741 2018. URL http: http///axiv...o abs///b///ta
aatpa/ass/111...//..//// 1803. 07416.
Yau-Shian Wang, Hung-Yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree structures into self-attention. arXiv preprint arXiv: 1909.06639, 2019.
Yijun Wang, Yingee Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and Tie- Yan Liu. Dual transfer learning for neural machine translation with marginal distribution regularization. In Thirty--Scoone AAAI Conference on Artificial Intelligence, 2018.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purposs code generation. arXiv preprint arXiv: httv::704.06999 2017.
Pengcheng Yin and Graham Neubig. Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720, 2018.
Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. Learning to compose words into sentences with reinforcement learning. arXiv preprint 16Xiv:1611.09100 2016.
11


Reviews:
[{"review_id": "S1xtQSg0Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural sequence modelling unit called METAGROSS.  In principle, the aim of this unit is to introduce recursive parametrization of  gating functions, building on the gated RNN paradigm.  The authors motivate this work by arguing that while gated-RNNs tackle vanishing gradient problems and facilitate learning long-range dependencies in sequences, improvements can be made with respect to learning on hierarchically-structured data.  The authors propose a method to do so by also learning the depth of the parametrization, and claim that the inductive bias that emerges from this configuration is beneficial to learning such tasks.\n\nI think the idea behind this work is sensible: introducing a meta-controller with recursive parametrization of gating functions for hierarchical tasks is sensible.  Another also strong point of this paper is that several experiments under different settings are presented, along with ablation studies and some model exploration.  The authors also show that integrating a non-autoregressive variant of the proposed meta-controller into the Transformer architecture can also be beneficial.  Results in general do show improvement over compared architectures.\n\nOn the negative side and besides empirical/experimental evidence, the paper would be much more convinving with some more insight into the model itself and some more qualitative evidence.  Figure 1 shows the architecture of the proposed method with a max depth of 3 indicating soft recursion with grayscale levels.  However, this figure is not referenced in the text and not explained further.  The non-autoregressive version (sec 2.3) simply does away with the dependence on hidden states and applies the proposed architecture directly on the input.  I think it would be very beneficial to see a simple toy example where the benefits of utilizing this meta-architecture can be qualitatively explained.  Finally, one can argue that this work is incremental, in the sense that it is a (relatively straightfwd) combination of meta-controllers with recursive architectures.  Differences and variations with respect to other methods in literature should be more clearly explained.\n\nsome more questions\n- although most experiments show some resilience with respect to the varying max depth parameter, have the authors noticed any problems and limitations arising from setting the max-depth to be very high?  for example, in Table 1 it seems that accuracy may drop when increasing max length.  I think that more discussions and ellaboration on this could be useful, as the authors also propose a way of learning the depth parameters and also state that this is task dependent.  \n- Figures 7-8-9 show variations of dynamic recursion on three databases.  Although these figures do show variation in learning (ranging from high activation fluctuations to static), it would be interesting to examine why this fluctuations occur in CIFAR  \n- which brings me to the second question:  although the results do not show this, could this task-dependent nature of the controller lead to more chances of overfitting on a given training set that may be noisy?\n- Although the authors perform ablation tests with multiple units and evaluate for max depth, we don't see many experiments with depth of more than 2 or 3 (besides table 1 - in many experiments the max depth is not mentioned).  Are the conclusions the same with all experiments wrt depth?\n- It would be useful to have a comment on model complexity", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gofkQqYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a variant of gating functions for recurrent neural networks and feed-forward layers of Transformer and apply it to variety of tasks including toy tasks such as sorting, tree traversal and more realistic tasks such as machine translation. The gating function is applied recursively for N number of steps and depth of recursion is learned softly in data-driven function. Authors show similar or slightly better performance of their approach when applied to LSTM and Transformer compared to vanilla LSTM and Transformer.  \n\nI have several comments regarding this work:\n\n1) I believe there is a typo in equation in section 2 describing parametrization of o^{n}_{t} where h_{t-1} has un-necessary upper-script {n}\n\n2) I would like to see the equations showing differences/similarities between MetaGross gating function and GRU/LSTM in Section 2 to better understand how MetaGross relates to previous work.\n\n3) Calling parallel version of MetaGross that computes gating values given entire sequence *non-autoregressive* is really correct since decoding for machine translation is still done in autoregressive fashion one token at a time. I would suggest the authors to remove word non-autoregressive and just stick with word parallel.\n\n4) The obvious connection with this work and work of Alex Graves on Adaptive Computation for Recurrent Neural Nets and many of its followups including Universal Transformers by Dehghani et al is missing from experimental comparison and is not even mentioned in related section. I believe that not mentioning these papers and not comparing to them empirically  this is a major drawback of this paper. \n\n5) I don't think that Task 3 (Logical Inference) contains a language vocabulary of six words because it is a natural english language unless I am misunderstanding something. \n\n6) What does EM stand for in Table 1. Would be great if you could include description of what EM, P(perplexity), n(depth of recursion) stands for in caption of Table 1\n\nOverall it is a good and well written paper, although I believe that the variants of recursively gated functions have been proposed and applied before (see comment 4). Would be open to discussions and raising scores if authors convince me otherwise. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJlXlGyJ9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: As no rebuttal has been posted I stand by my assessment.\n\nSummary\nThis papers proposes a recursive parameterization of gates in a recurrent model. Instead of directly conditioning gates on the input and previous hidden representation, the proposed model recursively calls itself to parameterize the gate. The recursion depth is dynamically determined (up to a predefined maximum recursion depth). The approach shows slight improvements over baselines on a range of tasks.\n\nStrengths\nSlight improvements on a wide range of downstream tasks\nQualitative analysis of dynamic recursion highlighting the adaptability of the method to different task properties\n\nWeaknesses\nFrom the equations in Section 2, I understand that a gate is conditioned not only on the input and previous hidden representation, but also the input and hidden representations from repeated application of the RNN cell for the given time step. As far as I understand, this is very closely related to ACT by Graves, Alex. \"Adaptive computation time for recurrent neural networks.\" 2016. They also have a dynamic way of determining how many recursive RNN cell applications per step should be performed.\nI am missing a clear description of differences of the proposed approach to the baselines tested in Section 3. At some point it is mentioned that a different optimizer was used (R-Adam). Are there any other confounding factors? I believe it is important to get clarity on these given that the difference between the model and baselines are very small.\nResults in Table 1 are highlighted in a misleading way. For example, stacked BiLSTM do as well for tree traversal (EM, n=5 and EM, n=10). For logical inference, there is a more recent paper investigating the limits of RNNs and I believe comparisons on that dataset could strengthen the paper: Evans, Richard, et al. \"Can Neural Networks Understand Logical Entailment?.\" ICLR 2018.\n\nMinor Comments\np1: \"the ability to reason deeply\" – I understand you mean literally \"deep\", but also \"reason\" is a loaded term.\np1: \"bears a totally different meaning\" should be made concrete\np3: I am a bit confused by the fact that Metagross is used to extend Transformers here given that you called it a recurrent (and recursive) model on the previous slide.\np4: What is the number of layers used in he stacked BiLSTM and is this the same as the maximum depth used in Metagross? I believe for a fair comparison they should be the same. I am also missing stacked LSTM in experiments on logical inference (Table 2).\nQuestions to Authors", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a recurrent architecture based on a recursive gating mechanism. The reviewers leaned towards rejection on the basis of questions regarding novelty, analysis, and the experimental setting. Surprisingly, the authors chose not to engage in discussion, as all reviewers seems pretty open to having their minds changed. If none of the reviewers will champion the paper, and the authors cannot be bothered to champion their own work, I see no reason to recommend acceptance.
Decision: Reject