Self-Instantiated Recurrent Units with Dynamic Soft Recursion
Authors: Aston Zhang
Yi Tay
Yikang Shen
Alvin Chan
Shuai Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: recurrent neural network
recursive neural network
LSTM

Abstract: While standard recurrent neural networks explicitly impose a chain structure on different forms of data, they do not have an explicit bias towards recursive self-instantiation where the extent of recursion is dynamic.  Given diverse and even growing data modalities (e.g., logic, algorithmic input and output, music, code, images, and language) that can be expressed in sequences and may benefit from more architectural flexibility, we propose the self-instantiated recurrent unit (Self-IRU) with a novel inductive bias towards dynamic soft recursion. On one hand, theSelf-IRU is characterized by recursive self-instantiation via its gating functions, i.e., gating mechanisms of the Self-IRU are controlled by instances of the Self-IRU itself, which are repeatedly invoked in a recursive fashion. On the other hand, the extent of the Self-IRU recursion is controlled by gates whose values are between 0 and 1 and may vary across the temporal dimension of sequences,  enabling dynamic soft recursion depth at each time step. The architectural flexibility and effectiveness of our proposed approach are demonstrated across multiple data modalities. For example, the Self-IRU achieves state-of-the-art performance on the logical inference dataset [Bowman et al., 2014] even when comparing with competitive models that have access to ground-truth syntactic information.

Self-Instantiated Recurrent Units with Dynamic Soft Recursion
Aston Zhang', Yi Tay?; Yikang Shen*, Alvin Chan", Shuai Zhang
Amazon Web Services AI, &Google Research Mila, Université de Montréal, NTU, Singapore, 'ETH Zürich azβastonzhang.. com
Abstract
While standard recurrent neural networks explicitly impose chain structure different forms of data, they do not have an explicit bias towards recursive self- instantiation where the extent of recursion is dynamic. Given diverse and even growing data modalities (e.g., logic, algorithmic input and output, music, code, images, and language) that be expressed in sequences and may benefit from more architectural flexibility, we propose the celf-instantiated recurrent unit (Self- IRU) with novel inductive bias towards dynamic soft recursion. On one hand, the Self-IRU is characterized by recursive self-instantiation via its gating functions, i.e.. gating mechanisms of the Self-IRU are controlled by instances of the Self-IRU itself, which are repeatedly invoked in a recursive fashion. On the other hand, the extent of the SeUf-IRU recursion controlled by gates whose values are between 0 and and may vary across the temporal dimension of sequences, enabling dynamic soft recursion depth at each time step. The architectural flexibility and effectiveness of our proposed approach are demonstrated across multiple data modalities. For example, the Self-IRU achieves state-of-the-art performance on the logical inference dataset [Bowman et al. 2014) even when comparing with competitive models that have access to ground ground-truth syntactic information.
Introduction
Models based on the notion of recurrence have enjoyed pervasive impact across various applica- tions. In particular, most effective recurrent neural networks (RNNs) operate with gating functions. Such gating functions not only ameliorate vanishing gradient issues when modeling and capturing long-range dependencies, but also benefit from fine-graie ane-grained control over temporal composition for
sequences (Hochreiter and Schmidhuber) 1997. Cho al. 2014).
With diverse and even growing data modalities (e.g., logic, algorithmic input and output, music, code, images, and language) that can be expressed in sequences and may benefit from more architectural flexibility, recurrent neural networks that only explicitly impose chain structure on such data but lack an explicit bias towards recursive self seff-instantiatinn may be limiting. For example, their gating functions are typically static across the temporal dimension of sequences. In view of such, this paper aims at studying an inductive bias towards recursive self-instantiatto where the extent of recursion is dynamic different time steps.
We propose a novel recurrent unit whose gating functions are repeatedly controlled by instances of the recurrent unit itself. Our proposed model is called the self-instantiated recurrent unit (Self-IRU), indicates modeling via the own instances of the model itself in a recursive where Specifically, two gates of the Self-IRU are controlled by Self-IRU instances. Biologically, fashion.
Work was done NTU.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

this design is motivated by the prefrontal cortex/basal ganglia working memory indirection (Kriete] al. 2013]. For example, a child Self-IRU instance drives the gating for outputting from its parent IRU instance.
Our proposed Self-IRU is also characterized by the dynamically controlled recursion depths. Specif- ically, we design a dynamic soft recursion mechanism, which softly learns the depth of recursive self-instantiation on per-time-step basis. More concretely, certain gates are reserved control the extent of the Self-IRU recursion. Since values of these gates are between and and may vary across the temporal dimension, they make dynamic soft recursion depth at each time step possible, which could lead to more architectural flexibility across diverse data modalities.
This design of the Self-IRU is mainly inspired by the adaptive computation time (ACT) Graves 2016) that learns the number of computational steps between an input and an output and recursive neural networks that operate on directed acyclic graphs. On one hand, the Self-IRU is reminiscent of the ACT, albeit operated at the parameter level. While seemingly similar, the Self-IRU and ACT are very different in the context of what the objective is. Specifically, the goal of the Self-IRU is to dynamically expand the parameters of the model, not dynamically decide how long to deliberate on input tokens in sequence. On the other hand, the Self-IRU marries the benefit of recursive reasoning with recurrent models. However, in contrast to recursive neural networks, the Self-IRU is neither
concerned with syntax-guided composition [Tai et 2013 Socher et al. 2013
D Dyer al. 2016
Wang and Pan 2020) nor unsupervised grammar induction [Shen et al.
2017
Choi et al.
2018
Yogatama et
2016 Havrylov et al. 2019).
Our Contributions All in all, sequences are fundamentally native the world, so the design of effective inductive biases for data in this form has far-reaching benefits across diverse range of real-world applications. Our main contributions are outlined below:
We propose the self-instamtiated recurrent unit (Self-IRU). It is distinctly characterized by a novel inductive bias towards modeling via the own instances of the unit itself in recursive fashion, where the extent of recursion is dynamically learned across the temporal dimension of sequences.
We evaluate the Self-IRU wide spectrum of sequence modeling tasks across multiple modalities; logical inference, sorting, tree traversal, music modeling, semantic parsing, code generation, and pixel-wise sequential image classification. Overall, the empirical results demonstrate architectural flexibility and effectiveness of the Self-IRU, For example, the Self-IRU achieves sate-of-thhear performance on the logical inference dataset (Bowman ett al., [014] even when comparing with competitive models that have access to ground-truth syntactic information.
Notation For readability, all vectors and matrices are denoted by lowercase and uppercase bold letters such as x and X, respectively. When a scalar is added to vector, the addition is applied element-wise (Zhang et al. 2021].
2
Method
This section introduces the proposed Self-IRU. The Self-IRU is fundamentally a recurrent model, but distinguishes itself in that the gating functions that control compositions over time are recursively modeled by instances of the Self-IRU itself, where the extent of recursion is dynamic. In the following. we begin with the model architecture that can recursively self-instantiate. Then we detail its key components such as how dynamic soft recursion is enabled.
2.1 Self-Instantiation
Given an imput sequence of tokens XI.... .y. the Self-IRU transforms them into hidden states throughout all the time steps: hi...., hr. Denoting by L the user-specified maximum recursion depth, the hidden state at time step is
h, = (clf-IRU(()) (x,, (L)

---

Self-IRU
b %-1
P
h
h
Self-IRU (1-1)
Ianh
0
p
F
F.
10
h
Figure 1: The self-instane instantiiaeed recurrent unit (Self-IRU) model architecture. Circles represent gates that control information flow from dotted lines, and squares represent transformations or operators.
(T) where Self-TRU(*) is instance of the Self-IRU model recursion depth L and h h is hidden state at time step -- and recursion depth L. In general, a Self-IRU instance at any recursion depth O 1 S T. returns hidden state for that depth:
Self-IRU h,2)) h,"
which involves the following computation:
F a Sel-IRU"-)) (x,, (((-1)) a. (m) F (x, h n22r))
(2.1)
( ( Self-RR (x,, h (1-1)
+ - 3(m)) ) p(() (x:: 111
o
(2.2)
z,
( tanh (!!)) (xx, b(_1))
(2.3)
c (1) 1 o 2 + (1-1()))z) )zz
(1
(2.4)
h o
(1) (
Xt:
(2.5)
where @ denotes the element-wise multiplication, denotes the sigmoid function, scalars "² and a(") are soft depth gates at time step and recursion node in the unrolled recursion paths, and F(!)
r!",, and (1())) are base transformations at recursion depth l. Without losing sight of the big picture, we will provide more details of such soft depth gates and base transformations later.
On a high level, Figure [ depicts the Sclf-IRU model architecture. We highlight that two gating functions of Self-IRU, the forget gate r in (2.1) and the output gate o, (2) in (2.2), recursively
controlled by instances of the Self-IRU itself. Therefore, we call both the forget and output gates the self-instantiation gates. The base case 0) for self-instantiation gates is
p(o) = o (
Xr. (
((i))) and o, (0) = =a ( (x,, (0)
At each recursion depth I, the candidate memory cell z at time step is computed in (2.3) Then in (2.4), the forget gate E.") controls the information flow from z; (I) and the memory cell c the previous time step to produce the memory cell e!") at the current time step t. As illustrated by the bottom arrow starting from x; Figure [1] the output gating (2.3) also adds a skip connection from residual networks to facilitate gradient flow throughout the recursive self-instantiation of the
Self-IRU (He al. 2016
2.2 Dynamic Soft Recursion
Now let us detail the soft depth gates ot and (5a)) 2.1 and 2.2 for time step and recursion node in the unrolled recursion paths. The index is used to distinguish nodes different positions

---

Time step
1+1
Self-IRU
Self-IRU (2)
0
(F)
Self-IRU
Self-IRU
Self-IRU
Self-IRU 11)
(FT)
DIFR) (RF)
a
(RR)
(FF)
BIRR)
(RF) #+1
DiRR
Self-IRU"
Self-IRU (0)
Self-IRU (0)
Self-IRU (
Self-IRU
(0)
Self-IRU"
(0)
Self-IRU (0)
Self-IRU
(
(n) Figure 2: Soft depth gates of and (3(h) for time step / and recursion node n (denoting F and R as the left child and the right child, respectively) control the extent of the Self-IRU recursion. The extent indicated by greyscale of any node at the beginning of an arrow along an unrolled recursion path. These gates are between 0 and and may vary across the temporal dimension, enabling dynamic soft recursion depth at each time step (here maximum depth L = 2).
in the recursion tree (e.g., in Figure 2 that is determined by the maximum recursion depth L. We propose learning them in data-driven fashion. Specifically, parameterize a!") and g(()) with
(n)) o [F(( (xx)) and ((()) =F (x:))
where F (xx) w! X + ( E R (* E {a, with weight parameters W(n and bias parameters b(.)) both learned from data.
Together with the sigmoid function a, these simple linear transformations of the input token x, are applied dynamically at each time step across the input sequence. Moreover, as shown in (2.1) and (..2) mathematically 0 < a; (.( control the extent recursion at each recursion node n, enabling soft depth along any unrolled recursive celf-instantiation path. Thus, ag") and s(") are called the soft depth gates.
Putting these together, Figure Jnrolls the recursive self celfannstantiatonn paths at two consecutive time steps to illustrate dynamic soft recursion depth. Specifically, the "softness" is indicated by greyscale of any node at the beginning of an arrow along an unrolled recursion path. In sharp contrast to multi-layer RNNs, Self-IRUs enable structures of seff-instantiation, where the extent of recursion is dynamic (to be visualized in Section 3.7
2.3 Base Transformations
At any recursion depth 1, r(") in (2.
(1)
in 2.2 and F. in (2.3 are base transformations of the
input and the hidden state h For example. we can model base transformations using RNN units (e.g., LSTM): at recursion depth I, for * E {f. 2) we have
F!") (x h
RNN! (x, h (0
Alternatively, we may also model base transformations with linear layers that only transform the input using learnable weight parameters (!(!) and bias parameters (1)) for E (f.
(x,)
+ (
The Self-IRU agnostic to the choice of base transformations and we will evaluate different choices in the experiments. We will discuss how the Self-IRU can be useful as (parallel) con-autoregressive model and connects to other recurrent models in the supplementary material.
Experiments
To demonstrate the architectural flexibility and effectiveness, we evaluate Self-IRUs wide range of publicly available benchmarks, perform ablation studies on the maximum recursion depth and base transformations, and analyze dynamics of soft depth gates.

---

3.1 Pixel-wise Sequential Image Classification
The sequential pixel-wise image classification problem treats pixels in images as sequences. We use the well-established pixel-wise MNIST and CIFAR- 10 datasets.
Table 1: Experimental results (accuracy) on the pixel-wise sequential image classification task.
Model
#Params
MNIST
CIFAR-IO
Independently R-RNN [ii al. 2018a
99.00
r-LSTM with Aux Loss
Irinh et al. 2018
98.52
72.20
Transformer (self-attention)
Trinh et
2018
98.90
62.20
TrellisNet [Bai et al.
2018b
(reported) (our run)
8.0M
99.20
73.42
TrellisNet Bar al.
2018b
8.0M
97.59
55.83
Self-IRU
0.9M
99.04
73.01
Table 1 reports the results of Self-IRUs against independently recurrent RNNs Li et al. 201 8a
r-LSTMss with aux loss [Trinh et al. 2018), Transformers (self-attention) (Trinh al. 201 2018] 8J. and TrellisNets (Bai et al. [0] On both the MNIST and CIFAR-10 10 datasets, the Self-IRU outperforms most of the other investigated baseline models, For the only exception, parameters of the Self-IRU only about 1/8 of those of the TrellisNet [Bai et al. 2018b) while still achieving comparable performance. This supports that the Self-IRU is a reasonably competitive sequence encoder.
3.2 Logical Inference
We experiment for the logical inference task on the standard datase proposed by Bowman et al. 22044. This classification task determine the semantic equivalence of two statements expressed with logic operators such as not, and, and or. As per prior work (Shen et al. (218). the model is trained on sequences with or fewer operations and evaluated on sequences
12 operations.
Table 2: Experimental results (accuracy) on the logical inference task (symbol i denotes models
with access to ground-truth syntax). The baseline results are reported from [Shen et al. 2018 The Self-IRU achieves state-of-the-ar ahe performance.
#Operations
Model
7
9
10
11
12
Trec-LSTM Tai et al.
2015
93.0
90.0
87.0
89.0
86.0
87.0
LSTM [Bowman et al.) 2014
88.0
85.0
80.0
78.0
71.0
69.0
RRNet Jacob al. 2018
84.0
81.0
78.0
74.0
72.0
71.0
ON-LSTM [hen et al.
2018
91.0
87.0
86.0
81.0
78.0
76.0
Self-IRU
97.0
95.0
93.0
92.0
90.0
88.0
We compare Self-IRUs with Tree-LSTMs Tai et al. 2015 LSTMs Bowman et al. 2014), RR- Nets (Jacob et al. 2018). and ordered-neuron (ON-) LSTMs TMs (Shen al. 2018 based the common experimental setting in these works. Table 2 reports our results on the logical inference task. The Self-IRU is strong and competitive model on this task, outperforming ON-LSTM by a wide margin (+12% on the longest number of operations). Notably, the Self-IRU achieves state-of-the-a per- formance on this dataset even when comparing with Tree-LSTMs that have access to ground-truth syntactic information.
3.3 Sorting and Tree Traversal
algorithmic tasks that solvable by recursion: sorting and tree traversal. In sorting, the input to the model a sequence of integers. The correct output is the sorted sequence of integers. Since mapping sorted inputs to outputs can be implemented in recursive fashion, we evaluate the Self-IRU's ability to model recursively structured sequence data, An example input-output pair would 9, 1, 10, 5, 3 1,3, 10. We evaluate on sequence length m {5, 10).
We also evaluate Self-IRUs
5

---

In the tree traversal problem, we construct a binary tree of maximum depth n. The goal is to generate the postorder tree traversal given the inorder and preorder traversal of the tree. This is known to arrive at only one unique solution. The constructed trees have random sparsity where trees can grow up to maximum depth n. Hence, these trees can have varying depths (models can solve the task entirely when trees are fixed and full). We concatenate the postorder and inorder sequences, delimited by special token. We evaluate on maximum depth n E {3, 4, 5, 8, 103. For n E {5, 8), we ensure that each tree traversal has at least 10 tokens. For n 10, we ensure that each path has at least 15 tokens. An example input-output pair would be 13, 15, 4, 7, 5, X, 13, 4, 5, - 7, 15, 13, 4,5.
We frame sorting and tree traversal as sequence-ter weeeennce-tt-squeeee to-seunnnce (Sutskever et al., 2014] tasks and evaluate models with measures of exact match (EM) accuracy and perplexity (PPL). We use standard mncoder-decoder architecture with attention [Bahdanau et al., 2014]. and vary the encoder module with BiL.SMM, stacked BiM,STMs, and ordered-neuron (ON-) LSTMs (Shen al. 2018
Table 3: Experimental results on the sorting and tree traversal tasks,
SORTING
TREE TRAVERSAL
Model
EM
PPL
EM
PPL
EM
PPL.
EM
PPL.
EM
PPL
EM
PPL
EM
PPL.
BILSTM
79.9
1.2
78.9
1.2
100
1.0
96.9
2.4
60.3
2.4
5.6
30.6
2.2
132.0
Stacked BiLSTM
83.4
1.2
88.0
100
98.0
1.0
63.4
2.5
5.9
99.9
2.8
225.1
ON-LSTM
90.8
1.1
87.4
1.1
100
1.0
81.0
1.4
33.7
288
5.5
52.3
2.7
173.2
SelF-IRU
92.2
1.1
90.6
100
63.4
5.6
20.4
288
Table 3 reports our results on the sorting and tree traversal tasks. In fact, all the models solve the tree traversal task with = 3. However, the task gets increasingly harder with a greater maximum possible depth and largely still remains a challenge for neural models today. On hand, stacked BiLSTMs always perform better than BiLSTMs and ON-LSTMs occasionally perform worse than standard BiLSTMs on tree traversal, while for the sorting task ON- perform much better than standard BiLSTMs. On the other hand, the relative performance of the Self-IRU is generally better than any of these basclines, especially pertaining to perplexity.
3.4 Music Modeling
Moreover, we evaluate the Self-IRU on the polyphonic music modeling task, i.e., generative modeling of musical sequences. We three well-established datasets: Nottingham, JSB Chorales, and Piano Midi Bawlaangee-Lewaddwwwwwe.o et al. 2012). The inputs are 88-bit (88 piano keys) sequences.
Table 4: Experimental results (negative log-likelihood) the music modeling task.
Model
Nottingham
JSB
Piano Midi
GRU CChnng al.
2014
3.13
8.54
8.82
LSTM Song et al.
2019
3,25
8.61
7.99
G2-LSTM
ct al.
2018b
3.21
8.67
8.18
B-LSTM Song et al.]
2019
3.16
8.30
7.55
TCN
Bai et
2018a
(reported)
3.07
8.10
TCN Bai et al.
2018a
(our run)
2.95
8.13
7.53
IeU
2.88
8.12
7.49
Table 4 compares the IRU with wide range of published works: GRU [Chung et al., 2014]. LSTM [Sng et al.) 2019 G2-LSTM (Li et al.] 2018b), B-LSTM (Song et al. 2019). and TCN Bai ct al. 2018a The Self-IRU achieves the best performance on the Nottingham and Piano midi datasets. It also achieves competitive performance on the JSB Chorales dataset, only underperforming the state-of-the-art by 0.02 negative log-likelihood.
3.5 Semantic Parsing and Code Generation
We further evaluate Self-IRUs on the semantic parsing (the Geo, Atis, and Jobs datasets) and code generation (the Django dataset) tasks. They are mainly concerned with learning to parse and generate structured data. We run our experiments on the publicly released source code of YYnn and Neubig

---

2018), replacing the recurrent decoder with our Self-IRU decoder (TranX Self-IRU), We only replace the recurrent decoder since our early experiments showed that varying the encoder did not yield any benefits in performance. Overall, our hyperparameter details strietly follow the codebase of Yin and Neubig 2018 i.e., we run every model from their codebase it is.
Table 5: Experimental results (aceuracy) on the semantic parsing (the Geo, Atis, and Jobs datasets) and code generation tasks (the Django dataset).
Model
Geo
Atis
Jobs
Django
Teq2 [Dong and Lapata
2016
87.1
84.6
31.5
LPN
Ling et al. 2016
62.3
NMT
Neubig
2015
45.1
YN17 Yin and Neubig
2017
71.6
ASN Rabinovich et al.
2017
85.7
85.3
ASN Supv. Attn.
RRabinovich al.
2017
87.1
85.9
TranX (iin and Neubig
2018
(reported in code)
88.6
87.7
90.0
77.2
TranX Yın and Neubig
2018
(our run)
87.5
87.5
90.0
76.7
TranX + Self-IRU
88.6
88.4
90.7
78.3
Table 5 reports the experimental results in comparison with the other competitive baselines such as Seq2Tree [Dong and Lapata 2016). LPN [Ling et al. 2016]. NMT [Neubig] 2015). YN17 (Yin) and Neubig, 2017]. ASN (with and without supervised attention) [Rabinovich et al. 2017]. and TranX [Yin and Neubig, 2018). We observe that TranX Self-IRU outperforms all the other approaches, achieving state-of-the-art perforaacce On the code generation task, TranX + Self-IRU outperforms TranX by +1.6% and +1% on all the semantic parsing tasks. More importantly, the performance gain over the base TranX method allows us to observe the ablative benefit of the Self-IRU that achieved by only varying the recurrent decoder.
3.6 Ablation Studies of the Maximum Recursion Depth and Base Transformations
Table 6 presents ablation studies of the maximum recursion depth (Section 2.1 and base transfor- mations (Section (.3) of Self-
Table 6: Ablation studies of the maximum recursion depth and base transformation on the semantic parsing (SP) and code generation (CG) tasks.
IRUs. The results are based on semantic parsing (Atis) and code generation (Django) tasks. We can see that their optimal choice is task dependent: (i) on the semantic parsing task, using the linear layer performs better than the LSTM for base transfor-
Max Depth
Base Transformations
SP
CG
Linear
88.40
77.56
Linear
88.21
77.62
3
Linear
87.80
76.84
L.TTM
86.61
78.33
2
LSTT
85.93
77.39
mations; (ii) conversely, the linear transformation performs worse than the LSTM on the code generation task.
On the whole, we also observe this across the other tasks in the experiments. Table 7 re- ports their optimal combinations for diverse tasks in the experi- ments, where the maximum re- cursion depth is evaluated = {0, 1, 3). As we can tell from different optimal com- binations in Table 7 choices of the maximum recursion depth and base transformations of Self- IRUs depend on tasks.
Table The optimal maximum recursion depth and base transfor- mations for different tasks in the experiments.
Task
Max Depth
Base Transformations
Image classification Logical inference
LSTM
2
LSTM
Tree traversal
1
LSTM
Sorting
I
LSTM
Music modeling
2
Linear
Semantic parsing
I
Linear
Code generation
1
LSTM
7

---

3.7 Analysis of Soft Depth Gates
Besides the task-specific maximum recursion depth and base transformations, empirical effectiveness of Self-IRUs may also be explained by the modeling flexibility via the inductive bias towards dynamic soft recursion (Section 2.2
We will analyze in aspects below.
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.5
0.6
0.4
0.4
0.4
0.4
0.2
(a) Initial (CIFAR-10)
(b) Epoch (CIFAR-IO)
Initial (MNIST)
(d) Epoch 10 (MNIST)
Figure 3: Soft depth gate values at initialization and training epoch 10 on the CIFAR-10 and MNIST datasets.
First, during training, the Self-IRU has the flexibility of building data- dependent recursive patterns of self- instantiation. Figure dissplay values of all the soft depth gates at all the three recursion depths on the CIFAR- 10 and MNIST datasets, depicting how recursive pattern of the Self-IRU is updated during training. For differ- ent datasets, the Self-IRU also flexibly learns to construct different soft recur- sive (via soft depth gates of values be- tween O and 1) patterns.
(a) Image classification (CIFAR-10)
Second, we want to find out whether the Self-IRU has the flexibility of softly learning the recursion depth on a per- time-step basis via the inductive bias to- wards dynamic soft recursion. Figure visualizes such patterns (i) for pixel- wise sequential image classification on the CIFAR-10 10 and MNIST datasets and (ii) for music modeling on the Notting- ham dataset. Notably, all the datasets have very diverse temporal composi- tions of recursive patterns. More con- cretely, the soft depth gate values fluc- tuate aggressively on the CIFAR-10 dataset (consisting of color images) in Figure 4a while remaining more stable for music modeling in Figure 4 More- over, these soft depth gate values remain totally constant on the MNIST dataset (consisting of much simpler grayscale images) in Figure 4b) These provide compelling empirical evidence for the architectural flexibility of Self-IRUs: they can adjust the dynamic construc- tion adaptively and can even revert to static recursion over time if necessary (such for simpler tasks).
6.1
180
470
(b) Image classification (MNIST)
3.54
Tamporaa
(c) Music modeling (Nottingham)
Figure 4: Soft depth gate values across the temporal dimen-
sion. "I." and "R" denote ot (n) and _ respectively (e.g.. "LLR" denotes the node at the end of the unrolled recursive path a. + at ((m))

---

The dynamic soft recursion pattern is made more intriguing by observing how the "softness" alters on the CIFAR-10 and Nottingham datasets. From Figure we observe that the soft recursion pattern of the model changes in rhythmic fashion, in line with our intuition of musical data. When dealing with pixel information, the recursive pattern in A adaptively according to the more complex polor-image information. Though these empirical results are intuitive, a better understanding of such behaviors may benefit from theoretical or biological perspectives in the future.
Related Work
The study of effective inductive biases for sequential representation learning has been prosperous research direction. This has spurred on research across multiple fronts, starting from gated recurrent models [Hochreiter and Schmidhuber 1997 Cho et al. 2014). convolution [Kim 2014]. to self-
attention-based models [Vaswani et al. 2017].
The intrinsic hierarchical structure native to many forms of sequences has long fascinated and
inspired researchers [Socher et al. 2013] [owmnan al., 2014 2016 Dyer et al., 2016). Nested
LSTMs use hierarchical memories Moniz and Krueger 2017] The study of recursive networks,
popularized by Socher al. [ 2013 J. has provided a foundation for learning syntax-guided composition.
Along the same vein, [Tai et al. 20155 proposed Free-L.STMM that guide L.TT composition with grammar. Recent attempts have been made to learn this process without guidance or syntax- based supervision Yogatama et al. 2016 Shen et al. 2017 Choi et al.
Havrylov et al. 2019 Kim 2018 propose structured gating mechanisms, imbuing the recurrent unit with rerestruct inductive bias. Besides, Tran et al.
2018
ct al, 2019]. Specifically,
undered-neuronn LSTMs Shen et al.
2018] showed that recurrence is important for modeling hierarchical structure. Notably, learning hierarchical representations across multiple time-scales [E] Hihi and Bengio, [996] (Schmidhuber
1992
Koutnik et al. 2014 Chung et al. 2016 Hafner et al. 2017] has also demonstrated effectiveness.
Learning an abstraction and controller over base recurrent unit is also another compelling direction. First proposed in fast weights by (Schmidhuber (1922], several recent works explored this notion. HyperNetworks (Ha et al., 2016) learn generate weights for another recurrent unit, i.e., form of relaxed weight sharing. On the other hand, RCRN [Tay et al. 2018)) explicitly parameterizes the gates of an RNN unit with other RNN units. Recent studies on the recurrent unit are also reminiscent of this particular notion BBaddbrry et al. 2016 Lei et al. 2018).
The fusion of recursive and recurrent architectures is also notable. This direction is probably the closest relevance to our proposed method, although with vast differences. Liu al. (2014) proposed recursive recurrent networks for machine translation that are concerned with the more traditional syntactic supervision concept of vanilla recursive networks, (Jacob et al.) 2018 proposed the RRNet, which learns hierarchical structures on the fly. The RRNet proposes to learn to split or merge nodes at each time step, which makes it reminiscent of other works (Choi et al. 2018 Shen et al. 2018). Lee and Osindero (2016) and Aydin and Güngör [2020) proposed to feed recursive neural network output into recurrent models. A Mevarez-Melis and Jaakkola (0166 proposed doubly recurrent decoders for tree-structured decoding. The core of their method is depth and breath- recurrence which is similar to our model. However, our Del is concerned with learning recursive self- instaatttannnnnnn, which is in sharp contrast to their objective of decoding trees.
Last, our work combines the idea of external meta-controllers [Schmidhuber, 1992 Ha et al. 2016 Tay et al., 2018) with recursive architectures. Specifically, our recursive parameterization is also form of dynamic memory that offers improved expressiveness in similar spirit to nemory-augmentt
recurrent models (Santoro et al. 2018 Graves et al. 2014 Tran et al. 2016).
5 Summary and Discussions
We proposed the Self-IRU that characterized by recursive instantiation of the model itself, where the extent of the recursion may vary temporally. The experiments across multiple modalities demonstrated the architectural flexibility and effectiveness of the Self-IRU. While there a risk of abusing the Self-IRU such for generating fake contents, believe that model is overall beneficial through effective understanding of our digitalized world across diverse modalities.
Acknowledgements.
We thank the anonymous reviewers for the insightful comments this paper.

---

References
David Alvarez-Melis and Tommi $ Jaakkola. Tree-structured decoding with doubly-recurrent neural networks. 2016.
Cem Rifki Aydin and Tunga Güngör. Combination of recursive and recurrent neural networks for aspect-based sentiment analysis using inter-aspect relations. IEEE Access, 8:77820-77832, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1rXiv:1409.044 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0127) 2018a.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv
preprint arXiv:1810.066 2018b.
Nicolas Boulanger-Lewandoove
Yoshua Bengio, and Pascal Vincent. Modeling temporal dependen- cies in high- nigh-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprini arXiv: (2ti::22066392 2012.
Samuel R Bowman, Christopher Potts, and Christopher D Manning. Recursive neural networks can learn logical semantics. arXiv preprint 14X6::1406. 140:.1002222 2014.
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. arXiv preprint arXiv:1603.06021, 2016.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. arXiv preprint 16.07766 2016.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Jihun Choi, Kang Min Yoo, and Sang-goo Lee. Learning to compose task task-specific tree structures. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Junyoung Chung. Caglar Gulcehre, KyungHyun Cho. and Yoshua Bengio. Empirical evaluation of gated recurrent ncural networks on sequence modeling. arXiv preprint 11Xiv:1412.3555, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint Iet.:610900000 2016.
Li Dong and Mirella Lapata. Language to logical form with neural attention. arXiv preprint arXiv: (tt::/661.000 0002000000.0.. 2016.
Chris Dyer, Adhigunaa Kuncoro, Miguel Ballesteros, and Noah Smith. Recurrent neural network grammars. arXiv preprint arXiv: arXiv: 1602.0777, 2016.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In Advances in neural information processing systems, pages 493-499, 1996.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint 1rXiv:1603.08983 2016.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. urXiv:1410.5.4 2014.
arXiv preprint
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: arXiv:1609.09: 1609.09109 2016.
Danijar Hafner, Alexander Irpan, James Davidson, and Nicolas Heess. Learning hierarchical informa- tion flow with recurrent neural modules. In Advances Neural Information Processing Systems, pages 6724-6733, 2017.
10

---

Serhii Havrylov, Germán Kruszewski, and Armand Joulin. Cooperative learning of disjoint syntax and semantics. arXiv preprint drXiv:1902.09393, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition,
pages 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, and Yoshua Bengio. Learning hierarchical structures on-the-fly with recurrent-recursive model for sequences. In Proceedings of The Third Workshop on Representation Learming for NLP, pages 154-158, 2018.
Yoon Kim. Convolutional neural networks for sentence classification, 2014.
Yoon Kim, Chris Dyer, and Alexander M Rush. Compound probabilistic context-free grammars for grammar induction, arXiv preprint arXiv:1906.1000 10225, 2019.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork mnn. arXiv preprint arXiv:1402.35/1 2014.
Trenton Kriete, David C Noelle, Jonathan D Cohen, and Randall C O'Reelll. Indirection and symbol- like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, (33941)):6330-113955 2013.
Chen- Yu Lee and Simon Osindero. Recursive recurrent nets with attention modeling for ocr in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2231-2239, 2016.
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1470-4481 2018.
Shuai Li, Wanging Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building longer and deeper rnn. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5457-5466, 2018a.
Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang. and Tie- Yan Liu. Towards binary- valued gates for robust 1stm training. arXiv preprint arXiv:1806.088 02988, 2018b.
Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomá' Kočiský, Andrew Senior, Fumin Wang, and Phil Blunsom. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744, 2016.
Shujie Liu, Nan Yang, Mu Li, and Ming Zhou, A recursive recurrent neural network for statistical ma- chine translation. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1491-1500, 2014.
Joel Ruben Antony Moniz and David Knueger. Nested Istms. In Asian Conference on Machine Learning, pages 530-544. PMLR, 2017.
Graham Neubig. lamtram: A toolkit for language and translation modeling using neural networks, 2015.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv: /rXiv:/704.05335 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski. Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pages 7299-7310, 2018
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compres- sion. Neural Computation, 4(2):234-242) 1992.

---

Yikang Shen, Zhouhan Lin, Chin- Wei Huang, and Aaron Courville. Neural language modeling by jointly learning syntax and lexicon. arXiv preprint arXiv:1711.0201: 2017.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville, Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint 11X.v91810.09556, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
Kyungwoo Song, JoonHo Jang, Il-Chul Moon, et al. Bivariate beta Istm. arXiv:1905.1052,, 2019.
arXiv preprint
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387. 2015.
lya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pages 3104-3112, 2014.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.0007 2015.
Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. Recurrently controlled recurrent networks. In Advances in Neural Information Processing Systems, pages 4731-4743, 2018.
Ke Tran, Arianna Bisazza, and Christof Monz. Recurrent memory networks for language modeling. arXiv preprint 1601.0/601.00220 2016.
Ke Tran, Arianna Bisazza, and Christof Monz. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv:1803.03355 2018.
Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le. Learning longer-term dependen- cies in rns with auxiliary losses. arXiv preprint arXiv: 1eX3vv8803300144 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1999-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/l 803.07416, 2018. URL http::/arxiv.. www.aaiwww....ooaeeooooooooooort httgg//bs/1800.. 07416
Wenya Wang and Sinno Jialin Pan. Syntactically meaningful and transferable recursive neural networks for aspect and opinion extraction. Computational Linguistics, 45(4):705-736, 2020.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. arXiv preprint arXiv: /704.01696, 2017.
Pengcheng Yin and Graham Neubig. Tranx: A transition-baute neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810002720, 2018.
Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. Learning to compose words into sentences with reinforcement learning, arXiv preprint arXiv:1611.09100. 2016.
Aston Zhang, Zachary C Lipton, Mu Li, and Alexander Smola. Dive into deep learning. arXiv preprint 100.:2226..144..... 2021.
12


Reviews:
[{"review_id": "hmexgQv5bLT", "reviewer": "Reviewer_yiWz", "summary": "This paper proposes a novel gated recurrent unit module that accomplishes a form of soft recursion through dynamic gating. The experiments evaluate the module across a range of tasks, and in particular show promising results on a logical inference task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a clever way to accomplish a form of soft recursion, by introducing a recurrent module, the 'self instantiated recurrent unit' (self-IRU), that can gate other instances of itself. I found the idea to be well motivated and the experiments seem to demonstrate improvements across a range of domains (though see the caveat below about statistical significance and multiple seeds), particularly on a logical inference task.\n\nI found some parts of the proposed module difficult to understand, and hopefully this can be clarified in the revision. In particular, what is the relation of index $n$ (for the depth gates) to the index $l$ (for the depth layer)? The paper states that $l$ takes on values from $1$ to $L$, the maximum allowed depth (a hyperaparameter), but what values do $n$ take on, and how does that relate to the recursion depth? It may help to clarify this if the values of $n$ and $l$ are labelled in Figure 2, i.e. node 0 and layer 0 could be labelled with superscripts $(n=0)$ and $(l=0)$. \n\nI'm also curious about a few of the design decisions, and am wondering whether the authors considered any of the following variations, or can offer any insight into the motivation behind these particular decisions:\n* Rather than having the input to $z_{t}^{(l)}$ come from the original input $x_{t}$, instead basing this on the output of the Self-IRU in the previous layer, as is done for the forget and output gates.\n* Sharing parameters for the base transformations $F$ across layers.\n* Computing the depth gates ($\\alpha$ and $\\beta$) using a recurrent layer instead of a single feedfoward layer.\n\nI did have two concerns. These concerns are the primary reason for my score, which I would be happy to raise if they are addressed:\n\n1) Are the results shown for a single random seed only, or averaged over multiple seeds? If a single seed, was only one seed used, or are these results for the best seed, and if so out of how many? I think the ideal approach would be to average over multiple seeds and present some measure of error, but at the very least this should be clarified. On some of the datasets the improvements were very small and it's not clear if they are even statistically significant.\n\n2) This seems somewhat unlikely to me, but I'm curious to what extent the results can be explained just by additional depth. How would the proposed approach compare to a multilayer LSTM with the same number of layers as a Self-IRU? One reason for thinking that this might partially explain the results is that the stacked BiLSTM performs almost identically to the Self-IRU on the tree traversal task.\n\nI found the results on the logical inference task very impressive. It makes a lot of sense that an inductive bias for recursion would be helpful for such a task. Have the authors considered visualizing the model's operation on a few examples from this task? That seems like it might be an intuitive way to convey how the model operates, and why it's useful for such tasks.\n\nMinor points:\n* It would be good to include more detail about hyperparameters, including the optimizer, learning rate, initialization etc. This could go in the Appendix.\n* I'm curious about what the relation might be, if any, to the following paper [Kriete et al.], in which one layer of a network is used to gate the output of another layer, and therefore accomplishes a form of indirection. I wonder if something similar could happen with the Self-IRU.\n\nKriete, T., Noelle, D. C., Cohen, J. D., & O’Reilly, R. C. (2013). Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, 110(41), 16390-16395.\n\n**Update and final summary**\n\nThis paper proposes a novel recurrent gated unit that implements a soft form of recursion. There is a thorough empirical evaluation across a range of tasks, showing that the proposed Self-IRU generally results in improvements. Though the improvements are small on some tasks, the improvements on the logical inference task are particularly impressive, especially against a baseline that has access to ground-truth syntactic information.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gWxnYZpeR0z", "reviewer": "Reviewer_kd1W", "summary": "The authors introduce a recursive neural network architecture, using a novel kind of unit termed a 'self-instantiated recurrent unit'.  This architecture enables the gate parameters within an LSTM-like architecture to themselves be determined by units at a lower recursive layer, and the level of recursion to vary dynamically.  The authors test the model empirically on multiple tasks, including image classification, logical inference and music sequence modeling. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The model architecture is an interesting novel variation on the LSTM architecture.  The ability to dynamically learn the recursive depth is also an interesting idea, and as the authors propose, may be interpretatively relevant In various domains.  The presentation is clear, and the authors offer a thorough experimental investigation of the architecture on multiple domains, achieving state-of-the-art results on several datasets, along with an ablation study, and analysis of the variation in recursive depth across domains, and comparisons with alternative architectures.\n\nMinor points:\n\n1. A potentially interesting comparison would be to compare the relative strengths of including more layers of recursive units versus more layers of 'stacked' units, i.e. whose h's do not connect back to the gate parameters, but feedforward to an output layer.  It would be possible to combine recursive and stacked layers in the same architecture, potentially with the recursive depth varying with the stack layer.\n\n2. The manuscript should be checked for typos before the final version. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not relevant.", "review_text": "", "extra_scores": {"ethical_concerns": "Not relevant.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "_EgLtbjB_E6", "reviewer": "Reviewer_nEcJ", "summary": "This paper proposes a new architecture for sequence modelling. The idea is roughly: rather than have stacked layers as in a stacked LSTM, to have an adaptive recursion at each time step. The paper states the equations for this model then mainly presents the performance on a range of problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It's hard to evaluate this sort of paper. The writing is very clear and the presentation is good. But on the other hand, it is relatively easy to present what is mainly a set of fairly straight-forward experimental evaluations.\n\nBut, this is not to detract from the creativity of the model. The idea is compelling, and the performance looks interesting. But then again, no code is provided, and the results are therefore not easily reproducible and must be taken with a grain of salt. As the paper is purely experimental this is a key consideration. The response to checklist question 3A (\"Did you include the code, data, and instructions needed to reproduce the main experimental results\") in the paper checklist is yes, and yet ... there is no code. I would increase my review score if the authors could provide, say, an anonymous download link to code that reproduces the main SOTA results.\n\nIt is also unclear if the experimental comparisons are appropriate. One obvious example is \"Nested LSTMs\" by Moniz and Krueger. This paper has a similar idea (albeit with what looks to me like a less compelling execution than that of the paper under review) but is not compared with and not even cited.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "XPJXD1BlJsH", "reviewer": "Reviewer_VoSb", "summary": "The paper proposes a new type of recurrent unit, analogous to GRU and LSTM, dubbed Self-IRU. The underlying update is similar to that used by LSTMs and GRUs. However, the authors suggest to additionally incorporate the output of a different Self-IRU in the forget and output gates, similar to stacking LSTMs or GRUs in multiple layers. The authors propose that this allows the model to learn a \"dynamic soft recursion\". The paper performs empirical experiments on pixel-wise image classification, a synthetic logical inference task, sorting, tree traversal, music modeling, and semantic parsing tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe recurrent unit is similar to existing units like GRU and LSTM, but with novel components. In particular, the idea of using previous layers as inputs into the forget and output gates appears to be novel. However, it doesn't seem exceptionally different from the common practice of stacking recurrent networks, although one important difference is that Self-IRU only uses the output of the previous layer for the forget and output gates, rather than throughout the unit.\n\n## Quality\n- The paper has very extensive empirical results comparing the proposed recurrent unit to previous work, showing that it can work well in practice.\n- It seemed odd that the previous layer's output is used directly for the forget and output gates, rather than having separate linear projections for the gates. More generally, it would improve the paper if there was more explanation about how it was decided where to incorporate the output of the previous layer inside the computation of the current layer.\n- I believe that the framing of the work as recursion is not appropriate. Typically, recursion in computer science involves taking a larger input, reducing or dividing it into some way for the recursive call, and then combining the outputs of the smaller problems. There are no such elements of recursion in this work. The recurrent unit proposed in this work does not follow this kind of structure, and instead more closely resembles the use of depth in neural networks.\n- There should be a more direct comparison of how the work conceptually differs from stacked recurrent networks. If you have a multi-layer LSTM, then each layer would also use the previous layer's output in computing the forget and output gates, as in this work.\n- The ablation studies in Section 3.6 suggest that the relationship between max depth, base transformation, and the resulting performance seems scattered and unintuitive. It would be preferable if such experiments could be crafted to convincingly show that the additions in this work compared to previous work result in benefits, or if it were possible to explain why  Currently, it seems like a portion of the empirical improvement exhibited by Self-IRU could be attributed to more degrees of freedom in hyperparameters allowing for a broader hyperparameter search.\n\n## Clarity\nThe paper is clear, although I would prefer if some of the framing was different, as mentioned above.\n\n## Significance\nSelf-IRU appears fairly straightforward to implement so I think others can easily build upon the work, both for applications and for future designs on recurrent units. It's not clear if the empirical improvement is large enough over existing alternatives so that it would see significant use in applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper checklist cites Section 3.7 for discussion of its limitations. Section 3.7 discusses how the soft depth gate values vary rhythmically for a music dataset and \"according to the more complex color-image information\" for CIFAR-10. Especially for the music dataset, it would be useful if the paper can demonstrate that the varied gate values somehow corresponds to discernible intervals within the music, or if a similar conclusion can be reached on some kind of synthetic dataset.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a self-instantiated recurrent unit that is related to the LSTM but with additional capabilities for soft recursive. The authors evaluate their method on a range of tasks including image classification, logical inference, sorting, tree traversal, music modeling, semantic parsing, and code generation.

The reviewers and I agree that the evaluations are quite extensive. It's also clear the method performs well. The model presentation is mostly clear, but there were still a number of queries and points of confusion that popped up in the back-and-forth (relation to stacked LSTM, why only hidden and output gates determined recursively, which parameters are shared, etc.). The author rebuttal helped in this regard, but these points weren't completely resolved. I agree with R-VoSb that the ablation analysis isn't as revealing as it should have been in illuminating the architecture choices. Also, I found that section 3.7 wasn't very developed and didn't add much understanding. I wish it was clearer why and how the architecture works.

That said, I recommend acceptance, based on the strength of the experiments and the cleverness of the architecture. I hope the exposition will be further improved in the final version, as there are many helpful comments from the reviewers.
Decision: Accept (Poster)