Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce a recursive neural network architecture, using a novel kind of unit termed a 'self-instantiated recurrent unit'.  This architecture enables the gate parameters within an LSTM-like architecture to themselves be determined by units at a lower recursive layer, and the level of recursion to vary dynamically.  The authors test the model empirically on multiple tasks, including image classification, logical inference and music sequence modeling. 
main_review: The model architecture is an interesting novel variation on the LSTM architecture.  The ability to dynamically learn the recursive depth is also an interesting idea, and as the authors propose, may be interpretatively relevant In various domains.  The presentation is clear, and the authors offer a thorough experimental investigation of the architecture on multiple domains, achieving state-of-the-art results on several datasets, along with an ablation study, and analysis of the variation in recursive depth across domains, and comparisons with alternative architectures.

Minor points:

1. A potentially interesting comparison would be to compare the relative strengths of including more layers of recursive units versus more layers of 'stacked' units, i.e. whose h's do not connect back to the gate parameters, but feedforward to an output layer.  It would be possible to combine recursive and stacked layers in the same architecture, potentially with the recursive depth varying with the stack layer.

2. The manuscript should be checked for typos before the final version. 
limitations_and_societal_impact: Not relevant.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new architecture for sequence modelling. The idea is roughly: rather than have stacked layers as in a stacked LSTM, to have an adaptive recursion at each time step. The paper states the equations for this model then mainly presents the performance on a range of problems.
main_review: It's hard to evaluate this sort of paper. The writing is very clear and the presentation is good. But on the other hand, it is relatively easy to present what is mainly a set of fairly straight-forward experimental evaluations.

But, this is not to detract from the creativity of the model. The idea is compelling, and the performance looks interesting. But then again, no code is provided, and the results are therefore not easily reproducible and must be taken with a grain of salt. As the paper is purely experimental this is a key consideration. The response to checklist question 3A ("Did you include the code, data, and instructions needed to reproduce the main experimental results") in the paper checklist is yes, and yet ... there is no code. I would increase my review score if the authors could provide, say, an anonymous download link to code that reproduces the main SOTA results.

It is also unclear if the experimental comparisons are appropriate. One obvious example is "Nested LSTMs" by Moniz and Krueger. This paper has a similar idea (albeit with what looks to me like a less compelling execution than that of the paper under review) but is not compared with and not even cited.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new type of recurrent unit, analogous to GRU and LSTM, dubbed Self-IRU. The underlying update is similar to that used by LSTMs and GRUs. However, the authors suggest to additionally incorporate the output of a different Self-IRU in the forget and output gates, similar to stacking LSTMs or GRUs in multiple layers. The authors propose that this allows the model to learn a "dynamic soft recursion". The paper performs empirical experiments on pixel-wise image classification, a synthetic logical inference task, sorting, tree traversal, music modeling, and semantic parsing tasks.
main_review: ## Originality
The recurrent unit is similar to existing units like GRU and LSTM, but with novel components. In particular, the idea of using previous layers as inputs into the forget and output gates appears to be novel. However, it doesn't seem exceptionally different from the common practice of stacking recurrent networks, although one important difference is that Self-IRU only uses the output of the previous layer for the forget and output gates, rather than throughout the unit.

## Quality
- The paper has very extensive empirical results comparing the proposed recurrent unit to previous work, showing that it can work well in practice.
- It seemed odd that the previous layer's output is used directly for the forget and output gates, rather than having separate linear projections for the gates. More generally, it would improve the paper if there was more explanation about how it was decided where to incorporate the output of the previous layer inside the computation of the current layer.
- I believe that the framing of the work as recursion is not appropriate. Typically, recursion in computer science involves taking a larger input, reducing or dividing it into some way for the recursive call, and then combining the outputs of the smaller problems. There are no such elements of recursion in this work. The recurrent unit proposed in this work does not follow this kind of structure, and instead more closely resembles the use of depth in neural networks.
- There should be a more direct comparison of how the work conceptually differs from stacked recurrent networks. If you have a multi-layer LSTM, then each layer would also use the previous layer's output in computing the forget and output gates, as in this work.
- The ablation studies in Section 3.6 suggest that the relationship between max depth, base transformation, and the resulting performance seems scattered and unintuitive. It would be preferable if such experiments could be crafted to convincingly show that the additions in this work compared to previous work result in benefits, or if it were possible to explain why  Currently, it seems like a portion of the empirical improvement exhibited by Self-IRU could be attributed to more degrees of freedom in hyperparameters allowing for a broader hyperparameter search.

## Clarity
The paper is clear, although I would prefer if some of the framing was different, as mentioned above.

## Significance
Self-IRU appears fairly straightforward to implement so I think others can easily build upon the work, both for applications and for future designs on recurrent units. It's not clear if the empirical improvement is large enough over existing alternatives so that it would see significant use in applications.
limitations_and_societal_impact: The paper checklist cites Section 3.7 for discussion of its limitations. Section 3.7 discusses how the soft depth gate values vary rhythmically for a music dataset and "according to the more complex color-image information" for CIFAR-10. Especially for the music dataset, it would be useful if the paper can demonstrate that the varied gate values somehow corresponds to discernible intervals within the music, or if a similar conclusion can be reached on some kind of synthetic dataset.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel gated recurrent unit module that accomplishes a form of soft recursion through dynamic gating. The experiments evaluate the module across a range of tasks, and in particular show promising results on a logical inference task.
main_review: This paper proposes a clever way to accomplish a form of soft recursion, by introducing a recurrent module, the 'self instantiated recurrent unit' (self-IRU), that can gate other instances of itself. I found the idea to be well motivated and the experiments seem to demonstrate improvements across a range of domains (though see the caveat below about statistical significance and multiple seeds), particularly on a logical inference task.

I found some parts of the proposed module difficult to understand, and hopefully this can be clarified in the revision. In particular, what is the relation of index $n$ (for the depth gates) to the index $l$ (for the depth layer)? The paper states that $l$ takes on values from $1$ to $L$, the maximum allowed depth (a hyperaparameter), but what values do $n$ take on, and how does that relate to the recursion depth? It may help to clarify this if the values of $n$ and $l$ are labelled in Figure 2, i.e. node 0 and layer 0 could be labelled with superscripts $(n=0)$ and $(l=0)$. 

I'm also curious about a few of the design decisions, and am wondering whether the authors considered any of the following variations, or can offer any insight into the motivation behind these particular decisions:
* Rather than having the input to $z_{t}^{(l)}$ come from the original input $x_{t}$, instead basing this on the output of the Self-IRU in the previous layer, as is done for the forget and output gates.
* Sharing parameters for the base transformations $F$ across layers.
* Computing the depth gates ($\alpha$ and $\beta$) using a recurrent layer instead of a single feedfoward layer.

I did have two concerns. These concerns are the primary reason for my score, which I would be happy to raise if they are addressed:

1) Are the results shown for a single random seed only, or averaged over multiple seeds? If a single seed, was only one seed used, or are these results for the best seed, and if so out of how many? I think the ideal approach would be to average over multiple seeds and present some measure of error, but at the very least this should be clarified. On some of the datasets the improvements were very small and it's not clear if they are even statistically significant.

2) This seems somewhat unlikely to me, but I'm curious to what extent the results can be explained just by additional depth. How would the proposed approach compare to a multilayer LSTM with the same number of layers as a Self-IRU? One reason for thinking that this might partially explain the results is that the stacked BiLSTM performs almost identically to the Self-IRU on the tree traversal task.

I found the results on the logical inference task very impressive. It makes a lot of sense that an inductive bias for recursion would be helpful for such a task. Have the authors considered visualizing the model's operation on a few examples from this task? That seems like it might be an intuitive way to convey how the model operates, and why it's useful for such tasks.

Minor points:
* It would be good to include more detail about hyperparameters, including the optimizer, learning rate, initialization etc. This could go in the Appendix.
* I'm curious about what the relation might be, if any, to the following paper [Kriete et al.], in which one layer of a network is used to gate the output of another layer, and therefore accomplishes a form of indirection. I wonder if something similar could happen with the Self-IRU.

Kriete, T., Noelle, D. C., Cohen, J. D., & O’Reilly, R. C. (2013). Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, 110(41), 16390-16395.

**Update and final summary**

This paper proposes a novel recurrent gated unit that implements a soft form of recursion. There is a thorough empirical evaluation across a range of tasks, showing that the proposed Self-IRU generally results in improvements. Though the improvements are small on some tasks, the improvements on the logical inference task are particularly impressive, especially against a baseline that has access to ground-truth syntactic information.
limitations_and_societal_impact: None.
review_title: null

Meta-review: This paper introduces a self-instantiated recurrent unit that is related to the LSTM but with additional capabilities for soft recursive. The authors evaluate their method on a range of tasks including image classification, logical inference, sorting, tree traversal, music modeling, semantic parsing, and code generation.

The reviewers and I agree that the evaluations are quite extensive. It's also clear the method performs well. The model presentation is mostly clear, but there were still a number of queries and points of confusion that popped up in the back-and-forth (relation to stacked LSTM, why only hidden and output gates determined recursively, which parameters are shared, etc.). The author rebuttal helped in this regard, but these points weren't completely resolved. I agree with R-VoSb that the ablation analysis isn't as revealing as it should have been in illuminating the architecture choices. Also, I found that section 3.7 wasn't very developed and didn't add much understanding. I wish it was clearer why and how the architecture works.

That said, I recommend acceptance, based on the strength of the experiments and the cleverness of the architecture. I hope the exposition will be further improved in the final version, as there are many helpful comments from the reviewers.
Decision: Accept (Poster)