Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Update: the authors have addressed my issues below and I have raised my score to 8.

***

This paper on recurrent neural networks goes back to Rosenblatt's continuous time dynamics model and uses a discretised version of that equation (equation (1) in the paper) to build an incremental version of the RNN, called incremental RNN, where the transition from hidden state h_{k-1} at step k-1 to next step's h_k is done using small incremental updates until the system achieves equilibrium. It claims that it manages to solve the vanishing gradient problem by keeping all gradients \frac{\partial h_k}{\partial h_{k-1}} equal to minus identity matrix. The algorithm is then extensively evaluated on a large number of tasks and compared with plain RNNs, LSTMs, and two recently published papers on antisymmetric RNN and FastRNN.

I need to admit that after reading the paper twice, I am not sure I understand how the method works exactly (how does inserting intermediary steps and variables g_0, g_1, ... g_T enable the system to reach equilibrium: is there an iterative evaluation until convergence?) and more worringly, how the single-step SiRNN differs from a normal RNN with an extra residual connection?

According to equation (5), for T=1 and g_0=0, we have:
g_T = g_1 = \eta_k^1 ( \phi (U (h_{k-1}) + W x_k + b) - \alpha h_{k-1} ).
If the gradients are vanishing in the normal RNN, why would they not vanish here for T=1?

Propositions 1 and 2 are for the case K=\infinity, and I could not understand the proof of theorem 1 that shows why \frac{\partial h_k}{\partial h_{k-1}} = - I. This seems to be the major contribution of the paper and should be given prominence.

What is missing is clear explanation, like (Bengio et al, 1994), of the identity gradient and of how the algorithm works. These questions could be solved by including code or pseudo-code explaining how to actually implement incremental RNNs.

There are also several important papers recently published that have approached the problems of continous-time dynamics and relaxation of hidden state to equilibria.
* The paper does not mention at all Neural ODEs [2] [3] where the state flows in a continuously differentiable way thanks to the continuous-time residual network ODE formulation. Moreover, isn't the idea of inserting a relaxation to equilibrium using ODEs already implemented in the ODE-RNNs [3]?
* How do incremental updates related to Adaptive Computation Time [4]?

For this reason, I am currently tending to reject the paper, but am open to change my score upon clarifications and links to other similar work.

Additional remarks:
The first paragraph of the paper explains the Elman RNN, not RNNs in general.
Please cite [1] alongside Bengio et al (1994) for the problem of the vanishing gradient.
Define alpha in equation (1)
Notation k and K is very confusing
Blue vs. green on Figure 2 is hard to read, and where is the new initialisation?
Why do you add h_k^K to h_{k-1} in equation (8)? I thought it was g_k^K?
Keep the same colours for all experiments in figure 4.

[1] Hochreiter (1991) "Untersuchungen zu dynamischen neuronalen Netzen"
[2] Chen, Rubanova, Bettencourt & Duvenaud (2018) "Neural Ordinary Differential Equations"
[3] Rubanova, Chen & Duvenaud (2019) "Latent odes for irregularly-sampled time series"
[4] Graves (2016) "Adaptive computation time for recurrent neural networks"

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose the incremental RNN (iRNN), which is inspired by the continuous-time RNN (CTRNN). Theoretically, the equilibrium point of iRNN exists and is unique. Furthermore, the norm of the Jacobian between two hidden states is always one, provided that the Euler iterations converge. The authors proved this property as well as the exponential convergence rate of the Euler iteration. These properties avoid the vanishing/exploding gradient problem typical in RNN with long sequences in theory. Empirically, the proposed method is compared with multiple RNN architecture on various tasks. 

The paper is clearly written and well organized. It starts with the existence and uniqueness of a fixed point; then provides the converge rate, and finally the main theorem of identity gradient. The main theorem provides a good theoretically guaranteed solution. The authors have done extensive experiments on several datasets compared against various popular and recently developed RNN-based models. The authors also provided some empirical experiments on the converge rate and the non-singularity of grad \psi.

The motivation of the paper could be improved. For example, despite the appealing theoretical properties. I did not fully understand the motivation in Equation 3. It would be great if the authors could further elaborate on it.

Typos:
Page 4 at “Upon computation, we see that”. The gradient on the left-hand side should be grad \psi? 
Page 4 at “at the equilibrium points we have”, the gradient of \psi should be the gradient of \phi?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors present a novel work to address the problem of signal propagation in the recurrent neural networks. The idea is to build a attractor system for the signal transition from state h_{k-1} to h_k. If the attractor system converges to a equilibrium, then the hidden to hidden gradient is an identity matrix. This idea is elegant. The authors verify the performance of Increment RNN on long-term-dependency tasks and non-long-term-dependency tasks.

The work successfully constructs a negative identity hidden to hidden gradient matrix but I still have a few concerns about the theory and the experiments. If the authors can address my concerns in the rebuttal, I am willing to increase my score. 

Theoretical concerns:
Even in the limit sense, the inner ordinary differential equation of g will converge to the equilibrium, it may not converge in the finite steps. Thus, the state-to-state gradient may be slightly away from the identity. And we know that (0.99)^T goes to infinity when T goes to infinity. In practice, the long-term-gradient problem may still exist in the incremental RNN.

Experimental concerns:
The theorem requires the norm of U to be bounded. But I cannot see how the authors bound the norm of U in the experiment section.

Clarity of writing:
There are a bunch of typos in the papers, especially the ones in the proof of Theorem 1. The proof of theorem 1 can be polished. The author swap the use of phi and psi several times in the proof. And gradient calculation on equation (5) should be expanded into more details. The authors also missed one U in the equation in the second last line of the proof. 

Overall, I think the paper is an interesting contribution to the community.

review_title: Official Blind Review #2

Meta-review: In this paper, the authors propose the incremental RNN, a novel recurrent neural network architecture that resolves the exploding/vanishing gradient problem. While the reviewers initially had various concerns, the paper has been substantially improved during the discussion period and all questions by the reviewers have been resolved. The main idea of the paper is elegant, the theoretical results interesting, and the empirical evaluation extensive. The reviewers and the AC recommend acceptance of this paper to ICLR-2020.
Decision: Accept (Poster)