Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents some novel contributions regarding recurrent neural networks. 
Building on the work of Chang et al. (2019),  the authors provide a global convergence result for the hidden representation of a family of recurrent neural networks using standard techniques from the Lyapunov analysis of dynamical systems.
The requirements of the theorem are met  (within the limits of discretization) by their proposed algorithmic scheme. 
Numerical evaluation on a variety of benchmarks shows that the proposed algorithm yields systematic improvement over other RNN approaches. 
For all of the above reasons, I recommend the acceptance of the paper. 

Some concerns to be addressed:
- The connection between stability and trainability or refer to Chang et al. (2019) if their analysis applies here. 
- specify the functions \sigma_min and \sigma_max used in Theorem 1
- specify the meaning of the one-arg function f(h^*) as opposed to the 2-arg f(h,t) appearing in Definition 1. 

review_title: A solid contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: I have increased my score to reflect the revisions

---------

This paper presents yet another architecture for fully connected RNNs or infinitely deep networks based on the integration of a continuous time dynamical system, where a projection of the weights is used to guarantee stability, hence a fixed point and a finite Lipschitz constant. 

Positives: The maths presented in the paper is correct and their results are nicer than the considered (not exhaustive) baselines.  

Negatives:

1) This idea has been widely explored and exploited by now. Adding a linear term and using a different solver is not enough in my opinion to make an innovative contribution. 

If you wish to present this as an ablation study, then perhaps you need to benchmark against existing solutions.
 
For instance, in this (missing) reference a very similar network is presented and analysed. 

@incollection{NIPS2018_7566,
title = {NAIS-Net: Stable Deep Networks from Non-Autonomous  Differential Equations},
author = {Ciccone, Marco and Gallieri, Marco and Masci, Jonathan and Osendorfer, Christian and Gomez, Faustino},
booktitle = {Advances in Neural Information Processing Systems 31},
pages = {3025--3035},
year = {2018},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/7566-nais-net-stable-deep-networks-from-non-autonomous-differential-equations.pdf}
}

I will refer to this as [1]. While this paper was about unrolling the stable RNN to generate a deep Lipschitz classifier, and was not used for sequence to sequence task, the architecture you propose and the claims are so much similar to [1] that this demands for a direct comparison. 

In the above paper, stability projections are presented for an architecture that is essentially the same, minus the additional linear component here. Paper [1] should be included as a baseline. You have it already implemented for fully connected layers. 

2) It is not very clear how this additional linear component would help in practise, as your architecture is fundamentally discretised with Euler which results in yet another generalised res-net. It has been shown that ResNet works much better than their predecessor,  Highway networks, because of the direct skip connection and better gradient flow. 
While the missing reference [1] (NAIS-Net) preserves that connection, It feels like your linear term would get rid of the skip connection and prevent the technique from being used in very deep networks or very long sequences due to vanishing gradients. 

3) What is the motivation for using a continuos-time approach? Your results are invalidated by the forward Euler, unless a very small hyperparameter epsilon is introduced. Why not just compute the projection in discrete time as done in [1]. Your stability will not hold for sparse in time data-points, because the Euler step would become too big and this is effectively an RNN that cannot handle different sampling time while preserving stability. 

4) It seems strange to compare to NODEs as they are meant to be used for something else. In particular, NODEs are designed to work without inputs but just by estimating the initial condition for the ODE and then "unrolling". 
If you add input signals to the NODE, which I guess is what you mean by "NODE RNN", how do you train it with the adjoint method? This should be made very clear in the paper. 

5) Your results are limited to a fully connected architecture, while [1] has shown a method to have a Lipschitz RNN for convolutional layers. Can you generalize to that as well?

I don't feel the contribution here is relevant enough to be included in the conference. 

If the above points are clarified in a convincing way and both the theoretical justification and the ablations performed with respect to [1], then I could consider improving my score. 

review_title: Missing references - Not clear what is the improvement over existing architecture 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed a new continuous-time RNNs which appears to be extremely similar to CT-RNN (Funahashi et al. 1993). They then constraint the network representation to account for learning long-term dependencies. 

Positive: The formulation of the constraint is very clear and sound. 

Positive: The analysis of the stability of the model and other properties is rigor enough and to the best of my knowledge sound and correct. 

Negative: From the experimental setting, reported values in tables, and code, it seems like the authors tuned the hyperparameters on the test set which I consider a bad practice that violates the code of conduct! I suspected this, and therefore, ran the code myself. With a few changes to the hyperparameters from the tuned one reported in the table the performance of the proposed model dropped significantly!

I would suggest the authors to create a fair testing scheme for all baselines and report the experimental results more accurately.
review_title: CT-RNNs with stability conditions for network parametrization

final_score: rating: 7: Good paper, accept
review_content: Considering a continuous time RNN with Lipschitz-continuous nonlinearity, the authors formulate sufficient conditions on the parameter matrices for the network to be globally stable, in the sense of a globally attracting fixed point. They provide a specific parameterization for the hidden-to-hidden weight matrices to control global stability and error gradients, consisting of a weighted combination of a symmetric and a skew-symmetric matrix (and some diagonal offset). The authors discuss numerical integration by forward-Euler and RK2, and thoroughly benchmark their approach against a large set of other state-of-the-art RNNs on various tasks including versions of MNIST and TIMIT.  Finally, they highlight improved stability of their RNN against parameter and input perturbations.

What I like about this paper is that it provides a solid theoretical basis and a principled, insightful parameterization that let’s one control separately the size of the real and the imaginary parts of the eigenvalues of A and W. The paper contains a number of interesting thoughts, and a really extensive comparison to other state-of-the-art models on several benchmarks.

In general, I feel however that this work is relatively close to the 2019 ICLR paper by Chang et al.; in several ways it feels like a more or less straightforward extension of this previous work.

It also remains a bit unclear to me how it’s ensured in practice that the matrices obey to the required conditions in the training process. Sect. 5 is not really about training, but just about numerically solving the ODE. From Appendix C it seems the scalar parameters $\beta, \gamma$ controlling the influence of the symmetric vs. skew-symmetric parts and the offset are not learned at all but just fixed after grid-search? The component matrices B, C, on the other hand it seems are not restricted at all but just initialized such that the theoretical conditions are likely, but not necessarily, met? This seems somewhat unsatisfying as there are in fact no guarantees that the global stability conditions will be met in practice, and tuning the model may require (potentially extensive) meta-parameter search?

Another drawback in my mind is that enforcing global fixed point dynamics is quite restrictive. For instance, this rules out cycles and many other interesting dynamics in the model’s intrinsic behavior. Apparently, from the authors’ empirical tests, this seems not to be required for solving this particular set of tasks. Which is somewhat puzzling to me as it appears this assumption should strongly curtail the model’s expressiveness.

In the tables and figures I missed statistics. No standard errors or confidence bands were provided, or how many runs were performed. If it’s all from a single run, can I be certain the numbers are not just lucky draws?

Nevertheless, given the overall convincing and extensive empirical results, I’m slightly leaning toward acceptance.

Minor issues:
- The authors sell the additional linear term in their RNN as a novelty, while in fact it’s rather standard in continuous RNN (older papers by Barak Pearlmutter, Song & Wang 2016, arxiv.org/abs/2006.02427, arxiv.org/abs/1910.03471)
- RK2 is generally not sufficient for more involved (stiff) dynamical problems; so the reason it works well here may lie in the fact that the model’s intrinsic dynamic is indeed very simple
- What is an unstable unit? I guess the authors mean that the RNN is not globally stable?
- Sect. 2,  dynamical systems inspired RNN: It may be important to note that formulating a RNN as ODE does not solve the exploding/vanishing gradient or stability problem per se (nor is it immediately clear to me why it should actually make it easier).
- Theorem 1: The $\sigma$ refer to the matrix eigenvalues in this case?
- What is a superset of skew-symmetric matrices?
- Second-to-last pg. of Sect. 7 was unclear to me, i.e. what exactly was done and evaluated here, maybe cos I’m not familiar with some of the cited methods.
review_title: Solid and well-done paper, but quite close to previous work, and with some caveats from a dynamical systems perspective.

Meta-review: Solid work on extending AntisymmetricRNN and expanding its expressivity while controlling the global stability of the recurrent dynamics. It contributes to the growing interest in continuous-time RNN formulations that can deal with exploding gradient problem, and worthy of ICLR poster presentation. Three reviewers were positive and one was slightly negative. Authors added additional experiments and strengthened the manuscript significantly during the review process.
Decision: Accept (Poster)