Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper explores a very important question in dynamical system identification of how to make recurrent neural networks (RNNs) learn both long-term and short-term dependencies without the gradient vanishing or exploding limitation. They suggest using piece-wise linear RNNs (PLRNNs) with a novel regularization technique.

The paper is well written and is very thorough with the necessary theoretical foundation, numerical experiments and analysis. 

I think the theory and results of this paper are significant and will be relevant to further our understanding of RNNs and system identification.

Major points:

1) L2 weight regularization can be easily applied to any of the RNN models used in the experiments. While other weight initialization schemes were compared to the paper's proposed model (rPLRNN), none of the other RNN models had similar regularization. This will shed some light on whether it is indeed the proposed regularization that matters or the full proposed model with PLRNN and a mix of regularized and non-regularized units.

2) It is not clear to me how one can choose the correct ratio of regularized vs unregularized units in the model. While the amount of regularization clearly helps in reducing training error as shown in Figure 3, increasing the ratio of regularized units in Figure S3C did not help the error past 0.1 and then larger values resulted in large increases such that the error at ratio 1 is equivalent to the error at ratio 0. Perhaps this observation is specific to the addition problem, but I feel that a discussion of the effect of this ratio on performance should be included for clarity. Additionally, the ratio of regularized units with best performance could potentially be different for different regularization amounts.

Minor Point:
g is not defined in equation 2
review_title: Excellent paper proposing a novel regularization technique to ReLU-based RNNs for dynamical system identification 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a type of regularization for piecewise linear RNNs (PLRNNs) that encourages the network to learn line or plane attractors. The paper argues, through mathematical analysis of the regularized network as well as numerical experiments, that this regularization alleviates the vanishing and exploding gradient problem and allows PLRNNs to reconstruct nonlinear dynamical systems with multiple timescales from noisy observations.

Major concern:
I found the paper interesting. My main concern has to do with the motivation of learning line or plane attractors. The paper argues that the proposed regularizer will improve performance *through a specific mechanism: encouraging plane attractors in the dynamics*. While the results in the experiments section are impressive, the paper as far as I can tell does not establish that the regularized PLRNNs have learned plane attractors.

For example, it could be that simply adding l2 regularization on all of the weights would also lead to better performance (rather than the specific regularization proposed), or perhaps the better performance arises from some other (undiscovered) mechanism. In fact, to show that the specific form of l2 regularization is what is useful, I think having a PLRNN with standard l2 regularization (applied to all of the weights) is a critical missing baseline.

To convince me that the benefit is really due to the given motivation (encouraging plane attractors), I want to see evidence that regularized PLRNNs have learned plane (or line) attractors, compared to unregularized PLRNNs (or better yet, PLRNNs with l2 regularization applied to all of the weights). This can be demonstrated in a number of ways, for example by showing eigenvalues of the Jacobian around fixed points of trained PLRNNs with and without the proposed regularization, on the three tasks in Fig. 2. If this was conclusively demonstrated, I would happily increase my rating.

For comparison, recent work showed that RNNs of multiple types (including vanilla RNNs and LSTMs) learned line attractors when solving an NLP task [1]. These line attractors were found in the networks after training, and did not have any special regularization encouraging them. I would appreciate if the authors would add some discussion comparing their regularization to these discovered line attractors.

[1] Maheswaranathan et al, NeurIPS 2019 (http://papers.nips.cc/paper/9700-reverse-engineering-recurrent-networks-for-sentiment-classification-reveals-line-attractor-dynamics)

Other concerns:
- How is Mreg (the number of regularized dimensions) chosen? How does the performance vary as a function of Mreg?
- What's g in eq(2)?
- I was a little surprised that the vanilla RNN worked so well in Fig 2A. I would have expected an LSTM to work as well, if not better. Do the authors have intuition for why this is?

Minor typos:
- Quotes around automatize in first and last paragraphs of section 1 are incorrect. should be `automatize' and `classical'. in latex, use the backtick character (`) for the first quote.
- The abbreviation "RNN" is used as if it is plural, but it is more commonly singular. (e.g. RNNs seem like instead of RNN seem like; or RNNs in their vanilla form instead of RNN in their vanilla form)
review_title: Interesting paper, but I'm not convinced that the proposed motivation (encouraging plane attractors) is the reason for the improved performance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a novel regularization term to PLRNN. PLRNN has nice numerical properties given its simple mathematical structure, but is able to capture complicate dynamics. It's also easy to establish a theoretical connection between PLRNN dynamics and the behavior of its gradients, which is nice. Given such a dynamic model, they design novel L2 terms pushing partial parameters towards zero, which leads to a line/plane attractor that allows slow time constants for long short-term memory. I think the idea is quite novel and interesting.

A few concerns:
1. It might be a bit jumpy for readers outside neuroscience to establish the connection between working memory and short-term memory. I would say making this point more explicit in the intro would be helpful.
2. Lack of definition of G at the beginning of 3.1.
3. It would be helpful to point out that eq 1 is PLRNN (with full name spell spelled out).
4. Section 4.1 is a bit confusing to me. It's unclear that why the higher the mse/cross entropy is, the better is the model should be; also the lower the Pcorrect value is, the better the model is. Moreover, I think it's still necessary to give the definition of all the model names included in the main text. Some are missing, e.g. iPLRNN, oRNN, maybe just very briefly. Just expect that not all readers would read the appendix.
5. The paper seems to care more about "interpretable", which is not clearly reflected in the paper. Figure 3 only shows the reconstruction. But it would be more interesting to visualize the latents of this neuron model's dynamic. It's mentioned that M={8,...,18} states were trained. How do they look like? Some are more line-attractors and some are more related to the fast spiking dynamics? Which state number is finally picked? Figure 3 only shows how influential \tao is. but there should be other insightful ablation studies to be done in order to understand the regularization term as well, i.e. how many latent states, how to split the two types, etc.
6. It would also be helpful to visualize the reconstruction and the latents for other RNN/LSTM models for the neuron model as well. That helps to show the advantage of the proposed model more clearly. 

In sum, I think the idea is quite interesting and practically useful. I also appreciate the theoretical analysis. But the experiment section is confusing by missing some explanations. And the presentation of the neuron model is not sufficient enough to prove that rPLRNN find interesting and interpretable dynamics. That's why I think it's a bit below a good paper. 





review_title: an interesting novel regularization term that encourages long short-term memory for RNN

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a regularization scheme for training vanilla Relu RNN to tackle the exploding and vanishing gradients issue. The work eases the analysis of RNN in the dynamical system point of view and connects the RNN dynamics and gradient theoretically. The experiments show the competitive performance comparing to LSTM.

The vanilla RNNs simplify the analysis as dynamical systems without those gates in LSTMs and GRUs while suffering exploding or vanishing gradients. The idea of tackling the issue  while keeping the simplicity sounds very interesting and useful.

I am leaning on accepting this manuscript if the authors could address my concerns.

- Sec 3.2. The authors mentioned the connection between the particular setting of RNN and working memory. This setting leads to a system without any autonomous dynamics that is not a typical model for working memory (e.g. attracting fixed-points, line attrators and etc.). I disagree that the space is neurally stable since all the states are sensitive to perturbation and cannot persist a stable memory.

- Sec 3.2. Following the so-called "neurally stable" setting, why the term on A in Eq.3 only regularizes the diagonal rather than penalizes the deviation from the identity matrix? This regularization does not lead to the tendency of A -> I mentioned in the text.
Furthurmore, the regularization does not guarantee either A -> I or W -> 0 so that the resulting Mreg subspace does not have the "memory" property.

- Eq.3. How is Mreg determined?

Minor:

- iPLRNN is used before defined.


review_title: Weak Recommendation to Accept

Meta-review: This paper describes a clever new class of piecewise-linear RNNs that contains a long-time scale memory subsystem. The reviewers found the paper interesting and valuable, and I agree. The four submitted reviews were unanimous in their vote to accept. The theoretical insights and empirical results are impactful and would be suitable for spotlight presentation.
Decision: Accept (Spotlight)