Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the recurrent neural networks with injecting noise to hidden states. They consider RNNs as data driven discretization of stochastic differential equations. Analysis for global stability is presented, and it is also shown that noise injection may improve training stability.
main_review: The analysis is based on the discretization of stochastic differential equations by adding the diffusion term on top of ordinary differential equations. With certain assumptions, the authors adopt this analysis to study the implicit regularization effect of noise injection schemes in the small noise regime, which promotes flatter minima, biases towards models with more stable dynamics, and favors larger classification margin in classification. 

The approach of injecting noise in learning the highly non-convex problems can be an effective approach for obtaining more stable and generalizable models. The authors demonstrate a theoretical evidence for RNNs with injecting noise in states by applying the popular tools of differential equations, which marginally promotes our understanding in this regime. The empirical study also shows promising results of the proposed approach compared with existing ones, where under higher noise scenario the proposed approach significantly outperforms.

Though theoretically this is an interesting problem and remains with largely unresolved issues, the presented study does not reveal much new insights in the problem. The paper is relatively easy to follow and clearly presented. The presented proof is mostly clear and seems correct, though I did not check all detailed analysis for the proof. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies RNNs that are trained with noise injection in the framework of ODEs.  The method is called NRNN. The authors provided Theorems to: (1) identify the form of implicit regularization of NRNN; (2) show the improved model stability; (3) provide the bounds for RNN classifiers. The experiments are conducted on two classification tasks: Sequential MNIST and heart beats classification.  
main_review: The idea of viewing noise injection as stochastic learning is interesting. In general, this work is well presented, the main claims are well warranted by theoretical analysis and empirical results. In terms of the noise level in eq.(4), how to ensure the value \epsilon is small? How do the different combinations of the tunable parameters \sigma_1 and \sigma_2 influence the overall performance in Table 1-3? 

The experimental results of robustness and resilience are interesting, NRNN presents better performance as compared to other methods while maintaining comparable predictive performance. I am wondering if this robustness can be held as well in the regression task?

limitations_and_societal_impact: no
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies noise injection in RNNs, which can be regarded as a discretized SDE driven by external inputs. The first main result of the paper is the derivation of the implicit regularization caused by noise injection (similar to noise injection for feed-forward models), which is shown to be related to the Hessian of the loss (promoting flatter minima) and the Jacobian of the dynamics (promoting slower/more stable dynamics). A further implication on the classification margin is presented. Lastly, the paper presents a result showing that noise injection can further improve the temporal stability of the RNN dynamics. These motivate the use of the noise injection for recurrent neural networks, and experiments are provided to justify the theoretical results.
main_review: This paper is well written and the topics is of interest to the RNN/time series modelling community. Noise injection and its regularization effect is worthy of investigation, and in this sense, the main results of this paper presents advancements in this area. Below, I list some questions that the authors may wish to address:

1. It is well established in non-recurrent (or non-deep) models that noise injection leads to implicit regularization. Thus, it may be good to summarize the new phenomena arises from the dynamical setting you consider. One effect I can see from Theorem 2 is that the margin depends on the Jacobian of the dynamics - stable dynamics gives larger margin. I thought this is interesting. Are there any other differences between this dynamical setting and the usual setting in supervised learning?

2. Theorem 2: the result suggests that “stable” dynamics have bigger margins. There seems to be much more straightforward ways to achieve this, i.e. setting A in (3) to be very negative. Is the theorem saying something more than this?

3. Experiments

   1. Since the earlier results suggest that RNN with noise injection can be understood as implicitly regularization with the form of the regularizer as in (16), what if one simply adds a term such as (16) to the loss function. Does it achieve the same effect?

   1. Are there experiments to demonstrate the additional stabilization effect of noisy RNNs, i.e. Theorem 3?

---

After rebuttal: thank you for responding to my questions and my evaluation is unchanged.
limitations_and_societal_impact: Not applicable
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides a theoretical framework for studying training of recurrent neural networks by injecting noise into hidden states. 
main_review: The paper is very well written and organized. Some valuable technical results are provided to study discretization schemes of stochastic differential equations modeling the injection of noise into networks’ hidden states. These results justify the regularization effect of the injection scheme. This is an important result also illustrated by some representative simulations. They also clearly explain what the limitations are, such as the RELU activation function does not satisfy the conditions required in this paper.
limitations_and_societal_impact: They didn't o it explicitely but I cannot see how this work can cause any negative societal impact.
review_title: null

Meta-review: This paper received overall favorable reviews. This paper was found to be well written and the topic to be of interest and innovative.


Please address reviewers comment in the final version.
Decision: Accept (Poster)