Noisy Recurrent Neural Networks
Authors: Soon Hoe Lim
N. Benjamin Erichson
Liam Hodgkinson
Michael W. Mahoney
Venue: NeurIPS 2021 Poster (2021)
Keywords: Recurrent neural networks
implicit regularization
stochastic differential equations
stability analysis
model robustness

Abstract: We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, it favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection can improve stability during training. Our theory is supported by empirical results which demonstrate that the RNNs have improved robustness with respect to various input perturbations.

Noisy Recurrent Neural Networks
Soon Hoe Lim Nordita, KTH Royal Institute Technology and Stockholm University soon. hoe. limesu.se
N. Benjamin Erichson University of Pittsburgh School of Engineering erichson@pitt. edu
Liam Hodgkinson
Michael W. Mahoney
ICSI and Department of Statistics UC Berkeley
ICSI and Department of Statistics UC Berkeley
liam. eyw..oogkiinsnnnbbeeeeee.......e edu
mmahoney@stat. beaboneeysttat..eekeeleeeye edu
Abstract
We provide general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can viewed as discretizations of stochastic differential equations driven by input data. This framework allows to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection improve stability during training. Our theory is supported by empirical results which demonstrate that the RNNs have improved robustness with respect to various input perturbations.
Introduction
Viewing recurrent neural networks (RNNs) as discretizations of ordinary differential equations (ODEs)
driven by input data has recently gained attention 27 16 49]. The "formulate in continuous time, and then discretize" approach (38] motivates novel architecture designs before experimentation, and it provides useful interpretation as a dynamical system. This, in turn, has led to gains in reliability and robustness to data perturbations.
Recent efforts have shown how adding noise can also improve stability during training, and con- sequently improve robustness [35]. In this work, we consider discretizations of the corresponding stochastic differential equations (SDEs) obtained from ODE formulations of RNNs through the addition diffusion (noise) term. We refer to these as Noisy RNNs (NRNNs). By dropping the noisy elements at inference time, NRNNs become stochastic learning strategy which, prove, has a number of important benefits. particular, stochastic learning strategies (including dropout) are often used natural regularizers, favoring solutions in regions of the loss landscape with desirable properties (often improved generalization and/or robustness). This mechanism is commonly referred to implicit regularization (40, 39 50), differing from explicit regularization where the loss is explicitly modified. For neural network (NN) models, implicit regularization towards wider minima is conjectured be a prominent ingredient in the success of stochastic optimization [67 28 Indeed, implicit regularization has been linked to increases in classification margins [47]. which can lead improved generalization performance [5T] A common approach to identify and study implicit regularization is to approximate the implicit regularization by an appropriate explicit regularizer
shall
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

40 39 1 6 21]. Doing so, we will see that NRNNs favor wide minima (like SGD); more stable dynamics; and classifiers with a large classification margin, keeping generalization error small.
SDEs have also seen recent appearances in neural SDES [59 [4], stochastic generalizations of neural ODES [ which can be seen as analogue of NRNNs for non-sequential data, with similar relationship to NRNNs as feedforward NNs do to RNNs. They have been shown to be robust in practice [35]. Analogously, we shall show that the NRNN framework leads to more reliable and robust RNN classifiers, whose promise is demonstrated by experiments on benchmark data sets.
Contributions. For the class of NRNNS (formulated first as a continuous model, which is then discretized), the following are our main contributions:
we identify the form of the implicit regularization for NRNNs through a corresponding (data- dependent) explicit regularizer in the small noise regime (see Theorem
we focus on its effect in classification tasks, providing bounds for the classification margin for the deterministic RNN classifiers (see Theorem [ in particular, Theorem 2
reveals that stable RNN
dynamics can lead to large classification margin;
we show that noise injection can also lead to improved stability (see Theorem 3 via a Lyapunov stability analysis of continuous-timg NRNNs;
we demonstrate via empirical experiments on benchmark data sets that NRNN classifiers are more robust to data perturbations when compared to other recurrent models, while retaining ofate-of-the-ar performance for clean data. Research code is provided here: https://github. cwm/ www.///rissoo///ooooooooooo errchson//oossRRW
Notation. We use ||0|| == || to denote the Euclidean norm of the vector e, and ||A||2 and || to denote the spectral norm and Frobenius norm of the matrix A, respectively. The ith element of a vector v is denoted by of or [u]', and the (i.j)-entry of matrix A by A or [A] For vector V = (u',..., )²), diag(v) denotes the diagonalization of e with diag(u)* = v² denotes the identity matrix (with dimension clear from context), while superscript T denotes transposition. For a matrix M, M = (M + M2 //2 denotes its symmetric part, Amin (M) and Xmax (M) denote its minimum and maximum eigenvalue respectively, maax (M) denotes its maximum singular value, and Tr(M) denotes its trace. For a function J Rnn - R such that each of its first-order partial derivatives (with respect to 2) exist, 0f R m=xx is the Jacobian matrix of f. For scalar- By function g : RR -> E. Vng is the gradient of g with respect to the variable h E R= and Hig the Hessian of g with respect to h.
2 Related Work
Dynamical Systems and Machine Learning. There are various interesting connections between machine learning and dynamical systems. Formulating machine learning in the framework of continuous-time dynamical systems was recently popularized by 62 Subsequent efforts focus on
constructing learning models by approximating continuous -itiiuuuuu--imeeeeee dynamical systems 19 29 48 and
studying them using tools from numerical analysis 36 64 69 68]. On the other hand, dynamical
systems theory provides useful theoretical tools for analyzing NNs, including RNNs [60 15 34
[6], and useful principles for designing NNs 23 [44. Other examples of dynamical systems inspired models include the learning of invariant quantities via their Hamiltonian or Lagrangian representations [37 22 10 7 (58). . Another class of models is inspired by Koopman theory, yielding [3].
models where the evolution operator is linear [56
45 33
Stochastic Training and Regularization Strategies. Regularization techniques such as noise injec- tion and dropout can help to prevent overfitting in NNs. Following the classical work [5] that studies regularizing effects of noise injection on data, several work studies the effects of noise injection
into different parts of networks for various architectures [25] 44, 35|
54 2 26 66 61]. In particular,
recently 66] studies the regularizing effect of isotropic Gaussian noise injection into the layers of
feedforward networks. For RNNs. [2] shows that noise additions on the hidden states outperform Bernoulli dropout terms of performance and bias, whereas (18) introduces a variant of stochastic
RNNs for generative modeling of sequential data. Some specific formulations of RNNs as SDEs were

---

also considered in Chapter 10 of [1]. Implicit regularization has also been studied more generally
than stochastic gradient based training of NNs [4 39
3 Noisy Recurrent Neural Networks
We formulate continuous-time recurrent neural networks (CT-RNNs)) at full generality as system of input-driven ODEs: for a terminal time T and an input signal x (z1)xe(o,7] E C((0, RR.).. the output gr E R4,, for t E [0, T' is a linear map of hidden states hG E Rd satisfying dh,- = (he,
y/ = Vh,
(1)
where V Rdd, and f I dh x RR + Rd is typically Lipschitz continuous, guaranteeing existence and uniqueness of solutions to
A natural stochastic variant of CT-RNNs arises by replacing the ODE in D by an Itô SDE, that is,
dh, f(h,, xt) + (h,, 24 )BB Vr Vhe.
(2)
where a dRdh x 12. + Rd and (B:)c20 an r-dimensional Brownian motion. The functions f, referred as the drift and diffusion coefficients, respectively. Intuitively, (2) amounts to noisy perturbation of the corresponding deterministic CT-RNN (I). At full generality, we refer to
the system (2) continuous time Noisy RNN (CT-NRNN). To guarantee the existence of a unique and {a( x.. he)ooe are uniformly
solution to (2). in the sequel, we assume that (J(,, Lipschitz continuous, and + xt). t h (h, z2)) are bounded in E [0, T] for each fixed RRdd, For further details, see Section in Supplementary Material (SM).
While much of our theoretical analysis will focus on this general formulation of CT-NRNNs, our empirical and stability analyses focus on the choice of drift function
fhh, Ah aWWh h + Ux + b),
where a + R a Lipschitz continuous scalar activation function extended to act on vectors pointwise, A, W Rd xdd, U E xdx and b dh Typical examples of activation functions include a(x) = tanh(z). The matrices A, W, U, V, are all assumed to be trainable parameters. This particular choice of drift dates back to the early Cohen-Grossber formulation of CT-RNNs, and was recently reconsidered in [16].
3.1 Noise Injections as Stochastic Learning Strategies
While precise choices of drift functions f are the subject of existing deterministic RNN theory, good choices of the diffusion coefficient a are less clear. Here, we shall consider a parametric class of diffusion coefficients given by:
= ((h,))=((ar4) + (fd z))),
(4)
where the noise level > 0 is small, and relative strength of additive noise and a multiplicative noise respectively.
and Ø2 ' 0 tunable parameters describing the
While the stochastic component is an important part of the model, one can set 0 at inference time. In doing so, noise injections in NRNNs may be viewed as learning strategy. A similar stance is considered in 35 for treating neural SDEs. From this point of view, we may relate noise injections generally to regularization mechanisms considered in previous works. For example, additive noise injection studied in the context of feedforward NNs in (6), in which case a Gaussian noise is injected to the activation function at each layer of the NN. Furthermore, multiplicative noise injections includes stochastic depth and dropout strategies special cases (36 35). By taking Gaussian approximation to Bernoulli noise and taking a continuous-timg limit, NNs with stochastic dropout can be weakly approximated by SDE with appropriate multiplicative noise, [6] All of these works highlight various advantages of noise injection for training NNs.
3.2 Numerical Discretizations
As in the deterministic case, exact simulation of the SDE in (2) is infeasible in practice, and one must specify numerical integration scheme. We will focus on the explicit Fuler- Maruyama (F-M) integrators [30], which are the stochastic analogues of Euler-type integration schemes for ODEs.
3

---

Let 0 = to < < .... .......6... tM = T be partition of the interval [0, Tİ. Denote 6m := Em+1 Im for each 1. ..... M 1, and 8 == (6m.). The E-M scheme provides family (parametrized by 6) of approximations to the solution of the SDE in (2:
1 +1 = nm + f(h 2. m6m (h 2m) V /mmm:
(5)
for m = 0, ,,,..,,,, M- 1, where (dm)m=0.....M
a given sequential data, the Em ~ NO 1) are
independent r-dimensional standard normal random vectors, and hg = hg. As A = max 6m +
the family of approximations (hm)) converges strongly to the Itô process (he) satisfying 2 (at rate
((VA) when the step sizes are uniform; see Theorem 10.2.2 (30].. See Section SM for details on the general case.
4 Implicit Regularization
To highlight the advantages of NRNNs over their deterministic counterpart, we show that, under reasonable assumptions, NRNNs exhibit a natural form of implicit regularization. By this, we mean regularization imposed implicitly by the stochastic learning strategy, without explicitly modifying the loss, but that, e.g., may promote flatter minima. Our goal is achieved by deriving appropriate explicit regularizer through perturbation analysis the small noise regime. This becomes useful when considering NRNNs as learning strategy, since we can precisely determine the effect of the noise injection as a regularization mechanism.
The study for discrete-time NRNNs is of practical interest and is our focus here. Nevertheless, analogous results for continuous-tie schemes. For this reason, we also study the continuous- covers general NRNNs, not necessarily those with the drift term (3) and diffusion term 4 that satisfy the following assumption, which is typically reasonable in practice. We remark that a ReLU activation will violate the assumption. However, RNNs with ReLU activation are less widely used in practice. Without careful initialization [31, (7), they typically suffer more from exploding gradient problems compared to those with bounded activation functions such as tanh.
NRNNS are also valuable for exploring other discretization
case in Section E in SM. Our analysis
Assumption A. The drift and diffusion coefficient of the SDE in 2 satisfy the following:
(i) for all E [" and æ E Rd, h + f(h, x)) and + /² (h, have Lipschitz continuous partial derivatives in each coordinate up to order three (inclusive);
iii for any h E Rdd, t + f(h, and t + (h, are bounded and Borel measurable on [0, T].
We consider rescaling of the noise a + ea in (2). where >> 0 is assumed to be a small parameter, in line with our noise injection strategies in Subsection 3.1
In the sequel, we let T denote the hidden states of the corresponding deterministic RNN model, satisfying
h mm+1 Th Jm (Ton 2m).
0.1....,MM M.....,---.. 1,
8m and denote the state-to-state Jacobians by with ng ho. Let A = max (U...E (U)
Jm I + 5m 05 (h 2m).
(7)
Oh
For m, k = =0,... also let
$m.k 3 -m Sr.
(8)
where the empty product is assumed to be the identity. Note that the $m,k are products of the state-to-state Jacobian matrices, important for analyzing signal propagation in RNNs (8). For the sake of brevity, we denote fm = fh fm) and om a (he, 2m) for m = 0, 1, M.
The following result, which is our first main result, relates the loss function, averaged over realizations of the injected noise, used for training NRNN to that for training deterministic RNN in the small noise regime.
Theorem (Implicit regularization induced by noise injection). Under Assumption
B((h()) = E(hhr) + ? [Q(h")) + R(
(9)
2

---

as é + O, where the terms Q and R. are given by
M Q(h)) = Vi(h
M
1,k
Σ
6m-1 10m
(10)
m=1
R(h²) Σ
tr 1
D 1,m Hjol 4 1,, 1m-1))
(11)
with Um a vector with the pth component:
[mm = m- 1 2,m Hnd lf mm2mmmmm1)m
-
(12)
for p l....,dd, .r. Moreover,
1Q S |((TTT) CRAA
(13)
for Co, CR O independent of A.
If the loss is convex, then it non-negative, but Q needs be. However, Q can be made negligible relative R provided that A is taken sufficiently small. This also ensures that the E-M approximations are accurate.
To summarize, Theorem T implies that the injection of noise into the hidden states of deterministic RNN is, on average, approximately equivalent to a regularized objective functional. Moreover, the explicit regularizer is solely determined by the discrete-time flow generated by the Jacobians
0fm 2h (Te)). the diffusion coefficients on, and the Hessian of the loss function, all evaluated along the dynamics of the deterministic RNN. We can therefore expect that the use of NRNNs as regularization mechanism should reduce the state-to-state Jacobians and Hessian of the loss function according to the noise level Indeed, NRNNs exhibit a smoother Hessian landscape than that of the deterministic counterpart (see Figure 3 in SM).
The Hessian of the loss function commonly appears in implicit regularization analyses, and suggests preference towards wider minima in the loss landscape. Commonly considered positive attribute 288. this, in turn, suggests a degree robustness in the loss to perturbations in the hidden states (65). More interesting, however, is the appearance of the Jacobians, which is indicative of a preference towards slower, more stable dynamics. Both of these attributes suggest NRNNs could exhibit a strong tendency towards models which are less sensitive input perturbations. Overall, can see that the use of NRNNs as a regularization mechanism reduces the state-to-state Jacobians and Hessian of the loss function according the noise level.
5 Implications in Classification Tasks
Our focus now turns to an investigation of the benefits of NRNNs over their deterministic counter- parts for classification tasks. From Theorem I it is clear that adding noise to deterministic RNN implicitly regularizes the state-to-state Jacobians. Here, we show that doing so also enhances an implicit tendency towards classifiers with large classification margin. Our analysis here covers general deterministic RNNs, although we also apply our results to obtain explicit expressions for Lipschitz RNNs.
Let SN denote a set of training samples 8,, = (Ln, Bn) for n = 1,..,,N N, where each input sequence 2n = (In, In,1...., 11) E X C Rdd has a corresponding class label yn e D (1...., ). Following the statistical learning framework, these samples assumed to be independently drawn from an underlying probability distribution on the sample space = X' x y An RNN-based classifier ((x:) is constructed in the usual way by taking
gd (xx)) argmax d,, (
(14)
where p²(x) c*' E e²d is the softmax function. Letting & denoting the cross-entropy loss, such classifier is trained from SN by minimizing the empirical risk (training error), RN(0²) = $ EN ((g((xn) vn), proxy for the true (population) risk (test error), R(g²)
5

---

B(n:)))yy (x) (x), v), with (x.y) E S. The measure used to quantify the prediction quality is the generalization enror (or estimation error), which is the difference between the empirical risk of the classifier on the training set and the true risk: GE(g") == (R(g") - RN (g")--
The classifier is function of the output of the deterministic RNN, which is an Euler discretization of the ODE ( with step sizes = (6m). In particular, for the Lipschitz RNN,
&m,k -m
(15)
where J 1 D.W), with D. d' ((WWh + U b)))ezg.
In the following, we let conv(X) denote the convex hull of X. We let Fo:m := (fo..... ..; f.)) so that I -0:M-1, and use the notation f[æ] to indicate the dependence of the function f on the vector Our result will depend on two characterizations of a training sample s; = (æ), y4).
Definition 1 (Classification Margin). The classification margin of a training sample 8; (ar,; y1) measured by the Euclidean metric d is defined as the radius of the largest d-metric ball in X centered at x; that is contained in the decision region associated with the class label yr, i.c. it is: (s;) = sup(a: d(æ;, Φ) S a + g" (x) = yi vzJ.
Intuitively, larger classification margin allows a classifier associate larger region centered on point æ; in the input space to the same class. This makes the classifier less sensitive to input perturbations, and perturbation of æ; is still likely to fall within this region, keeping the classifier prediction. In this sense, the classifier becomes more robust. In our case, the networks are trained by a loss (cross-ente entropy) that promotes separation of different classes in the network output. This, in turn, maximizes a certain notion of score of each training sample.
Definition (Score). For a training sample 8; = (æ;, Vi). we define its score as o(s;) = minjzv; v2(eg; 5²) [ 2 O, where e, E IRd the Kronecker delta vector with of = and eę- 0 for + S. [=.] == p(V p(Vhh (x:]) with Te [a:]] denoting the hidden state of the RNN, driven by the input sequence H; at terminal index M.
( [z)], and the decision boundary between class i and class j in the feature space is given by the hyperplane {= 5 z =z0}. A positive score implies that at the network output, classes are separated by a margin that corresponds to the score. However, large score may not imply large classification margin.
Recall that the classifier (xx) arg maxigl....dd,
Following the approach of (52 63 we obtain the second main result, providing bounds for classifi- cation margin for the deterministic RNN classifiers 9². We also provide a generalization bound in terms of the classification margin under additional assumptions (see Theorem 11 in SM).
Theorem 2. Suppose that Assumption A holds. Assume that the score (ss;) > 0 and
o(s;)
(88)== = C CEMD 8m sup |1 M [ølll 0,
(16)
where C ||V||2 mmxxm= 1.I...., M-
asike 1 .tm)
1,)
0 is independent of 8; (in particular,
C' V==IVlla [all|mmaxm=....... M- [ for Lipschitz RNNs), the &m,k are defined in 15 and the 8m are the step sizes. Then, the classification margin for the training sample s,:
7 (xx) Z 7(:))
(17)
Now, recalling from Section up O(22) and under the assumption that Q vanishes, the loss minimized by the NRNN classifer is, on average, E(hr,) + e R(hh), as - with regularizer
M
R((T5) = Σ 8m MMMMM11 - -.
(18)
where MT MM == Hig, I is the Cholesky decomposition of the Hessian matrix of the convex cross- entropy loss. The appearance of the state-to-state Jacobians in &m.. in both the regularizer (18) and the lower bound (16) suggests that noise injection implicitly aids generalization performance. More precisely, in the small noise regime and average, NRNNs promote classifiers with large

---

classification margin, an attribute linked to both improved robustness and generalization [63|. In this sense, training with NRNN classifiers is a stochastic strategy to improve generalization over deterministic RNN classifiers, particularly in learning tasks where the given data is corrupted (c.f. the caveats pointed out in (
Theorem Thhorem@iimpliss that the lower bound for the classification margin is determined by the spectrum of the P To make the lower bound large, keeping 6n and M fixed, the spectral norm of the 4 1 should be made small. Doing improves stability of the RNN, but may also lead to vanishing gradients, hindering capacity of the model to learn. To maximize the lower bound while avoiding the vanishing gradient problem,
should tune the numerical step sizes 8m and noise level e in NRNN appropriately. RNN architectures for the drift which help to ensure moderate Jacobians
(e.g.. M ll2 R 1 for all m (8) also remain valuable in this respect.
6 Stability and Noise-Induced Stabilization
Here we obtain sufficient conditions to guarantee stochastic stability of CT-NRNNS. This will also provide another lens to highlight the potential of NRNNs for improved robustness. A dynamical system is considered stable if trajectories which are close to each other initially remain close at subsequent times. As observed in [46] 42 77, stability plays essential role in the study of RNNs to avoid the exploding gradient problem, a property of unstable systems where the gradient increases in magnitude with the depth. While gradient clipping during training can somewhat alleviate this issue, better performance and robustness is achieved by enforcing stability in the model itself.
Our stability analysis will focus on establishing almost sure expomential stability (for other notions of stability, see SM) for CT-NRNNs with the drift function (3). To preface the definition, consider initializing the SDE at two different random variables ho and hig ;= hg + co. where Eo E R is a constant non-random perturbation with |eo| S The resulting hidden states, he and hú, are set to satisfy (2) with the same Brownian motion Br, starting from their initial values ho and hn- respectively. The evolution of e = h4 h2 satisfies
de, = Ac + Aar(e)dtt + (o,(e))dB.
(19)
(Wh, + +b)+bb and (o, = oh, + 21) - h4, x.). Since Aa, (0) 0, (0) = 0 for all t [0, = 0 admits a trivial equilibrium for (I9). Our objective is to analyze the stability of the solution 6; = O, that is, to see how the final state cr (and hence the output of the RNN) changes for arbitrarily small initial perturbation co O. To this end, we consider an extension of the Lyapunov exponent to SDEs at the level of sample path [41.
where Aar = aWW,,
Definition 3 (Almost sure global exponential stability). The sample (or pathwise) Lyapunov exponent of the trivial solution of (19) is A lim sup, log [|c/].. The trivial solution é = 0 is almost surely globally exponentially stable if A is almost surely negative for all cg E RRd..
For the sample Lyapunov exponent A(w), there is a constant C > O and a random variable 0 < T(W) < oc such that for all > T(W), |||]|| = IIA - hell < Ae almost surely. Therefore, almost sure exponential stability implies that almost all sample paths of (19) will tend to the equilibrium solution 0 exponentially fast, With this definition in tow, we obtain the following stability result.
Theorem 3. Assume that a is monotone non- -ecreaeea,,nng and on ||||| < [))y()))lr S [2]le]] all nonzero e E Rd,, E T). Then for any Eo E Rda, with probability one,
Φ + Amin (A S < vp + Lax (W) + Amax A (A
(20)
with o = -02 + of T and vi = -o +
a² T where La is the Lipschitz constant of
In the special case without noise (0) = negative definite and amin (Asym) > L.,.max (W), Theorem 3 implies that (2) exponentially stable. Most strikingly, and similar to 35 Theorem 3 implies that even if the deterministic CT-RNN not exponentially stable, can be stabilized through a stochastic perturbation. Consequently, injecting noise appropriately can improve training performance.
0), we recover case (a) of Theorem 1 in [16] when 4
7

---

7 Empirical Results
The evaluation of robustness of neural networks (RNNs partieular) an often neglected yet erucial aspect. In this section, we investigate the robustness of NRNNs and compare their performance to other recently introduced state-of-the-arr models on both clean and corrupted data. We refer to Section G in SM for further details of our experiments.
Here, we study the sensitivity of different RNN models with respect to a sequence of perturbed inputs during inference time. We consider different types of perturbations: (a) white noise: (b) multiplicative white noise: (c) salt and pepper; and (d) adversarial perturbations. To be more concrete, let be sequence. The perturbations in consideration are as follows.
Additive white noise perturbations are constructed as ë z+ Az, where the additive noise is drawn from a Gaussian distribution Ar (N o). This perturbation strategy emulates measurement errors that can result from data acquisition with poor sensors (where o can be used to vary the strength of these errors). Multiplicative white noise perturbations are constructed as x = x Ax, where the additive noise is drawn from Gaussian distribution Ar ~ (1,,oM)
Salt and pepper perturbations emulate defective pixels that result from converting analog signals to digital signals. The noise model takes the form (P X)
a, and TP(X max) = P(X min) a:/2, where (i, j) denotes the corrupted image and min and max denote to the minimum and maximum pixel values. The parameter a controls the proportion of defective pixels.
Adversarial perturbations are "worst-case" non-random perturbations maximizing the loss E(gg (X + AX),,) v) subject to the constraint that the norm of the perturbation ||X|| S consider the fast gradient sign method for constructing these perturbations (55]
We
We consider in addition to the NRNN three other RNNs derived from continuous-time models, including the Lipschitz RNN [16] (the deterministic counterpart to our NRNN), the coupled oscillatory RNN (coRNN) [49] and the antisymmetric RNN [7]. We also consider the exponential RNN [32], discrete-time model that uses orthogonal recurrent weights. We train each model with the prescribed tuning parameters for the ordered (see Sec. (711 and permuted (see SM) MNIST task. For the Electrocardiogram (ECG) classification task performed a non-exhaustive hyper-tuning parameter search. For comparison, we train all models with hidden-tee coadden-to-hiie to-hidden weight matrices dimension d₃ = 128. We average the classification performance over ten different seed values.
7.1 Ordered Pixel-by-Pixel MNIST Classification
First, we consider the ordered pixel-by-pinc by-ixel MNIST classification task [31]. This task sequentially presents 784 pixels to the model and uses the final hidden state predict the class membership probability of the input image. In the SM we present additional results for the situation when instead of ordered sequence fixed random permutation of the input sequence is presented to the model.
Table Table[sshows shows the average test accuracy (evaluated for models that are trained with 10 different seed values) for the ordered task. Here we present results for white noise and salt and pepper (S&P) perturbations. While the Lipschitz RNN performs best on clean input sequences, the NRNNs show an improved resilience to input perturbations. Here, we consider two different configuration for the NRNN. In both cases, we set the multiplicative noise level to 0.02, whereas we consider the additive noise levels 0.02 and 0.05. We chose these configurations they appear provide good trade-off between accuracy and robustness. Note that the predictive accuracy on clean inputs starts to drop when the noise level becomes large.
Table 1: Robustness w.r.t. white noise (a) and S&P (a) perturbations on the ordered MNIST task.
Name
clean
0.2
0.3
0.05
0.1
Antisymmetric RNN
97.5%
45.7%
22.3%
17.0%
77.1%
63.9%
42.6%
CoRNN
99.1%
96.6%
61.9%
32.1%
95.6%
88.1%
58.9%
Exponential RNN 32
96.7%
86.7%
58.1%
33.3%
83.6%
70.7%
43,4%
Lipschitz RNN [66
99.2%
98.4%
78.9%
47.1%
97.6%
93.4%
73.5%
NRNN (mult./ad. noise: 0.02/0.02)
99.1%
98.9%
88.4%
62.9%
98.3%
95.6%
78.7%
NRNN (mult./add. noise: 0.02/0.05)
99.1%
98.9%
92.2%
73.5%
98.5%
97.1%
85.5%

---

Noisy ANN Lipschiiz
0.8
Antisymmetric RNN
test accuracy
CORNN
EXPRNN
0.6
0.6
0.4
0.0
0.1
0.2
0.3
0.4
0.5
0.7
0.00
0.05
0.10
0.15
0.20
0.25
amount of noise
amount of noise
(a) White noise perturbations.
(b) Salt and pepper perturbations.
Figure 1: Test accuracy for the ordered MNIST task as function of the strength of input perturbations.
Table 2 shows the average test accuracy for the ordered MNIST task for adversarial perturbations. Again, the NRNNs show superior resilience even to large perturbations, whereas the Antisymmetric and Exponential RNN appear be sensitive even small perturbations.
Figure I summarizes the performance different models with respect to white noise and salt and pepper perturbations. The colored bands indicate +1 standard deviation around the average performance. In all cases, the NRNN appears to be less sensitive to input perturbations as compared to the other models, while maintaining state-of-the-arr performance for clean inputs.
7.2 Electrocardiogram (ECG) Classification
Next, we consider the Electrocardiogram (ECG) classification task that aims to discriminate between normal and abnormal heart beats of patient that has severe congestive heart failure (20). We use 500 sequences of length 140 for training, 500 sequences for validation, and 4000 sequences for testing.
Table shows the average test accuracy (evaluated for models that trained with 10 different seed values) for this task. We present results for additive white noise and multiplicative white noise perturbations. Here, the NRNN, trained with multiplicative noise level set 0.03 and additive noise levels set to 0.06, performs best both on clean as well as on perturbed input sequences.
Figure summarizes the performance of different models with respect additive and multiplicative white noise perturbations. Again, the NRNN appears to be less sensitive to imput perturbations as compared to the other models, while achieving state-of-the-ar aart performance for clean inputs.
Table 2: Robustness w.r.t. adversarial perturbations the ordered pixel- by-pxel MNIST task.
Name
r 0.01
0.05
r
r 0.15
Antisymmetric RNN
79.4%
24.7%
11.4%
10.2%
CoRNN 4g
97.5%
85.5%
55.9%
35.1%
Exponential RNN 32
94.5%
59.3%
19.7%
14.3%
Lipschitz RNN [6
98.1%
85.7%
58.9%
37.1%
NRNN (mult/add. noise: 0.02/0.02)
98.8%
94.3%
79.6%
58.3%
NRNN (mult/add, noise: 0.02/0.05)
98.8%
95.5%
86.8%
70.6%
Table 3: Robustness w.r.t. white (a) and multiplicative (a noise perturbations on the ECG task.
Name
clean
0.4
0.8
d' 0.4
GM
0.8
dxx
1.2
Antisymmetric
97.1%
96.6%
91.6%
17.0%
96.6%
94.6%
91.2%
CoRNN 4Y
97.5%
96.8%
92.9%
87.2%
93.9%
85.4%
78.4%
Exponential RNN
32
97.4%
95.6%
86.4%
76.7%
95.7%
89.4%
81.3%
Lipschitz RNN [6
97.7%
97.4%
95.1%
88.9%
97.6%
97.0%
95.6%
NRNN (mult/add, noise: 0.03/0.06)
97.7%
97.5%
96.3%
92.6%
97.7%
97.3%
96.5%

---

1.00
1.00
0.95
test accuracy
0.90
0.85
0.85
0.80
0.80
Noisy
Lipschitz
0.75
Antisymmetric RNN
0.70
0.70
CORNN
eXPRNN
0.65
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
amount of noise
amount of noise
(a) Additive white noise perturbations.
(b) Multiplicative white noise perturbations.
Figure 2: Test accuracy for the ECG task as function of the strength of input perturbations.
8 Conclusion
In this paper we provide a thorough theoretical analysis of RNNs trained by injecting noise into the hidden states. Within the framework SDEs, we study the regularizing effects of general noise injection schemes. The experimental results are in agreement with our theory and its implications, finding that Noisy RNNs achieve superior robustness state-of tate-off-hhe-rt generalization performance. We believe our framework can be used to guide the principled design of a class of reliable and robust RNN classifiers.
input perturbations, while maintaining
Our work opens up range of interesting future directions. In particular, for deterministic RNNs, it shown that the models learn optimally near the edge of stability [8]. One could extend these analyses to NRNNs with the ultimate goal of improving their performance. On the other hand, as discussed in Section although the noise shown here implicitly stabilize RNNs, it could negatively impact capacity for long-term memory 42 70]. Providing analyses to account for this and is the subject of future work.
the implicit bias due the stochastic optimization procedure [50 14]
Acknowledgements
are grateful for generous support from Amazon AWS. H. Lim would like to acknowledge Nordita Fellowship 2018-2021 for providing support of this work. N. B. Erichson, L. Hodgkinson, and W. Mahoney would like acknowledge the IARPA (contract W91 1991NF000033)) ARO, NSF, and and ONR via its BRC RandNLA for providing partial support of this work. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
References
[1] Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International Conference on Machine Learning, pages 233-244. PMLR, 2020.
[2] Raman Arora, Peter Bartlett, Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. In International Conference on Machine Learning, pages 351-361. PMLR, 2021.
[3] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael W. Mahoney, Forecasting sequential data using consistent Koopman autoencoders. In International Conference Machine Learning, pages 475-485. PMLR, 2020.
[4] Kaushik Balakrishnan and Devesh Upadhyay. Deep adversarial Koopman model for reaction- diffusion systems. arXiv preprint arXiv:2006.055/ 2020.
[5] Chris Bishop. Training with noise is equivalent to Tikhonov regularization. Neural Compu- tation, 7(1): 100:100811110 116, 1995.
10

---

[6] Alexander Camuto, Matthew Willetts, Umut Simsekli, Stephen J Roberts, and Chris C Holmes. Explicit regularisation in Gaussian noise injections. In Advances Neural Information Pro- cessing Systems, volume 33, pages 16603-16614, 2020.
[7] Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetricRNN: dynamical system view on recurrent neural networks. In International Conference on Learning Represen- tations, 2019.
[8] Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In International Conference on Machine Learning, pages 873-882. PMLR, 2018.
(9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571-65716 2018.
[10) Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. International Conference on Learning Representations, 2019.
[11) Michal Derezinski, Feynman T Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. In Advances in Neural Information Processing Systems, volume 33, pages 5152-5164, 2020.
[12] Adji Bousso Dieng, Rajesh Ranganath, Jaan Altosaar, and David Blei. Noisin: Unbiased regularization for recurrent neural networks. In International Conference on Machine Learning. pages 1252-1261. PMLR, 2018.
[13] Akshunna S. Dogra and William Redman. Optimizing neural networks via Koopman operator theory. In Advances Neural Information Processing Systems, volume 33, pages 2087-2097, 2020.
[14] Melikasadat Emami, Mojtaba Sahraee-Ardakan, Parthe Pandit, Sundeep Rangan, and Alyson Fletcher. Implicit bias of linear RNNs. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 2982-2992. PMLR, 2021.
[15] Rainer Engelken, Fred Wolf, and LF Abbott. Lyapunov spectra of chaotic recurrent neural networks. arXiv preprint arXiv:2006.0244: 2020.
[16] N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Representations, 2021.
[17] Benjamin Erichson, Michael Muehlebach, and Michael W Mahoney. Physics-informed autoencoders for Lyapunov-stable fluid flow prediction. arXiv preprint 1905 2019.
[18] Marco Fraccaro, Søren Kaae Sø nderby, Ulrich Paquet, and Ole Winther. Sequential neural mod- els with stochastic layers. In Advances Neural Information Processing Systems, volume 29, 2016.
[19] D. Gleich and M. W. Mahoney. Anti Anti-differentiating approximation algorithms: case study with min-cuts, spectral, and flow. In Proceedings of the 31st International Conference Machine Learning, pages 1025, 2014.
[20] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Kang Peng, and Eugene Stanley, Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. Circulation, 101(2))e2215 2000.
[21] Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: Lightweight adversarial training with data augmentation improves neural network training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2474-2482 2021.
11

---

[22] Samuel Greydanus. Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[23] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 3401)401400 2017.
[24] Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W Mahoney. Stochastic continuous normalizing flows: Training SDEs as ODEs. In Uncertainty in Artificial Intelligence (UAT), 2021.
[25] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, pages 646-661. Springer, 2016.
Jim, C Lee Giles, and Bill G Home. An analysis of noise in recurrent neural [26] networks: Convergence and generalization. IEEE Transactions on Neural Nenworks, 7(6):1424- 1438, 1996.
[27] Anil Kag, Ziming Zhang, and Venkatesh Saligrama. RNNs incrementally evolving an cquilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference Learning Representations, 2020.
[28] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.0485 2016.
[29] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In Advances in Neural Information Processing Systems, volume 33, pages 6696-6707, 2020.
[30] Peter E Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations, volume 23. Springer Science & Business Media, 2013.
[31] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. simple way initialize recurrent networks of rectified linear units. arXiv preprint arXiv: /504. 0094100,11,,
[32] Mario Lezcano- and David Caarunez-Rubioo Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pages 3794-3803 2019.
[33] Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional Koopman operators for model-based control. In International Conference on Learning Repre- sentations, 2019.
[34] Soon Hoe Lim. Understanding recurrent neural networks using nonequilibrium response theory.
J. Mach. Learn. Res., 22:47-1, 2021.
[35] Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao. Sanjiv Kumar, and Cho-Jui Hsich. How does noise help robustness? Explanation and exploration under the neural SDE framework. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition, pages 282-290,, 2020.
[36] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3276-3285. PMLR, 2018.
[37] Michael Lutter, Christian Ritter, and Jan Peters. Deep Lagrangian networks: Using physics as model prior for deep learning. arXiv preprint 1907.04907.049900 2019.
[38] Chao Ma. Stephan Wojtowytsch, and Lei Wu. Towards a mathematical understanding of neural setwork-based machine learning: What we know and what we don't. arXiv preprint arXiv:2009. 10713, 2020.
[39] M. W. Mahoney. Approximate computation and implicit regularization for very large-scale data analysis. In Proceedings of the 31 ACM Symposium on Principles of Database Systems, pages 143-154, 2012.
12

---

[40] M. W. Mahoney and L. Orecchia. Implementing regularization implicitly via approximate eigenvector computation. In Proceedings of the 28th International Conference on Machine Learning, pages 121-128, 2011.
[41] Xuerong Mao. Stochastic Differential Equations and Applications. Elsevier, 2007.
[42] John Miller and Moritz Hardt. Stable recurrent models. arXiv preprint vrxiv:/805./0369 2018.
[43] Jeremy Morton, Freddie Witherden, and Mykel Kochenderfer. Deep variational Koopman models: Inferring Koopman observations for uncertainty dynamics modeling and control. arXiv preprint arXiv:190..09770 2019.
[44] Hyconwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In Advances in Neural Information Processing Systems, pages 5109-5118. 2017.
[45] Shaowu Pan and Karthik Duraisamy. Physics-informed probabilistic learning of linear embed- dings of nonlinear dynamics with guaranteed stability. SIAM Journal Applied Dynamical Systems, 19(1):800-509 2020.
[46] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310-1318, PMLR, 2013.
[47] Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning III: Explaining the non-overfitting puzzle. arXiv preprint arXiv:/801.00/ 100173.00033,....000000000 2017.
[48] Alejandro F Queiruga, Benjamin Erichson, Dane Taylor, and Michael W Mahoney, Continuous-in-deph neural networks. arXiv preprint 20Xi::2008.02389, 2020.
[49] T. Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (coRNN); An accurate and (gradient) stable architecture for learning long time dependencies. In International Conference on Learning Representations, 2021.
[50] Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 2021.
[51] Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Generalization error of deep neural networks: Role of classification margin and data structure. In 2017 International Conference Sampling Theory and Applications (SampTA), pages 147-151. IEEE, 2017.
[52] Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions Signal Processing, 65(16):4265 440442655-280,,, 2017.
[53] David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pages 6976-6987, 2019.
[54] Qi Sun, Yunzhe Tao, and Qiang Du Stochastic training of residual networks: A differential equation viewpoint. arXiv preprint arXiv:1812.00114,, 2018.
[55] Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[56] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman invariant sub- spaces for dynamic mode decomposition. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 11 1130-1144 1140, 2017.
[57] Sachin S Talathi and Aniket Vartak. Improving performance of recurrent neural network with ReLU nonlinearity, arXiv preprint drXhv:1511.03771,
13

---

Communications in [58] Peter Toth, Danilo Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2019.
[59] Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09880 2019.
[60] Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Lyapunov exponents for RNNs: Understanding information propagation using dynamical systems tools. arXiv preprini arXiv:2006. 14123, 2020.
[61] Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In Proceedings of the 37th International Conference on Machine Learning, pages 10181-10192, 2020.
[62] E. Weinan. A proposal on machine learning via dynamical systems. Mathematics and Statistics, 5(1):1-11, 2017.
[63] Huan Xu and Shie Mannor. Robustness and generalization. Machine Learning, 86(3):391-423, 2012.
[64] Yibo Yang, Jianlong Wu, Hongyang Li, Xia Li, Tiancheng Shen, and Zhouchen Lin. Dynamical system inspired adaptive time stepping controller for residual network families. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6648-6655 2020.
[65] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian- -baed analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, pages 4954 4964, 2018.
[66] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint 1409:244 2014.
[67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[68] Huishuai Zhang, Da Yu, Mingyang Yi, Wei Chen, and Tie-yan Liu. Stability and convergence theory for learning ResNet: A full characterization. 2019.
[69] Jingfeng Zhang, Bo Han, Laura Wynter, Bryan Kian Hsiang Low, and Mohan Kankanhalli. Towards robust ResNet: A small step but giant leap. Proceedings of the Twenty-Eis Eighth International Joint Conference Artificial Intelligence, 2019.
[70] Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do RNN and LSTM have long memory? In International Conference on Machine Learning. pages 11365-1 11375. PMLR, 2020.
[71] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Lear- ing Hamiltonian dynamics with control. In International Conference on Learning Representa- lions, 2019.
14


Reviews:
[{"review_id": "OZUKUMKhKn", "reviewer": "Reviewer_NYKx", "summary": "The authors study the recurrent neural networks with injecting noise to hidden states. They consider RNNs as data driven discretization of stochastic differential equations. Analysis for global stability is presented, and it is also shown that noise injection may improve training stability.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The analysis is based on the discretization of stochastic differential equations by adding the diffusion term on top of ordinary differential equations. With certain assumptions, the authors adopt this analysis to study the implicit regularization effect of noise injection schemes in the small noise regime, which promotes flatter minima, biases towards models with more stable dynamics, and favors larger classification margin in classification. \n\nThe approach of injecting noise in learning the highly non-convex problems can be an effective approach for obtaining more stable and generalizable models. The authors demonstrate a theoretical evidence for RNNs with injecting noise in states by applying the popular tools of differential equations, which marginally promotes our understanding in this regime. The empirical study also shows promising results of the proposed approach compared with existing ones, where under higher noise scenario the proposed approach significantly outperforms.\n\nThough theoretically this is an interesting problem and remains with largely unresolved issues, the presented study does not reveal much new insights in the problem. The paper is relatively easy to follow and clearly presented. The presented proof is mostly clear and seems correct, though I did not check all detailed analysis for the proof. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "J8ay9NSE2aS", "reviewer": "Reviewer_vC4r", "summary": "This paper studies noise injection in RNNs, which can be regarded as a discretized SDE driven by external inputs. The first main result of the paper is the derivation of the implicit regularization caused by noise injection (similar to noise injection for feed-forward models), which is shown to be related to the Hessian of the loss (promoting flatter minima) and the Jacobian of the dynamics (promoting slower/more stable dynamics). A further implication on the classification margin is presented. Lastly, the paper presents a result showing that noise injection can further improve the temporal stability of the RNN dynamics. These motivate the use of the noise injection for recurrent neural networks, and experiments are provided to justify the theoretical results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and the topics is of interest to the RNN/time series modelling community. Noise injection and its regularization effect is worthy of investigation, and in this sense, the main results of this paper presents advancements in this area. Below, I list some questions that the authors may wish to address:\n\n1. It is well established in non-recurrent (or non-deep) models that noise injection leads to implicit regularization. Thus, it may be good to summarize the new phenomena arises from the dynamical setting you consider. One effect I can see from Theorem 2 is that the margin depends on the Jacobian of the dynamics - stable dynamics gives larger margin. I thought this is interesting. Are there any other differences between this dynamical setting and the usual setting in supervised learning?\n\n2. Theorem 2: the result suggests that “stable” dynamics have bigger margins. There seems to be much more straightforward ways to achieve this, i.e. setting A in (3) to be very negative. Is the theorem saying something more than this?\n\n3. Experiments\n\n   1. Since the earlier results suggest that RNN with noise injection can be understood as implicitly regularization with the form of the regularizer as in (16), what if one simply adds a term such as (16) to the loss function. Does it achieve the same effect?\n\n   1. Are there experiments to demonstrate the additional stabilization effect of noisy RNNs, i.e. Theorem 3?\n\n---\n\nAfter rebuttal: thank you for responding to my questions and my evaluation is unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6hutZZ0301H", "reviewer": "Reviewer_fjqb", "summary": "This paper provides a theoretical framework for studying training of recurrent neural networks by injecting noise into hidden states. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and organized. Some valuable technical results are provided to study discretization schemes of stochastic differential equations modeling the injection of noise into networks’ hidden states. These results justify the regularization effect of the injection scheme. This is an important result also illustrated by some representative simulations. They also clearly explain what the limitations are, such as the RELU activation function does not satisfy the conditions required in this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They didn't o it explicitely but I cannot see how this work can cause any negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "None I could find.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "0CcleVA-q-m", "reviewer": "Reviewer_m9sN", "summary": "This work studies RNNs that are trained with noise injection in the framework of ODEs.  The method is called NRNN. The authors provided Theorems to: (1) identify the form of implicit regularization of NRNN; (2) show the improved model stability; (3) provide the bounds for RNN classifiers. The experiments are conducted on two classification tasks: Sequential MNIST and heart beats classification.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of viewing noise injection as stochastic learning is interesting. In general, this work is well presented, the main claims are well warranted by theoretical analysis and empirical results. In terms of the noise level in eq.(4), how to ensure the value \\epsilon is small? How do the different combinations of the tunable parameters \\sigma_1 and \\sigma_2 influence the overall performance in Table 1-3? \n\nThe experimental results of robustness and resilience are interesting, NRNN presents better performance as compared to other methods while maintaining comparable predictive performance. I am wondering if this robustness can be held as well in the regression task?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received overall favorable reviews. This paper was found to be well written and the topic to be of interest and innovative.


Please address reviewers comment in the final version.
Decision: Accept (Poster)