Noisy Recurrent Neural Networks

1 Introduction . Viewing recurrent neural networks ( RNNs ) as discretizations of ordinary differential equations ( ODEs ) driven by input data has recently gained attention [ 7 , 27 , 16 , 49 ] . The “ formulate in continuous time , and then discretize ” approach [ 38 ] motivates novel architecture designs before experimentation , and it provides a useful interpretation as a dynamical system . This , in turn , has led to gains in reliability and robustness to data perturbations . Recent efforts have shown how adding noise can also improve stability during training , and consequently improve robustness [ 35 ] . In this work , we consider discretizations of the corresponding stochastic differential equations ( SDEs ) obtained from ODE formulations of RNNs through the addition of a diffusion ( noise ) term . We refer to these as Noisy RNNs ( NRNNs ) . By dropping the noisy elements at inference time , NRNNs become a stochastic learning strategy which , as we shall prove , has a number of important benefits . In particular , stochastic learning strategies ( including dropout ) are often used as natural regularizers , favoring solutions in regions of the loss landscape with desirable properties ( often improved generalization and/or robustness ) . This mechanism is commonly referred to as implicit regularization [ 40 , 39 , 50 ] , differing from explicit regularization where the loss is explicitly modified . For neural network ( NN ) models , implicit regularization towards wider minima is conjectured to be a prominent ingredient in the success of stochastic optimization [ 67 , 28 ] . Indeed , implicit regularization has been linked to increases in classification margins [ 47 ] , which can lead to improved generalization performance [ 51 ] . A common approach to identify and study implicit regularization is to approximate the implicit regularization by an appropriate explicit regularizer 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 40 , 39 , 1 , 6 , 21 ] . Doing so , we will see that NRNNs favor wide minima ( like SGD ) ; more stable dynamics ; and classifiers with a large classification margin , keeping generalization error small . SDEs have also seen recent appearances in neural SDEs [ 59 , 24 ] , stochastic generalizations of neural ODEs [ 9 ] which can be seen as an analogue of NRNNs for non-sequential data , with a similar relationship to NRNNs as feedforward NNs do to RNNs . They have been shown to be robust in practice [ 35 ] . Analogously , we shall show that the NRNN framework leads to more reliable and robust RNN classifiers , whose promise is demonstrated by experiments on benchmark data sets . Contributions . For the class of NRNNs ( formulated first as a continuous-time model , which is then discretized ) , the following are our main contributions : • we identify the form of the implicit regularization for NRNNs through a corresponding ( datadependent ) explicit regularizer in the small noise regime ( see Theorem 1 ) ; • we focus on its effect in classification tasks , providing bounds for the classification margin for the deterministic RNN classifiers ( see Theorem 2 ) ; in particular , Theorem 2 reveals that stable RNN dynamics can lead to large classification margin ; • we show that noise injection can also lead to improved stability ( see Theorem 3 ) via a Lyapunov stability analysis of continuous-time NRNNs ; • we demonstrate via empirical experiments on benchmark data sets that NRNN classifiers are more robust to data perturbations when compared to other recurrent models , while retaining state-of-the-art performance for clean data . Research code is provided here : https : //github.com/erichson/NoisyRNN . Notation . We use ‖v‖ : = ‖v‖2 to denote the Euclidean norm of the vector v , and ‖A‖2 and ‖A‖F to denote the spectral norm and Frobenius norm of the matrix A , respectively . The ith element of a vector v is denoted by vi or [ v ] i , and the ( i , j ) -entry of a matrix A by Aij or [ A ] ij . For a vector v = ( v1 , . . . , vd ) , diag ( v ) denotes the diagonalization of v with diag ( v ) ii = vi . I denotes the identity matrix ( with dimension clear from context ) , while superscript T denotes transposition . For a matrix M , M sym = ( M + MT ) /2 denotes its symmetric part , λmin ( M ) and λmax ( M ) denote its minimum and maximum eigenvalue respectively , σmax ( M ) denotes its maximum singular value , and Tr ( M ) denotes its trace . For a function f : Rn → Rm such that each of its first-order partial derivatives ( with respect to x ) exist , ∂f∂x ∈ R m×n is the Jacobian matrix of f . For a scalar-valued function g : Rn → R , ∇hg is the gradient of g with respect to the variable h ∈ Rn and Hhg is the Hessian of g with respect to h . 2 Related Work . Dynamical Systems and Machine Learning . There are various interesting connections between machine learning and dynamical systems . Formulating machine learning in the framework of continuous-time dynamical systems was recently popularized by [ 62 ] . Subsequent efforts focus on constructing learning models by approximating continuous-time dynamical systems [ 9 , 29 , 48 ] and studying them using tools from numerical analysis [ 36 , 64 , 69 , 68 ] . On the other hand , dynamical systems theory provides useful theoretical tools for analyzing NNs , including RNNs [ 60 , 15 , 34 , 7 , 16 ] , and useful principles for designing NNs [ 23 , 54 ] . Other examples of dynamical systems inspired models include the learning of invariant quantities via their Hamiltonian or Lagrangian representations [ 37 , 22 , 10 , 71 , 58 ] . Another class of models is inspired by Koopman theory , yielding models where the evolution operator is linear [ 56 , 43 , 17 , 45 , 33 , 4 , 3 , 13 ] . Stochastic Training and Regularization Strategies . Regularization techniques such as noise injection and dropout can help to prevent overfitting in NNs . Following the classical work [ 5 ] that studies regularizing effects of noise injection on data , several work studies the effects of noise injection into different parts of networks for various architectures [ 25 , 44 , 35 , 54 , 2 , 26 , 66 , 61 ] . In particular , recently [ 6 ] studies the regularizing effect of isotropic Gaussian noise injection into the layers of feedforward networks . For RNNs , [ 12 ] shows that noise additions on the hidden states outperform Bernoulli dropout in terms of performance and bias , whereas [ 18 ] introduces a variant of stochastic RNNs for generative modeling of sequential data . Some specific formulations of RNNs as SDEs were also considered in Chapter 10 of [ 41 ] . Implicit regularization has also been studied more generally than stochastic gradient based training of NNs [ 40 , 39 , 19 , 11 ] . 3 Noisy Recurrent Neural Networks . We formulate continuous-time recurrent neural networks ( CT-RNNs ) at full generality as a system of input-driven ODEs : for a terminal time T > 0 and an input signal x = ( xt ) t∈ [ 0 , T ] ∈ C ( [ 0 , T ] ; Rdx ) , the output yt ∈ Rdy , for t ∈ [ 0 , T ] , is a linear map of hidden states ht ∈ Rdh satisfying dht = f ( ht , xt ) dt , yt = V ht , ( 1 ) where V ∈ Rdy×dh , and f : Rdh × Rdx → Rdh is typically Lipschitz continuous , guaranteeing existence and uniqueness of solutions to ( 1 ) . A natural stochastic variant of CT-RNNs arises by replacing the ODE in ( 1 ) by an Itô SDE , that is , dht = f ( ht , xt ) dt+ σ ( ht , xt ) dBt , yt = V ht , ( 2 ) where σ : Rdh × Rdx → Rdh×r and ( Bt ) t≥0 is an r-dimensional Brownian motion . The functions f , σ are referred to as the drift and diffusion coefficients , respectively . Intuitively , ( 2 ) amounts to a noisy perturbation of the corresponding deterministic CT-RNN ( 1 ) . At full generality , we refer to the system ( 2 ) as a continuous-time Noisy RNN ( CT-NRNN ) . To guarantee the existence of a unique solution to ( 2 ) , in the sequel , we assume that { f ( · , xt ) } t∈ [ 0 , T ] and { σ ( · , xt ) } t∈ [ 0 , T ] are uniformly Lipschitz continuous , and t 7→ f ( h , xt ) , t 7→ σ ( h , xt ) are bounded in t ∈ [ 0 , T ] for each fixed h ∈ Rdh . For further details , see Section B in Supplementary Material ( SM ) . While much of our theoretical analysis will focus on this general formulation of CT-NRNNs , our empirical and stability analyses focus on the choice of drift function f ( h , x ) = Ah+ a ( Wh+ Ux+ b ) , ( 3 ) where a : R → R is a Lipschitz continuous scalar activation function extended to act on vectors pointwise , A , W ∈ Rdh×dh , U ∈ Rdh×dx and b ∈ Rdh . Typical examples of activation functions include a ( x ) = tanh ( x ) . The matrices A , W , U , V , b are all assumed to be trainable parameters . This particular choice of drift dates back to the early Cohen-Grossberg formulation of CT-RNNs , and was recently reconsidered in [ 16 ] . 3.1 Noise Injections as Stochastic Learning Strategies . While precise choices of drift functions f are the subject of existing deterministic RNN theory , good choices of the diffusion coefficient σ are less clear . Here , we shall consider a parametric class of diffusion coefficients given by : σ ( h , x ) ≡ ( σ1I + σ2diag ( f ( h , x ) ) ) , ( 4 ) where the noise level > 0 is small , and σ1 ≥ 0 and σ2 ≥ 0 are tunable parameters describing the relative strength of additive noise and a multiplicative noise respectively . While the stochastic component is an important part of the model , one can set ≡ 0 at inference time . In doing so , noise injections in NRNNs may be viewed as a learning strategy . A similar stance is considered in [ 35 ] for treating neural SDEs . From this point of view , we may relate noise injections generally to regularization mechanisms considered in previous works . For example , additive noise injection was studied in the context of feedforward NNs in [ 6 ] , in which case a Gaussian noise is injected to the activation function at each layer of the NN . Furthermore , multiplicative noise injections includes stochastic depth and dropout strategies as special cases [ 36 , 35 ] . By taking a Gaussian approximation to Bernoulli noise and taking a continuous-time limit , NNs with stochastic dropout can be weakly approximated by an SDE with appropriate multiplicative noise , see [ 36 ] . All of these works highlight various advantages of noise injection for training NNs .

Summary: This work studies RNNs that are trained with noise injection in the framework of ODEs.  The method is called NRNN. The authors provided Theorems to: (1) identify the form of implicit regularization of NRNN; (2) show the improved model stability; (3) provide the bounds for RNN classifiers. The experiments are conducted on two classification tasks: Sequential MNIST and heart beats classification.