Noisy Recurrent Neural Networks

1 Introduction . Viewing recurrent neural networks ( RNNs ) as discretizations of ordinary differential equations ( ODEs ) driven by input data has recently gained attention [ 7 , 27 , 16 , 49 ] . The “ formulate in continuous time , and then discretize ” approach [ 38 ] motivates novel architecture designs before experimentation , and it provides a useful interpretation as a dynamical system . This , in turn , has led to gains in reliability and robustness to data perturbations . Recent efforts have shown how adding noise can also improve stability during training , and consequently improve robustness [ 35 ] . In this work , we consider discretizations of the corresponding stochastic differential equations ( SDEs ) obtained from ODE formulations of RNNs through the addition of a diffusion ( noise ) term . We refer to these as Noisy RNNs ( NRNNs ) . By dropping the noisy elements at inference time , NRNNs become a stochastic learning strategy which , as we shall prove , has a number of important benefits . In particular , stochastic learning strategies ( including dropout ) are often used as natural regularizers , favoring solutions in regions of the loss landscape with desirable properties ( often improved generalization and/or robustness ) . This mechanism is commonly referred to as implicit regularization [ 40 , 39 , 50 ] , differing from explicit regularization where the loss is explicitly modified . For neural network ( NN ) models , implicit regularization towards wider minima is conjectured to be a prominent ingredient in the success of stochastic optimization [ 67 , 28 ] . Indeed , implicit regularization has been linked to increases in classification margins [ 47 ] , which can lead to improved generalization performance [ 51 ] . A common approach to identify and study implicit regularization is to approximate the implicit regularization by an appropriate explicit regularizer 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 40 , 39 , 1 , 6 , 21 ] . Doing so , we will see that NRNNs favor wide minima ( like SGD ) ; more stable dynamics ; and classifiers with a large classification margin , keeping generalization error small . SDEs have also seen recent appearances in neural SDEs [ 59 , 24 ] , stochastic generalizations of neural ODEs [ 9 ] which can be seen as an analogue of NRNNs for non-sequential data , with a similar relationship to NRNNs as feedforward NNs do to RNNs . They have been shown to be robust in practice [ 35 ] . Analogously , we shall show that the NRNN framework leads to more reliable and robust RNN classifiers , whose promise is demonstrated by experiments on benchmark data sets . Contributions . For the class of NRNNs ( formulated first as a continuous-time model , which is then discretized ) , the following are our main contributions : • we identify the form of the implicit regularization for NRNNs through a corresponding ( datadependent ) explicit regularizer in the small noise regime ( see Theorem 1 ) ; • we focus on its effect in classification tasks , providing bounds for the classification margin for the deterministic RNN classifiers ( see Theorem 2 ) ; in particular , Theorem 2 reveals that stable RNN dynamics can lead to large classification margin ; • we show that noise injection can also lead to improved stability ( see Theorem 3 ) via a Lyapunov stability analysis of continuous-time NRNNs ; • we demonstrate via empirical experiments on benchmark data sets that NRNN classifiers are more robust to data perturbations when compared to other recurrent models , while retaining state-of-the-art performance for clean data . Research code is provided here : https : //github.com/erichson/NoisyRNN . Notation . We use ‖v‖ : = ‖v‖2 to denote the Euclidean norm of the vector v , and ‖A‖2 and ‖A‖F to denote the spectral norm and Frobenius norm of the matrix A , respectively . The ith element of a vector v is denoted by vi or [ v ] i , and the ( i , j ) -entry of a matrix A by Aij or [ A ] ij . For a vector v = ( v1 , . . . , vd ) , diag ( v ) denotes the diagonalization of v with diag ( v ) ii = vi . I denotes the identity matrix ( with dimension clear from context ) , while superscript T denotes transposition . For a matrix M , M sym = ( M + MT ) /2 denotes its symmetric part , λmin ( M ) and λmax ( M ) denote its minimum and maximum eigenvalue respectively , σmax ( M ) denotes its maximum singular value , and Tr ( M ) denotes its trace . For a function f : Rn → Rm such that each of its first-order partial derivatives ( with respect to x ) exist , ∂f∂x ∈ R m×n is the Jacobian matrix of f . For a scalar-valued function g : Rn → R , ∇hg is the gradient of g with respect to the variable h ∈ Rn and Hhg is the Hessian of g with respect to h . 2 Related Work . Dynamical Systems and Machine Learning . There are various interesting connections between machine learning and dynamical systems . Formulating machine learning in the framework of continuous-time dynamical systems was recently popularized by [ 62 ] . Subsequent efforts focus on constructing learning models by approximating continuous-time dynamical systems [ 9 , 29 , 48 ] and studying them using tools from numerical analysis [ 36 , 64 , 69 , 68 ] . On the other hand , dynamical systems theory provides useful theoretical tools for analyzing NNs , including RNNs [ 60 , 15 , 34 , 7 , 16 ] , and useful principles for designing NNs [ 23 , 54 ] . Other examples of dynamical systems inspired models include the learning of invariant quantities via their Hamiltonian or Lagrangian representations [ 37 , 22 , 10 , 71 , 58 ] . Another class of models is inspired by Koopman theory , yielding models where the evolution operator is linear [ 56 , 43 , 17 , 45 , 33 , 4 , 3 , 13 ] . Stochastic Training and Regularization Strategies . Regularization techniques such as noise injection and dropout can help to prevent overfitting in NNs . Following the classical work [ 5 ] that studies regularizing effects of noise injection on data , several work studies the effects of noise injection into different parts of networks for various architectures [ 25 , 44 , 35 , 54 , 2 , 26 , 66 , 61 ] . In particular , recently [ 6 ] studies the regularizing effect of isotropic Gaussian noise injection into the layers of feedforward networks . For RNNs , [ 12 ] shows that noise additions on the hidden states outperform Bernoulli dropout in terms of performance and bias , whereas [ 18 ] introduces a variant of stochastic RNNs for generative modeling of sequential data . Some specific formulations of RNNs as SDEs were also considered in Chapter 10 of [ 41 ] . Implicit regularization has also been studied more generally than stochastic gradient based training of NNs [ 40 , 39 , 19 , 11 ] . 3 Noisy Recurrent Neural Networks . We formulate continuous-time recurrent neural networks ( CT-RNNs ) at full generality as a system of input-driven ODEs : for a terminal time T > 0 and an input signal x = ( xt ) t∈ [ 0 , T ] ∈ C ( [ 0 , T ] ; Rdx ) , the output yt ∈ Rdy , for t ∈ [ 0 , T ] , is a linear map of hidden states ht ∈ Rdh satisfying dht = f ( ht , xt ) dt , yt = V ht , ( 1 ) where V ∈ Rdy×dh , and f : Rdh × Rdx → Rdh is typically Lipschitz continuous , guaranteeing existence and uniqueness of solutions to ( 1 ) . A natural stochastic variant of CT-RNNs arises by replacing the ODE in ( 1 ) by an Itô SDE , that is , dht = f ( ht , xt ) dt+ σ ( ht , xt ) dBt , yt = V ht , ( 2 ) where σ : Rdh × Rdx → Rdh×r and ( Bt ) t≥0 is an r-dimensional Brownian motion . The functions f , σ are referred to as the drift and diffusion coefficients , respectively . Intuitively , ( 2 ) amounts to a noisy perturbation of the corresponding deterministic CT-RNN ( 1 ) . At full generality , we refer to the system ( 2 ) as a continuous-time Noisy RNN ( CT-NRNN ) . To guarantee the existence of a unique solution to ( 2 ) , in the sequel , we assume that { f ( · , xt ) } t∈ [ 0 , T ] and { σ ( · , xt ) } t∈ [ 0 , T ] are uniformly Lipschitz continuous , and t 7→ f ( h , xt ) , t 7→ σ ( h , xt ) are bounded in t ∈ [ 0 , T ] for each fixed h ∈ Rdh . For further details , see Section B in Supplementary Material ( SM ) . While much of our theoretical analysis will focus on this general formulation of CT-NRNNs , our empirical and stability analyses focus on the choice of drift function f ( h , x ) = Ah+ a ( Wh+ Ux+ b ) , ( 3 ) where a : R → R is a Lipschitz continuous scalar activation function extended to act on vectors pointwise , A , W ∈ Rdh×dh , U ∈ Rdh×dx and b ∈ Rdh . Typical examples of activation functions include a ( x ) = tanh ( x ) . The matrices A , W , U , V , b are all assumed to be trainable parameters . This particular choice of drift dates back to the early Cohen-Grossberg formulation of CT-RNNs , and was recently reconsidered in [ 16 ] . 3.1 Noise Injections as Stochastic Learning Strategies . While precise choices of drift functions f are the subject of existing deterministic RNN theory , good choices of the diffusion coefficient σ are less clear . Here , we shall consider a parametric class of diffusion coefficients given by : σ ( h , x ) ≡ ( σ1I + σ2diag ( f ( h , x ) ) ) , ( 4 ) where the noise level > 0 is small , and σ1 ≥ 0 and σ2 ≥ 0 are tunable parameters describing the relative strength of additive noise and a multiplicative noise respectively . While the stochastic component is an important part of the model , one can set ≡ 0 at inference time . In doing so , noise injections in NRNNs may be viewed as a learning strategy . A similar stance is considered in [ 35 ] for treating neural SDEs . From this point of view , we may relate noise injections generally to regularization mechanisms considered in previous works . For example , additive noise injection was studied in the context of feedforward NNs in [ 6 ] , in which case a Gaussian noise is injected to the activation function at each layer of the NN . Furthermore , multiplicative noise injections includes stochastic depth and dropout strategies as special cases [ 36 , 35 ] . By taking a Gaussian approximation to Bernoulli noise and taking a continuous-time limit , NNs with stochastic dropout can be weakly approximated by an SDE with appropriate multiplicative noise , see [ 36 ] . All of these works highlight various advantages of noise injection for training NNs .

Summary: This paper studies noise injection in RNNs, which can be regarded as a discretized SDE driven by external inputs. The first main result of the paper is the derivation of the implicit regularization caused by noise injection (similar to noise injection for feed-forward models), which is shown to be related to the Hessian of the loss (promoting flatter minima) and the Jacobian of the dynamics (promoting slower/more stable dynamics). A further implication on the classification margin is presented. Lastly, the paper presents a result showing that noise injection can further improve the temporal stability of the RNN dynamics. These motivate the use of the noise injection for recurrent neural networks, and experiments are provided to justify the theoretical results.