Improved memory in recurrent neural networks with sequential non-normal dynamics

1 INTRODUCTION . Modeling long-term dependencies with recurrent neural networks ( RNNs ) is a hard problem due to degeneracies inherent in the optimization landscapes of these models , a problem also known as the vanishing/exploding gradients problem ( Hochreiter , 1991 ; Bengio et al. , 1994 ) . One approach to addressing this problem has been designing new RNN architectures that are less prone to such difficulties , hence are better able to capture long-term dependencies in sequential data ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ; Chang et al. , 2017 ; Bai et al. , 2018 ) . An alternative approach is to stick with the basic vanilla RNN architecture instead , but to constrain its dynamics in some way so as to eliminate or reduce the degeneracies that otherwise afflict the optimization landscape . Previous proposals belonging to this second category generally boil down to orthogonalization of the recurrent dynamics , either at initialization or during the entire training period ( Le et al. , 2015 ; Arjovsky et al. , 2016 ; Wisdom et al. , 2016 ) . The basic idea behind these methods is that orthogonal transformations are isometries of the Euclidean space , hence they preserve distances and norms , which enables them to deal effectively with the vanishing/exploding gradients problem . However , this idea ignores the crucial effects of non-linearity and noise . Orthogonal transformations no longer preserve distances and norms in the presence of a non-linearity , suggesting that alternative transformations might be better suited to non-linear networks ( this point was noted by Pennington et al . ( 2017 ) and Chen et al . ( 2018 ) before , where isometric initializations that take the non-linearity into account were proposed ) . Similarly , in the presence of noise , norm preservation itself ceases to be the ideal objective . One must instead maximize the signal-to-noise ratio ( SNR ) of the propagated signal . In neural networks , noise comes in both through the stochasticity of the stochastic gradient descent ( SGD ) algorithm and sometimes also through direct noise injection for regularization purposes , as in dropout ( Srivastava et al. , 2014 ) . Previous work has shown that even in a simple linear setting , recurrent networks that maximize the SNR display strongly non-normal , sequential dynamics and orthogonal networks are highly suboptimal by this measure ( Ganguli et al. , 2008 ) . Motivated by these observations , in this paper , we investigate the potential of non-normal RNNs , i.e . RNNs with a non-normal recurrent connectivity matrix , in sequential processing tasks . Recall that a normal matrix is a matrix with an orthonormal set of eigenvectors , whereas a non-normal matrix does not have an orthonormal set of eigenvectors . This property allows non-normal systems to display interesting transient behaviors that are not available in normal systems . This kind of transient behavior , specifically a particular kind of transient amplification of the signal in certain non-normal systems , underlies their superior memory properties ( Ganguli et al. , 2008 ) , as will be discussed further below . Our empirical results show that non-normal vanilla RNNs significantly outperform their orthogonal counterparts in a diverse range of benchmarks.1 2 BACKGROUND . 2.1 MEMORY IN LINEAR RECURRENT NETWORKS WITH NOISE . Ganguli et al . ( 2008 ) studied memory properties of linear recurrent networks injected with a scalar temporal signal st , and noise zt : ht = Wht−1 + vst + zt ( 1 ) The noise is assumed to be i.i.d . with zt ∼ N ( 0 , I ) . Ganguli et al . ( 2008 ) then analyzed the Fisher memory matrix ( FMM ) of this system , defined as : Jkl ( s≤t ) = 〈 − ∂ 2 ∂st−k∂st−l log p ( ht|s≤t ) 〉 p ( ht|s≤t ) ( 2 ) For linear networks with Gaussian noise , it is easy to show that Jkl ( s≤t ) is , in fact , independent of the past signal history s≤t . Ganguli et al . ( 2008 ) specifically analyzed the diagonal of the FMM : J ( k ) ≡ Jkk , which can be written explicitly as : J ( k ) = v > Wk > C−1Wkv ( 3 ) where C = ∑∞ k=0 W kWk > is the noise covariance matrix , and the norm of Wkv can be roughly thought of as representing the signal strength . The total Fisher memory is the sum of J ( k ) over all past time steps k : Jtot = ∞∑ k=0 J ( k ) ( 4 ) Intuitively , J ( k ) measures the information contained in the current state of the system , ht , about a signal that entered the system k time steps ago , st−k . Jtot is then a measure of the total information contained in the current state of the system about the entire past signal history , s≤t . The main result in Ganguli et al . ( 2008 ) shows that Jtot = 1 for all normal matrices W ( including all orthogonal matrices ) , whereas in general Jtot ≤ N , where N is the network size . Remarkably , the memory upper bound can be achieved by certain highly non-normal systems and several examples are explicitly given in Ganguli et al . ( 2008 ) . Two of those examples are illustrated in Figure 1a ( right ) : a uni-directional “ chain ” network and a chain network with feedback . In the chain network , the recurrent connectivity is given by Wij = αδj , i−1 and in the chain with feedback network , it is given by Wij = αδj , i−1 + βδj , i+1 , where α and β are the feedforward and feedback connection weights , respectively ( here δ denotes the Kronecker delta function ) . In addition , in order to achieve optimal memory , the signal must be fed at the source neuron in these networks , i.e . v = [ 1 , 0 , 0 , . . . , 0 ] > . Figure 1b compares the Fisher memory curves , J ( k ) , of these non-normal networks with the Fisher memory curves of two example normal networks , namely recurrent networks with identity or random orthogonal connectivity matrices . The two non-normal networks have extensive memory capacity , i.e . Jtot ∼ O ( N ) , whereas for the normal examples , Jtot = 1 . The crucial property that enables extensive memory in non-normal networks is transient amplification : after the signal enters the network , it is amplified supralinearly for a time of length O ( N ) before it eventually dies out ( Figure 1c ) . This kind of transient amplification is not possible in normal networks . 1Code available at : https : //github.com/eminorhan/nonnormal-init 2.2 A TOY NON-LINEAR EXAMPLE : NON-LINEARITY AND NOISE INDUCE SIMILAR EFFECTS . The preceding analysis by Ganguli et al . ( 2008 ) is exact in linear networks . Analysis becomes more difficult in the presence of a non-linearity . However , we now demonstrate that the non-normal networks shown in Figure 1a have advantages that extend beyond the linear case . The advantages in the non-linear case are due to reduced interference in these non-normal networks between signals entering the network at different time points in the past . To demonstrate this with a simple example , we will ignore the effect of noise for now and consider the effect of non-linearity on the linear decodability of past signals from the current network activity . We thus consider deterministic non-linear networks of the form ( see Appendix A for additional details ) : ht = f ( Wht−1 + vst ) ( 5 ) and ask how well we can linearly decode a signal that entered the network k time steps ago , st−k , from the current activity of the network , ht . Figure 2c compares the decoding performance in a non-linear orthogonal network with the decoding performance in the non-linear chain network . Just as in the linear case with noise ( Figure 2b ) , the chain network outperforms the orthogonal network . To understand intuitively why this is the case , consider a chain network with Wij = δj , i−1 and v = [ 1 , 0 , 0 , . . . , 0 ] > . In this model , the responses of the N neurons after N time steps ( at t = N ) are given by f ( sN ) , f ( f ( sN−1 ) ) , ... , f ( f ( . . . f ( s1 ) . . . ) ) , respectively , starting from the source neuron . Although the non-linearity f ( · ) makes perfect linear decoding of the past signal st−k impossible , one may still imagine being able to decode the past signal with reasonable accuracy as long as f ( · ) is not “ too non-linear ” . A similar intuition holds for the chain network with feedback as well , as long as the feedforward connection weight , α , is sufficiently stronger than the feedback connection strength , β . A condition like this must already be satisfied if the network is to maintain its optimal memory properties and also be dynamically stable at the same time ( Ganguli et al. , 2008 ) . In normal networks , however , linear decoding is further degraded by interference from signals entering the network at different time points , in addition to the degradation caused by the nonlinearity . This is easiest to see in the identity network ( a similar argument holds for the random orthogonal example too ) , where the responses of the neurons after N time steps are identically given by f ( f ( . . . f ( f ( s1 ) +s2 ) . . . ) +sN ) , if one assumes v = [ 1 , 1 , 1 , . . . , 1 ] > . Linear decoding is harder in this case , because a signal st−k is both distorted by multiple steps of non-linearity and also mixed with signals entering at other time points . 3 RESULTS . 3.1 EXPERIMENTS . Because assuming an a priori fixed non-normal structure for an RNN runs the risk of being too restrictive , in this paper , we instead explore the promise of non-normal networks as initializers for RNNs . Throughout the paper , we will be primarily comparing the four RNN architectures schematically depicted in Figure 1a as initializers : two of them normal networks ( identity and random orthogonal ) and the other two non-normal networks ( chain and chain with feedback ) , the last two being motivated by their optimal memory properties in the linear case , as reviewed above . 3.1.1 COPY , ADDITION , PERMUTED SEQUENTIAL MNIST . Copy , addition , and permuted sequential MNIST tasks were commonly used as benchmarks in previous RNN studies ( Arjovsky et al. , 2016 ; Bai et al. , 2018 ; Chang et al. , 2017 ; Hochreiter & Schmidhuber , 1997 ; Le et al. , 2015 ; Wisdom et al. , 2016 ) . We now briefly describe each of these tasks . Copy task : The input is a sequence of integers of length T . The first 10 integers in the sequence define the target subsequence that is to be copied and consist of integers between 1 and 8 ( inclusive ) . The next T − 21 integers are set to 0 . The integer after that is set to 9 , which acts as the cue indicating that the model should start copying the target subsequence . The final 10 integers are set to 0 . The output sequence that the model is trained to reproduce consists of T − 10 0s followed by the target subsequence from the input that is to be copied . To make sure that the task requires a sufficiently long memory capacity , we used a large sequence length , T = 500 , comparable to the largest sequence length considered in Arjovsky et al . ( 2016 ) for the same task . Addition task : The input consists of two sequences of length T . The first one is a sequence of random numbers drawn uniformly from the interval [ 0 , 1 ] . The second sequence is an indicator sequence with 1s at exactly two positions and 0s everywhere else . The positions of the two 1s indicate the positions of the numbers to be added in the first sequence . The target output is the sum of the two corresponding numbers . The position of the first 1 is drawn uniformly from the first half of the sequence and the position of the second 1 is drawn uniformly from the second half of the sequence . Again , to ensure that the task requires a sufficiently long memory capacity , we chose T = 750 , which is the same as the largest sequence length considered in Arjovsky et al . ( 2016 ) for the same task . Permuted sequential MNIST ( psMNIST ) : This is a sequential version of the standard MNIST benchmark where the pixels are fed to the model one pixel at a time . To make the task hard enough , we used the permuted version of the sequential MNIST task where a fixed random permutation is applied to the pixels to eliminate any spatial structure before they are fed into the model . We used vanilla RNNs with N = 25 recurrent units in the psMNIST task and N = 100 recurrent units in the copy and addition tasks . We used the elu nonlinearity for the copy and the psMNIST tasks ( Clevert et al. , 2016 ) , and the relu nonlinearity for the addition problem ( because relu proved to be more natural for remembering positive numbers ) . Batch size was 16 in all tasks . As mentioned above , the scaled identity and the scaled random orthogonal networks constituted the normal initializers . In the scaled identity initializer , the recurrent connectivity matrix was initialized as W = λI and the input matrix V was initialized as Vij ∼ N ( 0 , 0.9/ √ N ) . In the random orthogonal initializer , the recurrent connectivity matrix was initialized as W = λQ , where Q is a random dense orthogonal matrix , and the input matrix V was initialized in the same way as in the identity initializer . The feedforward chain and the chain with feedback networks constituted our non-normal initializers . In the chain initializer , the recurrent connectivity matrix was initialized as Wij = αδj , i−1 and the input matrix V was initialized as V ∼ 0.9IN×d , where IN×d denotes the N ×d-dimensional identity matrix . Note that this choice of V is a natural generalization of the the source injecting input vector that was found to be optimal in the linear case with scalar signals to multi-dimensional inputs ( as long as N d ) . In the chain with feedback initializer , the recurrent connectivity matrix was initialized as Wij = 0.99δj , i−1 + βδj , i+1 and the input matrix V was initialized in the same way as in the chain initializer . We used the rmsprop optimizer for all models , which we found to be the best method for this set of tasks . The learning rate of the optimizer was a hyperparameter which we tuned separately for each model and each task . The following learning rates were considered in the hyper-parameter search : 8×10−4 , 5×10−4 , 3×10−4 , 10−4 , 8×10−5 , 5×10−5 , 3×10−5 , 10−5 , 8×10−6 , 5×10−6 , 3×10−6 . We ran each model on each task 6 times using the integers from 1 to 6 as random seeds . In addition , the following model-specific hyperparameters were searched over for each task : Chain : feedforward connection weight , α ∈ { 0.99 , 1.00 , 1.01 , 1.02 , 1.03 , 1.04 , 1.05 } . Chain with feedback : feedback connection weight , β ∈ { 0.01 , 0.02 , 0.03 , 0.04 , 0.05 , 0.06 , 0.07 } . Scaled identity : scale , λ ∈ { 0.01 , 0.96 , 0.99 , 1.0 , 1.01 , 1.02 , 1.03 , 1.04 , 1.05 } . Random orthogonal : scale , λ ∈ { 0.01 , 0.96 , 0.99 , 1.0 , 1.01 , 1.02 , 1.03 , 1.04 , 1.05 } . This yields a total of 7× 11× 6 = 462 different runs for each experiment in the non-normal models and a total of 9× 11× 6 = 594 different runs in the normal models . Note that we ran more extensive hyper-parameter searches for the normal models than for the non-normal models in this set of tasks . Figure 3a-c shows the validation losses for each model with the best hyper-parameter settings . The non-normal initializers generally outperform the normal initializers . Figure 3d-f shows for each model the number of “ successful ” runs that converged to a validation loss below a criterion level ( which we set to be 50 % of the loss for a baseline random model ) . The chain model outperformed all other models by this measure ( despite having a smaller total number of runs than the normal models ) . In the copy task , for example , none of the runs for the normal models was able to achieve the criterion level , whereas 46 out of 462 runs for the chain model and 11 out of 462 runs for the feedback chain model reached the criterion loss ( see Appendices B & C for further results and discussion ) . 3.1.2 LANGUAGE MODELING EXPERIMENTS To investigate if the benefits of nonnormal initializers extend to more realistic problems , we conducted experiments with three standard language modeling tasks : word-level Penn Treebank ( PTB ) , character-level PTB , and character-level enwik8 benchmarks . For the language modeling experiments in this subsection , we used the code base provided by Salesforce Research ( Merity et al. , 2018a ; b ) : https : //github.com/ salesforce/awd-lstm-lm . We refer the reader to Merity et al . ( 2018a ; b ) for a more detailed description of the benchmarks . For the experiments in this subsection , we generally preserved the model setup used in Merity et al . ( 2018a ; b ) , except for the following differences : 1 ) We replaced the gated RNN architectures ( LSTMs and QRNNs ) used in Merity et al . ( 2018a ; b ) with vanilla RNNs ; 2 ) We observed that vanilla RNNs require weaker regularization than gated RNN architectures . Therefore , in the word-level PTB task , we set all dropout rates to 0.1 . In the character-level PTB task , all dropout rates except dropoute were set to 0.1 , which was set to 0 . In the enwik8 benchmark , all dropout rates were set to 0 ; 3 ) We trained the word-level PTB models for 60 epochs , the character-level PTB models for 500 epochs and the enwik8 models for 35 epochs . We compared the same four models described in the previous subsection . As in Merity et al . ( 2018a ) , we used the Adam optimizer and thus only optimized the α , β , λ hyper-parameters for the experiments in this subsection . For the hyper-parameter α in the chain model and the hyper-parameter λ in the scaled identity and random orthogonal models , we searched over 21 values uniformly spaced between 0.05 and 1.05 ( inclusive ) ; whereas for the chain with feedback model , we set the feedforward connection weight , α , to the optimal value it had in the chain model and searched over 21 β values uniformly spaced between 0.01 and 0.21 ( inclusive ) . In addition , we repeated each experiment 3 times using different random seeds , yielding a total of 63 runs for each model and each benchmark . The results are shown in Figure 4 and in Table 1 . Figure 4 shows the validation loss over the course of training in units of bits per character ( bpc ) . Table 1 reports the test losses at the end of training . The non-normal models outperform the normal models on the word-level and character-level PTB benchmarks . The differences between the models are less clear on the enwik8 benchmark . However , in terms of the test loss , the non-normal feedback chain model outperforms the other models on all three benchmarks ( Table 1 ) . We note that the vanilla RNN models perform significantly worse than the gated RNN architectures considered in Merity et al . ( 2018a ; b ) . We conjecture that this is because gated architectures are generally better at modeling contextual dependencies , hence they have inductive biases better suited to language modeling tasks . The primary benefit of non-normal dynamics , on the other hand , is enabling a longer memory capacity . Below , we will discuss whether non-normal dynamics can be used in gated RNN architectures to improve performance as well .

Summary:  This paper proposes to explore nonnormal matrix initialization in RNNs. Authors demonstrate on various tasks (Copy/Addition, Permuted-SMNIST, PTB, enwik8) that chain-like nonnormal matrix initializations can outperform orthogonal or identity initialization in vanilla RNNs. However, nonnormal RNNs underperform their gated counterpart such as LSTM. Authors also show results where they use their initialization scheme in update gate of a LSTM.