Improved memory in recurrent neural networks with sequential non-normal dynamics
Authors: Emin Orhan
Xaq Pitkow
Venue:  (2020)
Keywords: recurrent neural networks
memory
non-normal dynamics

Abstract: Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period. The basic motivation behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve (Euclidean) norms and effectively deal with vanishing/exploding gradients. However, this ignores the crucial effects of non-linearity and noise. In the presence of a non-linearity, orthogonal transformations no longer preserve norms, suggesting that alternative transformations might be better suited to non-linear networks. Moreover, in the presence of noise, norm preservation itself ceases to be the ideal objective. A more sensible objective is maximizing the signal-to-noise ratio (SNR) of the propagated signal instead. Previous work has shown that in the linear case, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure. Motivated by this finding, here we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks. Our experimental results show that non-normal RNNs outperform their orthogonal counterparts in a diverse range of benchmarks. We also find evidence for increased non-normality and hidden chain-like feedforward motifs in trained RNNs initialized with orthogonal recurrent connectivity matrices. 

Published as a conference paper at ICLR 2020
IMPROVED MEMORY IN RECURRENT NEURAL NET- WORKS WITH SEQUENTIAL NON-NORMAN DYNAMICS
A. Emin Orhan' and Xaq P 2.3
NNw York University (eo4 1 (ennun.ueu) PRice University, Baylor College of Medicine
ABSTRACT
Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period. The basic motivation behind these methods that orthogonal transformations are isometries of the Euclidean space, hence they preserve (Fuclidean) norms and effectively deal with vanish- ing/exploding gradients. However, this ignores the crucial effects of non-linearity and noise. In the presence of non-linearity, orthogonal transformations longer preserve norms, suggesting that alternative transformations might be better suited non-linear networks. Moreover, in the presence of noise, norm preservation itself ceases to be the ideal objective. A more sensible objective is maximizing the signal- to-noise ratio (SNR) of the propagated signal instead. Previous work has shown that in the linear case, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamies and orthogonal networks are highly suboptimal by this measure. Motivated by this finding, here we investigate the potential of non-normal RNNs, i.c. RNNs with a non-norma recurrent connectivity matrix, in sequential processing tasks. Our experimental results show that non-normal RNNs outperform their orthogonal counterparts in a diverse range of benchmarks. We also find evidence for increased normality and hidden chain- like feedforward motifs in trained RNNs initialized with orthogonal recurrent connectivity matrices.
INTRODUCTION
Modeling long-term dependencies with recurrent neural networks (RNNs) hard problem due to degeneracies inherent in the optimization landscapes of these models, problem also known as the vanishing/exploding gradients problem (Hochreiter) 1991 Bengio et al. 1994). One approach to addressing this problem has been designing new RNN architectures that are less prone to such difficulties, hence are better able to capture long-tern mermteermae dependencies in sequential data (Hochreiter & Schmidhuber] 1997 Cho et al.) 2014 Chang et al. 2017 Bai al. 2018). An alternative approach is to stick with the basic vanilla RNN architecture instead, but to constrain its dynamics in some way so as to eliminate or reduce the degeneracies that otherwise afflict the optimization landscape. Previous proposals belonging to this second category generally boil down to orthogonalization of the recurrent dynamics, either initialization or during the entire training period (Le al. 2013 Arjovsky et al.) 2016; Wisdom et al., 2016). The basic idea behind these methods is that orthogonal transformations are isometries the Euchdean space, hence they preserve distances and norms, which enables them to deal effectively with the vanishing/explodingg gradients problem.
However, this idea ignores the crucial effects of non- con-linearity and noise. Orthogonal transformations no longer preserve distances and norms in the presence of non- non-linear,ty, suggesting that alternative transformations might be better suited to non-linear networks (this point was noted by Pennington et al.) (2017) and Chen et al. (2018) before, where isometric initializations that take the non-linearity into account were proposed). Similarly, in the presence of noise, norm preservation itself ceases to be the ideal objective. One must instead maximize the signal-to-noie noise ratio (SNR) of the propagated signal. In neural networks, noise comes in both through the stochasticity of the stochastic gradient descent (SGD) algorithm and sometimes also through direct noise injection for regularization purposes, as

---

Published as a conference paper at ICLR 2020
in dropout (Srivastava al., 2014). Previous work has shown that even in simple linear setting, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure (Ganguli et al. 2008).
Motivated by these observations, in this paper, we investigate the potential of non-normal RNNs, Le. RNNs with non-normal recurrent connectivity matrix, in sequential processing tasks. Recall that a normal matrix is a matrix with an orthonormal set of eigenvectors, whereas a non-normal matrix does not have an orthonormal set of eigenvectors. This property allows non-normal systems to display interesting transient behaviors that are not available in normal systems. This kind of transient behavior, specifically a particular kind of transient amplification of the signal in certain non-normal systems, underlies their superior memory properties (Ganguli et al.) (2008), as will be discussed further below. Our empirical results show that non- non-normal vanilla RNNs significantly outperform their orthogonal counterparts in diverse range of benchmarks
2
BACKGROUND
2.1
MEMORY IN LINEAR RECURRENT NETWORKS WITH NOISE
Ganguli et al. (0088) studied memory properties of linear recurrent networks injected with scalar temporal signal St, and noise Z₁
h; Wh 1 + v8, +
(1)
The noise is assumed to be i.i.d. with Z, NN I). Ganguli et al. (2008) then analyzed the Fisher memory matrix (FMM) of this system, defined as:
0
(u((s))
log pa (sgr) (br(sg))
as :
p(h((sge)
For linear networks with Gaussian noise, it is easy to show that J Jar() is, in fact, independent of the past signal history 8ge- Ganguli et al. (2008) specifically analyzed the diagonal of the FMM: (k) = JRk, which can be written explicitly as:
J(k) = v Tw
where C = W* is the noise covariance matrix, and the norm of W can be roughly thought of as representing the signal strength. The total Fisher memory is the sum of ((k) over all past time steps
Jont J(k)
(4)
A=
Intuitively, (k) measures the information contained in the current state of the system, h, about signal that entered the system & time steps ago, $t-k- Joot is then a measure of the total information contained in the current state of the system about the entire past signal history, sge.
The main result in Ganguli et al. (2008) shows that Joo = for all normal matrices W (including all orthogonal matrices), whereas in general Jeot S N, where N is the network size. Remarkably, the memory upper bound can be achieved by certain highly non-normal systems and several examples explicitly given in Ganguli al. (2008). Two of those examples are illustrated in Figure Th (right): a uni-directional "chain" network and chain network with feedback. In the chain network, the recurrent connectivity is given by 51-- = a-1 and in the chain with feedback network, it is given by W ad,, + 35,,++1, where a and 3 are the feedforward and feedback connection weights, respectively (here 8 denotes the Kronecker delta function). In addition, in order to achieve optimal memory, the signal must be fed at the source neuron in these networks, i.e. v =
Figure D compares the Fisher memory curves, Jkk., of these non- non-normal networks with the Fisher memory curves of two example normal networks, namely recurrent networks with identity or random orthogonal connectivity matrices. The two non-normal networks have extensive memory capacity, i.e. Jiat ~ O(N), whereas for the normal examples, Jhot 1. The crucial property that enables extensive memory in non-normal networks is transient amplification: after the signal enters the network, is amplified supralinearly for time of length O(N) before it eventually dies out (Figure Te). This kind of transient amplification is not possible in normal networks.
Code available at: https://gitthb.. com www.hhh/ooohnnomnnmnommmmmmomaaa -iint
2

---

Published as a conference paper at ICLR 2020
a
Normal
Non-normal
Identity
Orthogonal
Chain
Chain with feedback
Memory curve
Signal
20
Ideniiy
Orthogonel
Feodback chain
100
200
100
200
&
k
(times steps in the past)
Figure 1: Schematic diagrams of different recurrent networks and the corresponding recurrent connectivity matrices (upper panel). Memory curves, J(k) (Equation 3), for the four recurrent networks shown in The non-normal networks, chain and chain with feedback, have extensive memory capacity: Jeot ~ O(N), whereas the normal networks, identity and random orthogonal, have Joot = 1. Extensive memory made possible in non-normal networks by fransient amplification: the signal is amplified for a time of length O(N) before it dies out, abruptly in the case the chain network and more gradually in the case of the chain network with feedback. In b and c, the network size is N 100 for all four networks.
2.2
A TOY NON-LINEAR EXAMPLE: NON-LINEARITY AND NOISE INDUCE SIMILAR EFFECTS
The preceding analysis by Ganguli et al. (2088) is exact linear networks. Analysis becomes more difficult in the presence of a non-linearity However, we now demonstrate that the non-normal networks shown in Figure Th have advantages that extend beyond the linear case. The advantages in are due to reduced interference in these non-normal networks between signals
the non-linear entering the network at different time points in the past.
To demonstrate this with simple example, will ignore the effect of noise for and consider the effect of non-linearity on the linear decodability of past signals from the current network activity. We thus consider deterministic non-linear networks of the form (see Appendix A for additional details):
f(Wh:-1 vs:)
(5)
and ask how well we can linearly decode signal that entered the network k time steps ago, 8t-k. from the current activity of the network, h, Figure 2 compares the decoding performance in non-linear orthogonal network with the decoding performance the non-linear chain network. Just as in the linear case with noise (Figure the chain network outperforms the orthogonal network.
To understand intuitively why this is the case, consider chain network with Wij 65,t-1 and = 0, ... OJT, In this model, the responses of the N neurons after N time steps (at t N) given by f(xx), ((f(sN-)), Although the non- con-linearity f(-) makes perfect linear decoding of the past signal s impossible, one may still imagine being able to decode the past signal with reasonable accuracy as long as f(-)) is not "too non-linear". similar intuition holds for the chain network with feedback as well, as long as the feedforward connection weight, a, is sufficiently stronger than the feedback connection strength, 3. condition like this must already be satisfied if the network is to maintain its optimal memory properties and also be dynamically stable the same time (Ganguli al. 2008).
f())...(s))........
respectively, starting from the source neuron.
In normal networks, however, linear decoding is further degraded by interference from signals entering the network different time points, in addition to the degradation caused by the non- linearity. This casiest to see in the identity network (a similar argument holds for the random orthogonal example too), where the responses of the neurons after N time steps identically given
3

---

Published as a conference paper at ICLR 2020
a
Linear no noise
b
Linear noise
Non-linear + no noise
3
3
Orthogonal
0
R²
1.0
R
0.73
R²
-3
-3
0
3
-3
3
.3
0
$1
$1
$1
3
3
3
Chain
55' 0
'5
56
0
-3 R²
1.0
R² = 0.82
R 0.84
33
3
81
81
Figure 2: Linear decoding experiments. In linear network with no noise, the past signal 8 can be perfectly reconstructed from the current activity vector h100 using linear decoder. b When noise added, the chain network outperforms the orthogonal network predicted from the theory in Ganguli et al. (2008). e In completely deterministic system, introducing a non-linearity has similar effect to that noise. The chain network again outperforms the orthogonal one when the signal is reconstructed with linear decoder. As discussed further in the text, this is because the signal is subject more interference in the orthogonal network than in the chain network. All simulations in this figure used networks with N 100 recurrent units. In we used the e elu non-linearity for f(-) (Clevert et al. 2016). For the chain network, we assume that the signal is fed at the source neuron.
by S( [((ss))) sN), if one assumes v 1,, 1]_ Linear decoding is harder this case, because signal 8 & both distorted by multiple steps of non-linearity and also mixed with signals entering at other time points.
RESULTS
3.1 EXPERIMENTS
Because assuming an priori fixed non-normal structure for an RNN runs the risk of being too restrictive, in this paper, we instead explore the promise of non-normal networks as initializers for RNNs. Throughout the paper, we will be primarily comparing the four RNN architectures schematically depicted in Figure as initializers: two of them normal networks (identity and random orthogonal) and the other two non-normal networks (chain and chain with feedback), the last two being motivated by their optimal memory properties the linear case, as reviewed above.
3.1.1 COPY, ADDITION, PERMUTED SEQUENTIAL MNIST
Copy, addition, and permuted sequential MNIST tasks were commonly used as benchmarks in previous RNN studies (Arjovsky et al.) 2016 Bai et al. 2018 Chang al. 2017;; Hochreiter &
Schmidhuber 1997 Le al. 2015 Wisdom et al. 2016).
We now briefly describe each these
tasks.
The input is a sequence of integers of length T. The first 10 integers in the sequence define the target subsequence that is to be copied and consist of integers between 1 and 8 (inclusive). The next T integers are set to 0. The integer after that is set to 9. which acts as the cue indicating that the model should start copying the target subsequence. The final 10 integers are set 0. The
Copy task:
4

---

Published as a conference paper at ICLR 2020
output sequence that the model is trained to reproduce consists of - 10 Os followed by the target subsequence from the input that is to be copied. To make sure that the task requires sufficiently long memory capacity, we used a large sequence length, T 500, comparable to the largest sequence length considered in Arjovsky et al. (2016) for the same task.
Addition task: The input consists of two sequences of length T. The first one is sequence of random numbers drawn uniformly from the interval [0, 1]. The second sequence is an indicator sequence with Is at exactly two positions and Os everywhere else. The positions of the two is indicate the positions of the numbers to be added in the first sequence. The target output is the sum of the two corresponding numbers. The position of the first is drawn uniformly from the first half of the sequence and the position of the second is drawn uniformly from the second half of the sequence. Again, to ensure that the task requires sufficiently long memory capacity, we chose T = 750, which is the same as the largest sequence length considered in Arjovsky et al. (2016) for the same task.
Permuted sequential MNIST (psMNIST); This sequential version of the standard MNIST benchmark where the pixels are fed to the model one pixel at time. To make the task hard enough, we used the permuted version of the sequential MNIST task where a fixed random permutation is applied to the pixels to eliminate any spatial structure before they are fed into the model.
We used vanilla RNNs with = 25 recurrent units in the PSMNIST task and N 100 recurrent units in the copy and addition tasks. We used the elu nonlincarity for the copy and the PSMNIST tasks (Clevert et al.] 2016), and the relu nonlinearity for the addition problem (because relu proved to be more natural for remembering positive numbers). Batch size 16 in all tasks.
As mentioned above, the scaled identity and the scaled random orthogonal networks constituted the normal initializers. In the scaled identity initializer, the recurrent connectivity matrix initialized as
W = AI and the input matrix V was initialized as Vij AO 0.9/v/) In the random orthogonal initializer, the recurrent connectivity matrix was initialized as W = XQ, where Q is a random dense orthogonal matrix, and the input matrix V was initialized in the same way as in the identity initializer.
The feedforward chain and the chain with feedback networks constituted our non-normal initializers. In the chain initializer, the recurrent connectivity matrix was initialized as Wij = oo5y,-1 and the input matrix V was initialized as V ~ 0.9I where I denotes the N x d-dimensional identity matrix. Note that this choice of V is a natural generalization of the the source injecting input vector that was found to be optimal in the linear case with scalar signals to multi-diimenionall inputs (as long as N >> d). In the chain with feedback initializer, the recurrent connectivity matrix was initialized as W sy 0.998.,5-1 + 35,5+1 and the input matrix V was initialized in the same way as in the chain initializer.
We used the rmsprop optimizer for all models, which we found to be the best method for this set of tasks. The learning rate of the optimizer was a hyperparameter which we tuned separately for each model and each task. The following learning rates were considered in the hyper-parameed
search: 8x 10 5x 10 3 10 10 8x 10-5 5 5 x 10 15 3 10 -5, 10 5 8 10 5 10 x3x10 We ran each model each task 6 times using the integers from 1 to 6 as random seeds.
In addition, the following model-specific hyperparameters were searched over for each task:
Chain: feedforward connection weight, {0.99, 1.00, 1.01, 1.02, 1.03, 1.04, 1.05].
Chain with feedback: feedback connection weight, 3 E (0.01,0.02, 0.03, 0.04, 0.05, 0.06, 0.073.
Scaled identity: scale, À E {0.01, 0.96, 0.99, 1.0, 1.01, 1.02, 1.03, 1.04, 1.053.
Random orthogonal: scale, X E (0.01, 0.96, 0.99, 1.0, 1.01, 1.02, 1.03, 1.04, 1.05).
This yields a total of 7 x x 6- = 462 different runs for each experiment in the non-normal models and total of 9 x 11 x 6 594 different runs in the normal models. Note that we ran more extensive hyper-parameter searches for the normal models than for the non-normal models in this set of tasks.
Figure B-cc shows the validation losses for each model with the best hyper-parameter settings. The non-normal initializers generally outperform the normal initializers. Figure 33- shows for each model the number of "successful" runs that converged to validation loss below criterion level (which we set to be 50% of the loss for a baseline random model). The chain model outperformed all other models by this measure (despite having smaller total number of runs than the normal models).
5

---

Published as a conference paper at ICLR 2020
In the copy task, for example, none of the runs for the normal models able to achieve the criterion level, whereas 46 out of 462 runs for the chain model and 11 out of 462 runs for the feedback chain model reached the criterion loss (see Appendices B & flrr further results and discussion).
3.1.2 LANGUAGE MODELING EXPERIMENTS
To investigate if the benefits of non- normal initializers extend to more re- alistic problems, we conducted exper- iments with three standard language modeling tasks: word-level Penn Tree- bank (PTB), character-level PTB, and character-level enwik8 benchmarks.
50p
Addition-750
PsMNIST
Valication loss (croos only)
Validaiion loss (cross onl)
Bascline
Ideniiyy
Validation loss (mac)
Orthogonal
Chain
Fhich
For the language modeling experi- ments in this subsection, we used the code base provided by Salesforce Re-
Epoch
Epoch
search (Merity et al. 22018|b3 b): https:://gsthb..ooo
Capy -apy-500
Addition-750 750
PSMNIST
salesforce/ wwwborcee/wwwa22 awa-lsttn 1m
Number of contige.
We refer the reader MMrity al.
(2018a b) for more detailed de- scription of the benchmarks. For the experiments in this subsection, generally preserved the model setup used in MMeriyy et al. (2018a b), except for the following differences: 1) We replaced the gated RNN architectures (LSTMs and QRNNs) used in Merity eet al.) (0118 with vanilla RNNs; 2) We observed that vanilla RNNs require weaker regularization than gated RNN architectures. Therefore, the word-level PTB task, we set all dropout rates to O.1. In the character-level PTB task, all dropout rates except dropoute were set to 0.1, which was set to enwik8 benchmark, all dropout rates were set to 0; 3) We trained the word-level PTB models for 60 epochs, the character-level PTB models for 500 epochs and the enwi k8 models for 35 epochs.
Maniy
Ortigema
Orain
Pa essin
Sanlyy
Ortogenal
Cran
Fla crain
cv ovtiggmnal
Choao
Pl onsio
Figure 3: Results copy, addition, and PsMNIST bench- marks. a-c Validation losses with the best hyper-parameter settings. Solid lines are the means and shaded regions standard errors over different runs using different random seeeds. For the copy and addition tasks, we also show the loss values for random baseline models (dashed lines). For the PsMNIST task, the mean cross-entropy loss for random classifier is log(10) = 2.3, thus all four models comfortably outperform this random baseline right from the end of the first training epoch. d-f Number of "successful" runs (or hy- perparameter configurations) that converged to a validation loss below 50% of the loss for the random baseline model. Note that the total number of runs was higher for the normal models vs. the non-normal models (594 462 runs per experiment). Despite this, the non-normal models generally outperformed the normal models even by this measure.
the
We compared the same four models described in the previous subsection. As in Merity al. (2018a). we used the Adam optimizer and thus only optimized the a, 3, A hyper-parameters for the experiments in this subsection. For the typer-parameter in the chain model and the pyrae-parrmetter À in the scaled identity and random orthogonal models, we searched over 21 values uniformly spaced between 0.05 and 1.05 (inclusive): whereas for the chain with feedback model, we set the feedforward connection weight, to the optimal value it had in the chain model and searched over 21 3 values uniformly spaced between 0.01 and 0.21 (inclusive). times using different random seeds, yielding total of 63 runs for each model and each benchmark.
addition, we repeated each experiment
The results are shown in Tiguregamm in Table [ Figure shows the validation loss over the course of training in units of bits per character (bpc). Table I reports the test losses at the end of training. The non-normal models outperform the normal models on the word-level and character-level PTB benchmarks. The differences between the models are less clear on the enwik8 benchmark. However, in terms of the test loss, the non-normal feedback chain model outperforms the other models on all three benchmarks (T. T).

---

Published as a conference paper at ICLR 2020
a
PTB word
b
PTB char
enwik8 char
Identity
Validation loss (bpc)
Orthogonal
Validation loss (bpc)
Validation loss (bpc)
Chain
Fb. chain
1.9
500
Epoch
Epoch
Epoch
Figure 4: Results language modeling benchmarks. Solid lines are the means and shaded regions are standard errors over different runs using different random seeeds.
Table Test losses (bpc) on language modeling benchmarks. The numbers represent mean s.e.m. over independent runs. LSTM results from Merity et al. (2018a b).
MODEL
PTB WORD
PTB CHAR.
ENWIK8
IDENTITY
6.550 0.002
1.312 0.000
1.783
0.003
ORTHO.
6.557 + 0.002
1.312 + 0.001
1.843 + 0.046
CHAIN
6.514 0.001
1.308
0.000
1.803
0.017
FB. CHAIN
6.510 + 0.001
1.307 + 0.000
1.774 = 0.002
3-LAYER LSTM
5.878
1.175
1.232
We note that the vanilla RNN models perform significantly worse than the gated RNN architectures considered in Merity et al. (2018a b). We conjecture that this is because gated architectures are generally better at modeling contextual dependencies, hence they have inductive biases better suited to language modeling tasks. The primary benefit of non-normal dynamics, on the other hand, is enabling longer memory capacity. Below, we will discuss whether non-normal dynamics can be used in gated RNN architectures to improve performance as well.
3.2 HIDDEN FEEDFORWARD STRUCTURES IN TRAINED RNNS
We observed that training made vanilla RNNs initialized with orthogonal recurrent connectivity matrices non-normal. We quantified the non-normality of the trained recurrent connectivity matrices using measure introduced by Henriei (1962): di d(W)= = [ E, Pul2, where || ||r denotes the Frobenius norm and X is the 2-th eigenvalue of W. This measure equals 0 for all normal matrices and positive for nor-nlrmal matrices. We found that d(W) became positive for all successfully trained RNNs initialized with orthogonal recurrent connectivity matrices. Table repport the aggregate statistics of d(W) for orthogonally initialized RNNs trained on the toy benchmarks.
Although increased non-normality in trained RNNs is an interesting observation, the Henrici index, by itself, does not tell us what structural features in trained RNNs contribute this increased non-normality. Given the benefits of chain-like feedforward non-normal structures in RNNs for improved memory, we hypothesized that training might have installed hidden chain-like feedforward structures in trained RNNs and that these feedforward structures were responsible for their increased non-normality.
To uncover these hidden feedforward structures, we performed analysis suggested by Rajan et al. (016).. In this analysis, we first injected a unit pulse of input to the network the beginning of the trial and let the network evolve for 100 time steps afterwards according to its recurrent dynamics with no direct input. We then ordered the recurrent units by the time their peak activity (using a small amount of jitter break potential ties between units) and plotted the mean recurrent connection
7

---

Published as a conference paper at ICLR 2020
Table Henrici indices, d( (W), of trained RNNS initialized with orthogonal recurrent connectivity matrices. The numbers represent mean + s.c.m. over all successfully trained networks. We define training success as having a validation loss below 50% of random baseline model. Note that by this measure, none of the orthogonally initialized RNNs successful on the copy task (FigureB))
TASK
IDENTITY
ORTHOGONAL
ADDITION-7A
2.33 + 1.02
2.74 + 0.07
PSMNIST
1.01 + 0.12
2.72 + 0.08
weights, Wij. function of the order difference between two units, i - j. Positive values correspond to connections from earlier peaking units to later peaking units, and vice versa for negative i j values. In trained RNNs, the mean recurrent weight profile as function of i - j had an asymmetric peak, with connections in the "forward" direction being, on average, stronger than those in the opposite direction. Figure shows examples with orthogonally initialized RNNs trained on the addition and the permuted sequential MNIST tasks. Note that for a purely feedforward chain, and would be zero elsewhere. Although the
the weight profile would have a single peak at weight profiles for trained RNNs are not this extreme. the prominent asymmetric bump with a peak at a positive value indicates hidden chain-like feedforward structure in these networks.
a
Identity (Addition-750)
Orthogonal (Addition-750)
0.025
0.025
Mean rec. weight (W,
Trained Untrained
-0.025
-0.025
-99
99
-99
0
99
b
j
Identity (psMNIST)
Orthogonal (psMNIST)
0.025
Mean rec. weight (W/
Trained
Untrained
-0.025
-0.1
-24
24
-24
0
24
i
i-j
Figure 5: Training induces hidden chain-like feedforward structures in vanilla RNNs. The units first ordered by the time of their peak activity. Then, the mean recurrent connection weight is plotted as a function of the order difference between two units, - Results are shown for RNNs trained on the addition (a) and the permuted sequential MNIST (b) tasks. The left column shows the results for RNNs initialized with a scaled identity matrix, the right column shows the results for RNNs initialized with random orthogonal matrices. In each case, training induces hidden chain-like feedforward structures in the networks, as indicated by an asymmetric bump peaked at positive i - value the weight profile. This kind of structure either non-existent (identity) or much less prominent (orthogonal) in the initial untrained networks. For the results shown here, we only considered sufficiently well-trained networks that achieved validation loss below 50% of the loss for a baseline random model at the end of training. The solid lines and shaded regions represent means and standard errors of mean weight profiles over these networks.
8

---

Published as a conference paper at ICLR 2020
Table 3: Test losses (bpc) on language modeling benchmarks using 3-layer LSTMss (adapted from
Merity et al. (201)2200 (2018a b)) with different initialization schemes. Other experimental details were identical those described in 3.1.2 above. The numbers represent mean + s.c.m. over independent runs.
MODEL.
PTB WORD
PTB CHAR.
ENWIK8
ORTHO.
5.937 = 0.002
1.230 + 0.001
1.583 + 0.001
CHAIN
5.935 = 0.001
1.230
+ 0.001
1.586 + 0.000
PLAIN
5.949 = 0.007
1.245
= 0,001
1.584 + 0,002
MIXED
5.944 0.004
1.227 1 0.000
1.577 + 0.001
3.3 DO BENEFITS OF NON-NORMAL DYNAMICS EXTEND GATED RNN ARCHITECTURES"
So far, we have only considered vanilla RNNs. An important question is whether the benefits of non-normal dynamics demonstrated above for vanilla RNNs also extend to gated RNN architectures like LSTMs or GRUs (Hochreiter & Schmidhuber, 1997 Cho et al. 2014). Gated RNN architectures have better inductive biases than vanilla RNNs in many practical tasks of interest such as language modeling (e.g. see Table for a comparison of vanilla RNN architectures with an LSTM architecture of similar size in the language modeling benchmarks), thus it would be practically very useful if their performance could be improved through an inductive bias for non-normal dynamics.
To address this question, we treated the input, forget, output, and update gates of the LSTM archi- tecture as analogous to vanilla RNNs and initialized the recurrent and input matrices inside these gates the same way as the chain or the orthogonal initialization of vanilla RNNs above. We also compared these with more standard initialization scheme where all the weights were drawn from uniform distribution U(-vk, Vk) where & is the reciprocal of the hidden layer size (la- beled plain in Table This is the default initializer for the LSTM weight matrices in PyTorch: org/ docs/ stable /n. html # sstm| We compared these initializers in the language modeling benchmarks. The chain initializer did not perform better than the orthogonal initializer (Table 3 suggesting that non-normal dynamics in gated RNN architectures may not be as helpful as it is in vanilla RNNs. In hindsight, this is not too surprising, because our initial motivation for introducing non-normal dynamics heavily relied on the vanilla RNN architecture and gated RNNs can be dynamically very different from vanilla RNNs.
https://pyttrrr
When we looked the trained LSTM weight matrices more closely, we found that, although still non-normal, the recurrent weight matrices inside the input, forget, and output gates (i.e. the sigmoid gates) did not have the same signatures of hidden chain-like feedforward structures observed in vanilla RNNs. Specifically, the weight profiles in the LSTT recurrent weight matrices inside these three gates did not display the asymmetric bump characteristic of prominent chain-like feedforward structure, but were instead approximately monotonic functions of a qualitatively different kind of dynamics where the individual units are more persistent over time. The recurrent weight matrix inside the update gate (the tanh gate), on the other hand, did display the signature of a hidden chain-like feedforward structure (Figure (1). When we incorporated these two structures in different gates of the LSTMs, by using chain initializer for the update gate and monotonically increasing recurrent weight profile for the other gates (labeled mixed in Table 3 the resulting initializer outperformed the other initializers on character-level PTB and enwi k8 tasks.
(Figure (k-c), suggesting
4 DISCUSSION
Motivated by their optimal memory properties in simplified linear setting (Ganguli et al., 2008). in this paper, we investigated the potential benefits of certain highly non-normal chain-like RNN architectures in capturing long-term dependencies in sequential tasks. Our results demonstrate an advantage for such non-normal architectures as initializers for vanilla RNNs compared to the commonly used orthogonal initializers. We further found evidence for the induction of such chain- like feedforward structures in trained vanilla RNNs even when these RNNs were initialized with orthogonal recurrent connectivity matrices.

---

Published as a conference paper at ICLR 2020
a
Inpe
b
Forget gate
0.025
0.025
Mean rec. weight (W))
Trained Untrained
Mean weight (W
-0.025
-0.025
-1149
1149
-1149
1149
i
i-
d
Output
Update gate
0.025
0.025
Mean rec. weight (Wa)
Mean weight, (W
-0.025
-0.025
-1149
1149
-1149
1149
8
i-
Figure 6: The recurrent weight matrices inside the input, forget, and output LSTM gates do display the characteristic signature of a prominent chain-like feedforward structure. The weight profiles are instead an approximately monotonic function of j-j. j. The recurrent weight matrix inside the update (tanh) gate, however, does display an asymmetric chain- Mike structure similar to that observed vanilla RNNs. The examples shown in this figure are from the input (a), forget (b), output (c). and update gates (d) of the second layer LSTM in 3-layer LSTM architecture trained on the word-level PTB task. The weight matrices shown here were initialized with orthogonal initializers. Other layers and models trained on other tasks display qualitatively similar properties.
The benefits of these chain-like non-normal initializers do not directly carry over to more complex, gated RNN architectures such as LSTMs and GRUs. In some important practical problems such as language modeling, the gains from using these kinds of gated architectures seem to far outweigh the gains obtained from the non-normal initializers in vanilla RNNs (see Table D However, we also uncovered important regularities in trained LSTM weight matrices, namely that the recurrent weight profiles of the input, forget, and output gates (the sigmoid gates) in trained LSTMs display monotonically increasing pattern, whereas the recurrent matrix inside the update gate (the tanh gate) displays a chain-like feedforward structure similar to that observed in vanilla RNNs (Figure 6). We showed that these regularities be exploited to improve the training and/or generalization performance of gated RNN architectures by introducing them as useful inductive biases.
concurrent work ours also emphasized the importance of non-normal dynamics in RNNs (Kerg ct al. 2019). The main difference between Kerg et al. (2019) and our work is that we explicitly introduce sequential motifs RNNs initialization as a useful inductive bias for improved long-term memory (motivated by the optimal memory properties of these motifs in simpler cases), whereas their approach does not constrain the shape of the non non-normal part of the recurrent connectivity matrix, hence does not utilize sequential non-normal dynamics as an inductive bias. In some of their tasks, Kerg et al. (2019) also uncovered a feedforward, chain-like motif in trained vanilla RNNs similar to the one reported in this paper (Figure 55)
There is a close connection between the identity initialization RNNs (Le et al. 2015) and the widely used identity skip connections (or residual connections) in deep feedforward networks (He) et al. 2016). Given the superior performance of chain-like non-normal initializers over the identity initialization demonstrated in the context of vanilla RNNs in this paper, it could be interesting to look for similar chain-like non-normal architectural motifs that could be used in deep feedforward networks in place of the identity skip connections.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01277. 2018.
Y. Bengio, P. Simard, and Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural. Netw., 5:157-66, 1994.
S. Chang, Y. Zhang, W. Han, M. Yu, X. Guo, W. Tan, X. Cui, M. Witbrock, M.A. Hasegawa-Johnson, and T.S. Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems 30, 2017.
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and mean field theory of rnns: Gating enables signal propagation in recurrent neural networks, In International Conference on Machine Learning, pp. 872-881, 2018.
K. Cho, B. van Merriënboer, Ç Gülçehre, D. Bahdanau, Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using mn -ecoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representations (ICLR), 2016.
S. Ganguli, D. Huh, and H. Sompolinsky. Memory traces in dynamical systems. PNAS, 105(48): 18970-18975, 2008.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. In 33rd International Conference on Machine Learning, pp. 2978-2986 2016.
Peter Henrici. Bounds for iterates, inverses, spectral variation and fields of values of non-normal matrices. Numerische Mathematik, 4:24-40, 1962.
S. Hochreiter. Untersuchungen zu dynamischen neuronalen Neizen. PhD thesis, Institut f. Informatik, Technische Univ. Munich, 1991.
Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 1735:173- 1997.
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie. Non-normal recurrent neural network (nnRNN); learning long time dependencies while improving expressivity with transient dynamics. arXiv preprint /rX.v 2019.
Q.V. Le, N. Jaitly, and G.E. Hinton. A simple way to initialize recurrent networks of rectified linear units. 2015. URL peatps://arxite
00941
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis neural language modeling at multiple scales. arXiv:1803.088
2018a.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. In International Conference on Learning Representations (ICLR), 2018b.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in Neural Information Processing Systems (NIPS), pp. 4795- 2017.
11

---

Published as a conference paper at ICLR 2020
Kanaka Rajan, Christopher D Harvey, and David W Tank. Recurrent network models of sequence generation and memory. Neuron, 90(1):128 142, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958 2014.
Wisdom, T. Powers, J.R. Hershey, Le Roux, and L.. Atlas. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems 29, 2016.
A DETAILS AND EXTENSIONS OF THE LINEAR DECODING EXPERIMENTS
This appendix contains the details of the linear decoding experiments in section 2.2 and reports the results of additional linear decoding experiments. The experiments in section 2.2 compare the signal propagation properties of vanilla RNNs with either random orthogonal chain connectivity matrices. In both cases, the overall scale of the recurrent connectivity matrices is set to 1.01. The input weight vector is v [1, 0, O, ........ 0) for the chain model and V; (O, 1/v)) for the random orthogonal model (thus the overall scales of both the feedforward and the recurrent inputs are identical in the two models). The RNNs themselves are not trained in these experiments. At each time point, an i..d random scalar signal ~ N(00, 1) is fed into the network as input (Equation 5 We simulate 250 trials for each model and ask how well we can linearly decode the signal at the first time step, $1, from the recurrent activities at time step 100, h100. We do this by linearly regressing $1 on h100 (using the 250 simulated samples) and report the R² value for the linear regression in Figure 2
In simulations with noise (Figure 2b), an additional i.d.d. random noise term, Z NN(0,, σ),, is added to each recurrent neuron at each time step The standard deviation of the noise, d,, set 0.1 in the experiments shown in Figure 2b. To show that the results are not sensitive to the noise scale, we ran additional experiments with lower (a 0.01) and higher (o = 1) levels of noise (Figure (7). In both cases, the chain network still outperforms the orthogonal network. Note that these "linear + noise" " experiments satisfy the conditions of the analytical theory in Ganguli et al. 2008), so these results are expected from the theory.
As mentioned in the main text, the "non-linear + no noise" experiments reported in Figure 2c used the elu non-linearity. To show that the results are not sensitive to the choice of the non-linearity, we also ran additional experiments with tanh and relu non-linearities (Figure [8. As with the elu non- non-linea,i,y, the chain network outperforms the orthogonal network with the tanh and relu non-linearities as well, suggesting that the results are not sensitive to the choice of the non-linearity.
B
THE EFFECT OF THE FEEDBACK STRENGTH PARAMETER (3) IN THE CHAIN WITH FEEDBACK MODEL.
In this appendix, consider the effect of the feedback strength parameter, 8, for the chain with feedback model the context of the experiments reported in section 3.1. We focus on the PSMNIST task specifically. because this is the only task where the feedback chain model converges to low loss solution for a sufficiently large number of hhper- parameter configurations. For the addition and copy tasks, there are not enough successful typer-parameter configurations to draw reliable inferences about the effect of 33 (see Figure L-f). Figure 9 shows the validation loss at the end of training as function of :3 in the PSMNIST task. In this figure, we considered all networks that achieved validation loss lower than the random baseline model (i.c. < log(10) x 2.3) at the end of training (an overwhelming majority of the networks satisfied this criterion). Figure 9 shows that the final validation loss is a monotonically increasing function of 3 in this task, suggesting that large feedback strengths are harmful for the model performance.
C COMPARISON WITH PREVIOUS MODELS
In this appendix, compare our results with those obtained by previous models, focusing specifically on the experiments section 3.I.I (because the tasks in this section are commonly used as RNN benchmarks).
12

---

Published as a conference paper at ICLR 2020
a
b
linear + low noise
linear + high noise
3
Orthogonal
0
R²
0.97
R²
0.32
-3
-3
-3
-3
0
3
$1
$1
3
Chain
0
R
1.0
R²
0.38
-3
-3
-3
0
3
-3
0
3
81
81
Figure 7: Additional linear decoding experiments: a linear networks with low noise (a = 0.01) and linear networks with high noise (a = 1). In both cases, the chain network outperforms the orthogonal network suggesting that the results are not sensitive to the noise scale.
uRNN: We first note that our copy and addition tasks use the largest sequence lengths considered in Arjovsky al. (2016) for the same tasks (T 500 for the copy task and T 750 for the addition task). Hence our results are directly comparable to those reported in Arjovsky et al. (2016) (the random baselines shown by dashed lines Figure 3 are identical those in Arjovsky et al.
(2016) for the same conditions). The unitary evolution RNN (uRNN) model proposed in Arjovsky ct al. (2016) comfortably learns the copy-500 task (with 128 recurrent units), quickly reaching near-zero loss (see their Figure bottom right); however, it struggles with the addition task, barely reaching the half- baseline criterion even with 512 recurrent units (see their Figure 2, bottom right).
This difference in the behavior of the uRNN model in the copy and addition tasks is predicted by Henaff et al. (2066), where it is shown that random orthogonal and near-identity recurrent connectivity matrices have much better inductive biases in the copy and addition tasks, respectively. Because of its parametrization, uRNN behaves more similarly to random orthogonal RNN than near-identity RNN.
In contrast, our non-normal RNNs, especially the chain model, comfortably clear the half-baseline criterion both in copy-500 and addition-750 tasks (with 100 recurrent units), quickly achieving very small loss values in both tasks with the optimal that this is despite fact that our models use fewer recurrent units than uRNN model in Arjovsky et al. (2016) (100 vs. 128 or 512 recurrent units).
configurations (Figure 3bb)). Note
nnRNN: Kerg et al. (019)) report results the copy (7 = 200) and PsMNIST tasks only. They have not reported training success for longer variants of the copy task (specifically for T = 500). Kerg ct al. (2019) also have not reported successful training in the addition task, whereas our non-normal RNNs showed training success both in copy-500 and addition- 750 tasks (Figure bbb)
We conclude that our non-normal initializers for vanilla RNNs perform comparably to, or better than, the uRNN and nnRNN models in standard long-term memory benchmarks. One of the biggest strengths of our proposal compared to these previous models is its much greater simplicity. Both uRNN and NoRNN require complete re-parametrization of the vanilla RNN model NnnRNN even
13

---

Published as a conference paper at ICLR 2020
a
tanh no noise
b
relu no noise
3
3
Orthogonal
R²
0.43
R²
0.42
-3
-3
-3
0
3
-3
0
3
81
81
3
3
Chain
5T
55
R
0.80
R
0.83
-3
-3
-3
0
3
-3
0
3
81
81
Figure 8: Additional linear decoding experiments: tanh networks with noise and b relu networks with no noise. In both cases, the chain network outperforms the orthogonal network suggesting that the results are sensitive the choice of the non- -on-aiinari.y.
PSMNIST
Fb. chain
Validation loss
0.9
0.8
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Figure 9: Validation loss at the end of training function of the feedback parameter & in the PsMNIST task. All networks with a petter-than-randor loss at the end training are included in this figure. The solid line shows the mean and the shaded region represents the standard errors.
requires a novel optimization method). Our method, on the other hand, proposes much simpler, masy-to-implement, plug-and-play type sequential initializers that keep the standard parametrization of RNNs intact.
critical RNN: Chen et al. 2018) note that the conditions for dynamical isometry in vanilla RNNs are identical to those in fally-oonected feed- -orwwrdd networks studied in Pennington al. (017)). Pennington et al. (207)), in turn, note that dynamical isometry is not achievable exactly in networks
14

---

Published as a conference paper at ICLR 2020
with relu activation, but it is achievable in networks with tanh activation, where it essentially boils down to initializing the weights to small values. Pennington et al. (2017) give a specific example of a dynamically isometric tanh network (with
400, on 1.05, and σ 2.01 x 10-5) We set up a similar tanh RNN model, but were not able to train it successfully in the copy or addition tasks. Again, as with the nnRNN results, this shows the challenging nature of these tasks and suggests that dynamical isometry may not be enough for successful training in these tasks. A possible reason for this that although critical initialization takes the non-linearity into account, still does take the noise into account (i.e. it is not guaranteed to maximize the SNR).
LSTM, tanh RNN:
Consistent with the results in Arjovsky et al. (2016), we were not able to successfully train LSTMs or vanilla RNNs with tanh non non-linearity in the challenging copy-500 and addition-750 tasks. Therefore, these models were not included as baselines in section 3.1.1
psMNIST
100
Validation accuracy (%)
75
Fb. chain Chain Orthogonal Identity
50
25
0
64
Epoch
Figure 10: Validation accuracy in the PSMNIST task. The corresponding validation losses shown in Figure 3 the main text. Note that we used RNNs with n 25 recurrent units these simulations, these numbers (e.g. Arjovssy et al. (2016): Kerg et al. (2019))
directly comparable to those reported in some previous works
15


Reviews:
[{"review_id": "S1lp0zP-9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the sub-optimality of using orthogonal recurrent  matrix in RNNs with nonlinearity and noise, the authors look into non-normal alternatives, in particular matrices with chain-like structure in preserving memory in RNNs. The authors compare normal and non-normal RNNs on several sequential benchmark datasets, and show that non-normal RNNs perform better than their normal counterpart. \n\nThe paper is easy to follow. The novelty of the work is limited though. The chain structure was introduced in Ganguli et al. (2008). The work studies the benefit of initializing recurrent weights in nonlinear RNNs with these chain-like structures.\n\nChen et. al. (2018) already pointed out the limitation of orthogonal initialization alone for nonlinear RNNs, and proposed closed-form initialization for RNNs with different activation functions. It would be worthwhile to include a comparison to that method.\n\nIn experiment section 2.3.1, it would be helpful to include comparison of performance of chain with feedback using different beta values to confirm the intuition that stronger feedback strength would negatively impact the memory. \n\nResults in section 2.3.2 Table 1 are not exactly align with the story. Do the authors have any intuition on why the chain with feedback perform better than the chain variant. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyerkTJAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n This paper proposes to explore nonnormal matrix initialization in RNNs. Authors demonstrate on various tasks (Copy/Addition, Permuted-SMNIST, PTB, enwik8) that chain-like nonnormal matrix initializations can outperform orthogonal or identity initialization in vanilla RNNs. However, nonnormal RNNs underperform their gated counterpart such as LSTM. Authors also show results where they use their initialization scheme in update gate of a LSTM.\n\nComments:\nThe paper is well written and pleasant to read. The paper structure could be a bit improved. For instance, section 2 is named “Results” while 2.1 which take significant part of the section is about some prior results in (Ganguli et al. 2018). It would be better to have it under an explicit prior work section. \n\nThe description of the experiments reported in Figure 2. is a bit vague: what is the training/evaluation data?, do you train all the model parameters or only the linear layer?, what is the type of noise used? It is unclear to me how robust is the observation made in Figure-2. Do you see similar behavior with different noise-scale and other non-linearity such as tanh?\n\nThe experimental section provides convincing data showing that non-normal initialization schemes outperform orthogonal and identity initialization in vanilla RNN. However, it would be nice to add some comparisons with prior works. It is unclear how the current method compare with nn-RNN of (Kerg et al. 2019) and the unitary-RNNs. \n\n Why the score reported for the 3-LSTM in Table 3.  is underperforming 3-layer LSTM baseline used in (Merity et al., 2018), reported in Table 1.?  In addition, did you try saturating non-linearities for the RNN experiments?\n\nOverall, I think the method is promising, but comparison with prior work is missing. I would encourage the authors to compare their approach with unitary-RNN, and nn-RNN to better demonstrate the significance of their works.\n\nAdditional remarks:\n-\tSNR could be defined more precisely in the introduction. In particular, the introduction states that the stochasticity of SGD is a source of noise which is true. But the model presented in section 2 seems to focus mostly on input noise? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxc7hICKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The focus of this paper is on exploring non-normal initializations for training vanilla RNN for sequential tasks. They show on 3 different tasks, and a real-world LM task that  non-normal initializations of vanilla RNNs outperform their orthogonal counter-parts when particular forms of initialization are considered. \n\n\nAlthough the results for sequence task do not outperform the gated counterparts, the authors present an interesting exploration of initializing non-normal RNNs that outperform the orthogonal counterparts. It is good to see this line of work being explored as an alternative to exploring more complex architectures with many more parameters than necessary for the task. \n\nStrengths:\n    1. The paper explores non-normal RNNs and demonstrates on  3 synthetic tasks - copy, addition and pMNIST - how with careful initialization the proposed approach outperforms their orthogonal initialization counterpart. This line of experimentation is interesting as it potentially opens the door for more expressive modeling for sequential tasks by expanding the solution space of the weight matrices being learnt i.e orthogonal matrices are a special case.\n    2. The authors do a great job in motivating the paper, and the explanation is clear and easily understandable. The toy simulations in Section2.2 really helps drive the reasoning behind why chain initialization improves over orthogonal initialization.\n    3. Based on the insight from trained RNNs where the trained  weights exhibit a chain like structure, the authors attempt to modify the LSTM gate initializations well. However, they do not see any specific gain by doing so, and moreover they show analysis that demonstrate that the LSTM gates do not learn these chain like structures. However, they do have insight into the regularities of these learnt weights which could potentially open the door for more interesting initialization methods for training such gated architectures.\n\n\nIssues to be addressed in the paper:\n1. The plots are quite small and hard to follow. Can the authors enlarge these so they span the entire page? Also, for pMNIST it would be good to provide accuracy scores as well as a function of the training epochs.Finally, it would be good to include a comparison against LSTMs (and even Transformer networks) so it is easier for the reader to see where these approaches stack against architecture changes.\n2. The authors are missing a reference to this  work - http://proceedings.mlr.press/v48/henaff16.pdf  - which provides empirical analysis for the 3 synthetic tasks to test the ability of vanilla RNNs for solving long span sequential tasks. \n3. What about stability of these non-normal RNNs? For example, if we perturb the inputs to the training for the LM task how much variance do we see in the performance of these models? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to explore nonnormal matrix initialization in RNNs.  Two reviewers recommended acceptance and one recommended rejection.  The reviewers recommending acceptance highlighted the utility of the approach, its potential to inspire future work, and the clarity and quality of writing and accompanying experiments.  One reviewer recommending weak acceptance expressed appreciation of the quality of the rebuttal and that their concerns were largely addressed.  The reviewer recommending rejection was primarily concerned with the novelty of the method.  Their review suggested the inclusion of an additional citation, which was included in a revised version for the rebuttal but not with a direct comparison of results.  On the balance, the paper has a relatively high degree of support from the reviewers, and presents an interesting and potentially useful initialization in a clear and well-motivated way.
Decision: Accept (Poster)