The Dynamics of Signal Propagation in Gated Recurrent Neural Networks
Authors: Dar Gilboa
Bo Chang
Minmin Chen
Greg Yang
Samuel S. Schoenholz
Ed H. Chi
Jeffrey Pennington
Venue:  (2020)
Keywords: recurrent neural networks
theory of deep learning

Abstract: Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation as well as the spectral properties of the state-to-state Jacobians. By optimizing these quantities in terms of the initialization hyperparameters, we derive a novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe a beneficial effect on generalization performance using this new initialization.

Under review as a conference paper at ICLR 2020
THE DYNAMICS OF SIGNAL PROPAGATION IN GATED RECURRENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation well as the spectral properties of the state-to-ae stteeto--tttte Jacobians. By optimizing these quantities in terms of the initial- ization hyperparameters,
derive novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe beneficial effect on generalization performance using this new initialization.
INTRODUCTION
common paradigm for research and development in deep learning involves the introduction of novel network architectures followed by experimental validation selection of tasks. While this method- ology has undoubtedly generated significant advances the field, is hampered by the fact that the full capabilities of candidate model may be obscured by difficulties in the training procedure. It is often possible to overcome such difficulties by carefully selecting the optimizer, batch size, learning rate schedule, initialization scheme, or other hyperparameters. However, the standard strategies for searching for good values of these hyperparameters are not guaranteed to succeed, especially if the trainable configurations constrained to a oow-dineniional subspace of hyperparameter space, which can render random search, grid search, and even Bayesian hyperparameter selection methods unsuccessful.
In this work, we argue that for long sequence tasks, the trainable configurations of initialization hyperparameters for LSTMs and GRUs lie just such a subspace, which we characterize theoretically, In particular, we identify precise conditions on the hyperparameters governing the initial weight and bias distributions that are necessary to ensure trainability. These conditions derive from the observation that in order for network to be trainable, (a) signals from the relevant parts of the input sequence must be able to propagate all the way to the loss function and (b) the gradients must be stable (i.e., they must not explode or vanish exponentially). As shown in Figure 1, training of recurrent networks with standard initialization on certain tasks begins to fail as the sequence length increases and signal propagation becomes harder to achieve. However, as we shall show, suitably-chosen initialization scheme can dramatically improve trainability on such tasks.
We study the effect of the initialization hyperparameters signal propagation for very broad class of recurrent architectures, which includes as special cases many ofate-of-the-art RNN cells, including the GRU (Cho et al. (2014)), the LSTM (Hochreiter & Schmidhuber (1997)), and the peephole LSTM (Gers et al. (2002)). The analysis is based on the mean field theory of signal propagation developed in line of prior work (Schoenholz et al. (2016); Xiao al. (2018); Chen et al. (2018); Yang et al. (2019)), as well as the concept of dynamical isometry (Saxe et al. (2013); Pennington

---

Under review as a conference paper at ICLR 2020
1.0
0.8
Figure 1: Critical initialization improves trainability of recurrent networks. Test accuracy for peephole LSTM trained classify sequences MNIST digits after 8000 iterations. As the sequence length increases, the network no longer trainable with standard initialization, but still trainable using critical initialization.
Accuracy
0.6
0.2
Critical Initialization Standard inttialization
200
400
600
800
Sequence length
et al. (2017; 2018)) that is necessary for stable gradient backpropagation and which was shown to be crucial for training simpler RNN architectures (Chen et al. (2018)). We perform a number of experiments to corroborate the results of the calculations and use them to motivate initialization schemes that outperform standard initialization approaches on a number of long sequence tasks.
2
BACKGROUND AND RELATED WORK
2.1 MEAN FIELD ANALYSIS OF NEURAL NETWORKS
Signal propagation at initialization can be controlled by varying the hyperparameters of fully- connected (Schoenholz et al. (2016); Yang & Schoenholz (2017)) and convolutional (Xiao et al. (2018)) feed-forward networks, as well for simple gated recurrent architectures (Chen et al. (2018)). In all these cases, such control was used to obtain initialization schemes that outperformed standard initializations on benchmark tasks. In the forwwfd case, this enabled the training of very deep architectures without the use of batch normalization skip connections.
By forward signal propagation, we specifically refer to the propagation of correlations between inputs at the start of an input sequence through the hidden states of the recurrent network at later times, as will be made precise in Section 4.2. This appears to be a necessary but not sufficient condition for trainability the state-to-ste ttatt Jacobian matrix, as discussed Bengio et al. (1994). In our context, the goal of the backward propagation analysis is to improve the conditioning of the Jacobian by controlling the first two moments of its squared singular values. Forward signal propagation and the spectral properties of the Jacobian initialization can be studied using mean field theory and random matrix theory (Poole et al. (2016); Schoenholz et al. (2016); Yang & Schoenholz (2017; 2018); Xiao et al. (2018); Pennington et al. (2017; 2018); Chen et al. (2018); Yang et al. (2019)).
certain tasks. Another is the stability of the gradients, which depend on
As neural network training is nonconvex problem, using a modified initialization scheme could lead convergence different points in parameter space way that adversely affects the generalization error. We provide some empirical evidence that this does not occur, and in fact, the use of initialization schemes satisfying these conditions has a beneficial effect on the generalization error.
2.2 THE EXPLODING/VANISIING GRADIENT PROBLEM AND SIGNAL PROPAGATION IN RECURRENT NETWORKS
The exploding/vanishie gradient problem a well-known phenomenon that hampers training on long time sequence tasks (Bengio et al. (1994); Pascanu al. (2013)). Apart from the gating mechanism, there have been numerous proposals to alleviate the vanishing gradient problem by constraining the weight matrices be exactly or approximately orthogonal (Pascanu al. (2013); Wisdom al. (2016); Vorontsov et al. (2017); Jose et al. (2017)), or more recently by modifying some terms in the gradient (Arpit et al. (2018)), while exploding gradients can be handled by clipping (Pascanu et al. (2013)). Another recently proposed approach to ensuring signal propagation in long sequence tasks introduces auxiliary loss functions (Trinh et (2018)). This modification of the loss can be seen as form of regularization. Chang al. (2019) study the connections between recurrent networks and certain ordinary differential equations and propose the AntisymmetricRNN that can capture long term dependencies in the inputs. While many of these approaches have been quite successful, they
2

---

Under review as a conference paper at ICLR 2020
typically require modifying the training algorithm, the loss function, or the architecture, and such exist as complementary methods to the one we investigate here. We postpone the investigation of a combination of techniques to future work.
2.3 DYNAMICS BEYOND INITIALIZATION
While the analysis below applies to the network at initialization, it was recently shown that as the network width becomes large, parameters move very little from their initial values (Jacot al. (2018); Lee et al. (2019)). A non-trivial and surprising consequence is that covariance matrices like those studied here remain constant during training. While Jacot et al. (2018); Lee et (2019) do not specifically study RNNs, the analysis of Yang (2019) suggests the conclusions are likely to carry over. Moreover, even for finite-width networks, simply satisfying the conditions derived in this work at initialization can lead to a dramatic improvement in trainability as we show below.
3 NOTATION
We denote matrices by bold upper case Latin characters and vectors by bold lower case Latin characters. Dr denotes standard Gaussian measure. The normalized trace of random N N matrix A, Etr((A), denoted by T (A). o the Hadamard product, a-)) is a sigmoid function and both a (.),, tanh(-) act element-wise. We denote by D, diagonal matrix with the diagonal.
4 MEAN FIELD ANALYSIS OF SIGNAL PROPAGATION AND DYNAMICAL. ISOMETRY
4.1
MODEL DESCRIPTION AND IMPORTANT ASSUMPTIONS
Vanilla RNN
GRU Cho et al. (2014)
LSTM Hochreiter Schmidhuber (1997)
K
(/)
(
(r,o)
o(u))
(u tanh(c"+
(u';) s +(1-
f
(u'))
+(1 - ou,))) o tanh(u
1 Σ f a(u)) o (u?) tanh(u?)) ==l==
Table 1: A number of recurrent architectures written in the form 1. K is the set of pre-activation subscripts. f is the state update function in eqn. (Ia) and gx is the function in eqn. (Ic). The LSTM cell state is unrolled in order to emphasize that it can be written as a function of variables that are Gaussian at the large N limit. See Table 3 for additional architectures.
We begin with a general description of recurrent architectures that can be specialized to the GRU, LSTM and peephole LSTM among others. We denote the state of recurrent network at time t by E N with 8 ~ Do. and a sequence inputs to the network by (x)...,, 2" 2 J, x' R.-.
The state evolution of the network is given by
= f(s 1 {u]] ) (uf),2²)
(1a)
where is element-wise, affine function of s and {uf] is set of pre-activations up R. K defined for a set of subscripts K. They are given by
E
u
Wes Ugz + b
(1b)
where WA. U₁ NNN by R... We define additional pre-activations
Wx, (ut)ss + Ur,z' + bEg
(1c)
where RN + N is an element- wise function and u' is defined as in eqn. (1b) ! In cases where there is no need to distinguish between variables of the form (1b) and (Ic) we will refer to both as
Variables of the form (Ic) will be present only in the GRU (see Table 1).
3

---

Under review as a conference paper at ICLR 2020
- This state update equation describes all the architectures studied in this paper, well as many others, as detailed in Tables 1,3. The pre-activations u which are typically used to construct gates in recurrent network with the application of appropriate nonlinearities, will be of interest because they will tend in distribution to Gaussians as the network width is taken to infinity.
We assume Wk,55 (O /N),, Uk,Y NO v (fg/N),bb. NN pB) i.i.d. and denote O Ulo U Pe J. As in Chen et al. (2018), we make the untied weights assumption that Wx is
k
independent of s'. Although the assumption of untied weights may seem counter-intuitive in the context of recurrent networks, recall that we are only interested in characterizing various statistics of the network, not in computing the entire input-output map, for which using tied weights would certainly be crucial. For merely characterizing certain statistics, the untied weights assumption actually has long history of yielding correct predictions, and therefore is not all unnatural to adopt: 1) when computing the covariance of gradients in an MLP, the weights during backpropagation can be treated as independently sampled from the weights used during the forward propagation (Schoenholz et al., 2016; Yang & Schoenholz, 2017); 2) when computing the Jacobian singular value distribution of an MLP, the pre-activations h = Wx can be treated as the result of multiplying an iid copy w' of W (i.e. h W'x) (Pennington et al., 2017: 2018; Hayase, 2019); 3) a wide, randomly initialized MLP corresponds to the same Gaussian process whether or not its weights across layers are tied (Poole et al., 2016), and similarly, a simple RNN evolves the same whether or not its weights are tied across time, as long as its width is large (Chen et al., 2018). Indeed, we provide empirical evidence that calculations performed under this assumption still have considrrable predictive power, even in cases where it is violated.
4.2 FORWARD SIGNAL PROPAGATION
We now study how correlations between inputs propagate into the hidden states of recurrent network. This section follows closely the development in Chen et al. (2018). 8). We now consider two sequences
1
Σ>
of normalized inputs { {z}} with zero mean and covariance R = R
fed into two
1
copies of network with identical weights, and resulting in sequences of states {ss), (s)) We consider the time evolution of the moments and correlations
3/ = Biss:al
(2a)
Q¹ = Q2 =Bist, st
(2b)
Σイ C2 + (u²)² -Elsia
(2c)
where we define 2122 = Q2 (2t)².
Returning to the pre-activations defined in eqn. (1). We make the mean field approximation that the pre-activations are jointly Gaussian at the infinite width N + oo limit:
(
uhia whss
) ~N((
(N(((
[
1kk ukk
) (Qt - -At) ( c
C ( ))
(3)
where the second moment Q% is given by
Q = E Eu Whinl = Bis sta E = + p + - o - 2 p 1
(4a)
and defining 2 0% 1
E Eluk
kin ukennl -
of 3I
C2
+ 22) + +ERE, + P
ct =
(4b)
Q 1
Q r
The variables uigi w1 wb given by eqn. (1c) are also asymptotically Gaussian, with their covariance detailed in Appendix A. We will subsequently drop the vector subscript i since all clements are
Foor feed-forward networks, it has been shown using the Central Limit Theorem for exchangeable random variables that the pre-activations at all layers indeed converge in distribution to a multivariate Gaussian (Matthews al. (2018)).
4

---

Under review as a conference paper at ICLR 2020
identically distributed and f, gi act element-wise and the input sequence index in expressions that involve only one sequence.
For any / I t, the ung are independent of s' at the large N limit. Combining this with the fact that their distribution is determined completely by 1 - and that f is affine, one can rewrite eqn. (2a-c) using eqn. (la-c) as the following deterministic dynamical system
(4L S. Ct) = MM 0 C ......, prg Q2.C3
(5)
where the dependence on O and the data distribution has been suppressed. In the peephole LSTM and GRU, the form will be greatly simplified to M 2 C.-1). In Appendix E we compare the predicted dynamics simulations, showing good agreement.
One can now study the fixed points of eqn. (5) and the rates of convergence by linearizing the dynamics. The fixed point of eqn. (5) pathological, in the sense that any information that distinguishes two input sequences is lost. Therefore, delaying the convergence to the fixed point should allow for signals to propagate across longer time horizons. Quantitatively, the rate of convergence of eqn. (5) to fixed point gives an effective time scale for signal propagation from the inputs to the hidden state at later times.
While the dynamical system is wwo-dimensional and analysis of convergence rates should be per- formed by linearizing the full system and studying the smallest eigenvalue of the resulting matrix, in practice as in Chen et al. (2018) 8) this eigenvalue appears to always corresponds to the Cg direction". Hence, we assume convergence of Q ue, we need only study
C² = MMllis Qu, C!-+)
(6)
where Mc also depends on expectations of functions of (u.])...(u'. ( that do not depend on C² While this dependence is in principle on an infinite number of Gaussian variables as the dynamics approach the fixed point, .Mc can still be reasonably approximated in the case of the LSTM as detailed in Appendix D. We show that in the case of the peephole LSTM this map convex in Appendix C. This can be shown for the GRU by similar argument. It follows directly that it has single stable fixed point in these cases.
The rate of approach to the fixed point, XC; is given by linearizing around it. Setting Cj C2 + +et, we have
+ e²) C Mc(wi, + xc:e* + +((())))
= + 1 M.l.y....
(7)
The time scale of convergence the fixed point is thus given by
Ec: log XC:
(8)
which diverges as XC; approaches 1 from below. Due to the detrimental effect of convergence to the fixed point described above, it stands to reason that a choice of such that Xc: = 1 ò for some small 0 > 0 would enable signals to propagate from the initial inputs to the final hidden state when training on long sequences.
4.3 BACKWARDS SIGNAL PROPAGATION THE STATE-TO-STATE JACOBIAN
We now turn to controlling the gradients of the network. A useful object to consider in this case is the asymptotic state-to-state transition Jacobian
0s
J lim
t1000
Ös
J and powers of it will appear in the gradient as the covariance dynamics described Section 4.2 approach their fixed point (specifically, the gradient of a network trained on sequence of length T will depend on a matrix polynomial of order T in J), hence we desire to control the squared singular
"This observation is explained in the case of fully-connected feed-forward networks in Schoenholz et al. (2016).
5

---

Under review as a conference paper at ICLR 2020
value distribution of J. This distribution is described by moments of the form mart,, =((J"")") with mean and variance of the distribution given by
MJJ E
Σ
a
kEKU(U}
Ggr mJJ7,2 JJ = E
20.01 - 0g
E Σ
(10)
(.EEK(())
REKU(()
where the scalars {ax (a are architecture and hyperparameter dependent and are given in Appendix A.2.
Forward and backward signal propagation are in fact intimately related, as the following lemma shows:
Lemma 1. For a recurrent neural networks defined by (1), the mean squared singular value of the state-to-state Jacobian defined in (16) and Xc, that determines the time scale of forward signal propagation (given by (7)) are related by
MJJ7.1 XC2=12X,=1
(11)
Proof. See Appendix C.
one can find a setting of the hyperparameters such that MJJT, is close to and oj is small, then the spectrum of powers of J should remain well-conditioned ensuring that gradients will not explode or vanish.
4.4 DYNAMICAL ISOMETRY
Combining the results of Sections 4.2 and 4.3, we conclude that an effective choice of initialization hyperparamters should satisfy
Xc:
(12a)
MJJT
(12b)
G 0.
(12c)
We refer to these as dynamical isometry conditions. eqn. (12)a ensures stable signal propagation from the inputs the loss, and eqn. (12)b-c are motivated by the additional requirement of preventing the gradients from exploding/vanishing. Both of these conditions appear to be necessary in order for network be trainable. Combining equs. (9, 10, 11) find that if Σ, isometry conditions are satisfied dg 1, akye O, which can be achieved by setting Vk: : o 0 and taking μz co. This motivates the general form of the initializations used in the experiments.
the dynamical
We demand that these equations are only satisfied approximately since for a given architecture, there may not be value of & that satisfies them all. Additionally, even if such value exists, the optimal value of XC; for a given task may not be 1. There is some empirical evidence that the characteristic time scale deñindd by Xc; is much larger than that required for certain task, performance is degraded.
The typical values Σ, will depend on the data set, yet satisfying the dynamical isometry conditions is simplified if Σ, = 1 due to Lemma 1. It is thus natural choice, yet we acknowledge that more comprehensive treatment should consider the case of general E.,. We also find empirically that the results obtained under the Σ, assumption prove to be predictive and enable training on number of tasks without requiring a detailed analysis of 2.
4.5 THE STATIONARY DISTRIBUTION OF THE LSTM CELL STATE
In architectures that considered in previous works, well the peephole L.TTM and GRU, all the quantities required to calculate the dynamical isometry conditions through eqns. (9, 10) can be written terms of integrals over finite number of Gaussian variables the large width and time

---

Under review as a conference paper at ICLR 2020
10'
12?
18
Figure 2: Training accuracy the padded MNIST classification task described in 5.1 at different sequence lengths T and hyperparameter values On + a@, for networks with untied weights, with different values of Op O1 chosen for each architecture. The dark and light green curves are respec- tively 35, 66 where & is the theoretical signal propagation time scale in eqn. (8). As can be seen, this time scale predicts the transition between the regions of high and low accuracy across the different architectures and directions in hyperparameter space.
limit and are thus easy to evaluate. This is not the case in the standard LSTM due to the presence of the cell state. By unrolling the cell state update equation
c' = c'=o((')) c + (uf) tanh(u²))
(13)
in time, we find that c' depends on the entire sequence of uy from the first time step"
From eqn. 13 we find that the asymptotic cell state distribution is that of perpetuity, which is random variable X that obeys X - XY + Z where Y, Z are random variables and d denotes equality in distribution. The stationary distribution of a perpetuity generally does not have closed form, and neither does its likelihood (Goldie (1991)). In practice, one overcome this difficulty by sampling from the stationary cell state distribution, as described in Appendix D
EXPERIMENTS
5.1 PADDED MNIST CLASSIFICATION
The calculations presented above predict characteristic time scale & (defined in (8)) for forward signal propagation a recurrent network. It follows that on a task where success depends on propagation of information from the first time step the final T-th time step, the network will not be trainable for T' > &. In order to test this prediction, we consider a classification task where the inputs are sequences consisting of single MNIST digit followed by T' steps of i.i.d Gaussian noise and the targets are the digit labels. By scanning across certain directions in hyperparameter space, the predicted value of & changes. We plot training accuracy of network trained with untied weights after 1000 iterations for the GRU and 2000 for the LSTM, as a function of T and the hyperparameter values, and overlay this with multiples of Ç. As seen in Figure 2, we observe good agreement between the predicted time scale of signal propagation and the success of training. As expected, there are some deviations when training without enforcing untied weights, and we present the corresponding plots in the supplementary materials.
MNIST
CIFAR-10
5.2 UNROLLED MNIST AND CIFAR-10 10
standard LSTM
98.6
58.8
The calculation results motivate critical initial- izations that we test on standard long sequence benchmarks. The details of the initializations are presented in Appendix E. We unroll an MNIST digit into sequence of length 784 and train a critically initialized peephole LSTT with 600 hidden units. We also train critically initialized
critical LSTM
98.9
61.8
Table 2: Test accuracy on unrolled MNIST and CIFAR-10.
In contrast, the hidden state distribution of the GRU at time for instance can be expressed as an integral over {uf, - alone (which are Gaussian at the large fixed point) through eqn. 2.
7

---

Under review as a conference paper at ICLR 2020
digit
digits
digits
1.0
0.4
0.2
Ordical Mination Standard In inlioiaatiin
0.0
1 repetition
4 repetitions
24 repotitions
0.8
0.6
0.4
0.2
0.0
10
10a
10 10²
20a
10*
10' 3
20*
Iteration
Figure 3: Training accuracy for unrolled, concatenated MNIST digits (top) and unrolled MNIST digits with replicated pixels (bottom) for different sequence lengths. Left: For shorter sequences the standard and critical initialization perform equivalently. Middle: As the sequence length is increased, training with critical initialization is faster by orders of magnitude. Right: For very long sequence lengths, training with standard initialization fails completely (and is unstable from initialization in the lower right panel).
LSTM with hard sigmoid nonlinearities on unrolled CIFAR-10 10 images feeding in 3 pixels every time step. resulting in sequences of length 1024. We also apply standard data augmentation for this task. We present accuracy on the test set in Table 2. Interestingly, in the case of CIFAR-IO the best performance is achieved by an initialization with forward propagation time scale that is much smaller than the sequence length, suggesting that information sufficient for successful classification may be obtained from subset of the sequence.
5.3 REPEATED PIXEL MNIST AND MULTIPLE DIGIT MNIST
In order to generate longer sequence tasks, we modify the unrolled MNIST task by repeating every pixel certain number of times and set the input dimension to 7. To create a more challenging task, we also combine this pixel repetition with concatenation of multiple MNIST digits (either 0 or 1), and label such sequences by a product of the original labels. In this case, we set the input dimension to 1 12 and repeat each pixel 10 times. We train a peephole LSTM with both a critical initialization and a standard initialization on both of these tasks using SGD with momentum. the dimension of the label space is constant (and not exponential in the number of digits like in the latter). In both tasks, we observe three distinct phases. If the sequence length is relatively short the critical and standard initialization perform equivalently. As the sequence length is increased, training with a critical initialization is faster by orders magnitude compared to the standard initialization. As the sequence length is increased further, training with a standard initialization fails, while training with a critical initialization still succeeds. The results are shown in Figure 3.
this former task,
DISCUSSION
We have derived initialization schemes for recurrent networks motivated by ensuring stable signal propagation from the inputs to the loss and of gradient information from the loss to the weights. These schemes dramatically improve performance on long sequence tasks.
The subspace of initialization hyperparameters θ that satisfy the dynamical isometry conditions is multidimensional, and there is no clear principled way to choose preferred initialization within it. would be of interest to study this subspace and perhaps identify preferred initializations based on additional constraints. One could also use the satisfaction of the dynamical isometry conditions as
8

---

Under review as a conference paper at ICLR 2020
guiding principle in simplifying these architectures. A direct consequence of the analysis is that the forget gate, for instance, is critical, while some of the other gates or weights matrices can be removed while still satisfying the conditions. A related question is the optimal choice of the forward propagation time scale for given task. As mentioned in Section 5.2, this scale can be much shorter than the sequence length. It would also be valuable understand better the extent to which the untied weights assumption is violated, since it appears that the violation is non-uniform in O and to relax the constant 2 assumption by introducing a time dependence.
Another compelling issue is the persistence of the dynamical isometry conditions during training and their effect on the solution. has been recently shown that in sufficiently wide networks the gradient flow dynamics in function space are effectively linear (Jacot et al. (2018); Lee et al. (2019)) and under these same conditions the dynamical isometry conditions persist during training. Understanding the finite width and learning rate corrections to such calculations could help extend the analysis of signal propagation at initialization to trained networks in this regime.
REFERENCES
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg, Nan Rosemary Ke, Ioannis Mitliagkas, and Yoshua Bengio. h-detach: Modifying the Istm gradient towards better optimization. arXiv preprint arXiv:1810.03023, 2018.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Bo Chang, Minmin Chen, Eldad Haber, and Ed Chi. AntisymmetrieRNN: A dynamical system view recurrent neural networks. In International Conference on Learning Representations, 2019. URL www.t:tpp//t/...t. opw.reviee......e.. / net/ forum? wwwrummmddrrepoooo....
Minmin Chen, Jeffrey Pennington, and Samuel S Schoenholz. Dynamical isometry and mean field theory of rmns: Gating enables signal propagation in recurrent neural networks. arXiv preprint arXiv: 1806.05344 2018
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Felix Gers, Nicol Schraudolph, and Jürgen Schmidhuber. Learning precise timing with Istm recurrent networks. Journal of machine learning research, 3(Aug):115-145 2002.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and slatistics, pp. 249-256, 2010.
Charles M Goldie. Implicit renewal theory and tails of solutions of random equations. The Annals of Applied Probability, pp. 126-166, 1991.
Charles M Goldie and Rudolf Grübel. Perpetuities with thin tails, Advances in Applied Probability, 28(2):463-480, 1996.
Tomohiro Hayase. Almost surely asymptotic freeness for jacobian spectrum of deep network. arXiv preprint arXiv: /908..9900... 090.. 03901, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.0753 2018.
Cijo Jose, Moustpaha Cisse, and Francois Fleuret. Kronecker recurrent units. arXiv preprint 17Xiv:7005.///44 2017.

---

Under review as a conference paper at ICLR 2020
Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Jascha Sohl--Dcckttein, and Jeffrey Pennington. Wide neural networks any depth evolve as linear models under gradient descent. arXiv preprint arXhv:1922.067200 2019.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: 1804. 12711 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318, 2013.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in neural information processing systems, pp. 4785-4795, 2017.
Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. arXiv preprint 99990999 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in neural information processing systems, pp. 3360-3368, 2016.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 2013.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint 1611 012232, 2016.
Trieu H Trinh, Andrew M Dai, Thang Luong, and Quoc V Le. Learning longer-term dependencies in mms with auxiliary losses. arXiv preprint arXiv: 00000000044,,,. 2018.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference Machine Learning, pp. 3570-3578, 2017.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances Neural Information Processing Systems, pp. 4880-4888, 2016.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. arXiv preprint arXiv:1806.0553 2018.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in neural information processing systems, pp. 7103-7114, 2017.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 190.v:190..04460.. 2019.
Greg Yang and Sam S. Schoenholz. Deep Mean Field Theory: Layerwise Variance and Width Variation as Methods to Control Gradient Explosion. ICLR Workshop, February 2018. URL https: wttp://eeeeeeeeeeeeemeeme...e.e. .nt/ fow.mmidd=
00000.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A Mean Field Theory of Batch Normalization. In International Conference on Learning Representa- tions, 2019. URL https: /ttps://///e//eee.e//// let//forur?dd=SSytxoneeee"
10

---

Under review as a conference paper at ICLR 2020
Appendices
APPENDIX A DETAILS OF RESULTS
A.1 COVARIANCES OF u
The variables Ukiza Ubiii given by Ic are asymptotically Gaussian at the N - 00, with
Uhz = ore Di(ua)Da + Ukog +
Pk2
2
+ 2
1
(14a)
oh I (g )D (E.2 + m1²) +V/ + k2 R#, + Plaz
C
(14b)
82
Q 1
where DE is Gaussian measure on (uas Ub) corresponding to the distribution in eqn. (3).
A.2 MOMENTS OF THE STATE-TO-STATE JACOBIAN
The moments of the squared singular value distribution are given by the normalized traces
MJJa ((JJ")))
Since U", t < t is independent of s', if we index by k, k' the variables defined by eqn. (1b) and (Ic) respectively we obtain
Of
Of
Of
WAS (DEi(uE) + We De.Dgiiu D=r) )s.
J
+
W
+
(15)
Ost
du
du
Under the untied assumption We, W Kg are independent of s u at the large N limit, and are also independent of each other and their elements have mean zero. Using this and the fact that acts srement-wisee we have
MJJ = TJJJ")
E
(16)
kEKU(O)
where
D
k 0
D
uN is given by (1b)
(17)
(2 t0z5 oh (R(())
ok2
D
ug is given by (Ic)
and Do 0f Dí. a the paper are detailed in Appendix A.3.
85 The values of Do, Dj for the architectures considered in Bug
Controlling the first moment of JJJ not sufficient to ensure that the gradients do not explode or vanish, since the variance of the singular values may still be large. This variance given by
Jj MJJT mJ
11

---

Under review as a conference paper at ICLR 2020
The second moment MJJT, can be calculated from (15), and is given by
MJJT 2 = E
Σ
20 - do
(18)
kIIEKU(O)
where the ag are defined in eqn. (17). We arrange the scalars do, {at} into a vector a. In Appendix we show that the results of the calculations in this section match the empirical spectrum of the Jacobian.
For all the architectures considered in this paper, we find that Do o(uf) while Dj. finite as Vk: of - 0. Combining this with (11), (16), (18), we find that if Σ, = the dynamical isometry conditions are satisfied if ag 1, ak O, which can be achieved by setting Ak: o 0 and taking jy + oo. This motivates the general form of the initializations used in the experiments $ although there are many other possible choices of O such that the 0140 vanish.
Given the above general form of the dynamical isometry conditions for recurrent networks, now provide the detailed forms that apply the LSTM, peephole LSTM and GRU.
A.3 DYNAMICAL ISOMETRY CONDITIONS FOR SELECTED ARCHITECTURES
We specify the form of XO; and for the architectures considered in this paper:
A.3.1 GRU
auuj))(((pp)) + 03 ( tanhhu (anh(urzbl thihi
o' d'(uja)o''ugp)
X = E
ol a(ura)o(urn)) to hahio (unaa (
+o2 2 (1 r2
(u)T)) tanh' tanh((urza)) tanh' (1r2))
02(u1T) 03 (tanh" + Q ("(
a
0 02 h h²2 o(u;)) tanh 2 (ui )o (ur)
(1 ouu;) )² tanh (ur, )o² (un)
12
A.3.2 PEEPHOLE LSTM
au (utj)) + +0,0' (u (o (u tanh(u,,)) tanh(u,))
XC: = E
of C (uga) o' + 0: u ) (ug)) tanh' (uta) tanh (urs)
o²(ut))
(0²((1;) tanh? (u;)
of (:0²(u;)) tanh (oun?((u)) (at)
'In the case of the LSTM, we also want to prevent the output gate from taking very small values, as explained in Section A.3.
12

---

Under review as a conference paper at ICLR 2020
Figure 4: Squared singular values of the state-to-state Jacobian in eqn. (15) for two choices of hyperparam- eter settings ④ The red lines denote the empirical mean and standard deviations, while the dotted lines denote the theoretical prediction based on the calcu- lation described in Section 4.3. Note the dramatic difference in the spectrum caused by choosing an ini- tialization that approximately satisfies the dynamical isometry conditions.
Typical Hyperparameters
Approximate Dynamical Isometry
0.5
2.8
255.05
1.10 1.15
A.3.3
LSTM
aluja)a(ufj + oud tanh(c") tanh(c)) c' (o'(()))))
XC
E
tu (u tanh (ct) tanh'(ct)
+ofo (utt (uttt) tanh(ur, (anh(u))) +0 uta )o (u tanh' (u; tanh' (uft)
(2(
0i (wn) tanh" (c)) (ut tanh
of tanh (anh"(c*)c" o"" (Fo2(w)) tanh (c*))² tanh (uth((())) o0 (cnh"(c")
a
(
When evaluating in this case, we write the cell state as ct tanh
o(o'))
and
assume olo') for large t. The stability of the first equation and the accuracy of o
the second approximation are improved if of is not concentrated around 0.
A.4
RNN TABLE SUPPLEMENT
Minimal RNN Chen al. (2018)
peephole LSTM
Gers et al. (2002)
K
(Jrr}
fr, (u)) o tanh(s")
p'
a au o
(u o t-1
+(1
a())) xx
+o(u() tanh(uf))
Table Additional recurrent architectures written in the form 1. p' the output of the network at every time step. See Table I for more details.
APPENDIX B SQUARED JACOBIAN SPECTRUM HISTOGRAMS
To verify the results of the calculation of the moments of the squared singular value distribution of the state-to-state Jacobian presented in Section 4.3 we run an untied peephole LSTM for 100 iterations with i.i.d. Gaussian inputs. We then compute the state-to-state Jacobian and calculate its spectrum. This can be used to compare the first two moments of the spectrum to the result of the calculation, as well as to observe the difference between standard initialization and one close to satisfying the dynamical isometry conditions. The results are shown in Figure 4. The validity of this experiment rests on making an ergodicity assumption, since the calculated spectral properties require taking averages over realizations of random matrices, while in the experiment we instead calculate the moments by averaging over the
13

---

Under review as a conference paper at ICLR 2020
eigenvalues of a single realization. The good agreement between the prediction and the empirical average suggests that the assumption is valid.
APPENDIX C AUXILIARY LEMMAS AND PROOFS
Proof of Lemma 1. Despite the fact that each Ukai as defined in 1 depends in prin- ciple upon the entire state vector Sa, at the large N limit due to the isotropy of the input distribution we find that these random variables are ii.d. and independent of the state. Combining this with the fact that f is an element-wise function, it suffices to analyse a single entry of st f(s {ug).... H) which the large t limit gives
E [f(sa, (a)f Ug)] - (H²)
Mc(p', Q C.)
Q; (11))
where U Hutal E K, tha ~ N Qi - - and Ug is defined similarly (i.e. we assume the first two moments have converged but the correlations between the sequences have not, and in cases where depends on a sequence of {ug],.., {ut} we assume the constituent variables have all converged in this way). We represent Wha, Ubo via a Cholesky decomposition as
Uka
=ko + fi
(19a)
Σn Cia + (11((()220) + 1k
(19b)
where
Zkas 2kh
1)0, i..d.
We thus have duia
aC
Vt (2z)2
2ka
Ck Zkb
Combining this with the fact that
VI-Ct
(Dzg(z)z
DD for any g(2), integration by parts gives for any
91. 92
0 f DDiu "Dzka D 2krg1 (ett)())))))()))))) (2 8C₁
"iika kkaa D D2ib.1 (wka) Ogu(uie) ougo Ouyg 80c
(20)
E D Zzk D Dzuo
(wi.)) (gg()
Outa
Ou
Denoting Σ Q¹ - -2x)² and defining Ex, Ex, similarly, we have
dCka
-2 2
kaa
(21)
dC.
dCu, dC.
Ji *
Dt
gk. (ut 1hh (uth)
of (2) (wZ)) )²) Ogn lan(uia) dC's dC,
D
OCt
14

---

Under review as a conference paper at ICLR 2020
23
12
gk (u EgC₂
9bk (1%))
Di
tot
(b(u (a (1(s)
+/1²2 +2
where in the last equality we used 20. Using 20 again gives
(Mc Ca)
Eg /
(f(uiaa) Of(()s)
Dika D zkb Dzbb
aCk
duka
Ouio
We now note, using 4, that if C2 = 12 = 1 we obtain Ck = 1 and thus
dCk2
0 3
f g ( D
dC, C-11,1=11
l2
OMolK. C2,Os)
/
0 af(
= E
Dzk
)²
Oc₁
Ouk
0,=1E,=1
aMolk's O₂ C3s) Oc,
O5(s)
E
)²
Os
C.11E.=1
combining the above equations with 21 and comparing the result to 16 completes the proof.
Lemma 2. For any odd function g(x), ( x ) N ((#) H
1 C
)
Σ
05
C' < we have (g((x)(()) 2 0.
Proof of Lemma 2. For C 1 the proof is trivial. We now assume 0 < C < We split up I22 into four orthants and consider a point (a, b) with a, b > We have (())g(b) = gg(-=))(--b = =-9(a)g(-b) - > O. We will show > p(a, -b) + + where p is the probability density function of (xr y) and hence the points where the integrand is positive will contribute more to the integral than the ones where it is negative. Plugging these points into p(x,2) y) gives
that p(a, b) + p(-a(aa--0
p(a, b) + p(-a, a, b) e p(a, bb) + p(-a,b) - + (atp://woooeoooddd------------00
+ -
where is some positive constant that depends on the determinant of the covariance (since C < the matrix is invertible and the determinant is positive).
exp (2aCab) cosh (2ua - (a + b)) (bb++b)
exp (=p2Ca))) cosh (2paa(1 - C' b)) 2
15

---

Under review as a conference paper at ICLR 2020
where the last inequality holds for 0 < C < 1. It follows that the positive contribu- tion to the integral is larger than the negative one, and repeating this argument for every (a, b) in the positive orthant gives the desired claim (if a 0 or b = 0 the four points in the analysis are not distinct but the inequality still holds and the integrand vanishes in any case).
Lemma 3. The map 5 is convex in the case of the peephole LSTM.
Proof of Lemma 3. We have
M(C.) Sffh - (K))30% + pe
+ 12
1 r.r + 2 f f 1
-
Q- 1r2
From the definition of uhe and C we have
Ouks Oc%
Cj
O0k
C
o - M +2
V (xz)2
2ka
Zkb
Zka
2kh
CT2
act
C
O7 -1
k
and using / (Drg(x)x f Darg (x) we then obtain for any g(x))
9 80%
f D Dizaa kka D Dezuo( (asbg(ukiuu (u f D Drsog (u (uke) (oo (uken) aCe
Outy
(22)
=f -vt2) Dir Zuogg (g'((()))
8 MM..). uee
82
[ (1Q: (1Q (1t) + +x(e)) + / id0o / rarb
(Qr -x?
OC2
ac?
82 [ fale
((2² c
(1+)2 +(22) 2
oC. fafolQe - (HE)2)
OC%
82 f inie
J raT' + 0 inni 0 rrar + f in 002
82 frarb
ac2
aC. aC.
From 22 and non-negativity of some of the integrands
22 O f ((Qr - (HE)²)C + +x?) +
22 pfiii
/ (Tabh [ + /
82 frab
f inii 00%
dc2
dc?
From Lemma 2 we have f rarb 2 0 and thus have
2² ʃ gags ac2
a ʃ ""9h"" 2 for g f,i,r. We
PM(C.
20 0
Oc2
for 0 S C. 1.
Convexity of this map has a number of consequences. One immediate one is that the map has at most one stable fixed point.
16

---

Under review as a conference paper at ICLR 2020
APPENDIX D THE LSTM CELL STATE DISTRIBUTION
Algorithm LSTM hidden state moment fixed point iteration using cell state sampling
function FIXEDPPNNTITERATIONN"""""
"Q--1 O n2 Mhas))
2 <- CATCULLATRRX(Q0
Using 4
Initialize C E IR
for i f- to Mitn do
U, U,, uy < SAMPLEUS(Q ()
> Using
C U-U u,
b Using 13
end for
(n', Q%) <- (1 1, ,,,c,
Using
return Q5)
end function
As mentioned in the main text, the cell state differs substantially from other random variables that appear in this analysis since it cannot be expressed as a function of finite number of variables that are Gaussian at the large N and limit (see Table
1). Since at this limit the ut are independent, by examining the cell state update eqn. (13) we find that the asymptotic cell state distribution is that of perpetuity, which obeys
X 2 + Z
for some random variables Y, Z. Its stationary distributions will have heavy tails Goldie (1991). Due to the bounds on Y, 7 in the case of the LSTM, one expects based on results in (Goldie & Grübel (1996)) that the tails of the stationary cell state distribution will be exponential in the case of hard sigmoid nonlinearities or decay like x- in the case of soft sigmoid nonlinearities, and this tail behavior is indeed observed in simulations. Aside from the tails, the bulk of the distribution can take variety of different forms and can be highly multimodal, depending on the choice of O which in turn determines the distributions of Y, Z.
We overcome this difficulty by sampling from the stationary cell state distribution. For a given value of Qh. the variables u appearing in (13) can be sampled since their distribution is given by (3) at the large N limit. The update equation (13) can then be iterated and the resulting samples approximate well the stationary cell state distribution for a range of different choices of θ, which result in variety of stationary distribution profiles (see Appendix E3). The fixed points of (5) can then be calculated numerically as in the deterministic cases, yet care must be taken since the sampling introduces stochasticity into the process.
An example of the fixed point iteration eqn. (5) implemented using sampling is presented in Algorithm 1. The correlations between the hidden states can be calcu- lated in similar fashion. In practice, once the number of samples n. and sampling iterations Niters is of order 100 reasonably accurate values for the moment evolution and the convergence rates to the fixed point are obtained (see for instance the right panel of Figure 2). The computational cost of the sampling is linear in both n.: Niters (as opposed to say simulating neural network directly in which case the cost is quadratic in n.).
17

---

Under review as a conference paper at ICLR 2020
APPENDIX E ADDITIONAL EXPERIMENTS AND DETAILS OF EXPERIMENTS
E.I DYNAMICAL SYSTEM
We simulate the dynamics of Q in eqn. (5) for a GRU using inputs with Σ = for t 10 and Σ for 1 2 10. Note that this evolution is independent of the evolution of C. The results show good agreement in the untied case between the calculation at the large N limit and the simulation, as shown in Figure 5.
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.2
0.2
-0.2
-0.2
50
10
20
30
40
50
1.1
1.1
1.0
1.0
0.9
0.9
0.8
0.7
0.6
0.5
-2 1
0
2
4 6
-2
-1
1
3 5
14
Figure 5: Top: Dynamics of the correlations (6) for the GRU with 3 different values of μ function of time. The dashed line is the prediction from the mean field calculation, while the red curves are from simulation of the network with i.i.d. Gaussian inputs. Left: Network with untied weights. Right: Network with tied weights. Bottom: The predicted fixed point of (6) as a function of different μ.. Left: Network with untied weights. Right: Network with tied weights.
E.2
HEATMAPS
In Figure 6 we present results of training on the same task shown in Figure with tied weights, showing the deviations resulting from the violation of the untied weights assumption.
E.3 SAMPLING THE LSTM CELL STATE DISTRIBUTION
As described in Appendix D, calculating the signal propagation time scale and the moments of the state-to-state Jacobian for the LSTM requires integrating with respect to the stationary cell state distribution. The method for doing this is described in Algorithm 1. As is shown in Figure 7, this distribution can take different forms based on the choice of initialization hyperparameters θ, but in all cases we have studied the proposed algorithm appears to provide reasonable approximation to this distribution efficiently. The simulations are obtained by feeding network of width N 200 with i.i.d. Gaussian inputs.
18

---

Under review as a conference paper at ICLR 2020
12
GRU
peophale LSTM
LSTM
127
20
Figure 6: Training accuracy the padded MNIST classification task described in 5.1 at different sequence lengths T and hyperparameter values θ for networks with ried weights. The green curves multiples of the forward propagation time scale calculated under the untied assumption. We generally observe improved performance when the predicted value of & is high, yet the behavior of the network with tied weights is part of the scope of the current analysis and deviations from the prediction are indeed observed.
Sampling
Simulation
Figure Sampling from the LSTM cell state distribution using Algorithm 1, showing good agreement with the cell state distribution obtained by simulating network with untied weights. The two panels correspond to two different choices of
19

---

Under review as a conference paper at ICLR 2020
E.4 CRITICAL INITIALIZATIONS
Peephole LSTM:
Pi Hos p3, Py. Pr: P 1² 22
22 p:
vー 0
11
H 5
0 f. 0 0 10
LSTM (Unrolled CIFAR-10 task):
Pis Pr: oo Pi fz
2
por
f ve
0
v 2
0
2
H
r:
10-5
The value of ju was found by a grid search, since for this task information necessary to solve it did not require signal propagation across the entire sequence. In other words, classification of an image can be achieved with access only to the last few rows of pixels. The utility of the analytical results in this case, as mentioned in the text, is to greatly constrain the hyperparameter space of potentially useful initializations from theoretical considerations.
E.5 STANDARD INITIALIZATION
LSTM and peephole LSTM:
Kernel matrices (corresponding to the choice of v²): Glorot uniform initialization Glorot & Bengio (2010)
Recurrent matrices (corresponding to the choice of oi): Orthogonal initialization (i.i.d. Gaussian initialization with variance 1/N also used giving analogous perfor-
mance)
Hig pr Mox Pr. Pr. p3 =
E.5.1 LONG SEQUENCE TASKS
Learning rate scan: 8 equally spaced points between 10-3 and 10-5, Validation set: 10000 images for MNIST and CIFAR-10. In Table 2, the results for the standard LSTM on MNIST were reproduced from Arpit et al. (2018) and the results on CIFAR-I 10 were reproduced from Trinh et al. (2018).
20


Reviews:
[{"review_id": "r1xrrEojYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper touches the signal processing/long term propagation problem in gated recurrent neural networks from the mean field theory. The paper starts from a dynamic system view of the recurrent neural networks and calculates the time scale of converging to the fixed point. In order to avoid the system to converge to the fixed point, the authors utilize some initialization strategy to keep the time scale to infinity. The authors also relate the time scale to state-to-state Jacobians. \n\nThe paper is interesting but more details could be added to both theories and experiments. Given all those extra details, I can increase my scores.\n\nFor the theory:\n1. It is unclear how you go from equation (7) to (8). References or more explanations need to be added.\n2. It is unclear how the initializations in E.4 satisfies the criteria you define in the paper. More explanations can be added.\n3. This mean field approximation is still far away from practice. It is hard to believe the input in real life is Gaussian distributed vectors (sec 4.2). What will happen if the input distributions are not Gaussian? This should be discussed.\n4. This initialization only helps at the beginning of the training. What will happen if we do one backpropagation? This should be addressed.\n\nFor the experiments:\n1. The authors did not state the some of the experiment details in the papers, like what optimizer, regularization, learning rate.... To make better assessment of the experiments, those details should be added. Do you train/tune all the different initializations in the same way?\n2. I cannot find the description of Figure 1 anywhere in the paper. It is hard to believe LSTM did poorly on sequential MNIST unless giving more details since LSTM has been proved to perform okay on sequential MNIST in a bunch of papers[1]. \n3. What is the meaning of 112 dimension in 5.3? Does that mean you only choose the first 4 rows of MNIST images?\n4. Comparison over random seeds should be honestly justified for all the experiments.\n\nMinors:\n1. You have one missing \\Sigma_z in equation (4a).\n2. \\mu_s^2 in equation (4b) should be (\\mu_s^t)^2\n\nReferences:\n[1] Arjovsky, Martin, Amar Shah, and Yoshua Bengio. \"Unitary evolution recurrent neural networks.\" International Conference on Machine Learning. 2016.\n\n-------------\nI change my scores to weak reject . I agree with Reviewer 2 that the paper provide some insight from Physics and can be an interesting contribution to the community. However, I think all of the reviewers. including myself, find the paper hard to read for the general machine learning audience. And even though the authors mention that they will fix the text in the future, they do not change any text of the paper. I think writing is also important besides presenting interesting research ideas. Overall, I think the paper will be benefited from resubmission.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xoecU2Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The aim of this paper is to suggest randomized initializations for the various weights of a recurrent neural network (GRUs and various LSTMs are covered), such that training these networks gets to a successful start, when the model is trained on long sequences. Instead of being heuristic, their approach follows first principles of analyzing signal propagation through time, using ideas from statistical thermodynamics (mean field approximations). Some experiments, on toy datasets, validate their approach.\n\nI am quite intrigued by this paper. It is using interesting theory, shapes it to a practically highly relevant and difficult applied problem, and in the end comes up with a computable criterion of how to choose hyperparameters (means and variances of Gaussians to sample initial weights from). While the results in practice are still not too convincing, I am strongly in favour of giving this approach the benefit of doubt, as it could lead to practically very useful downstream work.\n\nThe main direction of improvement for this paper (given that experiments are what they are -- somewhat limited to toy situations right now) is to better explain the methodology to researchers not familiar with mean field methods. Most importantly, it is not explained in the main text how hyperparameters are really chosen in the end. Looking at Appendix E, I find some pretty basic choices, and no other alternatives considered. It is not explained why these choices satisfy the theory, why they'd be the only ones, etc. This creates a disconnect between the very nice (and seemingly useful) theory and its implications (they are not really well spelled out).\n\nHere is what I understood (and I am not specifically an expert on stat mech). The authors assume that the dimension of latent states (N) grows large. They assume that weights are sampled independently, and identical distributed in groups k (different cell types, weight vs bias), and that inputs are correlated with each other in each dimension. Based on these assumption, they follow Gaussians statistics through a number of time steps. In the limit, one gets a deterministic dynamical system, and as t -> infty, this may converge to a fixed point. In a very nice argument (which they could explain better), they state that such rapid convergence is bad news, because then information cannot spread across long time scales, so one has to find hyperparameters for which the system behaves \"critically\". A second arguments tries to keep gradient sizes (under MF assumptions) of O(1), so neither -> 0, not -> infty, which is again some \"critical\" range. Under their assumption, these critical conditions can be computed depending on the hyperparameters.\n\nUnfortunately, this is where the paper somewhat stops, it does not give specific methods for finding hyperpars that satisfy the criteria, at least not in the sense of characterising the whole space of such hyperpars (instead, in Appendix E, they just state some few settings that do). As a direction for future work, this would be very important. Another side question is whether for what the authors call \"trainability\", the only point that matters is whether for the initial weights, signals can spread and gradients are O(1). It is certainly necessary, I see that.\n\nDetailed comments:\n- Please fix Table 1, the expressions seem broken. What does \"r2\" mean in the GRU column?\n- At least for me, (1a) to (1c) really was too short. At least in the Appendix, please do explain how this gives GRU and LSTM\n- Please explain the untied weight assumption somewhere. s^t is a map of s^(t-1) and W_k, so how can W_k be independent\n   of all s^t? What are you really assuming here?\n- It took some repeated reading until I understood why the expressions in (2a) to (4b) do not depend on i, j, a, b (except\n  whether a = b or a != b). Explain that properly\n- The core of the whole approach seems to be first half of page 5. This seems like a very nice argument, but hard to understand. Try making it more crisp. I kind of get the rough idea why fast convergence over t would be bad, but would total divergence over t not also be bad?\n- In (12a-c), do you mean \"equal\" or \"approximately equal\"?\n- In 4.4: \"This motivates the general form of the initializations used in the experiments\": You have to make this more explicit. Why are your choices the only ones? Could there not be other choices satisfying (12a-c) approx, and be better?\n- Value of Sigma_z = 1: This seems odd to me, then your covariances are degenerate (rank 1 instead of 2). Please explain\n- Standard LSTM harder than GRU or peephole LSTM: Again, this sounds real interesting, but I did not get it from the explanation\n- I did not understand Figure 2. How are Theta_0, Theta_1 chosen?\n- As said above, the experiments are interesting, but somewhat artificial. Please do at least comment on real-world applications, and whether (and how) the ideas here would apply\n- Discussion: \"there is no clear principled way...\": Well, but practitioners need something. I'd disagree, at least one could attempt to navigate this space by global optimization techniques...\n\n\nADDITIONAL COMMENT:\n\nI tried to append the following as comment, but the (pretty broken) system would not let me, insisting that \"reader is not valid\" (???). Anyway, here it is. I hope I am allowed to add to my own review.\n\nI've seen the argument in reviews that assumptions made by this paper about independencies between weights and inner states are wrong, and therefore conclusions are not valid.\n\nFirst, such assumptions are indeed pretty common in such statistical mech analyses of learning methods. Second, you have to distinguish between weights after (random) initialization and after training. Of course, LSTM represents long term dependencies after training, but initialization is a different story.\n\nIf I was the AC for this paper, I'd ask somebody with at least some background in statistical mech to provide some additional opinions, as the reviewers (including myself) are not fully qualified.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxDSP-CtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a mean-field analysis of recurrent networks in this paper. I have a few concerns about this paper:\n\n(1) The most serious concern about their analysis comes from their assumption. They assume the weight W is independent on the state s_t (Page 4, Lines 5-6). The recursive structure is the most complicated part of the recurrent networks, and also its major difference from feedforward networks. In current networks, the hidden states become (or even heavily) dependent on the weight due to recursion.\n\nWhen making such an assumption, the recurrent networks just become similar to feedforward networks. The authors' claim that \"the untied weights assumption actually has long history of yielding correct prediction\" is not ungrounded and questionable.\n\n(2) The paper is not well written. Some assumptions are not explicitly stated. They are placed in the text without any highlight. Some theoretical statements are claimed without any rigorous proof. A few approximations are applied without clearly explaining about the resulting approximation errors. This is not acceptable, especially when the authors claim they are developing a \"THEORY\".\n\n(3) The experiments only consider the MNIST and CIFAR10 datasets. These datasets are mainly used for evaluating feedforward-type convolutional neural networks. Even though the authors might like their experiments, for the sake of the main stream users of recurrent networks. They should at least include experiments in conventional sequential prediction problems, e.g., speech, time series, machine translations.\n\n(4) Compared with other state of the art methods, their experimental results on CIFAR10 is too weak. I cannot believe such weak results can be used to make meaningful justifications.\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Using ideas from mean-field theory and statistical mechanics, this paper derives a principled way to analyze signal propagation through gated recurrent networks.  This analysis then allows for the development of a novel initialization scheme capable of mitigating subsequent training instabilities.  In the end, while reviewers appreciated some of the analytical insights provided, two still voted for rejection while one chose accept after the rebuttal and discussion period.  And as AC for this paper, I did not find sufficient evidence to overturn the reviewer majority for two primary reasons.

First, the paper claims to demonstrate the efficacy of the proposed initialization scheme on multiple sequence tasks, but the presented experiments do not really involve representative testing scenarios as pointed out by reviewers.  Given that this is not a purely theoretical paper, but rather one suggesting practically-relevant initializations for RNNs, it seems important to actually demonstrate this on sequence data people in the community actually care about.  In fact, even the reviewer who voted for acceptance conceded that the presented results were not too convincing (basically limited to toy situations involving Cifar10 and MNIST data).

Secondly, all reviewers found parts of the paper difficult to digest, and while a future revision has been promised to provide clarity, no text was actually changed making updated evaluations problematic.  Note that the rebuttal mentions that the paper is written in a style that is common in the physics literature, and this appears to be a large part of the problem.  ICLR is an ML conference and in this respect, to the extent possible it is important to frame relevant papers in an accessible way such that a broader segment of this community can benefit from the key message.  At the very least, this will ensure that the reviewer pool is more equipped to properly appreciate the contribution.  My own view is that this work can be reframed in such a way that it could be successfully submitted to another ML conference in the future.
Decision: Reject