Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Authors: Andrew Saxe, James L. McClelland, Surya Ganguli
ICLR 2014

Abstract: Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, thereby providing analytical insight into the success of unsupervised pretraining in deep supervised learning tasks.


Peer review:
[Review]:
This paper analyzes gradient descent learning in deep networks of linear units.  While such networks are trivially equivalent to 1 layer linear transforms, the authors argue that their learning dynamics should act as a tractable analogy for learning in non-linear networks.  Under certain simplifying assumptions they are able to produce some analytic expressions for how gradient descent learning acts in the idealized continuous time case.  

There is also an attempt to justify pre-training in terms of this analysis that I find not particularly convincing, as here the 'pre-training' amounts to solving the problem almost in the linear case but certainly not in the linear case.  

A new addition to this paper is a discussion non-linear networks and the role of initialization, which includes some interesting numerical simulations demonstrating that certain choices of scaling constant under a particular 'orthogonal' initialization scheme.  I like this part of the paper the best.  However, it is not the focus of the paper and should be developed further (perhaps in another paper?).

I should say that I reviewed this paper before.   It is in many ways improved from the original version.  A few problems still remain, and there are also many new ones (page 8 in particular). However, I'm mostly satisfied with it and would recommend acceptance.  I'm hoping the authors can answer my various question, as some parts of the paper either confused me and possibly contain mistakes.


In terms of the main content, the central issue I have with this paper is that the special initial conditions in which in analysis is done are essentially a partial solution to an SVD problem, which if it were solved fully, would give the optimal weights of the linear network in one step.  In particular, the assumptions used require that the weight matrices share singular vectors in a way that basically turns the optimization across all layers into a decoupled set of scalar optimizations.  In non-linear networks, where the optimal solution isn't given by an SVD, it is harder to believe that this kind of partial SVD would help as much as it does in the linear case (or perhaps at all).

I appreciate the attempt to make the analysis more rigorous in terms of 'learning speed'.  Although perhaps a better way to work out a realistic lambda would be to bound the error one gets by taking the continuous time approximation of the original difference equations and choose lambda based on when this error can be argued to have a negligible effect.  A lambda which has been chosen via some kind of curvature criterion will generally give a more stable iteration, but there is no guarantee that this will imply a reasonable correspondence between the discrete and continuous time versions, where mere stability isn't enough.





Abs/Intro:  'edge of chaos'?  This is only explained late into the paper, and sounds bizarre without any context.

Page 3:  'the input-output correlation matrix contains all of the information about the dataset used in learning'.  You mean just for learning linear networks/functions right?  Surely this isn't true in general...

Page 3: 'column of W^{21}'  Should this have a bar over it?

Page 4:  When you say that the fixed point structure of gradient descent learning was worked out in [19], you don't mention on what kinds of networks this analysis was done.

Page 5:  There are formatting issues at the top of the page.  Some text is cut off.

Page 6:  The word 'caveat' (used twice) doesn't seem appropriate here.  I usually don't think of special cases of definitions, or simplifying assumptions, as 'caveats'.

Page 7:  The bottom of the page contains formatting issues with the figure.

Page 7:  You should mention this is a classification task and also that the details are in appendix C.  The way it is written now it sounds like only the details of the choice of learning rate are in the appendix.

Page 8:  My experience is that optimization performance is only improved in the short term and maybe medium terms by pre-training vs standard carefully scaled inits.  In the longer term (which is what really matters in the end, as it dominates most of the run-time) the difference is less significant, if it is present at all. Improvement in generalization performance is less controversial, and in my opinion remains the best reason to try unsupervised pre-training.

I think it is quite reasonable to suspect that, insofar as the networks are approximately linear, perhaps unsupervised pre-training is having the boosting effect that your paper predicts that it might have.  But it seems reasonable to suspect that in later stages of optimization that the network may behave in a much less linear fashion as the units move out of their linear regimes, and so the optimization advantages of pre-training will diminish or even disappear.  This is consistent with the evidence from the literature.  

In Appendix D you cite some evidence from other papers.  But the results with Hessian-free optimization and other methods are consistent with what is written above (believe me, I know this work very well).  Look for example at the results for SGD in Figure 3 of the Chapelle & Erhan paper that you cite.  SGD reaches nearly identical KLs on MNIST after 700 dataset passes when you compare random initializations vs RBM pre-trained initializations.  The Curves results seem to favor pre-training a small bit with SGD, but the difference is small (maybe 1.5x faster), and would likely become insignificant in the longer term as random-init SGD appears to be catching up before the graph cuts off.    My experience, along with that of many other people who have studies these methods is that if you run these kinds of experiments longer, the approximate 1.5-2x speed increase with pre-training wanes significantly over time, eventually reaching near parity with well chosen random initialization methods, so that overall not much time is saved, if any.

Page 8:  The discussion about what pre-training should do didn't make sense to me in multiple places.

For example, previously you said that you were assuming Sigma^11 = I, but I guess this is no longer the case.  You then say that the product of the weights converges to Sigma^31(Sigma^31)^-1.   But how can this be true unless N2 >= N1,N3?  When N2 < N1,N3, the product of the weight matrices has rank at most N2, while Sigma^31(Sigma^31)^-1 will be full-rank in general.

You defined a^alpha to be a vector before, but now it is a 'strength'?

When you consider taking W21 = R2Q^T where R2 is 'an arbitrary orthogonal matrix', what does this actually mean?   Orthogonal matrices are square, and so is Q, so does that make W21 square (i.e. N1 = N2)?  And if R2 is truly arbitrary, then it is meaningless, as any orthogonal matrix M can be written as R2 Q^T for some R2 (take R2 = MQ).

When you say 'Now consider fine-tuning on...' are you now taking the input and output not to be equal.  This is confusing especially since this sentence appears in a paragraph which starts by saying that we are now talking about auto-encoding with input = output.

In what sense is 'W^21 = R2D1V11' a 'task'?   I suspect you meant something else here but this sentence is very awkwardly phrased.

I gave up trying to understand what this part was actually trying to say.


Page 9:  What is the error being plotted in the figure?

Page 9:  That initializing a linear network with part of the optimal SVD solution (which if you take the full solution, will instantly give the optimal error) isn't too surprising.  It would be much more interesting to see if these initialization schemes work well in the nonlinear case.

Page 9:  The Glorot initialization scheme uses uniform random numbers, and I think there is a factor of 6 somewhere in there.  Gaussians with 1/sqrt(N) is definitely not what they end up recommending.  Also, they did their calculations for tanh units, although it should probably work for linear ones.   In my experience, getting the scale constants and other details right matters a lot in practice!

Page 9:  How can you have the weight matrices be random orthogonal matrices?  Orthogonal matrices are square and the weight matrices don't have to be.

Page 11:  Merely scaling g very large won't be any good.  The 'activity' will propagate, yes, but the units will completely saturate.

Page 12:  'Variance' isn't really what eqn 21 is measuring.  That implies deviation from the mean, which doesn't have to be 0 here.  The units could all be completely saturating (this is bad), but eqn 21 would still have a non-zero finite value.

Page 12:  The analysis done in section seems interesting.  It does sound similar to work done on Echo State Networks (ESNs) and initializations.  Have you compared to that?  Sutskever et al. (2013) also report finding empirically that there was a critical range of scale parameters around which learning seemed to work best.  These scale parameters were also applied to weight matrices which had their spectral radius initially normalized to 1, which is similar again to the ESN papers. 

What is missing for me from this section is an application of these ideas to real networks.  Do these ideas improve on optimization performance?  A good comparison would be against a *properly implemented* Glorot-style initialization (see my above comments re. this), and/or something like the sparse initialization used in the HF papers.