Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Authors: Andrew Saxe, James L. McClelland, Surya Ganguli
ICLR 2014

Abstract: Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, thereby providing analytical insight into the success of unsupervised pretraining in deep supervised learning tasks.


Peer review:
[Review]:
* Brief summary of the paper:

This work is mostly a mathematical analysis of the dynamics of gradient descent learning in *linear* multi-layer networks, which surprisingly displays a complex non-linear dynamic. It delivers analytic results and novel insights regarding learning speed, how it is affected by depth, and regarding optimal initialization regimes, drawing connections to the benefits of pre-training strategies. In particular it shows that orthogonal initialization is a better strategy than the appropriately scaled random Gaussian initialization proposed in Glorot & Bengio AISTATS 2010. Mathematical results are empirically validated with linear networks trained on MNIST. Some empirical results using non-linear tanh networks are also provided, convincingly showing that some of the insights gained from the analysis of linear networks can apply to the non-linear case. 

* Quality assessment:

This is a very well written paper. I believe it offers a novel, thorough, and enlightening analysis of the dynamics of learning in linear networks. This is a worthy enterprise, and succeeds in uncovering novel insights that will likely also have an impact on practical approaches to training non-linear deep networks. It also paves the way to a more formal analysis of the dynamics of learning in non-linear deep nets.
I found the analysis of initialization strategies in sections 3 and 4 particularly interesting due to their practical relevance. 

My only concern, since this paper literally offers a lot, has to do with the ICLR policy on page limits (and text squeezing), that I am unsure about. To be checked.


* Pros and Cons:

+ Thorough mathematical analysis, conveying valuable novel insights on the dynamics of learning in deep linear networks
+ Evidence that these insights can have practical relevance for improving approaches to training deep non-linear networks
- TODO: Check ICLR policy on page limits??


* Detailed suggestions for improvements:

- Top of Figure 3 hides part of the above paragraph (a part that seemed important for understanding!)
- Bottom of caption of figure 3 too close to main text (almost overlaps)

- I find the term 'learning time' ambiguous or ill-chosen, as you do not take into account the fact that deeper networks usually involve more computation per epoch. I suggest you clearlry define it as meaning number of training iterations (epochs) upon first introducing it.

- Top of page 5: there is an ill-placed newline between 'logarithmically spaced' and the rest of the sentence.

- p 9: 
'mode strength was chosen to be small (u=0.001)'  ... because ... ? 

'We show (Fig 6A, left, red curve)' -> 'We empirically show (...'

- Figure 6: 'A Histograms of the singular values' should be 'B Histograms of the singular values'

- p 11: 
'netoworks' -> 'networks'

- Section 4:
'and phi(x) is any nonlinearity that saturates as ...' Any nonlinearity, really? This doesn't sound right, since any nonlinearity could incorporate some scaling that could override the role of gain factor g. You should more specifically characterize the properties of the nonlinearity, and state here which one(s) you will actually consider.

- Discussion:
'propery' -> 'property'

Appendix D.2:
'An analysis of this effect in deep linear networks' I believe you meant to write 'in deep nonlinear networks' since linear networks would all converge to equivalent solutions (at different paces) and thus could not exhibit any difference in generalization.