Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Authors: Andrew Saxe, James L. McClelland, Surya Ganguli
ICLR 2014

Abstract: Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, thereby providing analytical insight into the success of unsupervised pretraining in deep supervised learning tasks.


Peer review:
[Review]:
I'm not an official reviewer of this paper but I want to state for the record that I think it should be accepted. In the interest of full disclosure, I should make it clear that I'm a former co-author of the the lead author of this paper, but I don't think that has compromised my objectivity in this case. I also reviewed an earlier version of this work for NIPS 2013 without knowledge of its authorship and I also recommended that it be accepted then.

I don't intend for this comment to be a complete review, but I do want to argue against one of the main criticisms of this paper that I've seen made by the official reviewers, which is that deep linear models are uninteresting and that the algorithms studied are not used in practice.

Linear networks formed by composing matrix multiplies are an interesting model class in my opinion. They allow us to study some of the properties of deep networks without needing to commit to analyzing the most difficult cases.

Other papers have previously been published at leading conferences about using deep linear models to study deep learning. See for example this ICML 2012 paper from Geoffrey Hinton's group:
http://www.cs.toronto.edu/~tang/papers/deep_mfa.pdf
This paper chooses to study deep mixtures of factor analyzers (DMFAs) 'even though a DMFA can be converted to an equivalent shallow MFA by multiplying together the factor loading matrices at different levels'.

While it is true that the state of the art deep nets are not linear, so technically the exact method studied here is not used in practice, I think that the current state of the art deep nets are similar enough for this case to be interesting. For a wide variety of problems, rectified linear hidden units feeding into a linear classifier such as a softmax or SVM are the state of the art. I myself have also had a lot of success with my 'maxout' units, which are also locally linear. So quite a lot of the current neural net literature deals with networks whose functions are linear on domains of non-negligible size.

If you train a deep rectifier net and monitor how many hidden units change which linear piece they are are for each gradient step, this 'piece change rate' is quite small. I ran this experiment on a state of the art feedforward net on MNIST (trained with dropout and rectified linear units). I found that by the end of the first epoch, the piece change rate is less than 1%, and by the end of training it has fallen to less than 0.04%. This suggests that on short timescales it is actually quite reasonable to model training of state of the art deep nets as being linear. I suspect the fidelity of such a model breaks down over longer time scales, but if our model of short term learning dynamics allows us to improve the short term behavior of our learning algorithms, I suspect improving the short term behavior at every time step will result in an improvement in the long term behavior, even if we are not able to specifically predict that behavior accurately.

For these reasons, I suspect that insights gained on the model family that this paper studies will lead to improvements in training of non-linear deep nets.