Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Authors: Andrew Saxe, James L. McClelland, Surya Ganguli
ICLR 2014

Abstract: Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, thereby providing analytical insight into the success of unsupervised pretraining in deep supervised learning tasks.


Peer review:
[Review]:
In this paper, the authors try to analyze theoretically the dynamics of learning in deep neural networks. They consider the quite restricted case of deep linear neural networks. Such linear deep networks are not of practical interest, because the mapping y = Wx between the input vector x and output vector y in them can always be realized using a single weight matrix W which is the product of weight matrices of different layers, and therefore adding hidden layers does not improve their performance.

As the performance criterion the authors use the standard mean-square error between the output vectors of the networks and their target values (desired responses), which is in practice approximated by the respective squared error over all the training pairs. The gradient descent method applied to this criterion provides the batch learning rule (1) used in the paper. It should be noted that the authors' learning rule is not the batch version of the standard backpropagation algorithm. This is because already in the case of a single hidden layer it depends on the two weight matrices between the input and hidden layer, and between the hidden and output layer.

Despite the linearity of the studied deep networks, this learning rule has nonlinear dynamics on weights that change with the addition of each new hidden layer. The authors show by simplifying further their analyses that such deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initialization.

It is good that the authors know and refer to the paper by Baldi and Hornik from the year 1989, reference [19]. In this paper Baldi and Hornik showed that in a linear feedforward (multilayer perceptron) network with one hidden layer the optimal solution minimizing the mean-square error is given by PCA, more accurately by the subspace spanned by the principal eigenvectors of the correlation matrix of the input data, coincides with the data covariance matrix for zero mean data. However, Baldi and Hornik did not analyze the learning dynamics.

The paper suffers from lack of space. It extends to 10th page and with supplementary material to 15th page. Obviously because of space limitations figures are too small and there is no Section 1. Introduction, only plain text after the abstract.

Pros and cons:
--------------

+ The analysis on learning dynamics is novel and interesting. I am not aware of this kind of analyses on deep networks before this paper.
+ With their mathematical analyses, the authors are able to explain some phenomena observed experimentally in learning deep networks.
- The deep network analyzed is linear, with no practical interest.
- The learning algorithm(s) that the authors study are not used in practice.