Title: Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
Authors: Levent Sagun
Leon Bottou
Yann LeCun
Venue: ICLR 2017
Keywords: Optimization
Deep learning

Abstract: We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.

TL;DR: The eigenvalues of the Hessian of loss functions in deep learning have two components: singular bulk at zero that depends on the over-parametrization, and the discrete part that depends on the data.

Review discussion:
1- In Section 3, why did you look at the eigenvalues of the Hessian at the end of the training and not at different points during the training which could tell us more about the optimization?
2- In Figure 7, one could guess that in more complex cases, the magnitude of the learned weights is higher and that is why the top eigenvalue grows significantly. Did you try normalizing the weight (either normalizing the weight vector or normalizing the weights of each hidden unit separately) before calculating the Hessian?
We looked at the eigenvalues at the end of training to see the kind of critical points the training finds.
In other words, when we follow a gradient based method to locate a point with the norm of the gradient zero, in an ideal case where all eigenvalues are non-zero, the question would be to look at the Hessian there to describe the nature of the critical point judging by the number of negative eigenvalues it has. It turns out, in the case of neural networks most of them are zero or very small. This leads to a drastically different geometry for the bottom of the landscape. We completely agree that the Hessian throughout the optimization is also important from the point of view of the training, however, we believe that this is a different question than just wondering about the shape of the bottom of the landscape. We will gladly work on this aspect, as well.
In Figure 7, the norm of the weights of the parameters are, in fact, very similar to each other. We believe that the change in the sizes of eigenvalues should depend on a quantity other than the norm of the weights.
However, it's a great idea to perform a controlled experiment in which the norm of the weights for separate layers are monitored to show that this is indeed not caused by the norm of the weights.
In Figure 1, is this at the end of training?
As parameters, are you counting the number of hidden units, or number of weights?
How are hidden units distributed across layers for MNIST? Are the numbers of hidden units given in the text and legend the number per layer, or the total number?
What is Figure 1 left showing? With 2 hidden layers and 10 total hidden units (i.e. 784->5->5->10), I would have thought the Hessian would be of size (784*5+5*5+5*10)^2 which is much greater than 1600. In general, I'm struggling to see how Fig 1 left can be showing the full Hessian—even if this were linear regression off the 784-dimensional input, the Hessian would be of size 784^2.
What additional work is related to this paper?
- Yes, Figure 1 is at the end of training.
- We are counting the number of weights.
- They are per layer, but the ones in figure 1 have one hidden layer, not two, we apologize for the confusion. If the number of hidden units is k then the number of eigenvalues in figure 1 (right) is (784 + 1)*k + (k + 1)*k + (k + 1)*10
- The matrix in figure 1 (left) is much smaller due to the initial space constraints, it's when k = 2, so there are (784 + 1)*2 + (2 + 1)*2 + (2 + 1)*10 = 1606 in each axis, with a total of 1606^2 values. It would be a good idea to make this bigger by saving the entries of the Hessian separately for each column.
- The geometry of the bottom has been explored in another ICLR submission at https://arxiv.org/abs/1611.01540 in which the authors find paths between solutions which is intimately related to our work through the flatness of the landsacape at the bottom.
Another work that makes use of the ideas developed here is https://arxiv.org/abs/1611.01838.
Even though the main goal of that work is very different (convolve the loss function to obtain a smoother one with nicer training properties), the degenerate structure seems to have been inspirational.
What is the number of hidden units of the networks associated to the number of parameters of each system?
For the histograms, how is the range of the x-axis chosen?
In Figure 4: Were the histograms computed after convergence? If so, were the models regularized?
- Referring to the previous comment: If the number of hidden units is k then the number of eigenvalues in figure 1 (right) is (784 + 1)*k + (k + 1)*k + (k + 1)*10
- The x-axis is the numerical values for the eigenvalues, the range is linear and includes all the eigenvalues so there are no further eigenvalues above or below the limits in the histograms.
- Figure 4 is after convergence for all 5 systems, and the models are *not* regularized.
(We will add clarifications for all of these points in the text)
Studying the Hessian in deep learning, the experiments in this paper suggest that the eigenvalue distribution is concentrated around zero and the non zero eigenvalues are related to the complexity of the input data. I find most of the discussions and experiments to be interesting and insightful.
However, the current paper could be significantly improved.
Quality:
It seems that the arguments in the paper could be enhanced by more effort and more comprehensive experiments. Performing some of the experiments discussed in the conclusion could certainly help a lot. Some other suggestions:
1- It would be very helpful to add other plots showing the distribution of eigenvalues for some other machine learning method for the purpose of comparison to deep learning.
2- There are some issues about the scaling of the weights and it make sense to normalize the weights each time before calculating the Hessian otherwise the result might be misleading.
3- It might worth trying to find a quantity that measures the singularity of Hessian because it is difficult to visually conclude something from the plots.
4- Adding some plots for the Hessian during the optimization is definitely needed because we mostly care about the Hessian during the optimization not after the convergence.
Clarity:
1- There is no reference to figures in the main text which makes it confusing for the reading to know the context for each figure.
For example, when looking at Figure 1, it is not clear that the Hessian is calculated at the beginning of optimization or after convergence.
2- The texts in the figures are very small and hard to read.
Thank you for the comments and the review. Quantification of singularity is crucial but it may be tricky and even misleading given the degenerate structure, therefore we thought it would be equally important to lay out the observations first, and then as a separate work consider the quantification.
We revised the paper in an attempt to address most of the issues mentioned above and in other comments.
Also, we will add a comparison with another ML method, very soon.
The paper analyzes the properties of the Hessian of the training objective for various neural networks and data distributions. The authors study in particular, the eigenspectrum of the Hessian, which relates to the difficulty and the local convexity of the optimization problem.
While there are several interesting insights discussed in this paper such as the local flatness of the objective function, as well as the study of the relation between data distribution and Hessian, a somewhat lacking aspect of the paper is that most described effects are presented as general, while tested only in a specific setting, without control experiments, or mathematical analysis.
For example, regarding the concentration of eigenvalues to zero in Figure 6, it is unclear whether the concentration effect is really caused by training (e.g. increasing insensitivity to local perturbations), or the consequence of a specific choice of scale for the initial parameters.
In Figure 8, the complexity of the data is not defined. It is not clear whether two fully overlapping distributions (the Hessian would then become zero?) is considered as complex or simple data.
Some of the plots legends (Fig. 1 and 2) and labels are unreadable in printed format. Plots of Figure 3 don't have the same range for the x-axis. The image of Hessian matrix of Figure 1 does not render properly in printed format.
We realized that the text hasn't been clear in the experiments that we performed. We clarified the text.
Figure 1 and figure 2 (left) show the beginning and the end of the model with 10-hidden units which is consistent with figure 6. In both cases of the simple data and MNIST the initial point is chosen randomly on the surface of a sphere centered at zero with fixed  radius (the radius is depends on the number of hidden units).
The complexity of data can be tricky to describe, a data can be more complex for a certain model but less so for another one. Thank you for pointing this out, we clarified this point on the data complexity by the ease of separability in the text.
We also updated the axes and labels in fig 1 and 2. We changed the rendering format for the Hessian matrix increasing the sharpness and resolution. For figure 4 we would like to emphesize the two components of the spectrum, therefore we picked the widest possible representation for each case separately.
However, we would also like to note that another series of experiments will explore the scale of eigenvalues depending on the iteration number during the training.
This paper investigates the hessian of small deep networks near the end of training. The main result is that many eigenvalues are approximately zero, such that the Hessian is highly singular, which means that a wide amount of theory does not apply.
The overall point that deep learning algorithms are singular, and that this undercuts many theoretical results, is important but it has already been made: Watanabe. “Almost All Learning Machines are Singular”, FOCI 2007. This is one paper in a growing body of work investigating this phenomenon. In general, the references for this paper could be fleshed out much further—a variety of prior work has examined the Hessian in deep learning, e.g., Dauphin et al. “Identifying and attacking the saddle point problem in high dimensional non-convex optimization” NIPS 2014 or the work of Amari and others.
Experimentally, it is hard to tell how results from the small sized networks considered here might translate to much larger networks. It seems likely that the behavior for much larger networks would be different. A reason for optimism, though, is the fact that a clear bulk/outlier behavior emerges even in these networks. Characterizing this behavior for simple systems is valuable.
Overall, the results feel preliminary but likely to be of interest when further fleshed out.
This paper is attacking an important problem, but should do a better job situating itself in the related literature and undertaking experiments of sufficient size to reveal large-scale behavior relevant to practice.
Thank you very much for pointing out at the FOCI 2007 paper, it is certainly relevant, even though the main object is the Fisher information matrix rather than the Hessian of the cost function.
Dauphin et al. is interested only in saddle points near the path of training. These two works are indeed remotely related to the line of research that our work is invested in.
However, their objective and results are different.
We have added more references and clarified our contributions.
In light of the constructive feedback, we revised the paper with multiple edits and figure revisions. We also included a new experiment showing that the phenomena that we observe is not specific to the log-loss, that it holds in the case of MSE loss, as well.
We also noticed that the title can be misleading in that it may suggest that our work's focus is on the singularity of Hessian only.
Indeed, the good part of the work is related to the singularity, however, we have a second main message regarding the eigenvalue spectrum: the discrete part depends on data. We revised the title and parts of the body of the text to emphasize this point.

Meta-review:
role: PC
data: comment: This is quite an important topic to understand, and I think the spectrum of the Hessian in deep learning deserves more attention. However, all 3 official reviewers (and the public reviewer) comment that the paper needs more work. In particular, there are some concerns that the experiments are too preliminary/controlled and about whether the algorithm has actually converged. One reviewer also comments that the work is lacking a key insight/conclusion. I like the topic of the paper and would encourage the authors to pursue it more deeply, but at this time all reviewers have recommended rejection.

Decision: Reject