user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradient Descent Happens in a Tiny Subspace

Paper content:
We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training.
The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training.
A simple argument then suggests that gradient descent may happen mostly in this subspace.
We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.
Stochastic gradient descent (SGD) BID14 and its variants are used to train nearly every large-scale machine learning model.
Its ubiquity in deep learning is connected to the efficiency at which gradients can be computed BID15 BID16 , though its success remains somewhat of a mystery due to the highly nonlinear and nonconvex nature of typical deep learning loss landscapes .
In an attempt to shed light on this question, this paper investigates the dynamics of the gradient and the Hessian matrix during SGD.In a common deep learning scenario, models contain many more tunable parameters than training samples.
In such "overparameterized" models, one expects generically that the loss landscape should have many flat directions: directions in parameter space in which the loss changes by very little or not at all (we will use "flat" colloquially to also mean approximately flat).1
Intuitively, this may occur because the overparameterization leads to a large redundancy in configurations that realize the same decrease in the loss after a gradient descent update.One local way of measuring the flatness of the loss function involves the Hessian.
Small or zero eigenvalues in the spectrum of the Hessian are an indication of flat directions BID10 .
In , the spectrum of the Hessian for deep learning crossentropy losses was analyzed in depth.2
These works showed empirically that along the optimization trajectory the spectrum separates into two components: a bulk component with many small eigenvalues, and a top component of much larger positive eigenvalues.
3 Correspondingly, at each point in parameter space the tangent space has two orthogonal components, which we will call the bulk subspace and the top subspace.
The dimension of the top subspace is k, the number of classes in the classification objective.
This result indicates the presence of many flat directions, which is consistent with the general expectation above.In this work we present two novel observations:• First, the gradient of the loss during training quickly moves to lie within the top subspace of the Hessian.
4 Within this subspace the gradient seems to have no special properties; its direction appears random with respect to the eigenvector basis.•
Second, the top Hessian eigenvectors evolve nontrivially but tend not to mix with the bulk eigenvectors, even over hundreds of training steps or more. In
other words, the top subspace is approximately preserved over long periods of training.These observations are borne out across model architectures, including fully connected networks, convolutional networks, and ResNet-18, and data sets FIG1 , TAB0 , Appendices C-D).Taken
all together, despite the large number of training examples and even larger number of parameters in deep-learning models, these results seem to imply that learning may happen in a tiny, slowly-evolving subspace. Indeed
, consider a gradient descent step −ηg where η is the learning rate and g the gradient. The change
in the loss to leading order in η is δL = −η g 2 . Now, let g
top be the projection of g onto the top subspace of the Hessian. If the gradient
is mostly contained within this subspace, then doing gradient descent with g top instead of g will yield a similar decrease in the loss, assuming the linear approximation is valid. Therefore, we think
this may have bearing on the question of how gradient descent can traverse such a nonlinear and nonconvex landscape.To shed light on this mechanism more directly, we also present a toy model of softmax regression trained on a mixture of Gaussians that displays all of the effects observed in the full deep-learning scenarios. This isn't meant as
a definitive explanation, but rather an illustrative example in which we can understand these phenomenon directly. In this model, we can
solve the gradient descent equations exactly in a limit where the Gaussians have zero variance. 5 We find that the gradient
is concentrated in the top Hessian subspace, while the bulk subspace has all zero eigenvalues. We then argue and use empirical
simulations to show that including a small amount of variance will not change these conclusions, even though the bulk subspace will now contain non-zero eigenvalues.Finally, we conclude by discussing some consequences of these observations for learning and optimization, leaving the study of improving current methods based on these ideas for future work.
We have seen that quite generally across architectures, training methods, and tasks, that during the course of training the Hessian splits into two slowly varying subspaces, and that the gradient lives in the subspace spanned by the k eigenvectors with largest eigenvalues (where k is the number of classes).
The fact that learning appears to concentrate in such a small subspace with all positive Hessian eigenvalues might be a partial explanation for why deep networks train so well despite having a nonconvex loss function.
The gradient essentially lives in a convex subspace, and perhaps that lets one extend the associated guarantees to regimes in which they otherwise wouldn't apply.An essential question of future study concerns further investigation of the nature of this nearly preserved subspace.
From Section 3, we understand, at least in certain examples, why the spectrum splits into two blocks as was first discovered by .
However, we would like to further understand the hierarchy of the eigenvalues in the top subspace and how the top subspace mixes with itself in deep learning examples.
We'd also like to investigate more directly the different eigenvectors in this subspace and see whether they have any transparent meaning, with an eye towards possible relevance for feature extraction.Central to our claim about learning happening in the top subspace was the fact the decrease in the loss was predominantly due to the projection of the gradient onto this subspace.
Of course, one could explicitly make this projection onto g top and use that to update the parameters.
By the argument given in the introduction, the loss on the current iteration will decrease by almost the same amount if the linear approximation holds.
However, updating with g top has a nonlinear effect on the dynamics and may, for example, alter the spectrum or cause the top subspace to unfreeze.
Further study of this is warranted.Similarly, given the nontrivial relationship between the Hessian and the gradient, a natural question is whether there are any practical applications for second-order optimization methods (see BID7 for a review).
Much of this will be the subject of future research, but we will conclude by making a few preliminary comments here.An obvious place to start is with Newton's method BID7 .
Newton's method consists of the parameter update DISPLAYFORM0 .
There are a few traditional criticisms of Newton's method.
The most practical is that for models as large as typical deep networks, computation of the inverse of the highly-singular Hessian acting on the gradient is infeasible.
Even if one could represent the matrix, the fact that the Hessian is so ill-conditioned makes inverting it not well-defined.
A second criticism of Newton's method is that it does not strictly descend, but rather moves towards critical points, whether they are minima, maxima, or saddles .
These objections have apparent simple resolutions given our results.
Since the gradient predominantly lives in a tiny nearly-fixed top subspace, this suggests a natural low rank approximation to Newton's method DISPLAYFORM1 top .Inverting
the Hessian in the top subspace is well-defined and computationally simple. Furthermore
, the top subspace of the Hessian has strictly positive eigenvalues, indicating that this approximation to Newton's method will descend rather then climb. Of course,
Newton's method is not the only second-order path towards optima, and similar statements apply to other methods.

assistant: text: For classification problems with k classes, we show that the gradient tends to live in a tiny, slowly-evolving subspace spanned by the eigenvectors corresponding to the k-largest eigenvalues of the Hessian.