Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper derive a continuous model for SGD in linear regression settings to build understanding on some interesting phenomenon of deep neural network trained with SGD. 
main_review: **Strength**

1. The theoretical results are solid and well-motivated;
2. The experiments are thorough, and consistent with theoretical results;
3. The contribution of this work are carefully clarified.

**Weakness**

The only pity is that all the theoretical results are derived from a linear regression model. The impact of this work will significantly increase if these interesting theoretical results can be derived in more general cases.

**Further concerns**

I notice some disruptive claims are raised in the last paragraph. Please can you elaborate them in order to raise my score? The following two concern me the most:
- “the network stays within a local region” is wrong;
-  "Finally, the intuition that faster learning rates would lead to faster global displacement in parameter space is also wrong; instead induced velocity anti-correlations lead to slower global displacement"
summary_of_the_review: In this paper, the insight from the theoretical results can match the empirical observations pretty well. Their results certainly deepens our understanding on the learning dynamics of modern neural network.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The starting point of the present paper is the observation that the *parameters*
of a neural network continue to change when training the network with stochastic
gradient descent (SGD) even after the performance of the network has stabilised
at a final value. The authors study these "limiting dynamics" in linear
regression, which they model as an underdamped Langevin equation resulting in an
Ornstein-Uhlenbeck (OU) process and find an oscillatory dynamics in the
(position, velocity)-phase space. The authors discuss how to connect their
findings in this simple model with the dynamics of Resnet18 (see
below). Finally, they attempt to predict features of the diffusive behaviour of
the ResNet18 from their model (Sec. 8)
main_review: I welcome the general direction of the work: trying to understand the dynamics
of neural networks is a complex undertaking that a large community of
researchers is pursuing, so the study of simplified models is a promising
avenue.

### Setup of the study

In studying the *limiting* dynamics however, the authors limit themselves to a
setup where I don't see any immediate connections to learning or representations
of neural networks. The dynamics the authors describe happen *after* resuming
training of a pre-trained neural network, thus I feel like their setup restricts
the potential impact of the results of this study.

### Clarity / novelty of the results

I found the article hard to read at times because the authors repeatedly qualify
their observations as "surprising", "contrary to common intuition",
"nonintuitive", etc. I think these qualifiers can be mistaken as claims of
novelty etc. and would hence use them more sparingly. For example, the fact that
neural networks continue to move through their weight space has been
well-established for quite some time now, cf. for example [Jastrzebski et
al. '17, Chaudhari & Soatto, 18, Baity-Jesi et al. '18, and many more]. Hence I
didn't find the observation in Figure 1 "surprising" (p 2 after the equation),
which underlines the subjectiveness of these claims - or else I maybe reading
the figure incorrectly?

Another example: I would consider it well-established that OU processes whose
diffusion matrix is not isotropic do not follow the naïve Gibbs distribution,
but instead equilibrate in a modified potential (see for example Section 5.3
"Potential conditions" of Gardiner's "Handbook of stochastic methods" etc.)
Furthermore, modified losses arising through SGD dynamics have been studied in a
number of recent deep learning papers, some of which are cited by the authors 

Other claims about the significance of the results should equally be clarified
in my opinion, for example:

> The expectation that the training trajectory would reflect the underlying 
> anisotropy of the training loss driving the dynamics is also wrong;" (p. 9)

In my understanding, this study is only concerned with the *limiting* dynamics
of learning, and hence conclusions about the training cannot be drawn
immediately?

### Separation of experimental from theoretical results

The authors should be lauded for trying to connect their theoretical results
with empirical results on deep networks. Again though, I think the presentation
of the results should be revised to clarify which predictions are actually
derived from theory.  Take the exponent of anomalous diffusion (bottom of Fig
6): it cannot be estimated of the global displacement (12), as the authors
explain. Instead, the authors evaluate the dependence of the diffusion constant
on learning rate, batch size and momentum parameter directly from a simulation
by fitting a power-law to the empirical displacement. I would present this
result separately, as it is not a theoretical prediction, and thus presenting it
in a section entitled "Predicting the diffusive behaviour of the limiting
dynamics" could cause confusion in my opinion.

summary_of_the_review: I welcome the direction of the present study, trying to understand the dynamics
of deep neural networks via simpler models. However, I cannot recommend
acceptance of the present paper for three reasons: the implications of the
studied dynamics on learning or representations in neural network remain unclear
to me; the results derived from the Ornstein-Uhlenbeck process do not go beyond
the previous work to an extent that would warrant the publication at ICLR in my
opinion; and the unclear presentation of the results which doesn't always
clearly separate previous work from the presented results, or theoretical
predictions from numerical measurements (Fig. 6).

**Edit Nov 14th** I thank the authors for the lengthy replies to the points raised in this review. After careful consideration of their comments, and also taking into account the valuable points made by other referees, I keep my original score for now. I have increased my confidence from 4 to 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the long-time dynamics of deep neural networks. The author(s) (1) show some empirical findings related to the mean square displacement, (2) model SGD as an underdamped Langevin Equation, relate it to an Ornstein Uhlenbeck process in a linear regression setting, and use it to study the limiting dynamics of SGD, (3) use the Fokker-Planck formalism to show that the steady state weight distribution obeys a modified loss which is isotropic in the absence of L2-like regularization, (4) provide empirical evidence that their findings are relevant also beyond the context of linear regression.
According to the authors, these are the novelties provided by the paper:

(a) "We find empirically that long after performance has converged, networks continue to move through parameter space by a process of anomalous diffusion in which distance travelled grows as a power
law in the number of gradient updates with a nontrivial exponent."

(b) "We reveal an intricate interaction between the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix at the end of training that explains this anomalous diffusion."

(c) "we show that the key ingredient driving these dynamics is not the original training loss, but rather the combination of a modified loss, which implicitly regularizes the velocity, and probability currents, which cause oscillations in phase space"

(d) "We identify qualitative and quantitative predictions of this theory in the dynamics of a ResNet-18 model trained on ImageNet."

(e) "We uncover a mechanistic origin for the anomalous limiting dynamics of deep neural networks trained with SGD"

(f) "one of the most significant results of our analysis is that, depending on the relationship between the gradient noise covariance and the Hessian, the stationary distribution in phase space will generically violate detailed balance"

(g) 'Natural intuitions, such as “the network converges in parameter space” or “the network stays within a local region”, are wrong'

main_review: The paper is well written; a very pleasant reading. I find interesting the characterization of diffusion and displacement as a function of hyperparameters. Although it is not new that SGD obeys a modified loss, I find it insightful to see that this modified loss can be isotropic despite the loss being anisotropy, and to learn under which conditions this occurs. I also appreciate the clarity of the assumptions that are needed to obtain every single result, while maintaining the discussion at an intuitive level.


Major Criticisms.

- Result (a) is not new. The anomalous diffusion at the end of training was already shown in arXiv:1803.06969, exactly in the terms defined in this manuscript ("distance travelled grows as a power law in the number of gradient updates with a nontrivial exponent"). 

- Result (c). That SGD obeys a modified loss was also stated in previous literature, such as Ref.[21] or, in a different framework, arxiv:1803.01927.

- The title is very bold, inviting a rethinking of the limiting dynamics. I would say that this is because the authors assume that the deep learning community thinks that [statement (g) up here] “the network converges in parameter space”, or “the network stays within a local region”. These are however very naive intuitions that nobody with sufficient experience in deep learning theory would believe. Also, one of their main claims is [statement (f)] that detailed balance does not hold, implicitly suggesting that the results of e.g. Refs.[16,23] could be flawed. Again, nobody believes that detailed balance holds for SGD dynamics -not even at late stages of training-, since detailed balance is a sufficient condition (not even necessary, see e.g. https://aip.scitation.org/doi/full/10.1063/1.4863991) for equiibrium, but steady state distributions generally do not satisfy it. There is a whole section in Ref.[16] devoted to showing that SGD is out of equilibrium. Another example is arXiv:1803.06969, where they state that at the end of learning the system diffuses at the bottom of the landscape with a time-dependent diffusion constant (so this is clearly not an equilibrium process, and the network does not stay confined within a local region). In other words, I don't think that statements (f) and (g) add anything new to the current knowledge.

- The analysis of the limiting cycles is nice, but I do not see what we learn more than was already presented in Ref.[21]. The only novelty would be that the paths at the end of learning are not limit cycles, but rather space filling curves. However, in the way as it is presented, this looks more like a conjecture than an actual result. As for the description in terms of velocities, this is nice, but I don't find it surprising that the velocities oscillate, especially in the directions of the top 30 eigenvectors. A system confined by quadratic walls will have an oscillating velocity, as any harmonic oscillator. I would find it more surprising if the authors measured the same kind of oscillation in the direction of the bottom eigenvectors.


Minor comments:

- The authors project the limiting trajectory for the parameters onto the plane spanned by the top q 1 and bottom q 30 eigenvectors. I assume that  q1 and q30 were chosen in order to maximize the anisotropy on the related plane, in order to show that the trajectory is instead isotropic. I suggest to explain that this is the reason of the choice, since at first it wasn't clear to me why this was being done.

- Fig.3. It's nice to see that the trajectory is isotropic when no regularization is used. But why is the trajectory not centered around the minimum of the modified loss? Also, it looks like the minimum of test and modified losses coincide. Why is this?

- typo in section 8: trigonomentric

- Implicit regularization of the velocity trajectory. I assume that the authors mean that the velocity appears in the modified loss as an L2 regularization term. Is this right? Is there any further consequence to this?

- In figure 6 - bottom, the dots are measured from fits of the trajectory. The dashed line is c=1 diffusion, but this is a little confusing, because the dashed line in the top three plots is the theoretical prediction of Eq11. 
summary_of_the_review: The paper presents several concepts in a very clear and readable manner. However, the novelties presented are limited/incremental (certainly not as revolutionary as suggested by the title), and some of the results are either not new (anomalous diffusion in DNNs has been known for years now), not relevant (the lack of detailed balance in results relying on a stationary state distribution), or not well-supported (space filling curves instead of limit cycles). On the other hand, I do find some value in some of the results, such as the characterization of diffusion and displacement as a function of hyperparameters, and showing that under certain conditions the modified loss is isotropic.

The authors should reduce their claims regarding what is new in their work and adapt them to what is the real state of the art of the understanding of DNNs, and remove disputable statements such as "The major limitation of the stationary dynamics approach to analyzing SGD is its implicit assumption that the system is in detailed balance".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper uses a stochastic differential equation approximation of stochastic gradient descent momentum to model the neural networks dynamics. At the theoretical level the study focuses on linear regression. They showed that there is agreement between the results in the simpler model and deep neural networks. 
The most relevant fact is that the networks can have a super or sub diffusive behavior at the end of training. Unfortunately, their theory cannot explain this finding nor the practical consequences on the test loss.
main_review: In the current format, the paper suffers from several limitation that should be addressed by the authors. I believe that there are interesting results and I encourage the authors to answer my comments as I would be very happy to rise my recommendation score. 
The following comments try to follow the section in the submission:

 


The authors in their analysis of related works miss a large portion of the relevant literature. Discussing the analysis of the dynamics of neural networks: 
* The mean-field limit was introduced by three different independent works [Rotskoff, Vanden-Eijnden 2018] [Chizat, Bach 2018] [Song et al. 2018], only the ladder is cited.
* If the mean-field limit is mentioned then also the dynamical mean field theory approach should be cited. In the context of neural networks we have: [Mignacco et al. 2020][Mannelli et al. 2020][Mignacco et al. 2021].
* The dynamics of stochastic gradient descent and momentum were also obtained in dynamical mean field theory by the two papers by Mignacco above and [Mannelli, Urbani 2021].
* When mentioning the modelling of SGD using SDE, it is important to discuss the results of [Simsekli et al. 2019] where they showed that the jumps are fat-tail distributed and how that is compatible with the approximation.
* On the "empirical exploration" the authors refer only to [Papyan 2018], the results of [Ghorbani, Krishnan, Xiao 2019] that derived the same method independently and make thorough analysis of the dynamics.  In the same section also [Sagun et al. 2016] (ref [35]) should appear. 

On page 2, the authors say "Surprisingly, [..] the networks continue to move through parameter space" but this very well known. It known that stochastic gradient descent keeps moving after training, indeed this is at the basis of ref. [16]. What is not known is how is moving, and indeed this finding is interesting.

In figure 1 lower panel, there is only one value on x-ax. Please add at least a second one. I can guess that the next number after 10^4 would be 2*10^4 but this is not obvious a priori.

The authors should try to avoid to add adjectives to embellish the paper in the technical sections. These parts of the paper should be factual and reporting their results in a scientific way. 
For instance in section 8, describing figure 5 the authors say "As predicted, the spiral appears more evident ..", can you quantify how "spiral" the plot is? It is hard to conclude from that picture. Furthermore, that appears to be the result of one run. It should be good to average over many simulations to avoid the risk of observing random fluctuations.
The fact that the second figure appears "less evident" may indicate a limit in the theory. Since the authors are considering a SDE, higher order effects should appear more frequently for eigenvectors corresponding to smaller eigenvalues. 
It would be interesting to see what happens to the 1000th eigenvectors. Since ImageNet has 1000 classes, according to [Sagun et al. 2016][Papyan 2019] that eigenvector should be associated to irrelevant information and probably this behavior disappear. You could observe this also if you have the pretrained network in CIFAR100 or CIFAR10, by looking at the 100th and 10th eigenvector respectively.

An important aspect, that has not been discuss, is the effect on generalization. What are the practical consequences of these results? Although the theoretical framework can not be applied to the test dataset, the authors can verify the practical effects. 
Is being super/sub diffusive an advantage? Does the performance change? Answering these questions will bring value to the work.

The authors affirm that the dynamics brake detailed balance. Unfortunately I must have missed this part in the paper. Could they clarify?
Also, what are the practical implications of braking detailed balance?

In figure 6 the authors claim that their theory predicts the behavior of the displacement. It is not clear to me the procedure used. When you say "using this single estimate" what do you mean? Are evaluating \sigma^2 tr(H) in a single point for each plot and using this information in draw the line? It is not very clear.
On the second line, it is an overstatement to say that you predict the diffusion constant. Your model, by construction, is a correlated Brownian  motion with a drift, therefore the diffusion constant is 1. 
This is a pity, there is no explanation for the very interesting behavior the you observe.


summary_of_the_review: The paper presents an interesting result about the presence of sub/super diffusive behaviour at the end of training. Although the result may be of limited practical interest, it is a new finding and would be interesting to understand causes and consequences of that (not discussed in the paper). 
The paper has a large number of flaws that I believe can be amended to get a publishable result.
review_title: null

Meta-review: The authors study the limiting dynamics of a simple linear regression model. They use an underdamped Langevin equation which is quite common in the literature. Although the reviewers welcome the direction and the attempt to understand the dynamics of a simple model, the novelty of the paper is limited. As an example, the paper shows that the key ingredient driving these dynamics is not the original training loss, but a modified loss. As pointed out by the reviewers, this has already been pointed out in multiple papers. One important problem is the tendency of the paper to oversell the results (including the title), which makes it difficult to clearly separate the contributions made in the paper. After a discussion with the reviewers, the overall feeling did not change.

I can therefore not recommend acceptance. I strongly recommend the authors do a significant rewrite of the paper in order to clearly separate what contributions are truly novel and also improve the discussion of prior work.
Decision: Reject