Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper derives a counter term to the gradient flow ODE formulation that reduces the discretization error from Euler's method, which is gradient descent. When this correction term is expanded as a Taylor series, adding a select number of terms reduces the discretization order accordingly. This is then used to analyze the behavior of GD under symmetry constraints, specifically scale- and translation-invariant parameters. Specifically, this adds learning-rate-dependent correction terms to the decay rates of certain quantities, which matches gradient descent in practice.
strengths_and_weaknesses: Pros:
  - Quite an interesting take which attempts to correct the theoretical ODE formulation in order to match practice (as opposed to bridging this gap by using higher order solvers, for example). 
  - The motivation, process, and theoretical results are presented very well. I could follow understand every result (just the results; not the proofs) despite not being an expert in the theory of gradient descent.

Cons:
  - I imagine the theory (of estimating discretization error of ODEs) has been done before, though perhaps not in this exact context. The zero-th order term (Eq 9) has shown up in machine learning studies before.
  - It's not clear to me if we gain anything from using the higher order terms in the derivation, as the analysis and experimental results all use only the zero-th order term. 
questions: Can you clarify the connection between this correction term and the global truncation error of Euler's method (i.e. a bound on e_k)? I expected them to look very similar but seems to not be the case.

Depending on the step size, Euler's method can result in inconsistencies where the discretization completely diverges from the gradient flow, in particular when the step size is much larger than the Lipschitz constant of g, and results in a discretization that cannot be meaningfully modeled by any ODE. I imagine this would correspond to the correction term becoming infinite and the zero-th order approximation to the correction term becoming somewhat meaningless. Does the equation of motion reflect this kind of behavior in any way?

Can the correction term be used to correct the drift of the SDE formulation to stochastic gradient descent? 

Can Corollary 4.2 be experimentally verified?
limitations: .
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors derive an Equation of Motion, i.e., a continuous differential equation that matches the discrete time dynamics of gradient descent more clearly. They do so by adding a counter terms to Gradient Flow that cancels out higher order discretization errors in DNNs, and this counter term is derived using backward error analysis, more precisely it is the solution to Equation 6 in the paper. 

Given they are using backward error analysis, they are also able to quantify the discretization error for GD approximation of GF along with the counter term, and hence also provide a bound on the learning rate such that this discretization error is small. 

The authors apply their Equation of Motion to translation and scale invariant layers and show that their theoretical predictions better match GD.
strengths_and_weaknesses: My main concern is that this paper does not provide any interesting new result. The main novelty of the paper is the general form of the counter term (which is derived in Theorem 3.3), as opposed to previous work for example Barrett and Dherin (Implicit gradient regularization) which only uses the first order term as their regularization term. 

While the authors mention that they derive the discretization error (in Corollary 4.1), the precise formulation is not provided and the rate is provided as an upper bound (using Big-OH) which I believe is an artifact of standard series expansion results. 

Furthermore, for most of the results on the discretization error bounds and the upper bound on the learning rate, the authors assume that the counter term is either equal to zero or assume the first order counter term (i.e., the term in equation 9).

With the counter term equal to 0, the analysis matches a lot of the previous work (example Elkabetz and Cohen, [18]) and with the first order term the main theoretical results are very similar to the ones already established in Barrett and Dherin (who introduce the first order counter term as the implicit regularizer and the error is discussed in Theorem 3.1)

Besides this, the authors do apply their analysis to characterize the learning dynamics of scale and translation invariant layers and show that with the inclusion of the first order (adding higher order counter term is going to be computationally expensive) they are able to better predict the decay of parameter norm, which is interesting but not that surprising.
questions: In the current state the results seem to be very similar to the previous work (as I have mentioned in the previous comment), and besides the series expansion in Theorem 3.3, it is very hard to adjudge the precise novelty of the work, esp since most of the results discussed in the next sections assumes that the counter term is either 0 or they assume the first order counter term. I would really appreciate if the authors could clarify their contribution. 

Another minor comment, I think it would be easier to read the paper (esp the proofs) if the author provided a sketch after the main statement in the main paper as well as in the appendix. 
limitations: Yes the authors have discussed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with the discrepancy between the actual discretized gradient descent and its continuous version, i.e., a gradient flow, for describing the equation of motion of learning dynamics more precisely. The discrepancy error is formally introduced by using the backward error in numerical analysis. The authors derive a counter term, which can compensate for such a discrepancy of the gradient flow, thus can describe the actual discretized trajectories in a continuous manner. While the derived counter term is a complicated functional integral equation, it can be analytically solved (for all orders) by assuming the underlying solution is a power series. As an application, the authors use the derived dynamics with the proposed counter term for investigating scaling- and translation-invariant layers.
strengths_and_weaknesses: [Note]

Because I am not an expert on learning theory, my evaluation might not be exhaustive. Also, I did not read the proofs in the supplementary material carefully.

[Strengths]

To me, the derived counter term is novel and seems to be useful to predict and interpret complicated learning dynamics of deep neural network models. Although there have been previous studies that incorporate some correction terms with respect to the backward analysis error, to my knowledge they are restricted to 1-st order compensation $\frac{1}{4} \nabla || \nabla f(\theta)||$, which is generally called an implicit gradient regularization. The proposed counter term is generalized for higher orders and can recover the previous studies well as in (9). While the main result (8) seems to be a known technique in the numerical analysis field, I would like to give appropriate credit to the authors for the contribution that introducing such a technique to the deep learning field well. 

[Weakenesses]

The authors address a full-batch gradient descent only. The authors also mention such a limitation of this work in Conclusion and Limitations. I am not sure whether the approach can be easily generalized for the mini-batch stochastic gradient descent method.

While the authors theoretically prove high-order corrections are required to cancel the leading order of discretization error, it will be great if the authors (1) experimentally show the discrepancy between the GF with the proposed correction and that with a first order correction, and (2) demonstrate the former can approximate GD well compared the latter, e.g., in Figure 2 or Figure 4.

The proposed method is not guaranteed for GD with a large learning rate, thus cannot be used for explaining some interesting phenomena, e.g., the regularization effect of an initial large learning rate. However, I think it is not a crucial drawback of this paper, considering the essential assumption of GF.

The paper is very dense and thus hard to read. A journal format might be more suitable for a clear representation of this work.




questions: As I mentioned previously, it will be nice if the authors can experimentally elaborate on the usefulness of the proposed generalized counter term compared to the first-order counter term.

Is there any idea that can tighten the learning rate bound (13) more efficiently?

limitations: The authors discuss the limitations of the proposed method, e.g.,  the lack of concerning the mini-batch stochastic GD and other optimizers beyond GD, in Conclusion and Limitations section. It will be nice if the authors also address the questions raised above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is concerned with a theoretical understanding of modelling the dynamics of gradient descent with a differential equation. Previous work (Gradient Flow) describes the differential equation as:

$\frac{d\theta}{dt} = -\nabla_{\theta} L(\theta)$

Which is the Euler discretisation of Gradient Descent:

$\theta_{t+1} = \theta_{t} - \eta \nabla_{\theta}L(\theta_t)$

However, discretisation error exists, such that Gradient Flow and Gradient Descent diverge. This paper derives a counter term to Gradient Flow, labelled by $\xi$:

$\frac{d\theta}{dt} = -\nabla_{\theta} L(\theta) - \eta\xi(\theta)$

The counter term is a functional integral, the paper approximates the counter term with a series solution in $\eta$, with a recursive relationship existing to get from term $k$ to term $k+1$, the new dynamics are called Equation of Motion (EoM). A limit for the learning rate is also derived, which allows accurate simulation of gradient descent using (EoM) with larger step sizes.

Finally these findings are tested on scale-invariant layers and translation-invariant layers and the results support the theoretical findings.
strengths_and_weaknesses: Strengths:

This is not an area I have significant expertise in, however, overall this paper is very good in my opinion. Specifically:

- The paper shows many impressive theoretical results
- The experiments are extensive and support the theory
- The motivation for the paper is very clear

Weaknesses:

Again, I believe the paper is very good. I think it presents good theoretical results, with sufficient experiment to support this. The only weaknesses overall are in the writing style and presentation, I think the paper is quite math heavy currently, which makes it less accessible. However, this is down to personal preference. For example lines 239-245 (Definitions) feel like quite a complicated way of saying most of the meanings. One definition is $\alpha_{\mathcal{A}} = \alpha I_\mathcal{A} + I_{\mathcal{A}^{C}}$, but it is easier in my opinion to say $\alpha_{\mathcal{A}}$ is $\alpha$ for the parameters in layer $\mathcal{A}$ and $1$ for the others. This can be extended for most of the definitions in this paragraph. I think the best way the paper can be improved is by having as much intuition as possible in the main text, with theorems included, and then possibly having more mathematical detail in the appendix.

Other small points are the presentation of results. Figure captions don't have **Figure n** in line with the caption, but over to the left which feels weird. Table 1 could also be improved, rather than listing the decay rates, it might be more informative to list the differences (and relative differences) in decay rates between GF & GD and EoM & GD.
questions: The few questions/clarifications I have are:

- Line 240 (Definitions), a symmetry is defined as a transformation of parameters such that the loss function is unchanged. Is this definitely correct? Should it not be a transformation that leaves the predictions unchanged?
- Is it possible to run similar experiments for layers that do not have any symmetry, such as a standard linear layer? This could make a nice piece of future work.
- It seems that the majority of experiments were run using the first term in the expansion of $\xi$. Is it possible to run an ablation where we see how adding the next term affects results, and on and on until say the 5th term (to justify using only the first term)? I appreciate this requires huge amounts of memory for the higher derivatives.
- Apart from the theoretical contributions (which are already fantastic), do you envision any practical applications of this work?
limitations: The authors have been upfront with the limitations of their work. These are given in the conclusion and provide a nice avenue for future research, they are about different optimizers and how using minibatches are not accounted for in the current work. I cannot think of any further limitations.
review_title: null

Meta-review: metareview: Reviewers were unanimous in recommending that the paper be accepted, and I accordingly recommend the same.  I encourage the authors to take into account suggestions made by reviewers so as to further improve the text in the camera-ready version.
confidence: Certain
Decision: Accept