Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis
Authors: Taiki Miyagawa
Venue: NeurIPS 2022 Accept (2022)
Keywords: Discretization error
Gradient flow
Gradient descent
Equation of motion

Abstract: We derive and solve an ``Equation of Motion'' (EoM) for deep neural networks (DNNs), a differential equation that precisely describes the discrete learning dynamics of DNNs. Differential equations are continuous but have played a prominent role even in the study of discrete optimization (gradient descent (GD) algorithms). However, there still exist gaps between differential equations and the actual learning dynamics of DNNs due to discretization error. In this paper, we start from gradient flow (GF) and derive a counter term that cancels the discretization error between GF and GD. As a result, we obtain EoM, a continuous differential equation that precisely describes the discrete learning dynamics of GD. We also derive discretization error to show to what extent EoM is precise. In addition, we apply EoM to two specific cases: scale- and translation-invariant layers. EoM highlights differences between continuous and discrete GD, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD. Our experimental results support our theoretical findings.

Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis
Taiki Miyagawa NEC Corporation, Japan mi miyagawataikknec.co com
Abstract
We derive and solve an "Equation of Motion" (EoM) for deep neural networks (DNNs), differential equation that precisely describes the discrete learning dynam- ics of DNNs. Differential equations are continuous but have played a prominent role even in the study of discrete optimization (gradient descent (GD) algorithms). However, there still exist gaps between differential equations and the actual learn- ing dynamics of DNNs due to discretization error. In this paper, we start from gradient flow (GF) and derive a counter term that cancels the discretization er- ror between GF and GD. As result, we obtain EoM, continuous differential equation that precisely describes the discrete learning dynamics of GD. We also derive discretization error show to what extent EoM is precise. In addition, we apply EoM to two specific cases: scale- and translation invariant layers. EoM highlights differences between continuous-timg and discrete- time GD, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD. Our experimental results support our theoretical findings.
Introduction
Let us first explain our primary motivation for the present paper. In physics, of the fun- damental goals is to predict the dynamics of
Weight parameter
8
matter and its fundamental constituents. Specif- ically, "predict" here means to construct differ- ential equations that best describe the physical system under consideration and to solve them. Such differential equations are called Equations of Motion (EoM). An interesting question here may "What works (DNNs)?" That is, to what extent can we predict the discrete learning dynamics of DNNs by constructing differential equations? This is our research question.
Learning dynamics
Gradient flow
EoM
Counter
EoM for deep neural net-
Gradient descent
Step &
Figure Our approach. GF fails describing the learning dynamics of GD due to discretization error. Our counter term approach successfully cancels the discretization error between GF and GD and hence allows for a reliable analysis of GD.
Differential equations have played prominent role studying discrete optimization (gradient descent (GD) algorithms), although they are con-
tinuous 2 4
9 10 [I [2 [
15] 17 19 20 In the context of deep
learning, gradient flow (GF) and stochastic differential equations (SDEs) are used to analyze (stochas-
tic) gradient descent (SS)D)) Research targets include: convergence 6
4 17]
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

stability of optimization 19, optimization with constraints [19, convergent states [0], flatness of loss landscapes [17. empirical risk bounds [15,, and online PCA [... Various techniques for continuous analysis have been imported to the analysis of discrete GD algorithms.
However, there still exist gaps between differential equations and actual learning dynamies due to discretization error, which is the main interest of the present paper and is often missing in the literature above. To be specific, we focus on GF (() = - -g((()) as a continuous approximation of GD 0x+1 = 0% ng ((()), where ((t) E Id and 02 E R are the weight parameters of a DNN at time t E IR and step k E E, respectively, and g E R is a gradient vector. 1) E R is a learning rate and is regarded as the discretization step size when GF is discretized with the Euler method 21: O( = kn) = 02+1-0: Due this approximation, discretization error (or "continuation error") introduced, and thus GF cannot fully explain the dynamics of GD. For instance, we show that according to GF, the weight norm of a scale-invariant layer collapses to zero when we use weight decay, while GD does not show such behavior (Section
"
To fill the critical gap between GF and GD, we propose modifying GF to describe the learning dynamics of GD more precisely; i... add counter term & E IR to the gradient g of GF that cancels the discretization error (Figure T]. This idea is motivated by backward error analysis in numerical analysis [21] We derive functional integral equation that determines the counter term and solve it (Section 3 As a result, we obtain a more reliable differential equation, called EoM here, that describes the discrete learning dynamics of GD. Using the counter term, derive the leading order of discretization error (Section 4.1) to show to what extent GF and EoM are precise in describing GD's dynamics. This point is often missed in the literature on the continuous approximation of discrete GD algorithms 122 2 24
26 27 28). We further derive a sufficient condition for learning rates for the discretization error to be small (Section 4.2). We show that EoM well explains empirical results.
Furthermore, to show the benefits of EoM. we apply it to two specific cases: scale-invariant layers 29 30 and translation-nnvariinn layers 31 32 (Section 53. For scale-invariant layers, we show that better description of GD's discrete dynamics requires modifications the decay rate of weight norms that is previously derived in the continuous regime (SDEs) 33 In addition, we show that
EoM successfully reproduces the limiting dynamics (f + 0o) of weight norms and angular update 34 that are previously derived in the discrete regime, while GF cannot reproduce this result. For translation-invarial layers, we show that BoM rather than GF dramatically matches empirical results, indicating the importance of the counter term. To the best of our knowledge, no study analyzes the temporal evolution of translation translationinvariiant layers except for [31 and (32), where only the sum of weights is their focus, while we derive the dynamics of the whole weights.
Our contribution is four-fold. Our code and detailed experimental results are given as supplementary materials.
1. To fill the critical gap between GF and GD, we derive a counter term for GF that cancels the discretization error, and as result, we obtain EoM, a continuous differential equation that precisely describes the discrete learning dynamics of GD.
2. To show to what extent GF and EoM are precise in describing discrete GD dynamics, we derive the leading order of discretization error, as is often missed in the literature on the continuous approximation of discrete GD algorithms. We further derive sufficient condition for learning rates for the discretization error to be small.
3. We apply EoM two specific cases: scale-invariant layers and translation-invariant layers, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD.
4. Our experimental results support our theoretical findings.
Our work is the first step toward answering this research question: to what extent can we predict the discrete learning dynamies of DNNs by constructing differential equations (EoM for DNNs)? Also, our work helps researchers import continuous analysis to the discrete analysis of GD algorithms. In this sense, our work bridges discrete and continuous analyses of GD algorithms.
'See Supplementary Materials at hatts:://ppaaeesseeeeee////rrrrs

---

2
Related Work
The idea of approximating discrete-time stochastic algorithms with continuous equations dates back to stochastic approximation theory [
2
3 4 S Their primary focus is convergence analysis for
discrete-time algorithms, while our focus to predict the learning dynamics (temporal evolution)
of weight parameters, such as the decay rates of weight norms and effective learning rate of scale-
invariant layers. Our idea of the counter term is inspired by the backward error analysis developed
for numerical analysis [35]. This idea is now used to analyze discrete optimization [22] [2 [24] [1]
25 26 27 28). T8 is a pioneering work on discretization error analysis between GF and GD that
is based on the numerical analysis of the Euler method
[21]. They derive a sufficient condition for
learning rates for the discretization error to be small. This analysis is based on a bound (inequality), while we derive an explicit relationship between learning rates and discretization error as an equality,
Neural mechanies and Noether's learning dynamics [1 32 provide solution to a part of the afore- mentioned problem: to what extent can we predict the learning dynamics of DNNs by constructing differential equations? They derive (the breaking of) conservation laws of weight parameters using differential equations and provide the temporal evolution of the conserved quantities. The present work inspired by these studies but has crucial differences: 1) our focus is on the temporal evolution of all of the network parameters, not only the conserved quantities, 2) the gradient's 's correction for canceling the discretization error is not limited the first order, but all orders, and 3) the discretization error is explicitly provided in the present paper. See Appendix for more related studies.
3 Equation of Motion for Deep Neural Networks
In the following sections, we define EoM by modifying GF (Section 3.1) We show that the counter term satisfies a functional integral equation (Section 3.2), and then we solve it (Section 3.3
3.1 Our Approach and Definitions
We begin with simple idea: add counter term to GF to cancel discretization error, i.e.,
0(1) = ((()) - ())))))),
where O(t) E Rd is the vectorized weight parameters of a DNN at time E R, E N is the dimension of the weight, and ((t) denotes dttt//dt. Gradient g()(t)) is defined as (((()) := ((((())) + A0(), which consists of loss function [((()) and weight decay term (l)).. where À > 0 controls the strength of weight decay.
0 is small learning rate, and ((O(t)) E R is the counter term. Throughout this paper, we assume all functions are sufficiently smooth. We call Equation T the Equation of Motion (EoM) for DNNs, or simply EoM.
Our aim to find & that makes Equation T more reliable to precisely approximate GD 0 0 ng 0z), where 0₂ E IR is the weight at step k E Zzo- To do so, we first define the discretization
error between GF (T) and GD at step k:
eg = (kr)) O2 E Rd
and find & that makes e₁ small. Throughout this paper, we the standard Euler method to discretize GF: (tt) = (0(t m) (t))/n and kn, thus, 17 is identified with the discretization step size.
3.2 How to Determine Counter Term
We show that the leading order of e with respect to 1) is controlled by the counter term (Theorem and as a result, the counter term is determined via functional integral equation (Equation (6)
3.2
Our first theorem shows what the counter term should cancel. Theorem 3.1 (Recursive formula for discretization error). Discretization error e satisfies:
e - ex = =g(((((())) - g(((k))) ex)) + 1? S/' dsō(m(k + +))(1(++))))++1 - 8) - ((()(())))
=: =- -(((())))))((((())) - -x))= + A(O(kn))
(4)
3

---

Here, we defined ((0(k))) := 17² So (s + s))(1 -) -))_(((((kk)) R.. The proof is based on Taylor's s theorem and is given in Appendix [A.1 The right-hand side of Equation (3) tells us that the counter term (third term) should cancel the first and second terms. However, the following theorem states that the first term gives only subleading contributions with respect to 1)-
Theorem 3.2 (Leading order of discretization error). Suppose that ((0(kn)) O(n") and eo O for some y Therefore, the first term in the right-hand side of Equation 3 negligible compared with A.
Then es = O(n²) and (n(g(0((n)) - g(((k)) - -ex)) = (
ck+1 = c + ((0))))) - m(g(0(km)) g(0(kn) - ex))
= Ek + ((0()))) + O(n)+t)
(k: = 0, 2...
(5)
The proof is by induction and given in Appendix A.2 Therefore, the leading order of discretization error is (n))) and given by:
A(O(k)))) = 0 -
ds ds s)) <(O(k)) 0))
(6)
This is functional equation of because (t) contains & via Equation A solution to Equation
(6) for a large 7 gives a small A and thus gives a small ex via Equation 5
3.3
Solution Equation
How can we solve Equation 6 ? It is not easy to find an exact solution because Equation (6) is functional integral equation respect to r7:
36 37 38 39 therefore, we assume power series solution with
{(0(kn)) Σ 1 (o(())) + n (0(ki)) 1)² .....
o=0
In the following theorem, we successfully find solution for all orders of vy.
Theorem 3.3 (Solution of Equation 6) The solution to Equation ( of is given by
a+2
(-1)' DE, DR. ERj
(.) = Σ
Σ
(8)
i!
tk.=a-422
for a 0, 2.., where we use differential operators (Lie derivatives) Da = Ea- 10) (a 1,2, and Do : g()) V and also defined (0) E (a 1,2,... and E.( = g
The proof follows from the definition of the Lie derivative and is given in Appendix A.3 two orders of the solution are given by:
The first
(o) = 2 (g(0) ()) = Vllg(0)112
(9)
(0) = 2 (5) V)g(0) + 6 (g(0) v)
(10)
Discussions. As can be inferred from Equations 10). È. contains the a + 2ua-order derivative of the loss function. Therefore, the higher-order counter terms cancel the higher-order smoothness of the discretization error.
Here, we note that Equation can be found, e.g., in 15 as higher-order backward error analysis. However, our derivation above has independent contributions: 1) we clarify that the counter term cancels the leading order of discretization error (Theorem 3.2). and 2) find that the discretization error itself is also given by the counter term (Corollary in the next section).
Equation 9 often appears in the literature on backward error analysis 21 35 and its related topics 27 28 31]. Typically, & is added to gradients of continuous equations (e.g., SDE) to close the gap between continuous equations and discrete algorithms (e.g.. SGD) by canceling (at least first-order) discretization error. However, higher-order discretization error is neglected these studies. contrast, our solution error.
in machine learning, e.g., 41 23 24
(8) cancels all orders of discretization

---

4 Discretization Error
The question here is to what extent the continuous approximation (18) is precise; this point is often
27 28. In this section, we use the counter term (8) and quantify discretization error as a function of the loss function and its
missed in the literature on continuous approximation [22 23 24
11 25
26
derivatives (Section
We find that our result well explains empirical results. We further derive a sufficient condition for learning rates for the discretization error to be small (Section 4.2
4.1 Counter Term Gives Leading Order of Discretization Error
We show that the counter term gives the leading order of discretization error between GD vs. GF and EoM. The proof follows from Theorem 3.2 and 3.3 and is given in Appendix A.4
Corollary 4.1 (Leading order of discretization error is given by É.). Suppose that use & up to O)1)-1), i.e. & = Eo u + + -/7--1 1 E Z,o ( = for r 0). Then,
ek+1 = ex A(O(k))) 0() = + 22 011)+)).
(11)
First, Corollary 4.1 implies that the higher the orders of the counter term we use (large 7), the more precise EoM (1) is (small ex). Thus, GF (5 0) gives larger discretization error than EoM (5 & 0). Second, Corollary 4.1 gives the equality of the leading order of discretization error arbitrary steps.
This is not bound [18] nor asymptotic analysis (k + oo). Third, let us give an intuition by considering E = (GF). Then, Corollary 4.1
gives:
ex+1 = Eo + (H(((sn)) + (V)((((())) (O()))) + O(7)),
(12)
where H()) E Rddd is the Hessian of the loss function f with respect 0 and 7 E Rdxd is the identity matrix. Equation (12) suggests that 1) large learning rates lead to large discretization error and 2) steep loss functions (along the trajectory) lead to large discretization error.
Empirical result. We find Equation (2) well explains our empirical result. We compare Equa- tion 112 (up to (17²)) with the actual discretiza- tion error of GD and GF in Figure First, the gap between our theoretical prediction dis- cretization error (orange curve) and the actual discretization error (red curve) small because the range of relative error (|lex|//[(xs]) in this
0.16
0.14
TheorytSimulat-on)
Norm of discrediattin ertor
0.12
0.10
0.08
Exporiment
plot is only 0-0.01 (see also Figure [ in Ap- pendix F). Second, most of the discretization error for Theory (orange curve) and Experiment (red curve) is produced within the first 100 steps. We can understand this phenomenon with the help of Equation (1). It suggests that discretiza- tion error can be enhanced when the loss func- tion is non-smooth along the learning trajectory, which is likely to occur at the beginning of train- ing due random initialization. Therefore, large part of discretization error is produced in the early stage of training. Third, we see that most of the gap between Theory (orange curve) and Experiment (red curve) also comes from the first 100 steps; in fact, the green curve shows that there is a much smaller enhancement of the gap after the 100th step. The source of the gap is the higher-order term O(1²²) in Equation (12). consists of higher-order derivatives of the loss function (Theorem
0.04
Theory (Simulation) given Rue
0.02
C10
0.00
100
10:
10a
10a
Number terations
Figure 2: Theoretical prediction of discretiza- tion error of GF and GD (Equation (12) vs. actual discretization error of GF and GD. The larning rate and weight decay are 10 and 10-? 2 See Appendix F.2 for more results and details. See Section 6 for experimental settings.
8 and Corollary 4.1) and thus can be large when the loss function is non-smooth along the learning trajectory. Therefore, by the same logic as above, the early stage of training tends to produce gap between Theory (orange curve) and Experiment (red curve).
5

---

4.2 Discretization Error Bounds
We provide sufficient condition (an upper bound for r)) for GF and EoM follow GD up to a given step k, which helps infer desired learning rates (step sizes) for the discretization error to be small. We first consider { 0 (GF).
Corollary 4.2 (Learning rate bound for & 0). Let & = 0 and assume that eo = 0 Let and t be arbitrary positive numbers. If the step size satisfies
2
(13)
max OSPSE
(I((H((())) + X))((((())))
for some ls E (1,2,-., 1 then the discretization error be arbitrarily small:
||ex|| < <(E)
(14)
The proof follows from Equation (2) and is given Appendix A.5 We see that 1) there is no guarantee that the discretization enror is small unless the learning rate sufficiently small, 2) need small learning rates to keep the discretization error small for long period, and 3) we need small learning rates keep the discretization error small for non-smooth loss landscapes. This is consistent with our empirical results in Figure 3 and 4 in fact, 1) the discretization error blows up for large learning rate (1) 10-1 in Figure 3. increases as the number of steps increases (Figure and 3) most of it is produced in the early phase of training, where the objective function tends to be non-smooth, and the gradients tend be large.
We compare our bound (l3) with a bound given in 18 because, to our knowledge, only [8] provides bound for the step size with respect discretization error in the context of deep learning. In [8]. is proved that in essence, S </ Ct, where 3r. and "iee measure the non-smoothness of the loss function, and с depends
the spectrum of the Hessian. These factors are hard to compute analytically unless the loss function and network are simple, but the qualitative behavior of this bound the same as ours (13); i.e., both bounds become tight when the loss function is non-smooth.
We also derive a learning rate bound for & Eo (EoM) and the full statement is given in Corollary A.1 in Appendix A.6 which states that if 17 O V/V)), then ||e < O Therefore, larger step sizes are now allowed compared with Corollary 4.2 (GF) because of the non-zero counter term. Furthermore, we can show larger bounds for higher-order counter terms in similar way.
0.025
4-
LRLe-1
Morm of discretrration MFFOC
0.020
3.1e-3,w1 counter term (EoM)
18te-1L counter torm (GF)
0.015
0.010
LRIe:2 LR16
counter term
counter term
0.000
101
10:
10a 10a
0
10"
10
10a
10'
10*
Number of iterationn
Number llrations
Figure 3: Discretization error explodes for large learning rate (10-1). LR means learning rate.
Figure 4: Discretization error of GF and EoM. Figure is magnified. The counter term reduces discretization error expected, and smaller learn- ing rates give smaller discretization errors.
Weight decay is 1 Curves include both GF and EoM. Relative discretization error is also shown in Appendix F See Section o for experimental settings.
5 Application: Scale- and Translation-invarianl Layers
To show the benefits FoM, we finally apply our theory to two specifie cases: scale-invariant layers 29 30 and translation translationinvariinn invariant layers [31 32 Additionally, Appendix provides application

---

to broken conservation laws [31. In the following, we simply focus on & 0 and & Eo to analyze the differences between ş and & + O.
Definitions Let us first introduce our notation. A transformation ab of E RRd with parameter a E R. said to be symmetry transformation of loss function f if f((((0, a:) - f(0). IA {0, 1]dt denotes the indicator vector of subspace A R (e.g., A is linear layer in the DNN). For scalar E R. we define a = alA + L.: E R, where AA' is the complement of A. For a vector A E R" define A 0 @ I A Rd where is the Hadamard element-wise product. For the gradient operator V = (0/30,, 0/00a) T, we define V = I @ V. We also define rA = | and 0 = AAT
5.1 Learning Dynamics of Scale-invariant Layers
In this section, we focus that is invariant under the scale transformation eb(0, a) == 0.40 = af + (a > 0). For example, linear layer immediately before batch normalization layer is scale-invariant. We that for better description of GD's discrete dynamics, we need modifications to the decay rate of rA that previously derived in the continuous regime (33). In addition, we show that EoM successfully reproduces the limiting dynamics of r' and angular update [34] at E - oc that are previously derived in the discrete regime, while GF cannot. In Appendix C we additionally show that there are crucial differences between GD and GF via the effective learning rate of scale-invariant layers
layers. A scale-invariae layer A is defined subspace
29
EoM for r We construct the EoM for A (the EoM for O is given in Appendix foor completeness). E(O(t))
Theorem 5.1 (EoM for r' and solution). EoM ( gives 2 (t) ( - ( Specifically, this is equivalent to:
(t) = -2xr = 22(1) = rr) 2X
2X
(15)
for & = (GF) and
(²) = -((A+ + 7112
2 )r2 (t))) [V (
(16)
(t) = =r( -2(1+ +1 f drc 27 00(1+))00011 |V Ac(()))
(17)
(T)
for E = Eo (EoM).
The proof is based on Equations
and given in Appendix A.7 Equation 15) gives r (kr) =
3 (0)e - 2ndi (k E Zzo) at discretization; therefore, πa is regarded as the decay rate of TA (intrinsic learning rate 33 ). This is originally discussed in the continuous regime (SDE) 33 however, we
find that for a better description of the discrete dynamics of GD, the decay rate needs to be modified
from T7 to 171(1 + T (see the exponent of Equation (17)). This means that F in GD decays faster
than expected from a naive continuous dynamics (GF (15 and SDE 33
See Appendix G for
higher-order corrections.
Limiting dynamics. We next derive the limiting dynamics + o0) of T.A Corollary 5.1 (r'A equilibrium). When E 0 (GF), TA collapses zero as cc. When & Eo (EoM), assume that there exist two constants A+ 2 O and c. 2 0 such that (()) toon T and ([V + (()(1)) t- Cy.. Then 2.
y
Cx..
The proof follows from Theorem 5.1 and is given in Appendix A.8
The non-zero counter term successfully reproduces 30 2n/2Xc, 29 34]. which is originally derived in the discrete regime (SGD), although our approach is continuous (FoM (T). Without the counter term, we cannot explain
this behavior because GF gives (() to00 0( Vn/2Ac.).
We next derive the limiting dynamics of angular update (34), which is designed to measure the temporal evolution of scale-invariant networks. It is originally defined in the discrete regime:
7

---

cos As := Ak AAR+I, where Abk ==
11 That is, As represents single-step angular change in the weight parameters of the scale-invariant layers A. In the continuous regime, we can define cos (1) = (l) (l + n))
1430.
Corollary 5.2 (t) at equilibrium), Let use E= E Suppose that the assumptions in Corollary 5.1 are satisfied. The angular update at equilibrium, denoted by D., is given by cos A, = X-77 O and thus, A. /2nA + ((12)8/22.
1-nd
The proof is based on Corollary [.I] and given in Appendix A0 FoM successfully reproduces A. / (34, which is originally derived in the discrete regime (SGD), although EoM is continuous itself. On the other hand, GF cannot explain the limiting dynamics of At) because
when { O, rt) goes to zero as t + (Equation ).), and thus, cos A(t) ex(l) (x(t+a) is
rl(lt) ra (a(++))
ill-defined. In summary, there are gaps between GF and GD, and our discussion above indicates that the counter term is inevitable to describe the actual dynamics of GD.
5.2 Learning Dynamics of Translation-invarian Layers
Next, we apply EoM to translation-invane layers. To the best of our knowledge, study analyzes the temporal evolution of translation-invarian layers except for [3]] and [2], where only the sum of weights is their focus, while we derive the dynamics of the whole weights. A translation-invariant layer A is defined as layer that is invariant under the translation transformation vb(0, a) := 0 @+al, (a R). For example, linear layer immediately before the softmax layer is translation-invarian the following, we derive EoM and show that its theoretical prediction of decay rates dramatically matches empirical results, indicating the importance of the counter term. In Appendix D additionally discuss the differences between GF and GD in translation-invariant layers.
In
For convenience, first decompose 0 two vectors (Figure 5); rTro is orthogonal to f(0).. and 0 Anl is orthogonal to 0 TTT6 Here, note that f(( is orthogonal to 1A because of translation invariance; in fact, differentiating both sides of f(0 + a a1 = (0) with respect to a and setting
O, we have AA V (0) 0 (see also Lemma A.7 in Appendix A.III. Formally, we define OAI,
Oay and the projection matrix P as 0 == A 12 11 A. OAl (I P)O 0 A
and P = d 1 where d' is the dimension of A.
We construct the EoM for OrT (the EoM for Ylrl is given in Appendix D for completeness).
Theorem 5.2 (EoM for OAL)). EOM [ gives (t) = A1 n))'E((tt). Specifically, this is equivalent to θ (t) = 00 (f) 0 A -A (0) -e--- for & = (GF) and
(t) (A + nAa 1 (t) = (t) ( (A+ =42 for _ = Eu (EoM).
The proof is based on Equations I 9 and is given in Appendix A. O monotonically collapses to zero as + oo in either case of c 0 or E 7 O: thus, as / increases, the dynamics is restricted onto the subspace orthogonal e rr (Figure 5). The decay rate is corrected by the counter term from yn to ısλ + n² as is also done for T.A in Section 5.1 Therefore, the 0 of GD decays faster than T support our findings. In particular, Table shows that the decay rates predicted by EoM dramatically match those of GD, indicating the importance of the counter term.
that of GF. Figure [ and Table
Table 1: Decay rates of | The theoretical predictions by EoM (third column) dramatically match experimental results of GD (fourth column) much better than GF (second column), indicating the importance of the counter term. LR and WD mean learning rate and weight decay, respectively. The colors correspond to those in Figure
0 See Section for experimental settings.
(LR, WD)
Theory (GF)
Theory (EoM: Ours)
Experiment (GD)
(10 (10-1,10-3) (blue)
10 10-3
1.0005 x 10-3
L.0005003484995567 x 10-3
(10-1,10-3) (orange)
10-4
1.00005 10-4
1.000050018236335, 10-4
(10-2.10-2) (green)
10 10-4
1.00005 x 10
1.00004998099958888 x 10
(10-2,10-3) (red)
10-5
1.000005 x 10-5
1.0000049776814671 x 10 10
(10-3, 20=) (purple)
10-5
1.000005 x 10-5
1.00000504753122266 10-5
(10-3, (10-3,10-3, 10-3) (yellow)
10-6
1.0000005 10-6
1.0000005475000883 x 10-6
8

---

coanslation-invariinn layer
Aa
8
0.017
0.016
Decay
B
0.014
IRle-1, WD1e-2
LRle1 WDle-3
(f)
0.013
LR1e WD1e.2
LR1e-2, WD1e-3
LR1e-J, WD1e:2
0.011
LR1e-3, WD1e-3
Decay
0
10"
10'
10'
107
10
Number iterations
Figure 5: Learning dynamics of Figure 6: Decay of || (GD). ||0 monotonically de-
A is translation- 0Λ = (A2) AA2)_. TT decays to 0 invariant layer. LR and WD mean learning rate and weight (also shown in Figure O The decay of decay, respectively. Note that the orange and green curves GD is faster than that of GF (Theorem (LRie-1, WDle-3 and LRle-2, WDle-2) and the red and purple curves (LR WD1e-3 and LRle-3, WD1e-2) to- tally overlap. The decay rates of all curves are given in Table See Section 6 for experimental settings. I
translation-invarian layer. Here, cays to zero, as suggested by Theorem 5.2
5.2 As increases, the dynamics re-
stricted onto the subspace orthogonal to AAl
6 Experiment
We explain our experimental settings for Figures 26 and Table Our network consists of first linear layer, swish activation [48], second linear layer, batch normalization 44), third linear layer, and last softmax layer. Cross-entropy is used for the loss function. We note that the second linear layer is
and the last linear layer is translation invariant. The batch normalization uses fixed statistics to keep the scale invariance of the second linear layer. Swish is chosen to ensure diffcrentiability, None of the linear layers have a bias term. The dataset is the training set of MNIST [50), and thus, the batch size is 60,000. Gradient descent is used for the optimizer. We use 64-bits of precision for all computations. To simulate GF and EoM, we use a sufficiently small learning rate (10-5). The results produced from only one random seed to save computational costs, but we confirm that different random seeds lead to similar results. More detailed information is given in Appendix E and our code. In all experiments, we use & - Eo for EoM. We do not include higher-order counter terms, such as & because they require third and higher order derivatives of the loss function and are thus extremely memorr--consminng We could circumvent this issue, c.g., by applying Hessian-free optimization (5I), but this is out of our current scope.
Conclusion and Limitations
In this work, fill the critical gap between GF and GD, we add a counter term to GF and obtain EoM, continuous differential equation that precisely describes the discrete learning dynamics of GD. To show what extent GF and EoM are precise in describing GD's discrete dynamics, we derive the leading order of discretization error, as is often missed in the literature on the continuous approximation of discrete GD algorithms. We further derive a sufficient condition for learning rates for the discretization error to be small. We apply our theory to two specific cases, scale- and vrassation-invariant invariant layers, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD. Our experimental results support our theoretical findings.
Throughout this paper, we focus only on GD and GF to expose the ideas simply, and our study does not include stochasticity (e.g., SGD and SDE), acceleration methods (e.g., momentum and Nesterov [52)). ). or adaptive optimizers (e.g., Adam (53]). Nonetheless, they could be combined with our analy Amalysis. for example, using eиror analysis of SDEs [3 24], continuous-tima accelerated methods
13 14 55 [6], and continuous-time Adam (56). See Appendix G for more discussions.
Therefore, our study could be extended to import continuous analysis to the discrete analysis of various GD algorithms. this sense, our work bridges discrete and continuous analyses of GD algorithms.

---

Acknowledgment
We thank Shuhei M. Yoshida for his insightful comments on the dynamies of scale-invariant layers and the experimental settings. We also thank Hidenori Tanaka for his discussion that inspired us to start this study.
References
[1] Harold J. Kushner. Rates of convergence for sequential Monte Carlo optimization methods. SIAM Journal on Control and Optimization, 16(1):150-16) 1978.
[2] Harold J. Kushner and Dean S. Clark. Stochastic approximation methods for constrained and unconstrained systems. Springer-Vere New York, 1978.
[3] Harold J. Kushner and Adam Shwartz. An invariant measure approach to the convergence of stochastic approximations with state dependent noise. SIAM Journal on Control and Optimiza- tion, 22(1):13-2) 1984.
[4] L.. Ljung, G.C. Pflug, and H. Walk. Stochastic Approximation and Optimization of Random Systems. Oberwolfach Seminars. Birkhäuser Basel, 1992.
[5] H. Kushner and G.G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Stochastic Modelling and Applied Probability. Springer New York, 2003.
[6] Maxim Raginsky and Jake Bouvrie. Continuous-tin time stochastic mirror descent on a network: Variance reduction, consensus, convergence. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), pages 6793-6800, IEEE, 2012.
[7] Walid Krichene, Alexandre Bayen, and Peter Bartlett. Accelerated mirror descent in continu- and discrete time. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[8] Panayotis Mertikopoulos and Mathias Staudigl. Convergence to nash equilibrium in continuous games with noisy first-order feedback. In 2017 IEEE 56th Anmual Conference on Decision and Control (CDC), pages 5609-5614. IEEE, 2017.
[9] Walid Krichene and Peter I. Bartlett. Acceleration and averaging in stochastic descent dynamics. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[10] Qiang Liu. Stein variational gradient descent as gradient flow. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[11] Yuanyuan Feng, Lei Li, and Jian-Guo Liu. Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations. Communications in Mathematical Sciences, 16:777-789, 2017.
[12] Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d' d'Aspremont. Integration methods and optimization algorithms. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[13] Pan Xu, Tianhao Wang, and Quanquan Gu. Accelerated stochastic mirror descent: From continuous-time dynamics to discrete- time algorithms. In Amos Storkey and Fernando Perez- Cruz, editors, Proceedings the Twenty-Fire International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1087-1096. PMLR, 09-11 Apr 2018.
[14] Pan Xu, Tianhao Wang, and Quanquan Gu. Continuous and discrete-time accelerated stochastic mirror descent for strongly convex functions. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5492-5501. PMLR, 10-15 Jul 2018.
10

---

[15] Alnur Ali, Edgar Dobriban, and Ryan J. Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In ICML, pages 233 233-244, 2020.
[16] Nikola B Kovachki and Andrew M Stuart. Continuous time analysis of momentum methods. Journal of Machine Learning Research, 22(17): 2021.
[17] Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis. arXiv preprint arXiv:2106.02558 2021.
[18] Omer Elkabetz and Nadav Cohen. Continuous vs. discrete optimization of deep neural networks. In A. Beygelzimer, Dauphin, P. Liang. and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[19] Fanchen Bu and Dong Fui Chang. Feedback gradient descent: Efficient and stable optimization with orthogonality for DNNs. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[20] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? -a mathematical framework. In International Conference on Learning Representations, 2022.
[21] E. Hairer, S. P. Nørsett, and G. Wanner, Solving Ordinary Differential Equations I (2nd Revised. ): Nonstiff Problems. Springer-Verlatt Berlin, Heidelberg, 1993.
[22] Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2101-2110. PMLR, 06-11 Aug 2017.
[23] Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and dynamics of stochastic gradient algorithms Mathematical foundations. Journal of Machine Learning Research, 20(40):1-47, 2019.
[24] Yuanyuan Feng, Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu. Uniform-in-time weak analysis for stochastic gradient descent algorithms via diffusion approximation. Communications in Mathematical Sciences, 18(1):163-18) 2020.
[25] Wenqing Hu, Chris Junchi Li, Lei Li, and Jian-Guo Liu. On the diffusion approximation of nonconvex stochastic gradient descent. Annals of Mathematical Sciences and Applications, 2019.
[26] Jing An, Jianfeng Lu, and Lexing Ying. Stochastic modified equations for the asynchronous stochastic gradient descent. Information and Inference: A Journal of the IMA, (14),851-873, 11 2019.
[27] David Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2021.
[28] Samuel Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regular- ization in stochastic gradient descent. In International Conference on Learning Representations, 2021.
[29] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1006.05350, 2017.
[30] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2019.
[31] Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws deep learning dynamics. In International Conference on Learning Representations, 2021.
[32] Hidenori Tanaka and Daniel Kunin. Noether's learning dynamics: Role of symmetry breaking in neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.

---

[33] Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. In NeurlPS, 2020.
[34] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learning dynamics of neural network with normalization, weight decay, and SGD, 2021.
[35] Ernst Hairer, Christian Lubich, and Gerhard Wanner. Geometric Numerical Integration. Structure- Prestervng Algorithms for Ordinary Differential Equations. Springer, Berlin, 2nd ed. edition, 2006. ID: unige:12343.
[36] Ioan A Rus. On the problem of Darboux-lonescu. Universitatea Habes-Bolyai Faculty of Mathematics, 1981.
[37] Nicolaie Lungu and Ioan A Rus. On a functional volterra-fredholm integral equation, via picard operators. J. math. ineq, 3(4):519-527, 2009.
[38] Nguyen Thanh Long et al. On a nonlinear volmerea-hammrrsein integral equation in two variables. Acta Mathematica Scientia, 33(2):484 494, 2013.
[39] Tran Minh Thuyet, Nguyen Thanh Long, et al. nonlinear volmersa-hammrrseiin integral equation in three variables. Nonlinear Functional Analysis and Applications, 1932111 2014.
[40] Daniela Marian, Sorina Anamaria Ciplea, and Nicolaie Lungu. On a functional integral equation. Symmetry, 13(8):1321, 2021.
[41] Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2101-2110. PMLR, 06-11 Aug 2017.
[42] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 31, 2018.
[43] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. International Conference on Learning Representations, 2019.
[44] Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece. Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[45] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In International Conference on Learning Representations, 2020.
[46] Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank J Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. arXiv preprint arXiv:2202.00980. 2022.
[47] Simon Roburin, Yann de Mont-Marin, Andrei Bursuc, Renaud Marlet, Patrick Pérez, and Mathieu Aubry. Spherical perspective on learning with normalization layers. Neurocomputing, 487:66-74, 2022.
[48] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv: 17X.::1710.054411 2017.
[49] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448 456. PMLR, 2015.
[50] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yannn. lecun. confexdb/mnist, 2, 2010. License: Creative Commons Attribution-See Share Alike 3.0 license.
12

---

[51] James Martens et al. Deep learning via hessian- free optimization. In ICML, volume 27, pages 735-742, 2010.
[52] Nesterov Y. E. method for solving the convex programming problem with convergence rate O(1/k²). Dokl. Akad. Nauk SSSR, 269:543-544 1983.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint (42.6980 2014.
[54] Weijie Su, Stephen Boyd, and Emmanuel J. Candès. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1 4,,,,4,,110 2016.
[55] Jean-Francois Aujol, Charles Dossal, and Aude Rondepierre, Optimal convergence rates for Nesterov acceleration. SIAM Journal on Optimization, 29(4):311--3155 2019.
[56] Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM Journal on Optimization, 31(1):244-274, 2021.
[57] Emmy Noether. Invariante Variationssroblemee. Göttingen, Math-phys. Klasse, Seite 235-157, 1918.
Nachr. d. König. Gesellsch. d. Wiss. zu
[58] Emmy Noether. Invariant variation problems. Transport theory and statistical physics, 1(3):186- 207, 1971.
[59] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffree Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas. Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. License: Apache License 2.0. Software available from tensorflow.org.
[60]
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. Del Río, M. Wiebe, P. Peterson, P. Gééradd-aarchan K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-366 09 2020. License: BSD 3-Clause "New" or "Revised" License.
[61] Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[63] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 630-645 2016.
[64] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
IEEE, 2009.
[65] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. 2014.
[66] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. European conference computer vision, pages 740-755. Springer, 2014.
13

---

[67] Jian Deng. Strong backward error analysis for Euler- Maruyama method. Int. J. Numer. Anal. Model., 13:1-21, 2016.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sections 6 and and Appendix G
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section and Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] To save computational costs, we do not run experiments with multiple random seeds, but we confirm that different random seeds give similar
results, stated Section 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix F
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See our code.
(b) Did you mention the license of the assets? [Yes] See our code.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
See our code.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We do not such data.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data we are using do not include such information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "X_Fgcbk4OfT", "reviewer": "Reviewer_2BkU", "summary": "The authors derive an Equation of Motion, i.e., a continuous differential equation that matches the discrete time dynamics of gradient descent more clearly. They do so by adding a counter terms to Gradient Flow that cancels out higher order discretization errors in DNNs, and this counter term is derived using backward error analysis, more precisely it is the solution to Equation 6 in the paper. \n\nGiven they are using backward error analysis, they are also able to quantify the discretization error for GD approximation of GF along with the counter term, and hence also provide a bound on the learning rate such that this discretization error is small. \n\nThe authors apply their Equation of Motion to translation and scale invariant layers and show that their theoretical predictions better match GD.", "questions": "In the current state the results seem to be very similar to the previous work (as I have mentioned in the previous comment), and besides the series expansion in Theorem 3.3, it is very hard to adjudge the precise novelty of the work, esp since most of the results discussed in the next sections assumes that the counter term is either 0 or they assume the first order counter term. I would really appreciate if the authors could clarify their contribution. \n\nAnother minor comment, I think it would be easier to read the paper (esp the proofs) if the author provided a sketch after the main statement in the main paper as well as in the appendix. ", "limitations": "Yes the authors have discussed the limitations of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My main concern is that this paper does not provide any interesting new result. The main novelty of the paper is the general form of the counter term (which is derived in Theorem 3.3), as opposed to previous work for example Barrett and Dherin (Implicit gradient regularization) which only uses the first order term as their regularization term. \n\nWhile the authors mention that they derive the discretization error (in Corollary 4.1), the precise formulation is not provided and the rate is provided as an upper bound (using Big-OH) which I believe is an artifact of standard series expansion results. \n\nFurthermore, for most of the results on the discretization error bounds and the upper bound on the learning rate, the authors assume that the counter term is either equal to zero or assume the first order counter term (i.e., the term in equation 9).\n\nWith the counter term equal to 0, the analysis matches a lot of the previous work (example Elkabetz and Cohen, [18]) and with the first order term the main theoretical results are very similar to the ones already established in Barrett and Dherin (who introduce the first order counter term as the implicit regularizer and the error is discussed in Theorem 3.1)\n\nBesides this, the authors do apply their analysis to characterize the learning dynamics of scale and translation invariant layers and show that with the inclusion of the first order (adding higher order counter term is going to be computationally expensive) they are able to better predict the decay of parameter norm, which is interesting but not that surprising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MzXKD3ywbEb", "reviewer": "Reviewer_UB6r", "summary": "This paper is concerned with a theoretical understanding of modelling the dynamics of gradient descent with a differential equation. Previous work (Gradient Flow) describes the differential equation as:\n\n$\\frac{d\\theta}{dt} = -\\nabla_{\\theta} L(\\theta)$\n\nWhich is the Euler discretisation of Gradient Descent:\n\n$\\theta_{t+1} = \\theta_{t} - \\eta \\nabla_{\\theta}L(\\theta_t)$\n\nHowever, discretisation error exists, such that Gradient Flow and Gradient Descent diverge. This paper derives a counter term to Gradient Flow, labelled by $\\xi$:\n\n$\\frac{d\\theta}{dt} = -\\nabla_{\\theta} L(\\theta) - \\eta\\xi(\\theta)$\n\nThe counter term is a functional integral, the paper approximates the counter term with a series solution in $\\eta$, with a recursive relationship existing to get from term $k$ to term $k+1$, the new dynamics are called Equation of Motion (EoM). A limit for the learning rate is also derived, which allows accurate simulation of gradient descent using (EoM) with larger step sizes.\n\nFinally these findings are tested on scale-invariant layers and translation-invariant layers and the results support the theoretical findings.", "questions": "The few questions/clarifications I have are:\n\n- Line 240 (Definitions), a symmetry is defined as a transformation of parameters such that the loss function is unchanged. Is this definitely correct? Should it not be a transformation that leaves the predictions unchanged?\n- Is it possible to run similar experiments for layers that do not have any symmetry, such as a standard linear layer? This could make a nice piece of future work.\n- It seems that the majority of experiments were run using the first term in the expansion of $\\xi$. Is it possible to run an ablation where we see how adding the next term affects results, and on and on until say the 5th term (to justify using only the first term)? I appreciate this requires huge amounts of memory for the higher derivatives.\n- Apart from the theoretical contributions (which are already fantastic), do you envision any practical applications of this work?", "limitations": "The authors have been upfront with the limitations of their work. These are given in the conclusion and provide a nice avenue for future research, they are about different optimizers and how using minibatches are not accounted for in the current work. I cannot think of any further limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis is not an area I have significant expertise in, however, overall this paper is very good in my opinion. Specifically:\n\n- The paper shows many impressive theoretical results\n- The experiments are extensive and support the theory\n- The motivation for the paper is very clear\n\nWeaknesses:\n\nAgain, I believe the paper is very good. I think it presents good theoretical results, with sufficient experiment to support this. The only weaknesses overall are in the writing style and presentation, I think the paper is quite math heavy currently, which makes it less accessible. However, this is down to personal preference. For example lines 239-245 (Definitions) feel like quite a complicated way of saying most of the meanings. One definition is $\\alpha_{\\mathcal{A}} = \\alpha I_\\mathcal{A} + I_{\\mathcal{A}^{C}}$, but it is easier in my opinion to say $\\alpha_{\\mathcal{A}}$ is $\\alpha$ for the parameters in layer $\\mathcal{A}$ and $1$ for the others. This can be extended for most of the definitions in this paragraph. I think the best way the paper can be improved is by having as much intuition as possible in the main text, with theorems included, and then possibly having more mathematical detail in the appendix.\n\nOther small points are the presentation of results. Figure captions don't have **Figure n** in line with the caption, but over to the left which feels weird. Table 1 could also be improved, rather than listing the decay rates, it might be more informative to list the differences (and relative differences) in decay rates between GF & GD and EoM & GD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L-ZkcQZ90xd", "reviewer": "Reviewer_NdAp", "summary": "This paper derives a counter term to the gradient flow ODE formulation that reduces the discretization error from Euler's method, which is gradient descent. When this correction term is expanded as a Taylor series, adding a select number of terms reduces the discretization order accordingly. This is then used to analyze the behavior of GD under symmetry constraints, specifically scale- and translation-invariant parameters. Specifically, this adds learning-rate-dependent correction terms to the decay rates of certain quantities, which matches gradient descent in practice.", "questions": "Can you clarify the connection between this correction term and the global truncation error of Euler's method (i.e. a bound on e_k)? I expected them to look very similar but seems to not be the case.\n\nDepending on the step size, Euler's method can result in inconsistencies where the discretization completely diverges from the gradient flow, in particular when the step size is much larger than the Lipschitz constant of g, and results in a discretization that cannot be meaningfully modeled by any ODE. I imagine this would correspond to the correction term becoming infinite and the zero-th order approximation to the correction term becoming somewhat meaningless. Does the equation of motion reflect this kind of behavior in any way?\n\nCan the correction term be used to correct the drift of the SDE formulation to stochastic gradient descent? \n\nCan Corollary 4.2 be experimentally verified?", "limitations": ".", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n  - Quite an interesting take which attempts to correct the theoretical ODE formulation in order to match practice (as opposed to bridging this gap by using higher order solvers, for example). \n  - The motivation, process, and theoretical results are presented very well. I could follow understand every result (just the results; not the proofs) despite not being an expert in the theory of gradient descent.\n\nCons:\n  - I imagine the theory (of estimating discretization error of ODEs) has been done before, though perhaps not in this exact context. The zero-th order term (Eq 9) has shown up in machine learning studies before.\n  - It's not clear to me if we gain anything from using the higher order terms in the derivation, as the analysis and experimental results all use only the zero-th order term. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EJ1OTwhJ3xD", "reviewer": "Reviewer_BEXW", "summary": "This paper deals with the discrepancy between the actual discretized gradient descent and its continuous version, i.e., a gradient flow, for describing the equation of motion of learning dynamics more precisely. The discrepancy error is formally introduced by using the backward error in numerical analysis. The authors derive a counter term, which can compensate for such a discrepancy of the gradient flow, thus can describe the actual discretized trajectories in a continuous manner. While the derived counter term is a complicated functional integral equation, it can be analytically solved (for all orders) by assuming the underlying solution is a power series. As an application, the authors use the derived dynamics with the proposed counter term for investigating scaling- and translation-invariant layers.", "questions": "As I mentioned previously, it will be nice if the authors can experimentally elaborate on the usefulness of the proposed generalized counter term compared to the first-order counter term.\n\nIs there any idea that can tighten the learning rate bound (13) more efficiently?\n", "limitations": "The authors discuss the limitations of the proposed method, e.g.,  the lack of concerning the mini-batch stochastic GD and other optimizers beyond GD, in Conclusion and Limitations section. It will be nice if the authors also address the questions raised above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Note]\n\nBecause I am not an expert on learning theory, my evaluation might not be exhaustive. Also, I did not read the proofs in the supplementary material carefully.\n\n[Strengths]\n\nTo me, the derived counter term is novel and seems to be useful to predict and interpret complicated learning dynamics of deep neural network models. Although there have been previous studies that incorporate some correction terms with respect to the backward analysis error, to my knowledge they are restricted to 1-st order compensation $\\frac{1}{4} \\nabla || \\nabla f(\\theta)||$, which is generally called an implicit gradient regularization. The proposed counter term is generalized for higher orders and can recover the previous studies well as in (9). While the main result (8) seems to be a known technique in the numerical analysis field, I would like to give appropriate credit to the authors for the contribution that introducing such a technique to the deep learning field well. \n\n[Weakenesses]\n\nThe authors address a full-batch gradient descent only. The authors also mention such a limitation of this work in Conclusion and Limitations. I am not sure whether the approach can be easily generalized for the mini-batch stochastic gradient descent method.\n\nWhile the authors theoretically prove high-order corrections are required to cancel the leading order of discretization error, it will be great if the authors (1) experimentally show the discrepancy between the GF with the proposed correction and that with a first order correction, and (2) demonstrate the former can approximate GD well compared the latter, e.g., in Figure 2 or Figure 4.\n\nThe proposed method is not guaranteed for GD with a large learning rate, thus cannot be used for explaining some interesting phenomena, e.g., the regularization effect of an initial large learning rate. However, I think it is not a crucial drawback of this paper, considering the essential assumption of GF.\n\nThe paper is very dense and thus hard to read. A journal format might be more suitable for a clear representation of this work.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept