Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks
Authors: Hidenori Tanaka
Daniel Kunin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Noether's theorem
learning dynamics
symmetry
loss landscape
differential equation
modified equation analysis
conservation law
hessian
geometry
physics
gradient flow

Abstract: In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the "geometry of learning dynamics" in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify "kinetic symmetry breaking" (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether’s theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of implicit adaptive optimization, establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.

Noether's Learning Dynamies: Role of Symmetry Breaking in Neural Networks
Hidenori Tanaka'*, Daniel Kunin²
Physics & Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA, USA Stannfor University, Stanford, CA, USA
Abstract
In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been central design principle efficiently capture regularities the world, but the role of symmetry breaking is not well understood. Here, we develop theoretical framework to study the geometry of learning dynamics in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify kinetic symmetry breaking (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether's theorem known in physics take into account KSB and derive the resulting motion of the Noether charge: Noether's Learming Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces mechanism of implicit adaptive optimization, establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.
With the rapid increase in available data and computa- tional power, artificial neural networks have become an essential tool both in science and engineering. To obtain an accurate model, one has to overcome the challenge of tuning millions of randomly initialized parameters q RN to minimize the loss function (4) E R, measure of the discrepancy between the model's predictions from the in- put data and the provided truth. In minimizing the loss, the learning rule specifies how to iteratively update these parameters based on the recent trajectory and the local geometry of the non-convex loss landscape. We can rep- resent the repeated updates of the millions of parameters during learning as discrete movements a point in high-dimensional parameter space (Fig. T.). The architecture shapes the static loss landscape and the learning rule governs the motion of the model through parameter space during training. However, is the complex interaction of the architecture and learning rule which determines the final state of the trained model. Inspired by tools and ideas from classical mechanics (Fig. [p)), we develop a theoretical framework which unifies the effects of the architecture and learning rule the geometry of the learning dynamics.
Dynamics
Classical Mechanis
Parameter
Physical
Figure 1: Learning dynamics in anal- ogy to classical mechanics. (a) Learn- ing dynamics of neural network in pa- rameter space. (b) Motion of a particle in physical space.
Correspondence to: Hidenori Tanaka chidenori. www.denor....aneeettetttteeeeeee tanakaentt--eeeareeee com> Work done during an internship Physics & Informatics Laboratories, NTT Research, Inc.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Symmetry as a mathematical lens. One of the most powerful mathematical tools for studying the geometry of complex objects is symmetry. A symmetry of an object is a transformation that when applied to the object leaves certain properties unchanged. For example, consider how a circle can be rotated around its center without changing its overall appearance. In otherwords, a circle has a continuous rotational symmetry. In particular, the form of the symmetry transformation constrains the form of the object, the circle is the only 2D shape that has continuous rotational symmetry. Objects with symmetry are not restricted to geometric shapes, but could be functions or even dynamical systems. Recently, the lens of symmetry has been applied in the study of neural networks DI 2 A recent work [I identified an array of symmetries in neural networks to study the geometry of the loss function f()) in the parameter space. For example, batch normalization [6] defined as (N(gz) = (qx - Elgzi)/ V /ar[gaa) introduces a scale symmetry in the parameters immediately preceding the normalization layer, BN((1 + s)zz)= = BN(qr) for any positive scalar s E R+
Geometry of learning dynamics. What does the sym- metry of the loss function parameter space imply about the symmetry of the learning dynamics? In classical me- chanics, a symmetry of the potential function in space usually implies the equivariance of the trajectory, namely, a solution of the Newton's s cquation of motion (grey) re- mains solution under the symmetry transformations (red) as shown in Fig. 2 (a). This implies that the time evo- lution of the potential energy is invariant under symmetry transformations at any point in the dynamics. However, this intuition does not hold in modern neural networks. Consider the concrete example of a VGG1 I model with batch normalization layer after each convolutional filter. The scale symmetry of batch normalization implies the loss function is invariant to the scaling transformation on the convolutional filters g, f((1 + s)gz) = f(zz), however the learning dynamics are not. To demonstrate this we train two VGGI1 models on Tiny ImageNet with a standard initialization (grey) and a scaled initialization (red). Notice, that the scaling does not change the initial loss due to the scale symmetry, however it does effect the learning dynamics of the loss, as shown in the bottom of Fig. B(). In otherwords, even though the loss landscape observes scale symmetry at all steps in training, the learning dynamics do not. Understanding symmetry of the learning dynamics requires a new theoretical perspective.
(RR()))) = Rio G-
Loss functon A(I + th)) = Ra) c db VGG11 with Barchhorrm on Tiny Imaga/Net Scaled intiollaation
Rotsted initiaization Origin Initieization
Original Int
Time
12 Epochs
Figure 2: Symmetry of the loss + sym- metry of the learning dynamics. (a) Physical dynamics with rotational sym- metry. (b) Learning dynamics in param- eter space with scale symmetry.
Implicit acceleration and variational perspective on learning dynamics. In physics, variational formulations based on the Lagrangian and Hamiltonian provide a unifying framework study the role of symmetry in dynamical systems. However, there is still a gap between the tools and concepts in physics and the learning dynamics of neural networks. We identify the root cause of this gap to be the learning rule, namely the fact that the continuous-time limit of gradient descent is a first order differential equation in time, gradient flow dg dt
= f. unlike classical mechanics which is
governed by Netwon's 's second law m.yy = - V,, which is second-order in time. We bridge this gap by realizing that the effect of discretization, is introduce an implicit acceleration term proportional to the learning rate. Leveraging this connection, we build the mechanics of learning dynamics:
Sec. M From Newtonian to Lagrangian mechanics of learning via implicit acceleration. We show how gradient descent with a finite learning rate introduces implicit acceleration making it amendable to the Lagrangian formulation of accelerated optimizers [4]. Unlike in physics, kinetic symmetry breaking (KSB) is inherent in learning dynamics. We define kinetic symmetry breaking, where the kinetic energy corresponding the learning rule explicitly breaks the symmetry of the potential energy corresponding to the loss function. We derive Noether's learning dynamics (NLD) induced by the KSB. We generalize Noether's theorem in physics to derive NLD, which accounts for damping, the unique symmetries of the loss, and the non-Euclidean metric used in learning rules.
Sec.
Sec. 3
Sec.
KSB induces impicit adaptive optimization and enables successful learning. We leverage NLD to exactly solve for the time-evolution of the effective learning rate for networks with normalization layers, and establish an exact analogy between two seemingly unrelated components of modern deep learning: normalization and adaptive optimization.

---

From Newtonian to Lagrangian mechanics of learning
The fundamental law of mechanics the principle of least action [5] We first review how pioneering
works by Wibisono, Wilson, and Jordan 4 6 7 apply the variational principles of physics to model accelerated methods in optimization. We then show how gradient descent with a finite learning rate introduces implicit acceleration making it amendable to the Lagrangian formulation.
Variational formulation of classical mechanics. Just over century after Newton, Joseph- Lagrange demonstrated how Newton's 's equations of motion could be derived from variational framework of least action. Rather than integrating an equation of motion over time to get the path an object takes from point to point b, Lagrange prescribed that the path taken by the object is the path that minimizes the action, the path integral a function termed the Lagrangian, over all possible paths from to b. This principle of least action transformed mechanics from differential description to a variational description. Consider for example the simple setup of particle of mass m at position
IR- in N dimensional physical space moving according potential f(4) E IR
and Newton's s second law mğ = - ff, where qj = de is the acceleration. Let the Lagrangian be the function [(g, i, 1) mmq² - f')), the difference between the kinetic energy T = m and the potential energy V [q) of the particle. The principle of least action states that the time evolution of the particle's 's position g(t) is the function that minimizes the action S g L C(g. q. t)dt. In otherwords, g(t), with boundary conditions (to)) = and ((t1) = b, must satisfy the stationary 6$ = O, which is equivalent to satisfying the Euler-Lagrange equation (see SI Sec. for condition oq(1) the review),
(OL) dt 09
Oc
(1)
0q
Indeed, substituting the Lagrangian into the Euler- LagranLe equation, we recover the Newton's equation of motion mğ = -V ff confirming the consistency of this variational formulation.
Damping in optimization. In learning dynamics, one of the universal features is damping, a term dg proportional to the velocity q 7 in the equation motion. Intuitively, damping is needed for any optimization, since without damping the learning dynamics would conserve energy and not converge. We can incorporate damping into the previous Lagrangian example by introducing an explicit time-dependence L(g, t) e " coefficient [8] Substituting this Lagrangian into the Lager-Lag equation and dividing both sides by 6 ! we obtain the equation of motion of a damped particle mğ pa- f as intended.
f())), where j is the damping (or friction)
Non- Euelddenn geometry in optimization. Many optimization algorithms assume an underly- ing geometry. For example, consider gradient descent with learning rate n, The update equa- tion, qt + n) = (()) S/ is equivalent to solving the minimization problem, q(l + 7) arg ming [Vf(q(t)), g') + 1 lg' ((t) where (, represents inner product. In otherwords, can interpret each update of gradient descent trade-off (specified by 17) between mini- mizing the loss f()), while remaining close to its current location in Euclidean distance. For non-Euclidean measures of distance would obtain different optimization algorithms. Mir- ror descent generalizes this idea by replacing the distance measure by the Bergman divergence Da(y, x) h(y) h(xx) (Vh(x); y/ I)) where h((x)) is a distance-generating function. Indeed, when the distance generating function is l((x) = | then the Bregman divergence simplifies to be the squared Euclidean distance Dr. (y. x) 1/1x y/ and mirror descent is equivalent to gradient descent. The mirror descent framework encapsulates broad array of optimization methods such as gradient descent, natural gradient descent, and others 9..
7
M(z)
Fuler. Lagrange equation
Nesterov momentum Dumped physical motion Natural gradient flow GD momentum
log 2
log
log log1/4 1/4
log
$ [
Vf= V my + 3+
log
log
$ [
log
log
h(#)
4+ FFFF///=00
log 11.12
log 2011
2(1
401-8)
Table I: The Bregman Lagrangian 4 6 provides unified description of a family of learning rules
with time dependent parameters 0t, 3r. 7t and metric h(x) which provide the algorithmic degrees of freedom.
3

---

The Bregman Lagrangian. Integrating building blocks of optimization such as damping and non- Euclidean geometry into the variational framework of mechanics, Wibisono, Wilson, and Jordan introduced the Bregman Lagrangian 4 6
(l 4, cut (Da(g + c c000.)))))) e f())
(2)
For various choices of the time-dependent factors Ctg, Br. and 7/t the Bregman Lagrangian, under the principle of least action, defines continuous approximations to an array of learning rules such as Nesterov's accelerated gradient methods [TO, [ل] and natural gradient flow [2. The flexibility of this
framework makes our analysis generalize to a broad array of learning rules, as summarized table [
Implicit acceleration and the Lagrangian formulation of gradient descent. In the continuous- time limit of diminishing learning rate 7) + 0, gradient descent q(t + 77) q(t) - V becomes a first order differential equation, gradient flow i = - V.. Unlike Nesterov's 's momentum [II] the trajectory of gradient descent with heavy ball momentum 3 as applied in deep learning [12], follows Vʃ. Here, we apply modified equation analysis
gradient flow with the rescaling (1 - B) 1 to incorporate the effect of finite learning rate and leverage the fact that the discrete steps of gradient descent closely follow the continuous-time trajectory of the second-order differential equation time. We can informally see this by Taylor expanding an update with respect to small, but finite learning rate q(t + 1) q + 7/4 + T + ((r²) and then plugging this expression to the gradient descent updates g((++)- 9(1) - V.. We keep the first order in 1) and obtain second-order differential equation / +
vv-. We call the additional term zØ. implicit acceleration. This dynamics reduces to gradient flow q = f in the continuous-time limit as expected. (Note that a complementary approach is to view the effects of discrete step size as implicil gradient regularization
on the original loss f(4) + f(()= + //V | 1SI IG (7).) Performing a similar analysis [1
accounting for the effects of the heavy ball momentum with weight decay l yields second-order differential equation.
5 (1 + (1+ (1 B) + V ky=00 =0 0,
where 41 + q represents the implicit acceleration.
Indeed, we can connect this second-orde dynamics to the Bregman Lagrangian by considering the factors O22
2(1-0) log n(1+0) 3. log 9(++5) ") ot (()+8) 1, and the distance generating function
h(x) = 1 learning rate v,, heavy ball momentum B3, the loss function f(), and weight decay constant & is
2 The Lagrangian describing the learning dynamics under gradient descent with finite
D) [T + B) 1012 - ((4)) + (112))
4, = e00+a)
4
2 Symmetry and symmetry breaking in neural networks
The variational perspective of optimization, discussed in the previous section, provides a useful lens to characterizing the path a model takes through parameter space during training. To study the geometry of this path we will turn to the tools of symmetry. In particular, in this section we will discuss not only when symmetries exist in the loss function, but also when they are broken by the learning rule.
2.1 Symmetries of the loss function
There have been number of works studying symmetries of the loss function in parameter space, such as [1] who study discrete permutation symmetries or MI who study continuous differentiable symmetries. In this work we will focus on continuous differentiable symmetries found in modern deep learning architectures. Consider one one-parameter family of differentiable maps g - Q(g. s) parameterized by a scalar s E such that (q) possesses a differentiable symmetry if its invariant to the transformation f f(Q(q, s)) = f()) for all 8. The symmetries we consider this work are: translation symmetry where Q( = q + sn
gives an identity Q(q, 0) = 9. We say the potential
and ń is a time-independent vector, rotation symmetry where Q(g, s) = R(s)y and R(s)) is a rotation matrix, and scale symmetry where Q(g. 8) (1 + +8)g Translation symmetries occur neural network for parameters immediately preceding softmax function. Rotation symmetry is one of the

---

most fundamental symmetry groups in classical mechanics and occur in neural networks with weight decay 120 2 or word embeddings [22]. Scale symmetries appear ubiquitously neural networks
for parameters immediately preceding normalization layers, such as batch normalization [3] and its
variants [23). There is also related symmetry, defined as rescale symmetry in [. present in neural networks with continuous, homogeneous activation functions (e.g. ReLU, Leaky ReLU, linear).
2.2 Symmetry breaking in the kinetic energy
In the previous section, we introduced the Lagrangian formulation of learning dynamics, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. The kinetic energy of the Bregman Lagrangian, defined as
q. = ct Di(g + e "0 = e (hlq - - - h())) - (Vh(g), e "x4)), (5)
is as important as the loss function, but its properties are less discussed. Here, we ask if the Bregman kinetic energy respects the same symmetries as the loss function, whether Ti, (Q s), Qqq s),t) To (g, q, 1).
Euclidean kinetic energy. First, we consider when h(x) = 2 |xc and study the symmetry properties of the Euclidean kinetic energy,
TE(g, 2
(6)
For translation symmetry the velocity is invariant under the transformation Q= -j and thus indicating that the Euclidean kinetic energy is invariant under spatial translation. For rotation symmetry, the temporal differentiation commutes with the rotation Q Rtj, however the Euclidean kinetic energy invariant under spatial rotation because the norm is unchanged |Ộ| = (Rá, Rq) = l However, for scale symmetry Q (1 + s) and thus the norm |Q| (1 + s) 1// explicitly depends on s. Therefore, the kinetic energy explicitly breaks the scale symmetry present in the loss function. We call this explicit symmetry breaking of the Lagrangian by the kinetic energy term Kinetic Symmetry Breaking (KSB).
Non- Non-Euclidean kinetic energy. Next, we consider the setting of general non-Euclidean metric. e=*r()
The Taylor expansion of the Bregman kinetic energy with respect to the first term logg + e ()) in Eq results in the simplification,
(4 q, = (e /2) (q. (h(())a) O (1))).
Notice, the Euclidean kinetic energy only depends on the squared norm of the velocity I2 because the Hessian is the identity matrix V2 |// = I and the higher order derivatives zero. With non-Euclidean metries, such simplifications special to the Euclidean case do not necessarily apply and we should not expect that the Bregman kinetic energy respects any symmetries in general. However, the natural gradient ()] and its elegant approximations, such as (4), are examples of non- non-Euclidean optimization methods, which have invariance properties by construction. While the invariance properties of the natural gradient descent can be broken due to discretization (25), the fast and accurate approximation of natural gradients in modern large-scale neural networks is an active area of research 226 177]
3 Noether's learning dynamics
So far, we have seen how the Bregman Lagrangian provides a unified variational description of learning dynamics, and the scale symmetry, ubiquitous in modern neural networks, induces kinetic symmetry breaking (KSB). Here, we apply Noether's 's theorem to the Bregman Lagrangian and derive Noether's Learning Dynamics (NLD), unified equality that holds for any combination of symmetry and learning rules.
Noether's theorem. In 1918 Emmy Noether famously described one of the most fundamental theorems in modern physics: every differentiable symmetry of the action has an associated conserved quantity through time [28]. Noether's theorem not only proves the existence of, but also provides an explicit expression for, the conserved quantity, often called Noether charge. Focusing just on the symmetry properties of the Lagrangian, general form of Noether's 's theorem relates the dynamics of
5

---

the Noether charge (0,C,0,Q) to the change in the Lagrangian under an infinitesimal transformation O,C as
de 10, Q.Q) = a,C,
(8)
where we evaluate all derivatives at the identity element of the symmetry group (s = 0). We can derive this relationship as 0, = (0,C, 0,Q) + (0,C, = at (3,C, (,), where we plugged in the Euler-Lagrange equation 0,C d
at (O₂C) and rearranged the total time derivative. The most common application of Noether's theorem physics is when the Lagrangian, including both kinetic and potential energies, is symmetric under the infinitesimal transformation, 0, 0, implying Noether charge is conserved through time. For example, if we assume symmetry of the Lagrangian due to homogeneity or isotropy of space, Noether's ss theorem directly implies the conservation of momentum or angular momentum respectively. However, as shown in Sec. E. the Bregman kinetic energy for learning systems is often asymmetric to the symmetry of the potential function, implying that the resulting Noether charge not conserved.
Noether's learning dynamics. Here, we derive a general equation describing the sinetic-asymmetry- induced dynamics for the Noether charge in learning systems. If the Bregman kinetic energy does not respect the symmetry of the loss function, then Eq. g becomes
d dt (0,C, 0,Q) = 0,TT
implying kinetic symmetry breaking (0,T% # 0) induces motion of the Noether charge. To build better intuition for this motion and inspired by the definition of the generalized momentum, we define the Bregman momentum as,
D = e 7 V - Vh(g).
(10)
Expanding Eq. 9 by plugging in the definition of the Bregman Lagrangian Eq. 2 the Bregman kinetic
energy Eq. 5 and the Bregman momentum Eq 24 (details are described in SI Sec. B. we obtain Noether's 's Learning Dynamics (NLD),
charge
damping
kinetic asymmetry
Bon-Euclideun metric
(11)
+h ( 0,Q) (An. 0,Q) (A 0,Q) de
+ c (An - -e---=" 0,Q).
Each term in NLD has an intuitive meaning. Recall the conventional setting in classical mechanics, where the entire Lagrangian is symmetric, there is no damping, and the metric is Euclidean. In this case, the Noether charge is the inner product of momentum and the generator of the symmetry transformation (e a,Q). The first term of NLD, (An, Q,Q), is the inner product of the Bregman momentum and the generator of the symmetry transformation. In the setting of no damping, this terms is a generalization of the conventional Noether charge that encompasses non-Euclidean metrics. Indeed, in the Euclidean case, we can see that Ag Noether charge as expected. The second term of NLD. %e (An, a,Q),, represents the contribution of damping and the term becomes zero when Y is constant and there is no damping. The third term of NLD, (An, 0₂ Q), represents the contribution of kinetic symmetry breaking. Under the Euclidean metric, this term is proportional (4, a,Q). For translation symmetry, a 0 and this term is zero (A,, 0) = O. Similarly, for rotation symmetry, this term is zero (R)= R) = (ÿ. (0, = 0. Here, we used the fact that the generator of rotation is skew-symmetric matrix. However, for scale symmetry, Q n unique to learning systems, this term is proportional to lg] and thus present. The fourth term of NLD, eah (An, e -a ( 0,Q), represents the contribution of the non-Fuclidean geometry and is zero under the Fuclidean metric h(x)) 1 |0/22
ax and the term reduces the conventional
Noether's learning dynamics under gradient flow. Next, we show that in the over-damped limit of NLD, the charge (An, a,Q) itself, rather than its time-derivative becomes zero. Consider Eq when a₁- = - log m. Be = log m, 7t- Lt.
4 m dt (A,,0,0)) 0,Q) + n 0,Q) = =mā,T.
(12)
When 0,T 0. this equation describes how the charge decays exponentially with damping (An, 0,Q) # as in Fig.
3 In the mass-less limit + 0 with fixed friction μ,, the terms
proportional to m in Eq. 12 all vanish and we get
( h. 0,Q) = 0.
(13)

---

Under damped m >> μ
which states that the charge itself is zero, rather than its time derivative. Furthermore, in the Euclidean settings, the result simplifies to (j. 0,Q) = 0. As a physical analogy, this means that quantities such as momentum or angular momentum becomes zero in the over-damped limit. This formula unifies an array of conservation properties no- ticed under gradient flow [ 29 30
0,))
m/u
with a formal theoretical connection to Noether's ' theorem.
RI
32
Over damped m < μ
Overall, just like how Noether's 's theorem [28 unified an array of conservation laws and provided a theoretical foun- dation to discover new ones in physics, we have further unified conservation laws previously observed learning systems [1
Time: I
Figure 3: Noether's theorem with damping. Without KSB, the charge decays exponentially, but in the under- damped limit m/p > 1, the charge be- comes a conserved quantity. In the over- damped limit m - O, the charge expo- nentially diminishes to zero
31 52 and generalized these results for any combination of differentiable symmetries in neu- ral network architectures and learning rules (e.g., natural gradient descent, Nesterov's ' accelerated gradient descent, accelerated mirror descent, and cubic-regularized New- ton's 's method).
29 3C
4 Kinetic symmetry breaking induces implicit adaptive optimization
In the previous sections, we identified the mechanism of kinetic symmetry breaking and derived the Noether's 's learning dynamics (NLD). Here, we explore a role of symmetry breaking neural networks by applying NLD to study auto rate-tuning by normalization layers, originally studied by Arora, Li, and Lyu 63 In particular, we establish an exact theoretical analogy between two seemingly unrelated key components of modern deep learning: normalization and adaptive optimization.
Consider a neural network model with normalization layers (such as batch normalization [] group normalization <<] layer normalization (35 and others (23)), which introduces scale symmetry in the potential (loss) function f(11 + s)g) f(g). We train this model with gradient descent with a finite learning rate 17. momentum B, and weight decay k. We are interested in the evolution of the potential function through the learning dynamics f(q(t)). Due the scale symmetry of the network introduced by the normalization layer, the time evolution on the loss landscape only depends on the directional vector q = q/ /41 of the parameters throughout training, (((())) = f(((t))
What is the equation of motion for the directional vector? To solve for the dynamics, we first move to polar coordinate system q = r where r = [g] is the radial norm and q is the directional vector. This is akin to how we solve for the motion of a planet in a central field in classical mechanics. From Eq 4 the Lagrangian for this learning system in polar coordinates is,
Le =
Et C
m 2
2
+
m
2
))²²
k
2²
S(0) + A(t) (laP - -1),
(14)
2
2
where m unit norm constraint. The Lagrangian formulation facilitates coordinate transformations, because the form of the LagranLa equation is invariant under coordinate transformations. Taking advantage of this fact, we apply the Euler-Lagrange equation to derive the equation of motion for the directional vector as (see SI Sec. C for the derivation)
x(L+0) 2 1- 3, and C'g)) = Q/2 1 = O is a Lagrange multiplier enforcing the
1 Vf((), (2)
mă + ف1
(15)
where Vf(g) represents the gradient of f evaluated at q. Notice the striking similarity of this equation of motion with the equation of motion of the original parameters mă + u4 ff(()) This implies that the time evolution of the loss for network with normalization layers trained by gradient descent is equivalent to the time evolution of the loss for an identical network with unit filter norm trained by an adaptive optimizer which scales the gradients by factor of (1))
Strikingly, NL.D describes the motion of this adaptive scaling factor (1). When a, = - log m. and h(x)= = 1 | then the charge is (An, 0,Q) m(ğ.9) mā,, and Eq I1 including the effect
7

---

of weight decay k, implies
2m p
m0r u&,r²² -2kr²
V /(())
(16)
where we kept the first order terms in m and k. In the over-damped regime, ignoring inertia, we get a change in the squared parameter norm (1) as (see SI Sec. D for the derivation):
2m((1 + 3) r (t) V (I
31- [ -e- VV(((T)) (V + + 1
(17)
Each term in this implicit adaptive learning rate has an intuitive meaning. The first integral term keeps track of the recent history of the magnitude of gradient norms [ that the filters with fixed unit norm receive. The weight decay k controls the time-scale of the short-term memory of
the accumulated gradient norms through the exponential kernel e (t-T), When k all the gradients will be accumulated resembling an AdaGrad optimizer BG Notably, this integral term proportional the learning rate 7) implying that this mechanism of adaptive optimization is only present with finite learning rate. The second term represents an exponentially decaying memory of initialization. The finite filter norm at initialization y (0) prevents instability of learning dynamics avoiding division by zero.
So how does this adaptive scaling of the gradient step sizes compare with hand-designed adaptive Here, we develop a continuous time model of the explicit adaptive optimizer R.SPPrp and discover that the functional form of normalization layers' effective learning rate schedule Eq. 17 functionally agrees with that of RMSProp. The RMSProp algorithm is a recursive update rule expressed gu+1 qn 7 f(g)), where the gradient is vo.
optimizers successful in deep learning [36] E7 38)?
scaled by a factor of vG,- The factor G, keeps track of the history of the gradient norms following Gn+1 = pG,, + (1 - (q.)l where is a hyperparameter. Solving the continuous time dynamics 17 dG
dt = (1 p)G(t) - (1 ()())(((T)) of the adaptive scaling factor of R.MSProp.
we obtain
L. - (f((()))) +e -
VG(t) -
(18)
Strikingly, we find that this functional form induced by RMSProp exactly matches with the implicit adaptive learning rate schedule Eq to normalization layers suggesting potential benefits of the implicit adaptive optimization induced by normalization.
In summary, our theory has yielded the following new insights:
1. The optimization geometry of SGD with a finite step size momentum B and weight decay i together breaks the scale symmetry of the loss (e.g.. with normalization layers) and generates an implicit mechanism of adaptive optimization akin to RMSProp.
2. The implicit adaptive optimization mechanism exists when the learning dynamics is in the underdamped regime m/μ x n1(11 3) >> 0 and thus requires the learning rate to be finite.
3. Momentum significantly amplifies the effect of adaptive optimization by (1- 1900 folds as in Eq. 17 even in the standard setting of 8 = 0.9.
14.9
4. For scalc-invariant parameters, symmetry breaking due to weight decay 1 - k plays a role similar to the discount factor p for the cumulative gradient norm in RMSProp.
Finally, we empirically validate these predictions by training convolutional neural networks with batch normalization (VGG11) a large data set (Tiny-ImageNe) under various hyperparameter settings as in Fig. (see SI Sec. E In addition to the standard training with an "unconstrained filter norm" (pink), we perform ablation experiments with a "constrained filter norm" (blue). As concrete example, consider a convolutional filter E RD preceding the batch normalization layer. The loss is invariant under the scaling transformation of the filter norm f(1 + s)g) f(q). To test the presence and benefits of implicit adaptive optimization to the learning dynamics of the loss, we freeze the
.(t) el)) [a(0) after each dynamics of the filter norm by repeatedly performing the operation q(t) + [o(e]
step of optimization KK). This enforces the filter norm to be fixed [q()) = [g()) throughout training for any t. When trained with constant small learning rate 77 0.001, the final test accuracy (A-1)
8

---

(A-1)
Learning !
(B-1)
Momentum B
(C-1) Weight decay k
innconstraineed filter norm
Unconstralned filter norm
Test Test accuracy
Test Test accuracy
56-
Test accuracy
45
56-
28 o01
Constralned filter norm
mR
0.01
01
Constroined filter norm
== (1 - f01
16
Base cine:
30
k
10
(A-2)
1
0.001
7
0.03
(B-2)
B= 0.9
(C-2)
Constrained fittrr norm
Constrained filter norm
Loss
W
Loss
x
Inconstrained filter norm
Loss
2
k m 10
A = 0
Jnconstrained
Filter
20
42
6o
20 40
102
50
100
50
160
Epochs
Epochs
Epochs
Epochs
Epochs
Figure 4: Broken-symmetry induced dynamics the filter norm is a key to successful learning.
(VGG-11 models trained on Iiny-ImageNet.) (A-1) The final test accuracy and (A-2) the loss dynamies of models trained at various constant learning rates y with / = 0.9. (B-1) The final test accuracy and (B-2) the loss dynamics of models trained with various momentum 3 with learning rate drops. In agreement with our theory, the advantage of implicit adaptive optimization is present when trained with large learning rates and large momentum. (C-1) The final test accuracy and (C-2) the loss dynamics when the model is trained with various weight decay rates k. validating its benefits.
and loss dynamics (A-2) of models with unconstrained filter norm (pink) and constrained filter norm (blue) are almost identical. However, in large learning rate regime 7) 0.03, the models trained with unconstrained filter norm (pink) outperforms the models trained with constrained filter norm (blue) demonstrating the existence and benefits of implicit adaptive optimization. Similarly, we confirm our prediction that the presence of momentum 3 ~ 0.9 is essential to amplify the effects of implicit adaptive optimization, seen in test accuracy (B-1) and loss dynamics (B-2). Finally, we demonstrate the benefits of symmetry breaking due to weight decay h which acts in analogy with the discounting factor p for the cumulative gradient norms of RMSProp. Indeed, the final test accuracy (C-1) and the learning dynamics of the loss (C-2) are both enhanced by the presence of weight decay k in agreement with [3] Overall, the hyperparameters of the modern optimization learning rate, momentum, and weight decay- each play an important role in breaking the scale symmetry of the loss function and providing an mechanism of implicit adaptive optimization, a key to successful learning in neural networks.
Related works and future directions
Finite learning rates, modified gradient flow, and generalization. Recently, many works have demonstrated that initial large learning rate is key to successful training 40 41
42
A3 Ad]
Another line of works [1 [3 14. 15 16. 45 46) have incorporated the effect of a finite learning rate into gradient flow by adding modification term. A common approach applies modified equation analysis replace the original loss function f by modified loss function j = ʃ + 7 /|2 I1 LE! 15 where the additional term is called implicit gradient regularization. Intuitively, this term biases the learning dynamics towards wider minima with smaller gradient norms. To test this intuition empirically, recent works have explicitly added the gradient regularization term 4 /22 to the original loss and have shown that the models trained with the modified loss generalize well even with small learning rates [o or full-batch [77. In this work, we have taken a complementary approach and applied modified equation analysis in time add an implicit acceleration term 3 [ 45). When training with a finite learning rate, this implicit acceleration term gives rise to the implicit adaptation of the gradients. Remarkably, the two complementary pictures (i.e., implicit gradient regularization and implicit acceleration) offer a consistent view on how the large learning rates provide mechanisms to properly regularize the norm of gradients to avoid training instabilities. An idea for future work is to directly simulate gradient flow with implicit acceleration, instead of gradient flow with [15 16 Li or without +7 implicit gradient regularization, to keep the benefits of finite learning rates while avoiding the computations of the Hessian gradient products.

---

Geometry of optimization. The geometric property of optimization has long history of study. For are invariant under affine
example, natural gradient by Amari (9 and its approximations 2
4
transformations including scale transformation due to normalization layers
39 With the rise of deep learning, there has been recent developments to efficiently approximate and apply natural gradients to [x When training deep neural networks, we still commonly use gradient- based optimization methods represented by stochastic gradient descent with momentum [02). An array of adaptive optimization methods [36 Rr ESI have been proposed partially motivated as diagonal approximations of the Fisher Information Matrix (see recent discussions questioning the validity of the approximations [54)) A pioneering work [551 by Neyshabur et al. has identified that gradient descent does not respect the rescale symmetry introduced by ReLU activations, followed by works proposing solve such discrepancies by enforcing symmetry in the learning dynamics [56] 157. While above works have always tried to remove such broken symmetries, we demonstrated theoretically and empirically that the broken-symmetry can play an important role in deep learning. thus providing a unique counterexample. A future direction is to extend our analysis to the case of rescale symmetry of the homogemeous activation functions such as ReLU..
large neural networks (24 26 48 50 51 22
Conservation laws in neural networks. Balancing properties of network parameters have been identified in gradient flow dynamics of neural networks with linear 24 and ReLU EC [3 activation functions. These properties have been leveraged to better understand the role of over- wwver-arrmtterization 1291, implicit regularization [EO,, network pruning [31], continual learning [58]. and self self-supervised learning 32 A recent work identified yymmetry-inducee geometric structures of the loss and studied their roles under gradient flow dynamics with modified loss predicting the parameter dynamics M However, despite the ubiquity of conservation laws in deep learning, no formal connection to the Noether's theorem has been made due to the previous focus the gradient flow dynamics. Here, took an alternative path starting from a Lagrangian formulation, which encapsulates gradient flow as the mass-less limit. From this perspective, we unified all the conservation laws under the umbrella of Noether's s theorem, and provided its generalization for accelerated and non-Euclidean methods in optimization. In future works, we can harness the generality of our theory to investigate the dynamics of more complex learning rules such as natural gradient descent with or without Nesterov's s momentum in combination with any differentiable symmetries.
Auto-rate tuning by normalization layers. Batch normalization 3 and its variants
are essential train deep neural network models 6 providing multiple benefits as elegantly reviewed in [61] An important benefit of normalization layers is to enable stable training with large learning [33 rigorously studied the benefits of monotonically decreasing the
35
59
rates 2 A pioneering work
effective learning rate in simple setting without momentum nor weight decay. Another line of works inspiring an automatic learning rate scheduler 551 Li and Arora 366] theoretically studied the setting with weight
noticed the role of weight decay to effectively enforce higher learning rate 3 63 34
decay and momentum to design a exponential learning rate schedule that still leads to successful training counter- intuitr-ituiiivell.. In the present analysis, we leveraged the Lagrangian formulation to handle the complexity of modern practical optimizers and derived an exact solution describing the time-evolution of the effective learning rate in the presence of momentum and weight decay. By description of adaptive optimizers in parallel, we discovered that the
developing a continuous-tin functional form of the effective learning rate exactly matches with that of RMSProp. where momentum significantly amplifies the effect, and weight decay acts in exact analogy with the discounting factor. In the context of thorough empirical studies on adaptive optimizers |67 68 69), our results raise the hypothesis that the benefit of an adaptive optimizers may not be as significant in networks with normalization layers as long as the hyper-paramettrs, in particular momentum, are properly tuned.
6 Conclusion.
In nature, symmetry governs regularities, while symmetry breaking brings texture which include superconductivity [Z]], nucleon mass 72), ring attractor neural networks [
70
examples of
and the Higgs boson 76). Despite many attempts to bring symmetry to neural networks 55
symmetry breaking has rarely been discussed
design principle [22), or even treated as an obstacle to successful learning [55] (56] 55). In this work, we have discovered a novel role of explicit symmetry breaking in learning systems by applying a Lagrangian formulation neural networks. Overall, understanding not only when symmetries exist, but how they are broken is essential to discover geometric design principles in neural networks.
10

---

Acknowledgments and Disclosure of Funding
We thank Max Aalto, Kyosuke Adachi, Yuto Ashida, Fatih Dinc, Surya Ganguli. Kyogo Kawaguchi. Ekdeep Singh Lubana, Akiko Tanaka, Hajime Tanaka, Atsushi Yamamura, and Daniel Yamins for helpful discussions, D.K. thanks the Stanford Data Science Scholars program and NTT Research for support.
References
[1] Daniel Kunin, Javier Sagastuy- -Benna, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021. URL /ttttp:e//peereviess http:/oorummdddqqLLAAAAA..
[2] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021. URL
https://arxiv wwt.:ogg///baeaeeee
13478
[3] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pages 448- 456, Lille, France, 07-09 Jul 2015. PMLR. URL http:://procee///naeee mlr press/v37/ hoffe15.html
[4] Andre Wibisono and Ashia Wilson. On accelerated methods in optimization, 2015. URL
https://arxiv.om org/ abs/1509 1509.03616 03616
[5] Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz. Mechanics: Volume 1, volume 1. Butterworth--Heinmman. 1976.
[6] Andre Wibisono, Ashia Wilson, and Michael Jordan. variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences, (3344)):E7351- E7358, 2016.
[7] Ashia C Wilson, Benjamin Recht, and Michael Jordan, Iyapunov analysis of momentum methods in optimization. arXiv preprint 16X::: 2016. https:////ax org/ abs/1611.0263
[8] Kanai. On the quantization of the dissipative systems. of Theoretiall Physscss 440- 1948. URL http:://wwaaacccccoooooocoppppta login=true
[9] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2): 251-276, 1998.
[10] Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/k* 2). Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
[11] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nes- terov's accelerated gradient method: Theory and insights. In Advances in neural information processing systems, pages 2510-2518, 2014.
[12] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1144 1147, 2013.
[13] Nikola B. Kovachki and Andrew M. Stuart. Continuous time analysis of momentum methods. Journal of Machine Learning Research, 22(17):1 2021. URL http:://jmm: http:://mmmlrroro/ppeee ort.l/rrrrppppprs//// v22/19 466. html
[14] Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pages 2101-21 10, 2017.

---

[15] David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint MxXiv:2009.11662 2020.
[16] Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101. 2021.
[17] Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochas- tic training is not necessary for generalization. arXiv preprint arXiv:2 19, 2021.
[18] Berfin Simşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks:
Symmetries and invariances. arXiv preprint arXiv:2105. /2221, 2021.
[19] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296, 2021.
[20] Daniel Kunin, Jonathan M Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. arXiv preprint 1xXiv:1901.08160 2019.
[21] Arthur Jacot, François Ged, Franck Gabriel, Berfin Şimşek, and Clément Hongler. Deep linear networks dynamics: Low-rank biases induced by initialization scale and 12 regularization. arXiv preprint arXiv:2 15933, 2021.
[22] Robert Bamler and Stephan Mandt. Improving optimization for models with continuous symmetry breaking. In International Conference on Machine Learning, pages 423-432. PMI.R, 2018.
[23] Ekdeep Singh Lubana, Robert P Dick, and Hidenori Tanaka. Beyond batchnorm: Towards a general understanding of normalization deep learning. arXiv preprint arXiv:2106.05955 2021.
[24] James Martens and Roger Grosse. Optimizing neural networks with frantcder-factored approx- imate curvature. In International conference on machine learning, pages 2408-2417. PMLR, 2015.
[25] Yang Song, Jiaming Song, and Stefano Ermon. Accelerating natural gradient with higher-order invariance. In International Conference on Machine Learning, pages 4713-4722. PMLR, 2018
[26] James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:14122 193, 2014.
[27] Kevin Luk and Roger Grosse. A coordinate-free construction of scalable natural gradient. arXiv preprint arXiv: 1808. 10340, 2018
[28] Emmy Noether. Invariante variationsproblemme, math-phys. Klasse pp235-257, 1918.
[29] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. arXiv preprint arXiv: arXiv: /80p:060200 2018.
[30] Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep ho- mogeneous models: Layers are automatically balanced. In Advances Neural Information Processing Systems, pages 384-395, 2018.
[31] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, volume 33. 2020. URL httptt/ts////rooeiiingg
.wwrtiinnnnnnen............ www.ipu.......m cc/paper/ .ddf
[32] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning
dynamics without contrastive pairs. arXiv preprint arXiv:2102.068100 2021.
[33] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In International Conference on Learning Representations, 2019. URL.
https://openrevieet hitp://ww..eet/oooommmdddddaaaa -A9FX
12

---

[34] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[35] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization. arXiv preprint 107.v:160..06450. 2016.
[36] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[37] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):
26-31, 2012.
[38] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[39] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. International Conference on Learning Representations, 2019. URL
https://openrevieer http::///gaaarrree..errrrrrrrrtp
[40] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609 04836, 2016.
[41] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprint ArXiv:1907.04590 2019.
[42] Guillaume Leclerc and Aleksander Madry. The two regimes of deep network training, arXiv preprint arXiv:2002. 10376, 2020.
[43] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02215. 2020.
[44] Stanislaw Jastrzebski, Macicj Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09572 2020.
[45] Daniel Kunin, Javier Iigastuy-Brendl Lauren Gillespic. Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel LK Yamins. Rethinking the limiting dynamics of sgd: modified loss, phase space oscillations, and anomalous diffusion. arXiv preprint 1rXiv:2107.09133 2021.
[46] Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). arXiv preprint arXiv:2 12470, 2021.
[47] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gra- dient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021.
[48] Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for con- volution layers. In International Conference on Machine Learning, pages 573-582, PMLR, 2016.
[49] Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. In NIPS, pages 849-856. Citeseer, 2007.
[50] James Martens. Deep learning via hessian-free optimization. In ICML, volume 27, pages 735 735,772,220 2010.
[51] Yann Ollivier. Riemannian metrics for neural networks i: feedforward networks. Information and Inference: A Journal of the IMA, 108 2015.
[52] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584 2013.
13

---

[53] Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.
[54] Frederik Kunstner, Lukas Balles, and Philipp Hennig, Limitations of the empirical fisher approximation for natural gradient descent. arXiv preprint 1rXiv:1905.12555 2019.
[55] Behnam Neyshabur, Russ Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, pages 2422-2430, 2015.
[56] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. arXiv preprint arXiv:/511.0000 2015.
[57] Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pages
2017.
[58] Ekdeep Singh Lubana, Puja Trivedi, and Robert P Dick. Rethinking quadratic regularizers: Explicit movement regularization for continual learning. arXiv preprint arXiv:2102.02805, 2021.
[59] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizztioon to accelcrate training of deep neural networks. In Advances in neural information processing systems, pages 901-909, 2016.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[61] Roger Grosse Lecture 5: Adaptive gradient methods, normalization, and weight decay. CSC254/ Winter 202/ Topics in Machine Learning: Neural Net Training Dynamics, 2021. URL https: cs toronto. eeuee coaroosse/courrss//scsc/// 2021/
[62] Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL htttss:/pproceddimgg meggissneuripp..co ceeurippscc/ppper/2018 F1116/80722235334477776778888480
pdf
[63] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pages 2160-2170, 2018
[64] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint 17Xiv:1706.05550, 2017.
[65] Aitor Lewkowycz. How to decay your learning rate. arXiv preprint arXiv:2103. 2021.
[66] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In International Conference on Learning Representations, 2020. URL /ttl:tttts///operrrrii
het//forum?id=rrgg8Beee
[67] Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 190.5:1910 2019.
[68] Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley- benchmarking deep learning optimizers. In International Conference on Machine Learning, pages 9367 9376. PMLR, 2021.
[69] Zachary Nado, Justin M Gilmer, Christopher J Shallue, Rohan Anil, and George E Dahl. A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. arXiv preprint 10l.:2002006366,, 2021.
[70] David J Gross. The role of symmetry fundamental physies. Proceedings of the National Academy of Sciences,
1996.
14

---

[71] H Suhl, BT Matthias, and LR Walker. Bardeen- scieesce theory of superconductivity in the case of overlapping bands. Physical Review Letters, 3(12):552, 1959.
[72] Yoichiro Nambu and Giovanni Jona-Lasinio. Dynamical model of elementary particles based on an analogy with superconductiiity.. i. Physical review, 122(1):345, 1961.
[73] Shun-ichi Amari Dynamics of pattern formation in lateral-inhibition type neural fields. Biolog- ical cybernetics, 27(2):77-87, 1977.
[74] Rani Ben-Yishat R Lev Bar-Or, and Haim Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, 92(9):3844-3845 1995.
[75] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the drosophila central brain. Science, 356(6340):849-885 -539 2017.
[76] Peter Ware Higgs, Broken symmetries, massless particles and gauge fields. Phys. Lett., 12: 132-133, 1964.
[77] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics of deep neural networks with batch normalization and weight decay. arXiv preprint arXiv:2006.08419 2020.
[78] Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate, Advances Neural Information Processing Systems, 33, 2020.
15


Reviews:
[{"review_id": "eqgyu-ab3cL", "reviewer": "Reviewer_ngQE", "summary": "Based on the Euler-Lagrangian framework, this manuscript develops the theoretical method to model the learning dynamics of DNNs as symmetry breaking of the likelihood by the kinetic energy. \nThe model is expressed as the time evolution of Noether-current.\nAs a result, the author found that the learning dynamics of batch normalization are similar to RMSProp from the viewpoint of symmetry breaking.\nTheir findings deepen our understanding of batch normalization.\nThis study is unique not only because of the findings on batch normalization mechanics but also because it links the symmetry-breaking process of the likelihood function with the learning theory.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The study about elucidating implicit adaptive optimization from the viewpoint of symmetry is excellent and unique, so this manuscript has originality and significance worthy of publication.\nOn the other hand, I can not understand some points related to the effectiveness of the theory as below.\n\nThe first concern is that there may be few cases in DNNs where the symmetry of the potential function is known.\nFor example, in their theory building on BatchNorm, the authors state:\n\"We investigate this physical system because the learning dynamics\n264 of the most successful deep learning systems are exactly analogous to this system. For example, BatchNorm layers, ubiquitous in modern successful deep learning architectures, introduce scale symmetry in the parameter space q of the potential (loss) function.\"\nHowever, I can not understand that BatchNormalization introduce scale invariance when it is subjected to an activation function such as sigmoid? (I can understand this for if the parameters are the regression coefficients of linear regression.)\nAlso, I understand that the space q is a parameter of the BatchNorm layer when considering the dynamics of BatchNorm.\nAnd I understand that for the parameters of the other layers, the potential function is not symmetric to the scale transformation of these parameters.\nTherefore, the symmetry of the parameters of the other layers is unknown.\nFrom these understandings, I think it does not seem appropriate to compare the dynamics of RMSProp with those of DNN, since the space q in RMSProp is all the parameters of DNN.\nBy the way, isn't \"f((1+s)q)=q\" in L263 \"f((1+s)q)=f(q)\"?\n\nThe second is about the difficulty of the Lagrangian approach.\nIn general, it seems difficult to construct a Lagrangian that models the dynamics of a dissipative system.\nI understood that the Bregman Lagrangian was constructed based on the Lagrangian of a known dissipative system called damped oscillation.\nTherefore, when the time evolution of the error has a plateau, which is often the case in DNN training, the learning dynamics cannot be a damped oscillation, and that is why Bregman Lagrangian cannot be used.\nTherefore, I could not understand the validity of theoretical analysis assuming damped oscillations.\n\n\nThere are some minor concern as below:\n\n\nAround L168: Since q is supposed to be a generalized coordinate system, I think that authors should not make the assumption that $q = \\dot{q}$ and $Q = \\dot{Q}$.\nIn that case, I think that the premise of the discussion in this vicinity is no longer valid.\n\nL119: There is no definition of bra-ket which represents the inner product.\n\nL238: \",,\" might be typo\n\nL275: \"as$m\" might be \"as $m\"\n\nL295: \"| as\" might be \"as\"\n\nL331, L332: There is no apparent definition of \"unconstrained filter norm\" and \"constrained filter norm\".\n                    I think it is friendly for reader to describe these definition more clearly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have described the concerns about limitation in Main Review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12hours"}, "extra_text": {}}, {"review_id": "QW6pXf00_kv", "reviewer": "Reviewer_6ME4", "summary": "The gradient descent with a momentum can be regarded as the Newtonian equation with damping. The output of the batch normalization is invariant to the scale of the input (i.e., a scale symmetry). When a BN is inserted after a convolution, the whole network is invariant to the weight scale. A typical gradient descent algorithm ignores (i.e., breaks) this symmetry. Thanks to this symmetry breaking, the learning rate is implicitly adapted, and the adaptation is equivalent to RMSProp's.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly, the main result is very interesting. The perspective from the symmetry breaking is unique and provides a new insight to the reason why BN works. The reviewer still has several questions about experiment and relationship with physics.\n\n1) In the experiment, the VGG with BN is compared with a \"constrained filter norm\" variant, whose weight is normalized after each update. Even then, the gradient and the momentum term are calculated under the assumption of no constraints. This gap might affect the performance negatively. Hence, the performance degradation of the \"constrained filter norm\" variant might be insufficient to prove the effectiveness of the implicit adaptation by BN.\n\n2) How necessary is the perspective of the Noether's theorem and symmetry breaking? While a physical analogy is well known to the GD with momentum, the equivalence between (17) and (18) might be derived without it.\n\n3) Some prior works such as [34] are introduced in Section 5. In particular, [34] also provided the theoretical explanation of the reason why BN works. The relationship with and difference from prior works should be described in detail.\n\n4) The title might be excessive advertising. The theoretical analysis does not describe the general role of the symmetry breaking in deep learning, but it is applicable only to BN\n\n5) At line 62, the sentence of BN's symmetry might have a typo. Correctly, it might be BN((1+s)qx)=BN(qx).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The theoretical analysis is only applicable to the batch normalization, while the title seems to be more general. No negative societal impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "AdfVQapo8P", "reviewer": "Reviewer_YvtG", "summary": "The authors identify a \"kinetic asymmetry\" in the learning dynamics under gradient flow, where the kinetic energy is not invariant to the symmetries of the potential energy. An example is scaling: even when the network function is invariant to scaling or ReLUs or batch norm inputs, the learning dynamics are not. They derive a genealized version of Noether's theorem which takes this asymmetry into account and use it to study the implict adaptive optimization exhibited by batch normalization, showing that it is the same as that of RMSprop.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic and techniques of this paper are well outside of my comfort zone so I can only provide a high-level overview. From this high level, I find the premise of the paper exciting. The interaction between Lagrangian dynamics and optimization in machine learning is not new (e.g., the cited work of Wibisono, Wilson, and Jordan), but connections to  symmetry breaking and Noether's theorem, at least in the context of deep learning, seem to be original. The demonstrated consequence that the dynamics induced by batch normalization are the same as those of RMSprop is very cute. The paper is clearly written (although with quite a few misprints) and the provided examples are helpful.\n\nMy main objection is towards the generality of results and the corresponding claims. For example, the statement that \"kinetic symmetry breaking is a key design principle to the success of a learning system\" could be mollified. Symmetry breaking is certainly a beautiful perspective but it is one out of many.\n\nWhile the theory and the symmetry-broken Noether's law hold generally, the only example in the paper is the scale symmetry exhibited by ReLU activations and batch norm (since scaling does not preserve kinetic energy). Could the authors suggest or speculate other interesting symmetries in deep learning that do not preserve kinetic energy? Another question is about using your insights opportunely---since learning dynamics is not invariant under symmetries, symmetries may be used to improve performance or accelerate training. On a related note: is there a connection to quiver representations of neural nets https://arxiv.org/abs/2007.12213, https://arxiv.org/abs/2107.02550?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}, {"review_id": "3XpiY3x5jua", "reviewer": "Reviewer_L9t2", "summary": "The paper considers the role of symmetries in the learning dynamic by considering a continuous time formulation of the learning process in Lagrangian formalism. In such formulation there are \"kinetic\" and \"potential\" components to the dynamics, the former arising from the learning rules employed during optimization and the latter generated by the loss function landscape. Authors argue that for a large class of practically relevant network architectures there are symmetries that are present in the \"potential\" component, but not in the \"kinetic\" part.\n\nStarting with Bergman Lagrangian formulation of the learning dynamics, authors derive the equations for time evolution of the Noether's charge associated with symmetry transformation (assumed to be present) in the network architecture.\n\nFor the case on scale symmetry introduces by BatchNormalization, authors observe that in case of training with gradient descent + momentum + weight decay; the training dynamics effectively reduces to a learning dynamics with an adaptive scale where the scale is related to the associated Noether's charge. This suggests that kinetic asymmetry induces an implicit adaptive property to the optimization symmetry. A connection between RMSProp and implicit adaptation is established based on continuous time learning model.\n\nLastly, authors test theoretical predictions by training a set of models verifying theoretical predictions on Tiny ImageNet dataset. For comparison, the adaptive rescaling effect is artfully removed by renormalizing the parameters throughout training. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The role of symmetries in deep learning is an active research area. The paper studies a relevant problem of investigating how symmetry properties of the model, combined with asymmetry of the learning rules, give rise to non-trivial adaptive learning dynamics. To the best of my knowledge the presented results are novel and interesting. Connections between architecture and model components are often highly important in practical settings for guiding the design decisions and understanding the interactions between architectures and optimization.\n\nThe paper is clearly written and cites relevant related works. One caveat related to the presentation of the related work is that it does not acknowledge the fact that Noether's theorem has been used in recent works (e.g. reference [14]). The hypotheses are adequately tested. The work reads complete and insightful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work does not have negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following in the tradition of physics, the paper considers the implications of differentiable symmetries exhibited by a given learning task for the dynamics of the associated learning process. Although similar ideas have previously appeared in the literature ([14], in particular), an explicit study of the suggested Lagrangian framework is conceptually new. The reviewers found this framework interesting and insightful, and appreciated the unique perspective it provides on the connection between batch normalization and the popular optimization method RMSProp.
Decision: Accept (Poster)