Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks
Year: 2022

Abstract:
In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the geometry of learning dynamics in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify kinetic symmetry breaking (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether’s theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge: Noether’s Learning Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of implicit adaptive optimization, establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.

text: q0 q1
q2
Physical space
(b) Classical Mechanics
q0 q1
qN
Parameter space
(a) Learning Dynamics
Figure 1: Learning dynamics in analogy to classical mechanics. (a) Learning dynamics of a neural network in parameter space. (b) Motion of a particle in physical space.
With the rapid increase in available data and computational power, artificial neural networks have become an essential tool both in science and engineering. To obtain an accurate model, one has to overcome the challenge of tuning millions of randomly initialized parameters q 2 RN to minimize the loss function f(q) 2 R, a measure of the discrepancy between the model’s predictions from the input data and the provided truth. In minimizing the loss, the learning rule specifies how to iteratively update these parameters based on the recent trajectory and the local geometry of the non-convex loss landscape. We can represent the repeated updates of the millions of parameters
during learning as discrete movements of a point in high-dimensional parameter space (Fig. 1a). The architecture shapes the static loss landscape and the learning rule governs the motion of the model through parameter space during training. However, it is the complex interaction of the architecture and learning rule which determines the final state of the trained model. Inspired by tools and ideas from classical mechanics (Fig. 1b), we develop a theoretical framework which unifies the effects of the architecture and learning rule on the geometry of the learning dynamics.
* Correspondence to: Hidenori Tanaka <hidenori.tanaka@ntt-research.com> 2 Work done during an internship at Physics & Informatics Laboratories, NTT Research, Inc.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Symmetry as a mathematical lens. One of the most powerful mathematical tools for studying the geometry of complex objects is symmetry. A symmetry of an object is a transformation that when applied to the object leaves certain properties unchanged. For example, consider how a circle can be rotated around its center without changing its overall appearance. In otherwords, a circle has a continuous rotational symmetry. In particular, the form of the symmetry transformation constrains the form of the object, the circle is the only 2D shape that has continuous rotational symmetry. Objects with symmetry are not restricted to geometric shapes, but could be functions or even dynamical systems. Recently, the lens of symmetry has been applied in the study of neural networks [1, 2]. A recent work [1] identified an array of symmetries in neural networks to study the geometry of the loss function f(q) in the parameter space. For example, batch normalization [3] defined as BN(qx) = (qx E[qx])/ p Var[qx] introduces a scale symmetry in the parameters immediately preceding the normalization layer, BN((1 + s)qx) = BN(qx) for any positive scalar s 2 R+.
Geometry of learning dynamics. What does the symmetry of the loss function in parameter space imply about the symmetry of the learning dynamics? In classical mechanics, a symmetry of the potential function in space usually implies the equivariance of the trajectory, namely, a solution of the Newton’s equation of motion (grey) remains a solution under the symmetry transformations (red) as shown in Fig. 2 (a). This implies that the time evolution of the potential energy is invariant under symmetry transformations at any point in the dynamics. However, this intuition does not hold in modern neural networks. Consider the concrete example of a VGG11 model with a batch normalization layer after each convolutional filter. The scale symmetry of batch normalization implies the loss function is invariant to the scaling transformation on the convolutional filters q, f((1 + s)qx) = f(qx), however the learning dynamics are not. To demonstrate this we train two VGG11 models on Tiny ImageNet with
a standard initialization (grey) and a scaled initialization (red). Notice, that the scaling does not change the initial loss due to the scale symmetry, however it does effect the learning dynamics of the loss, as shown in the bottom of Fig. 2(b). In otherwords, even though the loss landscape observes scale symmetry at all steps in training, the learning dynamics do not. Understanding symmetry of the learning dynamics requires a new theoretical perspective.
Implicit acceleration and variational perspective on learning dynamics. In physics, variational formulations based on the Lagrangian and Hamiltonian provide a unifying framework to study the role of symmetry in dynamical systems. However, there is still a gap between the tools and concepts in physics and the learning dynamics of neural networks. We identify the root cause of this gap to be the learning rule, namely the fact that the continuous-time limit of gradient descent is a first order differential equation in time, gradient flow dqdt = rf , unlike classical mechanics which is governed by Netwon’s second law md
2q dt2 = rf , which is second-order in time. We bridge this gap
by realizing that the effect of discretization, is to introduce an implicit acceleration term proportional to the learning rate. Leveraging this connection, we build the mechanics of learning dynamics:
Sec. 1. From Newtonian to Lagrangian mechanics of learning via implicit acceleration. We show how gradient descent with a finite learning rate introduces implicit acceleration making it amendable to the Lagrangian formulation of accelerated optimizers [4]. Sec. 2. Unlike in physics, kinetic symmetry breaking (KSB) is inherent in learning dynamics. We define kinetic symmetry breaking, where the kinetic energy corresponding to the learning rule explicitly breaks the symmetry of the potential energy corresponding to the loss function. Sec. 3. We derive Noether’s learning dynamics (NLD) induced by the KSB. We generalize Noether’s theorem in physics to derive NLD, which accounts for damping, the unique symmetries of the loss, and the non-Euclidean metric used in learning rules. Sec. 4. KSB induces impicit adaptive optimization and enables successful learning. We leverage NLD to exactly solve for the time-evolution of the effective learning rate for networks with normalization layers, and establish an exact analogy between two seemingly unrelated components of modern deep learning: normalization and adaptive optimization.

heading: 1 From Newtonian to Lagrangian mechanics of learning
text: The fundamental law of mechanics is the principle of least action [5]. We first review how pioneering works by Wibisono, Wilson, and Jordan [4, 6, 7] apply the variational principles of physics to model accelerated methods in optimization. We then show how gradient descent with a finite learning rate introduces implicit acceleration making it amendable to the Lagrangian formulation.
Variational formulation of classical mechanics. Just over a century after Newton, Joseph-Louis Lagrange demonstrated how Newton’s equations of motion could be derived from a variational framework of least action. Rather than integrating an equation of motion over time to get the path an object takes from point a to point b, Lagrange prescribed that the path taken by the object is the path that minimizes the action, the path integral of a function termed the Lagrangian, over all possible paths from a to b. This principle of least action transformed mechanics from a differential description to a variational description. Consider for example the simple setup of a particle of mass m at position q 2 RN in N dimensional physical space moving according to a potential f(q) 2 R and Newton’s second law mq̈ = rf , where q̈ = d
2q dt2 is the acceleration. Let the Lagrangian be the
function L(q, q̇, t) = 12mq̇ 2 f(q), the difference between the kinetic energy T = 12mq̇ 2 and the potential energy V = f(q) of the particle. The principle of least action states that the time evolution of the particle’s position q(t) is the function that minimizes the action S[q] =
R t1 t0
L(q, q̇, t)dt. In otherwords, q(t), with boundary conditions q(t0) = a and q(t1) = b, must satisfy the stationary condition S q(t) = 0, which is equivalent to satisfying the Euler-Lagrange equation (see SI Sec. A for the review),
d
dt ✓ @L @q̇ ◆ = @L @q . (1)
Indeed, substituting the Lagrangian into the Euler-Lagrange equation, we recover the Newton’s equation of motion mq̈ = rf confirming the consistency of this variational formulation. Damping in optimization. In learning dynamics, one of the universal features is damping, a term proportional to the velocity q̇ = dqdt in the equation motion. Intuitively, damping is needed for any optimization, since without damping the learning dynamics would conserve energy and not converge. We can incorporate damping into the previous Lagrangian example by introducing an explicit time-dependence L(q, q̇, t) = e µm t( 12mq̇
2 f(q)), where µ is the damping (or friction) coefficient [8]. Substituting this Lagrangian into the Euler-Lagrange equation and dividing both sides by e µ m t, we obtain the equation of motion of a damped particle mq̈ + µq̇ = rf as intended.
Non-Euclidean geometry in optimization. Many optimization algorithms assume an underlying geometry. For example, consider gradient descent with learning rate ⌘. The update equation, q(t + ⌘) = q(t) ⌘rf , is equivalent to solving the minimization problem, q(t + ⌘) = argminq0 ⇥ hrf(q(t)), q0i+ ⌘ 1 12 |q 0 q(t)|2 ⇤ , where h , i represents inner product. In otherwords, we can interpret each update of gradient descent as a trade-off (specified by ⌘) between minimizing the loss f(q), while remaining close to its current location in Euclidean distance. For non-Euclidean measures of distance we would obtain different optimization algorithms. Mirror descent generalizes this idea by replacing the distance measure by the Bergman divergence Dh(y, x) = h(y) h(x) hrh(x), y xi where h(x) is a distance-generating function. Indeed, when the distance generating function is h(x) = 12 |x|
2, then the Bregman divergence simplifies to be the squared Euclidean distance Dh(y, x) = 12 |x y|
2 and mirror descent is equivalent to gradient descent. The mirror descent framework encapsulates a broad array of optimization methods such as gradient descent, natural gradient descent, and others [9].
The Bregman Lagrangian. Integrating building blocks of optimization such as damping and nonEuclidean geometry into the variational framework of mechanics, Wibisono, Wilson, and Jordan introduced the Bregman Lagrangian [4, 6],
L(q, q̇, t) = e↵t+ t Dh(q + e ↵t q̇, q) e tf(q) . (2)
For various choices of the time-dependent factors ↵t, t, and t the Bregman Lagrangian, under the principle of least action, defines continuous approximations to an array of learning rules such as Nesterov’s accelerated gradient methods [10, 11] and natural gradient flow [9]. The flexibility of this framework makes our analysis generalize to a broad array of learning rules, as summarized in table 1.
Implicit acceleration and the Lagrangian formulation of gradient descent. In the continuoustime limit of diminishing learning rate ⌘ ! 0, gradient descent q(t+ ⌘) = q(t) ⌘rf becomes a first order differential equation, gradient flow q̇ = rf . Unlike Nesterov’s momentum [11], the trajectory of gradient descent with heavy ball momentum as applied in deep learning [12], follows gradient flow with the rescaling (1 )q̇ = rf . Here, we apply modified equation analysis [1, 13] to incorporate the effect of a finite learning rate and leverage the fact that the discrete steps of gradient descent closely follow the continuous-time trajectory of the second-order differential equation in time. We can informally see this by Taylor expanding an update with respect to a small, but finite learning rate q(t+ ⌘) = q + ⌘q̇ + ⌘ 2
2 q̈ +O(⌘ 3) and then plugging this expression to the
gradient descent updates q(t+⌘) q(t)⌘ = rf . We keep the first order in ⌘ and obtain a second-order differential equation ⌘2 q̈ + q̇ = rf . We call the additional term ⌘ 2 q̈, implicit acceleration. This dynamics reduces to gradient flow q̇ = rf in the continuous-time limit as expected. (Note that a complementary approach is to view the effects of discrete step size as implicit gradient regularization on the original loss f(q)! f(q)+ ⌘4 |rf |
2 [1, 14, 15, 16, 17].) Performing a similar analysis [1, 13] accounting for the effects of the heavy ball momentum with weight decay k yields a second-order differential equation,
⌘ 2 (1 + )q̈ + (1 )q̇ +rf(q) + kq = 0, (3)
where ⌘2 (1 + )q̈ represents the implicit acceleration.
Indeed, we can connect this second-order dynamics to the Bregman Lagrangian by considering the factors ↵t = log ⌘(1+ )2 , t = log ⌘(1+ ) 2 , t = 2(1 ) ⌘(1+ ) t, and the distance generating function h(x) = 12 |x| 2. The Lagrangian describing the learning dynamics under gradient descent with a finite learning rate ⌘, heavy ball momentum , the loss function f(q), and weight decay constant k is
L(q, q̇, t) = e 2(1 ) ⌘(1+ ) t
 ⌘(1 + )
4 |q̇|2
✓ f(q) + k
2 |q|2
◆ . (4)

heading: 2 Symmetry and symmetry breaking in neural networks
text: The variational perspective of optimization, discussed in the previous section, provides a useful lens to characterizing the path a model takes through parameter space during training. To study the geometry of this path we will turn to the tools of symmetry. In particular, in this section we will discuss not only when symmetries exist in the loss function, but also when they are broken by the learning rule.

heading: 2.1 Symmetries of the loss function
text: There have been a number of works studying symmetries of the loss function in parameter space, such as [18, 19] who study discrete permutation symmetries or [1] who study continuous differentiable symmetries. In this work we will focus on continuous differentiable symmetries found in modern deep learning architectures. Consider a one-parameter family of differentiable maps q ! Q(q, s) parameterized by a scalar s 2 R such that s = 0 gives an identity Q(q, 0) = q. We say the potential f(q) possesses a differentiable symmetry if its invariant to the transformation f(Q(q, s)) = f(q) for all s. The symmetries we consider in this work are: translation symmetry where Q(q, s) = q + sn̂ and n̂ is a time-independent vector, rotation symmetry where Q(q, s) = R(s)q and R(s) is a rotation matrix, and scale symmetry where Q(q, s) = (1 + s)q. Translation symmetries occur in a neural network for parameters immediately preceding a softmax function. Rotation symmetry is one of the
most fundamental symmetry groups in classical mechanics and occur in neural networks with weight decay [20, 21] or word embeddings [22]. Scale symmetries appear ubiquitously in neural networks for parameters immediately preceding normalization layers, such as batch normalization [3] and its variants [23]. There is also a related symmetry, defined as rescale symmetry in [1], present in neural networks with continuous, homogeneous activation functions (e.g. ReLU, Leaky ReLU, linear).

heading: 2.2 Symmetry breaking in the kinetic energy
text: In the previous section, we introduced the Lagrangian formulation of learning dynamics, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. The kinetic energy of the Bregman Lagrangian, defined as
Th(q, q̇, t) ⌘ e↵tDh(q + e ↵t q̇, q) = e↵t h(q + e ↵t q̇) h(q)i hrh(q), e ↵t q̇i , (5)
is as important as the loss function, but its properties are less discussed. Here, we ask if the Bregman kinetic energy respects the same symmetries as the loss function, whether Th(Q(q, s), Q̇(q, s), t) = Th(q, q̇, t).
Euclidean kinetic energy. First, we consider when h(x) = 12 |x| 2 and study the symmetry properties of the Euclidean kinetic energy,
TE(q, q̇, t) = e ↵t
2 |q̇|2. (6)
For translation symmetry the velocity is invariant under the transformation Q̇ = q̇ and thus indicating that the Euclidean kinetic energy is invariant under spatial translation. For rotation symmetry, the temporal differentiation commutes with the rotation Q̇ = Rq̇, however the Euclidean kinetic energy is invariant under spatial rotation because the norm is unchanged |Q̇|2 = hRq̇,Rq̇i = |q̇|2. However, for scale symmetry Q̇ = (1 + s)q̇ and thus the norm |Q̇|2 = (1 + s)2|q̇|2 explicitly depends on s. Therefore, the kinetic energy explicitly breaks the scale symmetry present in the loss function. We call this explicit symmetry breaking of the Lagrangian by the kinetic energy term Kinetic Symmetry Breaking (KSB).
Non-Euclidean kinetic energy. Next, we consider the setting of a general non-Euclidean metric. The Taylor expansion of the Bregman kinetic energy with respect to the first term h(q + e ↵t q̇) in Eq.5 results in the simplification,
Th(q, q̇, t) = (e ↵t/2) ⌦ q̇,r2h(q)q̇ ↵ +O (e ↵t)2 . (7)
Notice, the Euclidean kinetic energy only depends on the squared norm of the velocity |Q̇|2 because the Hessian is the identity matrix r2 12 |x|
2 = I and the higher order derivatives are zero. With non-Euclidean metrics, such simplifications special to the Euclidean case do not necessarily apply and we should not expect that the Bregman kinetic energy respects any symmetries in general. However, the natural gradient [9] and its elegant approximations, such as [24], are examples of non-Euclidean optimization methods, which have invariance properties by construction. While the invariance properties of the natural gradient descent can be broken due to discretization [25], the fast and accurate approximation of natural gradients in modern large-scale neural networks is an active area of research [26, 27].

heading: 3 Noether’s learning dynamics
text: So far, we have seen how the Bregman Lagrangian provides a unified variational description of learning dynamics, and the scale symmetry, ubiquitous in modern neural networks, induces kinetic symmetry breaking (KSB). Here, we apply Noether’s theorem to the Bregman Lagrangian and derive Noether’s Learning Dynamics (NLD), a unified equality that holds for any combination of symmetry and learning rules.
Noether’s theorem. In 1918 Emmy Noether famously described one of the most fundamental theorems in modern physics: every differentiable symmetry of the action has an associated conserved quantity through time [28]. Noether’s theorem not only proves the existence of, but also provides an explicit expression for, the conserved quantity, often called Noether charge. Focusing just on the symmetry properties of the Lagrangian, a general form of Noether’s theorem relates the dynamics of
the Noether charge h@q̇L, @sQi to the change in the Lagrangian under an infinitesimal transformation @sL as
d dt h@q̇L, @sQi = @sL, (8)
where we evaluate all derivatives at the identity element of the symmetry group (s = 0). We can derive this relationship as @sL = h@qL, @sQi + h@q̇L, @sQ̇i = ddt h@q̇L, @sQi, where we plugged in the Euler-Lagrange equation @qL = ddt (@q̇L) and rearranged the total time derivative. The most common application of Noether’s theorem in physics is when the Lagrangian, including both kinetic and potential energies, is symmetric under the infinitesimal transformation, @sL = 0, implying Noether charge is conserved through time. For example, if we assume symmetry of the Lagrangian due to homogeneity or isotropy of space, Noether’s theorem directly implies the conservation of momentum or angular momentum respectively. However, as shown in Sec. 2, the Bregman kinetic energy for learning systems is often asymmetric to the symmetry of the potential function, implying that the resulting Noether charge is not conserved.
Noether’s learning dynamics. Here, we derive a general equation describing the kinetic-asymmetryinduced dynamics for the Noether charge in learning systems. If the Bregman kinetic energy does not respect the symmetry of the loss function, then Eq. 8 becomes
d dt h@q̇L, @sQi = e t@sTh, (9)
implying kinetic symmetry breaking (@sTh 6= 0) induces motion of the Noether charge. To build better intuition for this motion and inspired by the definition of the generalized momentum, we define the Bregman momentum as,
h ⌘ e t@q̇L = rh(q + e ↵t q̇) rh(q). (10)
Expanding Eq. 9 by plugging in the definition of the Bregman Lagrangian Eq. 2, the Bregman kinetic energy Eq. 5, and the Bregman momentum Eq 24 (details are described in SI Sec. B), we obtain Noether’s Learning Dynamics (NLD),
d dt chargez }| { h h, @sQi+ dampingz }| { ̇th h, @sQi = kinetic asymmetryz }| { h h, @sQ̇i + non-Euclidean metricz }| { e ↵th h e ↵tr2h(q)q̇, @sQi .
(11)
Each term in NLD has an intuitive meaning. Recall the conventional setting in classical mechanics, where the entire Lagrangian is symmetric, there is no damping, and the metric is Euclidean. In this case, the Noether charge is the inner product of momentum and the generator of the symmetry transformation he ↵t q̇, @sQi. The first term of NLD, h h, @sQi, is the inner product of the Bregman momentum and the generator of the symmetry transformation. In the setting of no damping, this terms is a generalization of the conventional Noether charge that encompasses non-Euclidean metrics. Indeed, in the Euclidean case, we can see that E = e ↵t q̇ and the term reduces to the conventional Noether charge as expected. The second term of NLD, ̇th h, @sQi, represents the contribution of damping and the term becomes zero when t is constant and there is no damping. The third term of NLD, h h, @sQ̇i, represents the contribution of kinetic symmetry breaking. Under the Euclidean metric, this term is proportional to hq̇, @sQ̇i. For translation symmetry, @sQ̇ = 0 and this term is zero h h, 0i = 0. Similarly, for rotation symmetry, this term is zero hR|s=0q̇, (@sR)q̇i = hq̇, (@sR)q̇i = 0. Here, we used the fact that the generator of rotation is a skew-symmetric matrix. However, for scale symmetry, @sQ̇ = q̇ unique to learning systems, this term is proportional to |q̇|2 and thus present. The fourth term of NLD, e↵th h e ↵tr2h(q)q̇, @sQi, represents the contribution of the non-Euclidean geometry and is zero under the Euclidean metric h(x) = 12 |x| 2.
Noether’s learning dynamics under gradient flow. Next, we show that in the over-damped limit of NLD, the charge h h, @sQi itself, rather than its time-derivative becomes zero. Consider Eq.11 when ↵t = logm, t = logm, t = µm t,
m d dt h h, @sQi+ µh h, @sQi = m@sT. (12)
When @sT = 0, this equation describes how the charge decays exponentially with damping h h, @sQi = e µ m t as in Fig. 3. In the mass-less limit m ! 0 with a fixed friction µ, the terms proportional to m in Eq. 12 all vanish and we get
h h, @sQi = 0, (13)
which states that the charge itself is zero, rather than its time derivative. Furthermore, in the Euclidean settings, the result simplifies to hq̇, @sQi = 0. As a physical analogy, this means that quantities such as momentum or angular momentum becomes zero in the over-damped limit. This formula unifies an array of conservation properties noticed under gradient flow [1, 29, 30, 31, 32] with a formal theoretical connection to Noether’s theorem.
Overall, just like how Noether’s theorem [28] unified an array of conservation laws and provided a theoretical foundation to discover new ones in physics, we have further unified conservation laws previously observed in learning systems [1, 29, 30, 31, 32] and generalized these results for any combination of differentiable symmetries in neural network architectures and learning rules (e.g., natural gradient descent, Nesterov’s accelerated gradient descent, accelerated mirror descent, and cubic-regularized Newton’s method).

heading: 4 Kinetic symmetry breaking induces implicit adaptive optimization
text: In the previous sections, we identified the mechanism of kinetic symmetry breaking and derived the Noether’s learning dynamics (NLD). Here, we explore a role of symmetry breaking in neural networks by applying NLD to study auto rate-tuning by normalization layers, originally studied by Arora, Li, and Lyu [33]. In particular, we establish an exact theoretical analogy between two seemingly unrelated key components of modern deep learning: normalization and adaptive optimization.
Consider a neural network model with normalization layers (such as batch normalization [3], group normalization [34], layer normalization [35], and others [23]), which introduces scale symmetry in the potential (loss) function f((1 + s)q) = f(q). We train this model with gradient descent with a finite learning rate ⌘, momentum , and weight decay k. We are interested in the evolution of the potential function through the learning dynamics f(q(t)). Due to the scale symmetry of the network introduced by the normalization layer, the time evolution on the loss landscape only depends on the directional vector q̂ ⌘ q/|q| of the parameters throughout training, f(q(t)) = f(q̂(t)). What is the equation of motion for the directional vector? To solve for the dynamics, we first move to a polar coordinate system q = rq̂ where r = |q| is the radial norm and q̂ is the directional vector. This is akin to how we solve for the motion of a planet in a central field in classical mechanics. From Eq.4, the Lagrangian for this learning system in polar coordinates is,
Lc = e µ m t
 m
2 ṙ 2 +
✓ m
2
˙̂q 2 k
2
◆ r 2 f(q̂) + (t) |q̂|2 1 , (14)
where m = ⌘(1+ )2 , µ = 1 , and C(q̂) = |q̂| 2 1 = 0 is a Lagrange multiplier enforcing the unit norm constraint. The Lagrangian formulation facilitates coordinate transformations, because the form of the Euler-Lagrange equation is invariant under coordinate transformations. Taking advantage of this fact, we apply the Euler-Lagrange equation to derive the equation of motion for the directional vector as (see SI Sec. C for the derivation)
m¨̂q + µ ˙̂q = 1 r2(t) rf(q̂), (15)
whererf(q̂) represents the gradient of f evaluated at q̂. Notice the striking similarity of this equation of motion with the equation of motion of the original parameters mq̈ + µq̇ = rf(q). This implies that the time evolution of the loss for a network with normalization layers trained by gradient descent is equivalent to the time evolution of the loss for an identical network with unit filter norm trained by an adaptive optimizer which scales the gradients by a factor of r2(t).
Strikingly, NLD describes the motion of this adaptive scaling factor r2(t). When ↵t = logm, and h(x) = 12 |x| 2, then the charge is h h, @sQi = mhq̇, qi = 12m@tr 2 and Eq.11, including the effect
of weight decay k, implies
m@ 2 t r 2 + µ@tr 2 = 2kr2 + 2m
µ2r2 |rf(q̂)|2, (16)
where we kept the first order terms in m and k. In the over-damped regime, ignoring inertia, we get a change in the squared parameter norm r2(t) as (see SI Sec. D for the derivation):
r 2(t) =
s 2⌘(1 + )
(1 )3
Z t
0 e 4k1 (t ⌧)|rf(q̂(⌧))|2d⌧ + e 4k 1 tr4(0). (17)
Each term in this implicit adaptive learning rate has an intuitive meaning. The first integral term keeps track of the recent history of the magnitude of gradient norms |rf(q̂(⌧))|2 that the filters with fixed unit norm receive. The weight decay k controls the time-scale of the short-term memory of the accumulated gradient norms through the exponential kernel e 4k 1 (t ⌧). When k = 0 all the gradients will be accumulated resembling an AdaGrad optimizer [36]. Notably, this integral term is proportional to the learning rate ⌘ implying that this mechanism of adaptive optimization is only present with a finite learning rate. The second term represents an exponentially decaying memory of initialization. The finite filter norm at initialization r4(0) prevents instability of learning dynamics avoiding division by zero.
So how does this adaptive scaling of the gradient step sizes compare with hand-designed adaptive optimizers successful in deep learning [36, 37, 38]? Here, we develop a continuous-time model of the explicit adaptive optimizer RMSProp and discover that the functional form of normalization layers’ effective learning rate schedule Eq. 17 functionally agrees with that of RMSProp. The RMSProp algorithm is a recursive update rule expressed as qn+1 = qn ⌘pGnrf(qn), where the gradient is scaled by a factor of p Gn. The factor Gn keeps track of the history of the gradient norms following Gn+1 = ⇢Gn + (1 ⇢)|rf(qn)|2, where ⇢ is a hyperparameter. Solving the continuous-time dynamics ⌘ dGdt = (1 ⇢)G(t) + (1 ⇢)|rf(q(⌧))|
2 of the adaptive scaling factor of RMSProp, we obtain
p G(t) = s 1 ⇢ ⌘ Z t 0 e 1 ⇢⌘ (t ⌧)|rf(q(⌧))|2d⌧ + e 1 ⇢ ⌘ tG(0). (18)
Strikingly, we find that this functional form induced by RMSProp exactly matches with the implicit adaptive learning rate schedule Eq.17 due to normalization layers suggesting potential benefits of the implicit adaptive optimization induced by normalization.
In summary, our theory has yielded the following new insights:
1. The optimization geometry of SGD with a finite step size ⌘, momentum , and weight decay k together breaks the scale symmetry of the loss (e.g., with normalization layers) and generates an implicit mechanism of adaptive optimization akin to RMSProp.
2. The implicit adaptive optimization mechanism exists when the learning dynamics is in the underdamped regime m/µ / ⌘/(1 ) 0 and thus requires the learning rate to be finite.
3. Momentum significantly amplifies the effect of adaptive optimization by 1+ (1 )3 = 1900 folds as in Eq. 17 even in the standard setting of = 0.9.
4. For scale-invariant parameters, symmetry breaking due to weight decay 1 k plays a role similar to the discount factor ⇢ for the cumulative gradient norm in RMSProp.
Finally, we empirically validate these predictions by training convolutional neural networks with batch normalization (VGG11) on a large data set (Tiny-ImageNet) under various hyperparameter settings as in Fig. 4 (see SI Sec. E). In addition to the standard training with an “unconstrained filter norm” (pink), we perform ablation experiments with a “constrained filter norm” (blue). As a concrete example, consider a convolutional filter q 2 Rn preceding the batch normalization layer. The loss is invariant under the scaling transformation of the filter norm f((1 + s)q) = f(q). To test the presence and benefits of implicit adaptive optimization to the learning dynamics of the loss, we freeze the dynamics of the filter norm by repeatedly performing the operation q(t) q(t)|q(t)| |q(0)| after each step of optimization [33]. This enforces the filter norm to be fixed |q(t)| = |q(0)| throughout training for any t. When trained with a constant small learning rate ⌘ ⇠ 0.001, the final test accuracy (A-1)
and loss dynamics (A-2) of models with unconstrained filter norm (pink) and constrained filter norm (blue) are almost identical. However, in a large learning rate regime ⌘ ⇠ 0.03, the models trained with unconstrained filter norm (pink) outperforms the models trained with constrained filter norm (blue) demonstrating the existence and benefits of implicit adaptive optimization. Similarly, we confirm our prediction that the presence of momentum ⇠ 0.9 is essential to amplify the effects of implicit adaptive optimization, as seen in test accuracy (B-1) and loss dynamics (B-2). Finally, we demonstrate the benefits of symmetry breaking due to weight decay k which acts in analogy with the discounting factor ⇢ for the cumulative gradient norms of RMSProp. Indeed, the final test accuracy (C-1) and the learning dynamics of the loss (C-2) are both enhanced by the presence of weight decay k in agreement with [39]. Overall, the hyperparameters of the modern optimization - learning rate, momentum, and weight decay - each play an important role in breaking the scale symmetry of the loss function and providing an mechanism of implicit adaptive optimization, a key to successful learning in neural networks.

heading: 5 Related works and future directions
text: Finite learning rates, modified gradient flow, and generalization. Recently, many works have demonstrated that an initial large learning rate is a key to successful training [40, 41, 42, 43, 44]. Another line of works [1, 13, 14, 15, 16, 45, 46] have incorporated the effect of a finite learning rate into gradient flow by adding a modification term. A common approach applies modified equation analysis to replace the original loss function f by a modified loss function f̃ = f + ⌘4 |rf | 2 [1, 14, 15], where the additional term is called implicit gradient regularization. Intuitively, this term biases the learning dynamics towards wider minima with smaller gradient norms. To test this intuition empirically, recent works have explicitly added the gradient regularization term ⌘4 |rf | 2 to the original loss and have shown that the models trained with the modified loss generalize well even with small learning rates [16] or full-batch [17]. In this work, we have taken a complementary approach and applied modified equation analysis in time to add an implicit acceleration term ⌘2 q̈ [1, 13, 45]. When training with a finite learning rate, this implicit acceleration term gives rise to the implicit adaptation of the gradients. Remarkably, the two complementary pictures (i.e., implicit gradient regularization and implicit acceleration) offer a consistent view on how the large learning rates provide mechanisms to properly regularize the norm of gradients to avoid training instabilities. An idea for future work is to directly simulate gradient flow with implicit acceleration, instead of gradient flow with [15, 16, 17] or without [47] implicit gradient regularization, to keep the benefits of finite learning rates while avoiding the computations of the Hessian gradient products.
Geometry of optimization. The geometric property of optimization has a long history of study. For example, natural gradient by Amari [9] and its approximations [24, 48] are invariant under affine transformations including scale transformation due to normalization layers [39]. With the rise of deep learning, there has been recent developments to efficiently approximate and apply natural gradients to large neural networks [24, 26, 48, 49, 50, 51, 52, 53]. When training deep neural networks, we still commonly use gradient-based optimization methods represented by stochastic gradient descent with momentum [12]. An array of adaptive optimization methods [36, 37, 38] have been proposed partially motivated as diagonal approximations of the Fisher Information Matrix (see recent discussions questioning the validity of the approximations [54]). A pioneering work [55] by Neyshabur et al. has identified that gradient descent does not respect the rescale symmetry introduced by ReLU activations, followed by works proposing to solve such discrepancies by enforcing symmetry in the learning dynamics [56, 57]. While above works have always tried to remove such broken symmetries, we demonstrated theoretically and empirically that the broken-symmetry can play an important role in deep learning, thus providing a unique counterexample. A future direction is to extend our analysis to the case of rescale symmetry of the homogemeous activation functions such as ReLU.
Conservation laws in neural networks. Balancing properties of network parameters have been identified in gradient flow dynamics of neural networks with linear [29] and ReLU [30, 31] activation functions. These properties have been leveraged to better understand the role of over-parameterization [29], implicit regularization [30], network pruning [31], continual learning [58], and self-supervised learning [32]. A recent work identified symmetry-induced geometric structures of the loss and studied their roles under gradient flow dynamics with modified loss predicting the parameter dynamics [1]. However, despite the ubiquity of conservation laws in deep learning, no formal connection to the Noether’s theorem has been made due to the previous focus on the gradient flow dynamics. Here, we took an alternative path starting from a Lagrangian formulation, which encapsulates gradient flow as the mass-less limit. From this perspective, we unified all the conservation laws under the umbrella of Noether’s theorem, and provided its generalization for accelerated and non-Euclidean methods in optimization. In future works, we can harness the generality of our theory to investigate the dynamics of more complex learning rules such as natural gradient descent with or without Nesterov’s momentum in combination with any differentiable symmetries.
Auto-rate tuning by normalization layers. Batch normalization [3] and its variants [35, 59] are essential to train deep neural network models [60] providing multiple benefits as elegantly reviewed in [61]. An important benefit of normalization layers is to enable stable training with large learning rates [62]. A pioneering work [33] rigorously studied the benefits of monotonically decreasing the effective learning rate in simple setting without momentum nor weight decay. Another line of works noticed the role of weight decay to effectively enforce a higher learning rate [39, 63, 64] inspiring an automatic learning rate scheduler [65]. Li and Arora [66] theoretically studied the setting with weight decay and momentum to design a exponential learning rate schedule that still leads to successful training counter-intuitively. In the present analysis, we leveraged the Lagrangian formulation to handle the complexity of modern practical optimizers and derived an exact solution describing the time-evolution of the effective learning rate in the presence of momentum and weight decay. By developing a continuous-time description of adaptive optimizers in parallel, we discovered that the functional form of the effective learning rate exactly matches with that of RMSProp, where momentum significantly amplifies the effect, and weight decay acts in exact analogy with the discounting factor. In the context of thorough empirical studies on adaptive optimizers [67, 68, 69], our results raise the hypothesis that the benefit of an adaptive optimizers may not be as significant in networks with normalization layers as long as the hyper-parameters, in particular momentum, are properly tuned.

heading: 6 Conclusion.
text: In nature, symmetry governs regularities, while symmetry breaking brings texture [70], examples of which include superconductivity [71], nucleon mass [72], ring attractor neural networks [73, 74, 75], and the Higgs boson [76]. Despite many attempts to bring symmetry to neural networks [2, 55], symmetry breaking has rarely been discussed as a design principle [22], or even treated as an obstacle to successful learning [55, 56, 57]. In this work, we have discovered a novel role of explicit symmetry breaking in learning systems by applying a Lagrangian formulation to neural networks. Overall, understanding not only when symmetries exist, but how they are broken is essential to discover geometric design principles in neural networks.
Acknowledgments and Disclosure of Funding
We thank Max Aalto, Kyosuke Adachi, Yuto Ashida, Fatih Dinc, Surya Ganguli, Kyogo Kawaguchi, Ekdeep Singh Lubana, Akiko Tanaka, Hajime Tanaka, Atsushi Yamamura, and Daniel Yamins for helpful discussions. D.K. thanks the Stanford Data Science Scholars program and NTT Research for support.

Review summaries:
The authors identify a "kinetic asymmetry" in the learning dynamics under gradient flow, where the kinetic energy is not invariant to the symmetries of the potential energy. An example is scaling: even when the network function is invariant to scaling or ReLUs or batch norm inputs, the learning dynamics are not. They derive a genealized version of Noether's theorem which takes this asymmetry into account and use it to study the implict adaptive optimization exhibited by batch normalization, showing that it is the same as that of RMSprop. 

The gradient descent with a momentum can be regarded as the Newtonian equation with damping. The output of the batch normalization is invariant to the scale of the input (i.e., a scale symmetry). When a BN is inserted after a convolution, the whole network is invariant to the weight scale. A typical gradient descent algorithm ignores (i.e., breaks) this symmetry. Thanks to this symmetry breaking, the learning rate is implicitly adapted, and the adaptation is equivalent to RMSProp's. 

The paper considers the role of symmetries in the learning dynamic by considering a continuous time formulation of the learning process in Lagrangian formalism. In such formulation there are "kinetic" and "potential" components to the dynamics, the former arising from the learning rules employed during optimization and the latter generated by the loss function landscape. Authors argue that for a large class of practically relevant network architectures there are symmetries that are present in the "potential" component, but not in the "kinetic" part.  Starting with Bergman Lagrangian formulation of the learning dynamics, authors derive the equations for time evolution of the Noether's charge associated with symmetry transformation (assumed to be present) in the network architecture.  For the case on scale symmetry introduces by BatchNormalization, authors observe that in case of training with gradient descent + momentum + weight decay; the training dynamics effectively reduces to a learning dynamics with an adaptive scale where the scale is related to the associated Noether's charge. This suggests that kinetic asymmetry induces an implicit adaptive property to the optimization symmetry. A connection between RMSProp and implicit adaptation is established based on continuous time learning model.  Lastly, authors test theoretical predictions by training a set of models verifying theoretical predictions on Tiny ImageNet dataset. For comparison, the adaptive rescaling effect is artfully removed by renormalizing the parameters throughout training. 

Based on the Euler-Lagrangian framework, this manuscript develops the theoretical method to model the learning dynamics of DNNs as symmetry breaking of the likelihood by the kinetic energy.  The model is expressed as the time evolution of Noether-current. As a result, the author found that the learning dynamics of batch normalization are similar to RMSProp from the viewpoint of symmetry breaking. Their findings deepen our understanding of batch normalization. This study is unique not only because of the findings on batch normalization mechanics but also because it links the symmetry-breaking process of the likelihood function with the learning theory.