Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
Authors: Daniel Kunin
Javier Sagastuy-Brena
Surya Ganguli
Daniel LK Yamins
Hidenori Tanaka
Venue: ICLR 2021 Poster (2021)
Keywords: learning dynamics
symmetry
loss landscape
stochastic differential equation
modified equation analysis
conservation law
hessian
geometry
physics
gradient flow

Abstract: Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion of a network in high-dimensional parameter space undergoes discrete finite steps along complex stochastic gradients derived from real-world datasets. We circumvent this obstacle through a unifying theoretical framework based on intrinsic symmetries embedded in a network's architecture that are present for any dataset. We show that any such symmetry imposes stringent geometric constraints on gradients and Hessians, leading to an associated conservation law in the continuous-time limit of stochastic gradient descent (SGD), akin to Noether's theorem in physics. We further show that finite learning rates used in practice can actually break these symmetry induced conservation laws. We apply tools from finite difference methods to derive modified gradient flow, a differential equation that better approximates the numerical trajectory taken by SGD at finite learning rates. We combine modified gradient flow with our framework of symmetries to derive exact integral expressions for the dynamics of certain parameter combinations. We empirically validate our analytic expressions for learning dynamics on VGG-16 trained on Tiny ImageNet. Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.

Published as a conference paper at ICLR 2021
NEURAL MECHANICS: SYMMETRY AND BROKEN CON- SERVATION LAWS IN DEEP LEARNING DYNAMICS
Daniel Kunin*, Javier Sagastuy-Brens Surya Ganguli, Daniel L.K. Yamins, Hidenori Tanaka#1
Stanford University i Physics & Informatics Laboratories, NTT Research, Inc.
ABSTRACT
Understanding the dynamics of neural network parameters during training is one of the key challenges building a theoretical foundation for deep learning. A central obstacle is that the motion of network in high-dimensional parameter space undergoes discrete finite steps along complex stochastic gradients derived from real-world datasets. We circumvent this obstacle through unifying theoretical framework based on intrinsic symmetries embedded in a network's 's architecture that are present for any dataset. We show that any such symmetry imposes strin- gent geometric constraints on gradients and Hessians, leading to an associated conservation law in the continuous-time limit of stochastic gradient descent (SGD), akin to Noether's 's theorem in physics. We further show that finite learning rates used in practice can actually break these symmetry induced conservation laws. We apply tools from finite difference methods to derive modified gradient flow, a differential equation that better approximates the numerical trajectory taken by SGD finite learning rates. We combine modified gradient flow with our frame- work of symmetries to derive exact integral expressions for the dynamics of certain parameter combinations. We empirically validate our analytic expressions for learning dynamics on VGG-16 trained on Tiny ImageNet. Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.
INTRODUCTION
Just like the fundamental laws of classical and quantum mechanics taught us how to control and optimize the physical world for engineering purposes, better understanding of the laws governing neural network learning dynamics can have a profound impact on the optimization of artificial neural networks. This raises a foundational question: what, if anything, can we quantitatively understand about the learning dynamics of large-scale, non-linear neural network models driven by real-wond woal-world datasets and optimized via stochastic gradient descent with a finite batch size, learning rate, and with or without momentum? ? In order to make headway on this extremely difficult question, existing works have made major simplifying assumptions on the network, such as restricting to identity activation functions Saxe al. (2013), infinite width layers Jacot et al. (208)), or single hidden layers Saad & Solla (1995). Many of these works have also ignored the complexity introduced by stochasticity and discretization by only focusing on the learning dynamics under gradient flow. In the present work, we make the first step in an orthogonal direction. Rather than introducing unrealistic assumptions on the model or learning dynamies, we uncover restricted, but meaningful, combinations of parameters with simplified dynamics that can be solved exactly without introducing major assumption (see Fig. [ To find the parameter combinations, we use the lens of symmetry to show that the training loss doesn't 't change under some transformation of the parameters, then the gradient and Hessian for those parameters have associated geometric constraints. We systematically apply this approach to modern neural networks to derive exact integral expressions and verify our predictions empirically large scale models and datasets. We believe our work is the first step towards a foundational understanding
Equal contribution. Correspondence kunin stanford.edu & hidenori www.oorrtttaaaaaaaa @nt-rrecccchh..com

---

Published as a conference paper at ICLR 2021
of neural network learning dynamics that is not based in simplifying assumptions, but rather the simplifying symmetries embedded in a network's ' architecture. Our main contributions are:
1. We leverage continuous differentiable symmetries in the loss unify and generalize geo- metric constraints on neural network gradients and Hessians (section B)
2. We prove that each of these differentiable symmetries has associated conservation law under the learning dynamics of gradient flow (section (
3. We construct a more realistic continuous model for stochastic gradient descent by modeling weight decay, momentum, stochastic batches, and finite learning rates (section S
4. We show that under this more realistic model the conservation laws gradient flow are bro- ken, yielding simple ODEs governing the dynamics for the previously conserved parameter combinations (section 6)
5. We solve these ODEs to derive exact learning dynamics for the parameter combinations, which we validate empirically on VGG-16 trained on Tiny ImageNet with and without batch normalization (section
RELATED WORK
The goal of this work is to construct a theoret- ical framework to better understand the learn- ing dynamics of state-of-the-arr neural networks trained on real-world datasets. Existing works have made progress towards this goal through major simplifying assumptions on the architec- ture and learning rule. (Saxe al. (2013) 2019) and Lampinen & Ganguli (2018) considered linear neural networks with specific orthogo- nal initializations, deriving exact solutions for the learning dynamics under gradient flow. The theoretical tractability of linear networks has further enabled analyses on the properties of loss landscapes Kawaguchil (2016), convergence Arora al.) (2018a): Du & Hu (2019), and
5
"Ter uaps
Tao dp Ago)
(a) Parameter Dynamics (b) Neuron Dynamics
Figure 1: Neuron level dynamics simpler than parameter dynamics. We plot the per- parameter dynamics (left) and per-channel squared Faclidean norm dynamics (right) for the convo- lutional layers of VGG-16 model (with batch normalization) trained Tiny ImageNet with SGD with learning rate 7 = 0.1, weight decay 10-4, and batch size S 256. While the pa- rameter dynamics are noisy and chaotic, the neuron dynamics smooth and patterned.
implicit acceleration by overparameterization Arora et (00T8)) Saad & Solla ( (1955) and Goldt et al. 2 studied single hidden layer architectures with non-lincarities in a student- teacher setup, deriving set of complex ODEs describing the learning dynamics. Such shal- low neural networks have also catalyzed recent major advances in understanding convergence
properties of neural networks Du al. (2018b); Mei al. (2018). Jacot al. (2018) considered infinitely wide neural networks with non non-lineen -ineeiiaassssss, demonstrating that the network's prediction becomes linear its parameters. This setting allows for an insightful mathematical formulation of the network's learning dynamics as form of kernel regression where the kernel is defined by the initialization (though see also Fort et al. (2020))- AArora et al. (2019) extended these results to convolutional networks Lee et al. (2019) demonstrated how this understanding also allows for predictions of parameter dynamics.
In the present work, we make the first step in orthogonal direction. Instead of introducing unrealistic assumptions, we discover restricted combinations of parameters for which we can find exact solutions, shown in Fig. [I] We make this fundamental contribution by constructing framework harnessing the geometry the loss shaped by symmetry and realistic continuous equations learning.
Geometry of the loss. A wide range of literature has discussed constraints on gradients originating from specific architectural building blocks of networks. For the first part of our work, we simplify, unify, and generalize the literature through the lens of symmetry.
The carliest works understanding the importance of invariances in neural networks come from the loss landscape literature Baldi & Hornik (989) and the characterization of critical points in the presence
2

---

Published as a conference paper at ICLR 2021
of explicit regularization Kunin al. (2019) More recent works have studied implicit regularization originating from linear Arora et al. (2018b) and homogeneous Du et al. (2018a) activations, finding that gradient geometry plays important role in constraining the learning dynamics. A different line of research studying the generalization capacity of networks has noticed similar gradient structures Liang et al. (2019). Beyond theoretical studies, geometrie properties of the gradient and Hessian have been applied to optimize Neyshabur et al.) (2015). interpret Bach et al. (2015), and prune Tanaka et al. (2020) neural networks.
Gradient properties introduced by batch loffe & Szegedy (2015). weight Salimans & Kingma (2016) and layer Ba et al.) (2016) normalization have been intensely studied. Van Laarhoven (2017) 7; Zhang et al.] (2018) showed that normalization layers are scale invariant, but have an implicit role in controlling the effective learning rate. Cho & Lee 2017); Hoffer et al. (2018); Chiley et al. (2019);
[i & Arora (2019); |Wan et al. (2020) have leveraged the scale invariance of batch normalization to understand geometric properties of the learning dynamics. Most recently, [Li et al. (2020) studied
the role of gradient noise in reconciling the empirical dynamics of batch normalization with the theoretical predictions given by continuous models of gradient descent.
Equations of learning. To make experimentally testable predictions on learning dynamics, we introduce a continuous model for stochastic gradient descent (SGD). There exists a large body of works studying this subject using stochastic differential equations (SDEs) in the continuous-ting
Mandf et al. (2015) 2017); Li et al. (2017), 7); Smith & Le (2017); Chaudhari & Soatto| (2018); Jastrzebski et al. (2017); Zhu al. (2018) An et al. (2018). Each of these works involves making specific assumptions on the loss or the noise in order to derive stationary distributions. More careful treatment of stochasticity led to fluctuation dissipation relationships at steady state without such assumptions Yaida (2018). In our analysis, we apply a more recent approach Li et al. (2017); ); [Barret] & Dherin (2020). inspired by finite difference methods, that augments SDE model with higher-order terms to account for the effect of a finite step size and curvature in the learning trajectory.
SYMMETRIES IN THE LOSS SHAPE GRADIENT AND HESSIAN GEOMETRIES
While we initialize neural networks randomly, their gradients and Hessians all points in train- ing, no matter the loss or dataset, obey certain geometric constraints. Some of these constraints have been noticed previously as a form of im- plicit regularization, while others have been leveraged algorithmically in applications from network pruning to interpretability. Remark- ably, all these geometric constraints can be un- derstood as consequences of numerous differ- entiable symmetries in the loss introduced by neural network architectures. A set of parame- ters observes differentiable symmetry in the loss if the loss doesn't change under a certain dif- ferentiable transformation of these parameters. This invariance introduces associated geometric constraints on the gradient and Hessian.
(a) Translation
(b) Scale
(c) Rescale
Visualizing symmetry. We visual- ize the vector fields associated with simple net- work components that have translation, scale, and rescale symmetry. In (a) we consider the vector field associated with a neuron ([w wz] x) where a is the softmax function. In (b) we con- sider the vector field associated with a neuron BN ([w1 W2] [x] x²)") where BN is the batch normalization function. In (c) we consider the vec- tor field associated with linear path w2wjz.
Figure 2:
Consider a function (0) where 0 E IRT This function possesses a differentiable symmetry if is invariant under the differentiable action 3b of group G on the parameter vector 0, i.e., if - (0, where a G. then F F0,,a)
If (w(0, a)) = (() for all (0, at). The existence of a symmetry enforces a geometric structure on the gradient, FF. Evaluating the gradient at the identity element of G (so that for all 0 (0, I) = 0)
yields the result,
S. 0 b a=1) = 0,
(1)
where (, denotes the inner product. This equality implies that the gradient ff is perpendicular to the vector field o, lo-, that generates the symmetry, for all 0. The symmetry also enforces a geometric structure on the Hessian, HF. Evaluating the Hessian at the identity element yields the
3

---

Published as a conference paper at ICLR 2021
result,
aa +
[8gda (o = 0,
(2)
which constrains the Hessian Hf. See appendix for the derivation of these properties and other geometric consequences of symmetry.
We will now consider the specific setting of a neural network parameterized by 0 E IRm the training loss [()), and three families of symmetries (translation, scale, and rescale) that commonly appear in modern network architectures.
Translation symmetry. Translation symmetry is defined by the group IR and action 0 + 0 + a1A where 1l A is the indicator vector for some subset A the parameters {01..... 0m). The loss function L thus possesses translation symmetry if (()) L(O + al for all a E RR. Such symmetry in turn implies the loss gradient dgL = g is orthogonal to the indicator vector 0g aa 1AA
(g. 1011) 0,
(3)
and that the Hessian matrix H = 0 L has the indicator vector in its kernel,
H = 0.
(4)
Softmax function. Any network using the softmax function gives rise to translation symmetry for the parameters immediately preceding the function. Let z = Wr + b be the input to the softmax function such that o(=); = E Notice that shifting any column of the weight matrix or the
et
bias vector by a real constant has no effect on the output of the softmax as the shift factors from both the numerator and denominator canceling its effect. Thus, the loss function is invariant w.r.t. (
BC this translation, yielding the gradient constraints (Dw. 1) 26 1) = O, visualized in Fig. for the toy model where t. E R?.
Scale symmetry. Scale symmetry is defined by the group GL (R) and action 0 + a ① where оA A 1.4. The loss function possesses scale symmetry if [(0) l(o @ 0) for all GL, (R). This symmetry immediately implies the loss gradient is everywhere perpendicular to the parameter vector itself 0a = 8 ④ 1 = AA.
(g.0~) 0,
(5)
and relates to the Hessian matrix, where (. = diag(1A)9 = 9AA. as
не + -A=0 0.
(6)
Batch normalization. Batch normalization leads to scale invariance during training. Let z = w + be the input to neuron with batch normalization such that BN(z) = =-F(2] where E[2] is the
Var(z)
sample mean and Var(z) is the sample variance given a batch of data. Notice that scaling w and b by a non-zero real constant has no effect on the output of the batch normalization as it factors from 2, E[2), and Var(2) canceling
effect. Thus, these parameters observe scale symmetry in the loss OC and their gradients satisfy ( ( w) + < oh b) = as has been previously noted by Ioffe & Szegedy
(2015): ): Van Laarhoven 2017). and visualized in Fig. for the toy model where w, b é K.
Rescale symmetry. Rescale symmetry is defined by the group GLY (R) and action 0 a ? 272 @ 0 where A₁ and A2 are disjoint sets of parameters. The loss function possesses rescale symmetry if C(0) = L(a,A, @ az, @ 6) for all a E GI. (R). This symmetry immediately implies the loss gradient is everywhere perpendicular to the sign inverted parameter vector 0.0/o= 0A, 0A22
(1A, 1A.).
(g.0A, - A 0
(7)
and relates to the Hessian matrix, where
= diag (14, - 11,99 = 9A, - -A as
H(O.. 0 +A. IA. = 0.
Homogeneous activation. For networks with continuous, homogeneous activation functions Φ(z) = ø'(2)² (e.g. ReLU, Leaky ReLU, linear), this symmetry emerges at every hidden neu- ron by considering all incoming and outgoing parameters to the neuron. For example, consider hidden neuron with ReLU activation Φ(2) = max(0, 2). such that w2o(w) w x + b) is the compu- tational path through this neuron. Scaling w1 and by real constant and w2 by its inverse has
4

---

Published as a conference paper at ICLR 2021
no effect the computational path the constants can be passed through the ReLU activation canceling their effects. Thus, these parameters observe rescale symmetry and their gradients satisfy
Brgg W2 0, as has been previously noted by Du et al. 2018a) Liang et all. 2019); Tanaka et al. (2020), and visualized in Fig. for the toy model where W1, W0 E IR and
ac
W1
+
b)
0C
b 0.
SYMMETRY LEADS TO CONSERVATION LAWS UNDER GRADIENT FLOW
We now explore how geometric constraints on gradients and Hessians, arising as a consequence of symmetry, impact the learning dynamics given by stochastic gradient descent (SGD). We will consider a model parameterized by 0, a training dataset {.1...., ,NN size N, and a training loss
[()) = き Σ z.) with corresponding gradient g(0) 8C
06
The gradient descent update with learning rate " is ((x+1) g(x) ng (0(m), which is for- ward Euler discretization with step size 1y of the ordinary differential equation (ODE)
d0
dt = (t)).
(9)
In the limit as - 0. gradient descent exactly matches the dynamics of this ODE, which is commonly referred to as gradient flow Kush- ner & Yin (2003). Equipped with continuous model for the learning dynamics, we now ask how do the dynamics interact with the geometric properties introduced by symmetries?
(a) Translation
(b) Scale
(c) Rescale
Figure 3: Visualizing conservation. Associated with each symmetry is a conserved quantity con- straining the gradient flow dynamics to face. For translation symmetry (a) the flow is constrained hyperplane where the intercept is conserved. For scale symmetry (b) the flow is con- strained sphere where the radius is conserved. For rescale symmetry (c) the flow is constrained hyperbola where the axes are conserved. The color represents the value of the conserved quan- tity, where blue is positive and red is negative, and the black lines are level sets.
Symmetry leads to conservation. Strikingly similar to Noether's theorem, which describes a fundamental relationship between symmetry and conservation for physical systems governed by Lagrangian dynamics, here we show that symmetries of network architecture have corre- sponding conserved quantities through training under gradient flow.
Theorem 1. Symmetry and conservation laws in neural networks. Every differentiable symmetry (a, 8) of the loss that satisfies (8, [0 de (=)) = 0 has the corresponding conservation law,
a (0, 0. = 0,
(10)
through learning under gradient flow.
To prove Theorem D consider projecting the gradient flow learning dynamics in equation 9 onto the generator vector field 00 associated with a symmetry. As shown in section the gradient of the loss g(0) is always perpendicular the vector field da ao Thus, the projection yields a differential equation, which can be simplified to equation (10). See appendix for complete derivation.
Application of this general theorem to the translation, scale, and rescale symmetries, identified in section 3 yields the following conservation law of learning.
Translation: (P,(t), 11 (0),
(11)
Scale: [0,(t))² = (0)
(12)
Rescale: |8A, (t) 10,,
(1) (1) 110 (0)
(13)
Each of these equations define conserved constant of learning through training. For parameters with translation symmetry, their sum (t), 1)) is conserved. effectively constraining their dynamics to hyperplane. For parameters with scale symmetry, their euclidean norm (1) is conserved,
5

---

Published as a conference paper at ICLR 2021
effectively constraining their dynamics to aphereloffe & Szegedy (2013) ; Van Laarhoven 2017 For
parameters with rescale symmetry, their difference in squared euclidean norm ( 1 (t)
A )))
is conserved, effectively constraining their dynamics to hyperbola Du et al. (2018a) In Fig. 3 we
visualize the level sets of these conserved quantities for the toy models discussed in Fig.
A REALISTIC CONTINUOUS MODEL FOR STOCHASTIC GRADIENT DESCENT
In section we combined the geometric constraints introduced by symmetries with gradient flow to derive conservation laws for simple combinations of parameters during training. However, empirically we know these laws are broken, demonstrated in Fig. I What causes this discrepancy? Gradient flow is too simple of continuous model for realistic SGD training. It fails to incorporate the effect of weight decay introduced by explicit regularization, momentum introduced by commonly used hyperaarameters stochasticity introduced by random batches, and discretization introduced by finite learning rate. Here, we construct a more realistic continuous model for stochastic gradient descent.
Modeling weight decay. Explicit regularization through the addition of an Le penalty on the parameters, with regularization constant A, is very common practice when training modern deep learning models. This is generally implemented not by modifying the training loss, but rather directly modifying the optimizer's 's update equation. For stochastic gradient descent, the result leads to the updated continuous model
d8 dt
90) A0.
(14)
Modeling momentum. Momentum is a common extension to SGD that uses an exponentially moving average of gradients to update parameters rather than a single gradient evaluation Rumelhart ct al. 1986). The method introduces two additional hyperparameters, a damping coefficient a and a momentum coefficient B, and applies the two step update equation, ((m+1) = @(") mn(n+1) where ((x+1) Bp(x) + (1 ())(()))). When a = B = 0, we regain classic gradient descent. In general, effectively reduces the learning rate and 9 controls how past gradients are used in future updates resulting in a form of "inertia" accelerating and smoothing the descent trajectory. Rearranging the two-step update equation, we find that gradient descent with momentum is first-order discretization with step size 1) a) of the ODF
dB
(1 (1-3) dt -
(15)
Modeling stochasticity. Stochastic gradients (+) arise when consider batch B of size S drawn uniformly from the indices (1...., .N forming the unbiased gradient estimate 9s(()) = $ Eies vl(0, When the batch size is much smaller than the size of the dataset, S < N, then we can model the batch gradient as an average of S i.i.d. samples from a noisy version of the true gradient g0). Using the central limit theorem, we assume (0) g()) is Gaussian random variable with mean 0 and covariance matrix (()). However, because both the batch gradient and true gradient observe the same geometric properties introduced by symmetry, the noise has a special low-rank structure. As we showed in section
3 the gradient of the loss, regardless of the batch, is orthogonal to the generator vector field o, وا a= associated with a symmetry. This implies the stochastic noise must also observe the same property. In order for this relationship to hold for arbitrary noise, then (00) = O. In other words, the differential symmetry inherent in neural network architectures projects the noise introduced by stochastic gradients onto low rank subspaces, leaving the gradient flow dynamics in these directions unchanged
Modeling discretization. The effect of discretization when modeling continuous dynamics is well studied problem in the numerical analysis of partial diffcrential equations. One tool commonly used in this setting, is modified equation analysis Warming & Hyett (1974), which determines how to better model discrete steps with a continuous differential equation by introducing higher order spatial or temporal derivatives. We present two methods based on modified equation analysis, which modify gradient flow to account for the effect of discretization.
A more detailed derivation for this ODE can be found in appendix SStochasticity and discretization can lead to non-trivial interactions requiring analysis using stochastic differential equation as explained in appendix F

---

Published as a conference paper at ICLR 2021
Modified loss. Gradient descent always moves in the direction of steepest descent on a loss function L at each step. however, due to the finite nature of the learning rate, it fails to re- main the continuous steepest descent path given by gradient flow. Li et al. (2017); Feng al. (2019) and most recently Barrett Dherin (2020), demonstrate that the gradient descent trajectory closely follows the steepest descent path of modified loss function L. The diver- gence between these trajectories fundamentally depends on the learning rate 1) and the curvature H. As derived in Barrett & Dherin (2020), and summarized in appendix DD this divergence is given by the gradient correction - HgH. which
(a) Modified Loss
(b) Modified Flow
Figure Modeling discretization. We visual- ize the trajectories of gradient descent and mo- mentum (black dots), gradient flow with and with- out momentum (blue lines), and the modified dy- namics (red lines) on the quadratic loss [(w) = wT 2.5 -1.51 w. On the left we visualize gradi- 2
is the gradient of the squared norm 7 /VC Thus, the modified loss Č = C+ V and the modified gradient flow ODE is
ent dynamics using modified loss. On the right we visualize momentum dynamics using modified flow. In both settings the modified continuous dy- namics visually track the discrete dynamics better than the original continuous dynamics. See ap- pendix D for further details.
d0 dt 0
! - (0))(()
(16)
See Fig. 4 for an illustrative example of this method applied to quadratic loss in IR2.
Modified flow. Rather than modifying gradient flow with higher order "spatial" derivatives of
the loss function, here we introduce higher order temporal derivatives. We start by assuming the existence of a continuous trajectory {(t) that weaves through the discrete steps taken by gradient descent and then identify the differential equation that generates the trajectory. Rearranging the update equation for gradient descent, 1,41 0, 7g(())) and assuming ((t) O, and ((t+)) 8.4 gives the equality -g( = (((+n))((() which Taylor expanding the right side results in the differential + y do
equation -g(Ot) = dt dr + O Notice that in the limit as 17 - O we regain gradient flow. For small 17, we obtain a modified version of gradient flow with an additional second-order term,
d0
d8
n 8 2 di
-
(17)
di
This approach to modifying first-order differential equation with higher order temporal derivatives was applied by Kovachki & Stuart (2019) to construct more realistic continuous model for momentum, as illustrated in Fig.
6 COMBINING SYMMETRY AND MODIFIED GRADIENT FLOW TO DERIVE EXACT LEARNING DYNAMICS
As shown in section 4 each symmetry results in a conserved quantity under gradient flow. We now study how weight decay, momentum, stochastic gradients, and finite learning rates all interact to break these conservation laws. Remarkably, even when using a more realistic continuous model for stochastic gradient descent, as discussed in scction we can derive exact learning dynamics for the previously conserved quantities. To do this we (i) consider realistic continuous model for SGD, (ii) project these learning dynamics onto the generator vector fields 0. la= associated with each symmetry, (iii) harness the geometric constraints from section (3 derive simplified ODEs, and (iv) solve these ODEs to obtain exact dynamics for the previously conserved quantities. For simplicity, we first consider the continuous model of SGD incorporating weight decay and modified loss, but not momentum and stochasticity F this setting, the exact dynamics, as fully derived in appendix for the parameter combinations tied to the symmetries are,
"See appendix P for a discussion how handle stochasticity in this setting using Itô calculus.
7

---

Published as a conference paper at ICLR 2021
Translation:
(0(t). 12) = = -0-(((00)))) (0(())),
(18)
10a(1)) - E =220000(0)) +1 +nI e-2 2 dt
Scale:
(19)
Rescale:
10x, - 10,2
(20)
--2M(1-)) + (0)
(19e2) 1- -190 12) dr
Notice how these equations are equivalent to the conservation laws derived in section when ? À 0. Remarkably, even in typical hyperparameter settings (weight decay, stochastic batches, finite learning rates), these solutions match nearly perfectly with empirical results] from modern neural networks (VGG-16) trained on real-world datasets (Tiny ImageNet), as shown in Fig. 5 We will now discuss each equation individually.
Translation dynamics. For parameters with translation symmetry, equation 18 implies that the sum of these parameters (t) 1)) decays exponentially to zero at a rate proportional to the weight decay. Equation 18 does not directly depend on the learning rate 7) nor any informa- tion of the dataset or task. This is due to the lack of curvature in the gradient field for these parameters (as shown in Fig. 2). This implies that at initialization we can deterministically pre- dict the trajectory for the parameter sum as sim- ple exponential functions with rate defined by the weight decay. The first row in Fig. [
A 0
A 10
==
10
0.30
Translation
(BV 13
0.00
00.55
-0.30
Scale
||
1.0
0.5
0.45
demonstrates this qualitatively, as all trajectories are smooth exponential functions that converge faster for increasing levels of weight decay.
Rescale
|| [|W
0.15
0.00
-0.5
Scale dynamics. For parameters with scale symmetry, equation 19 implies that the norm for these parameters (lAAl² is the sum of an expo- nentially decaying memory of the norm initial- ization and an exponentially weighted integral of gradient norms accumulated through training. Compared to the translation dynamics, the scale dynamics do depend on the data through the gra- dient norms accumulated throughout training. Without weight decay X = 0, the first term stays constant and the second term grows monotoni- cally. With weight decay
Time
steps)
Figure 5: Exact dynamics of VGG-16 on Tiny ImageNet. We plot the column sum of the final linear layer (top row) and the difference between squared channel norms of the fifth and fourth con- volutional layer (bottom row) of a VGG-16 model without batch normalization. We plot the squared channel norm of the second convolution layer (mid- dle row) of a VGG-16 model with batch normal- ization. Both models are trained on Tiny ImageNet with SGD with learning rate 7 0.1, weight decay X, batch size S 256, for 100 epochs Colored lines are empirical and black dashed lines are the theoretical predictions from equations (18). (I9). and (20). See appendix J for more details on the experiments.
0, the first term decays monotonically to zero, while the second term can decay or grow, but always stays pos- itive. The second row in Fig. 5 demonstrates these qualitative relationships. Without weight decay the norms increase monotonically as pre- dicted and with weight decay the dynamics are non-monen non-monotonic and present more complex be- havior. To better understand the forces driving
these complex dynamics, we can examine the time derivative of equation 19
dt (1)) = (1) + 19.1
(21)
From this equation decay (-2X(0xA())3 and a centrifugal effect due discretization (1 (9 The centripetal effect
that there is competition between centripetal effect due to weight
*See appendix D for more details on the experiments and how we compute the integrals terms in the exact solutions using stochastic gradients.
8

---

Published as a conference paper at ICLR 2021
due to weight decay is direct consequence of its regularizing influence, pulling the parameters towards the origin. The centrifugal effect due to discretization originates from the spherical geometry of the gradient field in parameter space because scale symmetry implies the gradient is always orthogonal to the parameter itself, each discrete update with a finite learning rate effectively pushes the parameters away from the origin. At the stationary state of the dynamies, these forces will balance leading the dynamics of these parameters to be constrained to the surface of high-dimensional sphere. In particular, at stationarity, then d. (1)) = which rearranging equation 21 gives the
condition (tt) = de de
/10| = v 2A. Consistent with the results of Wan et al. (2020), this implies that
at stationarity the angular speed (t) of the weights is constant and governed only by the learning rate 7) and weight decay constant X. When considering stochasticity explicitly, as explained in appendix [F] then these dynamics also depend on the covariance of the gradient noise ()) and batch size S.
Rescale dynamics. For parameters with rescale symmetry, equation 20 is the sum of an exponen- tially decaying memory of the difference in norms at initialization and an exponentially weighted integral of difference in gradient norms accumulated through training. Similar to the scale dynamics, the rescale dynamics do depend on the data through the gradient norms, however unlike the scale dynamics we have guarantee that the integral term is always positive. This leads to quite sophisti- cated, complex dynamics, consistent with the third row in Fig. 5 Despite the complexity, our theory, nevertheless, quantitatively matches the empirics. The only apparent pattern from the empiries is that for large enough weight decay, the regularization dominates any complexity introduced by the gradient norms and the difference in parameter norms decays exponentially zero
Harmonic oscillation with momentum. We will now consider continuous model of SGD
8 0
B 0.9
0 0.99
with momentum. As discussed in appendix G we consider the continuous model incorporat- ing weight decay, momentum, stochasticity, and
0.30
0.15
modified flow. Under this model, the solutions we obtain take the form of driven harmonic os- cillators where the driving force is given by the gradient norms, the friction is defined by the momentum constant, the spring coefficient is defined by the regularization rate, and the mass is defined by the the learning rate and momen- tum constant. For most standard hyperparameter choices, these solutions are in the overdamped setting and align well with the first-order solu- tions for SGD without momentum up to a time rescaling, as shown in the left and middle panel of Fig. G However, for large values of beta we can push the solution into the underdamped regime where we would expect harmonic oscil- lation and indeed, we can empirically verify our predictions, even at scale for VGG-16 trained on Tiny ImageNet, in right panel of Fig. 6
-0.15
-0.3
0.0
0.4
0.8
1.2
1.6
0.0
0.4 0.8 1.2
1.6
0.0 0.4 0.8 1.2 1.6
Time (y) stepp))
Figure 6: Momentum leads to harmonic oscil- lation. We plot the column sum of the final linear layer of VGG-16 model (without batch normalization) trained on Tiny ImageNet with SGD with learning rate 7 0.1, weight decay À = x 10 batch size S = 256 and momentum E {0, 0.9, 0.99). Colored lines are empirical and black dashed lines are the theoretical predic- tions from equations (34).
7 CONCLUSION
Despite being the central guiding principle in the exploration of the physical world Anderson (972);;
;
Gross (199)), symmetry has been underutilized in understanding the mechanics neural networks In this paper, we constructed unifying theoretical framework harnessing the geometric properties of symmetry and more realistic continuous equations for learning dynamics that model weight decay, momentum, stochasticity, and discretization. We use this framework to derive exact dynamics for meaningful combinations of parameters, which we experimentally verified on large scale neural networks and datasets. For example, in the case of VGG-16 model with batch normalization trained on Tiny- ImageNet (one of the model/dataset combinations we considered in section (6) there are 12, 751 distinct parameter combinations whose dynamics can analytically describe. Overall, this work provides a first step towards understanding the mechanics of learning in neural networks without unrealistic simplifying assumptions.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We thank Daniel Bear, Lauren Gillespie, Kyogo Kawaguchi, Brett Larsen, Eshed Margalit, Alain Studer, Sho Sugiura, Umberto Maria Tomasini, and Atsushi Yamamura for helpful discussions. This work was funded in part by the IBM-Watson AI Lab. D.K. thanks the Stanford Data Science Scholars program for support. J.S. thanks the Mexican National Council of Science and Technology (CONA- CYT) for support. S.G. thanks the James S. McDonnell and Simons Foundations, NTT Research, and an NSF CAREER Award for support. D.L.K.Y thanks the McDonnell Foundation (Understanding Human Cognition Award Grant No. 220020469), the Simons Foundation (Collaboration the Global Brain Grant No. 543061), the Sloan Foundation (Fellowship FG-2018 8- 10963). the National Science Foundation (RI 1703161 and CAREER Award 1844724), and the DARPA Machine Common Sense program for support and the NVIDIA Corporation for hardware donations.
REFERENCES
Jing An, Jianfeng Lu, and Lexing Ying. Stochastic modified equations for the asynchronous stochastic gradient descent. Information and Inference: A Journal of the IMA, 2018.
Philip W Anderson. More is different. Science, 177(4047)3393-339) 1972.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. arXiv preprint arXiv:1810.02281, 2018a.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparamettriiattin.. arXiv preprint arXiv: 882006609,, 2018b.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. arXiv preprint 0rXiv:1812.03980 2018c
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, pp. 8141-8150 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 2016.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation. PloS one, 10(7), 2015.
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009. /1 162, 2020.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-10. IEEE, 2018.
Vitaliy Chiley. Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. In Advances in Neurul Information Processing Systems, pp. 8433-8443, 2019.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5225-5235, 2017.
Simon S Du and Wei Hu. Width provably matters in optimization for deep linear neural networks.
arXiv preprint arXiv:1901.0853 2019.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances Neural Information Processing Systems, pp. 384-395, 2018a.
10

---

Published as a conference paper at ICLR 2021
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018b.
Yuanyuan Feng. Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu. Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation. arXiv preprint arXiv:1902.00633, 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Adv. Neural Inf. Process. Syst., 33, 2020.
Sepideh Kharaghani, Daniel M Roy,
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In Advances in Neural Information Processing Systems. pp. 6981-6991, 2019.
David Gross. The role of symmetry in fundamental physics. Proceedings of the National Academy of Sciences, 92(25)::4256-144) 1996.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623. 2017.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.
Nikola B Kovachki and Andrew M Stuart. Analysis of momentum methods. arXiv preprint /906::9906.0422 04285, 2019.
Daniel Kunin, Jonathan M Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. arXiv preprint arXiv:1901.08.66 2019.
Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applica- tions, volume 35. Springer Science & Business Media, 2003.
Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. In International Conference on Learning Representations (ICLR), 2018.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pp. 8572-8583, 2019.
Qianxiao Li. Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101-2110, 2017.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07455 2019.
Zhiyuan Li, Kaifeng Lyu. and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020.
11

---

Published as a conference paper at ICLR 2021
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). arXiv preprint arXiv:2102.12470, 2021.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 888-896, 2019.
Stephan Mandt, Matthew D Hoffman, and David M Blei. Continuous-time limit of stochastic gradient descent revisited. NIPS-2015, 2015.
Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873-4907 2017.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33)EE765 E7671, 2018.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, pp. 2422 2430,, 2015.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6688) 1986.
David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. Advances in neural information processing systems, 8:302-308, 1995.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901-909, 2016.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 13Xiv:13/2.6120, 2013.
Andrew M Saxe, James L McClelland, and Surya Ganguli. mathematical theory of semantic development in deep neural networks. Proc. Natl. Acad. Sei. U. S. A., May 2019.
Samuel I. Smith and Quoe V Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv: 1710.06451, 2017.
Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. In Advances in neural information processing systems, pp. 2510-251 151851 2014.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruming neural networks without any data by iteratively conserving synaptic flow. arXiv preprint (eXii:2006.0546) 2020.
Twan Van Laarhoven. I.2 regularization versus batch and weight normalization. arXiv preprint arXiv: 1rXiv:/706.053500 2017.
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics of deep neural networks with batch normalization and weight decay. arXiv preprint urXiv:2006.08411 2020.
RF Warming and BJ Hyett. The modified equation approach to the stability and accuracy analysis of finite-difference methods. Journal of computational physics, 14(2):159-179, 1974.
Sho Yaida. inncuuation-disipattii relations for stochastic gradient descent. arXiv preprint arXiv:1810.00004, 2018
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 1810. 12281, 2018
Zhanxing Zhu, Jingfeng Wu, Bing Yu. Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv: 1803.00 00/95 2018.
12

---

Published as a conference paper at ICLR 2021
A SYMMETRY AND GEOMETRY
Here we derive in detail the geometric properties of the loss landscape introduced by symmetry, as discussed in section 3 Consider function (0) where I Rm. This function possesses symmetry if it is invariant under the action v of a group G on the parameter vector 0, i.e., if 0 + (0 a) where a G. then F' (0, relationship between the gradient F and Hessian HF of the composition F(O, a) with the gradient ff and Hessian Hf of the original function f()). This relationship can be described by five constraints on Vʃ and Hf. Considering these general formulae when f(0) L(0), the training loss of neural network, yields fifteen distinct equations describing the geometrical relationships between architectural symmetries and the loss landscapes, some of which have been identified individually in existing literature (Table
(v)(0, a)) f(0) for all (0, Symmetry enforces geometric
Translation
Scale
Rescale
0g F 0g F
1,
6
VF
3,4
7,8,9
8 F
3.5
HF
0g 0 F
9
Table I: Unifying existing literature through symmetry. Here we provide references to existing literature describing geometric properties of either the gradient Hessian introduced by network's architecture. All of these properties can be unified as consequences of either a translation, scale, or rescale symmetry in the training loss.
1. loffe & Szegedy 2015) has motivated the effectiveness of Batch Normalization by its scale invariant property. In particular, they have noted that Batch Normalization will stabilize back-propagatie because "The scale does not affect the layer Jacobian nor, consequently, the gradient propagation. Moreover, larger weights lead smaller gradients, and Batch Normalization will stabilize the parameter growth."
2. Van Laarhoven (017) has then shown that the role of I. regularization when combined with batch loffe & Szegedy (2013), weight [Salimans & Kingma (2016) layer [Ba et al. (2016) normalization is not to regularize the function, but to effectively control the learning rale.
3. Zhang et al. (018) has thoroughly studied mechanisms of weight decay regularization and derived various geometric properties of loss landscapes along the way. AArora et al. (2018c) has theoretically analyzed the automatic tuning property of learning rate in networks with Batch Normalization.
4. Li et al. (020)) studied the interaction of weight decay and batch normalization in the setting of stochastic gradients.
5. Neyshabur et al. (2015) has identified that SGD is not rescale equivariant even when network outputs are rescale invariant. This is problem because gradient descent performs very poorly on unbalanced networks due to the lack of equivariance. Motivated by the issue, they have introduced a new optimizer, Path-SGD, that is rescale equivariant.
6. Arora et al. 2018b) proved that the weights of linear artificial neural networks satisfy strong balancedness property.
7. Du et al. 201 8a) studied implicit regularization in networks with homogeneous activation functions. To do that, they showed conservation law of parameters with rescale invariance.
8. Liang et al. (2019) have proposed new capacity measure to study generalization that respects rescale invariance of networks. Along the way, they showed geometric properties of gradients and Hessians for networks with rescale invariance. However, their results were restricted to a layer without biases.
9. Tanaka et al. 2020) have proved gradient properties of parameters with rescale invariance neuron level including biases.
13

---

Published as a conference paper at ICLR 2021
A.1 GRADIENT GEOMETRY
If a function /f posses a symmetry, then there exists a geometric constraint on the relationship between the gradients VF and f at all a))
0gF o.F
Oi Fo. Og.. Fd
%
VF=
The top element of the gradient relationship, Fg evaluated at any (0, a),, yields the property
Oayi V (/o.o)> V/lb
(22)
which describes how the symmetry transformation affects the function's is gradients despite leaving the output unchanged. The bottom element of the gradient relationship, 0. F, evaluated at the identity element of G' yields the property
V 8 (==1) = 0,
(23)
which implies the gradient Vf is perpendicular to the vector field 0a that generates the symmetry, for all In the specific setting when f(0) (()), the training loss of a neural network. these gradient properties are summarized in Table 2 for the translation, scale, and rescale symmetries described in section
Translation
Scale
Rescale
9 (v((0,)) o:) diag(o (V((, (W((,)) diag(o @ a o (v) AA, 8
(
1A
0
Table 2: Geometric properties of the gradient. The gradients of a neural network with either translation, scale or rescale symmetry observe certain geometric properties no matter the dataset or step in training.
Notice that the first row of Table Taalle[implie implies that symmetry transformations affect learning dynamics governed by gradient descent for scale and rescale symmetries, while it does not for translation symmetry. These observations are in agreement with Van Laarhoven (2017) who has shown that effective learning rate is inversely proportional to the norm of parameters immediately preceding the batch normalization layers and Neyshabur al. (2015) who have noticed that SGD not invariant to the rescale symmetry that the network output respects and proposed Path-SGI to fix the discrepancy.
A.2 HESSIAN GEOMETRY
If a function posses a symmetry, then there also exists a geometric constraint on the relationship between the Hessian matrices HF and Hf at all (0, a),
FFF 0,O.F) O,0₂F 8211
HF-
Fo + F( = 02 v0.0 vb + FOg Fa, (0 (0,
+ +
Hf 9)
The first diagonal element, 0g F. evaluated any (0, a), yields the property
0y V fl - (0 Hf Hfl Hfl
(24)
which describes how the symmetry transformation affects the function's Hessian despite leaving the output unchanged. The off-diagonal elements, O,O, F O. evaluated at the identity element of G yields the property
H|000th..=10..v0o.a + |0,0.lo=]0VV 0,
(25)
which implies the geometry of gradient and Hessian are connected through the action of the symmetry. Lastly, the second diagonal element, 02 F, represents an equality, evaluated at the identity element of G' yields the property
(0. (O..)).=))H/(((a/ ( (l 0,
(26)
14

---

Published as a conference paper at ICLR 2021
which combines the geometrie relationships in equation 23 and equation 25 In the specific setting when f0) = [()), the training loss neural network, these Hessian properties are summarized in
Table 2 for the translation, scale, and rescale symmetries described in section 3
Translation
Scale
Rescale
H())- H()(0, 0 =
diag diag(o3)HH
diag(a @ A, (1
H1A
Нθ + 94 01 AO
H(BA, 8A. 9A1 9A2
H1A
(0Al - - 0A.) + OA + 9.A. 9.20
Table 3: Geometric properties of the Hessian. The Hessian matrix of neural network with either translation, scale or rescale symmetry observe certain geometric properties no matter the dataset or step in training.
The translation, scale, and rescale symmetries identified in section 3 is not an exhaustive list of symmetries present in neural network architectures. For example, more general rescale symmetries can be defined by the group GLI (R) and action + a @ @a k2 @ 0, which occur in networks with quadratic activation functions. stronger form of rescale symmetry also occurs for linear networks under the action of the group GL (R) of & invertible matrices, noticed previously by Arora et al. (2018a); Du et al. (2018a). Interestingly, some of the gradient and Hessian properties for scale symmetry also be easily proven consequences of Euler's Homogeneous Function Theorem when k 0.
B CONSERVATION LAWS
Here we repeat Theorem ! and provide a detailed derivation.
Theorem. Symmetry and conservation laws in neural networks. Every differentiable symmetry (a, 8) of the loss that satisfies (6, (0,00v)a=1lg(0) = 0 has the corresponding conservation law
(0, 0, bb a=1) = 0 through learning under gradient flow.
Proof. Project the gradient flow learning dynamics, de
g(0), onto the vector field that generates
de
the symmetry 0, v. -== evaluated at the identity element,
d0
v== = (-g(0), =0. (
0 2
We can factor the left side of this equation as
d8 di
d
d (0, 0g
0g
(a=1)==)>
/ (0, 0 00,0, (aai))
a (a=1)
di
d
d0
(9, O, (m 0,
dt
d (0, 0 vl (a + (0, 8, (00000000==1000))
By assumption, (0, 18.000vla=1 0. implying d (0, (a-)) 0.
The condition 0. = 0 holds for the translation, scale, and rescale symmetries we
consider in section For translation symmetry, 0, 0, n- 0. For scale symmetry, O, 000
0
I and (0A. ()) 0. For rescale symmetry, ds a= -1 and (0..g(0AA))) A0 A.)A))) 0.
15

---

Published as a conference paper at ICLR 2021
C LIMITING DIFFERENTIAL EQUATIONS FOR LEARNING RULES
Here we identify ordinary differential equations whose first-order discretization give rise to the gradient descent and classical momentum algorithms. These differential equations can be understood as the limiting dynamics for their respective discrete algorithms as the learning rate + 0.
C.1 GRADIENT DESCENT
Gradient descent with learning rate v is given by the update equation
0x+1 02 (0{))>>
and initial condition 0g Rearranging the difference between consecutive updates gives
Bk+1 0x (0xx).
1
This is discretization with step size n the first order ODE
d 9 -
dt
where we used the forward Euler discretization gradient flow.
& 02 0
This ODE is commonly referred to
"
C.2 CLASSICAL MOMENTUM
Classical momentum with learning rate damping coefficient a, and momentum parameter (3, is given by the update equation
xk+1 Big + - a)g())))))
0x+1 = 02 1jūk+1,
and initial conditions Uo and some fg. The difference between consecutive updates
0x+1 02 1
= B1BBk 17(1 - ())((s)))
3 (Os, (x-1) n11 ())))x))
Rearranging this equation we get
0x41 02
Ox 02-1 -g(a)
77 a)
17(1 a)
This is discretization with step size 17(1 - (x) of the first order ODE
d
(1 3)
g()),
dt
where we used the forward Euler discretization 4 0,
- and backward Euler discretization
dl di
0x 0x-8 1ー-) We will refer to this equation as momentum flow. A more detailed derivation
for this ODE under Nesterov variants of classical momentum can be found in Su et al. (0014) and Kovachki & Stuart (2019).
'The default PyTorch implementation does not perform damping the gradient in the first momentum
buffer U1-
16

---

Published as a conference paper at ICLR 2021
D MODIFIED EQUATION ANALYSIS
dicrrete dyram cs
Gradient descent always moves in the direction of steepest descent on a loss function, however, due the finite nature of the learning rate, it fails to remain on the continuous steepest descent path. The divergence between the discrete and continuous trajectories fundamentally depends on the learning rate and the curvature of the loss. It thus natural to assume there exists more realistic continuous model for SGD that incorporates both these terms non-trivial way. How can we better model the discrete dynamics of gradient descent with a continuous differential equation?
dynomiss
Intuition from finite difference methods. To answer this question we will take inspiration from tools developed for finite difference methods. Finite difference methods are a class of numerical techniques for approximating deriva- tives in the analysis of partial differential equations (PDE). These approximations are applied iteratively to construct numerical solutions to PDE given some initial conditions, However, this discretization process introduces numerical artifacts, which can lead a significant difference between the numerical solution and the true solution for the PDE. method for understand- ing this difference by modeling the numerical artifacts as higher-order spatial or temporal derivatives modifying the original PDE. This approach can be used to construct mod- ified continuous dynamics that better approximate discrete dynamics, illustrated in Fig.
modified corcnuous dynamics
Figure 7: Circular motion. Consider the vector field f(x) = [---] and the discrete dynamics x++11 2t + (xa) (black dots), the continuous dynamics i f(x) (blue line), and the modified continuous dynamics ă = f(x) + Za. We visualize the trajectories given by these dynamics using the initial condi- tion In [1 [1 (white circle) and step size = 0.1. As we can see, the modified continuous trajectory better matches the discrete trajectory.
Modified equation analysis
D.1 MODIFIED Loss
Taking inspiration from modified equation analysis. Li ct al. (017)); Feng et al. 2019) and most recently Barrett & Dherin 2020). demonstrate that the trajectory given by gradient descent closely follows the steepest descent path modified loss function Σ, rather than the original loss C. As explained in Barrett& & Dherin (2020), assume there exists modificd vector field with corrections ga in powers of the learning rate to the original vector field g that the discrete dynamics follow. In other words, rather than considering the dynamics given by gradient flow, d -(0), we consider the modified differential equation,
dt 0 = - (0) (²)g2(())
Truncating the modified vector field up to the order r7 and using backward error analysis we can derive that the first-order correction = 1/2 Hg. which is the gradient of the squared norm | Thus, the truncated modified differential equation simply gradient flow on modified loss E L + 1
Convex quadratic loss. To illustrate modified loss, we will consider the trajectories of gradient descent w/: gradient flow vi()), and modified gradient flow w(l) on the convex quadratic loss L(w) = 1 wT Aw, where A > 0 is some positive definite matrix, as shown in Fig. 4 For a finite learning rate 7, and initial condition wo. gradient descent is given by the update formula d
w = wi AAw, Gradient flow is defined as di, _ = -Aù, a linear first-order differential equation. For the initial condition Wa. the resulting initial value problem can be solved exactly giving the gradient flow trajectory (i) = S
5-1 e - Swo. where A = S- AS is the diagonalization of the curvacture matrix. For learning rate the modified loss is [(w) w Aw + fww A and modified gradient flow is defined as d w resulting initial value problem can be solved exactly giving the modified gradient flow trajectory
Aw z A. For the initial condition wo. the
(1) S-Le -0-- AA) Swp-
17

---

Published as a conference paper at ICLR 2021
D.2
MODIFIED FLOW
Rather than modify gradient flow with higher order "spatial" derivatives of the loss function, here we introduce higher order temporal derivatives. We start by assuming the existence of continuous
trajectory 0(1) that weaves through the discrete steps taken by SGD and then identify the differential
equation that generates the trajectory. Rearranging the update equation for SGD, 0c+1 = 0r - nge (0:).
and assuming (() O, and 0(l + 7) - Oc+1, 1. gives the equality - -95( (0,) = 0it+n) - which
Taylor expanding the right side results in the differential equation (0,) Notice that in the limit as 7 + we regain gradient flow. For small 1). 17 < 1, we obtain modified version of gradient flow with an additional second-order term. This approach of modifying first-order differential equations with higher order temporal derivatives was applied by Kovachki & Stuart (2019)
de +
deo + O(n").
di
2 dr
to modify momentum flow, capturing the harmonic motion of momentum.
Convex quadratic loss. To illustrate modified flow, we will consider the trajectories of momentum We, momentum flow vi))) and modified momentum flow ((t) on the convex quadratic loss [(w) 2 Aw where A > 0 is some positive definite matrix, as shown in Fig. 4 For a finite learning rate 77, dampening = O, momentum constant B, and initial conditions do 0, Wo, then momentum is given by the pair of recursive update equations 1/+1 = Bv, + Aw, and wt++1 = Wt 7- at v = - Aw, a linear first-order differential equation. For given initialization Wp, the resulting initial value problem can be solved exactly as in the case of de
Momentum flow is defined as (1 (1-3) B) d
gradient flow. Modified momentum flow defined as ) 3) dr= to + (1 3) d い -AØ a linear second-order differential equation. For given initialization wtp and the assumed initial condition di (0) = 0, then the resulting initial value problem can be solved exactly as a system of damped harmonic oscillators.
dt
E DERIVING THE EXACT LEARNING DYNAMICS OF SGD
We consider a continuous model of SGD without momentum incorporating weight decay (cquation I4 and modified loss (equation I6), such that
č= + V
! "
4
where Ls L + 210 is the regularized loss. The gradient of the modified loss is,
Vī= = =C=VA= + " HAA -1+ n7
2 g + x 1
7 (Hg + *HO),
2 2
0 2
where we used vL2- = g + 10 and H2 = H + AI. Thus, the equation of learning we consider is
n d0 = - - = [+ 2
(x+
1/22
2 0 " (Hg - XHO).
(27)
dt
2
To incorporate the effect of stochasticity (equation 3T) in this equation of learning we could replace the full-batch gradient g and Hessian H with their stochastic batch counterparts ÅB and fl respectively. However, careful treatment of these terms using stochastic calculus is needed when integrating the resulting stochastic differential equation, which we discuss at the end of this section.
Translation dynamics. In the case of parameters with translation symmetry, the effect of discretiza- tion essentially leaves the dynamics for the constant of learning unchanged. Combining the geometric properties of gradient (/g, 1 0) and Hessian (H1, = 0) introduced by translation symmetry with the equation of learning (equation 27 gives the differential equation, d0
d (0.1A) 0,
dt +
V
( dt
(28)
where we used the simplification,
(9.1 10
H1 0 H
(vī,1A)
1
7d
(
n1121
(0, 1A) + "
2/
(Hg A) +X (H(A
2
A 7
(8, (8,
/
18

---

Published as a conference paper at ICLR 2021
and ignored the ( term set the weight decay constant À to be small as the learning rate 7 in practice. The solution to this differential equation is
(0(1), 1A) e - (0(0),
Scale dynamics. In the case of parameters with scale symmetry, the effect of discretization does distort the original geometry. A finite learning rate leads to centrifugal force that monotonically increases the previously conserved quantity (10)²), while weight decay acts as a centripetal force decreasing the quantity. Combining the geometric constraints on the gradient ((g, 8 0) and
Hessian (НθA = -94) introduced by scale symmetry with the equation of learning (equation 27 gives the following differential equation.
< dt
d0
+ VL, 0A
A -0--
1 d
|0
2
" 1eal 0,
(29)
2 dt
2
where we used the simplifications
(de 1x) = 1
2 de.
(9.H8 9-1 -19011=0
(V),0A)
1 + 77
(0-0x)
(^+ 1122 2
(0, (0,0A) + 7
2 (11g.0x 0A)
+1 +1(10007)
n17
X +
10a12
2 19al2
7
and ignored the 21)2²) term. The solution to this differential equation is
(0((t)²= = (0) + mf"" -2(tt-r) leal dr.
Rescale dynamies. In the case of parameters with rescale symmetry, the effect of discretization also distorts the original geometry. However, unlike in the sphere, the force originating from discretization can both increase decrease the previously conserved quantity 2 (1) Combining the geometric properties of gradient ((g, OA,)) - (g. AA,)) 0) and Hessian (HOA, Нθ + 9A,,
gA, = 0) introduced by rescale symmetry with the equation of learning (equation 27 gives the following differential equation,
d8 dt
+ V.
eA,
X de
dt =
dd)
(194,12 119da1))) = 0, (30)
2 dt
where we used the simplification,
(0.0a)) (1. 19.022)=0
A 1112
<VoΣ, - (Vī, 0A,)
1
(g.0Ar) (g. (Aa))
2 (10A, 2 10 (2)
2
SA1 tBA2)= IaA, 1 9A2
(n. + (3.9Ag)-0)
n
(g. HOA, - -HOA.
+ (0, HB -HOA
2
n!? + 2
(10A,1 2 -1
" (19., 2- 199,,113)
2
and ignored the ( term. This is the same differential equation as in equation 29 just with different forcing term. Thus, the solution to this differential equation is
(0) 10,, (0) +1 L' e -22(e-+) (leA. 2- - dr.
10A, -1, = = -2-
19

---

Published as a conference paper at ICLR 2021
F MODELING STOCHASTICITY
Stochastic gradients (0) arise when we consider a batch B of size S drawn uniformly from the indices {I, .I,..., ] forming the unbiased gradient estimate (r5(0) - $ Ligs V((0,x;) xz). When the batch size is much smaller than the size of the dataset, S < ., then we can model the batch gradient as an average of S i.i.d. samples from a noisy version of the true gradient g Using the central limit theorem, we assume (0) g(0) is Gaussian random variable with mean μr O and covariance matrix E() = 3 ( Under this assumption, the stochastic gradient update can be written as g(++1) = g(m) - (0(()))) + ys G where & is a standard normal random variable. This update is an Fuler-Maruyama discretization with step size v of the stochastic differential equation
T1 COO)a..
d0 -00dd V
(31)
where W, is standard Wiener process. Equation (31) has been derived as model for SGD in many
previous works Mandt et (2015). In order to simplify the analysis, many of these works have then
made additional assumptions on the covariance matrix Σ(0) G(O)G(OO),, such as ( (0)
where (0) is the Hessian matrix Jastrzebski et al. (017), Σ()) C where is some constant matrix Mandt et al. 5).15), and E(O)= where / is the identity matrix Chaudhari & Soatto (201 8). However, without any additional assumptions, the differential symmetrics intrinsic to neural network architectures add fundamental constraints on E.
As we showed in section 3 the gradient of the loss, regardless of the batch, is orthogonal to the generator vector field 0a 2 a== associated with a symmetry. This implies the stochastic noise must also observe the same property, (- vs / G()),, 0a = 0. In order for this relationship to hold for arbitrary no then G(B)TO, v a= = O. In other words, the differential symmetry inherent in neural network architectures projects the noise introduced by stochastic gradients onto low rank subspaces.
Scale Symmetry with Stochasticity. In the previous section, we performed our analysis using continuous-timg time model with deterministic gradients to facilitate calculations, and replaced them with stochastic batch gradients upon discretization when evaluated the results empirically. Instead, we can also directly model the stochastic noise arising from batch gradient with an Itô process to perform analogous analysis in continuous- time as pointed out by Li et al. (202T) in recent follow-up work.
First, we assume that the time evolution of network parameters O(t) during SGD training is described by the Itô process d0- = pdt + (O))dWW, where
g X0 (g + AHO) + 0(1)) + ((A)) + 0(2²).
Assuming a set of parameters A respect scale symmetry, then applying Itô's lemma to the function x²)()) yields the Itô process,
d(0,(1))= = (20 + " (TG di 2 / S
- -2 nlaul + " S G(eu)
=
Notice this is a deterministic ODE equivalent to the previously derived ODE with an additional forcing term accounting for the variance of the noise. We can perform analogous analysis for the case of translation and rescale symmetry.
We can also consider the effect of stochasticity without the complexity of stochastic calculus by considering the dynamics in the discrete setting. As explained in appendix [ this is possible for the case without momentum, but becomes much more complicated once we consider momentum well.
'We do not need to assume the noise is Gaussian in order for this property be true. However, we adopt this commonly accepted assumption contextualize our work within the literature modeling SGD with SDE.
20

---

Published as a conference paper at ICLR 2021
G DERIVING THE EXACT LEARNING DYNAMICS OF SGD WITH MOMENTUM
We consider a continuous model of SGD with momentum incorporating weight decay (equation 14 momentum (equation [5}. stochasticity (equation (1), and modified flow (equation t7p. As discussed in appendix|C we can model the effect of momentum by considering the forward Euler discretization P2+1-06 and the backward Fuler discretization -3 ---1-_11 These terms introduce the numerical m[I-a)
"I-aa"
artifacts 3(1=a) de 0 and - B df 0 respectively, as explained by the modified flow analysis in diz
zgz appendix D Incorporating these elements with weight decay gives the equation of learning,
7 a)) (1+ 3)
d²
d
- (1 B)
-) dt
-0 g(())
10 =
2
dt
Incorporating the effect of stochasticity into this equation of learning gives the Langevin equation,
n a)
2 (1+ + 3)dv =-( B)ddt A0dt g(0)dt + v In s G(O)av,
(32)
d0 = vdl,
where W. is a standard Weiner process. Compared to the modified loss route (described in the previous section), it is much more natural and simple account for the effect of stochasticity with modified flow.
Translation dynamics. Combining the geometric properties of gradient ((g, 1A) 0). Hessian (H1A- 0), and stochasticity (G(O)T14 = 0) introduced by translation symmetry with the Langevin equation of learning (equation 32) gives the differential equation.
17(1 a)
d
3) d= + +2) (0, 10, O.
d
2 (1 31+
+1 (1-
(33)
dt
This is the differential equation for a harmonic oscillator with y = (11 and w
V 7(1 (1--)(1+0)* Assuming the initial condition (0()), 1x)) t= = O. then the general solution (as derived in appendix H is
2A
y cosh (V12 (V1=-w9t) + 7 V Vクz-
sinh
(0(0), 1A)
<> w
(A()),1))
(1> + (0)()
? = w
e cos (Vw² >21) + -
sin (vw2-221) (0(0),
7 < w
(34)
Scale dynamics. Combining the geometric constraints on the gradient (g. = 0). Hessian A -9)), and stochasticity (O = 0) introduced by scale symmetry with the Langevin equation of learning (equation 32p gives the following differential equation
(1( (1) B) d
31 d
10 1/1 - -1) + 3)
d0
di2 +
2 dt +
+
(35)
4
2
dt
This the differential equation for a driven harmonic oscillator with 7 = a+-3) w
V 70 71- and f(t) 2 de di
4A
2 Assuming the initial condition d 10 1,- 0. then the
general solution (derived in appendix H is
sinh (72-=2((-T)) 2 d8 (T)
(0a(t)1R + So e ((t-))
dr <>
Va
(a)))
10a()) + S00 <-((T)))
- 7)2 d8 A (7)
di
2 dr
7 ==
(36)
(0(OR + So¹ (T(-r)
sin
V /- (c-r)
2 d8 (T)((T))
dr
7 <<w
Vi /2
Thhe derivation of this ODE uses A 0a) = -1 10x12 10xl
48
21

---

Published as a conference paper at ICLR 2021
where (8x(t) is the solution to the homogeneous harmonic oscillator
e cosh -V12-w91) + Vepーaz sinh ( |0x(0)2
2>w w
(0)())
e e1 + (0)122(0)2
7= w
en cos (vw2- Vwa_gz sin (w² (pt) 10x(0))
7 <<w
(37)
Rescale dynamics. Combining the geometric properties of gradient (g. A. -AA.) 0. Hessian
(O 0A₂)) DA, 9Ag 0), and stochasticity (G(0 A G(O A0 0) introduced by rescale symmetry with the Langevin equation of learning (equation 32 gives the differential equation,
(1l-aa(1 +
3) df
8) (1-
+ -
71 a) + B)
(1
dd.A: dt dt
d0
+
4
dt2
2 dt
2
(38)
This is the same harmonic oscillator given by equation 35 with the different forcing term f(t)
d0
A2 (²) The general solution given by equation (36) and (37) replacing |0| and 12
2
dt
dr
dt by A0 2 - A0 and
d0 d0
respectively.
dE
dt
H GENERAL SOLUTIONS FOR ODES
H.1 EXPONENTIAL GROWTH
Here we will solve for the general solution of the homogenous first-order linear differential equation,
= O. d + x(t)
dt
Assume solution of the form (t) eat Plugging this in gives the auxiliary equation +
Thus, the general solution to the differential equation with initial condition z0) is
I(()= (0).
Now will solve the inhomogenous differential equation,
d + + x x(t)= f(t).
dt
Multiply both sides by et and factor the left hand side using the product rule such that the differential equation simplies to d (e)tr()) ext f(t). Integrate this equation and using the fundamental 26 theorem of calculus rearrange to get the solution
x0(1 = M *00) + S e -(i-TT ((r)dr.
H.2 HARMONIC OSCILLATOOR
Here will solve the general solution for harmonic oscillator,
d2
+
2
+ 2 x(t) =.
dt
Assume a solution of the form æz()= = ad Plugging this in gives the auxiliary equation a² + 2-aa
2 = 0 with solutions a= = + / 2-2 w² Thus, the general solution to the oscillator equation
with initial conditions z00) and dz (0) is
x(1)= = 77
(CIeV - + Oze V----
22

---

Published as a conference paper at ICLR 2021
where C1, C2 constants
C = 7 V2 w² 2/72 w
O= - V22 w x(0).
z00),
2 2/72 w
Using hyperbolic functions the solution simplifies as
s(t) e (cosh (V>2 w2t)
7 sinh (V?2 (V?2-w²)) x(0).
V12 w
The form of this general solution implicitly assumes y > w. the overdamped setting. When y w. the critically damped setting, then the solution reduces to
xx()= = e E + C2l),
where C1 z00) and C2 (0( When y < w, the underdamped setting, then the solution reduces to
x(t) ~
72 (Ci cos (VWR-22t) + C2 sim (Vw2-
where C = z0) and C2 V (0))
H.3
DRIVEN HARMONIC OSCILLATOR
Here we will solve the general solution for a driven harmonic oscillator,
d2
2 + w²) x(t) = f(t).
dt
dt
First notice that if (t) solution to the driven harmonic oscillator, then
(t) is a solution to the homogenous harmonic oscillator and xx((t) a specific
xxt)= (t) + xa((),
is the general solution to the driven harmonic oscillator. We will use the Fourier transform to solve for st((t).
Let (a(T) = (T) and f(7) = (Vā))() be the Fourier transforms of ra(()) and f(t) respec-
tively. Applying the Fourier transform to the driven harmonic oscillator equation and rearranging gives
= + 2yér + -²) f(T).
which implies by the inverse Fourier transform that (t)( is the convolution,
(1(()) (G f) (t),
where G(t) is Green's function (the driven solution xt(() for the dirac delta forcing function dg),
e e-
G(t) O(t)
2² - w
which again using hyperbolic functions simplifies
sinh
(v>² w2t
G(t) = e()e
V22 w
This form of Green's function is again implicitly assuming > w. When 7 the function simplifies to
G(t) (tt)e
and when 7 < w, the function simplifies
sin (V22-1211 - 72t
G(t) = O(t)e W 72
Noticing that both G and f are only supported on [0, oo), their convolution can be simplified and the general solution for the driven harmonic oscillator is
(tt) = =n(() + /'' G(t (frrdd
23

---

Published as a conference paper at ICLR 2021
1 DERIVING DYNAMICS IN THE DISCRETE SETTING
In section we identified certain parameter combinations associated with network symmetries that conserved under gradient flow. However, as we explained in section | these conservation laws are not observed empirically. To remedy this discrepancy we constructed more realistic continuous models for SGD, incorporating weight decay, momentum, stochasticity, and finite learning rates. In section 6 we derived the exact dynamics for the parameter combinations under this more realistic setting, demonstrating near perfect alignment with the empirical dynamics. What would happen if we instead derived the dynamics for the purameler combinations directly in the discrete setting of SGD? Here, we will identify these discrete dynamics and discuss the relationship between the discrete equations and the continuous solutions.
Gradient descent with learning rate y and weight decay constant À is given by the update equation g(++1) (1 7))0(²) ng ()(())) ) and initial condition g(0), Using this update equation the sum of the parameters after + steps can be "unrolled" as,
(0(++1) 11) (1 m) ()))=+((00 (8(0) a1) 1
Σ (1- 1=0
1A)
Similarly, the squared Euclidean norm of the parameters after steps can be "unrolled" as,
(0(+1)) p== (1-nA) 1 ++)2² (1-mA))2mm))) (0)))))))
(1-mn))2m-1))
(g(()))))))))))))))))))))) 000)
2=
i=0
Combining these unrolled equations, with the gradient properties of symmetry discussed in section 3 gives the discrete dynamics for the parameter combinations.
Translation:
(pin) 11 (1 17))") (0(0) 1A)
(39)
Scale:
0mm) A = = -11-1)) -)))2mm
0
n-1
Σ
(1 - (1-
2n (2)
(40)
1=0
0(m))2
10(m))
Rescale:
(41)
A1
(1 - (1-1))22 2m
[0(0))
2 ( Ag
+ 1)
n-
Σ
(1 m//) ()))2(2-1-0)
(6) 2 9
(6) 9.22
20 presented in section and the discrete equations (39g, 400g (41) The exponential function with decay rate À from the continuous solutions are replaced by power of the base (1 - n7) in the discrete setting. The integral of exponentially weighted gradient norms from the continuous solutions are replaced by Riemann sum power weighted gradient norms in the discrete setting. This further confirmation that the continuous solutions derived, and the modified gradient flow equation of learning used, well approximate the actual empirics. While the equations derived in the discrete setting remove any uncertainty about the exactness of the theoretical predictions, they provide limited qualitative understanding for the empirical learning dynamics. This is especially true if we consider the learning dynamics with momentum. In this setting, the process of "unrolling" is much more complicated and the harmonic nature of the empirics, easily derived in the continuous setting, is hidden in the discrete algebra.
Notice the striking similarity between the continuous equations 18).
24

---

Published as a conference paper at ICLR 2021
J EXPERIMENTAL DETAILS
An open source version of our code, used to generate all the figures in this paper, is available at waabbb/cc/maaailknnnnnnenerre -mechaniss
Dataset. While we ran some initial experiments on Cifar- 100, the dataset used in all the empirical figures in this documents
Tiny Imagenet. It is used for image categorization consists of 100,000 training images at a resolution of 64 x 64 spanning 200 classes.
Model. We use standard VGG-16 models for all experiments with the following modifications:
The last three fully connected layers at the end have been adjusted for an input at the Tiny ImageNet resolution (64 64) and thus consist of 2048, 2048, and 200 layers respectively.
In addition to the standard arrangement of conv layers for the VGG-16, we consider variant where we add a batch normalization layer between every convolutional layer and its activation function.
Training hyperparameters Certain hyperparameters were varied during training. Below we outline the combinations explored. All models were initialized using Kaiming Normal, and no learning rate drops warmup used.
Model
Datnscd
Epochs
Batch size
Opt.
LR
Mom.
WD
Damp-
VGGG16
Tiny ImageNet
100
256
SGD
[0.1, 0.01]
[0, 0.001, 0.0005, 0.6001]
VGG wiBN
ImageNet
100
256
SGD
j0.1,
0.001, 0.0005, 0,0001]
Tiny ImageNet
0.999.0.99
0.0005, 0,0001
VGG-16 w/BN
Tiny ImageNet
100
128
SGDM
0.1
[0, 0.9. 6.99)
[0, 0.001, 0.0005, 0.0001]
Table 4: Training hyperparameters.
Counting number of symmetries. Here we explain how to count the number of symmetries VGG-16 model contains.
Scale symmetries appear once per channel every layer preceding batch normalization layer. For our VGG-16 model with batch norm: 2-64 1.64+-21222 1288 +-2-----220 2564----12+- 3--22+++- 4, 227.
Rescale symmetries appear layers as well as once per input neuron to a fully connected layer. Note that the sizes of the fully connected layers depend input image size and the number of classes. For our VGG- 16 model with and without batchnorm on Tiny ImageNet, this is: (2 64 + 128 3 256 3- 512 3 512 + 33 + (2048 1024 + 1024) 8, 323.
per channel where there are afine transforms between
Translation symmetries appear once input value to the softmax function, which for the case of classification is always equal to the number of classes, plus the bias term. For our 201.
case this is 200
# Params
# Scale
Rescale
# Translation
VGG-16 on Tiny ImageNet
18,067,464
0 4.227
8,323
201
VGG-16 w/BN on Tiny ImageNet
18,075,912
8,323
201
Table Counting the number of symmetries.
Computing the theoretical predictions. Some of the expressions shown in equations 188) 19 and (20 in section are not trivial to compute as they involve an integral of an exponentially weighted gradient term. To tackle this problem, we wrote custom optimizers in PyTorch with additional buffers to approximate the integral via a Riemann sum. At every update step, the argument the integral term was computed from the batch gradients, scaled appropriately, and accumulated in the buffer. Note that the above sum needs to be scaled by the learning rate, which is the coarseness of the grid of this Riemann sum. Checkpoints of the model and optimizer states stored at
25

---

Published as a conference paper at ICLR 2021
pre-defined frequencies during training. Our visualizations involve computing the right hand side of equations (18. (1). and (20) from the model states and the left hand side the same equations from the integral buffers stored in the optimizer states explained above. These two quantities are referred to as "empirical" and "theoretical" in the figures and are depicted with solid color lines and dotted lines, respectively.
J.1 ADDITIONAL EMPIRICS
In this work we made exact predictions for the dynamics of combinations of parameters during training with SGD. Importantly, these predictions were made the neuron level, but could be aggregated for each layer. Here plot our predictions both layer and neuron levels for VGG-16 models trained on Tiny ImageNet.
A
10-4
x= 10
10
0.30
0.15
0015
220
+30
Time (1) steps)
Figure 8: The planar dynamics VGG-16 on Tiny ImageNet. plot the column sum of the final linear layer of VGG-16 model (without batch normalization) trained on Tiny ImageNet with SGD with learning rate 7 0.1, weight decay À {0, 10 10-4, 5,5 x 10-4, 10 10-3), and batch size S = 256, Colored lines are empirical column sums of the last layer through training and black dashed lines are the theoretical predictions of equation 18
A == 0
A 10
= 10
== 10
theory
10
900
300
2
2
1
K324
+307
Time (1) steps)
Figure 9: The spherical dynamics of VGG-16 BN on Tiny ImageNet. We plot the squared Fuclidean norms for convolutional layers VGG-16 model (with batch normalization) trained on Tiny ImageNet with SGD with learning rate 7 0.1, weight decay À {0, 10-4, 5 10-4, 1103), and batch size 5 256. Colored lines represent empirical layer-wise squared norms through training and the white dashed lines the theoretical predictions given by equation (19..
26

---

Published as a conference paper at ICLR 2021
0
X 10-4
= 10
A 10
thoor
10
450
300
150
W
119 X30
+307
K32
Time (77 x steps)
Figure 10:
The hyperbolic dynamics of VGG-16 on Tiny ImageNet. We plot the difference between the squared Euclidean norms for consecutive convolutional layers of a VGG-16 model (without batch normalization) trained on Tiny ImageNet with SGD with learning rate 1j = 0.1, weight decay À E {0, 10--00+$.0 4,5 x 10 10 (0- and batch size S 256. Colored lines represent empirical differences consecutive layer-wist wise squared norms through training and the white dashed lines the theoretical predictions given by equation 20
==0 10
=
10-3
10
0.30
0.15
0.00
-0.30
0.30
(Wa, 1)
0.15
theory
0.00
0.15
0..0
0.30
0.15
0.00
0.15
0.30
12
16
4
8
12
16
4
8
12
16
8
12
16
Time (1) steps)
The planar dynamics of Momentum on VGG-16 on Tiny ImageNet. We plot column sum of the final linear layer of VGG-16 model (without batch normalization) trained on Tiny ImageNet with Momentum with learning rate 7 = 0.1, weight decay (0. 10-4, 5 x 10-1, 10-33, momentum coefficient 3 {0, 0.9, 0.99). and batch size S 128. Colored lines are empirical column sums of the last layer through training and black dashed lines are the theoretical predictions of equation 334..
Figure 11:
27

---

Published as a conference paper at ICLR 2021
=
x= 10
=
A 10-3
10
1000
500
250
theory
|| 1|?
1000
750
250
0.50 99
1000
750
500
0
12
12
12
8 12
Time (1) x steps)
The spherical dynamies of Momentum VGG-16 Tiny ImageNet. We plot the squared Euclidean norms for convolutional layers of a VGG-16 model (with batch normal- ization) trained on Tiny ImageNet with Momentum with learning rate 7j 0.1, weight decay E {0. 10-1, 5 x 10-1, 10-3), }, momentum coefficient / E {0, 0.9, 0.99). and batch size S = 128. Colored lines represent empirical layer-wise squared norms through training and the white dashed lines the theoretical predictions given by equation (36) and 37
Figure 12:
À
A= 10-4
A= 5 10-4
A = 10 -3
? == 10-1
|| 1|
450
300
150
0.90
450
312
300
150
8
12
16
0
8
12
16
0
8
12
16
0
8
12
16
Time (n x steps)
Figure 13: The hyperbolic dynamics of Momentum on VGG-16 on Tiny ImageNet. We plot the difference between the squared Euclidean norms for consecutive convolutional layers of a VGG-16 model (without batch normalization) trained Tiny ImageNet with Momentum with learning rate 7) = 0.1, weight decay À E {0. 10-4, 5x 10 .10-3), momentum coefficient 3 E {0, 0.9, 0.99), and batch size S 128. Colored lines represent empirical differences in consecutive layer-wise squared norms through training and the black dashed lines the theoretical predictions given by equation (36)
and 37 replacing 1012 and d8 by |0 10 and d8
d8 respectively.
dt
28

---

Published as a conference paper at ICLR 2021
1.75-
2.75
0.9
1:50
0.6
2.25
1.00
2.00
theor
1.6
1.35
2.4
1.20
2.2
2,0
1.0
2.55
2.8
2.6
2.25
2.25
24
2.4
2.10
1.95
2.0
1.95
x10
k10
K20
Time (1) x steps)
Figure 14:
The per-neuron spherical dynamics of SGD on VGG-16 BN on Tiny ImageNet. We
plot the per-neuron squared Euclidean norms for convolutional layers of VGG-16 model (with batch normalization) trained on Tiny ImageNet with SGD with learning rate = 0.1, weight decay À = O. and batch size S 256. Colored lines represent empirical layer-wise squared norms through training and the black dashed lines the theoretical predictions given by equation (19)
2.8
120
0.9
2.4
1.05
0.90
0.0
1.2
-0.8
0.75
0.30
1.12
0.4
0.15
1.12
0.2
0.96
0.96
IW
0.15
0.30
0.8
0.15
0.4
0.8
0.4
0.00
-0.15
0.0
Time (77 X steps)
Figure 15: The per-neuron hyperbolic dynamics SGD on VGG-16 on Tiny ImageNet. We plot the per-neuron difference between the squared Fuclidean norms for consecutive convolutional layers VGG-16 model (without batch normalization) trained on Tiny ImageNet with SGD with learning rate 7 0.1, weight decay À = 0, and batch size S = 256. Colored lines represent empirical differences in consecutive layer-wise squared norms through training and the black dashed lines the theoretical predictions given by equation 20
29

---

Published as a conference paper at ICLR 2021
B
0.9
B 0.99
6.56
1.2
6.59
0.6
0.4
5.25
theory
0.0
0.00
2.0
3.00
200
2.0
1.60
2.-
1.5
0.75
1.5
1.8
0.75
1.2
1.0
0.50
1/0
0.5
0.25
5.5
0.0
0:20
0.0
20
1.00
0.50
3.0
0.25
63
0.0 0.00
12
12
Time (1 X steps)
Figure 16: The per-neuron spherical dynamics of Momentum on VGG-16 on Tiny ImageNet. We plot the per-neuron squared Euclidean norms for convolutional layers (with batch normalization) trained on Tiny ImageNet with Momentum with learning rate 0.1, weight decay À O, momentum coefficient 3 (0.9, 0.99), and batch size 128. Colored lines represent empirical layer-wise squared norms through training and the black dashed lines the theoretical predictions given by equation (36) and (37).
VGG-16 model
0.9
0.99
20
1.00
2.0
15
15
10
0.50
0.5
-02
0.23
-02
0.3
theory
0.0
W
3.50
3.00
cmn 6-3
0.08
0.00
0.00
0.08
-0.15
0.00
0.05
-0
0.00
0.25
0.02
1.00
0.15
1.00
0.15
50 154
575
0.75
0.00
050
0.00
0.00
-0.00
52b
0.00
-3.08
4
13
16
12
16
12
6 12
16
12
Time (n steps)
Figure 17: The per-neuron hyperbolic dynamics of Momentum on VGG-16 on Tiny ImageNet. We plot the per-neuron difference between the squared Euclidean norms for consecutive convolutional layers of a VGG-16 model (without batch normalization) trained on Tiny ImageNet with Momentum with learning rate y = 0.1, weight decay À = O, momentum coefficient, 3 E (0.9, 0.99), and batch size S 128. Colored lines represent empirical differences in consecutive layer- wise squared norms through training and the black dashed lines the thcoretical predictions given by equation (36) and d0 d0 <40A2 respectively.
37 replacing 2 and 2r
dB by 10,,, 1 10A22l and
dt
30


Reviews:
[{"review_id": "wuK9tp_Qy_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the dynamics of the parameters while training a neural network via SGD. SGD is not studied directly but three continuous time approximations of SGD are considered: the classical gradient flow, a stochastic differential equation of Langevin type, and a \"modified\" gradient flow whose derivation has roots in the literature. For each dynamics, the authors show that some invariant properties of the loss function (which are often satisfied in practice) imply some invariant quantities for the dynamics.  \n\nOverall, the paper is rather clear. It tries to provide a physical meaning behind the dynamics of learning, which is an interesting question.\n\nHowever, I do not see any significant theoretical contribution. For instance, all derivations are simple differential calculus applications. Moreover, several models for SGD are used, each of them have their own invariant properties (which look alike) and then what? The technical contribution is not clear to me. Finally, the experimental contribution is rather mild because the numerical experiments are not discussed in the main text (except marginally in the conclusion). There is a lot of room for improvement (see below) and I don't recommend the paper for publication in this form.\n\n\nSpecific remarks:\n- The first paragraph of Section 3 is not necessary in my opinion (already explained in the intro). Moreover, the notation for the group action is a bit misleading, e.g. there is confusion between \\psi, \\psi(\\theta) and \\psi(\\theta,\\alpha)\n- Eq 2: the translation invariant suddenly applies only to a subset of the parameters.\n- Figure 1, 2 are not commented in the main text. Same for 3,4,5 (except in the conclusion)\n- The authors could recall Noether's theorem for comparison\n- The models used for SGD are not theoretically justified, except for the classical gradient flow, for which it is standard (Kushner & Yin 2003). For Eq (11) it starts to be sloppy (CLT + Forward Euler). For Eq (13), only intuition is provided.\n- Last paragraph of Page 5. It seems that the discussion applies for any \\xi (not necessarily Gaussian). Does it help to remove the Gaussian noise assumption?\n\n\n\n\n\nTo improve the paper, I suggest the authors to \n- clearly locate their work within the existing literature. This would help to understand why the questions answered in this paper are important, and to highlight their contribution.\n- Turn their paper into an experimental one. To this end, dedicate a whole section to **commented** numerical experiments. This does not mean adding more simulations, just explain them in the main text and how they contribute to the main message of the paper.\n\n\nMinor:\n\nPage 2: \"nornalization\"\nBetween Eq 3 and 4: Notation not defined\nPage 4: ReLU not defined (give the formula)\nSection 5.1, 2nd line. \\nabla is missing \nPage 6: graident\nPage 7 (two times): previosly \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mB9TwkPWk2J", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the learning dynamics of DNNs from the perspective of symmetry of some parameters. It is interesting to borrow ideas from physics, which I believe is a right way to go.\n\nSpecifically, the paper derives analytical form of parameters under the cases of translation, scale and inversion invariances, and also modified the underlying gradient flows to accommodate stochastic gradients. The results are very interesting, which I like a lot. However, I have some confusions about some results, which make me not able to give the paper a pass at this time. I will consider revising the score if the authors can clear me in the rebuttal.\n\n1. The papers talks about three invariances, and also given some examples of DNNs that satisfies these invariances. From my understanding, these invariances only apply to a very limited cases of DNN structures, e.g., the softmax layer. And in Figure 3-5, the authors verify the convolutional layers for these invariances. It is not clear to me why convolutional layers satisfy these invariances:  translation, scale and inversion. Am I missing something?\n\n2. In 6.1, a new solution for the translation invariance case is derived. However, it seems that the parameters following the solution still converge to zero? This is obviously not the case in practice.\n\n3. Third line below eq.16, it says \"there is a competition between the centripetal effect of weight decay and the centrifugal effect of discretization.\" I don;'t understand why the descretization has the centrifugal effect.\n\n\n==========\nAfter rebuttal: The rebuttal resolves most of my concerns. It is more clear to me that this is an interesting paper on describing the dynamics of DNN parameters in the training, which seems novel to me. I also realize that the closed form dynamics are not for individual parameters, but in terms of some statistics of the parameters, e.g., the sum of the parameters. This makes the results not as existing as what I thought. That is why I decide to raise my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a27_KGlIlo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: \n- This paper is very well-written and motivated. \n- The train of thoughts is explained very clearly, such that I (admittedly not being  an expert in this field) was able to follow. \n- The idea to unify invariances of the loss function by using symmetries and derive corresponding conservation laws (for $\\lambda =0$) in the gradient flow is very elegant. \n- By adapting a modification of the gradient flow from previous works that accounts for the discrete approximation of SGD, the derived theory was able to predict the behavior of the relevant quantities during training to a remarkable accuracy.\n\nCons: I did not find any major drawbacks of this work. Just two small questions:\n- Using $\\ell^2$ regularization on a problem with scale symmetry does not seem to make sense, because the cost function\n$$ \\mathcal{L}(\\theta) + \\lambda \\|\\theta\\|^2 $$ \nwill likely not have a minimizer as soon as $\\lambda >0$. Reducing the magnitude of any $\\theta$ that is optimal for $\\mathcal{L}$ reduces the regularization, but in the limit of $\\theta=0$ the loss might jump up. Thus, the costs are not lower semi-continuous.\n- Additionally, the scale symmetry seems to naturally lead to a discontinuous loss function $\\mathcal{L}$. Is there no problem in even defining the gradient flow for such a function? Which properties of $\\mathcal{L}$ do you need to derive the continuous gradient flow equations?\n\n\nOverall, I really enjoyed reading this paper. Since I am not an expert in the field, I cannot really judge the novelty/contribution, but aside from this aspect, I clearly recommend the acceptance of this work. \n\n-----------\n- There is a typo in Section 6.1 \"graident\"\n- I stumbled upon the NeurIPS 2020 paper \"Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate\" by Li, Lyu, and Arora. Based on the abstract, this seems to be a relevant related work. \n\n\n----------------\nAfter the rebuttal: I'd like to thank the authors as well as my fellow reviewers for the interesting discussions and corresponding clarifications. Summarizing my impressions from the discussion, the two main points of criticism are that the proposed analysis is not fully predictive (depends on the norm of the gradients that depend on the empirical data), but rather provides the laws that  govern the dynamics, and that the analysis is based on a time-continuous differential equation that seems to approximate SGD well instead of being applicable to the SGD iterates directly. The validity of the continuous dynamics is demonstrated in numerical results only.  I do agree that a fully predictive framework on SGD directly would be very intersting. Yet, I think the authors are taking important steps towards such a framework, and considering the fact that SGD often behaves surprisingly/unexpectedly (as also stated by R3), I am still quite impressed how accurately the theory matches the actual SGD behavior. For our understanding of how symmetries/invariances in the weights of network architectures influence the training, I believe this paper does provide interesting insights such that I recommend its acceptance. As for a final score, I could go down to a 7 to account for the concerns raised by my fellow reviews, but I think it would mainly reflect my uncertainty about my intuition that a fully predictive analysis on SGD directly might be infeasible, and this aspect should be reflected by the confidence rather than the rating. Thus, I'll give the authors the benefit of the doubt and keep my score, since I really enjoyed reading this paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LDk_3oOVHSp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The current work studies the implications of continuous symmetries of a DNN on its weight dynamics. Specifically, they consider linearly realized symmetries (such as a shift/translation to the logits), and use the fact that the loss/hessian/mini-batches, are insensitive to such shift to decouple their dynamics. They do so both in the continuous/vanishing-learning-rate case and for small learning rates and manage to provide accurate quantitative predictions for the dynamics of these quantities.\n\nOn the positive side, the current work makes accurate predictions on DNNs trained in a real-world setting (real datasets, convolutional layers, finite learning rates, mini-batches etc...) which is a very complicated problem. They do so with a refreshing toolbox, that of symmetries. \n\nOn the other hand, what makes their quantities tractable, seems to be the very fact that they have no impact on the DNNs final outputs. For example, this is why they are unaffected by the real-world dataset. While being a clever trick, it can be viewed as an inherent limitation of the approach: one understands quantities that have no bearing on what the DNN learns. I don't see, for instance, how one expects to perform architecture exploration (as mentioned in the discussion), using quantities that have no implications on the DNNs predictions. If the authors can argue that their approach has a broader horizon, it may increase its potential impact. \n\nTwo technical comments: \n\n1. I find the use of subset notation and definition of large theta confusing: It seems that small theta is the set of parameters but it is a subgroup of large theta. \n2. Inversion symmetry commonly refers to a discrete Z_2 symmetry, whereas the authors take it to be some representation of GL1.\n\n \n\n\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper offers a more systematic treatment of various symmetry-related results in the current literature. Concretely, the invariance properties exhibited by loss functions associated with neural networks give rise to various dynamical invariants of gradient flows. The authors address these dynamical invariants in a unified manner and study them wrt different variants of gradient flows aimed at reflecting different algorithmic aspects of real training processes. 

The simplicity and the generality of dynamical invariants are both the strength and the weakness of the approach. On one hand, they provide a simple way of obtaining non-trivial generalities for the dynamics of learning processes. On the other hand, they abstracts away the very structure of neural networks from which they derive, and hence only allow relatively generic statements. Perhaps the approach should be positioned more as a conceptual method for studying invariant loss functions. 

Overall, although the technical contributions in the paper are rather incremental, the conceptual contribution of using dynamical invariants to unify and somewhat simplify existing analyses in a clear and clean symmetry-based approach is appreciated by the reviews and warrant a recommendation for borderline acceptance. 

Decision: Accept (Poster)