Neural Mechanics:
Symmetry and Broken Conservation Laws in Deep Learning Dynamics

Daniel Kunin*, Javier Sagastuy-Brena, Surya Ganguli, Daniel L.K. Yamins, Hidenori Tanaka*(* equal contribution)

Stanford University, NTT Physics & Informatics Lab

Deep learning has been successful, but it’s inner-workings are still mysterious
Myriads of design choices of the deep learning systemshapes the trajectory that the millions of parameters take during training?

Architecture
ReLU or tanh?
Batch Normalization?
SoftMax?
Convolution?
Residual connection?

θN

θ0

θ1

Parameter space

Optimizer
How much weight decay?
How much momentum?
Learning rate schedule?
Batch-size?
Adaptive gradient?

Researchers and practitioners largely depend on try & error based heuristic search.
Better understanding of the mechanism is foundational in principled exploration of the vast design space.

Q. What, if anything, can we quantitatively understand about the learning dynamics ofstate-of-the-art deep learning models driven by real-world datasets?

Q. What, if anything, can we quantitatively understand about the learning dynamics ofstate-of-the-art deep learning models driven by real-world datasets?
This question is difficult because of…
1. millions of parameters
2. compositional non-linear functions
3. discrete updates by random batches of data
Existing works have simplified the problem by making major assumptions on the architecture…
Single Hidden Layer
[2]

[1]

y = θ f(θ x)David Saad and Sara Solla. Dynamics of on-line gradient descentlearning for multilayer neural networks. 1995.

Infinitely WideLinear Networks[L][2] [1]y = θ …θ θ xAndrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to thenonlinear dynamics of learning in deep linear neural networks. 2013.

θ

[l]

∈R

N [l]×N [l]

,N

[l]

→∞

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel:
Convergence and generalization in neural networks. 2018.

Q. What, if anything, can we quantitatively understand about the learning dynamics ofstate-of-the-art deep learning models driven by real-world datasets?
In this work we don’t introduce simplifying assumptions
on the architecture or optimizer!
Rather we identify and solve the simpler dynamics of parameter combinations.

conv. 2
conv. 3

conv. 6
conv. 7conv. 8Parameters: θiconv. 4conv. 5conv. 9conv. 10
conv. 11
conv. 12

combination dynamics

Training time: ttheoryParameter combinations: | | Wl(t) | |2

conv. 1parameter dynamics

Parameter combinations: | | Wl(t) | |2

VGG16

Training time: tTraining time: t

Q. What, if anything, can we quantitatively understand about the learning dynamics ofstate-of-the-art deep learning models driven by real-world datasets?
In this work we don’t introduce simplifying assumptions
on the architecture or optimizer!

Parameter combinations: | | Wl(t) | |2

Rather we identify and solve the simpler dynamics of parameter combinations.

Our theory matches
experiment exactly!

Training time: t

0

Q. Can we solve for complex learning dynamics of real deep learning models?
Part 1. Symmetry in the Loss Constrain Gradient and Hessian Geometries

Part 2. Symmetry Leads to Conservation Laws Under Gradient Flow

Part 3. A Realistic Continuous Model for Stochastic Gradient Descent
momentummodified flow

momentum flow<latexit sha1_base64="+evKJKuSgigGlDMLVGTUw4Rm5hU=">AAAB8nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdSDUPTisYK1hTSUzXbTLt1swu5EKKE/w4sHRbz6a7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaYZb7FEJroTUsOlULyFAiXvpJrTOJS8HY5up/X2E9dGJOoBxykPYjpQIhKMorX8bsiRkmvi1q961Zpbd2ciy+AVUINCzV71q9tPWBZzhUxSY3zPTTHIqUbBJJ9UupnhKWUjOuC+RUVjboJ8tvKEnFinT6JE26eQzNzfEzmNjRnHoe2MKQ7NYm1q/lfzM4wug1yoNEOu2PyjKJMEEzK9n/SF5gzl2AJlWthdCRtSTRnalCo2BG/x5GV4PKt7lu/Pa42bIo4yHMExnIIHF9CAO2hCCxgk8Ayv8Oag8+K8Ox/z1pJTzBzCHzmfP1JNj/Q=</latexit>= 0.9<latexit sha1_base64="8kS57V6elUqfsYO8R0ZLxNu1Nzg=">AAAB83icbZBNS8NAEIYn9avWr6pHL4tF8FQSEbQHoejFYwX7AU0om+2mXbrZhN2JUEr/hhcPinj1z3jz37htc9DWFxYe3plhZt8wlcKg6347hbX1jc2t4nZpZ3dv/6B8eNQySaYZb7JEJroTUsOlULyJAiXvpJrTOJS8HY7uZvX2E9dGJOoRxykPYjpQIhKMorV8P+RIyQ1xq7Var1xxq+5cZBW8HCqQq9Erf/n9hGUxV8gkNabruSkGE6pRMMmnJT8zPKVsRAe8a1HRmJtgMr95Ss6s0ydRou1TSObu74kJjY0Zx6HtjCkOzXJtZv5X62YYXQcTodIMuWKLRVEmCSZkFgDpC80ZyrEFyrSwtxI2pJoytDGVbAje8pdXoXVR9Sw/XFbqt3kcRTiBUzgHD66gDvfQgCYwSOEZXuHNyZwX5935WLQWnHzmGP7I+fwB0LWQNw==</latexit>= 0.99Part 4. Combining Symmetry and Modified Flow to Derive Learning Dynamics
eq. (39)
classifierSymmetry Constrain Gradient and Hessian Geometries
Symmetry: A function f(θ) posses a symmetry if it invariant under the action θ ↦ ψ(θ, α)of a group G on the parameter vector θ, i.e. if f(ψ(θ, α)) = f(θ) for any (θ, α).

Geometric constraints: If a function f(θ) posses a differentiable symmetry, then
Gradient
Hessian∂α f(ψ) = ⟨ ∇f, ∂αψ⟩ = 0

Example:∂θ∂α f(ψ) = Hf∂θψ∂αψ + ∂θ∂αψ ∇f = 0

f(x, y) = x 2 + y 2

Step 1. Identify symmetry:x
cos α −sin α
Rotation: ϕ(x, y, α) = (sin α cos α ) (y)Step 2. Evaluate gradient at identity:x⟨ ∇f, ∂αϕ⟩ = 0∇f = 2 [y]

y

y
x

−y
∂αϕ |α=0 = [ ]
x

y
x

x

Symmetry resides in all over the modern deep network architectures
Symmetry
ExampleTranslation

Scaleθ ↦ θ + α1α∈ℝθ ↦ αθ+α∈ℝ(θ1, θ2) ↦ (αθ1, α −1θ2)
α ∈ ℝ+

softmax

batchnorm

ReLU

σ(θx)i =e θi x∑j e θj x

BN(θx) =θx − E[θx]Var(θx)Rescaleθ2ReLU(θ1x) = θ2max(0,θ1x)

Gradient⟨g,1⟩ = 0⟨g, θ⟩ = 0⟨g1, θ1⟩ = ⟨g2, θ2⟩

Hessian⟨H,1⟩ = 0

⟨H, θ⟩ = − g

H(θ1 − θ2) + g1 − g2 = 0

Visualization

θ1

θ1
θ2

θ1
θ2

θ2

0

Q. Can we solve for complex learning dynamics of real deep learning models?
Part 1. Symmetry in the Loss Constrain Gradient and Hessian Geometries

Part 2. Symmetry Leads to Conservation Laws Under Gradient Flow

Part 3. A Realistic Continuous Model for Stochastic Gradient Descent
momentummodified flow

momentum flow<latexit sha1_base64="+evKJKuSgigGlDMLVGTUw4Rm5hU=">AAAB8nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdSDUPTisYK1hTSUzXbTLt1swu5EKKE/w4sHRbz6a7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaYZb7FEJroTUsOlULyFAiXvpJrTOJS8HY5up/X2E9dGJOoBxykPYjpQIhKMorX8bsiRkmvi1q961Zpbd2ciy+AVUINCzV71q9tPWBZzhUxSY3zPTTHIqUbBJJ9UupnhKWUjOuC+RUVjboJ8tvKEnFinT6JE26eQzNzfEzmNjRnHoe2MKQ7NYm1q/lfzM4wug1yoNEOu2PyjKJMEEzK9n/SF5gzl2AJlWthdCRtSTRnalCo2BG/x5GV4PKt7lu/Pa42bIo4yHMExnIIHF9CAO2hCCxgk8Ayv8Oag8+K8Ox/z1pJTzBzCHzmfP1JNj/Q=</latexit>= 0.9<latexit sha1_base64="8kS57V6elUqfsYO8R0ZLxNu1Nzg=">AAAB83icbZBNS8NAEIYn9avWr6pHL4tF8FQSEbQHoejFYwX7AU0om+2mXbrZhN2JUEr/hhcPinj1z3jz37htc9DWFxYe3plhZt8wlcKg6347hbX1jc2t4nZpZ3dv/6B8eNQySaYZb7JEJroTUsOlULyJAiXvpJrTOJS8HY7uZvX2E9dGJOoRxykPYjpQIhKMorV8P+RIyQ1xq7Var1xxq+5cZBW8HCqQq9Erf/n9hGUxV8gkNabruSkGE6pRMMmnJT8zPKVsRAe8a1HRmJtgMr95Ss6s0ydRou1TSObu74kJjY0Zx6HtjCkOzXJtZv5X62YYXQcTodIMuWKLRVEmCSZkFgDpC80ZyrEFyrSwtxI2pJoytDGVbAje8pdXoXVR9Sw/XFbqt3kcRTiBUzgHD66gDvfQgCYwSOEZXuHNyZwX5935WLQWnHzmGP7I+fwB0LWQNw==</latexit>= 0.99Part 4. Combining Symmetry and Modified Flow to Derive Learning Dynamics
eq. (39)
classifierSymmetry Leads to Conservation Laws Under Gradient Flow
Gradient flow: The gradient descent update θ (n+1) = θ (n) − ηg(θ (n)) with learning rate η is a forward
Euler discretization of the ODE known as gradient flow:dθ= − g(θ)
dt
How do these learning dynamics interact with the geometric properties introduced by symmetry?

Translation

Scale

Rescaleθ1θ1θ2⟨θA(t),1⟩ = ⟨θA(0),1⟩

θ1

2

θ2

| θA(t) | = | θA(0) |

2

θ2
2

2

2

| θA1(t) | − | θA2(t) | = | θA1(0) | − | θA2(0) |A version of Noether’s Theorem: Every symmetry* of a network architecture has a corresponding
conserved quantity through training under gradient flow. Projecting the gradient flow dynamics onto
the generator vector field generates an ODE, whose solution is a conservation law.

Emmy Noether (1882 - 1935)

d
θ,
∂
ψ
=
0
α
⟩
dt ⟨

2

*satisfying a mild assumption

Does this theory agree with empirics?
Theory (Gradient Flow)
Parameter combinations: | | Wl(t) | |2

Parameter combinations: | | Wl(t) | |2

Experiment

Training time: t

conv. 1
conv. 2
conv. 3
conv. 4conv. 5conv. 6
conv. 7conv. 8conv. 9conv. 10
conv. 11
conv. 12Training time: tParameter combinations: | | Wl(t) | |2

No, conservation laws are broken empirically!

Training time: t

Q. Why?
Gradient flow is too simple of a continuous model for SGD.
It fails to account for key building blocks of modern optimization:weight decay• momentum• stochasticity• finite learning rates•

0

Q. Can we solve for complex learning dynamics of real deep learning models?
Part 1. Symmetry in the Loss Constrain Gradient and Hessian Geometries

Part 2. Symmetry Leads to Conservation Laws Under Gradient Flow

Part 3. A Realistic Continuous Model for Stochastic Gradient Descent
momentummodified flow

momentum flow<latexit sha1_base64="+evKJKuSgigGlDMLVGTUw4Rm5hU=">AAAB8nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdSDUPTisYK1hTSUzXbTLt1swu5EKKE/w4sHRbz6a7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaYZb7FEJroTUsOlULyFAiXvpJrTOJS8HY5up/X2E9dGJOoBxykPYjpQIhKMorX8bsiRkmvi1q961Zpbd2ciy+AVUINCzV71q9tPWBZzhUxSY3zPTTHIqUbBJJ9UupnhKWUjOuC+RUVjboJ8tvKEnFinT6JE26eQzNzfEzmNjRnHoe2MKQ7NYm1q/lfzM4wug1yoNEOu2PyjKJMEEzK9n/SF5gzl2AJlWthdCRtSTRnalCo2BG/x5GV4PKt7lu/Pa42bIo4yHMExnIIHF9CAO2hCCxgk8Ayv8Oag8+K8Ox/z1pJTzBzCHzmfP1JNj/Q=</latexit>= 0.9<latexit sha1_base64="8kS57V6elUqfsYO8R0ZLxNu1Nzg=">AAAB83icbZBNS8NAEIYn9avWr6pHL4tF8FQSEbQHoejFYwX7AU0om+2mXbrZhN2JUEr/hhcPinj1z3jz37htc9DWFxYe3plhZt8wlcKg6347hbX1jc2t4nZpZ3dv/6B8eNQySaYZb7JEJroTUsOlULyJAiXvpJrTOJS8HY7uZvX2E9dGJOoRxykPYjpQIhKMorV8P+RIyQ1xq7Var1xxq+5cZBW8HCqQq9Erf/n9hGUxV8gkNabruSkGE6pRMMmnJT8zPKVsRAe8a1HRmJtgMr95Ss6s0ydRou1TSObu74kJjY0Zx6HtjCkOzXJtZv5X62YYXQcTodIMuWKLRVEmCSZkFgDpC80ZyrEFyrSwtxI2pJoytDGVbAje8pdXoXVR9Sw/XFbqt3kcRTiBUzgHD66gDvfQgCYwSOEZXuHNyZwX5935WLQWnHzmGP7I+fwB0LWQNw==</latexit>= 0.99Part 4. Combining Symmetry and Modified Flow to Derive Learning Dynamics
eq. (39)
classifier

Gradient flow is too simple, how can we construct a realistic continuous model for SGD?
Example: Quadratic Loss
Gradient Flow:⊺ℒ = θ Aθdθ
= − g(θ)
dtModeling weight decay (λ): Weight decay changes the trajectory from gradient flow
pulling the network to the origin in parameter space.

dθ= − g(θ) − λθdtModeling momentum (β): Momentum accelerates the learning dynamics rescaling
time, but leaves the trajectory intact.

dθ(1 − β)= − g(θ)
dt̂ (θ) as a noisy version of theModeling stochasticity: We model the batch gradient gℬfull batch gradient g(θ) such that,̂ (θ) = g(θ) + ϵ
gℬ̂ , ∂αψ⟩ = ⟨ϵ, ∂αψ⟩ = 0 for any batch ℬ.
where E[ϵ] = 0 and ⟨gℬBlue curve: gradient flow
Red curve: modified trajectory

A Realistic Continuous Model for Stochastic Gradient Descent
Modeling discretization: Gradient descent moves in the direction of steepest descent, but
due to a finite learning rate fails to remain on the continuous steepest descent path.
Q. Does there exist a “continuous equation of learning” that can accurately model theeffect of a finite learning rate?
A. Modified equation analysis is a method for modeling the discrepancy introduced bya discretization of a PDE with higher order “spatial” or “temporal” derivatives.
Modified Loss: Introduces higher order derivatives of the loss,
effectively modifying the loss landscape itself.

dθη= − g(θ) − H(θ)g(θ)
dt
2David G.T. Barrett and Benoit Dherin. Implicit Gradient Regularization. 2020.

Modified Flow: Introduces higher order temporal derivatives
modifying the flow directly.

dθη d 2θ
= − g(θ) −dt
2 dt 2
Nikola B. Kovachki, Andrew M. Stuart. Analysis Of Momentum Methods. 2019.

Blue curve: gradient flow
Red curve: modified trajectoryBlack dots: discrete SGD steps

0

Q. Can we solve for complex learning dynamics of real deep learning models?
Part 1. Symmetry in the Loss Constrain Gradient and Hessian Geometries

Part 2. Symmetry Leads to Conservation Laws Under Gradient Flow

Part 3. A Realistic Continuous Model for Stochastic Gradient Descent
momentummodified flow

momentum flow<latexit sha1_base64="+evKJKuSgigGlDMLVGTUw4Rm5hU=">AAAB8nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdSDUPTisYK1hTSUzXbTLt1swu5EKKE/w4sHRbz6a7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaYZb7FEJroTUsOlULyFAiXvpJrTOJS8HY5up/X2E9dGJOoBxykPYjpQIhKMorX8bsiRkmvi1q961Zpbd2ciy+AVUINCzV71q9tPWBZzhUxSY3zPTTHIqUbBJJ9UupnhKWUjOuC+RUVjboJ8tvKEnFinT6JE26eQzNzfEzmNjRnHoe2MKQ7NYm1q/lfzM4wug1yoNEOu2PyjKJMEEzK9n/SF5gzl2AJlWthdCRtSTRnalCo2BG/x5GV4PKt7lu/Pa42bIo4yHMExnIIHF9CAO2hCCxgk8Ayv8Oag8+K8Ox/z1pJTzBzCHzmfP1JNj/Q=</latexit>= 0.9<latexit sha1_base64="8kS57V6elUqfsYO8R0ZLxNu1Nzg=">AAAB83icbZBNS8NAEIYn9avWr6pHL4tF8FQSEbQHoejFYwX7AU0om+2mXbrZhN2JUEr/hhcPinj1z3jz37htc9DWFxYe3plhZt8wlcKg6347hbX1jc2t4nZpZ3dv/6B8eNQySaYZb7JEJroTUsOlULyJAiXvpJrTOJS8HY7uZvX2E9dGJOoRxykPYjpQIhKMorV8P+RIyQ1xq7Var1xxq+5cZBW8HCqQq9Erf/n9hGUxV8gkNabruSkGE6pRMMmnJT8zPKVsRAe8a1HRmJtgMr95Ss6s0ydRou1TSObu74kJjY0Zx6HtjCkOzXJtZv5X62YYXQcTodIMuWKLRVEmCSZkFgDpC80ZyrEFyrSwtxI2pJoytDGVbAje8pdXoXVR9Sw/XFbqt3kcRTiBUzgHD66gDvfQgCYwSOEZXuHNyZwX5935WLQWnHzmGP7I+fwB0LWQNw==</latexit>= 0.99Part 4. Combining Symmetry and Modified Flow to Derive Learning Dynamics
eq. (39)
classifierQ. How do weight decay, momentum, stochastic gradients, and finite learning
rates all interact to break these conservation laws?
2

Example: Consider the circular vector field on ℛ :f(x) =

1). Equation of learning.

0 −1x[1 0 ]

Discrete
xt+1 = xt + ηf(xt)xCircular flowdx
= f(x)dtf (x)⟨ f(x), x⟩ = 0

Modified circular flowdxη= f(x) + x
dt
2

x2

2). Project the learning dynamics onto the generator vector fields associated with a symmetry.
3). Harness the geometric constraints introduced by symmetry to derive simplified ODEs.
⟨ f(x), x⟩ = 0dx⟨ , x⟩ = ⟨ f(x), x⟩dt

x1⟨ f(x), x⟩ = 0dx
η
⟨ , x⟩ = ⟨ f(x), x⟩ + ⟨ x, x⟩
dt24). Solve these ODEs to obtain exact dynamics for the previously conserved quantities.
d 1
2|x| = 0
dt 2

d 1
η
2
2
|x| = |x|dt 2
2

| x |2 = | x(0) |2| x |2 = e ηt | x(0) |2

|x|

2

time

Q. How do weight decay, momentum, stochastic gradients, and finite learning
rates all interact to break these conservation laws?
To answer, we:
1. Consider a realistic continuous model for SGD, an equation of learning.

dθ = − F̃(θ)dt +ηG(θ)dWt,Sη
F̃(θ) = (1 + λ)g + λθ + (Hg + Hθ) .
2

2. Project the learning dynamics onto the generator vector fields associated with a symmetry.
3. Harness the geometric constraints introduced by symmetry to derive simplified ODEs.
⟨g, Hθ⟩ = − | g |2 −⟨g, θ⟩ = 0⟨g, θ⟩ = 0ηη22⟨F̃, θ⟩ = (1 + λ)⟨g, θ⟩ + λ⟨θ, θ⟩ + (⟨Hg, θ⟩ + ⟨Hθ, θ⟩) = λ | θ | − | g |2
2
4. Solve these ODEs to obtain exact dynamics for the previously conserved quantities.
2

d|θ|
η
2
2
= − λ|θ| + |g|dt
2
Translation

Scale

RescaleTheory (dotted lines) match the empirics (colored lines) perfectly!
VGG-16 trained on Tiny ImageNet with SGD

TheoryExperimentTranslation

Scale

RescaleTheory (dotted lines) match the empirics (colored lines) perfectly!
VGG-16 trained on Tiny ImageNet with SGD

Theory & ExperimentTranslation

Scale

RescaleTheory (dotted lines) match the empirics (colored lines) perfectly!
VGG-16 trained on Tiny ImageNet with SGD
Translation•⟨θ𝒜(t),1⟩ decays exponentially tozero at a rate proportional to theweight decay.
• Dynamics is independent oflearning rate and data due to thelack of curvature in the gradient
field

Scale

•

2

Norm | θ𝒜 | is the sum of an exponentially decaying memory of thenorm at initialization and an
exponentially weighted integral ofgradient norms accumulated
through training.

RescaleSimilar to the scale dynamics, therescale dynamics do depend on thedata through the gradient norms• No guarantee that the integral termis always positive.
•

Harmonic oscillation with momentum
Harmonic oscillator pulled by regularizer, pushed by gradients, with mass scaling with learning rate!

∝λ2∝ η|g|∝ η(1 + β)∝1−βHarmonic oscillation with momentum. When considering the learning dynamics of momentum, the solutions weobtain take the form of driven harmonic oscillators, where the optimization hyperparameters have physicalinterpretations.
<latexit sha1_base64="hFD5IUVqKEqyBKOfrpywKGa8kLw=">AAAB8HicbZBNSwMxEIZn61etX1WPXoJF8FR2RdCLUPTisYKtlXYp2TTbhibZJZkVSumv8OJBEa/+HG/+G9N2D9r6QuDhnRky80apFBZ9/9srrKyurW8UN0tb2zu7e+X9g6ZNMsN4gyUyMa2IWi6F5g0UKHkrNZyqSPKHaHgzrT88cWNFou9xlPJQ0b4WsWAUnfXYiThSckX8brniV/2ZyDIEOVQgV71b/ur0EpYprpFJam078FMMx9SgYJJPSp3M8pSyIe3ztkNNFbfheLbwhJw4p0fixLinkczc3xNjqqwdqch1KooDu1ibmv/V2hnGl+FY6DRDrtn8oziTBBMyvZ70hOEM5cgBZUa4XQkbUEMZuoxKLoRg8eRlaJ5VA8d355XadR5HEY7gGE4hgAuowS3UoQEMFDzDK7x5xnvx3r2PeWvBy2cO4Y+8zx9mwI95</latexit>=0<latexit sha1_base64="+evKJKuSgigGlDMLVGTUw4Rm5hU=">AAAB8nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdSDUPTisYK1hTSUzXbTLt1swu5EKKE/w4sHRbz6a7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaYZb7FEJroTUsOlULyFAiXvpJrTOJS8HY5up/X2E9dGJOoBxykPYjpQIhKMorX8bsiRkmvi1q961Zpbd2ciy+AVUINCzV71q9tPWBZzhUxSY3zPTTHIqUbBJJ9UupnhKWUjOuC+RUVjboJ8tvKEnFinT6JE26eQzNzfEzmNjRnHoe2MKQ7NYm1q/lfzM4wug1yoNEOu2PyjKJMEEzK9n/SF5gzl2AJlWthdCRtSTRnalCo2BG/x5GV4PKt7lu/Pa42bIo4yHMExnIIHF9CAO2hCCxgk8Ayv8Oag8+K8Ox/z1pJTzBzCHzmfP1JNj/Q=</latexit>= 0.9<latexit sha1_base64="8kS57V6elUqfsYO8R0ZLxNu1Nzg=">AAAB83icbZBNS8NAEIYn9avWr6pHL4tF8FQSEbQHoejFYwX7AU0om+2mXbrZhN2JUEr/hhcPinj1z3jz37htc9DWFxYe3plhZt8wlcKg6347hbX1jc2t4nZpZ3dv/6B8eNQySaYZb7JEJroTUsOlULyJAiXvpJrTOJS8HY7uZvX2E9dGJOoRxykPYjpQIhKMorV8P+RIyQ1xq7Var1xxq+5cZBW8HCqQq9Erf/n9hGUxV8gkNabruSkGE6pRMMmnJT8zPKVsRAe8a1HRmJtgMr95Ss6s0ydRou1TSObu74kJjY0Zx6HtjCkOzXJtZv5X62YYXQcTodIMuWKLRVEmCSZkFgDpC80ZyrEFyrSwtxI2pJoytDGVbAje8pdXoXVR9Sw/XFbqt3kcRTiBUzgHD66gDvfQgCYwSOEZXuHNyZwX5935WLQWnHzmGP7I+fwB0LWQNw==</latexit>= 0.99hWi , 1ieq. (39)<latexit sha1_base64="atoVGm3iYUKrQEQE9yIra0AbF2U=">AAAB/nicbZDLSsNAFIYn9VbrLSqu3AwWwYWURARdFt24rGAv0IQwmZ60QyeTMDMRSij4Km5cKOLW53Dn2zhNs9DWHwa++c85zJk/TDlT2nG+rcrK6tr6RnWztrW9s7tn7x90VJJJCm2a8ET2QqKAMwFtzTSHXiqBxCGHbji+ndW7jyAVS8SDnqTgx2QoWMQo0cYK7COPEzHkgLsBO8euJ4tbYNedhlMIL4NbQh2VagX2lzdIaBaD0JQTpfquk2o/J1IzymFa8zIFKaFjMoS+QUFiUH5erD/Fp8YZ4CiR5giNC/f3RE5ipSZxaDpjokdqsTYz/6v1Mx1d+zkTaaZB0PlDUcaxTvAsCzxgEqjmEwOESmZ2xXREJKHaJFYzIbiLX16GzkXDNXx/WW/elHFU0TE6QWfIRVeoie5QC7URRTl6Rq/ozXqyXqx362PeWrHKmUP0R9bnDwRFlN0=</latexit>

classifier

Time (⌘ ⇥ steps)<latexit sha1_base64="RMpDnB/7xPsMFt8zjK4vPgt6L00=">AAACEHicbVC7SgNBFJ2NrxhfUUubwSDGJuyKoGXQxjJCXpBdwuzkbjJk9sHMXTEs+QQbf8XGQhFbSzv/xsmj0MQDA+eecy937vETKTTa9reVW1ldW9/Ibxa2tnd294r7B00dp4pDg8cyVm2faZAiggYKlNBOFLDQl9DyhzcTv3UPSos4quMoAS9k/UgEgjM0Urd46iI8YFYXIdAxLbuAjLpoKk1njkZI9PisWyzZFXsKukycOSmROWrd4pfbi3kaQoRcMq07jp2glzGFgksYF9xUQ8L4kPWhY2jEzEovmx40pidG6dEgVuZFSKfq74mMhVqPQt90hgwHetGbiP95nRSDKy8TUZIiRHy2KEglxZhO0qE9oYCjHBnCuBLmr5QPmGIcTYYFE4KzePIyaZ5XHMPvLkrV63kceXJEjkmZOOSSVMktqZEG4eSRPJNX8mY9WS/Wu/Uxa81Z85lD8gfW5w+8z50G</latexit>

We plot the column sum of the final linear layer of a VGG-16
model (without batch normalization) trained on Tiny ImageNet.

Conceptual Overview
Classical Mechanics

v.s.

Neural Mechanics

θN

z

x

y

Physical space

θ0

θ1Parameter space

Forces:Gravity, Electric/Magnetic, Friction etc…

Forces:Gradients driven by real world dataset

Equation of motion:
Newton’s Law (F(x) = m∂2t x)

Equation of learning:
Modified gradient flow

Symmetries in Lagrangian:Translation in time/space, Rotation

Symmetries in the Loss function:
Translation, Scale, Rescale

Conservation laws:
Energy, momentum, angular momentum

Broken conservation laws:
Dynamics of parameter combinations

Conclusion and Future Work
Two “hammers” developed and used in this work:

1. Symmetry: A unifying theoretical framework explaining how a network’s architecture leads togeometric properties in the gradient and Hessian.
2. Modified Gradient Flow: A realistic continuous equation modeling SGD with weight decay,
momentum, stochasticity, and discretization.

Understanding Deep Learning:

Task

Learning
Rule

Model
Symmetry&

Modified Gradient Flow