Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Authors: Liu Ziyin
Yizhou Xu
Isaac L. Chuang
Venue: NeurIPS 2025 poster ()
Keywords: entropic loss
symmetry breaking
universal representation learning
physics
Primary area: deep_learning

Abstract: With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its variants. Building on the theory of parameter symmetries and an entropic loss landscape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading to a series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, (a) explain the universal alignment of neural representations between AI models and lead to a proof of the Platonic Representation Hypothesis, and (b) reconcile the seemingly contradictory observations of sharpness- and flatness-seeking behavior of deep learning optimization. Our theory and experiments demonstrate that a combination of entropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.

Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Liu Ziyin 13, Yizhou , Isaac Chuang'
Massachusetts Institute of Technology École Polytechnique Fédérale de Lausanne 3 NTT Research
Abstract
With the rapid discovery of emergent phenomena in deep learning and large lan- guage models, understanding their cause has become an urgent need. Here, we propose rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its vari- ants. Building on the theory of parameter symmetries and an entropic loss land- scape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, (a) explain the univer- sal alignment of neural representations between AI models and lead to proof of the Platonic Representation Hypothesis, and (b) reconcile the seemingly contra- dictory observations sharpness- and flatness-seeking behavior of deep learning optimization. Our theory and experiments demonstrate that combination of en- tropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.
Introduction
Modern neural networks trained with stochastic gradient descent (SGD) exhibit complex plethora of emergent behaviors emergence of capabilities [1, 2, 3], progressive sharpening and flattening [4,5], 5], phase-transition like behaviors [6, 7]. and universal representational alignment across models [8, 9, 10, 11] that are difficult to explain through loss minimization alone. These behaviors mirror phenomena found in physical systems finite temperature, suggesting that deep learning dynamics are shaped not just by explicit optimization but also by implicit forces arising from stochasticity and discrete updates. These implicit forces have long been associated with the phenomenon of "implicit bias" in deep learning [12, 13, 14, 15], but their precise mathematical nature remains often captured by entropic
elusive. In physics, such effects emerge from the system's 's statistical tendencies rather than its energy landscape alone [16]. The power of this framework lies in the notion of an effective entropy, which plays the role of potential whose gradients define the entropic force. Identifying this effective entropy not only reveals what the system is implicitly optimizing, but also opens the door leveraging theoretical tools from statistical physics to analyze and improve AI models.
forces that
Contributions.
We formalize this connection between stochastic learning dynamics and entropic forces through the lens of symmetry and representation learning to:
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
"Equal contribution.

---

1. Derive an entropic loss function and show that the entropic forces of SGD break contimuous parameter symmetrris while preserving discrete ones (Section 3).
2. Show that the symmetry breaking due to entropic forces gives rise to a family of equipartition theorems that predict the gradient alignment phenomena (Section 4).
3. Explain and unify two seemingly disparate but universal observations progressive sharpen- ing of the loss landscape and the emergence of universal representations - as consequences of entropic forces (Section 5).
Our theory establishes a principled framework akin to a thermodynamies of deep learning - that unifies several universal phenomena under single formalism. The results suggest that the entropic loss landscape, shaped by both optimization and entropy, plays a foundational role in understand- ing learning dynamics and emergent phenomena. Full derivations and experimental validations are provided in the appendix.
2 Related Work
Modified Loss and Effective Landscape. The concept of modified or effective losses has emerged as a critical framework for understanding the implicit biases induced by stochastic gra- dient descent (SGD) in deep learning, which differs from another line of work [17, 18, 19, 20, 21] which leverages the property of stationarity to analyze the stationary distribution of SGD. Ref. [22] introduced the notion of a modified loss analyze the discrete-time dynamics of SGD, demonstrat- ing how discretization implicitly alters the optimization landscape. Similarly, Refs. [23] and [24] extended the modified loss formulation to where there is gradient noise due to minibatch sampling. These works conducted numerical simulations to show that training on the effective loss really ap- proximates the original dynamics [23, 25, 26] and leads to similar generalization performances. In this work, we refer to this type of losses as entropic losses for their associations with theoretical physics. Still, these entropic losses remain poorly understood, and their significance for understand- ing emergent phenomena in deep learning is not yet appreciated. Our work finds the crucial link between the entropic loss and symmetry-brre breakkng dynamics, which is important for understanding the various intriguing nonlinear phenomena of representation learning.
Parameter Symmetry in Neural Networks. Parameter symmetries are shown to play a funda- mental role in shaping neural network training dynamics and their emergent properties [27, 28, 29, 7, 30, 31]. A series of works showed that continuous symmetries in the loss function give rise to conservation laws, which imply that the learning result of SGD training is strongly initialization- dependent [32, 28, 33]. More recent works showed how any stochasticity or discretization effect could break the symmetries in systematic way such that the learned solution is no longer depen- dent on the initialization, a hint of universality [34, 29, 7, 31]. Particularly, Ref. [29] developed the formalism of exponential symmetries and proved that any loss function with an exponential symme- try leads to a conmeery--reaking dynamics that converges to unique fixed points. This point can be seen the dynamical equivalence of our Theorem 2, which states that there is essentially no contin- uous symmetry in the entropic loss. In comparison, our framework takes a different perspective: we study the symmetry from loss landscape perspective and identify these symmetry-breakinn ten- dencies as entropic forces. This unified perspective enables us to understand the universal learning phenomena with greatly simplified analysis.
Effective Energy for Stochastic Gradient Learning
Define Exx, 0) to be the per-sample loss function. We can define the empirical risk
L(0)= EgI Ex, ()].
(1)
where (x, == ((x,0) 2 B represents the minibatch and 7 represents the weight decay. From a dynamical-syster perspective, for an infinitesimal learning rate, the loss function coincides with the Bregman Lagrangian of this dynamics, and so one can leverage the Lagrangian formalism to understand the training of gradient flow [35]. This is particularly attractive from a theory per- spective because modern theoretical physics are also founded on the Lagrangian formalism and this connection allows one to borrow physics intuitions to understand deep learning.
However, simply studying this loss function is insufficient to understand the learning dynamics of SGD at various learning rates y due to the stochastic discrete-time nature of SGD. This motivates the definition of an entropic loss 1 such that running n steps of update on (0) with learning rate n/

---

is the same as running one step of update on b, for any и. Taking the limit n - co, one can obtain "renormalized" loss function for which running gradient flow is the same as running gradient descent for the original loss. With this loss, it becomes possible again to leverage Lagrangian formalism to understand SGD training with discrete-time and stochastic learning. Because Φo coincides with running gradient flow on by one must have that
O, == b, + 1011 + 1f 2 + +
(2)
We can also consider the more general case where the learning rate is a fixed symmetric matrix with ||A|| = 17. The following theorem derives the entropic loss for this case. Many common algorithms, such as Adam, natural gradient descent, and even wide range of biologically plausible learning rules [36] can be seen as having matrix learning rate.
Theorem 1. (Entropic Loss) For fixed x, starting from Og run one-step gradient descent with A on (x. 8) to obtain 01_ Run n-step gradient descent with A/n on Φ2 (x, == (x( 0) + (x, 22 (z, 6) to obtain B,,- Then, assuming l (x, 8)ll M,
(2) 0). then, 0, = 01 + O(IIAIS + ||A11//n); 2. moreover, if (2, 8) (,,,, (1 then 6², 01 + +|AA//+
1. II, 0)
| ||A| M).
This M, also needs to hold in expectation with respect to the sampling of data points, and so one can define the expected entropic loss F (0) E[ or, (1, 0)], which is, up to the first order in A and y:
(0)
Exl(x, 0) +
Es VA
+O(IIAB)). (3)
4
leurning. symmetry
regularization
effective entropy due to discretization error, nuise, 8(0)
Specializing to the first order and with a scalar learning rate, this equation reduces to the "modified loss" previously derived in different contexts [22, 23, 24). The derivation has thermodynamie flavor as essentially computes the degree to which the dynamics irreversible, and the entropy term is the part that cannot be microscopically reversed. In Theorem 1, the first-order term in encourages the model to have a small gradient fluctuation. The second-order term, on top of gradient regularization, encourages the model move to flatter solutions; this term has been found to play a role in the edge of stability phenomenon [5]. However, since the first-order term is not yet well-understood, our work focuses on the first-order term in A. We also focus scalar learning rate A = n and will comment on the differences when the difference is essential.
Now, treating the original loss plus regularization as an energy, the dynamics of gradient flow on F contains an energy force and and entropic force: Å -n)(V L 70 VS). The VS term, the gradient of the effective entropy, will be called the "entropic force." " See Figure for an illustration of the entropic foree and example of how entropy evolves during training.
We prove that the entropic force term breaks almost any continuous symmetry L,, a key result that we will leverage to study progressive sharpening and universal representation learning.
Definition 1. A loss function f(x, 0) is said to be K K-invariant if items 1-3 are satisfied:
1. locality: K(0, A) XQ(0) + O(23) for differentiable Q:
2. consistency: (KK(O X), KO0 + x));;
3. invariance: K ((x,0) for all x, 0 and X R.
An entropic loss Fny is said to have the (robust) K K-invariance if there exists a neighborhood around n such that Fg,7 K-invariant.
Theorem 2. (Symmetry Breaking Under the Entropic Loss) Let f be K K-invariant. If F is also robustly K K-invariant, then, (1) (8, A) = =|@@| and (2) (V for all
This means that any symmetry or invariance that Fg, has must be norm-preserving transforma- tions. Essentially, this means that any invariance that is not rotation invariance must disappear. The following theorem shows that orthogonal discrete symmetries are preserved.
Theorem 3. Let 00 0.
T. If [x, 00) ((x, 0) for any 0 and x, then F7, (00) (0) for any
3

---

Learning at large and small learning rate
102
104
D(1²))
(
loss
10%
10-3
test
((7²))
(
entropy
50 VL
50'
VL
50
0 VF
V6 V..
Original dynamics
Original dynamics
Effective dynamics
Entropic
epoch
Figure I: Entropic forces due to discretization error and stochasticity. Left: The learning dynamics of SGD at a large learning rate (LLR) and small learning rate (SLR) is different. One can view the difference hetween LLR and SLR training as coming from an entropy termm, which is an order 7 force. After entropic correction, the difference between SLR and LLR is reduced to O(1)²) and it becomes possible to analyze LLR SGD with gradient flow. Right: An example of entropic effect in neural network training. ResNet18 trained on CIFAR-10 with learning rate decay at 100 and 150 epochs. At the first learning rate drop (black dashed lines), the gradient (entropy) increases. This is unexpected and can only be explained by the entropic loss, where a large learning rate penalizes the entropy, and thus decreasing the learning rate leads an increase in entropy. The second drop does not create too much effect because the learning rate is too small after the first drop.
Together with the previous result, this shows that when gradient noise or regularization is taken into account, the only relevant remaining symmetries are discrete symmetries. This implies that the results that are based on conservation laws for understanding SGD are questionable and can only hold the toy setting of an infinitesimal learning rate. The reason simple: Far does not have robust invariances at 7 O. Also, note that had we used a generic matrix learning rate (e.g., with Adam), the orthogonal invariances would also be broken. The meaning of these discrete symmetries can be understood through a framework similar to that proposed in Ref. [7] and is left to a future work.
Emergence of Gradient Balance and Equipartition Property
Lie group symmetries exist abundantly in nature and in modern neural networks' [27, 28, 29, 7, 32]. In thermodynamics, the existence of symmetries is crucial fact that leads to the emergence of hierarchical phenomena and phase transitions between them. In a sense, symmetry can be argued to be the "first-order" approximation of the level of hierarchies in the system [37]. The following theorem states that the entropic loss F breaks any nontrivial noncompact Lie group symmetries (also known as exponential symmetries [29]). For formality, we say that l (and L) has A-exponential symmetry if for any X R, any x and 0, and any matrix A, {xx 0) obeys E(x, e*A0) = E(2, 0).
Theorem 4. (Master Balance Theorem) If (xx,0) has an A-exponential symmetry, then any local minimum 0 of (3) satisfies
-nEs[EEER Vol(x,0")) A[E (2) + A = =0,
(4)
where := A=A" In addition, either (1) (e (n,((0)) for all X, or (2) there exists no 2 # such that eAg is a local minimum.
If we had chosen symmetric A, would have À = A. For anti-symmetric A, this theorem is trivial consistent with Theorem 2. Therefore, this theorem is a statement about non-compact Lie group symmetries that extends to infinity. The fact that every point is connected to point that satis- fies Eq (4) means that SGD can reach this condition easily. The meaning of this theorem is general gradient balance and alignment phenomenon. When 7 O, this equation states that the gradient along the positive spectrum of À must balance with the gradient along the negative spectrum. When y # O, there is, additionally, a tradeoff between gradient balance and weight balance. We apply this result to various neural networks in this section.
Many works have shown that when training with weight decay or when the model has small initialization, the weights of the layers become balanced, especially in homogeneous networks [38,
See Ref. [31] for detailed review.

---

39, 40]. Theorem 4 shows that the SGD in discrete-time or with stochasticity leads to completely different bias where the gradient noise between all layers must be balanced.
ReLU Layers
Consider a deep ReLU network trained on an arbitrary task:
f(z) D- W
(5)
where R(x) is a piece-wise constant the zero-one activation matrix functioning as the ReLU activa- tion. The entropy term can be written
D S(0) E
- Tr
where 9i Exets (0,x) and E shorthand of Eg. Namely, we can group the gradient covariance according to layer index i. The following theorem states that all layers must have a balanced gradient. The proof shows that while the learning term L invariant class of symmetry transformations, the entropy term is not and this creates systematic tendency for the parameters to reduce the entropy.
Theorem 5. (Layer Balance) For all local minimum of Eq. (3),
n(ETr[g - -9:9 W"," w,w
Specifically, for 7 = we have gradient balance [TT[ = [TT[[g9]] For v we have the standard weight balance Tr[W,w7] Tr[,,,7], Otherwise, the solution interpolates between gradient balance and weight balance.
Similarly, within every two neighboring layers, one can group the parameters into neurons:
Tr[ + TT[9--197111 Σ > (Tr(.h.. Tigg-11599411.)
(8)
where Jij.; is the incoming weights to the j-the neuron of the i-th layer, and gi+1A is the outgoing weights of the same neuron. The gradient for each neuron must also be balanced because there is rescaling symmetry in every neuron.
Theorem 6. (Neuron Balance) For all local minimum of Eq. (3) and any i, j.
nETr(gas:: gi (114599-115) SyTr[ws.g... Wi- 1: W(1...]
T
From a physics perspective, we have proved an equipartition theorem (ET). The elements in the matrix B[gig]] 71 can be seen the temperature (or, the average energy) felt by each parameter. The trace TrE[ggg"] ] thus gives the temperature of the layer. That different layers emerge to have the gradient second momentum is an explicit form of the ET and means that the entropy must be evenly spread out across every layer. Because the standard ET in physics is property of thermal equilibrium, our result may be seen as an extension of the physical law to the -quuo--equilibrium dynamics of learning. See Figure 2 for the emergence of layer and neuron balances in ReLU network. We train on the MNIST dataset, but the labels are generated by a teacher ReLU network and trained with MSE loss. Also, Appendix A for examples of training trajectories and for an example with a self-attention net.
Polynomial Network Now, consider the case where R(h) is a diagonal matrix such that (h) hd corresponds to polynomial activation. This type of network is also variant of homogeneous networks [38]. For these networks, one can show that the converge to a state where every layer's gradient norm is d times that of the previous layer, leading gradient exploding vanishing problem (See Appendix B.8).
Self Attention Consider the case when model has generic form: (xx, W,U) l(x, WU), where W and U are matrices. The loss can contain other trainable parameters, which we ignore. Define Gw EvsVV [xx W,U), GG = Eus Vul(, W.U), and one can prove the following relation:
Theorem 7. (Gradient Alignment) For all local minimum of Eq.(3) we have
TBB(G GG 4wWW -UUT..
(10)
5

---

rnbelance
leyer imbolencee
4.0
4.0
4.5
entropy
1e-3
1e-2
entrapy
1e-3
Te-2
Figure 2: Layer and neuron gradient balance during training of two-layer ReLU network. Here, every dot a fixed time during training, where bluer dots are closer to the initialization, and redder dots are closer to convergence. Left 1-2: The entropy is strongly correlated with the neuron balance. As entropy decreases, the neuron balance improves. In contrast, the loss is not correlated to entropic effects all. Right: Similarly, the layer balance also correlated with entropy and not with loss function value.
This theorem is thus applicable to matrix factorization, deep linear networks, and, more importantly self-attention layers. The self-attention logit is computed a, X WUX,, where W is the key matrix, U is the query matrix. The loss function is a function of ai viewed as a matrix: {((a;))) Let V W2 Then, applying this theorem reveals intriguing relation:
WEE[GGGy W. B[Gv
(11)
Interpolating Weight Balance and Gradient Balance For the theorems above, we have also studied how weight decay affects the balance conditions. We see that the weight decay creates something analogous: instead of gradient balance, weight decay encourages weight balance, and this effect often cannot be achieved together with gradient balance. Thus, there is tradeoff between gradient balance and weight balance. In reality, the network is somewhere in between, where the weight balance and gradient balance has to "balance" with each other. Also, this is generalized form of an equipartition theorem. If we regard the sum of the regularization and S as a "total" entropic potential r, then this means that every layer will contribute equal amount to I.?
5
Implications
Next, we apply these results to study the emergence of universal representations in neural networks, and the progressive sharpening phenomena in deep learning optimization.
5.1
Universal Representation Learning
Recent works found that the representations of learned models are almost universally aligned to different models trained on similar datasets [41, 42], and even the biological brains [43]. This interesting phenomenon has rather philosophical undertone and has been termed "Platonic Rep- resentation Hypothesis" [8]. Here, we say that the two neural networks have learned a universal representation if for all Z1, T2,
(12) where ha the activation of network in one of the hidden layers, and hB for network B. This is an idealization of what people have observed and the difference between the two sides is the "degree alignment." We leverage the entropic force formalism to identify an exact solution to the embedded deep linear network (EDLN) model:
ha(21) ( (=
((0,z)) = I|M, WW (
(13)
on datasets DM, {(Maa, v:)). where M1, M2, M3 are fixed but arbitrary invertible matrices. They have the following meaning:
Mi can be seen model of the layers coming after the deep linear network:
NNww, it helpful clarify the role the Le regularization term. Conceptually, it can either be regarded as a part of the loss function which we have done so far, or as a part of the entropic term. Treating it as part of entropy is sometimes conceptually preferred because, like the entropy, it also functions as a regularization of the parameter space, limiting the accessible states.

---

1.0
1.0
0.9
0.9
alggr ment
0.6
layer cross layer
allgr mermmnnt
0.6
input
some layer
0.6
0.6
0.5
0.5
500
1000
1500
2000
500
1000
1500
2000
Iteration
Iteration
Figure 3: The representations of two 6-layer networks independently trained on randomly transformed MNIST hecome perfectly aligned for every pair of layers. The figure shows the average alignment hetween the same different layers of two networks. This aligmment does not weaken if the input arbitrarily transformed (Theorem 8). The black dashed line shows the average alignment to input data, which is significantly weaker. Left: linear network. Right: tanh network.
0.8
Net. nip layer
15
Net A Attention
Net B Attertion
0.6
0.2
0.0
0
10
20
10
20
10
20
10
20
-0.2
mip layer
layer
Figure 4: Alignment of representations two VIT models pretrained on ImageNet. Net A: ViT-B (#parrm: 86M). Net-B: ViT-H (300M). We see that both mip layers and the self-attention layers have mutually aligned representations both with itself and the other with each other. In particular, the alignment with itself is slightly better than with the different model, and the alignment of later layers is better than that of the first layers. A similar result with larger models shown Appendix A.5.
.M₂ models the layers coming after the embedded network;
M3 models different views of the data, which is common in multimodal learning therefore, the two models are trained on two different (but related) datasets.
In Theorem 8, we will train two different models, each with their own and potentially different M.,
M2. M3. The arbitrariness of these three matrices implies universality.
The data is generated by ys = Va, & for i.dd noise e, Assuming that Ez, Ee, and L. := [E Dg "x,x," the following theorem characterizes the global minimum of the entropic loss for this network in the case y 0 and 17 0+.
Theorem 8 (Perfect Platonic Representation Hypothesis). Consider two deep linear networks A and
with weights of arbitrary dimensions larger than (ank(VEL, /Eer). Let model A train on D and model B Dass Moreover, the outputs of models are multiplied by M2, M2, respectively. Then, the global minimum of Eq. (3), every hidden layer of A is perfectly aligned with every hidden layer of B for any x, in the sense that
multiplied by M1, Mi, and the inputs
h 4 (x) Rg La (2)
(14)
for 1 S LA DA and 1 LB DB and any z, where is scalar and R U,U satisfying UFU, U2 I. h hx) A M2M3a, hz) I, MrM3x denote the output of the LA.LB-the the layer of network A and B, respectively.
Here, a perfect alignment means that h (x) differs from h (x) only by scaling and a rotation. Because of symmetry, SGD converges to state where all possible pairs of the intermediate layers of two different networks are mutually aligned, independent of the initialization. This an extraor- dinary fact because there exist infinitely many solutions that are not perfectly aligned, also due to symmetry. For example, take hg to be any hidden layer, and transform its incoming weight by A and its outgoing weight by A-1 This remains global minimum for L, but it is longer the case that there exists an orthogonal transformation R such that h hn. Therefore, for almost all global minima of L, there is no universal alignment between layers- yet, SGD prefers a universal solution due to the entropy term. The following theorem shows that weight decay will lead nonuniversal representation.

---

From physics and thermodynamies perspective, it is quite reasonable that the irreversibility of the dynamies leads the emergence of universal structures. A state is not universal if it contains information about its initial condition. Therefore, the irreversibility of the learning dynamics helps erase information about the parameter initialization, thereby enabling the learning of a universal solution. The following hypothesis can summarize this perspective:
Irreversibility enables universal representation learning.
Now, it is worthwhile to remark on the connection and difference between this result and the original PRH [8]. First, our theory provides strong support for the PRH. The original PRH only hypothe- sizes positive similarity between models, and it is unclear whether the alignment score can reach (which implies a perfect alignment) or will only be a small positive value. Our result shows that, in principle, it is possible to reach the perfect alignment limit. Secondly, this result also offers an alternative perspective on the representation alignment phenomenon to that the original PRH pa- per. In the old perspective, one regards having alignment as the default expectation, and positive alignment as something to be explained and understood. In our new perspective, the perfect align- ment is the default, and breaking away from it is something to be explained and understood which is exactly what we will demonstrate in the next theorem. We also refer to Ref. [44] for a more in- structive derivation of the proof and for detailed discussion of different ways break the perfect PRH.
Theorem 9. Consider the deep linear network (13) with widths larger than d rank(V). Let 17 =
and 7 + O.. At the global minimum of (3), we have
W: = U.PEUU
(15)
for i- 1,,"., D. UD and Un are given by the SVD of V MT' UDUUG where $ E ded contains the singular values. For 2D.... 1, U; are arbitrary matrices satisfying U.¹ U;- = Laxd- Moreover, E 5 {P.) c Raxd are diagonal matrices containing #1 and satisfy 1122 P. = la-
The universal representation property (Theorem 8) does not hold anymore. Therefore, gradient balances lead to universal representations, whereas weight balances do not. See Figure for an experiment with deep linear and nonlinear networks. A surprising aspect is that every layer can be aligned with every other layer. Because any deep nonlinear network is approximated by a linear network for small weight norm [45], one could say that any nonlinear network is, to first order in [0], a universal representation learner. In Figure 4, we compare the alignment between different self-attention layers of two differently sized vision transformers pretrained on ImageNet. Note that different layers of the same network and different networks also have significantly positive align- ments, consistent with the solution for the embedded deep linear net. That using weight decay does not lead to universal representations is supported by the result in Figure 10.
Theorem 8 is direct (perhaps the first) proof of the Platonic representation hypothesis, implying that for any Ii, 3 22, Eq. (12) holds. Importantly, the mechanism does not belong to any previously conjectured mechanisms (capacity, simplicity, multitasking [8]). This example has nothing to do with multitasking. The result holds for any deep linear network, all having the same capacity and the same level of simplicity, because all solutions parametrize the same input-output map. Here, the cause of the universal representation is symmetry alone: in the degenerate manifold of solutions, the training algorithm prefers a particular and universal one. This example showcases how symmetry is indeed an overlooked fundamental mechanism in deep learning.
5.2
The Sharpness Paradox
The tendency towards learning universal representations can imply a curse for training. For exam- plc, the sharpness of the loss landscape really depends on the distribution of the data, whereas the solution the model finds is independent of these distributions this could imply that these solutions can be quite bad in terms of, say, optimization properties. Meanwhile, paradox of the sharpness- seeking behavior of SGD has become explicit. On the one hand, the edge of stability (EOS) states that learning typically leads to sharper solutions, whereas a vast majority of works have shown that SGD training leads to flatter solutions [46, 47J. These two cannot happen simultaneously and the solution must be that the sharpness-seeking behavior of SGD is crtuation-depedeen This section formalizes this intuition and shows that universal representation learning can be intrinsically related to the edge of stability phenomenon, which is also ubiquitous in deep learning.
8

---

0.40
2.0
0.35
theory
1.8
0.35
thcory
edge stabilty
0.30
date balance
0.25
1.8
1.4
0.20
1.2
0.20
1.6
0.15
0.05
0.05
1.2
0.6
0.3
0.4
0.5
0.5
0.7
0.3
0.4
0.5
0.6
0.7 0.0
0.1
0.2
0.3
learning rate
Teamning rote
cata talance
Figure 5: The entropic theory predicts the boundary for the edge of stability (BOS) phenomenon [4]. The theory shows that the imbalance of features and the uncertainty labels make the model converge to sharper solutions. We run two-layer linear network trained on a regression task. Left panel plots the quantity T7maa at convergence. For stability, 77max must stay (approximately) below 2, and the black dotted line plots the thcoretical boundary for 7/maax 2. Middle: The same figure that emphasizes the phase boundary. The bluc-red boundary empirically defined by the condition 77max
with 0.1 - due random sampling, the actual edge of stability is slightly smaller than [20]. Right: We control the learning rate and balance of the label noise this experiment. As predicted, as data noise becomes more balanced, sharpness metric mmmax gets smaller, indicating better dynamical stability during training.
3.0
2.6
sharpess
sharpmess
starpness
2.8
2.8
2.4
2.6
2.2
2.4
4.0
4.5
0.043
0.045
4.5
5.0
0.019 0.020 0.021
Figure 6: Example of trajectory of training during the progressive flattening (Left 1, 2) and sharpening traiming, and
(Right 2) the experiment Figure Here. blue dots correspond earlier dots correspond to later in the training. During progressive flattening. the decrease in sharpness correlates directly with the entropy term (Left 1). whereas the training loss is independent of the sharpness (Left 2). For progressive sharpening (Right), the picture is more complicated. The training trajectory follows three phases. Phase I: flattening correlates with decrease entropy; phase II: sharpening correlates with decreasing entropy; phase III: sharpening correlates with an increasing entropy. The phase cannot he explained hy the saading-order entropic loss because, as the landscape becomes sharper, higher-order effects in start to dominate training. At the same time. the loss is never correlated with these effects (Right 4).
Definition 2. The total sharpness is defined as T(0) = THEV?
This definition is chosen for analytical tractability and has been used in prior works [48, 29J. T upper bounds the largest eigenvalue, and T/d lower bounds it, so it is good metric of stability and sufficient for the theorem we will prove. The following lemma shows that if there is an expo- nential symmetry in l, every local minimum of & connects without barrier to local minimum with arbitrarily high sharpness. can be seen as generalization of the result of Ref. [49] to general symmetries.
Lemma 1. (Sharpness Lemma) Assume that A is a symmetric matrix and E(x,e AA0) {(x, 0). Moreover, assume that AEV?((x, Then, limsuppxi--- ( (Tc
One can analytically solve for the sharpness of two-layer linear networks, and identify precise cause of the progressive sharpening effect.
Theorem 10. For a two-layer linear network E(x, 0) = [ly(x)) W, W2al with y(x) = Va+e e Rd
and Ex Ee = 0, Ex = ErxT, Σ. EeeT. Denote Ü.S' to be the SVD of V" V.V Ez and assume that the width of the network larger than rank(V'), Then we have
Tr TH[E
T(0) = d
- THE
Tr[S']+
(16)
at the global optimum of (3). Meanwhile, the minimal sharpness of the global minimum of is min 22,,,TrE,155.
(17)

---

where S is the singular values of V VVE
This result implies that SGD has no inherent preference for flatter minima. See Figure 5-6, where we train a two-layer linear network on linear regression task with 2d label y e RR. The labels y = V*x a + 6, for a ground truth matrix V and iid zero-mean noise such that E. = diag(1, @2)). where Øx E (0, 1) is called the "data balance." In Appendix A.6, we also train deep nonlinear network, and we see the same trend where improving balance in the label noise leads to flatter solutions.
As an example, we can choose Σn I, (d2 dy = d), which gives V = VL, ÜS. and thus
T(0)= = o (,,,
/ T
(18)
which can be arbitrarily large. Recall that the minimum of T(0), on the other hand, does not depend on Σ., the label noise covariance. Thus, the imbalance of the noise spectrum can lead to arbitrarily high sharpness. This could especially be a problem for language model training because there is a large variation in the randomness of tokens. Some words. like "the," could have a very low conditional entropy, while nouns or verbs can have high entropy. especially when there exist synonyms. Another example is choose M. I, which gives V VV ÜSÜT,, and thus
T(0) 2v 2 ""='""""
(19)
which is exactly the same as the minimal sharpness. This suggests that without the imbalance of the label noise alone, SGD indeed converges to the flattest solution.
Thus, an imbalance in the input feature can lead different sharpness-seeking behaviors. At the same time, if the loss function has scale invariance (f(x, 0) = E( A0) for any À e R), the learning dynamics leads to a flattening of the curvature during training (See Section B. Thus, entropic forces and symmetry are strong factors deciding the sharpness-seeking behavior of SGD.3
6 Conclusion
In this work, we have proposed an entropic-force perspective on neural network learning. We de- rive an entropic loss, which breaks continuous symmetries and preserves discrete ones, leading to universal behaviors such as gradient balance and alignment. The entropic loss suggests a poten- tial unifying perspective for understanding training: learning algorithms prefer solutions with the minimal gradient fluctuation. This perspective provides a unifying framework that explains several emergent phenomena in deep learning. including charpness-seekinee structure, as consequences of underlying symmetries and entropic forces. The framework offers predictive and explanatory power across architectures and scales, and points toward a more princi- pled, physics-inspired understanding of learning dynamics and emergent phenomena. Future work may extend this foundation to encompass higher-order corrections, richer architecture structures, and nonequilibrium dynamics of modern training procedures. A major limitation of our work is that we focused only on problems with explicit symmetries; an important future direction is to extend the results to cases with only approximate symmetries. Our experiments show that symmetry-fin systems qualitatively agree with symmetry preeeerri-reeeiiiggg systems, suggesting that there are underlying, hidden concepts yet to be discovered.
behavior and universal feature
An implication of Theorem 2 is that discrete symmetries such as Zg still remain in the loss function. This enables the possibilities of spontaneous symmetry breaking and phase transitions in neural networks. Prior works have studied phase transitions [7] without the entropy term and are inherently zero-temperature phase transitions. It may be possible to develop a theory of phase transitions based on entropic loss, similar to the classical Landau theory. Also, a striking aspect of our construction is its similarity to the actual formalism of thermodynamics in physics; our result motivates the development of a robust thermodynamics theory of deep learning.
VViewed together with the result in Section 5, one reaches an interesting and surprising conjecture: pro- gressive sharpening and universal representation alignment, with entirely different phenomenology. may have the same underlying cause, and could be two sides of the same coin.
10

---

Acknowledgment
The authors thank Prof. Phillip Isola and Prof. Tomaso Poggio for discussion. ILC acknowledges support in part from the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) through NSF Grant No. PHY-2019786. This work was also supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF - 1231216.
References
[1] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint Sxiv:2206.076822 2022.
[2] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 01226699-2772 2023.
[3] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307. 15936, 2023.
[4] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gra- dient descent on neural networks typically occurs the edge of stability. arXiv preprint arXiv:2103.0006 2021.
[5] Jeremy M Cohen, Alex Damian, Ameet Talwalkar, Zico Kolter, and Jason D Lee. Under- standing optimization in deep learning with central flows. arXiv preprint arXiv:2410.24206, 2024.
[6] Liu Ziyin and Masahito Ueda. Zeroth, first, and second-order phase transitions in deep neural networks. Physical Review Research, 5(4):043243, 2023.
[7] Liu Ziyin. Symmetry induces structure and constraint of learning. Forty-first International Conference on Machine Learning, 2024.
[8] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
[9] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.0694 140.094400 2024.
[10] Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, and Paul Pu Liang. Understanding the emergence of multimodal representation alignment. arXiv preprint arXiv:2502. 16282, 2025.
[11] Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani. Lu Cheng, and Mengnan Du. Large vision-language model alignment and misalignment: A survey through the lens of ex- plainability. arXiv preprint arXiv:2501 2501005000 2025.
[12] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[13] Tomer Galanti and Tomaso Poggio. Sgd noise and implicit low-rank bias in deep neural net- works. arXiv preprint arXiv: 055944 2022.
[14] Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gra- dient descent at the edge of stability. arXiv preprint arXiv:2209. 15594, 2022.
[15] Shuo Xie and Zhiyuan Li. Implicit bias adamw: lor -norm constrained optimization. In International Conference on Machine Learning, pages 54488-54510. PMLR, 2024.
[16] Rong-Gen Cai, Li-Ming Cao, and Nobuyoshi Ohta. Friedmann equations from entropic force. Physical Review D-Patiicles, Fields, Gravitation, and Cosmology, 81(6):06150, 2010.
[17] Sho Yaida. Fluctuationddisiiaaion relations for stochastic gradient descent. arXiv preprint arXiv: 1810000004 2018

---

[18] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. Advances in neural information processing systems, 32, 2019.
[19] Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations, 2022.
[20] Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. In International Conference on Machine Learning, pages 7045- 7056. PMLR, 2021.
[21] Liu Ziyin, Hongchao Li, and Masahito Ueda. Noise balance and stationary distribution of stochastic gradient descent. Physical Review E, 111(6):065303,, 2025.
[22] David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:209.111662 2020.
[23] Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12170 2021.
[24] Yatin Dandi, Luis Barba, and Martin Jaggi. Implicit gradient alignment in distributed and fed- erated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6454 6462, 2022.
[25] Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. arXiv preprint arXiv:2109.144 141 2021.
[26] Daniel Kunin, Javier Sagastuy-Brenal Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel LK Yamins. The limiting dynamics of sgd: Modified loss, phase- space oscillations, and anomalous diffusion. Neural Computation, 36(1):151-11) 2023.
[27] Xingguo Li, Zhaoran Wang, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, and Tuo Zhao Symmetry, saddle points, and global geometry of nonconvex matrix factorization. arXiv preprint arXiv:
1:5-1,2016
(28) Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accel- erated optimization. Advances in Neural Information Processing Systems, 35:16679-16690, 2022.
[29] Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equi- librium of stochastic gradient descent. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[30] Liu Ziyin, Yizhou Xu, and Isaac Chuang, Remove symmetries control model expressivity and improve optimization. The Thirteenth International Conference on Learning Representa- tions, 2025.
[31] Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry potentially unifies deep learning theory. arXiv preprint rxXv:2502.0$$300 2025.
[32] Hidenori Tanaka and Daniel Kunin. Noether's 's learning dynamics: Role of symmetry breaking in neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25646-25660. Curran Associates, Inc., 2021.
[33] Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows. 2023.
[34] Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. arXiv preprint 242505,,, 2023.
12

---

[35] Andre Wibisono, Ashia C Wilson, and Michael I Jordan. A variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences, 113(47) 2016.
[36] Liu Ziyin, Isaac Chuang, and Tomaso Poggio. Heterosynaptic circuits are universal gradient machines, 2025.
[37] Philip W Anderson. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science, (7(4404):333-399) 1972.
[38] Simon Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
[39] Andrey Polyakov. Homogeneous artificial neural network. arXiv preprint arXiv:2311.1797. 2023.
[40] Akshay Rangamani and Andrzej Banburski-Faud Fahey. Neural collapse in deep homogeneous clas- sifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4243-4247. IEEE, 2022.
[41] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225-236, 2021.
[42] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages
PMLR, 2019.
[43] Daniel LK Yamins, Ha Hong, Charles F CCdieeu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performannceopttmizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111233::619-8644 2014.
[44] Liu Ziyin and Isaac Chuang. Proof of perfect platonic representation hypothesis. arXiv preprint arXiv:2507.01009, 2025.
[45] Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of a deep linear network. In Advances in Neural Information Processing Systems, 2022.
[46] Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27444 27461, 2021.
[47] Lei Wu, Mingze Wang, and Weijie Su. The alignment property of SGD noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems, 35:4680-4693, 2022.
2022.
[48] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. In International Conference on Learning Representations, 2021.
[49] I. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp Minima Can Generalize For Deep Nets. ArXiv e-prints, March 2017.
13

---

100
2.0
1ayer1622
layer162
layer belance
0.8
1ayer2663
neuren balance
layer26.3
layer364
layer364
0.6
1.0
0.4
0.2
0.5
0.0
0.0
5000
15000
20000
5000
10000
15000
20000
Figure 7: 4-layer linear networks and weight decay trained teacher-student setting. Left: Layer imhalance for each layer, which verifies Theorem 5. Middle: Neuron balance for each layer, which verifies Theorem 6. The curves are smoothed and averaged over 5 runs better visualization. Right: Loss and entropy.
layer162
layer162
layer balence
16yer2633
layer364
neuron balance
layer566
layer364
20000
20000
epoch
epoch
Figure 8: The same setting as Figure 7, but for ReLU activation.
A Experiment
A.1 ResNet
For Figure 1, we train ResNet18 on CIFAR 10 using SGD with momentum 0.9, batchsize 128 and weight decay 5 x 10-4. The learning rate is 0.1 the beginning, 0.01 after the 100-th epoch and 0.001 after the 150-th epoch. The entropy is calculated by summing the gradient norm of all parameters. We obtain training accuracy 98% and test accuracy 88% at the end.
A.2 Gradient Balance
For Figure 2, we train on the MNIST dataset but the labels are generated by a teacher ReLU network and trained with MSE loss. Namely, the loss is
(f( v(x) cz
(20)
where v(x) is parameterized by random ReLU teacher network and is an i.i.d. Gaussian noise with 0.2 standard deviation. The training proceeds with SGD for 10s steps with a learning rate of 0.01 and batchsize of 200. Layer balance calculated by [ETr[g., -94-19+11 and neuron balance calculated by Σ, |ETr[9,: 91-1:-9 for the i-th layer.
Additional experiments on layer balance and neuron balance are presented Figures 7, 8 and for 4-layer linear networks, ReLU networks and simple self-attention networks, still for the teacher- student setting. The hidden dimensions are 256, 128, 64 for linear and ReLU networks, and 256 for self-attention. We present the evolution of layer and neuron balance along training. Figures and 8 suggest that layer balance and neuron balance approach zero during training, which verifies Theorems 5 and 6.
14

---

gradient imbalance
gradient imbalance
2.0
imbalance
5
4
3
1.5
2
1.0
2-
1
0.5
1
0
0.5
1.0
0
2000
4000
6000
8000
10000
entropy
loss
le1
epach
Figure 9: A self attention networks y (="UV and weight decay trained teacher-student
setting. The gradient imbalance is evaluated by ||G Gy GyGlllr. Left: Gradient imbalance is strongly correlated with the entropy. Middle: Gradient imbalance is weakly correlated with the entropy. Right: The evolution of entropy along training, which is averaged over runs.
A.3 Gradient Balance in Sell-Attention Nets
Figure 9 suggests that GJ Gu Gy approaches during training, and it is correlated with the entropy rather than the loss, which verifies Theorem
15

---

a alignment
alignment
10
1.0
10-1
1.0
0.8
0.8
weight decay
wnight decay
10"2
0.6
10"2
0.6
0.4
0.4
0.2
0.2
10-3
10-3
0.0
0.0
10' batchsice
10'
batchsize
Figure 10: The alignment of two 2-yayer ReLU networks independently trained on a teacher-stusee student setting. We measure the alignment for different batchsizes and weight decay. Left: SGD. Right: Adam.
1e-1
1e-1
lignmeent
aignment
2.5
3.0
4.0
0.95
1.00
1.05
1.10
1.15
entropy
1e-1
Figure 11: Universal alignment strongly correlated with the entropy but not loss. The setting the same as Figure 10, where we use SGD, weight decay 0 and batchsize 100.
alignment
alignment
1.0
weight decay
umight dacay
10
10
0.4
0.1
0.2
0.2
10
10"3_
batchsize
batchsize
Figure 12: The same setting as Figure 10, two-layer linear networks. Left: SGD. Right: Adam.
A.4 Unversal Representation Learning in MLP
For Figure 3, we train two independent 6-layer networks on MNIST. The networks have linear or tanh activation and 128 neurons each hidden layer, and for the second network, the input MNIST data is transformed by a random Gaussian matrix. We train the networks with Adam optimizer, learning rate 10 for 5 epochs. During training, we measure the representation alignment between every pair of layers, defined as the cosine similarity between the two sides of (12), averaged over the test set. We then plot the average alignment between the same or different layers of two networks. The input alignment denotes the average alignment between every layer representation and the input data.
In Figures 10 and 12, we test the influence of batchsize and weight decay on universal representa- tion teacher-student setting. In Figure 10, both the teachers and students are two-layer ReLU networks. In Figure 12, the student is replaced by a linear network. Their hidden dimensions are 100. Similar to Figure's 3, we measure the representation alignment between the middle layers of two independently trained networks, and the input of the second network is rotated by Gaussian matrix. We use random Gaussian data, and the labels are generated by teacher network. We train the student networks with SGD, learning rate 5 10-2 with Adam, learning rate 10" For both SGD and Adam optimizers, Figures 10 and suggest that universal alignment does not rely on the batchsize predicted, but disappears for large weight decay, which verifies Theorem 9, In Figure
16

---

11 I, we show that the increase of alignment is more correlated with the decrease of the entropy rather than the loss.
17

---

0.8
Net A mip layer
Net 8 mip layer
Attention
Altentton
15
0.4
10
0.2
0.0
-0.2
10
20
10
20
10
20
10
20
Net A mip layer
Net A mip layer
Net Attention
Net Attention
Figure 13: Alignment of representations of two larger VIT models pretrained on ImageNet. Net A: ViT-L (#param: 304M). Net-B: VIT-H (633M). This is similar Figure 4
A.5 Universal Representation in ViT
See Figure 13 for the alignment in Vision Transformer. The pretrained weights are taken from https:/ httes://dooee /wwc//doo/////oo/oooooo//cc/o/d/ peocs.pytooree pytorch. org/viivvv s on ma ieniin/oee 1s. html. We measure the CKA alignment between the two models or with itself with minibatch size of 300 images from the ImageNet dataset.
18

---

0.40
0.35
1d
063 balonce
0.25
12
0.15
-10
2966
1.2
0.05
0.8
0.3
0.4
0.5
0.5
0.7
0.0
0.1
Figure 14: Sharpness convergence for two-hiddee-layer ReLU network. The setting identical to that of Figure 5. Again, we see that higher imbalance the label leads to a sharper solution.
A.6 Edge of Stability
See Figure 5 and 6, where we train two-layer linear network on linear regression task with 2d label y IR. The labels V* x c, for ground truth matrix V* and iid zero-mean noise e such that Σ. diag(1, o²), where o₂ (0,1) 1) is called the "data balance."
We train with different learning rates and data balance. The training proceeds with SGD with batchsize of 32 for x 10a iterations. Input x e IR² is drawn from a standard Gaussian distribution, and the model has dimensions 2 - 10 2.
See Figure for an experiment with ReLU. Here, the architecture is has dimensions 10 10 + 2.
19

---

B Theory
B.1 Scale Invariance Leads to Flattening
The symmetry generator is A = I. We have
()) (0(
(21)
by Theorem 4. As π Fe, 2200) F Therefore, when we do gradient descent along
-V Fn,0, À monotonously increases. Meanwhile,
0(c*0) = ((=
(22)
decreases with and thus the sharpness decreases along training.
B.2 Proof of Theorem 1
Proof. For notational simplicity we drop the subscript y in the proof. When running gradient descent on f(x,0). 0), we have
01 = 0g - AV((r,
(23)
When running gradient descent on ΦA, we have
8, = 0o
((1A))))
(24)
and
02 = 01 (V((I, + (V) O(TAA)))))
00 vl(r, b1) 2/2 ( -
2A
(x,
(25)
(,, + 0(1A)))
2 の1 (1 n
Similarly we can obtain
A 0₂ 0 AV((2
(1) ( (
(26)
2 (I, 8)) (z, OCIAI' + /1A19/+
Therefore, we have 0/= 01 + O(IAAI + /n) if we choose (I, -V((E,
For small (3((x, 0), we have
(T (C O)))
2 VTT.:::00/0//CC00000660001000000 A)1))
(27)
Therefore, can choose
(2 2 -
(28)
to obtain 0' 0, + O(IIA|I + +||A||//= ||/13/n (|,++[1>)(7++ O))
B.3 Proof of Theorem 4
Proof. By the definition of the exponential symmetry,
(A)) 0) {(x,0),
(29)
Taking derivative w.r.t. À on (29), we have that 0) AA = 0. Then taking derivative w.r.t. 8, have that A Vol(x,0) vzl(z, 0) A0=0.
20

---

d Let ((A) := di (eA and 02 = ex Then we have
1()) ! EE V'L
A0
2
- Aaa
(30)
= -
2
= (T
"
2
where Σ(fx) = Es B Vol(2, Vo 0x]] is positive semi-definite. We also use
Tr{E((x) A-A"1 and (0x) A-A 0x 0. By [29, Lemma B.I], we have Tr{E(0x)A]
Tr(e 21 (0*) and thus
" Tr Ae 22A [(0')] - 22A A0*
T(A) =
2
(31)
= Σ !
2 uue 22p (nk m())) 2yyu;e 2np (n,
where Mi, are eigenvalues of eigenvectors of the symmetric matrix Á. Therefore,
='a)== = = 2A n; E(0")) n; 4ye 2A14 (n² 0, )) 2 0.
(32)
We have I(A) = 0 (which happens only À not full rank) or (A) strictly monotonic. As 0 is local minimum, we have I(0) O, which gives (4). Then we have [(A) = 0 ((A) = 0 iff = 0. which finishes the proof.
B.4 Proof of Theorem 2
Proof. By the definition of K-invariance, and taking derivative Vo of both sides of (I., ((,)))
6), we have
VoK(O, K(O ((tto)))(,,K((,A))) (())))) vol(x.0), where the Lh.s. follows from the chain rule. This imlics that
(33)
(I+ +VQ (e) vol(z,,)
(34)
and
CT,, (x
= - ([-AVQ++((A2)))) (1-AQQ-002)))00(tt
(35)
If FfFir is K K-invariant, the following Equation holds:
f(x, (/)(,)))))/V
K + (aC.://--0jjjl.-..e./----------- 11011 (36)
((, By the ((s A)) ((1, 0). and (36), have that
||Vol(z, ||0| (|V ((0.))
(37)
= = =1
(38)
Thus,
7' vV 0.
(39)
There are two cases: (1) "V vL- v 0, and (2) r/ ve==
For case (1), we are done. For case (2). the equation cannot hold for + dy because the first term is independent of y. Thus, we can only have case (I).
This means that for any
K(0,, X)[l |10|2 O(x2),
(40)
which is only possible if ((,, 2 This completes the proof.
21

---

B.5
Proof of Theorem 3
Proof.
By definition we have ||Oø||* ||011. Take derivative on both sides of ((x, 08) ((x,0)
we have
o Voollzz 08) vol(2,0)
(41)
Thus have
- 0), which gives [ vo0 = Vo 0)1 Combining the above results we
(42)
have Fn,7 Fg, (0).
B.6
Proof of Theorem 5
Proof. Rescaling symmetry implies that if we make the transform
W; W. W, e ~W
(43)
does not change.
This corresponds to the choice of
1
k=k=
kbba Akim
-1
k k=
(44)
0
otherwise
in Theorem 4, where the index klm corresponds to the m-th element of the I-th unit of the k-th layer. Then we have
7 (ETr[gg] - 9;9 4> - W.
(45)
This finishes the proof by setting ? = 0.
B.7 Proof of Theorem
Proof. We can choose A to be a rescaling matrix w.r.t. the j-th neuron of the i-th layer. Specifically, choose
;=k=ill=== k=k=i+l,m m =j
klm A
(46)
klm
otherwise
Theorem 4, which gives
WETTr(os:::fjj (e-1n99he1a Th Wi.j. wT, - Wi-1
MyTrrWi,j
(47)
B.8 Gradient Imbalance in Polynomial Networks
Theorem 11. (Neuron Balance) For all local minima of Eq. (3) and any i, j.
TETr(g.,.99 dgg- dge-1:-91-1.g] = 4-Trr[ws..uu dw 1nd w
(48)
This means that unless d 1, these networks will either have a noise or weight explosion problem. If y O, the gradient fluctuation grows like dD, exponential in depth D. When d < 1. later layers will have an exploding noise; when d 1, earlier layers will have an exploding noise. When both and 7 # 0, the sum of the noise and gradient norm will explode exponentially. In some sense, this implies that linear or sublinear types of activations are the only stable activations for deep neural networks.
Proof. We can still choose to rescaling matrix, but this time we should rescale the i 1-th layer more
Wids e e Wirled
dA Wailid
(49)
22

---

This corresponds to
-=k=ill=i=)
A
kita kim
d k=k=i+1, m j
(50)
otherwise
in Theorem 4, which gives
nET dg-- dge1::91-1;51 47/r[[us dw 10
(51)
This gives
(52)
by choosing y =
B.9
Proof of Theorem 7
Proof. The double rotation symmetry can be written
U EU W - e
(53)
where A is an arbitrary matrix. We thus choose the following generator
k k= 1, 1= i, j. or j, k == 2, 1= i, j. or = j, =
A klm
-1
(54)
0
otherwise
in Theorem 4, where h= corresponds to W and k = corresponds to U. This gives
nl Gwa, Gua GUaL Σ 47 Unk],
(55)
which finishes the proof.
B.10 Proof of Lemma 1
Proof. By definition, have
AA (A (0
(56)
and thus
T(e^A0) TH(e 22A 2MM
(57)
Let A == Es Mimm and thus
T(c^A0) = Ee 2AMi (nf EV E 0)n:).
(58)
As ARv?(((x, ( 0) # 0, there exists such that μ * 0 and n ( 0)n; * 0. Therefore, we have lim (T(c^A0) +o0 if μr < O, and lim (T(cAA0) = +oo if 14 >
B.11 Proof of Theorem 8
We first prove the following theorem, which we will leverage to prove Theorem 8.
Theorem 12. Let V' = VE,V such that = ÜS' its SVD and rank(V')) Assume that every layer has more than d hidden units. Then if y and nj 0*, at any global minimum of (3), we have
V M.W. Ü.DUB-I W, W.= -UUUU WrM,M UNEV
(59)
for 2,..., D 1, where U, are arbitrary matrices satisfying U. U. Iaxd. and E, E[xx" E. B[ec"'. Moreover,
Σ1 "11'"" D.2 IMMMM,,, M 20
ST
Tr[ME E. M, 20
Ep (T:=(T 2.2 HHMME, D-1
(60)
V
Σ, = Tr[MM2M,,,, MT
Ma
at la-
23

---

Proof. Consider two consecutive layers W and W1-1. Using Theorem 7, we have
BB[G Gw Gw, [
(61)
By the MSE loss ((r,y) () = 11y M, W. WT M2MMa this gives
W }6 W?" Bll. Ki-1
(62)
where ē EreB and A := Exs (V- M1 D...WWWWWWWW........... S.....i.M4 MMM2 satisfy E* = Er O, and thus Eas"
Σ2
18
and BEET" Ln
We use the fact that at the global minimum have Mi D...W. Mr V.
[8
and thus y - M1 WDW D.--Wi.MMMM
is independent of x. We denote Ê+11 Mr W D... W4-2. h4 == Wi-1..WW M2 M3 for i 2,3, -,, D 2, and ED == M1, h1 := M2M3.
Finally denote W1
W1 W1M2MMvV Σg and W V.,M,WD which gives 1+2 -.....-' W...-.W.... WWWWWWWW
V and
WT
WT
Wi+ W,
W
(63)
W W(WT ......I
for io = 2,3, D 2. For iori= we have
W W....WW... W3
WWWT
W2=
(64)
Tr W. WT WT W. W33
Tr [MMM
and for i= we have
W Tr M = WD
W WWWT D-2 WT W
(65)
Tr WD- WWWTT...WW W... D-2
Lemma 2 proves that we can decompose the matrices W1, W2, DD1,, W as
WD = E-D WD = UD-
Wi U1E
(66)
By minimizing (3) at ry 0*, we need to minimize
Elyy W W1 E||e| + - 2
(67)
Thus have
(V WV =0,
(68)
which gives
= W' Eg =V.
(69)
Then we obtain Up - Ü, Un= V and
-
(70)
We can assume Ep, Ej ddd because their ranks the same by (63), (64) and (65).
(63) gives
Σ 142 E2
EE... TrEE....
(71)
Tr(22 1+2
D2
and thus Σ, cla for -2,3, .., D and
Σ
Σ/
(72)
Tr
Tr
Σ
(64) and (65) give
Σ
D-1
(73)
Tr
Tr(EZ..EE E2
T.MMME MIT
TH[M E.MMEMM, M1
Tr
52
and thus
2 Σ
E
Σ
Σ
2
(74)
Tr[ TMMMMEM
TH[M E.M M1]
Tr[
Combining (70), (72) and (74), we finish the proof.
24

---

Now, we are ready to prove Theorem 8.
Proof. By Theorem 12, any global minimum of (3), the solution of Da-layer network for the dataset DM is given by
VE,,WA, ÜEDUbA-L W - UUE,U -,
(75)
for i = 2,,,, D 1, where U, are arbitrary matrices satisfying U? U, Lxxds and
Σ VS, ED Vs En x La
(76)
for some constants C1, C2, C3. The solution suggests that
h L A (x) W. M2M2a ULA V 15
(77)
Similarly
h (x) 1 MMMMa ULD VS
(78)
The proof is complete by comparing (77) and (78).
One might also consider the case where the minimal width of network B is dp d. In this case, we denote S' Bd containing top dæ values of S", Then we have
1 (x) n MyMM, UL, V/V
(79)
is now not fully aligned with , To calculate the alignment, the corresponding kernels are
Ka(z1,02) = h (x)) ) LA
(80)
and
KB(x1, (x) ",)"
(81)
for some constants C2 O. We then have
(KA, KA)F (X) (T{
(82)
Similarly we have
(83)
and
(Ka. = (aa
(84)
Therefore, the alignment is given by
(KA, KB)F
TY(S'S
2r(
(85)
VKK, (Kn KB)F (T((())9TT((5)
Tr[(s')2]
which is some value between and 1. In the end this section we proves the following technical lemma.
Lemma 2. Suppose that matrices Wi, W2, .., WD satisfy
Whe W 2.. D...
X.W.W.L..WWWY W.W. W.
(86)
for some X > and 1,2, 1, then we have write the SVD of W1, W2."., WD as
WD UDEDUD-1 WD-1 UD-1Eo- W1 UEE,UO.
(87)
where Ep.",, En the singular values.
Proof. Denote P. ==
We then have
W ++1 W. w............... = X;P:
(88)
and
WWW....W.W.......... "i,
(89)
25

---

which gives
A+t1 P. S P.S
(90)
X
where Si-1 W W+
Suppose that S VAVT, and thus have
A = cAAA,
(91)
where c:= Aul and A:= P.V. The (j.k) k) element gives A
A - cx)3A))
(92)
If Aia # 0, as A is semi-definite, we also have Asi: Abs * O, which gives ct, ci = Therefore, have AA AA, and thus
PiS = =
(93)
This shows that P; and Si+1 share the same eigenspace. Moreover, we have P. WiP,- W Denote W; U and W,+1 = U++1 V. As P.-1 share the same eigenspace with W W,W, we have P.-1 AU? for some diagonal matrix A, which gives P. U,E,A As Pa share the same eigenspace with WT W VT
2 V, we obtain V U.. which finishes the proof. C
B.12 Proof of Theorem 9
Proof. By Theorem 7 we have
W Wt.1 W.W."
(94)
for i= 1,,,., D 1, which suggests that W; =UEEU?, with Σ = Ei.1. At the global minimum we have (I ( 11 W; Mr which shows that the left side is the SVD of V VM, This finishes the proof.
B.13 Proof of Theorem 10
Proof. By Theorem 12 we have
TH ÜVSU, WVE TH U.V.VV
VEUU= T 4
(95)
TrE.,
By [31, Proposition 5.3] we have
S(0) = d, TH[,, 10
THEE
(96)
= dy
Tr[S'] -
TME
This finishes the proof of (17).
Meanwhile also calculate
Ut W
min dy|WW> U.W
||U
(97)
As UWE!/2_ VEL/2 == ÚSV the minimum is given by
1/4
-1/4 0,V3V
Tr dy
TrΣ,
U"= =
Ü SÚ,, w'y///
Ú,
(98)
dy
and
min 2dd,, TrS
(99)
where Ug is an arbitrary orthogonal matrix.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract explicitly states that out main contribution a theoretical frame- work for understanding the learning dynamics and emergent phenomena of neural networks trained with stochastic gradient descent (SGD) based entropic loss landscape.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed this in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a apeech- speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[Yes]
Justification:
All theorems are stated with the full set of assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We present all experiment details in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The experiments are only for demonstration and are straightforward to repro- duce following the description.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines thttps: /wni/nipsco
public/ guides Ceoe//odeeuuuna/ ss ssionP asionPolicy oolicy) for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//n i ps.. cc/ publ c/ gu i des / CCodeSubm i as onPol i ccy for more details. The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We disclose all hyperparameters of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We plot the standard error as the shaded area.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Personal computers are sufficient for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tt////errpppp/ wc/ppuliic/ttiiieuulleeeee..e
Answer: [Yes]
Justification:
We closely follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use pytorch. We use MNIST and CIFAR datasets, and the publically available pretrained weights from the pytorch website.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / ps wct.c//oonnnnnes//////ccc 2025/LLM) for what should or should not be described.
33


Reviews:
[{"review_id": "4zb95igdgP", "reviewer": "Reviewer_UAQt", "summary": "This paper tries to bridge the gap between theory and practice in deep learning. It studies a notion of entropic loss landscape and proves that the associated forces break continuous symmetries while preserving discrete ones, and gives a tradeoff between gradient and weight balancing. This also helps explain recent observed phenomena such as progressive sharpening and flattening in deep learning. Several experiments are run to support the claims.", "questions": "- What if the conditions on $\\phi_{\\eta}$ do not hold in Theorem 1 ? How often do they hold?\n- What if the learning rate is not a matrix ? You said Adam corresponds to a matrix, what about other optimizers ?\n- In Figure 2, is what you call entropy the $S(\\theta)$ ?\n- Does $\\phi_{\\eta}$ contain $\\gamma$ in its definition? If so, how? If not, how did the gamma appear when computing its expectation?\n- Can you say what the symmetries present (continuous and discrete) are in shallow linear network square loss for example ?\n- Are there some losses in practice that have A-exponential symmetry ?\n- How is theorem 2 saying that the continuous symmetry is broken ?\n- Does the sharpness result change if we choose the largest singular value of the hessian instead of the trace?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The research direction is interesting and may help explain some phenomena observed in practice in deep learning\n- Results are supported with experiments.\n\nWeaknesses:\n- The results rely on strong assumptions.", "was_revised": true, "final_justification": "My concerns were addressed", "rebuttal": "Thank you for your positive feedback and the detailed questions. We answer both the weaknesses and questions below. If you have additional questions, please raise them and we will be happy to address them.\n\n**Q1. The results rely on strong assumptions.**\n\nThank you for your comment. Our theoretical framework (Theorems 4-7) is derived primarily from the existence of exponential symmetries, which are intrinsic properties of modern neural architectures, and they are not assumptions. For instance: rescaling symmetries of ReLU layers and rotational symmetry of attention layers. Also, see our answer below to Q6.\nThese symmetries are structural characteristics of the architectures themselves, not additional assumptions imposed by our analysis. Consequently, our results reveal fundamental and **universal** properties that hold whenever these widely present symmetries exist, without requiring further restrictive conditions.\n\n**Q2. What if the conditions on \\phi_\\eta do not hold in Theorem 1 ? How often do they hold?**\n\nWe thank the reviewer for raising this important question, which we will clarify. Regarding Theorem 1: The function $\\phi_\\eta$ is explicitly defined in L94-95 as an \"equivalent\" loss capturing the leading-order effect of gradient descent dynamics. This is not an imposed condition but a mathematical construction derived from Taylor expansion. The only technical requirement is the boundedness of the third derivative, which is a standard smoothness condition.\n\n**Q3. What if the learning rate is not a matrix ? You said Adam corresponds to a matrix, what about other optimizers ?**\n\nFirst of all, almost all commonly used gradient-based optimizers can be interpreted as gradient descent with a matrix-valued learning rate (SGD is an identity matrix learning rate, natural gradient descent has the inverse Fisher Information as the learning rate matrix, Newton’s method has the inverse Hessian as the learning rate, SignSGD has the inverse gradient as the learning rate). Some exceptions do exist — such as coordinate descent — but those methods are less relevant for deep learning and therefore fall outside the scope of our analysis.\n\n**Q4. In Figure 2, is what you call entropy the $S(\\theta)$?**\n\nYes. Throughout our paper, the entropy $S(\\theta)$ refers to the third term in equation (3). For ReLU networks, such as those in Figure 2, $S(\\theta)$ takes the simplified form given in equation (6).\n\n**Q5. Does $\\phi_\\eta$ contain \\gamma in its definition? If so, how? If not, how did the gamma appear when computing its expectation?**\n\nAs defined in eq.(3), the entropy $S(\\theta)$ is independent of the weight decay $\\gamma$ (a technical note: if $\\gamma$ is very large, it can still appear in $S$ as a higher order term, but throughout our current work, $\\gamma$ is regarded as a small quantity that its effect on $S$ can be ignored.). The learning rate $\\eta$ and the weight decay $\\gamma$ determine the balance between the gradients and the weights in Theorem 4.\n\n**Q6. Can you say what the symmetries present (continuous and discrete) are in shallow linear network square loss for example ?**\n\nLinear networks exhibit a lot of symmetries, the most common of which is what we referred to as the double rotational symmetry, meaning that two consecutive layers can be jointly transformed by a invertible matrix without changing the network output. When the network output itself is invariant to these changes, it also implies that the loss function is invariant. Therefore, we will only discuss why and how the network output is invariant.\n\nConsider a two-layer linear network $f(U,W,x)=UWx$. We have $f(UQ,Q^{-1}W,x)= UQQ^{-1}Wx = UWx = f(U,W,x)$, for any invertible matrix Q. This is what we call the double rotational symmetry, and is what we used to derive our gradient alignment result (Theorem 7). In fact, the double rotational symmetry includes both continuous and discrete symmetries (as subgroups) in linear networks.\n\nFor example, the rescaling symmetry is a continuous symmetry: $f(\\lambda U,\\lambda^{-1}W,x)=f(U,W,x)$.\nPermutation symmetries are discrete symmetries: $(f(UP,P^{-1}W,x)=f(U,W,x)$ for any permutation matrix P. Of course, there is also the simplest type of discrete symmetry, the sign-flip symmetry (or, the $\\mathbb{Z}_2$ symmetry): $f(U,W,x)= f(-U,-W,x)$. Note that all these symmetries are present for any $x$. Also, note that the loss function is invariant to these transformations for the same reason: $||f(U,W,x) - y||^2 =||f(-U,-W,x) - y||^2 $.\n\nLastly, note that for a deeper network, different layers can have independent symmetries, and one can study them in isolation. For example, consider a three-layer linear network: $f(x) = UVW x$. There is a double rotation symmetry between $U$ and  $V$, and there is another double rotation symmetry between $V$ and $W$; our results can be applied to these two symmetries independently.\n\n**Q7. Are there some losses in practice that have A-exponential symmetry ?**\n\nYes, there exist many practical losses exhibiting A-exponential symmetry, particularly in common neural network architectures. This should already become quite clear given our answer to Q6 above, but we will name a few more examples. Our analysis in Section 4 identifies two important cases:\n\nRescaling Symmetry: In ReLU networks (Theorems 5-6), the function $f(U,W,x)=UReLU(Wx)$ satisfies $f(\\lambda U,\\lambda^{-1}W,x)=f(U,W,x)$ for any scalar $\\lambda\\neq0$. This symmetry exists in any architecture containing at least one ReLU layer, such as $f_1(UReLU(Wf_2(x)))$ where $f_1,f_2$ are arbitrary sub-networks.\n\nDouble Rotational Symmetry: As discussed in Q6, linear and self-attention layers exhibit this property between the key and query matrices (Theorem 7), where the loss remains invariant under simultaneous orthogonal transformations of certain parameter groups.\n\nThese represent concrete instances of A-exponential symmetry in practice, with the specific form depending on the network architecture's structure and activation functions, and independent of the chosen loss. Thus, any network containing at least one self-attention layer has one double rotation symmetry: If a network has 10 self-attention layers, then it also has ten independent double rotation symmetries.\n\n**Q8. How is theorem 2 saying that the continuous symmetry is broken ?**\n\nTheorem 2 states that if the loss $\\ell$ is K-invariant but K does not satisfy conditions (1) or (2), then the associated $F$ is not K-invariant. In other words, the symmetry must disappear unless K corresponds to rotational invariance.\n\n**Q9. Does the sharpness result change if we choose the largest singular value of the hessian instead of the trace?**\n\nNo, the main conclusion remains the same. We used the trace of the Hessian for analytical convenience. For example, Lemma 1 also implies that the largest singular value also diverges under the stated assumptions (because divergence of the trace of a PSD matrix happens if and only if its largest singular value diverges)."}, {"review_id": "Dmh3mXqBvm", "reviewer": "Reviewer_oJQv", "summary": "This paper introduces a theoretical framework that explains various emergent behaviors in deep learning by modeling stochasticity in the learning process caused by discrete time updates as governed by the entropic forces. The authors derived an entropic loss function for neural network learning and showed that this loss function breaks continuous symmetries while preserving discrete ones. The paper provides a theoretical explanation for various emergent phenomena in deep learning including gradient and weight alignment, universal feature structure, and sharpness seeking behaviors.", "questions": "1.\tI’m not sure I understood Definition 1. First, the weight decay gamma does not appear in the definition. Second, do you mean that for a given function K satisfying locality and consistency, the loss function satisfies the invariance condition? In that case maybe switch the order of the 3 conditions. \n2.\tThe entropic loss is initially denoted as phi but later denoted as F, make sure the notations are consistent or state explicitly otherwise.\n3.\tThe gradient balancing result is interesting. However, I do not fully understand its connection to the results with small initialization, in your case, the weight balance requires eta goes to zero, so entropic force goes to zero and the system can be viewed as continuous gradient flow, but there’s no assumption on the initialization, so I’m wondering where this difference comes from.\n4.\tIn Figure 2, the ranges of loss/entropy/imbalance are all relatively small, is this because you only select time points at later stage of training, i.e. close to equilibrium? Or is there some other reason why this is happening? \n5.\tIn earlier relevant works on deep linear networks (i.e, Hanin & Zlokapa, Li & Sompolinsky), they also discussed and calculated the hidden representations, especially the average of the inner product of the hidden layer activation, across layers after training. Their calculation would be at the equilibrium of the loss, where no entropic loss is considered, and also averaged over all possible solutions, whereas your results are about learning with SGD and discrete dynamics without averaging over the solution space. While these two frameworks are very different, I’m wondering whether some results can be connected, i.e, can your results reduce to theirs in certain limits? Does the representation in your case also have the inner product aligning with the target output?\n6.\tIn Figure 3, did you experiment with different types of nonlinearity to see how sensitive the alignment is? For example, how would alignment for a ReLU network look like? \n7.\tSome typo in Figure 4, I suppose these are mlp/attention layers and self/other alignment respectively but the axis labels are all the same. \n8.\tIn section 4, does the balance result depend on the MSE loss? If not, why would you do a student-teacher setup with MSE loss on MNIST, it would seem more natural to simply train the original MSE dataset with CE loss?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The perspective of using an entropic loss to understand stochasticity in the learning dynamics is quite interesting and novel. The paper then proceeds with rigorous theoretical proofs to demonstrate interesting implications of this perspective. Using this entropic loss, the paper shows that the loss breaks all continuous symmetry and preserves discrete ones, this is in stark contrast with previous works that focus on the equilibrium of continuous gradient flow without incorporating the stochasticity induced by discrete time dynamics. The framework also provides explanations for previously observed behaviors including weight and gradient alignment, universal feature structure and sharpness seeking. All results are supported with sufficient numerical experiments. \nThis paper not only provides a new insights on understanding learning dynamics in discrete time with mathematical rigor, but also discusses thoroughly the potential implications of the theoretical results and their connections to widely observed phenomenon. Therefore, I believe this paper is a valuable addition to theoretical understanding of ML and is of interest to a broad audience. \n\nWeaknesses: \nWhile the paper shows empirical evidence in realistic settings. Many of the main theorems (8-10) are established for deep linear networks, or particular types of settings (theorems 5-6) I think it is still insightful as long as the theoretical results can predict behaviors in more complicated settings, as the authors show, but it would be nice to have a summary of results or discussions which part of the results are applied to which types of networks/loss etc.", "was_revised": true, "final_justification": "My questions regarding connections to relevant works, clarifying the assumptions of the theorems and better justify the setups of some of the experiments have been resolved by the rebuttal. I keep my score unchanged, this paper provides valuable insights in understanding the learning dynamics of neural networks in discrete time and is a valuable addition to the field of theoretical ml.", "rebuttal": "Thank you for your positive feedback and the detailed questions. We answer both the weaknesses and questions below.\n\n**Q1.While the paper shows empirical evidence in realistic settings. Many of the main theorems (8-10) are established for deep linear networks, or particular types of settings (theorems 5-6) I think it is still insightful as long as the theoretical results can predict behaviors in more complicated settings, as the authors show, but it would be nice to have a summary of results or discussions which part of the results are applied to which types of networks/loss etc.**\n\nThank you for the suggestion. Having a table to summarize the applicability of these results would be great. Theorems 1–4 are general and do not depend on specific network architectures or loss functions. Theorems 5–6 apply to arbitrary ReLU layers, Theorem 7 applies to linear or self-attention layers, and Theorems 8–10 concern deep linear networks trained with MSE loss. We will add a summary table or paragraph to clarify these conditions in the main text.\n\n**Q2. I’m not sure I understood Definition 1. First, the weight decay gamma does not appear in the definition. Second, do you mean that for a given function K satisfying locality and consistency, the loss function satisfies the invariance condition? In that case maybe switch the order of the 3 conditions.**\n\nWe appreciate your thoughtful comments. You are right on both points. The weight decay parameter $\\gamma$ was inadvertently included in Definition 1 and will be removed. Indeed, the invariance condition of the loss function holds when the function K satisfies both locality and consistency. As suggested, we will reorder the conditions to first state locality and consistency for K, then the invariance property. This restructuring will better reflect the logical flow of our definition. Thank you for catching these important points.\n\n\n**Q3. The entropic loss is initially denoted as phi but later denoted as F, make sure the notations are consistent or state explicitly otherwise.**\n\nThank you for catching this. The entropic loss F is defined as the expectation of $\\phi$ (lines 97–98), so they are related but not identical. \n\n**Q4.The gradient balancing result is interesting. However, I do not fully understand its connection to the results with small initialization, in your case, the weight balance requires eta goes to zero, so entropic force goes to zero and the system can be viewed as continuous gradient flow, but there’s no assumption on the initialization, so I’m wondering where this difference comes from.**\n\nThis is an insightful observation. The key point is that Theorem 4 does not rely on any assumptions about initialization or learning rate. It describes the structure of a local minimum of equation (3), arising purely from symmetry and weight decay — independent of optimization dynamics or initialization scale. \n\nWhile the entropic force vanishes as $\\eta \\to 0$, the weight balance induced by symmetry remains nontrivial, as discussed in lines 155–158. This is because the weight balance is due to weight decay, and already encourages the weights to be small (thus, sometimes leading to a similar effect to a small init.).\n\n**Q5.In Figure 2, the ranges of loss/entropy/imbalance are all relatively small, is this because you only select time points at later stage of training, i.e. close to equilibrium? Or is there some other reason why this is happening?**\n\nThe small ranges in loss, entropy, and imbalance are task-specific. In fact, the absolute values of them are not quite meaningful. For this task, we have a small initial imbalance (blue points). During training, entropy and imbalance decrease further. The plot includes the entire training trajectory, not just the equilibrium phase.\n\n**Q6.In earlier relevant works on deep linear networks (i.e, Hanin & Zlokapa, Li & Sompolinsky), they also discussed and calculated the hidden representations, especially the average of the inner product of the hidden layer activation, across layers after training. Their calculation would be at the equilibrium of the loss, where no entropic loss is considered, and also averaged over all possible solutions, whereas your results are about learning with SGD and discrete dynamics without averaging over the solution space. While these two frameworks are very different, I’m wondering whether some results can be connected, i.e, can your results reduce to theirs in certain limits? Does the representation in your case also have the inner product aligning with the target output?**\n\nIndeed, our settings are quite different. Both Hanin & Zlokapa, Li & Sompolinsky consider the Bayesian setting, where the noise is thermal and equivalently generated by an isotropic Gaussian process. Our work directly deals with the actual SGD noise coming from the minibatch sampling. Many predictions of the two theories are fundamentally and qualitatively different, for example, the Bayesian type theory (such as Hanin & Zlokapa, Li & Sompolinsky) predicts a nonvanishing stationary probability measure everywhere, but the actual trajectories of SGD typically occupy a small subset of the entire space, because the entropic contribution is not negligible even at small \\eta and \\gamma. \n\nThere are a few aspects that the two types of theories do agree on. For example, as shown in the proof of Theorem 8, the learned representation h(x) satisfies $h(x)^T h(x) \\propto (Vx)^T (Vx)$, indicating alignment with the target output. Exploring deeper connections with prior equilibrium analyses is an interesting future direction.\n\n\n**Q7.In Figure 3, did you experiment with different types of nonlinearity to see how sensitive the alignment is? For example, how would alignment for a ReLU network look like?**\n\nYes, we experimented with ReLU networks and found that the alignment pattern is nearly identical to that shown in the left panel of Figure 3. We used tanh purely for illustration, and will clarify this in the caption.\n\n**Q8.Some typo in Figure 4, I suppose these are mlp/attention layers and self/other alignment respectively but the axis labels are all the same.**\n\nThank you for pointing this out. The first and third panels in Figure 4 are self-attention layers. We will correct the labels accordingly in the revision.\n\n**Q9.In section 4, does the balance result depend on the MSE loss? If not, why would you do a student-teacher setup with MSE loss on MNIST, it would seem more natural to simply train the original MSE dataset with CE loss?**\n\nThanks for the comment. Both settings are fine, but we wanted to be consistent with other experiments and chose to have a more consistent setting with other experiments. We will also include the more natural setting in the final revision."}, {"review_id": "kISEC14dnX", "reviewer": "Reviewer_zJhT", "summary": "This paper introduces a new theory for understanding the stochasticity and discretization effects arising from stochastic gradient descent (SGD). A key idea is approximating a single update of the original loss with n steps of updates of a loss, named entropic loss. When  $n \\rightarrow \\infty$, the discrete update is converted to a continuous update. The discretization error and noise effects are further decomposed from the entropic loss and termed effective entropy, and the gradient of the effective entropy, named entropic forces, is shown to break almost any continuous parameter symmetries while preserving discrete ones. As two example applications, the entropic forces are used to explain two observations: the progressive sharpening of the loss landscape and the emergence of universal representations", "questions": "- The entropic force appears to result from both stochasticity and discretization effects. Is it possible to analyze these two effects independently, or do they interact in a way that complicates their separation?\n- When $n \\rightarrow \\infty$, the discrete update transitions to a continuous update for the effective entropy. Apart from the ability to use Lagrangian formalism, are there other practical advantages of this continuous update?\n- In Figure 1, should the pink arrows be red instead?", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths And Weaknesses:\n\nI have limited knowledge in this specific area and the overall manuscript appears satisfactory to me. \n\nQuality:\n\n- [+] The theoretical demonstrations are clear and seems technically sound.\n\nClarity:\n\n- [+] The presentation of the idea is clear.\n\nSignificance& Originality:\n\n- [+] I think this work is novel and can be a contribution to the Machine learning community.\n- [+] New theory is introduced for understanding the learning dynamics.", "was_revised": true, "final_justification": "My concerns are addressed and I will keep my original score", "rebuttal": "Thank you for your positive feedback and the detailed questions. We answer both the weaknesses and questions below.\n\n**Q1. The entropic force appears to result from both stochasticity and discretization effects. Is it possible to analyze these two effects independently, or do they interact in a way that complicates their separation?**\n\nThank you for the insightful question. Yes, these two effects can be separated. The entropy term can be written as \n$$\\eta\\mathbb{E}_\\mathcal{B}||\\mathbb{E}\\_{x\\in\\mathcal{B}}\\nabla \\ell(x,\\theta)  ||^2=\\eta[\\mathbb{E}\\_x\\nabla \\ell(x,\\theta)]^2+\\eta Var\\_\\mathcal{B}(\\mathbb{E}\\_{x\\in\\mathcal{B}}  \\nabla \\ell(x,\\theta))$$\n\nThe first term corresponds to the discretization effect with strength \\eta, and the second term corresponds to the stochastic effect with strength $\\eta/|\\mathcal{B}|$. Thus, we can analyze each term separately. \n\n**Q2. When $n\\to\\infty$, the discrete update transitions to a continuous update for the effective entropy. Apart from the ability to use Lagrangian formalism, are there other practical advantages of this continuous update?**\n\nIn our formalism, the continuous-time formalism serves primarily as a theoretical tool for understanding \nSGD training. For example, [16] uses it to argue that directly optimizing the effective loss via gradient descent does reproduce the same behavior as SGD. From a practical standpoint, however, SGD remains more effective and easier to implement than such continuous dynamics.\n\n**Q3. In Figure 1, should the pink arrows be red instead?**\n\nThank you for catching that — yes, the pink arrows should be red. We will correct this in the revised version."}]

Author rebuttal:

Meta-review: This paper develops a framework for understanding learning dynamics in deep networks, with connections to symmetry breaking in representation learning, loss landscape sharpness, and emergence of universal representations.  After discussion with authors, all reviewers lean toward accept.  There is a consensus that the paper is technically solid and includes sufficient experimental evidence in real settings.  However, theorems involve strong assumptions and are for simpler settings or toy models (e.g., deep linear networks).  The Area Chair agrees with the reviewer consensus that the overall contribution justifies recommending acceptance.
Decision: Accept (poster)