Title: Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: entropic loss

symmetry breaking

universal representation learning

physics
Abstract: With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its variants. Building on the theory of parameter symmetries and an entropic loss landscape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading to a series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, (a) explain the universal alignment of neural representations between AI models and lead to a proof of the Platonic Representation Hypothesis, and (b) reconcile the seemingly contradictory observations of sharpness- and flatness-seeking behavior of deep learning optimization. Our theory and experiments demonstrate that a combination of entropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.

Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning

Abstract

With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its variants. Building on the theory of parameter symmetries and an entropic loss landscape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading to a series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, (a) explain the universal alignment of neural representations between AI models and lead to a proof of the Platonic Representation Hypothesis, and (b) reconcile the seemingly contradictory observations of sharpness- and flatness-seeking behavior of deep learning optimization. Our theory and experiments demonstrate that a combination of entropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.

1 Introduction

Modern neural networks trained with stochastic gradient descent (SGD) exhibit a complex plethora of emergent behaviors - emergence of capabilities [1, 2, 3], progressive sharpening and flattening [4, 5], phase-transition like behaviors [6, 7], and universal representational alignment across models [8, 9, 10, 11] - that are difficult to explain through loss minimization alone. These behaviors mirror phenomena found in physical systems at finite temperature, suggesting that deep learning dynamics are shaped not just by explicit optimization but also by implicit forces arising from stochasticity and discrete updates. These implicit forces have long been associated with the phenomenon of 'implicit bias' in deep learning [12, 13, 14, 15], but their precise mathematical nature remains elusive. In physics, such effects are often captured by entropic forces -macroscopic forces that emerge from the system's statistical tendencies rather than its energy landscape alone [16]. The power of this framework lies in the notion of an effective entropy , which plays the role of a potential whose gradients define the entropic force. Identifying this effective entropy not only reveals what the system is implicitly optimizing, but also opens the door to leveraging theoretical tools from statistical physics to analyze and improve AI models.

Contributions. We formalize this connection between stochastic learning dynamics and entropic forces through the lens of symmetry and representation learning to:

39th Conference on Neural Information Processing Systems (NeurIPS 2025).

Derive an entropic loss function and show that the entropic forces of SGD break continuous parameter symmetries while preserving discrete ones (Section 3).

Show that the symmetry breaking due to entropic forces gives rise to a family of equipartition theorems that predict the gradient alignment phenomena (Section 4).

Explain and unify two seemingly disparate but universal observations progressive sharpening of the loss landscape and the emergence of universal representations - as consequences of entropic forces (Section 5).

Our theory establishes a principled framework - akin to a thermodynamics of deep learning - that unifies several universal phenomena under a single formalism. The results suggest that the entropic loss landscape, shaped by both optimization and entropy, plays a foundational role in understanding learning dynamics and emergent phenomena. Full derivations and experimental validations are provided in the appendix.

2 Related Work

Modified Loss and Effective Landscape. The concept of modified or effective losses has emerged as a critical framework for understanding the implicit biases induced by stochastic gradient descent (SGD) in deep learning, which differs from another line of work [17, 18, 19, 20, 21] which leverages the property of stationarity to analyze the stationary distribution of SGD. Ref. [22] introduced the notion of a modified loss to analyze the discrete-time dynamics of SGD, demonstrating how discretization implicitly alters the optimization landscape. Similarly, Refs. [23] and [24] extended the modified loss formulation to where there is a gradient noise due to minibatch sampling. These works conducted numerical simulations to show that training on the effective loss really approximates the original dynamics [23, 25, 26] and leads to similar generalization performances. In this work, we refer to this type of losses as entropic losses for their associations with theoretical physics. Still, these entropic losses remain poorly understood, and their significance for understanding emergent phenomena in deep learning is not yet appreciated. Our work finds the crucial link between the entropic loss and symmetry-breaking dynamics, which is important for understanding the various intriguing nonlinear phenomena of representation learning.

Parameter Symmetry in Neural Networks. Parameter symmetries are shown to play a fundamental role in shaping neural network training dynamics and their emergent properties [27, 28, 29, 7, 30, 31]. A series of works showed that continuous symmetries in the loss function give rise to conservation laws, which imply that the learning result of SGD training is strongly initializationdependent [32, 28, 33]. More recent works showed how any stochasticity or discretization effect could break the symmetries in a systematic way such that the learned solution is no longer dependent on the initialization, a hint of universality [34, 29, 7, 31]. Particularly, Ref. [29] developed the formalism of exponential symmetries and proved that any loss function with an exponential symmetry leads to a symmetry-breaking dynamics that converges to unique fixed points. This point can be seen as the dynamical equivalence of our Theorem 2, which states that there is essentially no continuous symmetry in the entropic loss. In comparison, our framework takes a different perspective: we study the symmetry from a loss landscape perspective and identify these symmetry-breaking tendencies as entropic forces. This unified perspective enables us to understand the universal learning phenomena with a greatly simplified analysis.

3 Effective Energy for Stochastic Gradient Learning

Define ℓ ( x, θ ) to be the per-sample loss function. We can define the empirical risk as

where ℓ γ ( x, θ ) ∶= ℓ ( x, θ ) + γ ∣∣ θ ∣∣ 2 , B represents the minibatch and γ represents the weight decay. From a dynamical-system perspective, for an infinitesimal learning rate, the loss function coincides with the Bregman Lagrangian of this dynamics, and so one can leverage the Lagrangian formalism to understand the training of gradient flow [35]. This is particularly attractive from a theory perspective because modern theoretical physics are also founded on the Lagrangian formalism and this connection allows one to borrow physics intuitions to understand deep learning.

However, simply studying this loss function is insufficient to understand the learning dynamics of SGDat various learning rates η due to the stochastic discrete-time nature of SGD. This motivates the definition of an entropic loss ϕ η such that running n steps of update on ϕ η ( θ ) with learning rate η / n

is the same as running one step of update on ℓ γ for any x . Taking the limit n →∞ , one can obtain a 'renormalized' loss function for which running gradient flow is the same as running gradient descent for the original loss. With this loss, it becomes possible again to leverage Lagrangian formalism to understand SGD training with discrete-time and stochastic learning. Because ϕ 0 coincides with running gradient flow on ℓ γ , one must have that

We can also consider the more general case where the learning rate is a fixed symmetric matrix Λ with ∣∣ Λ ∣∣ = η . The following theorem derives the entropic loss for this case. Many common algorithms, such as Adam, natural gradient descent, and even a wide range of biologically plausible learning rules [36] can be seen as having matrix learning rate.

Theorem 1. (Entropic Loss) For fixed x , starting from θ 0 run one-step gradient descent with Λ on ℓ γ ( x, θ ) to obtain θ 1 . Run n -step gradient descent with Λ / n on ϕ Λ ( x, θ ) ∶= ℓ γ ( x, θ ) + ϕ 1Λ ( x, θ ) + ϕ 2Λ ( x, θ ) to obtain θ ′ n . Then, assuming ∣∣∇ 3 ℓ γ ( x, θ )∣∣ ≤ M ,

if ϕ 1Λ ( x, θ ) = 1 4 ∇ ℓ γ ( x, θ ) T Λ ∇ ℓ γ ( x, θ ) , then, θ ′ n = θ 1 + O (∣∣ Λ ∣∣ 3 +∣∣ Λ ∣∣ 2 / n ) ;

moreover, if ϕ 2Λ ( x, θ ) = 1 2 ∇ ℓ γ ( x, θ ) T Λ ∇ 2 ℓ ( x, θ ) Λ ∇ ℓ γ ( x, θ ) , then θ ′ n = θ 1 + O (∣∣ Λ ∣∣ 4 +∣∣ Λ ∣∣ 2 / n + ∣∣ Λ ∣∣ 3 / n +∣∣ Λ ∣∣ 3 M ) .

This ϕ η also needs to hold in expectation with respect to the sampling of data points, and so one can define the expected entropic loss F η,γ ( θ ) = E [ ϕ η ( x, θ )] , which is, up to the first order in Λ and γ :

effective entropy due to discretization error, noise, ∶= S ( θ )

Specializing to the first order and with a scalar learning rate, this equation reduces to the 'modified loss' previously derived in different contexts [22, 23, 24]. The derivation has a thermodynamic flavor as it essentially computes the degree to which the dynamics is irreversible, and the entropy term is the part that cannot be microscopically reversed. In Theorem 1, the first-order term in Λ encourages the model to have a small gradient fluctuation. The second-order term, on top of gradient regularization, encourages the model to move to flatter solutions; this term has been found to play a role in the edge of stability phenomenon [5]. However, since the first-order term is not yet well-understood, our work focuses on the first-order term in Λ . We also focus on a scalar learning rate Λ = ηI and will comment on the differences when the difference is essential.

Now, treating the original loss plus regularization as an energy, the dynamics of gradient flow on F contains an energy force and and an entropic force: ˙ θ = -η (∇ L + γθ + ∇ S ) . The ∇ S term, the gradient of the effective entropy, will be called the 'entropic force.' See Figure 1 for an illustration of the entropic force and an example of how entropy evolves during training.

We prove that the entropic force term breaks almost any continuous symmetry of L , a key result that we will leverage to study progressive sharpening and universal representation learning.

Definition 1. A loss function ℓ ( x, θ ) is said to be K -invariant if items 1-3 are satisfied:

locality: K ( θ, λ ) = θ + λQ ( θ ) + O ( λ 2 ) for a differentiable Q ;

consistency: K ( K ( θ, λ ) , λ ′ ) = K ( θ, λ + λ ′ ) ;

invariance: ℓ ( x, K ( θ, λ )) = ℓ ( x, θ ) for all x , θ and λ ∈ R .

An entropic loss F η,γ is said to have the ( robust ) K -invariance if there exists a neighborhood around η, γ such that F η,γ is K -invariant.

Theorem 2. (Symmetry Breaking Under the Entropic Loss) Let ℓ be K -invariant. If F is also robustly K -invariant, then, (1) ∥ K ( θ, λ )∥ = ∥ θ ∥ and (2) ∇ T ℓ ∇ Q ( θ )∇ ℓ = 0 for all θ .

This means that any symmetry or invariance that F η,γ has must be norm-preserving transformations. Essentially, this means that any invariance that is not rotation invariance must disappear. The following theorem shows that orthogonal discrete symmetries are preserved.

Theorem 3. Let OO T = I . If ℓ ( x, Oθ ) = ℓ ( x, θ ) for any θ and x , then F η,γ ( Oθ ) = F η,γ ( θ ) for any θ .

Learningatlarge and

smalllearningrate

102

101

0(n²)

(u)o

DSS

100

train

10-1

test

0(n²)

0(n)

entropy

△θ =-nVL

△0'：

=

n

2

VL

△0'：

=

n

2

VF

7△-△

10-2

0

50

100

150

200

Originaldynamics

Originaldynamics

Effectivedynamics

EntropicForce

Together with the previous result, this shows that when gradient noise or regularization is taken into account, the only relevant remaining symmetries are discrete symmetries. This implies that the results that are based on conservation laws for understanding SGD are questionable and can only hold in the toy setting of an infinitesimal learning rate. The reason is simple: F η,γ does not have robust invariances at η = γ = 0 . Also, note that had we used a generic matrix learning rate (e.g., with Adam), the orthogonal invariances would also be broken. The meaning of these discrete symmetries can be understood through a framework similar to that proposed in Ref. [7] and is left to a future work.

4 Emergence of Gradient Balance and Equipartition Property

Lie group symmetries exist abundantly in nature and in modern neural networks 1 [27, 28, 29, 7, 32]. In thermodynamics, the existence of symmetries is a crucial fact that leads to the emergence of hierarchical phenomena and phase transitions between them. In a sense, symmetry can be argued to be the 'first-order' approximation of the level of hierarchies in the system [37]. The following theorem states that the entropic loss F breaks any nontrivial noncompact Lie group symmetries (also known as exponential symmetries [29]). For formality, we say that ℓ (and L ) has a A -exponential symmetry if for any λ ∈ R , any x and θ , and any matrix A , ℓ ( x, θ ) obeys ℓ ( x, e λA θ ) = ℓ ( x, θ ) .

Theorem 4. (Master Balance Theorem) If ℓ ( x, θ ) has an A -exponential symmetry, then any local minimum θ ∗ of Eq. (3) satisfies

where ˜ A ∶= A + A T 2 . In addition, either (1) F η,γ ( e λA θ ∗ ) = F η,γ ( θ ∗ ) for all λ , or (2) there exists no λ ≠ 0 such that e λA θ ∗ is a local minimum.

If we had chosen a symmetric A , we would have ˜ A = A . For an anti-symmetric A , this theorem is trivial, consistent with Theorem 2. Therefore, this theorem is a statement about non-compact Lie group symmetries that extends to infinity. The fact that every point is connected to a point that satisfies Eq (4) means that SGD can reach this condition easily. The meaning of this theorem is a general gradient balance and alignment phenomenon. When γ = 0 , this equation states that the gradient along the positive spectrum of ˜ A must balance with the gradient along the negative spectrum. When γ ≠ 0 , there is, additionally, a tradeoff between gradient balance and weight balance. We apply this result to various neural networks in this section.

Many works have shown that when training with weight decay or when the model has a small initialization, the weights of the layers become balanced, especially in homogeneous networks [38,

39, 40]. Theorem 4 shows that the SGD in discrete-time or with stochasticity leads to a completely different bias where the gradient noise between all layers must be balanced.

ReLU Layers Consider a deep ReLU network trained on an arbitrary task:

where R ( x ) is a piece-wise constant the zero-one activation matrix functioning as the ReLU activation. The entropy term can be written as

where g i = E x ∈B ∇ W i ℓ ( θ, x ) and E is a shorthand of E B . Namely, we can group the gradient covariance according to layer index i . The following theorem states that all layers must have a balanced gradient. The proof shows that while the learning term L is invariant to a class of symmetry transformations, the entropy term is not - and this creates a systematic tendency for the parameters to reduce the entropy.

Theorem 5. (Layer Balance) For all local minimum of Eq. (3) ,

Specifically, for γ = 0 we have gradient balance E Tr [ g i g ⊺ i ] = E Tr [ g j g ⊺ j ] . For η = 0 , we have the standard weight balance Tr [ W i W T i ] = Tr [ W j W T j ] . Otherwise, the solution interpolates between gradient balance and weight balance.

Similarly, within every two neighboring layers, one can group the parameters into neurons:

where g i,j, ∶ is the incoming weights to the j -the neuron of the i -th layer, and g i + 1 , ∶ ,j is the outgoing weights of the same neuron. The gradient for each neuron must also be balanced because there is a rescaling symmetry in every neuron.

Theorem 6. (Neuron Balance) For all local minimum of Eq. (3) and any i, j ,

From a physics perspective, we have proved an equipartition theorem (ET). The elements in the matrix E [ g i g T i ] can be seen as the temperature (or, the average energy) felt by each parameter. The trace Tr E [ g i g T i ] thus gives the temperature of the layer. That different layers emerge to have the gradient second momentum is an explicit form of the ET and means that the entropy S must be evenly spread out across every layer. Because the standard ET in physics is a property of thermal equilibrium, our result may be seen as an extension of the physical law to the out-of-equilibrium dynamics of learning. See Figure 2 for the emergence of layer and neuron balances in a ReLU network. We train on the MNIST dataset, but the labels are generated by a teacher ReLU network and trained with an MSE loss. Also, see Appendix A for examples of training trajectories and for an example with a self-attention net.

Polynomial Network Now, consider the case where R ( h ) is a diagonal matrix such that R ii ( h ) = h d corresponds to a polynomial activation. This type of network is also a variant of homogeneous networks [38]. For these networks, one can show that the converge to a state where every layer's gradient norm is d times that of the previous layer, leading to a gradient exploding or vanishing problem (See Appendix B.8).

Self Attention Consider the case when a model has a generic form: ℓ ( x, W, U ) = ℓ ( x, WU ) , where W and U are matrices. The loss can contain other trainable parameters, which we ignore. Define G W = E x ∈B ∇ W ℓ ( x, W, U ) , G U = E x ∈B ∇ U ℓ ( x, W, U ) , and one can prove the following relation:

Theorem 7. (Gradient Alignment) For all local minimum of Eq. (3) , we have

le-6

le-6

le-4

1e-4

neuronimbalance

5

neuronimbalance

5

layerimbalance

layerimbalance

4

4

8

8

3

3

6

6

2

2

4

4

2

4.0

4.5

2

4.0

4.5

entropy

1e-3

loss

le-2

entropy

le-3

loss

le-2

This theorem is thus applicable to matrix factorization, deep linear networks, and, more importantly, self-attention layers. The self-attention logit is computed as a ij = X T i WUX j , where W is the key matrix, U is the query matrix. The loss function is a function of a ij viewed as a matrix: ℓ ({ a ij }) . Let V = W 2 W 1 . Then, applying this theorem reveals an intriguing relation:

Interpolating Weight Balance and Gradient Balance For all the theorems above, we have also studied how weight decay affects the balance conditions. We see that the weight decay creates something analogous: instead of gradient balance, weight decay encourages weight balance, and this effect often cannot be achieved together with gradient balance. Thus, there is a tradeoff between gradient balance and weight balance. In reality, the network is somewhere in between, where the weight balance and gradient balance has to 'balance' with each other. Also, this is a generalized form of an equipartition theorem. If we regard the sum of the regularization and S as a 'total' entropic potential Γ , then this means that every layer will contribute an equal amount to Γ . 2

5 Implications

Next, we apply these results to study the emergence of universal representations in neural networks, and the progressive sharpening phenomena in deep learning optimization.

5.1 Universal Representation Learning

Recent works found that the representations of learned models are almost universally aligned to different models trained on similar datasets [41, 42], and even to the biological brains [43]. This interesting phenomenon has a rather philosophical undertone and has been termed 'Platonic Representation Hypothesis' [8]. Here, we say that the two neural networks have learned a universal representation if for all x 1 , x 2 ,

where h A is the activation of network A in one of the hidden layers, and h B for network B. This is an idealization of what people have observed - and the difference between the two sides is the 'degree of alignment.' We leverage the entropic force formalism to identify an exact solution to the embedded deep linear network (EDLN) model:

on datasets D M 3 = {( M 3 x i , y i )} i , where M 1 , M 2 , M 3 are fixed but arbitrary invertible matrices. They have the following meaning:

M 1 can be seen as a model of the layers coming after the deep linear network;

1.0

1.0

0.9

0.9

alignment

0.8

samelayer

alignment

0.8

crosslayer

5333

0.7

0.7

input

samelayer

0.6

0.6

crosslayer

0.5

0.5

-input

0

500

1000

1500

2000

0

500

1000

1500

2000

iteration

iteration

20

0.8

layer

Net B mlp layer

NetA Attention

NetBAttention

0.6

15

Net A mlp I

0.4

10

0.2

5

0.0

0

-0.2

0

10

20

0

10

20

0

10

20

0

10

20

Net A mlp layer

Net A mlp layer

NetAAttention

NetAAttention

M 2 models the layers coming after the embedded network;

M 3 models different views of the data, which is common in multimodal learning - therefore, the two models are trained on two different (but related) datasets.

In Theorem 8, we will train two different models, each with their own and potentially different M 1 , M 2 , M 3 . The arbitrariness of these three matrices implies universality .

The data is generated by y i = V x i + ϵ i for i.i.d. noise ϵ i . Assuming that E x i = E ϵ i = 0 and Σ ϵ ∶= E ϵ i ϵ T i , Σ x ∶= E x i x T i , the following theorem characterizes the global minimum of the entropic loss for this network in the case γ = 0 and η → 0 + .

Theorem 8 (Perfect Platonic Representation Hypothesis) . Consider two deep linear networks A and B with weights of arbitrary dimensions larger than rank ( √ Σ ϵ V √ Σ x ) . Let model A train on D M 3 and model B on D M ′ 3 . Moreover, the outputs of models are multiplied by M 1 , M ′ 1 , and the inputs are multiplied by M 2 , M ′ 2 , respectively. Then, at the global minimum of Eq. (3) , every hidden layer of A is perfectly aligned with every hidden layer of B for any x , in the sense that

for 1 ≤ L A < D A and 1 ≤ L B < D B and any x , where c 0 is a scalar and R = U 1 U T 2 satisfying U T 1 U 1 = U T 2 U 2 = I . h L A A ( x ) ∶= Π L A i = 1 W A i M 2 M 3 x , h L A A ( x ) ∶= Π L A i = 1 W A i M ′ 2 M ′ 3 x denote the output of the L A , L B -the layer of network A and B, respectively.

Here, a perfect alignment means that h L A A ( x ) differs from h L B B ( x ) only by a scaling and a rotation. Because of symmetry, SGD converges to a state where all possible pairs of the intermediate layers of two different networks are mutually aligned, independent of the initialization. This is an extraordinary fact because there exist infinitely many solutions that are not perfectly aligned, also due to symmetry. For example, take h B to be any hidden layer, and transform its incoming weight by A and its outgoing weight by A -1 . This remains a global minimum for L , but it is no longer the case that there exists an orthogonal transformation R such that h A = h B . Therefore, for almost all global minima of L , there is no universal alignment between layers - yet, SGD prefers a universal solution due to the entropy term. The following theorem shows that weight decay will lead to a nonuniversal representation.

From a physics and thermodynamics perspective, it is quite reasonable that the irreversibility of the dynamics leads to the emergence of universal structures. A state is not universal if it contains information about its initial condition. Therefore, the irreversibility of the learning dynamics helps erase information about the parameter initialization, thereby enabling the learning of a universal solution. The following hypothesis can summarize this perspective:

Irreversibility enables universal representation learning.

Now, it is worthwhile to remark on the connection and difference between this result and the original PRH [8]. First, our theory provides strong support for the PRH. The original PRH only hypothesizes a positive similarity between models, and it is unclear whether the alignment score can reach 1 (which implies a perfect alignment) or will only be a small positive value. Our result shows that, in principle, it is possible to reach the perfect alignment limit. Secondly, this result also offers an alternative perspective on the representation alignment phenomenon to that of the original PRH paper. In the old perspective, one regards having no alignment as the default expectation, and positive alignment as something to be explained and understood. In our new perspective, the perfect alignment is the default, and breaking away from it is something to be explained and understood - which is exactly what we will demonstrate in the next theorem. We also refer to Ref. [44] for a more instructive derivation of the proof and for a detailed discussion of different ways to break the perfect PRH.

Theorem 9. Consider the deep linear network (13) with widths larger than d ∶= rank ( V ) . Let η = 0 and γ → 0 + . At the global minimum of (3) , we have

for i = 1 , /uni22EF , D . U D and U 0 are given by the SVD of M -1 1 V M -1 3 M -1 2 ∶= U D SU 0 , where S ∈ R d × d contains the singular values. For i = 2 , /uni22EF , D -1 , U i are arbitrary matrices satisfying U T i U i = I d × d . Moreover, Σ = S 1 / D . { P i } D i = 1 ⊂ R d × d are diagonal matrices containing ± 1 and satisfy Π D i = 1 P i = I d .

The universal representation property (Theorem 8) does not hold anymore. Therefore, gradient balances lead to universal representations, whereas weight balances do not. See Figure 3 for an experiment with deep linear and nonlinear networks. A surprising aspect is that every layer can be aligned with every other layer. Because any deep nonlinear network is approximated by a linear network for a small weight norm [45], one could say that any nonlinear network is, to first order in ∥ θ ∥ , a universal representation learner. In Figure 4, we compare the alignment between different self-attention layers of two differently sized vision transformers pretrained on ImageNet. Note that different layers of the same network and different networks also have significantly positive alignments, consistent with the solution for the embedded deep linear net. That using weight decay does not lead to universal representations is supported by the result in Figure 10.

Theorem 8 is a direct (perhaps the first) proof of the Platonic representation hypothesis, implying that for any x 1 , x 2 , Eq. (12) holds. Importantly, the mechanism does not belong to any previously conjectured mechanisms (capacity, simplicity, multitasking [8]). This example has nothing to do with multitasking. The result holds for any deep linear network, all having the same capacity and the same level of simplicity, because all solutions parametrize the same input-output map. Here, the cause of the universal representation is symmetry alone: in the degenerate manifold of solutions, the training algorithm prefers a particular and universal one. This example showcases how symmetry is indeed an overlooked fundamental mechanism in deep learning.

5.2 The Sharpness Paradox

The tendency towards learning universal representations can imply a curse for training. For example, the sharpness of the loss landscape really depends on the distribution of the data, whereas the solution the model finds is independent of these distributions - this could imply that these solutions can be quite bad in terms of, say, optimization properties. Meanwhile, a paradox of the sharpnessseeking behavior of SGD has become explicit. On the one hand, the edge of stability (EOS) states that learning typically leads to sharper solutions, whereas a vast majority of works have shown that SGD training leads to flatter solutions [46, 47]. These two cannot happen simultaneously - and the solution must be that the sharpness-seeking behavior of SGD is situation-dependent. This section formalizes this intuition and shows that universal representation learning can be intrinsically related to the edge of stability phenomenon, which is also ubiquitous in deep learning.

0.40

2.0

0.40

0.35

theory

1.8

0.35

theory

2.2

edgeofstability

b

ance

0.20

0.30

0.25

0.15

0.10

1.6

1.0

1.4

1.2

b

=

0.30

0.25

0.20

0.15

0.10

M

xew

1.6

2.0

1.4

1.8

0.05

0.8

0.05

1.2

0.6

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0.0

0.1

0.2

0.3

learningrate

learningrate

databalance

3.0

3.0

2.6

2.6

sharpness

sharpness

sharpness

S

2.8

sharpness

2.8

2.4

2.4

2.6

2.6

2.2

2.2

2.4

2.4

6

Q

4.0

4.5

5.0

0.043

0.045

4.5

5.0

0.0190.0200.021

entropy

loss

entropy

loss

Definition 2. The total sharpness is defined as T ( θ ) = Tr E ∇ 2 ℓ ( x, θ ) .

This definition is chosen for analytical tractability and has been used in prior works [48, 29]. T upper bounds the largest eigenvalue, and T / d lower bounds it, so it is a good metric of stability and sufficient for the theorem we will prove. The following lemma shows that if there is an exponential symmetry in ℓ , every local minimum of ℓ connects without barrier to a local minimum with arbitrarily high sharpness. It can be seen as a generalization of the result of Ref. [49] to general symmetries.

Lemma 1. (Sharpness Lemma) Assume that A is a symmetric matrix and ℓ ( x, e λA θ ) = ℓ ( x, θ ) . Moreover, assume that A E ∇ 2 ℓ ( x, θ ) ≠ 0 . Then, limsup ∣ λ ∣→+∞ ∣ T ( e λA θ )∣ = +∞ .

One can analytically solve for the sharpness of two-layer linear networks, and identify a precise cause of the progressive sharpening effect.

Theorem 10. For a two-layer linear network ℓ ( x, θ ) = ∣∣ y ( x )-W 1 W 2 x ∣∣ 2 with y ( x ) = V x + ϵ ∈ R d y and E x = E ϵ = 0 , Σ x = E xx T , Σ ϵ = E ϵϵ T . Denote ˜ US ′ ˜ V to be the SVD of V ′ ∶= √ Σ ϵ V √ Σ x and assume that the width of the network is larger than rank ( V ′ ) . Then we have

at the global optimum of (3) . Meanwhile, the minimal sharpness of the global minimum of ℓ is

where ˆ S is the singular values of V √ Σ x .

This result implies that SGD has no inherent preference for flatter minima. See Figure 5-6, where we train a two-layer linear network on a linear regression task with a 2d label y ∈ R 2 . The labels y = V ∗ x + ϵ , for a ground truth matrix V ∗ and iid zero-mean noise ϵ such that Σ ϵ = diag ( 1 , ϕ x ) , where ϕ x ∈ ( 0 , 1 ) is called the 'data balance.' In Appendix A.6, we also train a deep nonlinear network, and we see the same trend where improving balance in the label noise leads to flatter solutions.

which can be arbitrarily large. Recall that the minimum of T ( θ ) , on the other hand, does not depend on Σ ϵ , the label noise covariance. Thus, the imbalance of the noise spectrum can lead to arbitrarily high sharpness. This could especially be a problem for language model training because there is a large variation in the randomness of tokens. Some words, like 'the,' could have a very low conditional entropy, while nouns or verbs can have high entropy, especially when there exist synonyms. Another example is to choose Σ ϵ = I , which gives V ′ = V √ Σ x = ˜ US ′ ˜ U T , and thus

which is exactly the same as the minimal sharpness. This suggests that without the imbalance of the label noise alone, SGD indeed converges to the flattest solution.

Thus, an imbalance in the input feature can lead to different sharpness-seeking behaviors. At the same time, if the loss function has scale invariance ( ℓ ( x, θ ) = ℓ ( x, λθ ) for any λ ∈ R ), the learning dynamics leads to a flattening of the curvature during training (See Section B.1). Thus, entropic forces and symmetry are strong factors deciding the sharpness-seeking behavior of SGD. 3

6 Conclusion

In this work, we have proposed an entropic-force perspective on neural network learning. We derive an entropic loss, which breaks continuous symmetries and preserves discrete ones, leading to universal behaviors such as gradient balance and alignment. The entropic loss suggests a potential unifying perspective for understanding training: learning algorithms prefer solutions with the minimal gradient fluctuation. This perspective provides a unifying framework that explains several emergent phenomena in deep learning, including sharpness-seeking behavior and universal feature structure, as consequences of underlying symmetries and entropic forces. The framework offers predictive and explanatory power across architectures and scales, and points toward a more principled, physics-inspired understanding of learning dynamics and emergent phenomena. Future work may extend this foundation to encompass higher-order corrections, richer architecture structures, and nonequilibrium dynamics of modern training procedures. A major limitation of our work is that we focused only on problems with explicit symmetries; an important future direction is to extend the results to cases with only approximate symmetries. Our experiments show that symmetry-free systems qualitatively agree with symmetry-preserving systems, suggesting that there are underlying, hidden concepts yet to be discovered.

An implication of Theorem 2 is that discrete symmetries such as Z 2 still remain in the loss function. This enables the possibilities of spontaneous symmetry breaking and phase transitions in neural networks. Prior works have studied phase transitions [7] without the entropy term and are inherently zero-temperature phase transitions. It may be possible to develop a theory of phase transitions based on entropic loss, similar to the classical Landau theory. Also, a striking aspect of our construction is its similarity to the actual formalism of thermodynamics in physics; our result motivates the development of a robust thermodynamics theory of deep learning.

Acknowledgment

The authors thank Prof. Phillip Isola and Prof. Tomaso Poggio for discussion. ILC acknowledges support in part from the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) through NSF Grant No. PHY-2019786. This work was also supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF - 1231216.

References

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems , 36:28699-28722, 2023.

Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936 , 2023.

Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065 , 2021.

Jeremy M Cohen, Alex Damian, Ameet Talwalkar, Zico Kolter, and Jason D Lee. Understanding optimization in deep learning with central flows. arXiv preprint arXiv:2410.24206 , 2024.

Liu Ziyin and Masahito Ueda. Zeroth, first, and second-order phase transitions in deep neural networks. Physical Review Research , 5(4):043243, 2023.

Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning , 2024.

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 , 2024.

Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940 , 2024.

Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, and Paul Pu Liang. Understanding the emergence of multimodal representation alignment. arXiv preprint arXiv:2502.16282 , 2025.

Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, and Mengnan Du. Large vision-language model alignment and misalignment: A survey through the lens of explainability. arXiv preprint arXiv:2501.01346 , 2025.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

Tomer Galanti and Tomaso Poggio. Sgd noise and implicit low-rank bias in deep neural networks. arXiv preprint arXiv:2206.05794 , 2022.

Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594 , 2022.

Shuo Xie and Zhiyuan Li. Implicit bias of adamw: ℓ ∞ -norm constrained optimization. In International Conference on Machine Learning , pages 54488-54510. PMLR, 2024.

Rong-Gen Cai, Li-Ming Cao, and Nobuyoshi Ohta. Friedmann equations from entropic force. Physical Review D-Particles, Fields, Gravitation, and Cosmology , 81(6):061501, 2010.

Sho Yaida. Fluctuation-dissipation relations for stochastic gradient descent. arXiv preprint arXiv:1810.00004 , 2018.

Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems , 32, 2019.

Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations , 2022.

Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. In International Conference on Machine Learning , pages 70457056. PMLR, 2021.

Liu Ziyin, Hongchao Li, and Masahito Ueda. Noise balance and stationary distribution of stochastic gradient descent. Physical Review E , 111(6):065303, 2025.

David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162 , 2020.

Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 , 2021.

Yatin Dandi, Luis Barba, and Martin Jaggi. Implicit gradient alignment in distributed and federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6454-6462, 2022.

Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. arXiv preprint arXiv:2109.14119 , 2021.

Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel LK Yamins. The limiting dynamics of sgd: Modified loss, phasespace oscillations, and anomalous diffusion. Neural Computation , 36(1):151-174, 2023.

Xingguo Li, Zhaoran Wang, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, and Tuo Zhao. Symmetry, saddle points, and global geometry of nonconvex matrix factorization. arXiv preprint arXiv:1612.09296 , 1:5-1, 2016.

Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated optimization. Advances in Neural Information Processing Systems , 35:16679-16690, 2022.

Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Liu Ziyin, Yizhou Xu, and Isaac L Chuang. Remove symmetries to control model expressivity and improve optimization. The Thirteenth International Conference on Learning Representations , 2025.

Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry potentially unifies deep learning theory. arXiv preprint arXiv:2502.05300 , 2025.

Hidenori Tanaka and Daniel Kunin. Noether's learning dynamics: Role of symmetry breaking in neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 25646-25660. Curran Associates, Inc., 2021.

Sibylle Marcotte, R´ emi Gribonval, and Gabriel Peyr´ e. Abide by the law and follow the flow: Conservation laws for gradient flows. 2023.

Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. arXiv preprint arXiv:2306.04251 , 2023.

Andre Wibisono, Ashia C Wilson, and Michael I Jordan. A variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences , 113(47):E7351-E7358, 2016.

Liu Ziyin, Isaac Chuang, and Tomaso Poggio. Heterosynaptic circuits are universal gradient machines, 2025.

Philip W Anderson. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science , 177(4047):393-396, 1972.

Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems , 31, 2018.

Andrey Polyakov. Homogeneous artificial neural network. arXiv preprint arXiv:2311.17973 , 2023.

Akshay Rangamani and Andrzej Banburski-Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4243-4247. IEEE, 2022.

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems , 34:225-236, 2021.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519-3529. PMLR, 2019.

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619-8624, 2014.

Liu Ziyin and Isaac Chuang. Proof of a perfect platonic representation hypothesis. arXiv preprint arXiv:2507.01098 , 2025.

Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of a deep linear network. In Advances in Neural Information Processing Systems , 2022.

Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems , 34:27449-27461, 2021.

Lei Wu, Mingze Wang, and Weijie Su. The alignment property of SGD noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems , 35:4680-4693, 2022.

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. In International Conference on Learning Representations , 2021.

L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp Minima Can Generalize For Deep Nets. ArXiv e-prints , March 2017.

1.0

2.0

layer1&2

layer1&2

balance

0.8

layer2&3

layer3&4

neuronbalance

1.5

layer2&3

layer3&4

0.6

1.0

layer

0.4

0.2

0.5

0.0

0.0

0

5000

10000

15000

20000

0

5000

10000

15000

20000

5

8

layerl&2

layerl&2

layerbalance

4

layer2&3

neuron balance

6

layer3&4

layer3&4

layer5&6

3

4

2

2

1

0

5000

10000

15000

20000

0

5000

10000

15000

20000

A Experiment

A.1 ResNet

For Figure 1, we train ResNet18 on CIFAR 10 using SGD with momentum 0 . 9 , batchsize 128 and weight decay 5 × 10 -4 . The learning rate is 0 . 1 at the beginning, 0 . 01 after the 100 -th epoch and 0 . 001 after the 150 -th epoch. The entropy is calculated by summing the gradient norm of all parameters. We obtain training accuracy 98% and test accuracy 88% at the end.

A.2 Gradient Balance

For Figure 2, we train on the MNIST dataset but the labels are generated by a teacher ReLU network and trained with an MSE loss. Namely, the loss is

where y ( x ) is a parameterized by a random ReLU teacher network and ϵ x is an i.i.d. Gaussian noise with 0 . 2 standard deviation. The training proceeds with SGD for 10 4 steps with a learning rate of 0 . 01 and batchsize of 200 . Layer balance is calculated by ∣ E Tr [ g i g T i -g T i + 1 g i + 1 ]∣ and neuron balance is calculated by ∑ j ∣ E Tr [ g i,j, ∶ g ⊺ i,j, ∶ -g i + 1 , ∶ ,j g ⊺ i + 1 , ∶ ,j ]∣ for the i -th layer.

Additional experiments on layer balance and neuron balance are presented in Figures 7, 8 and 9 for 4-layer linear networks, ReLU networks and simple self-attention networks, still for the teacherstudent setting. The hidden dimensions are 256 , 128 , 64 for linear and ReLU networks, and 256 for self-attention. We present the evolution of layer and neuron balance along training. Figures 7 and 8 suggest that layer balance and neuron balance approach zero during training, which verifies Theorems 5 and 6.

5

imbalance

5

4

2.0

imbalance

anc

4

3

1.5

gradient i

b

E

3

2

1.0

gradient

adier

2

1

ra

0.5

1

6

0

0

0

1

2

0.5

1.0

0

2000400060008

800010000

entropy

1e1

loss

le1

A.3 Gradient Balance in Self-Attention Nets

Figure 9 suggests that G T U G U -G V G T V approaches zero during training, and it is correlated with the entropy rather than the loss, which verifies Theorem 7.

alignment

alignment

10-1

1.0

10-1

1.0

0.8

0.8

decay

decay

0.6

0.6

10-2

10-2

weight

weight

0.4

0.4

0.2

0.2

10-3

10-3

0.0

0.0

102

102

batchsize

batchsize

le-1

le-1

4

4

3

3

alignment

alignment

2

2

1

1

0

0

2.0

2.5

3.0

3.5

4.0

0.95

1.00

1.05

1.10

1.15

entropy

le-1

loss

alignment

alignment

10-1

1.0

10-1

1.0

0.8

0.8

decay

decay

0.6

0.6

weightc

10-2

weight(

10-2

0.4

0.4

0.2

0.2

10-3

10-3

0.0

0.0

102

102

batchsize

batchsize

A.4 Unversal Representation Learning in MLP

For Figure 3, we train two independent 6-layer networks on MNIST. The networks have linear or tanh activation and 128 neurons in each hidden layer, and for the second network, the input MNIST data is transformed by a random Gaussian matrix. We train the networks with Adam optimizer, learning rate 10 -4 for 5 epochs. During training, we measure the representation alignment between every pair of layers, defined as the cosine similarity between the two sides of (12), averaged over the test set. We then plot the average alignment between the same or different layers of two networks. The input alignment denotes the average alignment between every layer representation and the input data.

In Figures 10 and 12, we test the influence of batchsize and weight decay on universal representation in a teacher-student setting. In Figure 10, both the teachers and students are two-layer ReLU networks. In Figure 12, the student is replaced by a linear network. Their hidden dimensions are 100 . Similar to Figure 3, we measure the representation alignment between the middle layers of two independently trained networks, and the input of the second network is rotated by a Gaussian matrix. We use random Gaussian data, and the labels are generated by the teacher network. We train the student networks with SGD, learning rate 5 × 10 -2 or with Adam, learning rate 10 -4 . For both SGD and Adam optimizers, Figures 10 and 12 suggest that universal alignment does not rely on the batchsize as predicted, but disappears for large weight decay, which verifies Theorem 9. In Figure

11, we show that the increase of alignment is more correlated with the decrease of the entropy rather than the loss.

20

0.8

Amlplayer

NetB mlp layer

Net A Attention

NetBAttention

0.6

15

0.4

10

0.2

Net

5

0.0

0

-0.2

0

10

20

0

10

20

0

10

20

0

10

20

Net A mlp layer

NetAmlplayer

NetAAttention

NetAAttention

A.5 Universal Representation in ViT

See Figure 13 for the alignment in Vision Transformer. The pretrained weights are taken from https://docs.pytorch.org/vision/main/models.html . We measure the CKA alignment between the two models or with itself with a minibatch size of 300 images from the ImageNet dataset.

0.40

2.0

0.35

1.8

1.8

0.30

1.6

balance

0.25

1.4

1.6

data

b

0.20

0.15

1.2

1.4

1.0

e

0.10

1.2

0.05

0.8

0.6

1.0-

0.3

0.4

0.5

0.6

0.7

0.0

0.1

0.2

0.3

learningrate

databalance

A.6 Edge of Stability

See Figure 5 and 6, where we train a two-layer linear network on a linear regression task with a 2d label y ∈ R 2 . The labels y = V ∗ x + ϵ , for a ground truth matrix V ∗ and iid zero-mean noise ϵ such that Σ ϵ = diag ( 1 , ϕ x ) , where ϕ x ∈ ( 0 , 1 ) is called the 'data balance.'

We train with different learning rates and data balance. The training proceeds with SGD with a batchsize of 32 for 4 × 10 4 iterations. Input x ∈ R 2 is drawn from a standard Gaussian distribution, and the model has dimensions 2 → 10 → 2 .

See Figure 14 for an experiment with ReLU. Here, the architecture is has dimensions 2 → 10 → 10 → 2 .

B Theory

B.1 Scale Invariance Leads to Flattening

The symmetry generator is A = I . We have

by Theorem 4. As d dλ F η,γ ( e λ θ )∣ λ = 0 = θ T ∇ F η, 0 . Therefore, when we do gradient descent along -∇ F η, 0 , λ monotonously increases. Meanwhile,

decreases with λ , and thus the sharpness decreases along training.

B.2 Proof of Theorem 1

Proof. For notational simplicity we drop the subscript γ in the proof. When running gradient descent on ℓ ( x, θ ) , we have

When running gradient descent on ϕ Λ , we have

and

Similarly we can obtain

Therefore, we have θ ′ n = θ 1 + O (∣∣ Λ ∣∣ 3 +∣∣ Λ ∣∣ 2 / n ) if we choose ϕ 1 ( x, θ ) = 1 4 ∇ ℓ ( x, θ ) T Λ ∇ ℓ ( x, θ ) For small ∇ 3 ℓ ( x, θ ) , we have

.

Therefore, we can choose

B.3 Proof of Theorem 4

Proof. By the definition of the exponential symmetry,

Taking derivative w.r.t. λ on (29), we have that ∇ θ ℓ ( x, θ ) T Aθ = 0 . Then taking derivative w.r.t. θ , we have that A T ∇ θ ℓ ( x, θ ) + ∇ 2 θ ℓ ( x, θ ) Aθ =0.

Let I ( λ ) ∶= d dλ F η,γ ( e λA θ ∗ ) and θ λ ∶= e λA θ ∗ . Then we have

where Σ ( θ λ ) ∶= E B [ E x ∈B ∇ θ ℓ ( x, θ λ )][ E x ∈B ∇ θ ℓ ( x, θ λ )] T is positive semi-definite. We also use Tr [ Σ ( θ λ ) A -A T 2 ] = 0 and ( θ λ ) T A -A T 2 θ λ = 0 . By [29, Lemma B.1], we have Tr [ Σ ( θ λ ) ˜ A ] = Tr [ e -2 λ Σ ( θ ∗ ) ˜ A ] , and thus

where µ i , n i are eigenvalues of eigenvectors of the symmetric matrix ˜ A . Therefore,

We have I ( λ ) ≡ 0 (which happens only if ˜ A is not full rank) or I ( λ ) strictly monotonic. As θ ∗ is a local minimum, we have I ( 0 ) = 0 , which gives (4). Then we have I ( λ ) ≡ 0 or I ( λ ) = 0 iff λ = 0 , which finishes the proof.

B.4 Proof of Theorem 2

Proof. By the definition of K -invariance, and taking derivative ∇ θ of both sides of ℓ ( x, K ( θ, λ )) = ℓ ( x, θ ) , we have

where the l.h.s. follows from the chain rule. This imlies that

and so

If F is K -invariant, the following Equation holds:

By the assumption, ℓ ( x, K ( θ, λ )) = ℓ ( x, θ ) , and (36), we have that

There are two cases: (1) η ∇ T ℓ ∇ Q T ∇ ℓ -γQ T θ = 0 , and (2) η ∇ T ℓ ∇ Q T ∇ ℓ -γQ T θ ≠ 0 .

For case (1), we are done. For case (2), the equation cannot hold for γ + dγ because the first term is independent of γ . Thus, we can only have case (1).

This means that for any θ

Thus,

which is only possible if K θ, λ θ

∥ ( )∥ 2 = ∥ ∥ 2 . This completes the proof.

B.5 Proof of Theorem 3

Proof. By definition we have ∣∣ Oθ ∣∣ 2 = ∣∣ θ ∣∣ 2 . Take derivative on both sides of ℓ ( x, Oθ ) = ℓ ( x, θ ) , we have

Thus we have

which gives ∣∣∇ Oθ ℓ ( x, Oθ )∣∣ 2 = ∣∣ O -T ∇ θ ℓ ( x, θ )∣∣ 2 = ∣∣∇ θ ℓ ( x, θ )∣∣ 2 . Combining the above results we have F η,γ ( Oθ ) = F η,γ ( θ ) .

B.6 Proof of Theorem 5

Proof. Rescaling symmetry implies that if we make the transform

in Theorem 4, where the index klm corresponds to the m -th element of the l -th unit of the k -th layer. Then we have

This finishes the proof by setting γ = 0 .

B.7 Proof of Theorem 6

Proof. Wecan choose A to be a rescaling matrix w.r.t. the j -th neuron of the i -th layer. Specifically, we choose in Theorem 4, which gives

B.8 Gradient Imbalance in Polynomial Networks

Theorem 11. (Neuron Balance) For all local minima of Eq. (3) and any i, j ,

This means that unless d = 1 , these networks will either have a noise or weight explosion problem. If γ = 0 , the gradient fluctuation grows like d D , exponential in depth D . When d < 1 , later layers will have an exploding noise; when d > 1 , earlier layers will have an exploding noise. When both η and γ ≠ 0 , the sum of the noise and gradient norm will explode exponentially. In some sense, this implies that linear or sublinear types of activations are the only stable activations for deep neural networks.

Proof. We can still choose A to be a rescaling matrix, but this time we should rescale the i + 1 -th layer more

L does not change.

This corresponds to the choice of

This corresponds to in Theorem 4, which gives

This gives

by choosing γ = 0 .

B.9 Proof of Theorem 7

Proof. The double rotation symmetry can be written as

where A is an arbitrary matrix. We can thus choose the following generator

in Theorem 4, where k = 1 corresponds to W and k = 2 corresponds to U . This gives

which finishes the proof.

B.10 Proof of Lemma 1

Proof. By definition, we have and thus

Let A ∶= ∑ i µ i n i n T i , and thus

As A E ∇ 2 ℓ ( x, θ ) ≠ 0 , there exists i such that µ i ≠ 0 and n T i E ∇ 2 ℓ ( x, θ ) n i ≠ 0 . Therefore, we have lim λ →+∞ ∣ T ( e λA θ )∣ = +∞ if µ i < 0 , and lim λ →-∞ ∣ T ( e λA θ )∣ = +∞ if µ i > 0 .

B.11 Proof of Theorem 8

We first prove the following theorem, which we will leverage to prove Theorem 8.

Theorem 12. Let V ′ = √ Σ ϵ V √ Σ x such that V ′ = ˜ US ′ ˜ V is its SVD and rank ( V ′ ) = d . Assume that every layer has more than d hidden units. Then if γ = 0 and η = 0 + , at any global minimum of (3) , we have for i = 2 , /uni22EF , D -1 , where U i are arbitrary matrices satisfying U T i U i = I d × d , and Σ x = E [ xx T ] , Σ ϵ = E [ ϵϵ T ] . Moreover,

Proof. Consider two consecutive layers W i and W i + 1 . Using Theorem 7, we have

By the MSE loss ℓ ( x, y ) = ∣∣ y -M 1 W D /uni22EF W 1 M 2 M 3 x ∣∣ 2 , this gives

where ˜ x = E x ∈B x and ˜ r ∶= E x ∈B [ y -M 1 W D /uni22EF W 1 M 2 M 3 x ] satisfy E ˜ x = E ˜ r = 0 , and thus E ˜ x ˜ x T = Σ x ∣B∣ and E ˜ ϵ ˜ ϵ T = Σ ϵ ∣B∣ . We use the fact that at the global minimum we have M 1 W D /uni22EF W 1 M 2 M 3 = V , and thus y -M 1 W D /uni22EF W 1 M 2 M 3 x is independent of x . We denote ξ i + 1 ∶= M 1 W D /uni22EF W i + 2 , h i ∶= W i -1 /uni22EF W 1 M 2 M 3 for i = 2 , 3 , /uni22EF , D -2 , and ξ D ∶= M 1 , h 1 ∶= M 2 M 3 .

for i = 2 , 3 , /uni22EF , D -2 . For i = 1 we have

and for i = D -1 we have

Lemma 2 proves that we can decompose the matrices W ′ 1 , W 2 , /uni22EF , W D -1 , W ′ D as

By minimizing (3) at η = 0 + , we need to minimize

Thus we have which gives

Then we obtain U D = ˜ U , U 0 = ˜ V and

(64) and (65) give

and thus

Combining (70), (72) and (74), we finish the proof.

We can assume Σ D , /uni22EF , Σ 1 ∈ R d × d because their ranks are the same by (63), (64) and (65). (63) gives

and thus Σ i = cI d for i = 2 , 3 , /uni22EF , D -2 and

Now, we are ready to prove Theorem 8.

Proof. By Theorem 12, at any global minimum of (3), the solution of a D A -layer network for the dataset D M is given by

for i = 2 , /uni22EF , D -1 , where U i are arbitrary matrices satisfying U T i U i = I d × d , and

for some constants c 1 , c 2 , c 3 . The solution suggests that

Similarly

The proof is complete by comparing (77) and (78).

One might also consider the case where the minimal width of network B is d B < d . In this case, we denote ¯ S ′ ∈ R d B × d B containing top d B values of S ′ . Then we have

It is now not fully aligned with h A ( x ) . To calculate the alignment, the corresponding kernels are

and

for some constants c 1 , c 2 > 0 . We then have

and

Therefore, the alignment is given by

which is some value between 0 and 1 .

In the end of this section we proves the following technical lemma.

Lemma 2. Suppose that matrices W 1 , W 2 , /uni22EF , W D satisfy

for some λ i > 0 and i = 1 , 2 , /uni22EF , D -1 , then we have write the SVD of W 1 , W 2 , /uni22EF , W D as

where Σ D , /uni22EF , Σ 1 are the singular values.

Proof. Denote P i ∶= ( W i /uni22EF W 1 )( W i /uni22EF W 1 ) T . We then have

Similarly we have and

which gives

where S i + 1 ∶= W T i + 1 W i + 1 .

Suppose that S i + 1 = V Λ V T , and thus we have

where c ∶= λ i + 1 λ i and A ∶= V T P i V . The ( j, k ) element gives

If A jk ≠ 0 , as A is semi-definite, we also have A jj , A kk ≠ 0 , which gives cλ 2 i = cλ 2 j = 1 . Therefore, we have A Λ = Λ A , and thus

This shows that P i and S i + 1 share the same eigenspace. Moreover, we have P i = W i P i -1 W T i . Denote W i = U i Σ i U T i -1 and W i + 1 = U i + 1 Σ i + 1 V . As P i -1 share the same eigenspace with W T i W i , we have P i -1 = U i -1 Λ U T i -1 for some diagonal matrix Λ , which gives P i = U i Σ i ΛΣ i U i . As P i share the same eigenspace with W T i + 1 W i + 1 = V T Σ 2 i + 1 V , we obtain V = U i , which finishes the proof.

B.12 Proof of Theorem 9

Proof. By Theorem 7 we have

for i = 1 , /uni22EF , D -1 , which suggests that W i = U i Σ i U T i -1 with Σ 2 i = Σ 2 i + 1 . At the global minimum we have U D ( Π D i = 1 Σ i ) U T 0 = Π D i = 1 W i = M -1 1 V M -1 2 , which shows that the left side is the SVD of M -1 1 V M -1 2 . This finishes the proof.

B.13 Proof of Theorem 10

Proof. By Theorem 12 we have

By [31, Proposition 5.3] we have

This finishes the proof of (17).

Meanwhile we can also calculate

As UW Σ 1 / 2 x = V Σ 1 / 2 x ∶= ˆ U ˆ S ˆ V , the minimum is given by

and

where ˆ U 1 is an arbitrary orthogonal matrix.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract explicitly states that out main contribution is a theoretical framework for understanding the learning dynamics and emergent phenomena of neural networks trained with stochastic gradient descent (SGD) based an entropic loss landscape.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed this in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems are stated with the full set of assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present all experiment details in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The experiments are only for demonstration and are straightforward to reproduce following the description.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We disclose all hyperparameters of experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot the standard error as the shaded area.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Personal computers are sufficient for all our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We closely follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use pytorch. We use MNIST and CIFAR datasets, and the publically available pretrained weights from the pytorch website.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This research does not involve LLMs as any important, original, or nonstandard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/ 2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new theory for understanding the stochasticity and discretization effects arising from stochastic gradient descent (SGD). A key idea is approximating a single update of the original loss with n steps of updates of a loss, named entropic loss. When  $n \\rightarrow \\infty$, the discrete update is converted to a continuous update. The discretization error and noise effects are further decomposed from the entropic loss and termed effective entropy, and the gradient of the effective entropy, named entropic forces, is shown to break almost any continuous parameter symmetries while preserving discrete ones. As two example applications, the entropic forces are used to explain two observations: the progressive sharpening of the loss landscape and the emergence of universal representations\n\nSTRENGTHS AND WEAKNESSES:\nStrengths And Weaknesses:\n\nI have limited knowledge in this specific area and the overall manuscript appears satisfactory to me. \n\nQuality:\n\n- [+] The theoretical demonstrations are clear and seems technically sound.\n\nClarity:\n\n- [+] The presentation of the idea is clear.\n\nSignificance& Originality:\n\n- [+] I think this work is novel and can be a contribution to the Machine learning community.\n- [+] New theory is introduced for understanding the learning dynamics.\n\nQUESTIONS:\n- The entropic force appears to result from both stochasticity and discretization effects. Is it possible to analyze these two effects independently, or do they interact in a way that complicates their separation?\n- When $n \\rightarrow \\infty$, the discrete update transitions to a continuous update for the effective entropy. Apart from the ability to use Lagrangian formalism, are there other practical advantages of this continuous update?\n- In Figure 1, should the pink arrows be red instead?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a theoretical framework that explains various emergent behaviors in deep learning by modeling stochasticity in the learning process caused by discrete time updates as governed by the entropic forces. The authors derived an entropic loss function for neural network learning and showed that this loss function breaks continuous symmetries while preserving discrete ones. The paper provides a theoretical explanation for various emergent phenomena in deep learning including gradient and weight alignment, universal feature structure, and sharpness seeking behaviors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The perspective of using an entropic loss to understand stochasticity in the learning dynamics is quite interesting and novel. The paper then proceeds with rigorous theoretical proofs to demonstrate interesting implications of this perspective. Using this entropic loss, the paper shows that the loss breaks all continuous symmetry and preserves discrete ones, this is in stark contrast with previous works that focus on the equilibrium of continuous gradient flow without incorporating the stochasticity induced by discrete time dynamics. The framework also provides explanations for previously observed behaviors including weight and gradient alignment, universal feature structure and sharpness seeking. All results are supported with sufficient numerical experiments. \nThis paper not only provides a new insights on understanding learning dynamics in discrete time with mathematical rigor, but also discusses thoroughly the potential implications of the theoretical results and their connections to widely observed phenomenon. Therefore, I believe this paper is a valuable addition to theoretical understanding of ML and is of interest to a broad audience. \n\nWeaknesses: \nWhile the paper shows empirical evidence in realistic settings. Many of the main theorems (8-10) are established for deep linear networks, or particular types of settings (theorems 5-6) I think it is still insightful as long as the theoretical results can predict behaviors in more complicated settings, as the authors show, but it would be nice to have a summary of results or discussions which part of the results are applied to which types of networks/loss etc.\n\nQUESTIONS:\n1.\tI’m not sure I understood Definition 1. First, the weight decay gamma does not appear in the definition. Second, do you mean that for a given function K satisfying locality and consistency, the loss function satisfies the invariance condition? In that case maybe switch the order of the 3 conditions. \n2.\tThe entropic loss is initially denoted as phi but later denoted as F, make sure the notations are consistent or state explicitly otherwise.\n3.\tThe gradient balancing result is interesting. However, I do not fully understand its connection to the results with small initialization, in your case, the weight balance requires eta goes to zero, so entropic force goes to zero and the system can be viewed as continuous gradient flow, but there’s no assumption on the initialization, so I’m wondering where this difference comes from.\n4.\tIn Figure 2, the ranges of loss/entropy/imbalance are all relatively small, is this because you only select time points at later stage of training, i.e. close to equilibrium? Or is there some other reason why this is happening? \n5.\tIn earlier relevant works on deep linear networks (i.e, Hanin & Zlokapa, Li & Sompolinsky), they also discussed and calculated the hidden representations, especially the average of the inner product of the hidden layer activation, across layers after training. Their calculation would be at the equilibrium of the loss, where no entropic loss is considered, and also averaged over all possible solutions, whereas your results are about learning with SGD and discrete dynamics without averaging over the solution space. While these two frameworks are very different, I’m wondering whether some results can be connected, i.e, can your results reduce to theirs in certain limits? Does the representation in your case also have the inner product aligning with the target output?\n6.\tIn Figure 3, did you experiment with different types of nonlinearity to see how sensitive the alignment is? For example, how would alignment for a ReLU network look like? \n7.\tSome typo in Figure 4, I suppose these are mlp/attention layers and self/other alignment respectively but the axis labels are all the same. \n8.\tIn section 4, does the balance result depend on the MSE loss? If not, why would you do a student-teacher setup with MSE loss on MNIST, it would seem more natural to simply train the original MSE dataset with CE loss?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper tries to bridge the gap between theory and practice in deep learning. It studies a notion of entropic loss landscape and proves that the associated forces break continuous symmetries while preserving discrete ones, and gives a tradeoff between gradient and weight balancing. This also helps explain recent observed phenomena such as progressive sharpening and flattening in deep learning. Several experiments are run to support the claims.

STRENGTHS AND WEAKNESSES:
Strengths:
- The research direction is interesting and may help explain some phenomena observed in practice in deep learning
- Results are supported with experiments.

Weaknesses:
- The results rely on strong assumptions.

QUESTIONS:
- What if the conditions on $\phi_{\eta}$ do not hold in Theorem 1 ? How often do they hold?
- What if the learning rate is not a matrix ? You said Adam corresponds to a matrix, what about other optimizers ?
- In Figure 2, is what you call entropy the $S(\theta)$ ?
- Does $\phi_{\eta}$ contain $\gamma$ in its definition? If so, how? If not, how did the gamma appear when computing its expectation?
- Can you say what the symmetries present (continuous and discrete) are in shallow linear network square loss for example ?
- Are there some losses in practice that have A-exponential symmetry ?
- How is theorem 2 saying that the continuous symmetry is broken ?
- Does the sharpness result change if we choose the largest singular value of the hessian instead of the trace?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03