Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
Authors: Luca Di Carlo
Chase Goddard
David J. Schwab
Venue: ICLR 2026 Poster (2026)
Keywords: Optimization
Mode Connectivity
Generalization
Entropy
Curvature
Flatness
Sharpness
Primary area: optimization
TL;DR: We show that even when minima may be connected by a path of low loss, such paths often exhibit a dynamical barrier produced by entropic forces.

Abstract: Modern neural networks exhibit a striking property: basins of attraction in the loss landscape are often connected by low-loss paths, yet optimization dynamics generally remain confined to a single convex basin and rarely explore intermediate points. We resolve this paradox by identifying entropic barriers arising from the interplay between curvature variations along these paths and noise in optimization dynamics. Empirically, we find that curvature systematically rises away from minima, producing effective forces that bias noisy dynamics back toward the endpoints — even when the loss remains nearly flat. These barriers persist longer than energetic barriers, shaping the late-time localization of solutions in parameter space. Our results highlight the role of curvature-induced entropic forces in governing both connectivity and confinement in deep learning landscapes.

Published as a conference paper at ICLR 2026
ENTROPIC CONFINEMENT AND MODE CONNECTIVITY IN OVERPARAMETERIZED NEURAL NETWORKS
Luca Di Carlo
Chase Goddard
Joseph Henry Laboratories
Joseph Henry Laboratories
Lewis-Sigler Institute
Princeton University
Center for the Physics of Biological Function
Princeton, NJ, USA
Princeton University Princeton, NJ, USA
Lucadc@@rincetenn.ee.
David Schwab
Initiative for the Theoretical Sciences The Graduate Center, CUNY
New York, NY, USA
ABSTRACT
Modern neural networks exhibit a striking property: basins attraction the loss landscape often connected by low-loss paths, yet y optimization dynamics generally remain confined to single convex basin (Baity-Jesi et al. 2019; Juneja et al. 2023) and rarely explore intermediate points. We resolve this paradox by identifying fynng entropic barriers arising from the interplay between curvature varia- tions along these paths and noise in optimization dynamies. Empirically, we find that curvature systematically rises away from minima, producing effective forces that bias noisy dynamics back toward the endpoints - even when the loss remains nearly flat. These barriers persist longer than energetic barriers, shaping the late- time localization of solutions parameter space. Our results highlight the role of cevature-inducee entropic forces in governing both connectivity and confinement in deep learning landscapes.
INTRODUCTION
Deep neural networks trained, in the overparametrized regime, exhibit a number of surprising and counterintuitive properties. One of the most striking is the observation that distinct solutions, found with standard optimization algorithms, are often connected by low-loss paths in parameter space (Garipov et al., 2018 Draxter et al. 2018 Frankle al. 2020 Such mode connectivity results im- ply that the landscape is far less rugged than once assumed: minima that appear isolated are, in fact, linked by paths of low, nearly constant loss. At the same time, however, optimization dynamics dis- play seemingly contradictory behavior. Standard training with stochastic gradient descent (SGD), with without momentum, converges parameter space corresponding to these paths (Baity-Jesi et al. 2019).
well-defined minimum and rarely explores regions of
We argue that this paradox can be resolved by recognizing the role of entropic forces generated by curvature variations along connecting paths. Although the loss may be nearly flat along these paths, the curvature of the landscape typically increases away from found minima, producing effective barriers that bias stochastic dynamics back toward the endpoints. These barriers emerge from the interaction between fluctuations induced by SGD noise and the Hessian spectrum along low-energy paths. In this way, regions of parameter space that are energetically connected become effectively disconnected.
"Equal Contribution

---

Published as a conference paper at ICLR 2026
1.1 RELATED WORK
Our work aims to unify insights from two areas: First, we draw on insights from a body of work that shows SGD has an implicit bias towards flatter minima the strength of which increases with smaller minibatch size.
This behavior attributed to the higher noise levels in gradient estimates, which act form of implicit regularization preventing convergence to sharp minima and instead favoring wider. flatter basins that generalize better (Keskar et al. 2017). Several works have formalized this intuition: Jas- trzebski et al. (2018) Smith & Le| interpret the minibatch noise as an effective temperature that enables exploration and escape from sharp valleys: Wei & Schwab (2019) and Xie et al. (202T) further show that the resulting dynamics can be modeled stochastic process biased toward flat regions. Collectively, these studies demonstrate that curvature and generalization are deeply inter- twined with the stochastic geometry of the optimization trajectory. We leverage these insights to conduct deeper analysis of mode connectivity in neural network training. Garipov et al. (2018)
and Draxler al. (2018) showed the existence of nonlinear paths of low-loss between different minima found by training with different random seeds. Subsequently, (Franlle et al. 2020) showed that when the training dynamics of two networks tied together early in training, the resulting minima, found after training is complete, are linearly connected by paths of low-loss. Follow-up work has analyzed in more depth how linear mode connectivity emerges (Singh al.) 2024
Zhou
et al. 2023). and such work has important implications for model merging (Ainsworth et al
2023
Singh & Jaggi 2020) and weight-space ensembling (Izmailov al. 2018 Wortsman et Gagnon-Audet et al. 2023 Wortsman et al., 2022).
2021
1.2
CONTRIBUTIONS
Our main contributions are as follows:
We show empirically that the curvature along minimum-loss paths between minima in- creases away from the endpoints.
We argue that such a "bump" in the curvature leads to an entropic barrier, and that such entropic barriers lead to confinement of solutions even when the loss is low along a path in parameter space.
We show that despite the existence of low-loss connecting paths between solutions, entropic forces confine models to specific regions of parameter space.
We show that entropic barriers between minima persist longer than energetic barriers, when considering models that shared the first & epochs of training, suggesting that both energetic and entropic forces are responsible for the final region of parameter space that a model ends up in.
2 BACKGROUND: ENTROPIC FORCES AND CURVATURE
It is well established in statistical physics that the state of system is governed not only by energetic forces-derived from gradients of an energy or potential- but also by entropic forces, arising from thermal fluctuations. In the context of neural networks, the energy landscape defined by the train- ing loss, and its gradient directs the deterministic component of learning. However, the stochasticity introduced by finite learning rates and minibatch sampling induces an effective temperature, making entropic contributions to the trajectory of the model non-negligible. As a result, optimization can be biased toward broader, flatter regions of the landscape- not because they are lower in loss, but because they occupy larger volume in parameter space. We illustrate this idea in simple toy model. Consider Brownian particle evolving in a two-dimensional potential
1 ) ,(())a",
Φ = VV + E(t),
V y) =
g(v)>0
(1)
where & is white delta correlated Gaussian noise, (E,(t) = 2T (t). and g(y) is an arbitrary positive function of the coordinate y. In the analogy to deep learning, V plays the role of the task loss function, while the noise & arises from SGD noise due to minibatching and finite learning rate. To leading order one identify T x n/ B3 for learning rate y and batch size B,
2

---

Published as a conference paper at ICLR 2026
(A)
(B)
Figure Curvature produces an entropic force. (A) Illustration of potential V(r, 0) with circular minimum at r = 1, where the curvature varies with angle. At zero temperature (7 = 0), the angular distribution uniform, P(0) = 1/(2x). ). At finite temperature, thermal fluctuations bias the system toward flatter regions (yellow) rather than sharper ones (red). (B) Example of a Brownian particle diffusing along the ridge of a loss landscape, lighter colors correspond to larger times. En- tropic forces generated by fluctuations push the particle toward flatter directions, effectively favoring broader regions of the landscape.
Mandt et al. 2017,Smmi (Smith al. 2020 Liu et al. 2021). though the precise relationship depends on local curvature and other details of the loss landscape (Ziyin et al. 2021). this simple motivating example we assume is white and Gaussian, even though in deep networks the SGD noise is neither perfectly white nor perfectly Gaussian. this analogy. y corresponds soft modes (directions with nearly flat curvature where the loss hardly changes), and represents the stiff modes (directions associated with large eigenvalues of the Hessian).
For fixed y. the distribution of is Gaussian with variance (x²) ('(y))- When ø relaxes much faster timescale than y y-for instance when the curvature of the potential along the z-direction much larger than that along the y-direction- may average the fast variable obtain effective dynamics for y:
1 dg(v) gly) dy
0y (x + - -T
+ g.
This equation be rewritten gradient-flow dynamies generated by effective potential (y))
(v) Ver I' (v)).
(E(+) 2T 5(t - t))
dy
The resulting stochastic dynamics converge to Boltzmann-like stationary distribution (Gardiner ct al. 2004).
Ver( T
P(y) exp
(4)
Equation 3 reveals the key mechanism: the force is proportional to the negative derivative of g effectively driving the system towards smaller values g(y). corresponding flatter directions in We call these forces entropic because they are proportional to the effective temperature 7 and therefore vanish in the absence of noise. This familiar principle statistical physics, where the thermodynamic state of system is determined by competition between minimizing energy and maximizing entropy, with the temperature controlling the relative importance of the two. In Fig. 1 illustrate this effect with example potentials whose curvature varies with one of the coordinates, showing how the resulting entropic force pushes the system toward flatter regions.
In deep neural networks, these forces expected to grow stronger the effective temperature increases, making them more prominent for large learning rates and small minibatches, as we illus- dominate, potentially
trate in Section4.1 When entropy outweighs energy, entropic forces
3

---

Published as a conference paper at ICLR 2026
driving optimization to climb the loss landscape. An example of this phenomenon in deep network is also shown in Section 4.1
This minimal example is far from capturing the full complexity of the dynamics in regions of approx- imately constant low loss of real deep neural networks. Nevertheless, it illustrates how stochasticity interacts with curvature to favor flatter minima. Even when the training loss is near zero, these entropic barriers can effectively confine solutions to specific regions of parameter space. In the re- mainder of this paper, we show empirically that entropic forces arising from the curvature of loss landscapes in real networks trained on natural images produce qualitatively similar behavior.
3 METHODS
We begin by training a collection of mmage-classifiation models on CIFAR-10 using both Wide ResNet and ResNet architectures, each initialized from different random seeds obtain diverse set of distinct minima. For every pair of minima under study, we construct a low-loss path connecting them using the AutoNEB algorithm of Draxler al. (2018).
Once the MEP is obtained, we analyze its geometric and dynamical properties. First, we measure curvature along the path using several complementary Hessian-based statistics, allowing us to char- acterize how flatness varies between the endpoints and the interior of the path. Second, we study optimization dynamics constrained to the MEP by projecting SGD updates onto the nearest path seg- ment. This controlled setting isolates how stochasticity interacts with curvature: models initialized along the MEP exhibit systematic drift toward flatter regions, revealing the action of entropic forces whose strength grows with the effective noise level. We then proceed to analyze this phenomenon at a finer scale by studying linearly connected minima, following the approach of Frankle et al. (2020).
3.1 TRAINING DETAILS
Unless otherwise specified, all experiments conducted Wide ResNet- 16-4 Zaaoruyko & Komodakis (016) trained on the CIFAR- dataset (Krizhevsky) 2009). Following standard practice (Zaorrykko & Komodakis (0166). we use stochastic gradient descent (SGD) with momentum / 0.9, weight decay regularization w = 5 x 10-4 and an initial learning rate of n 0.1. Models are trained for 200 epochs with a batch size of 256, and the learning rate reduced by a factor of 5 at 30%, 60%, 80%, and 90% of the total training epochs. We apply mild data augmentation consisting of random horizontal flips and random crops with pixel padding followed by resizing to the original 32 32 resolution.
3.2 MINIMUM ENERGY PATII
To explore the structure of the loss landscape between different solutions, we identify low-loss connecting paths using the Automatic Nudged Elastic Band (AutoNEB) algorithm introduced by Draxler et al. (2018). In brief, the algorithm initializes k intermediate pivots along the straight-line path between elastic springs, while minimizing the loss orthogonal to the path.
minima and optimizes their positions such that they evolve if connected by
Since the loss along the straight segments between pivots may still be high, AutoNEB dynamically adds new pivots whenever the loss along a segment exceeds predefined threshold. This adaptive refinement ensures smooth, low-loss path is found. Following Draxler al. (2018), we refer to such paths as minimum energy paths (MEPs), by analogy with physical systems. Importantly, the optimization dynamics is such that does not change the length of the segments composing the MEP, so when a new pivot is inserted between two existing pivots, the resulting segments remain shorter than the original. In the following, the relative position along the MEP is reported in terms of pivot index, normalized by the total number of pivots. Note that this parameterization does not reflect the actual metric distance along the path, as the pivot density and length are non-uniform see Figure in Appendix A.3
Unless otherwise specified, all MEPs shown in the paper are computed using a sequence of refine- ment cycles with decreasing learning rates. Specifically, we run four cycles each with the following parameters: 10.1, 10), (5 x 10-?, 5), (10-2 -2 5), and (10-3, -3 5). where each tuple denotes (learning rate, number of epochs).
4

---

Published as a conference paper at ICLR 2026
3.3 CURVATURE MEASURES
A natural measure of the curvature of the loss landscape is the Hessian of the loss function, defined H = (O). More precisely, is the spectrum of the Hessian that captures the local geometry of the landscape. However, if the model has N parameters, then H E RNXN, making it intractable to compute or store explicitly for modern networks. Instead, we use three independent summary statistics of the Hessian spectrum, each providing tractable yet informative proxy for curvature.
We estimate the maximum eigenvalue of the Hessian, Amax ( using the power iteration method (see, e.g.. Yao et al. (2020)). Crucially, this method requires only Hestian-vector products, which can be computed efficiently via automatic differentiation in ((N) time. The update rule for the power method is:
(n+1)) Hulm ||Host))1*
OL)())
where He Σ 3 00,00g ug.
(5)
After few iterations, (()) converges the dominant eigenvector, and Amax R |
We estimate the trace of the Hessian and part of its spectrum using its connection to the Fisher Information Matrix near minimum. Specifically, when H" is local minimum and the model is well-calibrated, the Hessian can be approximated by the Fisher Information Matrix:
F(0*) = E( (so( y)sy (ar, v)]
so v) Ve log po (x)
(6)
6-
where s0(2, y) is the score. This equivalence discussed further in the Appendix A.2 this expression, we can compute the trace of the Fisher-and hence approximate the trace of the Hessian- by summing the diagonal elements of the outer product sosg.
From
As third measure, we compute the Fisher matrix on a small random subset of the training dataset of size E, and perform singular value decomposition (SVD) on the resulting score matrix, which has shape N x (C),,, where N is the number of parameters and C the number of classes. This procedure efficiently estimates the leading components of the curvature spectrum without requiring full-batch computation or explicit construction of the full Hessian matrix.
3.3. A NOTE ON REPARAMETERIZATTIO
Dinh et al. (2017) showed that symmetries in the architecture of networks allow deep networks to be re-parameterized without changing the function computed by the network. While this observation potentially makes the Hessian a poor tool for studying generalization, we note that when considering SGD optimization dynamics
still the Hessian and not reparameterizztioo invariant measure that governs the dynamics of the system. Particularly, for any symmetry Ta that leaves the function computed by the network the same, we have that V.L(T,6) 0) and flat directions induced by symmetry do not induce gradient.
4 RESULTS
4.1 ENTROPIC CONFINEMENT
In Figure 2. show the loss (C) and the curvature- quantifee by the the trace Tr(H) (A) and the maximum eigenvalue Amax (H) (B) of the Hessian-along MEPs connecting different pairs of minima of Wide ResNet-16-4. Interestingly, the loss along the MEP often lower than at the endpoints. This behavior likely arises because each pivot is pulled downward both by the loss gradient (locally minimizing energy) and by the coupling to the neighboring pivots. This effectively lowers the noise experienced by the system effectively allowing to reach deeper minima. Despite the absence of loss barriers along the MEPs, we observe sharp rise curvature along the MEED
'A small dip is visible near the endpoints Figure B)B), where estimated maximum eigenvalue of the Hessian brietly decreases. We believe this artifact is due to the estimation procedure: computing the Hessian away from an exact minimum introduces a correction proportional to the norm of the gradient. This effect is stronger at the ends of the MEP where the loss is slightly higher. Interestingly, this dip is not present in the estimates based the Fisher Information Matrix (panel (A) and (D)
5

---

Published as a conference paper at ICLR 2026
(A)
(D) 30
8000
pivor 0-33
(70)
pivot 1-33
6000
MEP 1-2
MEP 2-3
25
pivot 3-33
pivor 17-33
4000
pivot
20
32-33
60
eigenvslue
pivot 33-33
15
(C)
20
10-
Loss
0.0015
5-
0.0010
1.0 0
0.0
0.6 0.8
10P
10
10P
Relative position on the MEP
rank
Figure 2: Entropy induces barriers between minima. (A,B) Curvature along minimum energy paths (MEPs) connecting different minima, measured via the trace of the Hessian (A) and the maximum eigenvalue of the Hessian (B). Numbers indicate distinet minima found via independent training runs, markers indicate pivot points; different colors correspond to different pairs of minima, and marker shapes denote MEPs found via different instantiations of the AutoNEB random seed. (C) Cross entropy loss along MEPs connecting different pairs of regular minima. (D) Spectrum of the Hessian along MEP 1-2, estimated singular value decomposition (SVD) of the score matrix computed on E' = 1024 training examples. As we move into the interior of the MEP, the entire spectrum shifts upward, reflecting an increase the curvature in all directions along the path.
measured either via Amax or the Hessian trace. The curvature decreases only near the endpoint minima. As argued in Section 2 such variations in curvature generate entropic forces that bias optimization toward flatter regions,
in the absence of explicit loss barriers.
One might argue that the increase in sharpness along the MEP is simply a consequence of the decreasing loss effective reduction in learning rate, especially given prior work suggesting a relationship between sharpness and learning rate (Cohen et al.) 2021). We argue that this is not the case: while the loss drops between the first and second pivots, it then remains approximately eigenvalue and
constant along the rest of the MEP. In contrast, both sharpness rise. This indicates that the observed increase in curvature is not merely a byproduct of lower loss or implicit regularization, but rather reflects a genuine change in the geometry of the optimization landscape.
trace of the ceassian-conting
4.1.1 MEASUREMENT ENTROPIC FORCE
To directly observe these entropic effects, we initialize models specific points along given MEP and study how stochastic gradient descent pushes them along the path. We use a variant of SGD that projects updates back onto the nearest linear segment of the MEP, ensuring that dynamies remain constrained to the path (see Section A. for details). Without this projection, standard SGD causes the models to leave the MEP and wander along other directions
the loss landscape that are not aligned with the MEP. This projected training variant is only used for the experiments in Figures and R In this context, to fairly compare experiments that use different learning rates, we plot results against an effective time, defined as the product of the number of optimizer updates and the learning rate, teff = (optimizer updates) x 77.
As shown in Figure A[A), when a model is initialized along the MEP, it is pushed back toward the nearest (and relatively flatter) endpoint of the path. Models starting deeper within the MEP take longer to relax to the endpoints. We also note that entropic forces drive the optimization back towards the first pivot despite the fact that the loss actually increases along this path, illustrating a scenario where entropic force is stronger than energetic force. This observation can be understood

---

Published as a conference paper at ICLR 2026
(A)
(B)
(C)
0.176
relative Enelidian position on MBP
relative Enelldiia position on the MEP
relative Enelidian position on the MEP
0.8
0.150
0.6
0.125
0.100
0.076
0.2
0.060
starling point
batch
0.0
2-3
27-28
22-25
batch wise 24
bateh sine 32
0.025
0.01
0.008
16-17
batch 64
0,000
0.02
10P
10
102
10a
100
102
10
10
10"
10'
10
10'
Figure 3: Relaxation dynamics induced by entropic forces. (A) Relaxation dynamics along the MEP for Vanilla projected SGD (batch size 16, learning rate 0.02) for models initialized different points along the MEP (We use MEP 1-2 from Figure 2). The legend shows the two closest pivots each starting point. Models initialized deeper along the MEP take longer to relax the endpoint. (B, C) Models are initialized between the second and third pivots of the MEP, and trained using projected SGD constrained to the path (see Section A. The y-axis shows the relative Fu- clidean distance along the MEP, where and correspond to the endpoints of the path. The entropic force drives the models back toward the endpoints. (B) Models trained with learning rate v 0.02 for increasing batch sizes. Relaxation to the endpoint is faster for smaller minibatches, indicating that entropic forces are stronger for smaller batch sizes. (C) Models trained with minibatch size 16 for increasing learning rates. Relaxation to the endpoint is faster for larger learning rates, indicating that entropic forces are stronger at higher effective temperatures. Different curves the same color correspond to different realizations of the SGD noise.
(A)
relalive Euclidian position the MEP
0.08
0.08
0.07
0.06
relative Euclddion position on
0.06
0.05
0.05
0.04
reference
SGD
0.03
sdam lt 2-
0.03
wdam lr 1--08
4o-04
2e
0.01
nesterov 1e-02
0.01
nesteroe
10a
10-1
10P
10a
optimizer updates
tuF
Figure 4: Relaxation dynamics induced by entropic forces for different optimizers. Relaxation dynamics along the MEP for projected dynamics using Adam (green) and SGD with Nesterov mo- mentum (red), compared vanilla SGD (black). We plot the results against the number of updates (A) and the effective time (B). The effect of the entropic forces seems to be more prominent for both Adam and SGD with Nesterov momentum.
7

---

Published as a conference paper at ICLR 2026
(A)
(C)
(D) 10
Init
Spliting Epoch: 0
Lass
Spliting Epoch: 15
Max Heseian Expnnaauun
Loss
10-1
Solitting Epoch,
Splitting Epoch
10"3
Spliting Epoch:
10a
Instabilitt
P
10k-
(B)
Amon
10.-
Moan
101
10r
z013
10'
30
Splitting spoch
20
0.0
0.6
40
Relative position along linear path
Splitting epoch
Figure 5: Entropic barriers are relevant later in training. (A) Linear mode connectivity schematic (Frankle al., 2020). We train network to epoch k, then produce two networks via different data ordering, and measure the loss along a linear path. (B) The average loss along such a path goes down as k increases, decreasing rapidly with k. (C) Top: The loss profile along linear paths for various k. Bottom: The curvature profile, measured by the maximum Hessian eigenvalue, for various k. (D) We plot the instability (The relative change along the path) of the loss and the curvature. For small k the loss exhibits larger instability, while for larger k, the curvature exhibits larger instability.
more easily through a statistical physics lens: the noisy dynamics drive the system to minimize not the energy but the free energy the system balances the effects of energy and entropy.
In Figure BB) and (C), we show how minibatch size and learning rate affect the dynamics along the MEP. As expected for a genuine entropic force, its strength increases with the effective temperature. Accordingly, relaxation is faster for smaller minibatches, as shown in Figure B), and for larger leaming rates (Ir), as shown Figure B.))
C). In the Appendix, Figure we show explicitly how the entropic force scales with the batch size.
In Figure 4 we investigate how the choice of optimizer affects the entropic force. We show that (projected) Adam and SGD with momentum both respond more strongly to changes in curvature than vanilla SGD. This suggests that the effect of entropic forces may become more important when using adaptive optimizers using momentum.
The increase of the curvature along the MEP adds nuance to the idea that the loss landscape consists of one large "valley" containing all the parameter configurations with low loss: Although minima in such a valley may be connected energetically, our experiments suggest that such a valley is ef- fectively broken up into disconnected regions by entropic barriers. We emphasize that entropic "barriers" are not barriers in the most literal sense of the word: the model is not dynamically forbid- den from crossing such barriers. Rather, the noisy dynamics ensure that crossing an entropic barrier is statistically extremely unlikely. We therefore say that the model is "effectively forbidden" from crossing such barrier.
4.2 LINEAR MODE CONNECTIVITY
Although we have argued that entropic forces separate the low-loss region of parameter space into regions effectively confined by entropic barriers, we have not yet addressed how and when these confined regions are chosen along the course of training. In this section we will take steps towards answering this question through the lens of linear mode connectivity. Following the methods of Frankle et al. (2020), we train M networks with shured data order up until epoch k; which we will call the splitting epoch. After epoch k, each of the M networks sees an independent ordering of the data and can then potentially move away from its "siblings." the other M 1 networks. The sibling networks are then trained until convergence. All networks trained in this section use the ResNet-20 architecture (He et al. 2015), unless otherwise noted.
The crucial observation in Frankle et al. 2020) is that once l becomes sufficiently large, the sibling networks become connected by linear paths of low loss, implying that they converge the same region of parameter space. Interestingly. & does not have to be very large compared to the number of epochs required for convergence before linear mode connectivity is observed. In Figure 5
8

---

Published as a conference paper at ICLR 2026
(A)
(8)
10
Loss
Max Hessian Eigenvalue
Loss
10-1
10-2
Instability
10a
10
Amax
10
Spiltting Epoch.
Spiltting Epoch: 16
10a
Splitting Eppch::
Spiliting Reoch 41
10
0.0
0.2
0.4
40 Splitting epoch
Relative position along linear path
Figure 6: Entropic barrier behavior persists across datasets & architectures. (A) The average loss along linear interpolating path goes down the splitting epoch k increases, for a ResNet-110 trained on CIFAR- (B) Entropic barriers become more relevant late training for a ResNet- 110 trained CIFAR-100.
reproduce these experiments, and measure the curvature along the linear, low-loss paths between converged siblings. Similarly to the nonlinear case (Section 4.1. we see bump in the curvature along these paths. Additionally, we notice that these entropic barriers persist for larger values of than their energetic counterparts, implying that entropic forces contribute relatively more to the final stages the model's localization a region of parameter space. To see this, we plot the instability function of & (Figure 5D) The instability measures the relative change, max/min, in the metric (loss or curvature) along the linear path. For small values of k. the loss has the larger instability, while for larger values of the curvature exhibits greater instability.
In Figure 6 we show that this behavior persists across datasets & architectures, and repeat our analysis for ResNet-110 trained CIFAR-100. We similar behaviors in the loss and in the curvature (Figure (6) across both datasets, emphasizing that these trends are not dataset-specific. We also see similar behavior Figure for CIFAR-I00, where entropic barriers become relatively more important over the course of training (See Figure 6B). In the Appendix Figure 9 we show that a ResNet-20 trained on CIFAR- 100 also exhibits similar behavior.
DISCUSSION
Entropic Confinement. Our results provide new insight into the global geometry of the loss land- scape. While prior work has emphasized that minima are often connected by low-loss paths, forming single broad "valley" of solutions (Garipov et al.) 2018 Frankle et al. 2020), our findings reveal that these paths are not flat when entropic forces are taken into consideration (Figure Instead, they exhibit systematic increases in curvature away from their endpoints, producing localized "bumps" in sharpness. This observation refines the valley picture: the basin of low-loss solutions is structured by curvature variations that give rise entropic barriers.
We show that the forces produced by curvature variations along connecting paths consistently drive optimization dynamics back toward flatter regions near the minima (Figure In particular, models initialized away from a minimum but constrained to remain on the path show persistent drift back toward the endpoint, even though the loss profile nearly flat. We also observe that smaller batches and larger learning rates accelerate relaxation, showing that the strength of the entropic force de- pends on the noise level. Entropic forces are not necessarily negligible we show empirically that they can drive models up loss gradient.
Entropic Linear Mode Connectivity. Our analysis of linear mode connectivity further shows that entropic forces play an important role late in training. As the splitting epoch increases, energetic

---

Published as a conference paper at ICLR 2026
barriers along linear paths decrease, but curvature barriers persist for longer into training (Figure Figure 6). This suggests a two-phase picture of training: early dynamics are dominated by energetic forces that drive the model into a low-loss basin, while later entropic forces become more rele- vant. Our experiments have important implications for late-time dynamics of deep network training, basin selection, and parameter-space ensembling techniques. In particular, earlier work including Altuntas et al. (2025), demonstrates that the final basin network ends up in is highly sensitive to perturbations the weights, especially early in training. These findings suggest that even small contributions from entropic forces could have an outsized effect on the model's long-term fate.
Confinement and Generalization. Our findings may also provide insight into generalization properties of overparameterized models. Empirically, models trained with SGD tend to find a gener- alizing solutions and not overfit the data, even after many epochs of training. This occurs dexpite the fact that the loss landscape is energetically flat, raising the question of why optimization dynamics do not diffuse into regions of parameter space that overfit the training data.
We posit that generalizing minima may be effectively disconnected from overfit minima via en- tropic barriers. Entropic barriers could make paths to such regions effectively inaccessible: even when overfitting solutions are connected to flatter ones by low-loss paths, entropic forces could shield the generalizing solutions by repelling SGD away from regions of parameter space that do not generalize. Our results suggest that this is promising avenue for future work. In fact, there is evidence that models in similar convex basins of attraction share generalization properties (Juneja ct al. 2023).
Weight-space averaging. Our work also provides new lens through which view weight-space ensembling techniques. The study of global loss landscape features, such as mode connectivity (Draxler al. 2018 Frankle et al.] 2020). has been erucial in developing methods like Stochastic Weight Averaging (SWA) (Izmailov et al., 2018 Wortsman et al.] 2021). Our findings suggest more nuanced picture the global landscape: techniques like SWA may be averaging minima that, while energetically connected within single low-loss valley, may be effectively disconnected by the entropic barriers we observe. This would imply that the SWA solution cannot be easily found by diffusive optimization dynamics at the bottom of valley in the loss landscape. A valuable avenue for future work would be to analyze the connectivity properties of these averaged minima to better understand how weight-space averaging is able construct solutions with favorable generalization properties.
Limitations & Future Work We note that there is a large space of low-loss paths connecting min- ima, and that the methods used here to find such paths (AutoNEB & linear interpolation) introduce a source of bias in the paths consider. While we acknowledge that this bias could impact the gen- erality of our conclusions, we also observe similar qualitative profiles in the curvature across these two methods, even though they introduce different sources of bias. We believe it is an important and promising direction for future work to investigate how to sample the space of paths in a more principled manner.
6 CONCLUSION
We identify key geometric feature of neural network loss landscapes and its impact on optimization dynamics. Our central finding is that low-loss paths connecting distinct minima consistently exhibit a rise in curvature away from their endpoints. We show that this variation, when coupled with the inherent noise of stochastic gradient descent, gives rise to entropic barriers. We demonstrate empirically that these barriers generate effective forces that confine the optimizer to flatter regions near the minima, even when the path is energetically favorable.
Our experiments exploring the curvature along linearly mode-connected networks reveal that the mechanism of entropic confinement is particularly relevant during the later stages of training, shap- ing the final localization and stability of the learned solution. Our results establish these curvature- induced forces key element in understanding the behavior of stochastic optimizers. This geo- metric perspective offers new insights into how the landscape itself guides the discovery of stable and well-generalizing models, providing a promising direction for future research.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement. We do not foresee any direct ethical concerns arising from this work. Our study focuses on better understanding optimization in machine learning, without direct deployment in sensitive application domains. We note that have used language models to polish the text of this manuseript in places.
Reproducibility Statement. We have provided descriptions of all algorithms, models, and experi- mental setups in the main text and appendix. Training procedures and dataset details are documented to facilitate replication. When the author list is unblinded, we will release our codebase to enable full reproducibility of our results.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models mod- ulo permutation symmetries. In The Eleventh International Conference on Learning Representa- lions, 2023. URL htttt:///peenreeeii www.ewwwweeeee....eee wor/moormmmmmmm 3dd=:0smmmmmmmPPP
Gül Sena Altutaş, Devin Kwok, Colin Raffel, and David Rolnick. The butterfly effect: Neural net- work training trajectories are highly sensitive to initial conditions. In Forty-second International Conference on Machine Learning, 2025. URL https: httpp::/openrerriitt /ooo/oppeeviiiieeee
net/ / forum? ? id=
1.1
Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, GÅ@rard Ben Arous, Chiara Cam marota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing dynamics: deep neural networks versus glassy systems*. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124133, dec 2019. doi: 10. 10.1088/1742
URL //tx:://dxxdd. .org/
Jeremy Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. URL https: http://////ppeeeeeeeeeeeeeeeetee fow.://t/oooendd///e=ee-
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. International Conference on Machine Learning, pp. 1019-1028, PMLR, 2017.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1309-1318. PMLR, 10-15 Jul 2018. URL httpt:tt////.. /tpps://pree httt:s://proceedinggs mlr.
html
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. Proceedings of the 37th International Conference on Machine Learning, ICML'20.. JMLR.org. 2020.
Jcan-Christophe Gagnon-Audes Ricardo Pio Monti, and David Schwab. AWE: Adaptive weight- space ensembling for few-shot fine-tuning, In ICLR 2023 Workshop Mathematical and Em- pirical Understanding of Foundation Models, 2023. URL
PirrrM I
Crispin W Gardiner et al. Handbook of stochastic methods, volume 3. springer Berlin, 2004.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wil- son. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httppppss//prrroeennn...errrrrr File Bll.app0rrceeellloab--eeeeebbba3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL.

---

Published as a conference paper at ICLR 2026
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. CoRR, abs// 1803.05407, 2018. URL hRtp::///aaxix org/ abs/ 1803. 05407
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Ben- gio, and Amos Storkey, Finding flatter minima with sgd, 2018.
net Dortmo 2in waa.d-rr1F99CCCC
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connec- tivity reveals generalization strategies. The Eleventh International Conference on Learning Representations, 2023. URL https://opeettee http:///eeeeeeeeneeettte.ee net
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. https:/ /wppp/eeeiiiieeeeeeeeeeeeeeeeeee
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. In International Conference on Machine Learning, pp. 7045-7056. PMLR, 2021.
Stephan Mandt, Matthew Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1-35, 2017.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 22045-22055. Curran Associates, Inc.,
2020. URL httpps://proceeettttttt neurips wwterrisscccpppe cc/paper files/paper/2020/ Fwle/ftelelliee (tl.://22766110664000ceee2222223 pappeeeeeeeeeee.oooaaaea.eeaeeee pdf
Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard Schölkopf, and Thomas Hofmann. Landscaping linear mode connectivity. In High-dimenssoonal Learning Dynamics
2024: The Emergence of Structure and Reasoning, 2024.
Forum?id Derumm?d=OSSMqvPP11..
Samuel Smith. Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning, pp. 9058-9067. PMIR, 2020.
Samuel L Smith and Quoe V Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
Mingwei Wei and David J Schwab. How noise affects the hessian spectrum in overparameterizee neural networks, 2019. URL httpt::ttp///tttee www.aaiwww....ooooooooooooo http://bs///111. 101bb///991001995
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903 2021.
httts::/aaxxi http:///axxivoorg/aa////////
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Lud- Wig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accu- racy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965 23998. PMLR, 17-23 Jul 2022. URL https: http::://pxocceeingg .mlr. press/ v1 62. 222/wor.smaa22... html
12

---

Published as a conference paper at ICLR 2026
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamies: Stochastic gradient descent exponentially favors flat minima. In International Conference on net wtt.:/neee//ommmmdid/ee.....
Learning Representations, 2021. URL
iCoYGo
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp. 581-590. IEEE, 2020.
Sergey Zagonuyko and Nikos Komodakis. Wide residual networks. CoRR, abss/6655007444 2016. URL http:://axiiii wwtp:/ggggabbbaaaeee htt.//11555000158888
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity, 2023. URL https://arxiv.orr
abs/2307.02235
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength minibatch noise in sgd. arXiv preprini arXiv:2102.053733 2021.
A APPENDIX
A.1 k-STEP PROJECTED SGD
In order to directly measure the effect of entropic forces in a controlled setting, we use a modified version of SGD. Our algorithm deals with two conflicting considerations: First, we would like to limit the scope of our observation to models that lie on a linear path, or more generally models that lie on MEP. However, we would also like run optimization in such a way that entropic forces arising from curvature are still relevant to the optimization dynamics. The key observation is that if we were to run SGD on a line in parameter space, projecting back the line after each optimization step, we would remove the effect of entropic forces, which arise from noisy multi-step optimization dynamics |Wii Schwab (2019). Motivated by this, we propose a natural algorithm that trades off between these two considerations by taking multiple SGD steps between before projecting the parameters back to the liner path (or MEP).
Algorithm k-step projected SGD
Input: A model fo(r), a loss function L(0, x, y). an integer k, path pivots t,, 01 ... ON SGD learning rate 77. SGD batch size B. while not converged do for i to i do
Draw a batch b + {I,, Vi U; ~ D
0 - 0 v/ L) end for Project 0 onto the closest segment (across n²) connecting On and 0 end while
In this way, trades off the effect of entropic forces (large k) vs how close to the linear, low-loss path optimization stays (small k). In Figure use & = 15 and run the algorithm along the MEP 1-2. The number steps plotted on the horizontal axis is the "raw" number of SGD steps i.e. the number of times the parameter vector of the network updated.
A.2 THE FISHER TRICK FOR ESTIMATING THE HESSIAN
Computing the full Hessian of the training loss is intractable for modern neural networks due to both memory and runtime constraints. The Hessian matrix has O (N3) parameters, where Np is the number of parameters of the network, hence even just computing and storing the Hessian matrix is prohibitive. A common workaround is to exploit the equivalence between the Hessian of the loss and the Fisher Information Matrix (FIM) at minimum.
13

---

Published as a conference paper at ICLR 2026
If, in the case of image classification, the loss is the negative oog-likelihod,
(()) -- Σ
logpo x).
(2.1)ED
At any parameter vector that minimizes the loss function [()), we have that (yl()) [log(po(yl=))) 0, taking the derivative of this equation with respect to 0 and using the log-derivative trick we have:
(pl()))) Vo (og(p(()(==)] Epllv m)
Vapo [r]] (ul [V log(po(y)>)) Vo log(po (M) [))
(7)
Therefore at any minimum 0 of the loss, the Hessian the loss coincides with the Fisher informa- tion matrix F(()).
F(0*) E (m.a)DD Vo log po x) V Po x))
This identity allows approximate Hessian eigenvalues using stochastic estimates of the FIM. In practice the FIM is easier to estimate than the Hessian, since can be decomposed into a product of low-rank matrices.
A.3 SUPPLEMENTARY FIGURES
(A)
(B)
3.5
25
2.5
20
15
1.5
10
5
0.5
Relative position on MEP
Relative position on MEP
Figure Properties of MEPs constructed with AutoNEB. The AutoNEB algorithm updates the positions of pivots without changing the lengths e of the segments connecting them, and adds new pivots along segments where the loss is not well-approximated by the linear interpolation, see Draxler et al. (2018). As result, the Euclidean distance between consecutive pivots is not constant, and pivots tend to be denser near the middle of the MEP. (A) Relative Euclidean distance of each pivot measured from the first pivot, illustrating the cumulative distance along the MEP. (B) Lengths of individual segments between consecutive pivots, showing that the spacing A is non-uniform. Note that the values in (A) correspond to the cumulative sum of the segment lengths shown in (B).
14

---

Published as a conference paper at ICLR 2026
relative Euclidien position on the MEP
0.10
bs 20
0.08
be 22
0.06
brelxx
102
0.04
10"
107
102
10a
22
21
26
fef
Batch size
Figure 8: Relaxation dynamics induced by entropic forces. (A) Relaxation dynamics along the MEP using Vanilla projected SGD (see Section A. with learning rate v = 0.02 for models initialized at the second pivot of the MEP. Different colors indicate different minibatch sizes, and different curves correspond to different realizations.
(B) Dependence of the characteristic relaxation time, defined as the time required for the relative distance along the MEP to decrease by factor of The relaxation time appears to well described by a growing exponential.
(A)
(B)
Loss
Max Hossian Eigenvalue
Loss
10-1
10a
Instability
10
Amax
10
10
Splitting Epoch:
Splitting Epoci:
Spliting Epoch: 25
Splitting Epoch:
0.0
0.2
0.4
0.6
0.8
1.0
20
40
60
Relative position along linear path
Splitting epoch
Figure 9: Entropic barrier behavior persists across datasets. (A) The average loss along a linear interpolating path goes down & increases, for ResNet-20 trained on CIFAR-100. (B) Entropic barriers become more relevant late in training for a ResNet-20 trained on CIFAR-100.
15


Reviews:
[{"review_id": "gBpkQroZpB", "reviewer": "Reviewer_rx5A", "summary": "Minibatch training introduces stochasticity, which is also modulated by the discrete step size.\nThis paper studies so-called *entropic forces*, an implicit bias that progressively nudges optimization toward flatter regions of the loss landscape, strengthening as update noise increases.\nSeen through the lens of mode connectivity, this driving force is proposed as an dynamic barrier between disconnected minima regions.\nIndeed, by sampling low-loss paths between minima with AutoNEB, they observe that curvature rises away from the endpoints.\nWhen models are initialized slightly off these endpoints, path constrained dynamics pull them back, revealing a curvature-driven “gatekeeping” effect that intensifies with smaller batch sizes.\nReusing a setup of linear mode connectivity, the authors further argue that the relative strength of the entropic force grows in the later stages with respect to the loss.", "questions": "1. **Subsection 4.1** you mention that the loss drop may result from the elastic coupling between pivots. Could you elaborate on how this coupling contributes to a further decrease in loss?\n1. **Figure 3 setup**: what behavior do you observe if the parameters are not projected back onto the MEP during training? This question seems important to assess the scope of entropic forces in a real training process for example.\n1. **Figure 2 curvature plateau** – Around the midpoint of the MEP, curvature metrics appear nearly constant, with even a local minimum of $\\lambda_{\\text{max}}$ on Fig 2B. . Should we interpret this as a region where entropic forces vanish, leaving the parameters unbiased toward the endpoints? Have you tested initializations in this region?\n1. **AutoNEB bias** AutoNEB minimizes both loss and path length: \"the spring energy grows quadratically with the total length of the path\" (quote from Drawler et al). Could this introduce a bias toward paths that wrap around high-loss regions or exhibit higher curvature? How sensitive do you believe your findings are to the specific properties of the AutoNEB algorithm?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This paper presents an interesting study curvature-induced biases in neural network optimization.\nIt offers valuable insights into how stochasticity combined with geometry can influence the parameter dynamics, mostly towards the end of training when the loss' role is lessened.\nThe paper contributes to explain why certain solutions, though equivalent from the point of view of the train loss, are preferred and therefore challenges our understanding of the implicit biases of optimizers.\nThe results could have implications for late-stage optimization dynamics, low-loss manifolds of solutions, stability within the zero-loss regime, and connections to phenomena such as double descent or grokking.\nAn original point of the paper lies in the fine-grained, directional analysis of local curvature effects beyond more general arguments given in previous literature (e.g. Edge of Stability, catapult).\nThe work also raises an interesting question on overfitting, that could be countered by such entropic forces and hence foster better generalization.\nIn that sense it might also shed more light on why flatness measures have been successful despite not being principled.", "weaknesses": "While the paper is conceptually interesting and seems technically sound, it suffers from awkward or overly elliptical phrasing, despite no apparent space constraints (see questions and minors below for specifics).\nThis gives the impression that the manuscript was not thoroughly reread, which also somewhat undermines confidence in the results.\n\nThe paper sometimes seems to overreach:\n- line 155: The reference to “the constant loss non-linear path” overstates uniqueness. In addition, identifying paths with the Auto NEB algorithm means incorporating any bias towards particular properties these paths might have, and this is not discussed. (see my question about it below)\n- line 320 & 406: the claim that the entropic forces are responsible for the localization of the parameter when the *splitting epoch* is high appears is not fully supported.\nFigure 4 only suggests a relative increase in the importance of entropic forces, not that they overtake loss dynamics. A clearer analysis of the trade-off between energetic and entropic components would strengthen this point.\n\nInterpretation of Fig 2 c.: while earlier works have shown that intermediate models can outperform endpoints (e.g. Model Fusion via Optimal Transport by Singh and Jaggi or Git Re-Basin: Merging Models modulo Permutation Symmetries by Ainsworth et al), the abrupt loss drop in Figure 2(c) seems unusual. The explanation given by the authors line 260 is not very convincing as it should also applies in earlier works published since the work by Draxler et al. (see also my related question below)\n\nExperimental details are reported but code is currently not provided.\n\nLimitations include:\n1. not discussing quantitatively the tradeoff between entropic forces and loss which makes it difficult to quantify the effect of entropic forces on a typical training\n1. while the batchsize point of view is interesting, it would have been great to also investigate the learning rate since it is mentioned as the other source of stochasticity in section 2\n1. while making broader connections and mentioning potential applications, the paper focuses only on the mode connectivity/linear mode connectivity setup\n\n## Minor and additional feedback\n- duplicated paragraphs in 4.\n- line 621 trimmed or spurious full stop before \"in practice\" ?\n- line 233 equation ??\n- fig 2: plots inverted w.r.t. caption\n- line 93: $\\eta$ and $B$ are not introduced\n- line 100: isn't there a missing $T$ in the variance formula ?\n- line 70: when the loss is near $0$ is a bit imprecise, is it \"when the loss is near zero along the path of solutions\"\n- line 100 and equation 2: I think it would be better to adopt a consistent notation for the variance\n- the whole toy model example is a bit too elliptical (also while $g_y$ is understandable is it not clearly defined and adds to the reader disambiguation effort)\n- line 208 missing link (?)\n- The sign of $V_{eff}$ might be wrong in equation $3$ since it leads to $P(y)\\propto \\text{exp(ln }g(y))$ and so the dynamics would be attracted towards high curvature region instead of repulsed.\n- $m$ of figure 3B is called $\\rho$ in the main text \n- abstract: \"yet optimization dynamics remain confined to one solution\" different parameters can be sampled at the end of training so this is a bit unclear\n- line 130: 'entropic' is spelled 'entorpic'\n- line 249: 'affect' is written instead of 'effect'\n- line 596: \"on along\" is redundant", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCQzzhCS1h", "reviewer": "Reviewer_im1k", "summary": "This paper provides a dual view on why neural network optimization remains localized near certain minima despite low-loss connecting paths between different solutions. The authors investigate the training dynamics using the AutoNEB algorithm (Draxler) and show that while energetic barriers (arising from the loss) remain low along these paths, entropic barriers emerge from the interaction between curvature variations and stochastic gradient descent noise, manifested as a \"curvature bump\" away from minima. They demonstrate that these curvature-induced entropic barriers persist longer than traditional energetic barriers and play a key role in determining which regions of parameter space are dynamically accessible during training.", "questions": "- I believe the pivots in the AutoNEB procedure are initialized equidistantly along a linear interpolation between minima (following Draxler et al.), but the subsequent relaxation and potential addition/removal of pivots means they no longer maintain equal spacing. Could the authors clarify: how does the relative position translate into the relative metric position between the two points?  \n- The existence of entropic barriers along low-loss paths raises intriguing questions about weight averaging techniques like Stochastic Weight Averaging (SWA), model soups, and linear mode connectivity ensembles. If curvature increases along connecting paths between minima, what does this imply for intermediate points obtained by weight averaging?\n\nMinor comments\n\n- The phrase \"bottom of the loss landscape\" sounds somewhat unnatural, you could consider using more precise terminology like \"low-loss regions,\" \"basins of attraction,\"  \n- Missing refs on Lines 208, 233  \n- First two paragraphs of 4.2 repeated twice verbatim   \n- MEP 1-2 could be clarified in the figure", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides an interesting and fresh perspective on how optimization is biased towards certain solutions, driving interesting connections to the implicit bias of SGD (though the relationship to the literature could be expanded here)  \n- The paper is sound. They use trace and max. Eigenvalue of the Hessian and using the SVD of the Fisher, providing robust evidence that curvature systematically increases along connecting paths.  \n- The observations relating to the spawning experiments in Frankle are particularly insightful. I would encourage the authors to expand on this connection and engage deeply with the literature on the chaotic early phase of neural networks (e.g. Fort et al, 2020, Altıntaş et al, 2025). The controlled noise setting in Altıntaş could be an interesting addition to this work.", "weaknesses": "- While the core findings are convincing, the work would benefit significantly from additional ablation studies. Given that the experimental setting (ResNet-20, Wide ResNet-16-4) is not prohibitively expensive, several natural extensions would strengthen the paper:  \n- How do entropic barriers manifest when learning rates are reduced? Do they become weaker (less noise)?  \n- Algorithms like K-FAC, Shampoo, or natural gradient descent explicitly account for curvature. Do these methods navigate entropic barriers differently?   \n- I think an experiment in at least Cifar-100 scale would be needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhQtW4ZUXq", "reviewer": "Reviewer_ukzS", "summary": "This paper explores the paradox that neural network minima are connected by low-loss paths, but SGD confines itself to individual solutions. The authors propose that entropic barriers, following from the interplay between curvature of the landscape and noise in the optimization, explain the confinement. They run experiments on ResNet architectures trained on CIFAR-10 to demonstrate that the curvature systematically increases along low energy paths between minima. This creates effective forces that bias SGD back to the low curvature endpoints. They show that the magnitude of these forces inversely scales with batch size and argue that these forces persist longer than energetic barriers, becoming dominant in later stages of training.", "questions": "1. Why does starting from 0.7 relaxes the minimum in relative position 0 in Fig.3A? Does it converge to the one in position 1 for higher initializations? It would be interesting to have a more complete profile of what's going on for different starting points (see comments above).\n2. What do you think would happen if you initialized in the middle of a path and then perform SGD without projecting to the path?\n3. Do the curvature bumps persist when regularization is added to the loss?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper's motivation is well presented and the introduction/literature review clearly set up the problem.\n- The paper connects two well-known phenomena: mode connectivity and the low curvature bias of gradient-based optimizers. These two nicely merge in the paper's analysis, giving an intuitive and convincing picture that the proposed explanation is actually behind the  phenomenon observed. \n- I think that the experimental setups are interesting and well-designed. I particularly like the idea of initializing in the middle of the path and looking at the relaxation when performing gradient descent projected on such curve.", "weaknesses": "1. My main concern with this work is that, while the ideas are very interesting, the experimental validation is too limited in scope to fully validate it. Given that the paper makes the strong claim that these energy barriers are a \"univeral\" phenomenon, the experiments should definitely be performed on more than two architectures (Wide ResNet-16-4 and ResNet-20) and one dataset (CIFAR 10) like the paper currently does.\n\n2. Overall, the feeling I get from reading the paper is that a lot of space is used for things that are not central to the argument, like the details about curvature measures and their computation (which could have been an appendix) and the toy model that, in its current form, doesn't really help in conveying the idea (more below).  This space, in my opinion, would have been better spent by expanding the experimental setup.\n\n3. It is not clear how the analytical toy model helps in understanding the phenomenon. First, the derivation feels disconnected. It is not clear how the authors go from Eq. 2 to Eq. 3. Expanding in an appendix would help but the main text section should still be adjusted to be more readable. Furthermore, I don't understand what we are getting from the analytical part of toy model described in page 2. Unless I am missing something, the main message i.e. \"regions with smaller curvature $g(y)$ (flatter directions in $x$) contribute larger entropy and are statistically favored, even if the original energy $V(x, y)$ is minimized elsewhere\" doesn't seem to follow directly from the analytics. It is however clear enough from Fig. 1B.\n\n4. The claim that \"entropic barriers persist longer than energy barriers\", which is based on Figure 4D seems a bit overreaching given that it results from only one set of experiment.\n\n### Minor weaknesses\n\n1. I think that the sentence at L249, stating that \"symmetries do not have any affect on optimization dynamics.\" is incorrect. There are plenty of works that study the relationship between symmetries and optimization dynamics. To mention only a few, see [1-3].\n2. \"delta correlated Gaussian noise\" undefined at L091\n3. In eq. 2 $g_y$ is the derivative of $g$ w.r.t. $y$?\n4. Paragraph duplicated at line 357\n5. Figure 2 caption has a ??\n6. Page 4 has score(?) after equation 5\n7. Missing space at line 56\n\n\n\n- [1]  Du, Simon S., Wei Hu, and Jason D. Lee. \"Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced.\" Advances in neural information processing systems 31 (2018).\n- [2] Kunin, Daniel, et al. \"Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics.\" arXiv preprint arXiv:2012.04728 (2020).\n- [3] Zhao, Bo, et al. \"Symmetries, flat minima, and the conserved quantities of gradient flow.\" arXiv preprint arXiv:2210.17216 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CoRrqvQ7gF", "reviewer": "Reviewer_zHdP", "summary": "The paper describes an entropic mechanism by which stochastic optimization is biased towards flatter minima. The proposed mechanism is that flatter minima take up larger portion of weight space, and since training is noisy, entropic considerations would predict that the posterior distribution of network parameters would be localized in low-curvature zones.  This extends a line of research about the implicit bias of various optimizers, the connection between local curvature and generalization properties, and the connectivity of the low loss manifold of deep networks.", "questions": "I wrote my questions in the weaknesses box.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "I really loved the central idea of the paper. It offers a simple conceptual model and provides concrete and convincing numerical demonstrations. The question it addresses is timely and important, and the answer is at once original, simple and insightful, even if not rigorously proven. The presentation is relatively clear, though I think much of it is wrongly phrased, see below. I think this is an important contribution in this sub field, that will likely generate interesting follow up works.", "weaknesses": "Despite the praises above, my final recommedation is a weak reject because I think the manuscript has some fundamental flaws, and proofreading is more than sloppy. However, as I wrote in the Strengths section above, I think the central idea is (probably) valid and interesting and that this manuscript would be a good contribution to the field once the flaws are fixed. I'd be very happy to upgrade my assessment pending the authors response.\n\n1. The main weakness I see is in the framing. The terminology used is at times inaccurate and at times plainly wrong and contains what I think are fundamental mistakes in interpertation. However, the general idea is, I think, valid. The manuscript would be much stronger (and my grade would be higher) if this were corrected. I'll try to as explicit as i can with this point:  \n  - The main disagreement I have with the framing has to do with how the authors distinguish energy and entropy in DYNAMICAL terms.  For example, the opening statement of Sec. 2 is that \"the dynamics of a system are governed not only by energetic forces—derived from gradients of an energy or potential—but also by entropic forces, arising from thermal fluctuations\". Similarly, the authors write that \"entropy causes optimization to climb up the loss\" (line 138) or that \"entropic forces dynamically confine models\" (line 72) or similar statements scattered throughout the manuscript.  \n This is a delicate point: The basic tenet of stat mech is that the Boltzmann distribution arises naturally *even if the underlying dynamics is purely deterministic and energy driven*, the canonical example for which is the micro-canonical ensemble, (sorry for the pun). When the authors write that the system is \"dynamIcally confined\" to high entropy regions, they use this term exactly in the opposite meaning of what \"dynamical confinement\" would mean in the stat mech literature -- namely places that have non-vanishing Boltzman weight and that therefore one should be able to probe if dynamics were ergodic. But one cannot probe them because of dynamic (=energetic, not entropic) constraints.\n * Explicitly: in the stat mech jargon, dynamical confinements pertain to *energy* barriers, while entropic confinements pertain to *free energy*. This is a crucial difference, which is central to the authors' main point.  \n(a trivial textbook example: consider an ideal gas in a container. It is *perfectly dynamically possible* for the gas to be concentrated in one half of the container, but such configurations have very low entropy (high free energy) and therefore it is *thermodynamically* forbidden, while being *dynamically allowed*)\n\n- A similar mistake happens when the authors write that \"it is possible for entropic forces to be the stronger than energetic forces, leading to a scenario where entropy causes optimization to climb the loss.\" (line 138). However, this is generically the case. It is *always true* that \"entropic forces\" drive dynamics up the energy landscape (driving them down the energy landscape would result in negative temperature which is not strictly forbidden by thermodynamic, but is quite an esoteric condition which I think is irrelevant here). Even in the toy example given by the authors of a valley with a varying curvature, $V=g(x)y^2$, the basic fact that $\\langle x^2 \\rangle\\ne0$ stems from the fact that at finite temperatures (hence finite entropy) the dynamics drive the system up the quadratic well.\n\nI think both these mistakes, and similar ones, stem from the same basic misunderstanding. This framing should be fixed.\n\n2. The main result of the paper is Fig. 2 and the curvature variation along the MEP. However, I suspect that the custom SGD the authors use might affect this result. At the very least, the authors should present how the curvature along MEP looks like if one uses the method of Frankle /Draxler directly, and show how their result changes with $k$ (defined in appendix A1, and, as far as I understand, is confusingly not the same $k$ as in Fig. 4). \n\n3. There are typos, missing references, and two paragraphs are repeated verbatim twice in Sec. 4.2. Appendix A2 terminates in the middle of a sentence (!). This is, frankly, not serious. I know well how submission to these conferences work, but this almost insulting for the reviewers and ACs and all the people who took the time to read your paper seriously and comment on it constructively. The minimum you could do is read it before you submit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)