Almost Bayesian: Dynamics of SGD Through Singular Learning Theory
Authors: Max Hennick
Stijn De Baerdemacker
Venue: ICLR 2026 Poster (2026)
Keywords: singular learning theory
SGD
gradient noise
gradient descent
Fokker-Planck
training dynamics
Bayes
Bayesian
Primary area: learning theory
TL;DR: We examine the long runtime dynamics of SGD as diffusion on porous media using tools from singular learning theory.

Abstract: The nature of the relationship between Bayesian sampling and stochastic gradient descent in neural networks has been a long-standing open question in the theory of deep learning. We shed light on this question by modeling the long runtime behaviour of SGD as diffusion on porous media. Using singular learning theory, we show that the late stage dynamics are strongly impacted by the degeneracies of the loss surface. From this we are able to show that under reasonable choices of hyperparameters for vanilla SGD, the local steady state distribution of SGD (if it exists) is effectively a tempered version of the Bayesian posterior over the weights which accounts for local accessibility constraints.

Published as a conference paper at ICLR 2026
ALMOST BAYESIAN: DYNAMICS OF SGD THROUGH SINGULAR LEARNING THEORY
Max Hennick
Stijn De Baerdemacker
Department of Mathematics and Statistics University of New Brunswick & TrojAl mhenni & ca
Department of Mathematics and Statistics, Department of Chemistry University of New Brunswick st jn. debae rdemacker wwwarunberrrrrrrreeee
ABSTRACT
The nature of the relationship between Bayesian sampling and stochastic gradient descent in neural networks has been a long-standing, open question the theory of deep learning. We shed light on this question by modeling the long runtime behaviour of SGD as diffusion on porous media. Using singular learning theory, show that the late stage dynamics are strongly impacted by the degeneracies of the loss surface. From this we are able to show that under reasonable choices hyperparameters for SGD, the local steady state distribution of SGD effectively a tempered version of the Bayesian posterior over the weights which accounts for local accessibility constraints. We then empirically verify the porous diffusion picture across multiple models and datasets, and provide experimental evidence of the steady state- Baye-Ba posterior correspondence.
INTRODUCTION
One of the core problems in developing a scientific theory of deep learning models is giving descriptive theory of how the internal model structure evolves during training as the model gains "knowledge" about its training distribution ((McGrath et al.) 2022). (Olsson et al.) 2022 and how this evolution relates to the generalization ability of deep learning models. Classical methods for understanding model generalization such as the Bayesian Information Criterion (Schwarz 1978) fail
to give accurate descriptions of the generalization behavior deep learning, due to its "singular"
nature (Wei al. 2023).
This has lead recent research to utilize Watanabe's 's singular learning theory (SLT) Watanabe 2009)
the basis for studying deep learning models. The key result of singular learning theory is the widely applicable Bayesian information criterion (Watanabe) 2012) which (broadly speaking) says that the generalization error of model with parameter controlled by the learning coefficient X(w), which corresponds to the "complexity" of some local area around the parameter. Measuring how this method to study the emergence of structure within neural networks (Lau et al. 2024). (Wang et al. 2024ah) and has given very promising results.
quantity evolves over time has been proposed
Despite this, it not clear how the dynamical picture of SGD interacts with the purely Bayesian description of SLT. It has been shown that there is seemingly some relationship between Bayesian sampling of parameter space of neural networks and SGD, both experimentally (Mingard et al. 2020).
and theoretically under assumptions of non-degeneracy of minima of the loss (Mandt et al. 2016b)
(which is known to be false in general). Here we extend this connection to the more general case by describing the late stage training dynamics of SGD using fractional Bokker-Planck equation which can be solved explicitly under reasonable assumptions. We show that the steady- solution of this equation is related to the purely Bayesian posterior by tempering probabilities based accessibility constraints determined by the learning coefficient. Potential practical applications of the results presented here are discussed in appendix

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 SINGULAR LEARNING THEORY
Our work relies upon results coming from singular learning theory ( (Watanabe 2012). (Watanabe
2022), |Watanabe| 2024). (Watanabe 2009)) the known relationship between inference and thermo- dynamies (LaMont & Wiggins 2019). and the application of singular learning theory to the study of deep learning. referred to developmental interpretability ( ((Wang et al. 2024b), (Wang et al.) 2024a). (Chen al. (.23)). We make particular use the estimation methods for the local learning coefficient introduced in (Lau et al. 2024) using (van Wingerden et al. 2024)..
2.2 GRADIENT NOISE AND SGD DYNAMICS
The methods used here are related to the Stochastic Gradient Noise model (SGN) of SGD ((Zhou al.] (202T), (Battash & Lindenbaum, (023). (Nguyen et al. 2019), (Simsekli al. 2019). (Mignacco
& Urbani 2022) ) due to the relationship between the Fokker-Plance equation and the Langevin equation used in SGN. This framework has been used, for example, to model escape times from local minima (Xie et al. (202T)
Other works have studied the diffusive-like dynamics of SGD (Fjellström & Nyström 2022). and even modeled SGD an -hateinnUhleenbekk process to relate the dynamics of SGD back to the purely Bayesian case (Mandt et al.] 2016b). However, this framework requires that the minima the loss be quadratic, which means cannot accurately capture the behaviour of SGD in neural networks due to the degeneracy of local minima. Furthermore, other works have also found connections between SGD) and fractal geometry ((Camuto et al., 202T), (Simşekli al., (02T)) by the of iterated function systems and Feller processes. Although related to the results here, the formalisms used are significantly different and the exact relationship is not straightforward.
The most closely related to the work here is (Chen et al. 2021 who show that many networks seem super-diffusive near initialization and decay into sub-diffusion over time. They also give relationship to a type of fractal diffusion to explain this. However, they give no theoretical results, relying entirely on experimental results draw conclusions. Our work instead focuses on rigorous theoretical model that allows us to develop theory about the long runtime nature of SGD which explains the observations made previously, and we provide experimental results verify theoretical predictions.
3
FRACTIONAL DYNAMICS OF DEEP LEARNING
3.1 GRADIENT NOISE AND THE FOKKER-PLANCK EQUATION
Consider a neural network defined by some set of parameters w E W (where we assume W is compact throughout) and let X' be the of tuples (I,,, f(z;)) where f is the oracle that describes our decision problem. Denote the loss function by L X' x W + IR and set L|X, w] Ex [(x w)]. Letting X.. C X' be randomly sampled subset of possible inputs, the empirical loss on X., will then be denoted L. [Xm, w| = Ex.,. [Lr w)]. For the purposes of the theoretical analysis, we will assume that we are working the large batch size regime so that the estimation noise of the loss (and gradient) doesn't 't dominate the dynamics of the system.
3.1.1 GRADIENT NOISE AND
There is extensive literature which attempts to capture the dynamics of SGD by decomposing the weight updates (under some abuse of notation) into the form
dw dt
- C(( + Ew.-
where is is the population loss, t is the timestep, 7 is the learning rate, and Dw, is random vector (which we will assume in this work is an anisotropic Gaussian). This what is generally referred to as Langevin stochastic differential equation. Systems governed by such SDEs have a displacement R(t) x 12, meaning they diffuse like Brownian motion.
2

---

Published as a conference paper at ICLR 2026
However, most works which examine the weight dynamics don't agree with this model. It has been found that networks trained under SGD can behave super-diffusively early in training, becoming sub-diffusive as traiming continues (Chen al. 2021). Our experiments agree with this, finding that the displacement of neural network weights after long run times are described well by a power
law like R(t) & to for 2 (Bouchaud & Georges 1990) (an example of which can be seen in figure DD. Similarly, it has been observed that the weight movement of SGD (with momentum and weight decay) can have weight displacement that scales logarithmically like R(t) x Int (Hoffer al., (018). This behaviour cannot be captured by the traditional Langevin equation and requires the introduction of a subordination term. Such non- non-Brownian diffusion is collectively referred to as anomalous diffusion.
Average SGD displacement over normalized time
Average R(t)
Brownian reference (a 0.5)
10
R(t) sqrt(MSD(t))
8
4
2
Fitted displacement exponent a: 0.441
0.0
0.2
0.6 Normalized time (t)
0.4
0.8
1.0
Figure 1: Mean weight displacement of collection of fully connected neural networks trained using SGD on a randomly generated Moons dataset (Pedregosa et al.) (00T)) compared with expected displacement in the case of Brownian motion. It can be seen that this displays anomalous diffusion corresponding to early super-diffusion followed by late stage sub-diffusion.
To tackle this problem, we move into a formalism which is dual to the SDE picture, being the Fokker-Planck equation. Intuitively, the SDE picture describes the stochastic evolution of a single run while the Fokker-Plann equation is what describes the deterministic evolution of the probability distribution over parameter space over time determined by the SDE.
We now give the Fokker-Pe -lanck- equation (FPE) in weight space (that is, V V.)::
0p(w, 1) V SDD(((ww (D(w ()w,e, t) (p(,,
at
where is probability density function (density of states), D is the diffusion coefficient, is scalar (usually called friction), and a loss function which physical sense acts as a potential energy.
While the standard FP equation describes the behaviour of standard Brownian motion, one can hande the sub-diffusive case by introducing the (Caputo) fractional derivative operator(Diethelm) 2019) Do where 0 < is real number. Letting f be some arbitrary (differentiable) function of / the Caputo fractional derivative operator defined as
1 [([ [T]-a)
f'(t)
fPf(()
dr
(3)
aa) (t-7)a
3

---

Published as a conference paper at ICLR 2026
We now define the (time) fractional Fokker-Planck equation (FFPE) for SGD
as:
Dj p(w,() 1) = V (D(w, ()Vp(w, 1) - (pp(w, 1) V Lm (te))
Where the X,, is dropped from the loss expression for simplicity. We note here that this equation itself does not directly describe ultra-slow diffusion (that is, where the displacement R(t) x In t). However, ultra-slow diffusion appears "in the limit" power law sub-diffusion (Kochubei 2008).
discussion of this as well as the role of the fractional derivative are given in in the appendix.
One might now try to modify this to a "time-space fractional" Fokker-Planee equation to account for the potential super-diffusive behaviour early in training. However, we are interested in studying the steady states of the system, and under some very mild assumptions (namely that the probability distribution doesn't It lose mass over training) the steady state of the system does not depend on this early stage of training. and can be captured by solving the given time fractional FPE (Barkai 2001 , (Metzler et al. 1999 P. However, in our case, still run into difficulty since the diffusion coefficient is a location-dependent inhomogeneous diffusion tensor (that is, different dimensions have distinet diffusion coefficients) instead of a single scalar. Luckily, are able to approximate the diffusion tensor as single scalar function late in training.
will discuss in the next section,
3.2
FRACTAL DIMENSIONS AND SUBDIFFUSION
3.2.
SINGULAR LEARNING THEORY AND FRACTAL DIMENSIONS
In order to capture the local geometric structure that impacts the diffusive process we make use of singular learning theory (Watanabe) 2009) via the local learning coefficient (LLC) (Lau al.) 2024). We give a brief introduction to these ideas here, but a more substantial introduction is given in appendix A
Consider our loss function to be the Kullback-Leibiee divergencd Km [w]. Consider then the ball B,(w") ) of radius about some "true parameter" w" such that K[w"] 0. Letting be some arbitrarily small constant, and denote the set of parameters which have loss Km, [w] < within the ball (ww"") of radius r as (w"", c). Consider then the singular integral
v(e) = Sm..ur plw)
(w=
where p(w) is some arbitrary choice of prior distribution on the parameter space. Now letting 0 < a < 1 be some arbitrary constant, the local learning coefficient Lau et al. (2024) is defined as
log
V(cc) V(c)
X(u*) lim c=0
(6)
log(a)
We then have that asymptotically as e + 0 (under some mild assumptions):
V(e) A(w")
In the diffusion picture, the LLC behaves as a localized mass (Minkowskii Bonnlii--ouggand) fractal dimen- sion which determines the geometry of (potentially degenerate) near critical points. The nature of this relationship is discussed in greater depth in appendix B.I
3.3
LAWS OF FRACTAL DIFFUSION AND SGD
3.3.1 THE SPECTRAL DIMENSION
While the LLC captures the geometry of the loss, we all need to capture the dynamics of SGD on this geometry. To this end utilize second fractal dimension which describes the trajectory of particles
under a potential called the spectral dimension d, (Millln et al. 2021). (Bouchaud & Georges
'ince this is a continuous time formulation, it is more accurate to call it stochastic gradient flow, however the former is known to he reasonably well-approximated by the latter (Li ct al. 2019).
2The super-diffusive component matters for studying things like relaxation and crossover time. "e can just as well use the log loss (as it only differs by additive constant) but using the KI -ivveiveeee simplifies the analysis.
4

---

Published as a conference paper at ICLR 2026
1990)). We start with the definition in the "homogeneous" case (e.g when the fractal dimension of the medium is the same everywhere) and then adapt it to our multifractal case. If we consider the LLC as being the scaling exponent for the volume of "goo parameters" in particular area, the spectral dimension determines then the volume of states that the diffusive process over that area can actually reach over some period of time (in our case, the volume of states SGD can actually reach in that area). We define this dimension below.
Definition 3.1 (Spectral Dimension (Bouchaud & Georges I ) Let Vt(t) be the total volume of occupied states which result from running diffusive process porous media from some initial distribution. The spectral dimension dg is then defined as:
Va(t) ~
In non-homogeneous systems have that the spectral dimension changes different timescales (t,..., bm] where we have different scaling exponents d, (t(44) such that
(ttt) =t d,(4)
(9)
/. belongs to timescale 4,- If the variation between these different dimensions too large, the theoretical framing becomes more difficult. Luckily, we find that for SGD this spectral dimension is well-captured by single constant over training. Thus for vanilla SGD we can use what is known as the asymptotic spectral dimension defined as:
V.(1)
as 1 0
(10) More
and simply take d, = d. ((ben Avraham & Havlin 2000). Palladin & Vulpiani 1987)).
information and intuition about the spectral dimension provided in appendix B.2
3.3.2 WEIGHT DISPLACEMENT AND FRACTAL DIMENSIONS
We would now like to figure out the relationship between the local fractal dimension (the local learning coefficient) and the spectral dimension.
Definition 3.2 (Walk Dimension (Paladin & Vulpiani 1987), (Bouchaud Georges 1990 )) Let
R(t) be the displacement of a particle at time t. The walk dimension is defined by
R(t)=
Faok
(11)
with dwalk being the walk dimension with dwalk 2 for sub-diffusion.
We note here that these values are defined almost identically even when the process displays initially super-diffusive dynamics (details in appendix[B.3
It is known that in particular regimes the walk dimension takes on particular form. This is known as the Alexander- Alexander-Orbuch Orbuch (AO) relation and relates the walk dimension to the fractal dimension of the medium (the L.C. for us) and the spectral dimension. While originally stated in the context of homogeneous media this relation known to hold locally for porous media which are homogeneous on sufficiently small scale (Hambly et al. 2002). Restating these results in framing gives:
Theorem 3.1. The walk dimension at point on the loss surface can be given as
d (1) 2X(w))
(12)
d,
critical points.
The idea that neural networks trained by SGD are close to some critical point is direct result of the prevalence degenerate saddle points of the loss surface (Daauphin et al. 2014), (Advani et al.. 2020), (Fukumizu & Amari, 2000), (Choromanska et al. 2015)). In cases where there are nearby saddle points (which is more common in early training), this does relation does not need to hold as the diffusion is dominated by the gradient behaviour. This is the sense in which the relation is local. However, as noted, so long as this behaviour is largely isolated to early training it does not impact the theoretical results.
5

---

Published as a conference paper at ICLR 2026
3.3.3 DIFFUSION COEFFICIENTS AND LOCAL BEHAVIOUR
We would like to use these fractal dimensions define a diffusion coefficient. Importantly, we find that the diffusion coefficient is reasonably approximated by a scalar function. We also should expect that late in training, the localized dynamics nearby degenerate points should be directly proportional to the volume of low loss parameters. We state the results informally below, with the formal results and proofs in appendix D
First we state the following theorem:
Theorem 3.2 (Small Scale Dynamics are LLC Dependent), Let w E W be a point such that the Hessian of the loss H Hww") is positive semidefinite. Let W = Brr, w²) be small area about w². If the diffusion coefficient D along degenerate directions is isotropic then for fixed error tolerance & the first passage time T(E) through W is x DC
<dw")) where C is the "capacitance" of the escape set.
An important thing to note about the above theorem is that this is what would be considered a "pore scale" model of the diffusion, and is determined by the small scale dynamics and as such is less usefuul for experimentally capturing the behaviour. For experimental purposes we develop a more coarse- coarse-grained theory which relies on the following result:
Lemma 3.1 (Diffusion Coefficient Approximation (informal)). For reasonable choices of the learning rate in the large batch size regime, the diffusion tensor can be approximated by a scalar function for long runtimes.
Since the steady state is determined by the long-runtimg dynamics, we can study the FFPE with scalar diffusion coefficient.
To study the coarse- grrrsedg diffusive behaviour physics-inspired scalar diffusion coefficient for porous media that captures the essential behaviors of the diffusion at some choice of measurement scale called the characteristic length scale &. Under some assumptions we have the following:
Lemma 3.2. Letting& & be some characteristic length scale, the diffusion coefficient can be approxi- mated as Dg = 2d-uskk dank
A thing note is that the choice of & is effectively how far are zooming out and averaging over the local dynamics, which gives a scaling law, not an exact relation. A general practice is to pick value of & which is large enough average out the fluctuations in an area but not so large that it starts to ignore large scale changes in structure. The effect of choice of & is shown in section 4 One may also notice that this implies that the diffusion coefficient is higher for a small LLC which seemingly contradicts the pore pore-scale diffusion derived earlier where low LLC was slower. However, this is explained through the spectral dimension (as we shall see) as is bounded above by the LLC, meaning that small LLC is only faster if the dynamics allow very free movement inside of the domain.
Combining lemma 3.2 with the definition of the walk dimension given earlier get: Corollary 3.1 (Fractal Effective Diffusion Coefficient). Let E be a choice of the characteristic length scale. One can then define the effective (local) diffusion coefficient for length scale as
2A(wa) d,
De(w)
(13)
3.4 STATIONARY STATES THE SGD FORKER- EQUATION
We now present thcoretical results about the diffusive process with proofs in appendix D In appendix Dwe provide a brief discussion of impacts when particular assumptions about the system are not met and how the results here can be extended.
Assuming some fixed scale ç, using the effective diffusion coefficient, we can actually find the local steady-state solutions for the SGD Fractional Bokker-Planche equation (if it exists):
Lemma 3.3. Consider a subset W C W such that the effective diffusion coefficient De is (approx- imately) constant on W. Suppose then that there exists steady-state solutions of the SGD-FFPE
on this subset with true parameter(s) w" so Di (w" 1) = O. The steady-state pa(w] Xm))
) = (w(
-mll]
distribution is then given by Ps (1(

---

Published as a conference paper at ICLR 2026
Note that the above holds even if Dr does not have the form given in definition 3.1 long it is simply a scalar. However, if does have the form, due to the definition of Dç. the above condition that it be constant is actually simply saying that X(w) be locally constant in W which tends to be the case away from phase transitions (Wang et al. 2025). We can also get from this a relationship with the Bayesian posterior perspective of singular learning theory.
Corollary 3.2. Letting y 1 for simplicity, if C. is the log-loss and w E W then
Ps ps)))DDs x p(Xm/w)
(14)
so
p(w) ((w)po(w)mm
(15)
ZmDe
where ZmD; is the partition function and is an arbitrary choice of prior.
This explains the observed relationships between Bayesian sampling and SGD seen in (Mingard et al.) (020). We can see that SGD effectively scales the likelihood of certain states of the underlying purely Bayesian distribution at the measurement scale & based on how accessible they are to the model under the optimization process. That is, the distribution of solutions found by SGD from some initial distribution concentrate more heavily in particular areas than the Bayesian posterior since SGD simply cannot reasonably reach those areas.
Another important aspect of the local learning coefficient that it can be considered the quantity that "bounds" the movement of network weights. For notational simplicity let w(t) be the parameters of the system at time f, so we can formally state the above as:
Lemma 3.4. Suppose the loss function is non-convex and non-constant on W. Then with spectral dimension dg as t - with fractal dimension X(w(t)) on W W, the inequality dg S X(w(t)) holds (in the small learning rate regime).
In the above lemma the timescale condition used to account for the fact that at early times such sub-diffusive processes can appear nearly linear. Given the above, we get the following corollary:
Corollary 3.3. For time as t + oc, we have d, S X(w(t)) where
X(w(t)) = lim 1 //" A(w(t))dt 7-000 7
(16)
Notice that since small A(w) implies greater local volume, but larger d, implies that the volume spreads faster over time, large local volumes trap the spread of SGD, slowing it down. This aligns with previous research examining the eigenvalues of the Hessian of the loss (Sagun et al. 2016). In the next section we will show that the above result holds experimentally as well examine other properties of our fractal diffusion theory of SGD.
4
EXPERIMENTAL RESULTS
4.1 DIFFUSIVE BEHAVIOUR
Here we present experimental results to validate the diffusive theory across multiple model architec- tures and tasks. Namely we look small language models trained on the TinyStories dataset(Eldan Li 2023), vision models trained on Tiny Imagenet (Le & Yang 2013), as well as extensive ablations on MNIST dataset (Deng) 2012) with fully connected architectures with ReLU activations and batch normalization. More extensive experimental details can be found in appendix G
To compute the LLC we utilize the estimator provided by (van Wingerden et al., 2024). To compute the spectral dimension d, we first compute the value log(R(t) ((()) where R((t) is the total weight displacement at time f. We then find d, by solving the linear regression problem
d, log( 2X(w) log(t) +
(17)
where e is simply offset term.

---

Published as a conference paper at ICLR 2026
Spectral Dimension vs. ULC
LLC Spectral DKn
Pearson corril, Jinal, lambde, barj optimizergge
0.5
0.6
(Estenated
62
tutchues
40
20
0,80
Learning nabe
Spetra Dimonsion
(b) Correlation between learning coefficient (average) and total weight displacement (MNIST).
(a) Visualization of lemma
3.4 (MNIST)
Figure 2: In a) we check that the result of lemma 3.4 holds. In b) we check that independent of our choice of diffusion model, the total displacement and average learning rate strongly correlated in the large batch, low learning rate regime.
Model name
X
dg
y2
TinyStories- IM
32
21,422
0.33
0.98
TinyLlama-155
76.1
48.3
0.32
0.98
TinyStories-33M
39.3
38.7
0,49
0.98
ResNet18
72.05
0.57
0.004
ResNet34
73.5
0.62
0.004
2
VGG16
159.7
0.14
0.001
Table Results for different models.
Using this setup, we are able to experimentally test the result of lemma 3.4 and corollary 3.3 which can be seen in figure for an extensive collection of various models over MNIST, as well as various vision and language models in [ We also check the accuracy of the sub-diffusion model.
We find that in general, the sub-diffusive prediction is very accurate for are trained to convergence. In particular we note that despite our theory not explicitly accounting for adaptive optimizers and learning rate schedulers, the dynamics vision models fine-tuned using an initial adaptive optimizer, followed by low learning rate SGD are well-predicted by the theory. Furthermore, by taking pretrained language models which have already been trained convergence in the weights and then continuing training on their initial pretraining dataset agrees with the predictions of the theory. More results are available in appendices and
models tested which
4.2 POSTERIOR CONCENTRATION
In order check the results of lemma 3.3 and corollary 3.2 we train large number of identical fully connected networks on generated moons dataset (Pedregosa et al. (01T]) using SGD. To compare the distribution of solutions found via SGD vs. the (local) Bayesian posterior, we use SGLD (Welling & Teh 2011) to approximate the Bayesian posterior. We then identify clusters of solutions, and identify the concentrations SGD and Bayesian solutions within each cluster. To select the scale check how the choice of & impacts the KL-divergence between the empirical SGD distribution and the theoretical SGD distribution (figure 3). We can see in figure 4 that the solutions found by SGD do tend to concentrate around lower LLC areas. Figure 4b and table 2 shows how the tempering of the distribution SGD solutions effectively agrees with approximate Bayesian posterior of SGL.D.
for tempering,
8

---

Published as a conference paper at ICLR 2026
between empirical theeretical densities
Kijampp theory)
KLithcory| || cmp)
15-
XL fLdivereec
10
-2.00
-1.75
-1.50
-1.25
-1.00
-0.75
-0.50
-0.25
0.00
log10/st)
Figure 3: Kl.-divergences between the empirical vs. theoretical distribution for different choices of g-
40
2. 13
(a) LLC Histogram
(b) The cluster concentrations
Figure 4: a) Shows the histogram of local learning coefficients of solutions found by SGD. Notice that predicted by the theoretical results, they tend to concentrate near lower LLC values (better generalizing solutions). b) The probability concentrations of solutions found by SGD (blue), the approximate Bayesian posterior (orange), and the tempered SGD distribution (green) for each cluster. Notice that despite SGD itself preferring the cluster Cl, after tempering (f = 0.5), the tempered SGD steady state distribution almost entirely agrees with the Bayesian posterior. Statistical measures can be seen in table 2
DISCUSSION
5.1 LIMITATIONS
While we believe our theory is useful and tends to capture dynamics of most optimizers empirically. it does not explicitly take into account complex dynamics of adaptive optimizers like Adam (Kingma Ba 2017) as adaptive optimizers can exhibit multiple spectral dimensions over the course of training meaning the theory here is incomplete and should occur as a "special case" of a more general theory.
Some experimental results and discussions around this be seen in appendices and H
Another limitation to consider that we assume the existence an approximate steady state. While this is common practice in the study of SGD ((Pesme et al.) 2020). (Mandt al.) (2016a). (Mandt et al. 2018). In general, SGD iterates do not converge to exact equilibria, but under standard assumptions and suitable learning-rate schedules (or sufficiently small learning rate) they approach

---

divergence for the Published as a conference paper at ICLR 2026
Metric
Value
K. ([Bayys] Tempered SGD)
0.009
Wass (Bayes, Tempered SGD)
0.002
JS (Bayes, Tempered SGD)
0.003
Table 2: The KL divergence, the Wasserstein distance, and the approximated Bayesian posterior and the tempered SGD distribution.
the set stationary points and attain iterates with small gradient norm. One can theory have instances where no approximate steady state exists since label noise can in theory produce equilibrium flow through states, so SGD might have a non-equilibrim steady state with probability flow driven by said noise. While examining such situations is outside the scope of this work, it is an important avenue of future work to examine a) the time to equilibrate of SGD and b) if it does not equilibrate, can we quantify its non-equilibrium steady state?
5.2 CONCLUSION AND AVENUES FOR FUTURE WORK
Here we have argued that the long runtime dynamics of SGD are captured by taking the corresponding Pokker-Plance equation to describe diffusion on porous geometry. This porous geometry corre- sponds is described by the learning coefficient, drawing a direct relationship between the dynamics of SGD to Bayesian statistics via singular learning theory. Our experimental results validate this claim.
We believe our theory helps provide insight into the learning process and adds the groundwork needed to build foundational theory of learning dynamics. Our theory says that the learning process is governed partially by the model's behavioral phases as described by the learning coefficient. This opens up framework for studying emergence and phase transitions during training by considering properties the dynamical system. Adapting this framework explicitly to adaptive optimizers and checking how this impacts the diffusive structure is important avenue for future work.
5.2. REPRODUCIBILITY STATEMENT
To encourage reproducibility we provide source code for the experiments included along with extensive documentation in appendices
REFERENCES
nickyprottinyllama-155 Hugging Face huggingface.cv https://huggingface. co/
n nickypro/tinyleema 5M a. [Accessed 16-11-2025)
voneneddan/ingySooris--MM Hugging Face huggingface.co.
http:s////huggggggaccccccc Iorsnee_dan/TTiyyytoor.. 1M, b. [Accessed 16-11-2025].
vonceelan/TingStorc-3333-- Hugging Face - http://www....ppee///gggngnn...e rroreneldan//iinyttorress
[Accessed 16-11-2025).
Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020. ISSN 0893-6080.
doi: Cotm://cceenooe/reree//////66631 com/
URL https://www sci www.eciiiiccccccccc..ccc.oo
E. Barkai. Fractional fokker-planck equation, solution, and application. Phys. Rev. E, 63:046118 Mar 2001. doi: 10. 101/10/yy.eeeE..3..0000
URL (tttst::: //ts::///nnkk..... aps owwps//rr////00000
1103/PhysRevE, 63. 046118
Barak Battash and Ofir Lindenbaum. Revisiting the noise model of stochastic gradient descent, 2023.
/ wttp:/www..rraas//33...o. 92749
URL httph: http::///aaavvv.eea
John Baxter. Classical potential theory and its probabilistic counterpart (j. doob). SIAM Review, 27
(3):460-46) 1985. doi: 10.1137/102712.
10

---

Published as a conference paper at ICLR 2026
Daniel ben Avraham and Shlomo Havlin. Diffusion and Reactions in Fractals and Disordered Systems. Cambridge University Press, 2000.
Jean-Philippe Bouchaud and Antoine Georges. Anomalous diffusion in disordered media: Sta- tistical mechanisms, models and physical applications. Physics Reports, 195(4):227-293, 1990. ISSN 0370-1573. doi: otg.a//c.......ooo.o....oooooooa
wttp:////0....000000000007000000 URL https /ww..sciencedirer. coteeettccocccaaeeeeeeeeeeee
Alexander Camuto, George Deligiannidis, Murat Erdogdu, Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. Fractal structure and generalization properties of stochastic optimization algorithms. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18774-18788. Cur- ran Associates, Inc., 2021. URL /tt.a//tssr/ppppcedddannsaaaoppp
Liam Carrll.. Distilling singular learning theory, 2023. URL httpp::/wwww.eeeteeeepp
com/
3/cz7XjvCIssCeepdiit Accessed: 2025-01-20.
Guozhang Chen, Cheng Kevin Qu, and Pulin Gong, Anomalous difflusion dynamics of learning in deep neural networks, 2021. htt.ltttsss/axx..ooooo/o/////
10588
Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian phase transitions in toy model of superposition, 2023. URL https::/arxiv..ot
abs/ Abss23100 06301
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks, 2015. httpt//ps///.....oorgg//////////
0233
Doina Cioranescu and Patrizia Donato. An Introduction Homogemization. Oxford University Press, 11 1999. ISBN 9780198565543. doi: 10. 10111999100007777709555555544330
URL
attps::/doi..
0001
Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, 2014. httt/t//t//:/txiitt org/ abs/ 11....106..... 2572
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. JEEE Signal Processing Magazine, 29(6):141-144 2012. doi: 10.1 10.10///P..212.22211122
Kai Diethelm. General theory of caputo-type fractional differential equations, 2019.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
07759
Carmina Fjellström and Kaj Nyström. Deep learning, stochastic gradient descent and diffusion maps, 2022. URL hhtth:::: http//axvvvoo UR//ttpss//rrrrrrrrr////////0000
K. Fukumizu and S. Amari. Local minima and plateaus in hierarchical structures multilayer perceptrons. Newral networks the official journal of the International Neural Network Society, 13 3:317-27, 2000. doi: 10.1016/S0893. (00000883006000000000005555
B. M. Hambly, Jun Kigami, and Takashi Kumagai. Multifractal formalisms for the local spectral and walk dimensions. Mathematical Proceedings of the Cambridge Philosophical Society, 132(3): 555-571, 2002. doi: 10.1017/503350044100 10100006111111881
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
03385
Flad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks, 2018. URL /tL.:ttta//aaxx..orgo///////
1705. 08741
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https: //aaxiv//xvvv www.worggaaee wttp/b////11..../ 6980

---

Published as a conference paper at ICLR 2026
W. Kinsner. A unified approach to fractal dimensions. In Fourth IEEE Conference on Cognitive Informatics, 2005. (ICCI 2005)., pp. 58-72, 2005. doi: U0.1009COGIIFF200...555 1532616.
Anatoly N. Kochubei. Distributed order calculus and equations of ultraslow diffusion. Journal of
Mathematical Analysis and Applications, 340(1):252-2811 April 2008. ISSN 0022-247X. doi:
10. 10.1016/j, 1...1//6..jaaaa... jma.:ijaaaaaa..... ww...000007000000 08.02.. URL http://dx doi /ttp://////////......../////....
1016 jmaa 2007.000000
024
Colin H. LaMont and Paul A. Wiggins. Correspondence between thermodynamics and inference. Physical Review E, 99(5), May 2019, ISSN 2470-0053. doi: 10. 1..110/phyarevee99.0552111 URL
/xx doi http::/........oro......111/1p//
99. 99.021440
Edmund Lau, Zach Furman, George Wang. Daniel Murfet, and Susan Wei. The local learning
coefficient: A singularity-aware complexity measure, 2024. URL vttps://arxiv.cr
abs/ 12108
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge, 2015. URL https://api semanticschol arr,
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40); 1-47, 2019. URL WtRttbttp://// jm wttp://ppa./////////....
html
Yong Shun Liang and Wei Yi Su. geometric based connection between fractional calculus and fractal functions. Acta Mathematica Sinica, English Series, 4002):537-567, Feb 2024. ISSN 1439-7617. doi: 10.10077s101144-233-000 1663-33 URL https:://oi..rrrtt s 61C114-023-1663-
Stephan Mandt, Matthew Hoffman, and David Blei. variational analysis of stochastic gradient algorithms. In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings The 33rd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 354 363, New York, New York, USA, 20-22 Jun 2016a. PMLR. URL https //proceedings. mlr. m.l.... press / v48 11388/mandt.11 html
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. A variational analysis of stochastic gradient algorithms, 2016b. URL httL:tttt//ssetettt org/ abs wwtg:///////....o./....oo
Stephan Mandt, Matthew Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference, 2018. htt.::tttt:/aaaaa
org/ orgg/bb///////..///////// 0b44 04289
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47), November 2022. ISSN 1091-6490. doi: 10. 00.1073/pmas.22266551133 URL http::///aa doi www.dd.........oo. /tt0g///.111/////.. 2206625119
Ralf Metzler, Eli Barkai, and Joseph Klafter. Anomalous diffusion and relaxation close thermal equilibrium: A fractional fokker-planck equation approach. Phys. Rev. Leur 000:3533-3561 May 1999. doi: 10. 1011000ysyeeee.ee.... 82.35635.. URL /tttps:///iikk aps wawpp.orr...oo wptpppppg//////...1111.
PhyskevLett 82. 3363.
Francesca Mignacco and Pierfrancesco Urbani. The effective noise of stochastic gradient de- scent. Journal of Statistical Mechanics: Theory and Experiment, 2022(8)083405, August 2022. ISSN 1742-5468. doi:
174225468//c8444
Ana P. Millán, Giacomo Gori, Federico Battiston, Tilman Enss, and Nicolò Defenu. Complex networks with tuneable spectral dimension as a universality playground. Physical Review Research, 3(2), April 2021. ISSN 2643-1564. doi: 10. 103
URI htt.
7/dx. doi. org/ 10. 10..111///yyyyeeeeeeeer....c
023015
Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, and Ard A. Louis. Is sgd a bayesian sampler? 15191
well, almost, 2020. /
12

---

Published as a conference paper at ICLR 2026
Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, and Gael Richard. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise, 2019. URL https
//axxiv.orgl abs wttpg//aaa//...o/a//ee/ae /1066 09069
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse. Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. http:/22222iii-ottttee-lllle and and-induction heweriioonnhadddddded...mmm html.
http::/transformer-crrcci
Giovanni Paladin and Angelo Vulpiani. Anomalous scaling laws in multifractal objects. Physics Reports, 156(4) 1987. ISSN 0370-1573. doi: https///ooi.or///
1016/0370- 15073770-11573077711110-44 URL https: www.wsccence.... rect. com/ coom/sciencee
axticle/ss1/037711777711111
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-lear: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Scott Pesme, Aymeric Dieuleveut, and Nicolas Flammarion. On convergence--dignoottic based step sizes for stochastic gradient descent, 2020.
2000//00abbb000000053533 00534
Sidney Redner. A Guide to First-Passage Processes. Cambridge University Press, 2001.
Levent Sagun, L.. Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv: Learning. 2016.
Gideon Schwarz. Estimating the Dimension of a Model. Annals of Statistics, 6(2):461-464, July 1978.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Umut Simsekli, Levent Sagun, and Mert Gurbuzzalaban. A tail-index analysis of stochastic gradient noise in deep neural networks, 2019.
06053
Samuel L. Smith and Quoe V. Le. A bayesian wwwpspeeeiiveeoongeeeniiiiiiiiie descent, 2018. URL httpt:t/////o http:///aaaaaa..oor//////111.0.0
and stochastic gradient
Zhou Songping. On the fractional calculus functions aype of weierstrass function. Chinese Annals of Merteeeatticssseriiss A,
12341 9022
FRANK B. TATOM. The relationship between fractional calculus and fractals. Fractals, 03(01): 217-229, 1995. doi: 10.1 142/80218348090000175 URL /
50218348X95000175
S W Tyler and W Wheatcraft. Fractal processes in soil water retention. Water Resources Research; (United States), 26:5, 05 1990. ISSN ISSN WRERA. doi: 10.1129/WR02200050011017 URL
https: hetps://wwee www. ost gov wtto///bll///5440000000
Stan van Wingerden, Jesse Hoogland, George Wang, and William Zhou. Devinterp. 2024.
https
George Wang, Matthew FarrugiaRRoberts, Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewise development of transformers. In High- dimensional Learming Dynamics 2024: The Emergence of Structure and Reasoning. 2024a. URL htttps://pperre/e
net htt.s//orummiiio 20ab
George Wang, Jesse Hoogland, Stan Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient, 2024b. URL
029884
13

---

Published as a conference paper at ICLR 2026
George Wang, Jesse Hoogland, Stan Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient. In The Thirteenth International Conference Learning Representations, 2025. URL https::/openreviee/
d=SUc UOWnnd
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. 2009.
URL https
//api. str.an/seeattttssseeeeeee.. www.aaaat.cccccc...ooooooo.... wttpa//wa.a....ooooooooooooaaaa.
Sumio Watanabe. A widely applicable bayesian information criterion, 2012. URL https://
arxiv. abs 1208.6338 6338
Sumio Watanabe. Recent advances in algehraic geometry and bayesian statistics, 2022. URL
Sumio Watanabe. Review and prospect of algebraic research in equivalent framework between statistical mechanies and machine learning theory, 2024. URL https:://////// httttp:://axxv.org/// /bss
2406. 10234]
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems, ((112)):0443-10888 December 2023. ISSN 2162-2388 doi: 10.1 1009//m/s..222..3111111
URL
http://dx.com doi. wwt. 1109//NN1111 2022. 31 3167409.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML' 1. pp. 68 1-88, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306
Hermann Weyl. On the volume of tubes. American Journal of Mathematics, 61(2):661-472 1939. ISSN 00029327, 10806377. URL http::/ jstor .org wttgsttalllooooooooo.oo........o httg:/staaleee233333333 2233155
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima, 2021. URL https: pictps://arii:: //arxiv
brg/abs/2002. 03495
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Hoi, and Weinan E. Towards theoretically understanding why sgd generalizes better than adam in deep learning. 2021. URL https:
7/arxiv org. wttp:/rgaabb///oooaaa 05622000.00627 05627
Umut Şimşekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization neural networks* Journal of Statistical Mechanics: Theory and Experiment, 2021 December 2021. ISSN 5468. doi: acc3a
A SINGULAR LEARNING THEORY BASICS
Here we give informal introduction to singular learning theory. For more in-depth but still accessible introduction, we recommend the Distilling Singular Learning Theory series of blog posts (Carroll 2023), along with the the seminal work by Watanabe [Watanabe] 2009). For us, it's mostly important to understand the problems that singular learning theory solves. To do this, we first must consider classical idea in machine learning, the Bayesian Information Criterion. The BIC used to determine which model from a set of different models is likely to generalize the best. Let aw be model with free parameters w the collection of models, trained over m datapoints and denote minimum loss achievable by au as Ln (ag) The BIC says that we should select the model from our collection of models which minimizes the following:
BIC nLn (a.) d
log
(18)
This more-or-less says that should choose the simplest model that fits data.
The caveat about the BIC however makes the assumption that the models we care about are "regular statistical models". There are two key things that required for statistical model to be
14

---

Published as a conference paper at ICLR 2026
regular. First, the model must be identifiable, which effectively means that any set of parameters for a are unique in that if aw, (I) Ows (z) then wi = W22 Second, the Fisher Information matrix near the true parameters ahe must be positive definite. This condition is casiest to understand if assume the loss is the KL-ddivereence (or log loss), as it corresponds to saying that the Hessian of the loss H (T,,, (a.)) is non-degenerate, having only non-zero eigenvalues.
This fact is key for how one derives the BIC. While the formal derivation of the BIC is straightforward, it is time consuming and there's a much simpler way to intuitively see why it matters. First, the non-degeneracy of H (L,()))) means the geometry of the loss surface is a paraboloid about a We want to then measure how many configurations of a have loss less than c, so we want measure the volume of a paraboloid of height in our parameter space. The nice thing about a parabaloid is that volume is half of that of the cylinder that encloses This can be computed straightforwardly from the d-dimensional volume of tubes formula Weyl 1939): c
Va(2c)
(19)
v det H (Lm (o%))
Here Va is the volume of the d-sphere. This formula is effectively where the d comes from in the BIC. Now, one might notice that if we are considering potentially degenerate local minima, this formula cannot be applied since the degeneracy of the Hessian means the determinant is O. In this case, the BIC not well defined either. Models with degenerate minima are called singular models. In some sense, most model classes are singular. Neural networks for instance are highly singular and generally admit many equivalent parametrizations for computing the same function. Singular learning theory attempts to handle this problem by finding method for computing the volume of degenerate local minima.
This is done by considering the singular integral Watanabe 2009)
V(c)= J J(
p(w)dw
(20)
where p(w) a prior distribution over the parameter space so p(w) behaves as a measure, and is the population loss. Unlike the quadratic minima case. there is no straightforward volume formula one can use here. However, as is shown in (Watanabe| (209)), as e - 0 asymptotically this integral is:
V(e) = c (- logc)""-1 + (-e> e)g-))))
(21)
where À is learning coefficient and m the mulriplicity. One can define this integral at level sets which are non-true parameters by shifting the value in the integrand like {w W|0 L(w) << e).
In short, one can show that the volume about a local minima scales with the height according to the learning coefficient À so the volume of the degenerate minima scales & e as c - WWtanabe 2009). This can be used to derive the Widely Applicable Bayesian Information Criterion (Watanabe 2012 which given by:
WBIC = nI.n + X logg
(22)
The natural interpretation of À is the "effecttve dimension" of a model. We note here well that it is relatively common to treat the multiplicity as taking the value m = 1 to simplify working with singular models as the relative contributions for most applications are negligible in their effects.
A.1 THE LOCAL LEARNING COEFFICIENT
In the above, we discussed the global learning coefficient of (Watanabe 2009). A local version of this was defined in (Lau et al., 2024). This given straightforwardly by simply restricting from the singular integral over the whole set of e-true parameters to some local neighborhood of some parameter of interest w One normally considers ball of some radius r about said parameter B, B,(") ) and then defines the local singular integral as
(WEBB.(w)) - du(w)
Vi- (c)
(23)
15

---

Published as a conference paper at ICLR 2026
where du((w is the standard Lebesgue measure. The local learning coefficient and local multiplicity are of the same form as the global case:
Vi (c) A(w (-w") (-log m)
(24)
When looking at the local learning coefficient, one must make a choice of scale r. However, we note that increasing cannot increase the value of X(w"). This comes from fundamental property of the learning coefficient which says that the learning coefficient in an area is the smallest of all possible local learning coefficients
O. The reasoning for this is non-trivial, and the interested reader is referred to [Watanahe] 2009) for details.
We now briefly explain how the local learning coefficient is computed in practice. Suppose that define some distribution over (r(w"). The local learning coefficient estimator of (Lau et al. 2024) is given
\(w*)
logn (w.) - (ww)
n
(25)
This accords with our intuition that if w" is simple/flat then perturbing the value of w² should not change the loss. For in-depth experimental results for the accuracy of this estimator we refer readers
(Lau et al. 2024) and (Wang et al. 2024b).
A.1.1 DEPENDENCE OF THE LLC ON r
While this question is addressed more formally in appendix B of (Lau et al. 2024), we address this point here informally. Under the standard assumptions of Singular Learning Theory (Watanabe) 2009), the local learning coefficient some fixed "pointwise" learning coefficient. This is because as r becomes small, the behaviour of the B, (w") ) is dominated by the leading order of the singular expansion about w². Shrinking r cannot change the monomial exponents obtained after resolution of singularities; those exponents are geometrie invariants of the germ of L at the point.
a ball of radius around true parameter w' converges to
loss [(w) for some
B FOKKER- EQUATIONS AND FRACTAL DIMENSIONS
B. THE MASS DIMENSION
In section 3.2.1 we claim that the local learning coefficient is effectively a mass dimension. To see this, let's start by considering large collection of monniineraactnng) particles diffusing through an arbitrary fractal media. An important thing to note here is that diffusion on fractal media is actually a special case of the more general "diffusion on porous media" where the volume of the pores scales like fractal dimension. Keeping notational consistency, we are interested in the valid states in some ball B ((w"). ). To measure this, we need something called the "characteristic linear dimension" which we can scale asymptotically. In porous media, this is something like the "pore diameter" (since particles can occupy any point pore). For consistency again, we denote this value as 6.
The mass dimension is then the fractal dimension that determines the relative volume of the pores to the total volume as we restrict the diameter of the pores by taking c < One way to see what this is doing is to consider the mass dimension of an empty sphere (that is, the whole thing is pore and nothing is there to impede a particle). As we take to we end up with every possible point being a pore, so the relative volume is 1.
So we can imagine that for some ball centered about a reference point B(w*). ), and we care about the volume of states a particle could exist in within this ball, usually denoted as A (e). The relative volume is given identically to the learning coefficient case like
(
(26)
M(B(w*)
We get the fractal dimension which determines the relative volume (w"" as:
M(c) x (y("))
(27)
as é + 0 asymptotically. One can see that this coincides with the definition of the local learning coefficient( 2005), (Bouchaud & Georges 1990)). From a fractal geometric viewpoint, the
16

---

Published as a conference paper at ICLR 2026
normal mass dimension is computed as a "Minkowski sausage" which can be thought of similar how the volume of a "sausage casing" wrapping a pore scales as you decrease the radius. The learning coefficient is similar, except it uses different "gauge function" since instead like basins, so we look at how the volume of water in the basin changes as we decrease the height of the water. This fractal dimension has been used to model the diffusion of water through ground soil of different types (Tyler& & Wheatcraft] [1900.. They find diffusion is much slower and has larger fractal dimension through clay-like media where there are very few channels to move through, where more sand-like media has faster diffusion and a lower fractal dimension. This is conceptually the same as the results given here.
pores are not tubes, but are
B.2 THE SPECTRAL DIMENSION
The spectral dimension is easiest to understand if you think "how fast does diffusion manage to explore new places?" rather than "what is the geometric dimension of the space?". In the main text we introduce the spectral dimension the scaling exponent of the "volume of visited/occupied states". That is, (t) simply denotes the volume of states which have been visited by the process at time t. The spectral dimension is then:
(1()) ~ t
(28)
Intuitively d, says how fast diffusion fills out the medium you are diffusing on, and that sense is the "dimension that diffusion sees".
In this work, the medium is (a region of) parameter space, and the diffusive process the long- cong-run stochastic motion of SGD (modeled via the Fokker-Planck equation). The spectral dimension then measures how quickly SGD can spread over the set of weight configurations that are dynamically accessible from some initial condition. To see this bit more formally we will discuss the spectral dimension of normal Brownian motion.
B.2.1 SPECTRAL VS. GEOMETRIC DIMENSION
On a flat surface like Rd, random walk driven by Brownian motion has displacement R(t) x t so the region which gets explored after time t can be seen straightforwardly to have volume which simply scales with the displacement, since the diffusion is uniform in d dimensions. That we have:
(a(t) x R(t)
(29)
so
Vi(() x 1
(30)
One can see though that by introducing obstructions into this free space changes the diffusion rate. Moving through medium with many obstructions changes the geometry that diffusion experiences, which can be very different from the naive ambient dimension. Narrow channels, dead ends, bottlenecks, and local degeneracies all slow down or redirect the random walk. The process might live in very sigh-dimensional ambient space, but only much smaller effective set of directions is actually accessible on the timescales we care about. In such cases one typically has that d, A d. In our setting, the ambient dimension is the number of parameters, while the local "mass dimension" of low-loss regions is given by the local learning coefficient X(w). Consider this in the same context the Brownian motion example. Since the learning coefficient captures the volume of low loss states, a diffusive process on those states can only ever access at most that many states, bounding the spectral dimension. So the spectral dimension literally captures the number of states SGD can actually visit over some time frame.
B.2.2 WHY SGD HAS A SPECTRAL. DIMENSION
For simplicity we are going to imagine a very localized picture of SGD here. Suppose we initialize in some area W which has learning coefficient À which approximates the volume of low loss points in W. The spectral dimension d, of this area tells us how efficiently the SGD-induced diffusion spreads into that volume over time. The walk dimension d ties the two together and governs the displacement scaling.
From the perspective of diffusion theory, d, is therefore the right quantity to describe the effective dimensionality of SGD dynamics.. It is "spectral" because, in principle, it could be read off from
17

---

Published as a conference paper at ICLR 2026
the spectrum of the Fokker-Plande operator governing SGD; in practice, estimate it through the observed power-law scaling of displacement, which is equivalent information in the regime we study.
B.3 IMPACT OF EARLY SUPER-DIFFUSIIN DYNAMICS ON DIFFUSION EXPONENTS
Given the displacement equation
R(I) (S)
(31)
we can give rather straightforward way to incorporate the super-diflusive component. If given trajectory goes from super-diffusive to sub-diffusive, there exists a crossover time te where the dynamics change. Given this, let r(t) be the super-diffusive component such that for te we have r(t) = r(tc). Letting I, (t) = 1 for fort 2 te we can then write the displacement as
R(t) ~ I, (f) - "ma + r(t)
(32)
similar trick works for the volume. This means that behaviour without directly impacting the exponents if one accounts for the crossover time. In general, we find experimentally in many cases that the initial super-diffusive regime is short enough that the dynamics are still well approximated by the entirely sub-diffusive equations.
account for the early super-diffusive
C TOWARDS PRACTICAL APPLICATIONS
While the results here largely theoretical, we believe they provide important avenues and insights for practical applications. We discuss some of these below.
C.1 TRANSFER LEARNING AND ROBUSTNESS
C.1.1 PARAMETER CHOICE FOR TRANSFER LEARNING
task. The In transfer learning. you start from pretrained minimum and fine-tune with SGD value at initialization tells you how wide that basin is. If one maintains record the weight displacement from pretraining, one can estimate how the effect the new data distribution has on d, during the initial steps. These tell you how "wide" and "connected" the accessible region is, which can inform how aggressively dimension is low, but the loss is high, you are likely stuck in a wide flat basin, so one might increase the learning rate decrease the batch size.
tune the learning rate and batch size. For example the spectral
C.1.2 ROBUST MODEL. SELECTION
Our theory indicates that model parameters with low but a high relative spectral dimension d, represent models which had more movement within the same large basin. Selecting for such models might result in more robust generalizing models as the minima they exist in are "flat"
C.2 LEARNING RATE SCHEDULERS AND OPTIMIZERS
C.2.1 DESIGNING LEARNING RATE SCHEDULERS
Warmup and decay can be viewed as shaping d, over time. This suggests the potential for structural schedule design: e.g. maintain higher d, early (more exploration), then lower d, later (stronger localization).
C.2.2 EVALUATING OPTIMIZERS
Another application is evaluating optimizers for particular structural properties. That is, look at how the spectral dimension or the learning coefficient change over time and compare these with SGD to better understand how the optimizer impacts generalization behaviour.
18

---

Published as a conference paper at ICLR 2026
C.3 APPROXIMATE BAYESIAN INFERENCE
One can potentially apply the theory here to calibrating uncertainty in SGD. In practice, "Bayesian" approximations often assume Langevin dynamics with quadratic minima. Our theory gives way to correct for degeneracy and accessibility so that posterior variances and predictive intervals reflect the actual dynamics of SGD, not an idcalized model.
D PROOFS
D.1 LLC DETERMINES PORE SCALE DYNAMICS NEAR DEGENERATE POINTS
The proof of this relies on results from the theory of first passage times, homogenization, and porous diffusion, namely the estimation of mean first passage times of porous media. However, the components needed are straightforward to state. First we need the following definition (Baxte 1985):
Definition D.1. Let c Rd. The Newtonian capacity Cap(K) defined as
inf{ /
Vu dx u E C u 2 on K}
(33)
Intuitively: if K "big" or "accessible," you can spread charge thinly and keep fields weak which makes them low energy, large capacity; if it's tiny shielded, fields must be intense - high energy. small capacity. Next we give a known result of first passage times in porous media(Redner) (001).
Theorem D.1. Let Q Id be a porous domain and let dXx. be brownian process on St and let r be the set of all absorbing walls in Q. If we consider then I' to be the subset of absorbing walls with area less than ò then the mean first passage time of Brownian particle through Qt is (asymptotically) proportional to
12
T
(34)
(T))
as 5 >> O where D is the diffusion coefficient of the Brownian process.
Intuitively this says that for isotropic noise the time spent some domain proportional to how big the domain is and how large the escape walls are. We can use this to get the mean first passage time for SGD around degenerate saddle points under some simplifying assumptions.
Theorem D.2. Let W be a point such that the Hessian of the loss H (w*) is positive semidefinite. Let W = Brr, w²) be a small area about w² with local learning coefficient X(w*) with W. = {w e WIC(w) < c). assume
have small istropic noise D and that there is reflective boundary at height e along the wall, and some escape set (absorbing boundary) I(W) then the time it takes to traverse distance with error tolerance inversely proportional to the LLC.
Proof. First note that in the above picture, we effectively have Brownian motion along a submanifold We {w e WIC(w) < E), which we can treat as diffusion through a porous media where the pores have height e. Then from theorem D.Ilw we know that asymptotically as 5 - 0 we should have
We
T(W:)
(35)
(WWp(T&(WW))
and since we can take Wa| = VE). This gives the desired result.
While the above result requires isotropic noise and zero gradient, analogous results likely hold under weak anisotropy and weak gradients by considering potential driven Brownian motion on submanifold which is tilted behave like a porous media.
D.2 DIFFUSION COEFFICIENT HAS SCALAR APPROXIMATION
First we would like to prove that the diffusion coefficient is well-approximated by constant. To do this we prove handful of results. the following let 7 & be a small learning rate and let n denote the batch size. Letting Dr be the Caputo fractional derivative, take
Dg w: - (we-1) Ew
(36)
19

---

Published as a conference paper at ICLR 2026
to be the overdamped Langevin equation with
Swn=1 V 2D 2D(
(37)
where dW, is a d-dimensional Wiener process. Let T be the time to equilibration for an instance of the system. Then as + T, assume for almost all eigenvalues e, of the Hessian 1 of Cn (wt-1) we have e; < 1 are Ry O (which has been shown in (Sagun et al. 2016)). Furthermore, we start with the assumption that the full diffusion tensor is proportional to the Hessian so D(w,-1) x (wx-1)) which has been shown previously (both experimentally and rigorously under particular assumptions, see (Xie et al. 2021) and (Smith & Le 2018)) We start with the following:
Lemma D.1 (Effective Diffusion Tensor is Low Rank). As t + T, assuming L is in C2, the diffusion tensor is well-approximated by an effective diffusion tensor Dellw) with rank deff & d with w E R4
Proof. Since 11 x -1) and we know that almost all eigenvalues are R O, the result theorem. That is, approximating via the
follows almost immediately from the Eckart- Eckart-Young- truncated eigendecompossiion
Dx = Σ e.q.
T
(38)
and considering the ordering |eil 2 |02l 2 ...eel we get the approximation error
DD - D
Σe
c
(39)
k>
and if e, ss for all i k, then Σish of xz
Lemma D.2. As t + T. taking D(w))= = 2 (ww) Xie et al.
2021 p for batch size n and learning
rate y, For any there exists some choice of y and n such that there is scalar value with
ID(w)) - aIll < e
(40)
Proof. Since (wr) is symmetric, it can be rewritten as QAQT" D(we) where A diag(e (1, ...y ex) We can then take
D(w)) al = Q(A at)Q² and by the unitary invariance of the Frobenius norm we get:
(41)
[D(w,) - al = [(A al)ll
(42)
which is
Σ (e. - a)²
(43)
Then since for almost all i, e, we have
ca² + Σ (ej-oo) - a)²
(44)
where the sum is over all non-zero eigenvalues and c is the number of eigenvalues. Let a" argmin arg + E,(e, a)? Notice that since e, is eigenvalue of H (wx) we can rewrite it ej = Ree where e', is the corresponding eigenvalue in the unscaled Hessian.
Letting ef be the largest unscaled eigenvalue, notice that as 7 - 0 and/or + co that the value for e, dominates the sum, and all the other ej go to 0, so the sum is then
~ (d- 1]a² + (E1 a)²
(45)
so setting a E₁ we get
II(A aT)ll ~ (d - {)
(46)
= (d- (d-
(47)
and since the learning rate and the batch size can be made arbitrarily small/large, our result follows.
n
20

---

Published as a conference paper at ICLR 2026
We now prove the general form of the scalar diffusion coefficient some effective scale. This is well-known result within the diffusion literature (Bouchaud & Georges (990) but we include it here for completeness. We prove it for the homogeneous case. The inhomogeneous case follows from application of this to a restricted sub-domain.
Lemma D.3. Let & be some choice of length scale and dwalk be the walk dimension. The diffusion coefficient can be approximated by a scalar as Dg dist.
Proof. The effective diffusivity is defined as Dg
length traversal time Since R(() ~ 1. we get that
length
((t))ua so setting R(t) = & and rewriting t(E) as the time t such that R(t) ç. we get ~ Edhm can write
Dg
(48)
Cdoat
(49)
as desired.
D.3 STEADY STATES
Here we will give proofs of the results given in section
Lemma. Consider a subset of the parameter space W C. W such that the effective diffusion coefficient Dş is (approximately) constant on W. Suppose then that there exists steady state solutions
-76m [w]
on this subset w" so 0p(w" ,t) = 0. The steady-state distribution is then given by (w()) x
Proof. First, by definition of the steady state we have D2 p(w, t) 0 which reduces the fractional FPE to effectively the normal FPE, so we must solve the following PDE:
V (D(w, P(w,, t) <p(( L..
(50)
Now under the assumption that for all wi we E W that D(w)) D(w2), then the long-term behavior of the diffusion coefficient at length scale & can be approximated by the effective diffusion coefficient given in definition 3.1 giving
0 V (D&
up(w, L. [w])
(51)
One can also that the values of De and L. [w] are not dependent p (that is, the change in the probability of w does not change the loss or geometric properties determining diffusion at w) meaning that the SGD-FFPE reduces to linear partial differential at steady state solutions. The solution is then readily obtained by solving the normal Fokker-Planck equation, which is simply the
-y6
Boltzmann distribution for the system giving (w)) x
Pc
desired.
Corollary. Letting 7 = for simplicity, if L is the log-loss, then
m. mDDs x p(Xmmluu (w)
(52)
so
(( (()) ZmnD
Xm))
(53)
where De is the partition function. and p is the prior.
Proof. First, note that the empirical negative log loss
m
Cm [w]
p w)
(54)
m
This is a dimensionless quantity, however, we can consider the coarse coarse-graining graining of the parameter space by some scale & so that B(w,g) E). By taking the appropriate choice of measurement scale
21

---

Published as a conference paper at ICLR 2026
(given general regularity assumptions about the structure of the loss surface implicit in singular learning theory) we have that if w1, W2 E ((w, {) then Lm [w1] = Lm [w/] Now consider that:
mm (w)
II
p(vs [T,, w)
(55)
1=1
ppXmmlu (l/ww
(56)
Now given the result of lemma 3.3 one gets p. (w) that
for partition function Za. We then get
Z.
-Ggalo) [m]]
(e
JmDe -mmmmm())
(57)
p( pP (w)
(58)
Letting ZDD; be the appropriate partition function, the result then follows from application of Bayes' theorem.
Lemma. Suppose the loss function & is non-convex and non-constant on W. Then with spectral dimension dg as + oo with fractal dimension X(w(t)) on W W, the inequality de S >(w(t)) holds (in the small learning rate regime).
Proof. Consider two points w1; we be two points visited in the long timescale regime at times 4 /2 separated by distance R. If we suppose that there exists linear path connecting wi to w2 along the manifold and we remove all other paths linking the two points we have diffusion on a linear structure. Now using the definition of the walk dimension, following the are A of this restricted structure
gives (1) xx /. Busk but since this restricted structure has only a single path, it has walk dimension dwalk 2. Now, suppose that this is true for any pair of points. Notice that this implies that all points are connected by linear path at arbitrary distances along the loss manifold meaning the loss surface would have dwalk = 2. Furthermore this would imply that the loss does not change for any choice of parameter, violating the fact that it is non-constant so we must have dwalk > 2. Now since 2x(w) have de d d.
2x(w) and clearly if dwalk > 2 then 2d(w) X(w) so dg S X(w) dumk
duat
Corollary. For time t as t + co, we have dg S X(w(t)) where
lim 1 / X(w(())dt Xw((()
(59)
Proof. Let To be the time such that for all T > To the inequality of lemma 3.4 holds. Consider then a time T' >> To and consider the integral
Xw(t))dt= S." ((w(t)dtt+ + I' A(w(t))dt
(60)
and since To is finite we can take the first portion of this integral be a constant (since we know that the LLC is bounded above by where d is the number of free parameters):
[
X(w(t))dt= =
(61)
By the result of lemma 3.4 we have that for all times greater than To. we must have
X(w(t)dt 2 f" d,dt
(62)
and since d, is constant
(w(l))dl 2 (T - 7o)ds
(63)
which means that by adding C' to both sides and dividing by T get
1
(T - To)d C
T A(v(t))dt
(64)
T
T
From this get
1
L'" X(w(t))dt 2 d, (C (o- To)d
(65)
T
T
where the term (C-ro)d, vanishes T - oo since C must be finite.
22

---

Published as a conference paper at ICLR 2026
D.4 EXTENDED RESULTS
Below we give an extra result which explains a domain where the theory can fail, as well as discussion about complexities regarding how the diffusion process relaxes towards stationary states.
In the following result, consider the batge-batch small learning rate regime, What happens if instead of the long runtime being sub- aub-diffusive has a small linear component?
Proposition D.I. Suppose that R(()) ~ I. Foath ee for small constant and has no stationary distribution.
Proof. The proof is straightforward. Let 72 be the crossover time where for all > -2, cl ( so on long timescales the small linear term dominates the sub-diffusive term, so R(() for / > 72
can be effectively approximated as R((1) ~l + R((2)) + (( and as + oo the constant term of dominates. However, stationary distribution must be independent of time. This cannot be the case however as at any point in time the diffusive process has non-trivial movement away from initialization so the distribution p(w, t) spreads continuously over all timescales.
The thing that causes the problem with linear diffusion is if the space is unbounded. If one bounds the space with a reflective boundary one can recover a stationary state but the dynamics become more complicated. One way approach studying this system would be assume that the process eventually reaches global minima and that such minima form a connected submanifold. One could then consider certain directions on the manifold to be confining, and others to be free. Processes on this submanifold can be studied using tools like Morse- Baote theory.
E HOMOGENIZATION
Ultimately the theory presented here relies on the process of homogenization, which is a well-known technique in the study of diffusion. We will give basic informal overview here, but full treatment can be found (Cioranescu & Donato 1999). We will then discuss how the method used for estimating the local learning coefficient in (Lau al. 2024) is related to homogenization.
Homogenization a process used to understand diffusive processes where the underlying governing structure can have small but rapid variations small scales. These fluctuations might matter for diffusing particle on short length/time scales but they should effectively average out at some larger scale. A bit more formally, if we imagine something like a chemical concentration e (2; which is diffusing according to the PDE
Oct = V
Ot (D(
(67)
where the diffusion D coefficient varies rapidly when < 1. However, if D is bounded, then homogenization theory tells us that there some other function c given by e - 0 such that there is some effective PDE:
Oc Øt = v (D VV
(68)
where D is an effective diffusion coefficient which only varies over a much larger scale. This is effectively taking the PDE and averaging out the fluctuations over particular scale to get something that is easier model. When performing a homogenization one normally picks a scale that they are "averaging over". This scale can be picked somewhat arbitrarily but making the scale too large or too small can negatively impact how accurately one captures the dynamics of the system. If one takes the scale too small, homogenization is not effective. If one takes the scale too large, you start to ignore how the distribution of fluctuations can change in different areas of the media, leading to an inaccurate theory.
There is a sense in which the local learning coefficient estimation introduced in (Lau et al. 2024) is related to homogenization. For a particular value w in the parameter space (which is assumed to be local minima) and a ball Bs(w") ) of radius about w", they define the learning coefficient estimator
A(w" .0 mT [T.m (wr) Iam (w" ']]
(69)
23

---

Published as a conference paper at ICLR 2026
where Bs(w") and _ log The choice of is effectively the scale which one is homogenizing, and the estimate of the LLC is akin to the average fluctuation over that area. This is also why when trying to accurately estimate the LLC is recommended to not make å too large.
F ROLE OF THE FRACTIONAL DERIVATIVE
F.1 THE FRACTIONAL DERIVATIVE
The Caputo fractional derivative
(-" f'(t) Dtf(t) r(1 - a) ()- dr
(70)
is essentially like derivative with memory of past derivatives, weighted by power law decay in time controlled by a. To see this, one can consider two extreme cases. First, taking a - 0 you get the total net change of f(t) f()). Taking between the time t and the start time, a effectively controls how quickly you ignore the past.
1 you recover something more akin to the "slope"
If we want to see how it induces power law subdiffusion consider the linear function f(1) at + b. Assuming 0 < a < 1, we find
f(1(
11-0
(71)
r(2 a)
So notice that as a + 0 the process becomes more linear, so a controls how "sublinear" the process is.
F.2 FRACTAL DIMENSION AND FRACTIONAL DERIVATIVE
The relationship between the fractal dimension and the fractional derivative operator has been subject of investigation for nearly 3 decades, starting with (TATOM 1995). The authors used numerical simulations to study the relationship between the fractional derivative and the fractal dimensions of particular curves, finding linear relationship between the order the fractional operator and the fractal derivative. Since then, extensive theoretical results have been proven for different types of special functions (see (Liang Su 2024) for an overview). was proven in (Songping 2004) that there is a linear relationship between the Minkowski mankowski-BBuligand
Bouligand dimension of Weierstrass function and Minkowski-Bil Binkowski-Bouligand dimension of its corresponding fractional calculus. We hypothesize that the fractional derivative in the FFPE for SGD accounts for the change in X(w) as one moves through the parameter space. More concretely:
Hypothesis F1 (Shared Slopes). Let (tt) be the fractional derivative exponent time t. The value af) dA(w))
dt
Since at) is effectively local property of point that is related the derivative about the point, this should be unsurprising as the learning coefficient directly describes degenerate directions the space.
24

---

LR 0.1, Batch Size 64, Weight 0.001 Published as a conference paper at ICLR 2026
16
12
weight
0.1. Batch Size 32, Weight Decay 0.001
0.1, Batch Size Weight 0.001
50
250
limestep
(a) Jump in Weights with SGD Optimizer
15
LR 0.1, Batch Size 32, Weight Decay 0.001
Estinate
50
100
150
200
250
Timestep
(b) Drop LLC
Figure 5: The corresponding changes in weight vs. the LLC.
As mentioned in section 3 we see more complex dynamics when we operate in the grokking regime. Experimentally we see (figrre55 55)) that the appropriate choice of hyperparameters result in sudden large jumps in weight space (and the LLC) when the batch size is sufficiently small. The general sub-diffusive behaviour of these systems is captured by the fractional derivative in time D However, the large jumps indicate the need for a fractional derivative in space to fully account for grokking behavior. This could be done by introducing fractional Laplacian operator to equation 3.1. however we don't explore this analytically here. We do note however that the introduction the space fractional derivative is effectively the same Levy noise Langevin equation.
25

---

Published as a conference paper at ICLR 2026
Am
98
96
Accuracy
94
92
90
Trin LR- WD-0.001
Test LR=0.1, BS-64, WD=0.001
Train LR-0.1, 65-32, WD-0.001
lest LR=0.11, BS=32, WD=0.001
50
limestep
Figure 6: Train/test accuracy over time. Notice that sudden jumps the accuracy correspond to sudden jumps in the weights and the LLC.
We believe that this is evidence that the concept of stage boundaries and developmental stages introduced in (Wang et al. 2024a) is seemingly very natural way to discuss stages of learning. They suggest delineating phases of learming by critical points in the (noise mitigated) LLC evolution curve. Our experiments indicate that the rate of change of the local learning coefficient should roughly capture the impact of the time and space fractional derivatives. Discontinuities (or very sharp changes) seemingly account for the action the spatial fractional derivative, while more stable changes seemingly relate to actions of the time fractional derivative.
G EXPERIMENT DETAILS
For all experiments, we use the following configuration for the learning coefficient estimation:
Table 3: Hyperparameters for LLC Estimation
Hyperparameter
Value
optimizer_lr
le-5
optimizer localization
100.0
SGLD
num_chaing
aum_draws
400
nun_burnin_steps
0
jum_steps_bw_draww
G.1 MNIST
To investigate our theory using the MNIST dataset, we take a subset of 10000 images, and create 50/50 train-test split. We then conduct two different sets of experiments. The first set of experiments are ran on 50 identical models with different random initializations, each trained for 100 epochs with batch size 256 and a learning rate of 0.001. For the other set of experiments we run against set of 18 different architectures which vary in depth and layer widths, training these for 250 epochs but with the other parameters fixed (a table of architectures is provided appendix TT. We found for our purposes that it is sufficient to use basic set of hyperparameters for the estimator (appendix D. We compute the LLC every 100 steps, as well as log the displacement of the network from its initial position. We then take the final LLC to be the average over the last 10 estimates. We also performed extensive ablation experiments over optimizers and parameters on MNIST, whose details can be found in appendix H
26

---

Published as a conference paper at ICLR 2026
G.2 TINY IMAGENET
Tiny ImageNet (Le & Yang 2015 is subset of 200 classes of the full ImageNet dataset, which have been down-sampled to 64 x 64 pixels. In our experiments we rescale the images back to 224 x 224 well as apply the standard Imagenct normalization. We conduct experiments the following models pretrained on ImageNet:
ResNer18 (He et al.
2015)
ResNer34 (He et al.
2015)
VGGII SSmonyyn & Zisserman, 2015
To conduct our experiments we finetune the above mentioned models after replacing the original output layer with an output layer with 200 neurons. In order train this network we follow general fine-tuning practices. That is, we freeze the original weights, and fit the new classification head using the Adam optimizer for maximum 20 epochs with learning rate of 0.001 and a weight decay of 0.0001, with batch size of 128. If the loss does not decrease more than 0.0001 over 3 epochs, stop training, switching to SGD with weight decay and learning rate 0.00001 for 2000 steps with batch size 128. We note here that vision experiments the language model or MNIST experiments. This was done to test the theory on the "fine-tuning" stage of model development.
conducted slightly differently than
G.3 TINYSTORIES
The TinyStories dataset (Eldan & Li (0233 was selected as it allows us to explicitly test our theory on late stage training without training models from scratch, but where there are multiple reasonably sized pretrained models which we can compute the LLC for multiple times throughout training. In particular we run experiments on the following models from HuggingFace trained on TinyStories:
roneneldann/TnnStoriees// IM hug
nickpprottinyllama-155M hhug
roneneldann/funStoriis--- hhug C)
Each these models are trained for 1000 steps with a batch size of 16, with a learning rate of 0.00001, with the L.C computed every 100 steps.
G.4 POSTERIOR CONCENTRATION
To test the posterior concentration predictions we use simple toy model and dataset where one can reasonably approximate the Bayesian posterior. use the moons dataset (Pedregosa et al. 201T) with 512 samples, noise ratio of 0.2, and batch size of 128. Using this train 2 hidden layer ReLU network where each hidden layer has 64 neurons. Each model is trained using SGD with learning rate of 0.01 for total of 200 epochs. We do this for 500 random initializations of the network on the same dataset. At the end training for each model, we compute the LLC,, discarding Since
any non-converged training solutions found by SGD and use these to seed SGLD. In particular, since the Bayesian posterior will concentrate around the model with the lowest loss and lowest learning coefficient, samples which have the lowest loss and the lowest LLC. For each run of SGLD we take draw 1000 samples with 200 burn steps, and 10 steps per sample with learning rate of 0,00001.
theory is largely about the local posterior, take the
use the SGD
H MNIST ABLATIONS
Ablations were ran across optimization various hyperparameters for a fully connected network trained on MNIST to better understand the effects hyperparameter choices have on the diffusion characteristics. Experiments are ran for both SGD and Adam to test if the theory is effective for adaptive optimizers. We highlight some of these experiments here.
27

---

Published as a conference paper at ICLR 2026
Optimizer
dg (mean)
dg (std)
Alinal (mean)
Alinal (std)
Test Acc (mean)
adam
0,4061
0.9068
3.0957
5.7533
90.4297
sgd
7.8165
10.2494
12.5270
11.8393
94.0592
Spectral dimension d,, optimbber-igd, wd=2.0, momentum-@.0
Arsu, optiniber-igd, wdw0 momeroum-0.00
spedial mieesti
6.0
6m0
Learing
Learing
(a) Spectral dimension
(b) learning coefficient (final)
Figure 7: Learning coefficient and spectral dimension for vanilla SGD with varying batch size and learning rate.
Spectral dimension
optimizer=agd, Peerson=-0.0B8 Spearman= 0.200
50
40
30
20
10-
0.0
0.2
0.4
0.6
Momentum
Figure 8: Correlation between momentum and spectral dimension.
dimonsion waddm,,
3.0
64-
3.5
66
Learinng rata
Learning rube
(a) Spectral dimension
(b) Learning coefficient (fina)
Figure 9: Learning coefficient and spectral dimension for Adam with varying batch size and learning rate.
28

---

Published as a conference paper at ICLR 2026
Spectral dimension d, vs Batch size
Atsul VS Batch size
optimizer=sdd Pearson=0. 332, Speerman=0.434
optimizer-sgd, Pcarson-O. Speamann-0400
50
50
40
10-
10-
100
120
200
120
(a) Spectral dimension
(b) Learning coefficient (final)
Figure 10: Correlation Learning coefficient and spectral dimension for with batch size for SGD
Spectral dimansion d, vs Learning rate
Area vs Leamingg rate optim ssdiimizer-sdd,, Pearson=-0,069 Spearman- 0.033
aptimizer-sgd, Pecrson--O 2,060 Spearman--0.40
50-
dimansion
40
20
10
10-
0-
0.00
0.02
0.04
0.08
0.00
02
04
06
0.08
0.10
(a) Spectral dimension
(b) Learning coefficient (final)
Figure 11: Correlation of Learning coefficient and spectral dimension with learning rate for SGD
We note that interestingly when using Adam, the spectral dimension seems have a stronger correlation with performance than the learning coefficient as can be seen in figures 12 and 13 Another interesting phenomena that supports our theory is that there relatively little correlatio between À and the learning rate, but there is relatively substantial correlation between the spectral dimension de and the learning rate (figure IT] which aligns well with the theory. We note that the correlation between À and the batch size reflects the sensitivity of the empirical LLC estimator of LLau et al. 2024) to the batch size.
Final test accuracy Spectral dimension d, opti optimizer=sdd, Pearson-0.33 00.00, Spearman-
Final accuracy dimension optimizer madmmaam Pearson-0.320, Spearman-
100
Fiail be1 Sccurcyy
Fhatl bezt sccrscy
90
80
70-
30
50
1.0
2.0
3.0
dimensioo d,
dimension d,
(a) SGD test accuracy
(b) Adam test accuracy
Figure 12: Correlation of spectral dimension with test accuracy for Adam and SGD.
29

---

Published as a conference paper at ICLR 2026
Final test eccuracy V5 Anea
Final test accuracy Vs Ansal
opémizee=sgd, Pearson=0.413 Spearman=0 0.442
optimizer=adam, Pearson=0 0.66, Spearma= 084
100
100
Final test accurac)
Finil test accurac)
90
88
0
20
30
40
50
10
15
25
30
Avail
(a) SGD test accuracy
(b) Adam test accuracy
Figure 13: Correlation the final learning coefficient with test accuracy for Adam and SGD.
Observed Predicted finel R
Observed Predicted final R
optimizer=sgd. Pearson-O, 389 Spearman-0.977
ootimizer-adam, Pearson Spearman-0,
Observed froe
Observed firel
8000
20
10
20
50
2000
4000
6000
8000
Predicted hnall
Predicted tinal
(a) Predicted vs. true displacement for SGD
(b) Predicted vs. true displacement for Adam
Figure 14: Predicted vs. true displacements.
Predictive corripredicted observed zer-5dd
128
0190
0.00
0973
Leaming
Figure 15: The correlation between the predicted displacement and the true displacement.
30

---

Published as a conference paper at ICLR 2026
Pearson cortt, finel, lembaa, fine
optimizer-sg
0.5
0.0
0.4
02
40
-02
-04
0361
Leamtng note
Figure 16: The correlation between the final displacement and the final vs. batch size and learning rate.
lamida optiriiiiieeeee
02
b00ch
-02
-04
32
32
-08
-08
(a) The correlation between the final displacement and(b) The correlation between final displacement and the final vs. batch size and learning rate for SGD. the final vs. batch size and learning rate for adam.
Figure 17: Correlations between the displacement and learning coefficient.
H.0.1 SLT AND ADAM
Note that in figure i7 seems that while the learning coefficient correlates very strongly with SGD, Adam does not. We suggest that this is because the LLC measures a quantity associated with the same Riemannian metric over the data as is used by SGD while adam effectively changes the metric structure via preconditioning. It is well known that variable-metris methods, including adaptive optimizers like Adam, can be interpreted as performing gradient descent in Riemannian metric defined by positive-definite preconditioner matrix. In the case of Adam, this preconditioner the diagonal matrix built from the running average of squared gradients. Recent work has made this connection explicit, showing that Adam can be viewed as an approximate natural-gradient method using a diagonal empirical Fisher information matrix as data-dependent metric, the singularity structure of the Adam metric likely different.
I ADDITIONAL EXPERIMENTS AND RESULTS
I.1 DIFFUSION PREDICTION ACCURACY
If the weight diffusion is indeed fractal, should expect that using the spectral dimension as estimated by equation 17 we should be able to accurately predict the movement the weights. While this is in the presented in table I they the volume of models trained We can see the histogram of the R² scores in figure 18] An important thing note here that these estimations don't explicitly account for the early super-diffusive
presented in the main body for MNIST due to
31

---

Published as a conference paper at ICLR 2026
behaviour seen during initial training, large periods of super diffusion should decrease the accuracy. However, other settings where we can account for early super-diftusion behaviours, the predictions become nearly exact. An instance of this is in the case of the Tiny ImageNet models where explicitly factor out the adaptive training component and simply fit to the SGI) component at the end of training (as theory explicitly cares about late training stages of SGD). This shift in dynamics can be seen in figure [9 However, in instances where we can start with SGD, if the model is already near equilibrium, we can see the sub-diffusive behaviour very early with vanilla SGD. An example of this can be seen in figure 20
Distribution of R Values
Mean 0.755
+1 0.164
Frequency
4
2
0.7
RR Score
Figure 18:
The distribution of the R2 scores for identical MNIST models.
Weight displacement a=0.00, d 5=5.7, Ra=1.00
32
Displacement R(t)
30
28
26
R(
24
22
20
18
1000
2000
3000
4000
5000
6000
7000
Timestep
Figure 19: Displacement of the ResNet18 model on Tiny ImageNet. A sharp transition from Adam to SGD can be near step 5000, at which point the dynamics become distinctly subdiffusive.
32

---

Published as a conference paper at ICLR 2026
Observed vs predicted displacement R²=0.98
Observed R(L)
Predicted R(t) from log-log
0.020
0.015
R(L)
0.010
0.005
0.000
200
400
600
800
1000
Timestep
Figure 20: Displacement of the TinyStories-33m model along with the displacement predicted by the theory.
L.1.1 TINYSTORIES USING ADAM
Model name
dg
y2
TtooSStries-1N IM
32.05
36.6
0.57
0.99
Tlama 15M
77.02
48.06
0.31
0.97
TinyStories-33M
40
31.95
0.39
0.98
Table 4: Results for language models trained using the Adam optimizer.
Note that while the theory does seem to hold for Adam in some cases, it is less consistent, reflecting the more complex way in which Adam interacts with the geometry.
I.2 AVERAGE SPECTRAL DIMENSION
We also check the result of corollary 3.3
This can be seen in figure 21
I.3 CIFAR EXPERIMENTS
We ran additional experiments using various convolutional architectures on CIFARIO to explore how the super-diffusive component impacts the theoretical results. We find that while these models display stronger super-diffusive behaviours (increasing intensity with model size) the super-diffusive behaviour attenuates quickly enough that the theoretical results still hold, even when the super- diffusive behaviour is not factored out. Note as well that the runtimes were not as long, so one might expect the super-diffusive component to have larger impact. In the following restrict ourselves to subset of 10000 samples from the CIFARIO dataset where all models are initialized using Xavier initialization, with 0 bias and ReLU activations.
33

---

Published as a conference paper at ICLR 2026
Spectral Dimension vs. LLC (Average)
LLC Spectral Dim
LLC (avg) with error
50
LLC (Average)
40
20
10
15
25
Spectral Dimension
Figure 21: The average LLC vs. the spectral dimension. These results align with corollary
3.3
Table 5: Spectral Dimension vs. LLC across runs for convolutional architectures
Num Params
Spect. Dim
LLC
Channels
Batch
Epochs
LR
1184
0.085
2.585 0.779
32
64
50
0.001
19936
3.374
5.929 0.963
32, 64
64
50
0.001
94304
4.887
11.829 + 1.960
32, 64, 128
64
50
0.001
372928
7.483
16.176 2.175
64, 128, 256
64
50
0.001
ConvNet Architectures (CIFAR) Weight Displacement
Parameters
Total Model Parameters 19936
Total Model Parameters 94304
10a
Weight Dispoccment (log]
6x
-0
10"
10
(log)
Figure 22: Weight displacement over time across subset of the CIFARIO dataset for Convolutional Architectures.
We also investigated the weight displacement for residual architectures. These can be seen table where the first number is the number of channels in between residual blocks.
34

---

Published as a conference paper at ICLR 2026
Table 6: Spectral Dimension vs. LLC across runs for residual architectures
Num Params
Spect. Dim
LLC (final)
Channels
Batch
Epochs
LR
1216
0.085
2.585 0.779
32
64
50
0.001
208448
3,481
14.729 + 4.394
16, 32, 64
64
50
0,001
167424
4.534
10.408 + 2.127
32, 64
50
0.001
831616
5.795
24.756 7.614
32, 64, 128
64
50
0.001
872640
9.652
37.731 + 8.332
16, 32, 64, 128
50
0.001
Residual ConvNet Architectures (CIFAR) Weight Displacement
Total Model Parameters 208448
lotal Model Perameters 872640
Model Parameters 1216
Weight Displocement (log)
Total Model Parameters 167424
Total Model Parameters 831616
100
10
Timestep (log)
Figure 23: Weight displacement over time across a subset of the CIFARIO dataset for residual architectures.
we look at both figures 23 and 22 we can see that larger models exhibit more super-diffusive behavior.
35

---

Published as a conference paper at ICLR 2026
J
MNIST TABLES
Table 7: Spectral Dimension vs. LLC with hyperparameters for varying model architectures.
Num Params
Spect. Dim
LLC (Final)
Layers
242304
2,910
30.514
784, 256, 128, 64
125568
3.749
31.424
784, 128, 64, 128, 64
117376
7.373
40.128
784, 128, 64, 32, 64, 32, 64
244384
8.310
38.866
784, 256, 128, 64, 32,
127296
8.661
33.361
784, 128, 64, 128, 64, 32
126112
12.458
33.385
784, 128, 64, 128, 64,
121472
12.909
61.920
784, 128, 64, 32, 64, 32, 64, 32, 64
118016
12.996
23.551
784, 128, 64, 128
109184
13.763
20.324
784, 128, 64
125568
14.603
73.419
784, 128, 64, 32, 64, 32, 64, 32, 64, 32, 64
234752
15.021
23.299
784, 256, 128
143680
15.838
51.999
784, 128, 64, 128, 64, 128, 64, 32
141952
19.357
44.844
784, 128, 64, 128, 64, 128, 64
134400
20.268
36.134
784, 128, 64, 128, 64, 128
143872
20.925
41.541
784, 128, 64, 128, 64, 256
244032
21.719
35.879
784, 256, 128, 64, 32
158336
24.465
71.359
784, 128, 64, 128, 64, 128, 64, 128, 64
In table g we can see instances where the spectral dimension is negative. This related to the model having negative LLC (which occurs generally when there are no near-stable solutions). When negative LLCs occur, it is almost always near initialization where the model also displays super-diffusive displacement (as is likely moving strongly along steep gradient). The negative spectral dimension is a remnant of how the spectral dimension is computed in our case. One can reasonably remove occurrences of a negative LLC and perform the estimation which would capture the behavior of the model on the fractal landscape it traverses away from initialization (which is shown to theoretically valid in appendix B.33). transparency we do do this as illustrates that SGD has complex multifaceted dynamies.
36

---

Published as a conference paper at ICLR 2026
Table 8: Spectral Dimension vs. LLC across runs with different initializations.
Num Params
Spect. Dim
LLC
Channels/Layers
Batch
Epochs
LR
110912
-0.327
19.553
784, 128, 64,
256
100
0.001
110912
-0.252
18.440
784, 128, 64, 32
256
100
0.001
110912
0.048
21.552
784, 128, 64, 32
256
100
0.001
110912
0.163
20.071
784, 128, 64,
256
100
0.001
110912
0.356
19.785
784, 128, 62,
256
100
0.001
110912
0.455
20.136
784, 128, 62,
256
100
0.001
110912
1,462
22.654
784, 128, 62,
256
100
0.001
110912
2.396
21.117
784, 128, 62,
256
100
0.001
110912
2,401
20.125
784, 128, 62
256
100
0.001
110912
2.407
19.733
784, 128, 64,
256
100
0.001
110912
3.790
19.628
784, 128, 64,
256
100
0.001
110912
3.824
19.682
784, 128, 62,
256
100
0.001
110912
3.839
21.184
784, 128, 62,
256
100
0.001
110912
3.882
21.821
784, 128, 62,
256
100
0.001
110912
3,899
20.301
784, 128, 64,
256
100
0.001
110912
5.013
22.756
784, 128, 62.
256
100
0.001
110912
5.881
20.381
784, 128, 64,
256
100
0.001
110912
5.968
20.591
784, 128, 62,
256
100
0.001
110912
6.410
22.074
784, 128, 64,
256
100
0.001
110912
6,556
20.343
784, 128, 62,
256
100
0.001
110912
6.713
20.250
784, 128, 62,
256
100
0.001
110912
6,801
21.056
784, 128, 62,
256
100
0.001
110912
7.036
21.386
784, 128, 62,
256
100
0.001
110912
7.240
19.237
784, 14, 64,
256
100
0.001
110912
7.311
21.323
784, 128, 62,
256
100
0.001
110912
7.360
20.886
784. 128, 62,
256
100
0.001
110912
7.794
21.724
784, 128, 64,
256
100
0.001
110912
8.293
23.803
784, 128, 64,
256
100
0.001
110912
8.739
20.638
784, 128, 62,
256
100
0.001
110912
8.749
21.454
784, 128, 64,
256
100
0.001
110912
8.796
22.440
784, 128, 62,
256
100
0.001
110912
8.812
22.730
784, 128, 64,
256
100
0.001
110912
9.081
21.760
784, 128, 62.
256
100
0.001
110912
9.408
19.758
784. 128, 62,
256
100
0.001
110912
9.605
21.916
784, 128, 64,
256
100
0.001
110912
9.795
20.633
784, 128, 64,
256
100
0.001
110912
9,838
20.142
784, 128, 64,
256
100
0.001
110912
9.898
22.295
784, 128, 64,
256
100
0.001
110912
10.230
22.213
784, 128, 64,
256
100
0.001
110912
10.742
22.556
784, 128, 64,
256
100
0.001
110912
11.060
23.296
784, 128, 64,
256
100
0.001
110912
11.966
20.767
784, 128, 64,
256
100
0.001
110912
12.204
20.973
784, 128, 64,
256
100
0.001
110912
12.308
19.877
784. 128, 62,
256
100
0.001
110912
13.253
20.701
784, 128, 64,
256
100
0.001
110912
13,889
22.577
784, 128, 64,
256
100
0.001
110912
13.907
21.487
784, 128, 64,
256
100
0.001
110912
13.952
23.129
784, 128, 64,
256
100
0.001
110912
14.258
21.386
784, 128, 64,
256
100
0.001
110912
15.516
23.590
784, 128, 64, 32
256
100
0.001
37

---

Published as a conference paper at ICLR 2026
K COMPUTE RESOURCES
All experiments were run on a single Nvidia RTX A4000 GPU, a single Intel Xeon W-2223 CPU, and 32GB of physical RAM. The individual runs vary greatly in compute times, ranging from minutes to ~ hour, with the total compute time at x 30 hours.
L
LLM USAGE DISCLOSURE
LLMs were used in this work primarily to locate results (formulas, etc) in papers and textbooks.
38


Reviews:
[{"review_id": "dlmxMggrUf", "reviewer": "Reviewer_psyE", "summary": "This paper proposes a theoretical connection between stochastic gradient descent (SGD) dynamics and almost Bayesian behavior, by analyzing the steady-state of SGD through the lens of fractional diffusion and spectral geometry. The authors introduce the concept of local learning capacity (LLC) as an analog of an inverse temperature, and relate it to the spectral dimension of the loss landscape via a family of scaling laws. They provide a mathematical derivation of these relations and test them empirically on toy data and small-scale MNIST experiments.", "questions": "- Can you clarify the dependence of V on ρ and r in Eqs. 5–7? It currently looks like it depends on those implicitly, but the notation hides it. Also, in Eq. 6, you say λ is defined for an arbitrary choice of a – does that mean λ is independent of a? If yes, can you show why?\n- Why is Definition 3.3 a definition and not a theorem? It seems to express a relation between previously defined quantities, so it’s not really definitional. Please clarify or justify.\n- The Alexander–Orbach relation is usually derived for homogeneous fractal media. Under what assumptions does it hold here, and how do you know those hold for your loss landscapes?\n- How exactly is the scale ξ chosen for D_ξ in Eq. 13? Is it a fixed number, dataset dependent, or tuned? Have you checked sensitivity to it?\n- You introduce the “near-stability hypothesis” (Hyp. 3.1). Have you tried to test this empirically (e.g., by measuring distances to nearby minima or metastable states)?\n- Could you add at least some quantitative evaluation for the LLC–spectral-dimension relation? e.g., correlation coefficients, confidence intervals, or plots across architectures or datasets. Right now, it’s impossible to tell how strong the relation is.\n- Please fix the citation formatting (use \\citep and \\citet correctly) and typos throughout.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Ambitious conceptual link between stochastic dynamics, spectral geometry, and Bayesian ideas\n- The LLC notion is at least intuitively appealing and might motivate future work on quantifying local \"temperature\" or effective capacity in SGD\n- Some of the derivations are mathematically nontrivial and show that the authors know the literature on stochastic processes and singular learning theory\n- The paper is generally readable and nicely typeset, with good figures (especially Fig. 1 for intuition)", "weaknesses": "- **Steady-state assumption**: many derivations rely on SGD reaching or approximating a steady-state distribution. The authors mention that this may not actually exist or be reachable in practice, but then they still use it for all the main results. This feels like a big logical gap that should be discussed much more clearly.\n- **Notation and definitions**: this part is honestly confusing.  \n  - V(ε) is first defined as a function of ε (Eq. 5), but later it takes a ball B(w*, ε) as argument (Eq. 7). It’s unclear if that’s the same V or a new one.  \n  - The dependence on ρ and r is dropped mid-derivation.  \n  - “walker dimension” vs. “walk dimension”: these seem to be the same thing, just inconsistent terminology.  \n  - Definition 3.3 really reads like a derived result (it relates quantities already defined before), so calling it a definition doesn’t make sense. It should be a theorem or corollary, or at least justified.\n- **Assumptions**: The “near-stability hypothesis” (Hyp. 3.1) is just stated but never tested. Likewise, the use of the Alexander–Orbach relation assumes homogeneity that is unlikely to hold for neural loss landscapes. There’s no discussion of what happens if it fails.\n- **Experiments are extremely limited**:  \n  - Only a toy Moons example (few hundred points) and small MNIST subset are shown.  \n  - No error bars, no statistical tests, no correlation coefficients... so the claim that LLC correlates with the spectral dimension is just a visual impression.  \n  - The authors even note that the bound is loose, so it’s unclear if the theoretical quantities actually predict anything.  \n  - There are no ablations or sensitivity studies (learning rate, batch size, the coarse-graining scale ξ, etc.).\n- **Practical relevance unclear**: given the weak and small-scale experiments, it’s hard to see what we should take away practically. The theory could still be interesting, but it would help to discuss possible uses or ways to estimate LLC in large-scale setups.\n- **Minor stuff**: lots of typos (e.g., “Kullback-Liebler”, “whcih”) and inconsistent citation style ((double parentheses)) all over the place. Would be good to fix that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8rqncL1Lq", "reviewer": "Reviewer_v9AS", "summary": "The paper models long-time diffusion dynamics of SGD using the fractional Fokker–Planck equation (FFPE), which can describe sub-diffusive behavior induced by degenerate loss landscapes on a fractal geometry. The authors derived the specific solution of the steady-state distribution of FPPE based on a defined effective diffusion coefficient. They connected it to a tempered Bayesian posterior, modulated by local accessibility constraints determined by the local learning coefficient (LLC). Moreover, they demonstrated that the spectral dimension of SGD is bounded by the fractal dimension, and empirically verified this on MNIST.", "questions": "1. To assess the empirical fidelity of the “almost Bayesian” claim, can you quantify how closely the steady-state distribution $p_{s}(w)$ matches a true Bayesian posterior of SGLD samples as in Figure 5 (e.g., via KL divergence / Wasserstein distance)?\n2. Have you attempted to compute LLC or spectral dimensions on larger or more complex networks (e.g., CNNs on ImageNet)?\n3. The fractional Fokker–Planck framework only focuses on sub-diffusion. How would the model extend to super-diffusive or Lévy-like early dynamics, and would this modify the steady-state form?\n4. Can you provide any quantitative evidence (e.g., LLC temporal stability plots) supporting the near-stability hypothesis 3.1 during late training?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper employs fractional SDE and spectral dimension analysis in a coherent framework, which introduces a novel insight to bridge the gap between SGD and Bayesian methods. All lemmas and corollaries are provided with detailed proofs.\n2. The idea of modeling anomalous diffusion using the FFPE basis grounded in fractal geometry is reasonable and effective.", "weaknesses": "1. The experimental validation only utilizes the small MNIST and CIFAR datasets without larger-scale input, which limits the empirical generality of the proposed inequality $d_{s} \\leq \\lambda(w(t))$.\n2. Experiments use only vanilla SGD without momentum or adaptive variants (Adam, RMSProp). It is unclear whether the proposed fractional diffusion framework can be extended to these more practical optimizers.\n3. There is no ablation study about how hyperparameters (e.g., batch size, weight decay, epochs) affect the spectral dimension or LLC evolution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLd9j64Sjb", "reviewer": "Reviewer_zNv7", "summary": "The paper analyses the behaviour of SGD for NN optimization using concepts from singular learning theory. The key result is a connection between fractal dimension and local behavior of SGD as quantified by a local learning coefficient (LLC), and another one is a link between the diffusive process induced by SGD and a tempered version of the Bayesian posterior, with higher concentration for parts of the parameter space that are easier to reach from an initial condition. The theory is validated using relatively simple empirical experiments that validate the theories related to fractal dimension and LLC.", "questions": "What can a practitioner learn about your work? Would it be useful to track the spectral dimension in some scenarios and use the information to guide some design choices? What would we gain over tracking LLC? I am very much willing to adapt my score once understanding better the practical significance of the results, but would like to remark that the paper could well be published even as a purely theoretical paper.\n\nThe estimated LLC is effectively constant for spectral dimensions between 0 and 15 in both Figure 2 and 3. Why is this? Is the lower bound overly loose for low spectral dimensions, or is this near-constant behavior explained by something else?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The research question is important: SGD is very widely used and the previous theoretical analysis of its properties relies on simplifying assumptions, some of which are relaxed here, and hence there is clear need for improved understanding. Even though the paper uses somewhat heavy theoretical machinery that may not be accessible for all readers, the key results are understandable for the broader community. I am not an expert with this range of theoretical tools myself, and hence cannot directly evaluate the significance of the contribution, but I find the concrete results convincing.\n\nThe empirical experimentation is well justified, with clear explanation of why the experiment is conducted like it was. It is on a small scale, but sufficient for shedding light on the theory.", "weaknesses": "It is somewhat difficult to grasp the importance of the spectral dimension, as it is not directly linked to quantities people usually consider. In effect, one needs to be already familiar with Lau et al. (2024); this is not directly a limitation as it is valuable to build on the LLC measure, but the paper would work better as a stand-alone piece if LLC was explained and illustrated a bit more detail here.\n\nThe empirical validation is limited to confirming the bounds of Lemma 3.4 and Corollary 3.2 that are both about the spectral dimension. As someone who found Corollary 3.1 a more tangible contact point I would have liked to see something brought from Appendix C to the main paper. I understand the choice though, as that result is more difficult to communicate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KOk62ekJd", "reviewer": "Reviewer_C7eA", "summary": "The paper studies the dynamics of SGD algorithms using as tools the singular learning theory. The dynamics of SGD can be approximately modeled using a Langevin SDE, with the exception that the dynamics are super-diffusive during early training and sub-diffusive afterwards, which the authors model using the time fractional Fokker-Planck equation. To capture the local geometry, the local learning coefficient (LLC) and the spectral dimension are utilized, with the LLC essentially acting as a mass dimension and the spectrial dimension determining the reachable area. The resulting stationary state of the Fokker-Planck equation was derived, demonstrating the connection to the Bayes posterior.", "questions": "I am not familar with the topic of this paper, as such I do not have strong opinions on it. Please see the Weaknesses listed above.\n\nMinor:\n\nLine 065: to related -> to relate", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The theoretical analysis is rigorous.\n2. Empirical evidence verifies some theoretical claims.", "weaknesses": "1. One of the main claim of the paper is the connection of the stationary distribution of SGD dynamics to the Bayes posterior, yet this was not empirically validated.\n2. I found limited background provided on singular learning theory; I think this would be beneficial for readers not that familar with singular learning theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)