Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian
Authors: Yiran Zhang
Weihang Xu
Mo Zhou
Maryam Fazel
Simon Shaolei Du
Venue: ICLR 2026 Poster (2026)
Keywords: over-parameterization
global convergence
non-convex optimization
mixtures of Gaussians
score-based generative models
Primary area: optimization

Abstract: Score matching has become a central training objective in modern generative modeling, particularly in diffusion models, where it is used to learn high-dimensional data distributions through the estimation of score functions. Despite its empirical success, the theoretical understanding of the optimization behavior of score matching, particularly in over-parameterized regimes, remains limited.
  In this work, we study gradient descent for training over-parameterized models to learn a single Gaussian distribution. Specifically, we use a student model with $n$ learnable parameters, motivated by the structure of a Gaussian mixture model, and train it on data generated from a single ground-truth Gaussian using the population score matching objective.
  We analyze the optimization dynamics under multiple regimes. When the noise scale is sufficiently large, we prove a global convergence result for gradient descent, which resembles the known behavior of gradient EM in over-parameterized settings.
  In the low-noise regime, we identify the existence of a stationary point, highlighting the difficulty of proving global convergence in this case. 
  Nevertheless, we show convergence under certain initialization conditions: when the parameters are initialized to be exponentially small, gradient descent ensures convergence of all parameters to the ground truth. 
  We further give an example where, without the exponentially small initialization, the parameters may not converge to the ground truth.
  Finally, we consider the case of random initialization, where parameters are sampled from a Gaussian distribution far from the ground truth. We prove that, with high probability, only one parameter converges while the others diverge to infinity, yet the loss still converges to zero with a $1/\tau$ rate, where $\tau$ is the number of iterations. We also establish a nearly matching lower bound on the convergence rate in this regime.
  This is the first work to establish global convergence guarantees for Gaussian mixtures with at least three components under the score matching framework.

Published as a conference paper at ICLR 2026
CONVERGENCE DYNAMICS OF OVER-PARAMETERIZED SCORE MATCHING FOR A SINGLE GAUSSIAN
Yiran Zhang
Weihang Xu University of Washington Seattle, WA 98105, USA xuwhêcs wa washington.eer edu: edu
Institute for Interdisciplinary Information Sciences Tsinghua University
Beijing 100084, China
changyir22mails.eesee wwwggguuu.ssinnaa...a.. en
Mo Zhou
Maryam Fazel
University of Washington Seattle, WA 98105, USA
University of Washington Seattle, WA 98105, USA
2ochoul7@cs washington. edu
mfazel eduu
Simon Shaolei Du University of Washington Seattle, WA 98105, USA ssduêcs washington..dd
ABSTRACT
Score matching has become a central training objective in modern generative mod- eling, particularly in diffusion models, where it is used to learn high-dimensional data distributions through the estimation of score functions. Despite its empiri- cal success, the theoretical understanding of the optimization behavior of score matching, particularly over-parameterizze regimes, remains limited. work, we study gradient descent for training over-parameterized models to learn a single Gaussian distribution. Specifically, we use a student model with n learnable parameters, motivated by the structure of Gaussian mixture model, and train it on data generated from a single ground-truth Gaussian using the population matching objective. We analyze the optimization dynamics under multiple regimes. When the noise scale is sufficiently large, we prove global convergence result for gradient descent, which resembles the known behavior of gradient EM in over-parameterized settings. In the low-noise regime, we identify the existence of a stationary point, highlighting the difficulty of proving global convergence in this case. Nevertheless, we show convergence under certain initialization conditions: when the parameters are initialized to be exponentially small, gradient descent ensures convergence of all parameters to the ground truth. We further give an exam- ple where, without the exponentially small initialization, the parameters may not converge to the ground truth. Finally, consider the case of random initialization, where parameters are sampled from Gaussian distribution far from the ground truth. We prove that, with high probability, only parameter converges while the others diverge to infinity, yet the loss still converges to zero with 1/T rate, where is the number of iterations. We also establish nearly matching lower bound on the convergence rate this regime. This is the first work to establish global convergence guarantees for Gaussian mixtures with least three components under the score matching framework.
this
INTRODUCTION
Diffusion models (Song & Ermon 2019 Song al.) 2020) have become leading framework for generative modeling, achieving state-of-the-an atr results across wide range of visual computing tasks
Work done while Yiran Zhang was visiting the University of Washington.

---

Published as a conference paper at ICLR 2026
(Po et al., 2024). They form the foundation of modern image and audio synthesis systems such as DALL-E2 (Ramesh et al., 2022) and Imagen (Saharia et al. 2022). At the core of this framework is the idea of learning score functions. i.e., the gradients of the log-density of intermediate noisy distributions, in order to approximate the reverse of a forward diffusion process.
While diffusion models are empirically successful, their theoretical understanding remains limited. A substantial body of recent work has studied the convergence properties of diffusion- -baeed sampling perfect or approximate score oracle (De Bortoli et al.
algorithms under the assumption of access
202T; Block et al. 2020; Chen et al., 2023a; De Bortoli 2022 Lee et al. 2022] Liu et al. 2022
Pidstrigach 2022; Wibisono & Yingxi Yang 2022 Chen et al. [022] 2023c Lee et al. 2023 Chen
et al. 2023b; Li et al., [023a] [Benton et al., 2023). These works establish rigorous convergence
guarantees for sampling and restoration tasks but rely on access to ideal score functions.
More recently, (Shah et al. (203)) analyzed the optimization landscape of the denoising diffusion probabilistic model (DDPM) training objective in the setting where both the student and ground-true distributions are mixtures of two Gaussians. They showed that there exists an algorithm that performs gradient descent at two different noise scales and successfully recovers the ground-truth parameters, assuming random initialization and minimum separation between the components. Their result demonstrates that, in the
and rell-separated regime, gradient descent on the DDPM objective can achieve global recovery. Since the DDPM training objective coincides with the score matching loss at the population level (see Appendix A of et al. (2022)) their result can also be interpreted
global convergence guarantee for score matching this specific setting. However, their approach requires the number of model parameters to match the true number of Gaussians (e.g., two), which is often unknown in advance. In practice, diffusion models are typically overparameterized, using far more parameters than needed. Therefore, we focus on overparameterized score matching, where an oversized network learns a single Gaussian.
The Gaussian mixture model has also been studied in the context of the gradient EM algorithm, whose connection the score matching objective was established in Shah et al. (2023). Recently, Xu et al.) (2024) provided a global convergence guarantee for gradient EM in the werrpparametrrized setting, which further highlights the practical and theoretical importance of this regime. These developments motivate the central question of this work:
Can we prove the global convergence of gradient descent on the score matching objective in the over-parameterized setting?
In this work, we consider the setting where the ground truth is single Gaussian distribution, and the student network has n learnable parameters, following the setup in Shah et al. (2023). A formal definition is given in Section 1.3 Our main contributions are as follows:
We first analyze the regime where the noise scale is sufficiently large. Building on techniques from the gradient EM analysis in Xu et al. (2024), we prove convergence of gradient descent on the score matching objective (Theorem 2. This result extends the connection between DDPM and EM observed in (hah al. 2023) the perrpparameterized setting.
We then consider the more challenging small-t regime. We begin by showing the existence stationary point with nonzero loss, which shows that global convergence cannot be guaranteed from arbitrary initialization (Theorem 3.1).
Nevertheless, we show that if all student parameters initialized to be exponentially close to zero, then all parameters converge to the ground truth (Theorem 3.2 Our analysis introduces a technique for tracking the evolution of the geometric center of the parameters. We further complement this result with a counterexample, showing that such exponentially small initialization is necessary for full parameter recovery (Theorem 3.4..
Finally, we study the case where the goal is only to minimize the training loss, not nec- essarily to recover all parameters. We show that when student parameters are initialized independently from a Gaussian distribution centered far from the ground truth, then with high probability, one parameter converges to the ground truth while the others diverge to infinity, yet the loss still vanishes (Corollary 3.6). This regime requires a delicate analysis of the gradient dynamics. We first show that one parameter converges to the ground truth faster
2

---

Published as a conference paper at ICLR 2026
than the others. Then, we prove that once this parameter is sufficiently close to the ground truth, the minimum distance between the ground truth and all other parameters increases. We further show that the loss in this setting converges at rate O(1/T), and establish a nearly matching lower bound of 9 / for any constant é >> O (Theorem 3.7). in sharp contrast to the linear convergence rate in the exactly parameterized case (Shah al. 2023)
1.1 RELATED WORKS
Theory of Score Estimation. Several works have studied the theoretical aspects of score-bases generative modeling. One line of work investigates how well score-based models can generate samples from complex distributions, Koehler& & Vuong (2023) showed that Langevin diffusion with data-dependent initialization can learn multimodal distributions, such as mixtures of Gaussians, available. Another line of work examines the optimization behavior of score-based models. [i et al. (203b)); Hann et al. (024)) Wang et al. (2024) provided convergence guarantees for gradient descent in diffusion model training. The statistical complexity of score matching has also been investigated. Koehler et al. (2022) established connections between the statistical efficiency of score matching and functional properties of the underlying distribution. Pabbaraju et al. (2023) analyzed score matching for neepolynomial distributions, and Wibisono al. (2024) derived minimax optimal rates for nonparametric score estimation in high dimensions. Denoising diffusion probabilistic models (DDPMs), introduced by Ho et al. (2020), are a widely used framework of score-based generative modeling, where training is typically performed using the score matching objective at varying noise levels. In the context of distribution learning, [Shah itt al. (2023) analyzed gradient descent on the DDPM training objective and proved recovery of a two-componett Gaussian mixture under suitable initialization and separation assumptions. Chen et al. (2024) showed that there exists an algorithm capable of learning k-component Gaussian mixtures using the score matching objective, although their focus is statistical feasibility rather than the convergence behavior of gradient-base based optimization.
provided that accurate score estimates
Learning Gaussian Mixtures. There is large body of work on learning Gaussian mixture models. Recent results such as Gatmiry et al.] (2024); CCen et al. (2024) focus on designing computationally efficient algorithms that recover Gaussian mixtures with small estimation error. Many classical approaches rely on a well-separatedness assumption, where the centers of the Gaussians are assumed to be sufficiently far apart (Liu & Li 2022. Kothari et 2018 Diakonikolas et al. 2018). Another line of work studies the Expectation Maxiiinzanaiiiiiiiiioonn (EM) algorithm, which is closely related to Gaussian mixture estimation (Daskalakis et al., 2017 Xu et al.) 2016: Wu Zhou 2021: Dwivedi jet al. 2020). Jin al. (2016) showed that EM may fail to achieve global convergence for mixtures with more than two components. In contrast, Xu al. (2024) established global convergence of gradient EM in over- parammteeetee setting, where the ground truth is single Gaussian and the learner uses multiple components.
Theory of Over-parameterizer Teacher-Student Settings. Over-pane paramttrrizaaiinn is popular topic in recent theoretical work, with focus on both landscape and algorithmic properties. A common result is the slowdown of convergence, observed in different settings such as Gaussian mixtures (Dwivedi et al.] (2018; Wu & Zhou| 2021). two-layer neural networks (Xu & Du 2023
Richert et | 2022) and nonconvex matrix sensing problems (Xiong et al. 2023 Zhang al. 2021
Ding et al. 2024, /Zhuo et al., 2024).
1.2 KEY CHALLENGES
In this section, we introduce several unique challenges that distinguish our analysis from prior work.
Cubic gradient terms. Unlike the gradient EM framework studied in Xu et al. (2024), where the update dynamics involve only linear terms, the gradient the score matching loss in our setting contains cubic interactions in the parameters. When the student parameters μ4 are far from the ground truth, these cubic terms dominate the gradient direction, making the dynamics substantially more difficult to control. Prior work such as Shah et al. (2023) only analyzed the regime where the parameters are already close to the ground truth, thereby avoiding this complication. Extending the analysis beyond the wrau-ground-truth regime requires new techniques.
3

---

Published as a conference paper at ICLR 2026
Multiple convergence regimes. A second challenge arises from the fact that, results show, the over- peeeparaaeteriz student model does not guarantee that all parameters converge to the ground truth. Instead, different initialization schemes lead to qualitatively different convergence behaviors: in some cases, all parameters converge to the ground truth, while in others only a single parameter converges and the remaining ones diverge. This multiplicity of possible regimes makes the analysis intricate, since we must carefully characterize the conditions under which each type of convergence occurs.
1.3 PRELIMINARIES
Diffusion Model Background. We begin by reviewing the background on diffusion models. Let go denote the data distribution on IR, and let Xo ~ go be a random variable drawn from it. The two main components in diffusion models are the forward process and the reverse process. The forward process transforms samples from the data distribution into noise, for instance via the Orhstein-UUllnbeekk
(OU) process:
dx, X, dt d2dW, with Xo 20:
where (Wi) is standard Brownian motion in Rd We use qe to denote the distribution of Xe, the solution to the OU process at time t. Note that for X, Gt.
X = exp(-I)Xo + /1 exp( 22) with Xo ~ qo, Ze ~N0,0,
Here (p, Σ) denotes the Gaussian distribution.
The reverse process then transforms noise into samples, thus performing generative modeling. Ideally, this could be achieved by the following reverse-time stochastic differential equation for terminal time T':
'x;-" {X;- vV In dt + v2dW, with X 91'
where now W, is the reversed Brownian motion. In this reverse process, the iterate X; is distributed according to gr for every [0, T", so that the final iterate Xf is distributed according the data distribution qu. The function Va In known as the score function, depends on the unknown data distribution. practice, it is approximated by minimizing the score matching loss:
C, (s,) = Ex.-g. (X,) V, In gn (X,))]]
Throughout this paper, || || denotes the Euclidean (L.2) norm.
The setup for ground truth Gaussians is as follows (in this paper analyze the case of 1):
m
NMA Ia),
= 40
m
where j E RRd, 1. Shah et al. 2023 a simple calculation showed that
Ni la), where vi, js exp(- -tt,
and
m Σ
exp (pp(-11 2//e112/2)
Va (=)
w (x) -x, x,, where (()(())
Σ exp(-11 13 1 222/2)
Our Setting.
Motivated by Shah et al. (2023) and the optimal score form above, we model the student network using learnable parameters Ml -.,.... An Rd follows:
(x( Σ (.)( - where (4,, = fas exp((tt) t), wr,((x) = exp(- exp(-11o-1h.t11/11
E exp(-1|x -|1x
In this paper, consider the over-parameterrize setting. The ground truth consists of single Gaussian component, i.e., m = 1, and we denote its mean by j jí. Therefore,
Va ( 14 x, where 14 = p exp(
4

---

Published as a conference paper at ICLR 2026
The number of learnable parameters in the student model is 2. In this case, 9: NN Ia). the loss is
(((s)) = Ex.ng. (X)) - Va l = BB-N(a
Σ wr 11) 14
In this paper, we analyze gradient descent applied to the population loss. Since we focus on optimization for fixed f, we treat the loss (x(8)) as function of the variables jigt for i ,,..., and directly run gradient descent on these variables. Given step size > 0, the update rule is
Pit (T+1)
where denotes the iteration index. When the context is clear, abbreviate Mi, by b4 and abbreviate wi. by w;. We denote uy as the value of at the 7-th iteration of gradient descent. We use (T) denote the loss the 7-th iteration. Also, when the context clear, we simply use C to denote (st). We define the function v(x) == Ei= W/ (z)x), which depends on the current parameters l4 (i.e., P4 = Pi, for fixed t).
WARM UP: CONVERGENCE UNDER LARGE NOISE REGIME
We begin by analyzing the setting where the noise scale is large. In this case, we prove the following convergence guarantee:
Theorem 2.1. Ler M = max, [1 - μ and suppose t > logn + log M + 2. If the step size
satisfies S O xズ then after T iterations of gradient descent, the loss satisfies
n² d2
(((s((T)) S
TJT
The proof of the above theorem is deferred Appendix When is large enough, we prove an 0(1 (T) convergence rate in Theorem [.1 matching the best known rate for gradient EM (Xu et al. 2024) in this pre-paaraeterizee setting. Our update is closely related gradient EM in this regime, as we explain below.
Proof Sketch. For such t, by definition, || 42 S M exp( S 3n In the following, use H4 to denote pi,e.
For each we can find that
(s(st) BewNV(u
Σ
We (x)))) -M²
exp( (pp--111---01 12112/2)
= =EewNigddd
p4 14
==1 expl - || M4 12,12/2)
27
exp( exp(-1|z ||0 (w: 122)
= =Ey~Ni0JJa
(14: 14)
(1)
2j=1 exp( (11 14; 11
Therefore, without loss generality, we may assume μo by shifting each μ4 to μ4 - pt. In the following, we assume that n4 = 0. and [ < 3n We have the following calculation for the gradient of loss:
Lemma 2.2 (Part of Lemma B.I Let x ~ N(0, Ia), n² = O. We define v(x) = E. ws (2))) Then, we have
V C = 2E,
wx(x) u - 2wx(2)0(2)0(xx) ms
-2w;(x) Σ (x(xx) (v(x) ps) μj + 3w;(z) (o(z) ((())))(())))))))))))))))))))))) v(x)
5

---

Published as a conference paper at ICLR 2026
Notice that when the parameters ||/||| are sufficiently small, intuitively, the first term wy (z))(() dominates the gradient expression, as the remaining terms involve cubic interactions in 14 and can be viewed higher-order corrections. Also, it is known in Xu et al. (2024) that the first term wy (z))(a) the same as the gradient of the population loss in gradient EM. Therefore, by a similar proof as prove the convergence result. This connection highlights
Theorem 2 in Xu al. (2024), the relationship between DDPM training and the gradient EM algorithm in the over-paranin wver-parameterized setting; The connection in the well-separated regime was also previously discussed by [Shah al. (2023).
3 CONVERGENCE FOR SMALL NOISE
We now consider the small-noise regime. Since we analyze a fixed noise scale write pi,r as p and ji as μ when no ambiguity arises. Gradient descent is applied to optimize these parameters Ms.. Therefore, for each theorem in this section about 14 ss the corresponding result works for
ju = exp(() 14 and jx exp(()µ² in the original setting. In the small-/ regime, we can view exp(() as a small number. Since a nonzero amounts to uniform rescaling of the parameters, we set 1 = 0 in this section without loss of generality.
3.1 UPRM-UP: A STATIONARY POINT WITH NONZERO LOSS
In this subsection, we present a simple example demonstrating the existence of point where the loss is nonzero but the gradient vanishes-specinee
local maximum.
Theorem 3.1. Let 2 and p* For there is a point μ (M1,-. ". ,n) where the loss is nonzero but the gradient is zero.
Theorem 3.1
demonstrates the existence of a stationary point with nonzero loss, indicating that gradient descent may fail to converge when initialized arbitrarily. To establish meaningful convergence guarantees, it is therefore necessary to impose conditions on the initialization. This highlights the difficulty of analyzing the small-noise regime, where the optimization landscape becomes more complex. The proof is deferred Appendix [.I]
Proof Sketch. The main intuition of the proof is to consider a configuration where some μ4 = 0 and others satisfy || - oc, making their contribution negligible and yielding near-zero loss. Since the loss is also zero at the origin, but strictly positive at some intermediate points, continuity implies the existence of local maximum.
Based on this intuition, we outline the proof as follows. Let e = (1,0,0, ,0) E Rd. We consider the case where μ4 sei, μl2 -se1, = (0, -.. ,0) for where a positive real number.
Then, by symmetry, we have C,, = for all 2 3. Moreover, by Lemma 2.2 the gradients Vpu & and Vuz vanish on all coordinates except the first (because each gradient linear combinations of the J4 "s). By symmetry, we have V,, C V -=
Notice that the losses for maximize the loss. Therefore, for this s, the gradient vanishes for each μ4:-
and s +oo are both zero, by continuity, there must be an to
In what follows, consider two cases: (1) all μ4 initialized close in Section 3.2 and (2) random Gaussian initialization in Section 3.3
3.2 INITIALIZATION EXPONENTIALLY CLOSE TO O ENSURES PARAMETER CONVERGENCE
In this section, we consider the case where the initialization exponentially close to We have the following theorem.
Theorem 3.2. Let Mo == ||~*|| 2 0. Assume the initialization satisfies (1+,;01 < 10 exp (10))dM))
for all i. Then, for step size 17 = (.t.zz) ), gradient descent ensures that each μ4 converges to μ*.
Moreover, there exists 0
(n(log n log Mo) such that for any T > T,
Co(so(T)) V-(T -
where 7 S 7)
5

---

Published as a conference paper at ICLR 2026
Theorem 3.2
establishes that, under exponentially small initialization, all μ4, converge the ground truth. Moreover, once the parameters are sufficiently close, the loss decreases at a rate of ((1vvF).
The proof is deferred to Appendix D.2
Proof Sketch. The main intuition of the proof is to maintain a reference point that converges to the ground truth and to show that all parameters stay close to this point throughout training.
To be more precise, the proof proceeds as follows. By eq. Tb. we can change the case to J = 0, and there exists |||||| = Mo such that for each i, 11 ulll 10d (10)) By applying an
orthonormal transformation (i.e., a rotation in IR)), we may assume without loss of generality that 1 M9e1. Recall that e₁ = (1,0, ... ,0) E TR...
We maintain value A initialized by A Mo and the fact that as long A Z on we always
have || A S cxpl 100 (A())))) We first need the following lemma which shows
the gradients for each μ4 are close.
Lemma 3.3 (See also Lemma D.2. Let A 1. K = 10000nd B 10
Assume that for any i S n, 44 Aeill S B. Then any i, we have that
1
6KBA
l| -Ae
71
n
Let the update of be ( A AT)
Then by this lemma, we can bound
|| (((+1)
A(r+)), by 1| (1(r+1) A(r+)) $ | (7) - + nllV - e
Therefore,
prove that as long A(r)
1 always have || A <
10endexp( 109ndexp(10Pnd (( by induction. When A(t) S
6n we can view it as the initialization in
Theorem 2.I. so directly applying Theorem 2.1 gives the result.
One may notice that requiring ||wi|| to be exponentially small is too strong. However, we can prove that the exponential term is necessary. To establish this exponential dependence, we analyze the scenario where Mo is large.
Theorem 3.4. Let Mo > 10 Vdl no and p* = (Mo, 0,... ,0) Assume that the initialization is 1i (0) (ea, 0, 0, 0) and pr (U) (0. 0. 0. ... 0) for i = 2, Then, as long as the step size n 1/(00))), Pr converges to μ², and ||ll|| - 0o for 2, 2,
where 6o = exp( MM0
Theorem 3.4 shows that there is an initialization where all parameters are exponentially close to but not all parameters converge to the ground truth. By the above two theorems, we can see that although it is possible for all parameters to converge to the ground truth, the requirement for the initialization is very strict. This result illustrates that over over-porane parameterization under small noise can lead to unstable convergence behavior. The proof of this theorem is deferred to Appendix [D4
Proof Sketch. Similar to the discussion above, we can assume that the ground truth is 0 and the initialization 141 = (M, - 60, 0, O), /22 = (Mo, 0, ........... ,).
In each step let μ1 = (M - c, 0, 0), f2 = (M, 0, ... 0). Through careful analysis, we can lower bound the rate of increase in e and upper bound the rate of decrease in during each iteration.
At last, when is large enough, Theorem 3.5 gives the result by considering different Mo. O
3.3 RANDOM INITIALIZATION FAR FROM GROUND TRUTH STILL ENSURES LOSS CONVERGENCE
In this section, we consider the case where the initialization random and far from the ground truth. Specifically, we show that if each parameter is initialized independently from N(u, Ia), where is far from μ²" then with high probability, exactly one μ4 converges to the ground truth, while all other Hyys 's diverge to infinity.
Theorem 3.5. Let Mg = || μ²ll 2 /d n 10 (μ is an arbitrary vector in Rd)). Let the initialization satisfy: (1) |n -ull & M2/3 for all j. and (2) there exists such that || 1r
7

---

Published as a conference paper at ICLR 2026
Il ---11+ Mo for all + in. Then, as long as the step size 1) < 1, we have that pio converges to y², and I| oc for + in. Also, there exists ( /n) such that when T, the loss (o(su) converges with rate
1 Lo(so(T)) - a T)
Note that when all μr 's are randomly initialized from A (N, Ia), the probability that the initialization satisfies both conditions is at least (1 - n² MM (see Lemma D.3). Therefore, we have the following corollary:
Corollary 3.6. Lei all pi be initialized by (u, Ia) and step size < 1. Let Mo -11u -uu--"1l /d 10 Then with probability at least (1 n² m2 M, there exists T = O(M3/M/33)) /n) such that
when T > T, the loss Lo(sa) converges with rate
Lo(so(T)) n(T TT
Moreover, one μi converges to y², while the others diverge to infinity.
We also have lower bound for the convergence rate, which shows that the O(1/T) convergence bound is nearly optimal.
Theorem 3.7. Let Mo- = 11--~~11 10 n0 and nj < 1. Assume that the initialization satisfies:
(1) || -il S / for all j. and (2) there exists in such that (1 -n'll 2 11 -p" Mo 1/3 for all j # io- Lete > 0 be constant. Then.for any >> 0 (it can depend on n, d, Mo. but not on T), when T large enough, we have
C
Lo(so(T)) 2 7l+e*
The above results show that if the parameters are initialized independently from Gaussian distribution centered far from the ground truth, then with high probability, only one parameter converges while the others diverge to infinity. We further prove that the loss converges at rate of ((1/T), and establish a nearly matching lower bound of ((1/r+to) for any constant e >> O.
Comparing the results in this section with those in Section 3.2 observe sharp contrast between the two regimes; one where all parameters converge, and one where only a single parameter does. This suggests that analyzing the training dynamics for initializations that lie between these two extremes may be challenging. Understanding this intermediate regime remains an interesting open question.
The full proofs of the above three results are complicated and can be found in Appendix D.3 with a central preliminary lemma in Appendix C Below we give proof sketch:
Proof Sketch. The proofs of the above theorems proceed by analyzing the training dynamics two stages. In the first stage, show that || decreases more rapidly than the other ||ار|| values. As result, Mig becomes very close to the ground truth yz",, while the other μ4 's remain far from μ²"
In the second stage, we prove that once pio is sufficiently close μ² and the others far, the minimum distance minzzio ||/ | increases over time. Also, 14io remains close to μ²"
Here we discuss how to prove the above theorems in more detail. Similar the analysis in the previous sections, we can modify the setting such that the ground truth is μz 0, and all μas are initialized near Mae1 (i.e., p M.ez). We can prove that in the initialization, with high probability:
1. all 44's satisfy ||m Moll < M
2. there exists i such that for any j , i, we have ||
|| > ||m]|| + M,
Without loss of generality, we assume that the the second condition is 1.
We analyze the training dynamics in two stages. At the end of the first stage, we have |||||| S 3M5 and for each é 2 2, || 2 Mn/2. In the second stage, in each iteration we let min ||]] We prove that we always have that [|m 1/M3, and M always increases.
8

---

Published as a conference paper at ICLR 2026
A3
160
1.2
2095
7500
37530
20006
tooation
torstion
(a) Norms || vs. iteration for M
(b) Norms | vs. iteration for M
0
30
logiito ation)
oglieeraiion
(c) Log-log plot of loss vs. iteration for M 4 (d) Log-log plot of loss iteration for M 6
Figure Experimental results illustrating two convergence behaviors. (a, c) For M 4. all parameters converge to the ground truth, consistent with Section 3.2 (b, d) For M 6, only one
parameter converges while the others diverge, consistent with Section 3.3 The loss curve (d) exhibits convergence rate close (11/r) in the final phase.
Stage 1. At this stage, we need to show that || decreases rapidly, while each μ4 with 2 2 changes slowly.
We first prove that in this stage, Vus Based on this fact, we can prove that in Ollog Mo/1) iterations, || becomes less than sxy while decrease of || for i 2 2 are very small. Therefore, after Ollog Mo/1) iterations, we have || S 8M3 and ||l|| 2 2 for other
n: L is near μ4 and Vu. for 2 2 are exponentially small.
Stage 2. At this stage, consider the update in iteration. Let M min || In the first Me. Therefore, we have || 1/M3 first. We prove that the property maintains and M increases by induction. Thus, we always have M 2 10 vd 10 and 14 's diverge for i 2. Moreover, by careful analysis the speed M increases, loss convergence rate.
iteration of this stage, have M > 2
prove the
4 EXPERIMENTS
In this section, we conduct numerical experiments to illustrate the two convergence behaviors predicted by our theory. We set d d333 3, and take the ground-truth Each parameter μи is initialized independently as (M, 0, 0) + 10-? 7 where 2; (o, Ia). The values of M are indicated the corresponding captions. The results are shown Figure !
be (0, 0, 0).
Training is performed using gradient descent with learning rate 0.01, for 20,000 iterations, and batch size of 20,000. Figures (a) and (b) show the evolution of || while Figures (c) and (d) display the log-log plot of the loss over iterations. Panels (a) and (c) correspond to the setting of Section 3.2 where all parameters converge to the ground truth. Panels (b) and (d) illustrate the behavior under the regime of Section 3.3 where only one parameter converges and the others diverge. Although the perturbation fixed at 10 10-? 7 it exceeds the critical threshold exp( (xp(-poly(M)) when M = resulting in qualitatively different outcome. Notably, in panel (d), the log-log loss curve approaches a linear slope approximately - after a warm-up stage, consistent with a convergence rate close to O(1/7).

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we study the problem of learning a single Gaussian distribution using an over- parameterized student model trained with the score matching objective. We show that when the noise scale is large, gradient descent provably converges to the ground truth. In contrast, when the noise scale is small, the loss may still converge, but the training dynamics become unstable and highly sensitive initialization. This highlights the importance of large noise in ensuring stable and predictable training dynamics in score-based generative models.
Although we focus exclusively on the case where the ground truth is single Gaussian component, our analysis reveals that even this seemingly simple case exhibits rich and subtle dynamics under over-paramettrizztion. Extending the theory to more complex ground-truth distributions, such as multi-component Gaussian mixtures, remains a challenging and important direction for future work.
There are also several additional directions suggested by our analysis. First, in practical diffusion models, training performed using time-averaged score matching loss that aggregates gradients from multiple noise levels. Our results provide characterization of the gradient dynamics at fixed time, and understanding how these fixed-time components interact under a full f-averaging scheme is interesting and nontrivial next step. Second, while theoretical study focuses gradient descent, score-based models are trained in practice using stochastic or adaptive gradient methods. Developing a theoretical framework that captures the behavior of these optimizers in the over-parameterized regime is an important open problem.
6 ACKNOWLEDGEMENTS
W. Xu, M. Zhou, and M. Fazel were partially supported by the award NSF TRIPODS II DMS- 2023 166. M. Fazel's work was supported in part by awards CCF 2212261, CCF 2312775, and the Moorthy Family professorship. S. Du's work supported in part by NSF CCF 2212261, NSF IIS 2143493, NSF IIS 2229881, Alfred P. Sloan Research Fellowship, and Schmidt Sciences AI 2050 Fellowship.
REPRODUCIBLLITY STATEMENT
All theoretical settings and assumptions are stated Section 1.3 Section 4
The experiment details are in
We also provide our code in the supplementary materials.
REFERENCES
Joe Benton, George Deligiannidis, and Arnaud Doucet. Error bounds for flow matching methods. arXiv preprint arXiv:2305. 260. 2023.
Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto- encoders and langevin sampling. arXiv preprint arXiv:2002.00107 2020.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pp. 9735-4773.. PMLR, 2023a.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215. 2022.
Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. Advances in Neural Information Processing Systems, 36:68552-68575. 2023b.
Sitan Chen, Giannis Daras, and Alex Dimakis. Restoration-degradaee beyond linear diffusions: A non-asy con-asymptotic analysis for ddim-type samplers. In International Conference on Machine Learning, pp. 4462-4484. PMLR, 2023c.
10

---

Published as a conference paper at ICLR 2026
Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404.18891 2024.
Constantinos Daskalakis, Christos Tzamos, and Manolis Zampetakis. Ten steps of em suffice for mixtures of two gaussians. In Conference on Learning Theory, pp. 704-710. PMLR, 2017.
Valentin De Bortoli, Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.052: 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances Neural Information Processing Systems, 34:17695-17709, 2021.
Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1047-1060, 2018.
L. Ding, D. Drusvyatskiy, M. Fazel, and Z. Harchaoui. Flat minima generalize for low-rank matrix recovery. IMA Journal Information and Inference, 2024. To appear.
Raaz Dwivedi, Koulik Khamaru, Martin J Wainwright, Michael Jordan, et al. Theoretical guarantees for em under misspecified gaussian mixture models. Advances in Neural Information Processing Systems, 31, 2018
Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Martin Wainwright, Michael Jordan, and Bin Yu. Sharp analysis of on Artificial Intelligence and Statistics, pp. 1866-1876. PMLR, 2020.
for weakly identifiable models. In International Conference
Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404./8869 2024.
Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arXiv:2401.15604, 2024.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6851 2020.
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin Wainwright, and Michael Jordan. Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences. Advances in neural information processing systems, 29, 2016.
Frederic Koehler and Thuy-Duong Vuong. Sampling multimodal distributions with the vanilla score: Benefits data-based initialization. arXiv preprint 1rXiv:2310.01775 2023.
Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. arXiv preprint arXi::2210.0077 2022.
Pravesh K Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Proceedings of the 50th Anmual ACM SIGACT Symposium on Theory of Computing, pp. 1035-1046, 2018.
Beatrice Laurent and Pascal Massart, Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp. 1302-1338, 2000.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882, 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pp. 946-985. PMLR, 2023.
Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards faster nom-asymptotic convergence for diffusion-based generative models. arXiv preprint arXiv:2306.099: 002221.1.000.... 2023a.

---

Published as a conference paper at ICLR 2026
Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36:2097-2127, 2023b.
Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1248-1261, 2022.
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. 14699, 2022.
Chirag Pabbaraju, Dhruv Rohatgi, Anish Prasad Sevekari, Holden Lee, Ankur Moitra, and Andrej Risteski. Provable benefits of score matching. Advances in Neural Information Processing Systems, 36:61306-61333 2023.
Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865 2022.
Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. In Computer Graphics Forum, volume 43, pp. e15063. Wiley Online Library, 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022.
Fredericke Richert, Roman Worschech, and Bernd Rosenow. Soft mode in the dynamics of over- realizable online learning for soft committee machines. Physical Review E, 105(5). 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 01.36479-364999 2022.
Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. 2023.
Advances in Neural Information Processing Systems, 36:
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basre generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Charles M Stein. Estimation of the mean a multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Yuqing Wang, Ye He, and Molei Tao. Evaluating the design space of diffusion-based generative models. arXiv preprint arXiv:2406.12838 2024.
Larry Wasserman. All of statistics: a concise course in statistical inference. Springer Science & Business Media, 2013.
Andre Wibisono and Kaylee Yingxi Yang. Convergence in kl divergence of the inexact langevin algorithm with application to acore-based generative models. arXiv e-prints, pp. 2r111-2211, 2022.
Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. In The Thirty Seventh Annual Conference Learning Theory, pp. 4958-499) PMLR, 2024.
Yihong Wu and Harrison H Zhou. Randomly initialized algorithm for two-component gaussian mixture achieves near optimality o(yn) iterations. Mathematical Statistics and Learning, 4(3), 2021.
12

---

Published as a conference paper at ICLR 2026
Nuoya Xiong. Lijun Ding, and Simon S Du. How over-parameeerizztion slows down gradient descent in matrix sensing: The curses of symmetry and initialization. arXiv preprint arXiv:2310.01769, 2023.
Ji Xu, Daniel Hsu, and Arian Maleki. Global analysis of expectation maximization for mixtures of two gaussians. Advances in Neural Information Processing Systems, 29, 2016.
Weihang Xu and Simon Du. Over-paranen parameterization exponentially slows down gradient descent for learning a single neuron. In The Thirty Sixth Annual Conference on Learning Theory, pp- 1155-1198, PMLR, 2023.
Weihang Xu, Maryam Fazel, and Simon S Du. Toward global convergence of gradient for over-parameterized gaussian mixture models. arXiv preprint arXiv:2407.004900 2024.
Jialun Zhang, Salar Fattahi, and Richard Y Zhang, Preconditioned gradient descent for over- parameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems, 34:5985-5996, 2021.
Jiacheng Zhuo, Jeongyeol Kwon, Nhat Ho, and Constantine Caramanis. On the computational and statistical complexity of over-parameterized matrix sensing. Journal of Machine Learning Research, 25(169):1-4,,2 2024.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
1.1
Related Works
Key Challenges
1.3
Preliminaries
2
Warm Up: Convergence under Large Noise Regime
3
Convergence for Small Noise
3.1
Warm-Up: A Stationary Point with Nonzero Loss
Initialization Exponentially Close to 0 Ensures Parameter Convergence
3.3
Random Initialization Far from Ground Truth Still Ensures Loss Convergence
Experiments
Conclusion
10
Acknowledgements
10
7 Reproducibility Statement
10
Additional Preliminaries
15
B Proofs for Section 2
15
Omitted Prool for Theorem
16
A Preliminary Lemma for Appendix D.3
20
D Proofs for Section 3
36
D. Proof of Theorem
36
D.2 Proof of Theorem 3.2
37
Proof of Section
39
D.4 Proof for Theorem 3,4
42
LLM Usage Declaration
44
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL PRELIMINARIES
The following Gaussian tail bound is a direct consequence of Lemma 1 in Laurent & Massart (2000).
Lemma A.1 (Gaussian Tail Bound). Let d be a positive integer. For any positive real number t vd.
we have
(o. |||=||2 2 1) S exp 1-(1-vaf
We also need the following Mill's incquality, which can be found in Wasserman (2013 Lemma A.2 (Mill's Inequality). For any t >> O, we have
1 exp( exp(-t2/2)
I - 7 1
Pr
We need the following Stein's 's lemma:
Lemma A.3 (Stein's 's Lemma, Stein 1981 For (N,, Ia)) and differentiable function
: R + R, we have
(x 1)] 2 (
if the two expectations in the above identity exist.
B PROOFS FOR SECTION
In this section, we denote C(r) as the (81) at iteration т. We define
exp (-11x -111x-17777 12/22)
u (z)x)((((x)
Σラ= exp
(-1|s- - 14, 12/2
Whenever there is no ambiguity, we omit the superscript (r) for notational simplicity. Notice that without loss of generality, we assumed that μ Thus the loss
Σ we (x:
L = G=EENNOLLL
First, we need prove the following lemma:
Lemma B.1. Assume that μ¹º O. Let (00 la). and let μ4 be the parameters of the student network. We define v(x) = Σ, (, Then, we have
V nx L 2F
[0 (x) + (x(x))(T)) (m=v(xx) (xc ni]
= 2 wx(x)(()) + (x)x) Σ (1())uyH - (w)(2))()))(()) ps
(w) Σ (5( (u(x)] p3) ny + 3w; (o())T(())) o(x)
Proof. By symmetricity, we just need to prove the case when
Direct calculation shows the following fact:
[wh (z) wi (2)) (x m1), if 1,
V ny (x)
{ wi (I) 1
(x( (x W1),
otherwise.
Also, direct calculation shows that
V (x)) wx(x)) wi(x))((x)) m.) v(x)).
15

---

Published as a conference paper at ICLR 2026
Thus, have
wi(x)/4 My (w)(x)
A1 L = 2Eg
w1 (x) Σ wi(x)pp +
Σ
0u1
2E,
[wi (2) + v(x) (wi (x))((()) (a -1)
2B, W1 (z) + wi (z)x(x) (M1 (m))) (z -mi)
By Stein's 's lemma (Lemma A.3), we have Ex
[w] (x))(() (n1 - [xz))
V wi (x) Σ wr(r) (11 ((x))
Ex
Eg
wi (1(x)(11 - u(x))u(x) (M1 v(x)) + w1 (x)) Σ
- v(x))
w1 Σ (1)) Σ (mn
"unst)a-EXus - e(z)))
-E in (z)((()) v()))o()) (11- - v(x)) + w (2) Σ (x)())p (m v(x))
(1()))(())(() (m1 v(x)) w1(x) v(x) (,)) ( v(x))
Therefore, by combining the two equations above, we have
VA, C -E
(z))()( + (z))(())) ( v()) (o - m1)
-E,,
w (z))((( (x))u(()) (m - v(x)) + (I()((( (#1 v(x))
mi(x))(s) (1)()))-w(r)))) (x) (m v(x)) v(x) + un E (x(()))114 (m vxx))
2
-w1(s)0(2)0(s) (MI -()-)(x)) wi (x)x(x) (ew
-2E,
wi(x))(a)) w (x) Σ wn(x))u A1 2w1 (z) p
2wn (x) o(x)
()w5(()) (0oTTTTyws((1 3 (x))(( (3
B. OMITTED PROOF FOR THEOREM
We need the following lemma, which is the Lemma 18 in Xu et al. (2024). Lemma B.2. Let be constant such that 0 < c
<
3d
1.
We have
E (exp (c][æ][)] + 5
Now, we prove the local smoothness of the loss function.
Lemma B.3. Let μ (fi.... be the parameters of the student network. Let ô (8,...,, km)) be the perturbation of the parameters. Let ê = max; ||6:1I. . = max; ||wel], Assume that ó, satisfy the following conditions:
1
8 8<p< and <
3
12d'
16

---

Published as a conference paper at ICLR 2026
Then for each i, we have
|V 5)
v.C(p)) & 1528vd,
where V C is the gradient of & with respect to the ith vector (14; + 6; or p4)-
Proof. For any i, we define wx(p)pp)) as the ith weight of the student network with parameters μ.
Then, for any x, have
||- (14++))))
+6;11
exp
w + 6)
Σn exp
[Jag +
18- Mis 11
exp (16.11[[1=]1| + [((l) exp (- __1,,11"
exp
Σn exp
|| Ms
exp (-116x [lxxx))) exp
exp
16
2
S exp (20(1=z)] + pp) + 8*) ()(())))))
Similarly, we have
exp
| 2
exp (- [[]=|] + (Ixl)))) exp
116:11?
Σx exp
|| 1ut-pell !1? exp (116 [|[|==]|||| (lx exp
16x111
exp
(-20([|==| + pp) 82) (x(p))
Thus, by Lemma B.
we have
[[ + 6) V.((H)ll
2E
Ies (2l + (0)1,(00l1 5) (=(0))))(((0111 p+ Σ wa (2l 6) + 6).
Σ + wall 5)
w +)) +0)+ Σ
5.k
+ Σ
wy(x/pp ())x((/l +5)r((01 (1x+ -)0-101)
j.k.
+ 8 (wx 5) w=(x/11))>² 1² 18 > |wy 8) (1y((/)))11 ]
<2Em~
exp (48(1|=| + + + 26 - 11) w0y(2/14) + 8 + 558p² Σ
+ (exp (26([[=] + p) 8²) 1) wx(2)p)) p² + 18 Σ (exp (26([|=] + 82) 1) w,(z/ ppa
<2En~V(0.1a)
(exp (45[||z]1 [|]l|=+|++p))) +p) 28 1) + 88 + 26 exp (25[[1 +p) + 82) 1) p²
<2EE <2E,,~ 4 (exp (40[1I=] p) + 28²) - 1) + 86
<168 + 8p -E,-A(0.L [886p - 482 + 1) exp (401==]1) - 1]
<165 8p
(85p + 482 + +1) (1 + 205 d) -1)
<1528vd.
Here the second last inequality is from Lemma B.2
17

---

Published as a conference paper at ICLR 2026
By the above lemmas, can prove Theorem 2.1 We just need prove the following lemma:
Lemma B.4. Assume that [i 1 & 3n for s Sisi n. If the step size satisfies S 0 (m)zz)
then after iterations of gradient descent, the loss satisfies
n d2
(e(s((T) S
TIT
Proof. Direct calculation from Lemma B.1 shows that
Σ u V WucC
((, v(x) + (mr(x)es(x)) (4. 21)²= Σ us(xx) (v(x) [2)² + 3 (v(x) v(z)) 2
(2)
Notice that when || < 3.
(v(x) (v(x)Tpi)² ")))>sloois <|10(2)11 11mi112 < $ [1((2)))
Thus, we have
Ene V C2 Em
0(x) T v(x) 4 Σ wx(x)) (w(z) (1.)
o(x) T v(x) - 4 Σ wi(x)) ||0())11
Be~N((,Ta))
N(0.1))
[o(x) T o(x)]
Let p max By Lemma 12 in Xu at. (2024). we have
exp(-8p"
v(x)] 2 Ω
n2d(1 (nd(1++p) (p11++pd))))) /d)2
Therefore, for some constant 0 <e<l, 1.
V
n?2(( PV d p
We assume that
n S 1200n
Now we use induction on T to prove that:
1. For any i, during training, E, 11 3.
2. For any T, we have (())_
& C/F9
The result is obvious when 7 = 0. Assume it is true for 7. Then in this step, S
1 3.
We define S as:
exp(-8p²)
5 5= u V L2c.
n2d(1+ ( pV
18

---

Published as a conference paper at ICLR 2026
Define U(T) = Ex 1l1 111 Let p4 = 14 below. We have
(5)
U 1) -U(T)= Σ 11 7/ -
-2n Σ u4 V nz C+ n Σ 11
-2ns + "¹ Σ | Vi, C12
< -218 + 1 Σ ((D - 8p B (w:s)) 2
S 221S + 4np²p²
Therefore, if U(T) < 1/5, then p < 1/5. As 17 < 1/100, we have
U(T 1) - (T) + 4n/²²² s 1/3. 1/5, have p 1/(m). Then, we have
Otherwise, (f)
3n²d(1 + pV 12 p 2 20p²
Thus
U(r + 1) & (()) 2 + (T)).
Combining the two cases, we have proved the first part of the induction.
For the second part, use L((x) denote the loss function with input μa. We have
((++1) c(r) =L(u(T) - -VVC(((T)) c(u(T))
(VL(((T)- -VCCIN)))) ds
(VCUM()) ds
+
(VC(H(T)) - -IVV(((T))) ds
- [- + of (CCC))))-VC(t-SSNNNNCCE V((u(T)) ds.
Notice that for any i, (p((T)) is the loss function with input (T)(T,, which is at most 2p + 8p" - 3p by the equation of Lemma B.I Thus, for any
1
sn| (C; Sp, 12d'
Therefore, combining Lemma
B.3
we have
(((N))))-V((())snnNYYV V((M(T))
(V.L(H(T)) - (.V(((())-VV.(((()>>--))))))())) -w V( v.L(U(T)))
(V)C(((T)) - V.C(H(T) STV -IV (VLC(A(T))I
Sa2ddsn| - |[VC(M(T)]I
19

---

Published as a conference paper at ICLR 2026
Thus, have
c(++1)- -(()
- --
1
Notice that
[VC(M(T)\|
2
S
2 c
dp pd
12n (ctr))
3/2
np
12n
Therefore, we have
( 1011-2'' ((()) - 5ー [[(u()))]
1
300m de (c(T)))
c²
This implies that
(c(r1)² - -C(rrt)²²
1 (C(++1)²
(C(+))
(C(T)) (C(r+1))²
2 (c(r+l)) (C(r) - ((++))
(L(r)) (c(r))²((((+t))) (c(r+1))² à²
30000 (C(r))) s)
(C(+)) 3
20a
30 /1
Thus, by induction hypothesis, we have
2
(LI++1)) 2
1 (T + 1).
This gives the desired result for loss convergence. As is bounded by 1/3, by
exp(-8p²)
L = C v(x) 2 Ω
n2d(1 (10[1++00 pva)+
y converges to 0. Therefore all μi converges to O.
Proof of Theorem 2.7 Notice that we have assumed that μ²" 0. Ast > log log M 2, for any
i, we have [| (0) exp(-t) < 3m Therefore Theorem 2.1 follows by Lemma B.4
C A PRELIMINARY LEMMA FOR APPENDIX D.3
In this section, prove the following lemma that is used in Appendix D.3 We also assume that μ² 0 in this section.
Lemma C.I. Let Mo 2 10 /0 Let the initialization be || S 1/M and minz2 || Mo- Then, as long as the step size < we have that fn converges O, and || converges to co for i = 2, 2.... n. Moreover, there exists T = O(M3/n such that when T > T, the loss Lo(so) converges with rate
Lo(so(T)) T[[ T))
Also, let > 0 be constant. Then for any e (it can be relavent to n, d, My. but not relavent to T), when 7 is large enough, we have
Co(su(T)) 7ltc*
20

---

Published as a conference paper at ICLR 2026
We consider the gradient one iteration. Let M
min | [
We use mathematical
induction to prove the following result:
1. .M doesn't decrease in the training process.
2. During training, we always have || || < 1/(
Now under these two assumptions, we analyze the gradient in one iteration.
First we need the following lemma that gives upper bound for B[w=((=)] (and also for coordinates for E[u, (x)) with j > 1 by symmetry):
Lemma C.2. Let D > 1 be a positive real number. Assume that || < DE. then
l| := s D.
3 8
g²
2 exp
8²
2s
(x [w2(x)] s exp
+
1)3
Proof. Without loss of generality, we assume that μl (s, 0, 0) by rotating all the μi
For any x E Rd let xj be the first coordinate of æ. We consider two cases:
Case 1: 01 S 2 ||-||| < s.
We have
1 2 I [12113))
exp
w2(x))
Σ, exp
(-2 111 - Mi 12)
exp (- | || 12)l3
exp (- - 21l12) + exp (- -111 - 12)
+ exp (11 (1 ) x + 42
| (12))
1
1 1+exp
(-821
|2:||
82
D
D3
1 1 +xcxp(
(-871
2s
+ (2)
139
28
< exp
83 +
=8 8
D3
2
Therefore, the total contribution of this part is
2s
En-,(0,11) [w)(x)] < Bn--(((11)
1 11
SD/2 exp
SE1
+
D3
2
1//
2s
1
exp
80
+
82
exp
dx
D)3
2
2m
2
28
1//2
(I)
s)²
exp
exp
dr
D3
V
2
2s
exp
Pr
s/2]
D3
(0,1
28
82
< exp
exp
D3
8
2
28
exp
D3
The second last inequality is from Lemma
A.2
21

---

Published as a conference paper at ICLR 2026
Case 2: Now consider the case when 21 > s/2 ||||| >> probability of this part is at most
By
Lemmas
A.2
and B.2
the total
2 exp (-(s- v2)2/2) + +- exp
3
s²
exp
8
As ||2||| 1 1, the total contribution of this part is at most
exp
2
Corollary C.3. Let D 10 be a positive real number. Assume that |
DJ.
|122|
D, then
D2
1x-N(o.1a) [w2(x))
exp
D
Proof.
We just need to notice that 2 D and
s² 2s
1
2D
D
ln
8 D
8 D
8
Corollary C.4. Let D > 10 be a positive real number. Assume that || < DT. || = 2 D, then
5
(
82
E (w2(x)] S
cxp
7
Proof.
This is simply due to the fact that
s²
52
52
s2
8
D3
8
56
7
For each æ E Rdd let ø denote the vector obtained by removing the first coordinate of x (so I E Rd-1). Also, let 21 be the first coordinate of æ. Recall that v(x) is defined v(x) E. (;(x), We define IRd + R the first coordinate of v(x), and U2 R Rd as the rest coordinates of v(z). For each 1 5 i n, we define a; as the first coordinate of μus and 6, as the rest coordinates of prove
14 (thus a, RR,,, bg Rd-1). In the following,
inequalities based these notations.
Lemma C.5. Let M > 10. Assume that || < 1/M and || M for any i 2 2. P2
(D,0... .......... .0), where M < D S M + Ma- Then, for each > 2, we have
w (z)
D2
22 x ID aa) + 1 1リ |
cxp
W2
2
Moreover, if | 2 we have
(x(x)
D
<
21
| M2 at
(2)
M
Proof. Notice that by (0..... ,0). By definition, we have
wx(
D
2 1| - x TT (M2 11,))
exp
Wy (x:)
D
a2/2 (D a:) |b:112/2 + .. TTby
exp
D
/2/2 x (D a:) |b:111 2 /2 ||ا|| [1ba1)
S exp
22

---

Published as a conference paper at ICLR 2026
Therefore, always have
wi (x)2) (2
D
S cxp
d2 x (D a;) 1|6,10/ /2 | [1b.1]
2
D
/2 - 21 a;) |ز||
S exp
Moreover, if at +| Therefore,
M², by the fact that +1b:lb=1 2 M we have |bb:1| 2 v M - at 2 |
wx(xx) (2(
D
I exp
/2 21 a:) ||b: |b::12/2 + ||ح|| [[bal]
D
11 - a.) + | V M - -
I exp
M3
Lemma C.6. Let M 1000. Assume that || S 1/M and || > M for any i2 p
(D, O,..... 0). where M < D < M + MS. Then, for each i > 2, if |b, > 6||=||, 0.4M f x1 S
0.6.M, we have
wi(x) wz(x)
S exp ( ||b:12
30
Proof. For any > 2, we have
(x)x)
(I1 a;)2
+ (01 - D)2
|
| - bull3
exp
W2
2
2
D
s exp
/ - 2D a;) 1|6,11 /2 ||ج|| [10.11
D
/2 (D a.) ||: 12/33)
S exp
When |6:1| < 2VM, we have
||b:11
lail 2 v
M
|b M
> T1-
2M
Therefore, have
w. (I)
D2
M
|b,l12
||b.
< exp
I1
(D- M2
16:113
W0 (x)
2
3
|b:11
|bb.112
s exp
1+
21
6
2M
||b.112
S exp
30
When
|6:1| > 2/M,, we have
(x(x)) w2(x) < cxp
2 x²
||b:||2
M
||b:l12
|bb:11
D
I exp
1+
exp
1
2
3
30
L
Lemma C.7. Let y ~ N(0, la 1d-1).
Then for any vector b IRd-1, with probubility at least 1
we have
1
16,11
D.
f
+
< 4 log n + 2.
4
23

---

Published as a conference paper at ICLR 2026
Proof. We assume that b (s, 0, ... .0). where ||6|| by rotating the coordinates. Let the first coordinate of x be y. Then the left hand side s(y 8/2) + T
2²
Notice that by Lemma A.2
with probability at least 2n we have y S 22 In this case,
1 b:
|
1
s(y - s/2) + + g²
b._
+ 2
+
2
4
8²
+ +2s +2svlogn +2
4
< 4 log + 2.
So the result follows.
Lemma C.8. Let M > 1000n Vd. Assume that |MI|| S 1/M3 and || 2 M for any i 2 2.
H2 D....... 0). where M S D I M +
1 Ma. Then for any M S In S M/2 + 1/M, with
probability at least over I ~ N(0, Ta-1), we have w2(x) 2 1/(10n²) and
M
(1- 100m 1 M
S en (x))
10
Proof. In the proof of this lemma, we always assume that We assume that for any 2, we have
M
S σ1 S M/2 + 1/M, | < VM.
2
|6:11 4 log n +
by +
4
By Lemma C.7 when #1 is fixed, then with probability at least n-2 we have the above inequality holds for all 2. Also, the probability that ||=||| 2 VM is at most exp(-(VM - v2) S 2n Therefore, with probability at least - 1. we have the assuptions above hold. We just need to prove the desired result under the above assumptions.
2n
For any i >>>>>>>> 2, we have
W/ (z)
(I1 -a)2
[ -
D)2
1
exp
+
W2 (x)
2
2
2
2
D
1
))
exp
/2
21(D a:) bJ
b:
2
2
When |b S 6/M, we have
|ail
2 V M - b3 > Z1
Therefore,
D2
D
M |
0, - 2D - -a,) 5 2
D /M² [(b.12))
D1
1b,112
|
11
21
2M
|b:l12
< 2 +
4
By eq. 33. we have
wx(x)
< exp (4 + 2) < 10n².
W2y
Otherwise, when [|b,]| > 6,M, we have |bb|l| 6 ||||| By Lemma C.6 we have (x(x)/u2(x) S 3.
Therefore, for any 2, we have
wi wx(a))
< 10m 4
(2(x)
24

---

Published as a conference paper at ICLR 2026
Also, we have
W1
(
|1
||
+ x (MI x
122))
exp
022
(x)
1
D
M
1
1
exp
D
||5l||
2M3
2
2
M
M3
1
D
exp
2
1/5.
2M3
M
M
On the other hand, we have
W1 (z)
11
||2|
exp
+ x
(M1 (22
W2 (x)
2
2
D2
M
S exp
D
|
2
M
D
1
S exp
22
2M3
M
implying wi (z) 1 2/3, w2(x) 2 1/(10n²).
By the fact that w/ (z) + Ei>2 (x) 1, have (x) 2 1/(50n²).
Also, we need to upper bound all (i > 2) such that < M/2. By Lemma C.5
(x)
D2
-2/22-211 x xI a.) + + s exp
D
of
Ma.
|
exp
W2
2
2M3
2
In this case, when a; S M/2 we have
w. (x)
D
M
Ma.
|
< exp
(2
2M3
2
2
Ma;
M
M
< exp
< exp
|a:| +
5
2
When M2/2 a; S M/2, we have of
+ ||
D By Lemma C.5
we have
w.
D
< exp
x (D
a:) + |ل||| M a?
W222
M3
D
M
S
exp
4 + |
S expl 1-2/3/3
M
Therefore, the absolute value of the total contribution of all wx(raa such that as < 2/2 is at most n exp( (0M2(222//) 22)
When a; 2 M + 1, by Lemma C.5
we have
(x)x))
D
d./2
21(D - a:) + -111111)
exp
W2(2)
2
M²
(M au) + 1 13]
exp
14
2
2
2
(a; 2x, )2/2 1 I1a; + +20 - +z 1313
1
exp
1+
2
M
M
M
M
exp
05 +
A exp
(-M2//) /ac.
Therefore, the total contribution of all (x(r)) such that a; M + 1 is most nexp (-M2/6)
25

---

Published as a conference paper at ICLR 2026
Now can bound (z). On the one hand, by w1 S 2/3, we have
(I)
Σ Wi( (x)aaa
M (1 wi (x) w2(x)) nexp (-M2/5)
wi (x)
+ (2(x) +
M3
2
M
M
nex -M22/5)
2
3
M3
M
10
On the other hand, we have
U1 (1() = Σ ws(1)a
S w (x)
+ (1 - wi (x))
+ (1)
+ +=xxp(-MM²))
M3
M
M
n (ex((-M3)
50m
M.
100n
These two inequalities give the desired result.
Lemma C.9. Assume that || we have [[][]]] & 4||-|||
and || 2 M for any i 2. Then for any ||=||| > 1000n
Proof. First notice that for any such that |p:|| > 3|| we have
w/ (.(z))
||4 xz|2
|| z01
exp
W1 (x)
2
221/z111//9 (1 + ||x:1)2
S exp
exxp
11/711//7
S exp
||=1|| P) (nallall))
Therefore, we have
||b()]]| < 1 + exp(- ||=]] + 3| < 4||=]|]
Now we have prepared to prove the following lemma, which establishes that for each μa with i 2 the gradient update is directed toward increasing its value.
Lemma C.10. Let M min;22 l| such that M > 10a Vd 10, Assume that || < 1/M/ and || 2 M for any 2. For any i if |al < M then μ Vn,C s
2 10 exp( M2/8).
Proof. Without loss of generality, we assume that i 2 and μz2 (D,0, ,0) by rotating all the py's s. Then < D < M + MT.
Recall that
p. L BanN(o,1a) [w;(z))(x)) + (x(x))(())) (M4 v(x)) (2 pi)
26

---

Published as a conference paper at ICLR 2026
Let
S (1n
S = (2) (D v1 (01 D)]
S3 [E.. (b2 - v2(x)) (21 D)}.
Thus the first coordinate of each of them.
S3. Now we just need to give an upper bound for
Upper bound S₁: Notice that S E; Bon. First, by corollary C.3 have
7
(-
M
M
1e (w2)))) (x)]a1 S (eo-:roo.)))(2((()))) &
M exp
an S
eep
M
8
For any i 2 2, we have || 2 M. If ||s|| < 2M, then we have
7 M
M (-
M
(ww.)o,.))(((()))... I (w== S
exp
a, 14 14exp
(88)
8
If [|/]| 2M, then by corollary C.4 and a; S |plll we have
5
||4:11
M
(ew-)ro...)ll2(())0(((()))) < (e
||mil| exp
as S 5exp
(-8 8
7
Therefore,
A22
S S 14nexp
(-8
Upper bound Sz: Recall that
S2 = Bo..
vz (2) - D)}.
We first consider the total contribution of the negative w2( un (x) (D (x)) (x1 D). Lemma C.8 we consider the case where M
By
2 < xı < M/2+1/M, M/10- I v1(x) S (1 100n
1 M
and u2 (x) 1/(10n²). In this case, 0 S 0n (æ), S D. Thus we have
1
1
M
M
1
M
(z) (D
<
1
M
10n
100m5
100n
2
M
3000n
Also, for each 2ı such that M/2 s x1 < M/2 + 1/M, the density of xı is at least
(M/2 + 1/M) 1/M)²
1
exp (- M
exp
2
8
8
Therefore, the total contribution of the negative w2 (2))))) (x) (D (I:) (x1 D) is most
asa
.²3
24000m To exp
Then we consider the total contribution of the positive (z(x)e (D - un (x)) (x) DD). When
|||||| 2M/33 for any i such that |||4||| > 3 | we have
we (x)
||m - x|| || x1/
exp
W1 (x)
2
||ma|1
||
S exp
+ ||
(lluall + (1
2
2
||mx112
exp
||mell
exp
(-11u:11 112//7).
27

---

Published as a conference paper at ICLR 2026
Thus, as M > 1000v n the sum of these (x)]||| is at most n||z||| (-1|=|1//7) N |||||
Thus [|=( & 4||x/||- The total contribution of the positive wz (2)(z)) (x) (1- (z)) (m) - D) under
this case is at most
(2M/3 3 + +h vaj²) +
Eo 11/2222/ 50
x0|
5
exp
50(2M/3 + 1)
2m
k=0
(2M/3)²)
- 2 V2TT
exp
50(2M/3+11 1)s
2
< exp( (up(-M22/6))
When ||>||| < 23/3, we have 11 < 2M/3 D. To ensure that (2()))1(z) (D (x)) (2) D)
positive, we need vi (x) < 0r un (x) D.
For any > such that || 2M, we have
(I)
||u 11
|1
- x1
exp
11 (z)
2
2
(1lill 2M/3322
M2
S exp
S exp
/llxell.
2
3
Therefore, we have [x1(2)] S (v(x)) $ 3M.
Notice that
W2 (x)
|122 x|
|14 - x1133
(-2 - Ma +1).
M²
exp
S exp
101 (2)
2
When 2 - M/3, the total contribution of the positive wy
(2(2))o (I) (D v2(x)) (x1 - D) is at most
M/3
M
Mxj + 1) 301
exp
exp
2M/3
2IT
2
M/3
1
(M
<60M
exp
dz
2T
2
-60M
Pr
> 2M/3]
(2M
2M2
<90M² exp
= -00M exp
(-9
2
9
Now focus on the case where M/3 < 21 most (notice that 11 and ||ā||| are independent)
2M/3. When
|
VM, the contribution is at
exp (-(VM- (VV2/2
M/2
1
o (-2
M
+ Ma +1)
30M -d
exp
exp
M/3
V2T
2
2
M/2
-0M3 dr
exp
2
M/3
(exp(-M
90M3.
Pr
(y > M/2)
M2
M
$180M exp (-8
8
3
Otherwise, we have have wxx
||||||
/ M. For any i> > 1 such that a, 0, by Lemma
C.5 if < -M/2,
D
< exp
22/2
x (D- a.) |=
w2(
2
D
of
M
3M
<exp
|
2
3
2
2
exp
(-M2 /3) /aa.).
28

---

Published as a conference paper at ICLR 2026
If a, -M/2, we have af + ||]1| < D By Lemma C.5
have
w/ (a)
D
21 az) + | M aa?
exp(1- MM01-MM/33MMMM S exp ( 1M2//4 /M.
W2 2
M2
Therefore, we have (z(2), 2 2
3 by summing up all negative ag 's Therefore, the contribution of the positive (2(x)(1(2)) (D 21 (z)) (21 D) the case that (1) < 0 most
1 (22 M3 2M s Ez- [w2(x)] exp(-M2/88)
The last inequality is due to corollary and M 1000.
When (1())= > D, consider any
> such that > M + 2. By Lemma C.5
we have
(;(x))
D2
/ 2 x (D a.) + |
exp
W2 (
2.M
(a: D) 11
I exp
(M DD)
M
$ exp
/|a.].
Therefore, by summing up all contributions of a; > M + 2. we have (x) S M + Therefore,
contribution of the positive w2 (z))x (2) (D - 2i (x)) (œ1 D) in the case that (z) D is at most
10M²] s 10M [x>] S 140M exp( (-2 /s).
We sum up all the cases. The positive contribution is at most
M
M
1mp(-M//00 + 90M exp( 2002222//// /9) + 180M exp
8
3
+ exp(-M² + 140M exp(-M2
M
18000n16
exp
8
Therefore, we have
2
.2²
So 5
48000n 10
exp
8
Upper bound S3: Recall that
S3 ===Bao -ro 22(2)( D)).
First, similar as discussed above, when || > 0.6M,
|| > 3|1=// we have
w
||m - x||
| x=|
exp
(x)
2
||m211
||
exp
+ ||z||| (llaall + (1
2
2
||mell
exp
||m|l|
exp
(-11p=113
Therefore, we have [|=())] < 4||x]|| The total contribution this part is at most
(0.6M + h vdj²
Ez [1
50
exp
50(0.6M +
2
2
k==0
(0.6M)²
-50(0.6M + 1)3
exp
2m
2
exp(
0.15.M2).
29

---

Published as a conference paper at ICLR 2026
Now consider the case where ||>||| < 0.6M.. Thus 21 < 0.6... For any |
2M, we have
wy (x)
- 0.6M)²
+ (0.6M)²
(-N²*
M2
W2 (x) I exp
S exp
(-3
/
/llpsll-
2
2
3
Therefore, have [22(z)] S.M. As ||a|||
< 0.6M, we have
W2 (2(x) = exp
Il
||
x
M
Mx1 +1
I exp
W1 (z)
2
2
Therefore, the total contribution of 21 S 0.4M is at most
0.4M
1
exp
11
exp
M
+
Mr
+1
300
V2TIT
2
0.4M
1
(M
x²1²
<60M
exp
dr
V2T
2
<60M3
Pr
0.6.M]
1-N10,1)
(0.6.M)²
<100M exp
100M exp (-0.18M3).
2
Now we consider the case where 0.4M < σ1 < 0.6.M. When ||||| > VM, the contribution is at most (notice that 1 and ||ब|| are independent) By Lemma C.6 for any i > 2, if ||::l| 2 6||ž]|, we have
(x) W2 (x)
S exp (1 |bx.1(2)
30
Therefore, if ||ا|| > VM, we have ||2((x)]| S 7|| Then for fixed 2 the expected Vy (x)) 12 (2))
most
M + +E, By-N((0,
[1 Mul>>< 711811]
(k+ VM - va(²)
<M +
exp
7(k + VM + 1)3
2T
2
k=0
(VM M vdj²
SM + 2
exp
7(VM M + +1²
<2M
Therefore, the total contribution of this part is at most
[M² M
E
1o.4a<Zz <0.6M min
cxp
(-2
+
-MM++1)
( 02(2))<1 - D)
M²
Mx +1 H
<4M
min 1, exp
2
<4M²
M/2
1
exp
exp
M
+ Mx + +1) dø +
|M12 V2T
0.6M
exp
2 (_27
dæ
22T
2
2
2
00
$16M²2 Pr
[y
> 0.5M]
(o.1)
(0.5M)²
<32M exp
32.M exp (-M2/8).
2
Therefore, by combing all the cases, we have
S1 S exp(p exp(-0.15.M/))
100M2. exp (-0.18M²) + 32M exp (-M2/8) 50M
550Mexp exp (-M2/8)
30

---

Published as a conference paper at ICLR 2026
Now combine all the three parts.
(- (-8 8
M
M
M
S + S2 + Sz E 14n exp
exp
48000n 10 exp
+ 500 exp (-
M
<
10 exp(-M2//))
which means
M3
H2 V
m2
cxp(-10
M /
10 10
We also need the following lemma to bound the norm of gradient.
Lemma C.11. Let M = minz22 || such that M > 10 vd noo Assume that | |
<
1/M
and
|| 2 M for any i 2 2. Then for any i 1, we have V All S 700n exp
M²²
g
Proof. Without loss of generality, we assume that = 2 and μt = (D,0, ... 00 by rotating all the N4 s. Here D > M. Recall that by Lemma B.T we have
C C = 2 w2(x)u(z) + w2(x) Σ (oy(x))(kk) p2 2w2(())(z))((z)
(w2(xx) Σ (x) (u(x) ps) My + 3w2(x) (v(x) v(x)) ((()))((x)) v(x)
=2 (2(2))(()) + w2(x) Σ ae (ey(r))u p22 - 2w2(x) Σ
j.k
2wz(z) Σ
()(x)ux(z) (MT (y) us + 3w2(2) Σ
(x())xx())) (M] pa) pi
J,k
j.k.
we have [x 3/ exp (-a²² Also, for any
By corollary C.3 for any
2 such that ||m > 2M, by corollary C.4 have (1x S Twell
exp
Therefore, for any i 2 2, we have
(o < 7M exp (-
M2
(-8
Therefore, we consider the right hand side of eq. 4p. For each (M, uz)pu item, we use the maximum ||m||| among Mi, Pas Pa to upper bound the norm. Therefore, we have
V u2 L < 18E [=2(x)]||ma]]
60 Σ Ex(
<>1
< 700nM2 exp
8
Now we consider the gradient update for μ4-
Lemma C.12. Let M min || such that M > 100 Vd 20 Assume that ||1|| < 1/M3 and ||ll|| 2 M for any i 2. Then for any nj 1, we have
M
|| - v/ A11 S (1 -))l + + 2500M exp
8
31

---

Published as a conference paper at ICLR 2026
Proof. First, consider case where ||>||| we have
<
M 2
VM. In this case, for any i > 1, || 2 M.
Wi (x)
[1;
x1
|1 - x²|
I exp
W1 x
2
2
M 2
+
V
M
M
M
M
S exp
2
2
S exp(-MM MMMMM MM.
Therefore, this case, we have wi (x) 2 0.99 as sum of all other (i('''s is at most 0.01. As M is at least 0.99, we have
probability of || & M
S = E. (1. 2 0.5.
Therefore, by the form of V ny L as stated in Lemma
B.1 we have
|| 2Sp =
28 w1 (x)
wi (x) + w/ (x)
Σ wy (1)))1y0)) i1 (x1(2))(2))(()) jn
1>1
2wn (x) Σ Wy (v(x) (3) Mi 3wn (I)) (u(x:) ( v(x)) o(0z)
Ex (w: (z)) MGll + 2 [x)(2)² 4w1 3w1 (z) |
+ 36
Ex [w,
(5)
The last equation is by writing
Ex wi (x)
Σ wy w i - (0)())()))((z) mn
2wn Σ (y(xx) (u(a:) (4)) My + (x)(x) (w(x) (x(x)To(x)) v(x)
Ex
wi (x)
Σ wy(t))ugTMn pn - 2wo (x)
Σ
j.k
Σ (c) (#I ms) uu 1 J.K.l
- 2w (x)
Ew wwy(x)ulo (mTPs) Ps + 3wi (2)
j.k
Then for each item, find the maximum #4 in the item and use || upper bound I1[u pae Now we bound eq.
(). By corollary C33 for each i, we have
7
2
((, (x)) S M
exp
Also, by corollary C.4 for each such that ||m := > 2M, we have
5
22 7
E [wx(x)]
exp
s
Therefore,
.2²
M2
Ex (x)]la1111 Σ
Eg [wx [x))))]
38-M
M exp
8
<
2400M nexp
(-18)
8
32

---

Published as a conference paper at ICLR 2026
As for the remaining term, notice that [2)(2)² - Aw0 (2x) + 3w, (x) s 11-110 w1(2)) - 3wi(x)1 s 2(1 - 11 (z))
Therefore,
2 EB [w) (x) 2 4w0 (x)
+ 3wi (z) ||
<4E, [1 wi [1]]]] "lmll" 1|
=411/1 ||
Σ
Ey (x))]
{>1
4
7
M
M3
M
exp
8
M
SM² exp
Combining these two terms, we have
exp (-Ar²)
.22
C - - s 2500m exp
8
Therefore, have
(-M²²
_22
112 1 41 <1 2 + 2500n exp
8
1/2
$ (1 |m|l + 2500m exp
8
Lemma C.13. Lef 1) and M > 10 10%vd. n0
Assume that || 1/M3, || 2 M
(- M²2
for any i 2 2 and || || > 5000nM2 exp
Then for any Mo such that M s Mo S
M + 700mM exp(-M* - M /8), we have
|| - V ns Cl
||
S (1-1/44
Mό expl - M2/8)
M exp (28)/8)
Proof. Lete Mo M < J. By Lemma C.12 we have
(1 (1-m) 11-1)lma111 + 2500m exp (- Mr2
|| V/ ClI
<
|M ||
||
< (1 11)
17
2
1 17
2
Combining with
M3 (xp(-M331
M2
cM
2
exp
+
2 exp (-M2/8)
M
4
8
E
2
cM
1 ++2
M
2
4
eM
7
2 1
5
10"
the result follows as (1 (1-11) 101nn/11)( n/44 2 (1 - 2))
The following lemma gives bound for the loss.
33

---

Published as a conference paper at ICLR 2026
Lemma C.14. Let M min | such that M 10ºvd no 7500n exp(- M2/8/ /8). Then for L 1(,,() [1 we have
Assume that || 1
M (exp(-M218) < < 14n ex/
5000
Proof. First, notice that
E Ep-N(0,11)
Σ Bew.voozsluu(e))yy()) (44 ps)
w
|
12 +
E (w)
1>1
10 M exp( M2 + 2n 7M exp(-M2/8)
15nM exp( M² /8).
The last inequality is by corollary
C.3
As for the lower bound, assume that || (M, 0.... 0) by rotating all the p4's. Then by Lemma C.8 for M/2 < 21 < M/2+ + 1/M, with probability at least 0.5, we have [[v())] || 2 M/10.
Therefore, as the probability of such æı is at least
(M/2 + 1/M)2
M
0.5
exp
exp
2T
2
M
50M
8
we have C
.M exp( (2((M2//8)
5000
We use the following lemma to characterize the updates of μr for all i 2.
Lemma C.15. Let M min 2 || such that M > 10 Vd n0 Assume that |p| < ||m||| Z M for any i 2 2. Then for any TJ < 1, we have
1/M3 and
M
M + 72 (xtpn///xxppMMMM111p
max ll 122
M n. vI
exp(-M220)
2
10
10
Proof. For any 2 2, if ||:1|| < M + Mr then by Lemma C.10 we have
M
|| 1
|| + 2nj4 Vp,h L 2 M +
2n
10 exp(- exp(-M2/88
Thus
M
|14 1/V mz Cl| M 2. 10 exp( exp(-
Otherwise, if ||p
M X75 by Lemma C.11
have
M
||4 7/ M M3 T 7 exp( (2 M + 2 10
(xp(-M2/8))
Therefore, the second inequality follows. The first inequality a simple corollary of Lemma C.11
Now ready to prove Lemma C.1
Proof of Lemma C.7 We always use M to denote min;22 [1,
In the first iteration, we have |// < 1/M3 and || (0) || 2 M for any 2 2. The following
induction can prove that for any always have [|] 1/(M(1) and || 2 M)( for
34

---

Published as a conference paper at ICLR 2026
any > 2. We just need notice that if it holds for 7, then we have M(++1) 2 M(() because of
Lemma C.15 Then if || [ || > 5000nMM2 exp(-MM2/p))))) we have (we use M to denote M(r))
|| [|u (++1)
||m|" V - CI1
A1
(M exp( 10)
$ (1 / 4) (M) (x)r)))exp(((MMM0
1- 7/4
(Mir)
(Mi++1)y3
The second last inequality is because M(++1) M and they are both large enough. The last inequality is because M(++1) < Mlr) +n- 700m. (2xp(m2//) Therefore, the induction holds for T + 1.
By Lemma C. | || can only increase by at most 17 2500n ex2 in each iteration. If is larger than 5000nMM² 2:000mmMeppppMMMM exp(-M²/S)) then it decreases Therefore, once it becomes at most 7500n exp(-M 22) it will never exceed this value again. By Lemma C.13 the gradient descent uses T = (M///M/ /n) iterations to make | I 7500m xxp(-M²/8) Then it always holds.
By Lemma C.I5
have
M(++1) M(r) + nn.
exp(-
Therefore,
M(++1) exp (- (A^(+))) 2 /8)
< <Ma()) +n (7) (es(-((M(7)) ( (1 (1-1) (o0(0(("))ee(((((((())) //8))
<M(') exp(- exp(=(M(()))) //8) (1 (1- (0x(MM)))) expp))
<M exp(-(M(()) /8) - 14mn (M expl exp(
Let G, 14 (2 Therefore for any 2 T, we have G+41 S G. -G, S
So
G न + G-Gr G
1 +1
We can use induction to prove that for any t > T, G. 1 --7,50 5o
(4 expl exp( v '1(T= - T)
Therefore, by Lemma C.14 when >> T, we have
((T) S n(T - T)
1
Also, we have
M(++1) exp (-(artou1)*) 18)
2 (M +n. 77000(()))))) exp( ( (((6))2/88)) (8)) (xp(-((4(1))//)) (1- (alpa//())xxp(((AM(1)
2M - (MM(-(()()))//8))) (1 (1- 700n(a)())) (1pp0//)))eep((111110 ((2(()))2/))
-M(r) exp(- (MM((()))))/))) 700m)(Mr(7))² (M exp( (x/(2>((p(()))))))/)))))))
35

---

Published as a conference paper at ICLR 2026
Therefore, for any 0 0.5 and constant K (K can be relavent to n, d, Mo. but not r), when M(r) is large enough, we have
MM(++)) (- (M(r+1)) 1-c
(M (xx)eex(((((())]]) (1 800m)( (wwpa//)))(/////ppp/pp (xp-
(M (MM))))3)( +
2K
Therefore, when is large enough, we have
K1/(1-c)
(
5000
5 1/(1-c))
Therefore, the theorem follows.
D PROOFS FOR SECTION 3
Still, for each i, we can find that
27
C((s:) = BENNCut.
Σ
wi (2) (x) 14/
expl - 1| /i,eel//22
= -ReNNiiiaa
ja exp(-x) - 11" exp(pp
j=1 exp(- || M4j.1 12/2)
n1
expl exp(-11x- (ui. - -11)11//22
(is - 10 exp((()
2j=1 j=1 exp( 1op0-1aa 1 (1y, 14 12)112/2)
Therefore, the training loss is equivalent to the case when μ²" 0, and the initialization is j A4s 11"
As only consider the case where t O, descent is on the loss L == (s())). Based on the above analysis. we just need to consider the case where 1r" 0 and fu; is initialized "close to" Mpe (Ma, O,... .0.. (actually it is initialized as -11", but by applying an orthonormal transformation, we just need to consider this case).
μr to denote μ4,0 for simplicity. The gradient
D.1 PROOF OF THROREM
Let e 01, 00 E Rdd We consider the case where 141 S61, 122 -s61, 14 (0,.--,0) for 2 3, where is positive real number.
Recall that by Lemma B.I
we have
V n4 L = 2E, (x(x))(() + wi(x) Σ Wy (t)())yuJ is (2x(2))(())((1))
- 2wx (x) Σ (z(x) (u(x) (s) uj + (w; (v(x) Te()) o(0)
which is a linear combination of all μı we know that V L can only have nonzero component in the first coordinate.
36

---

Published as a conference paper at ICLR 2026
Let
= + wx(x)) Σ wy (y(x))up) pi - (2x(2))()))(()) p
(w); Σ ()( (v(x)
(o(x)))))))))))))))))))))))))) m4) + (w)(2) (v(x) T v(z)) o(x),
have Then for i >> 2, we always have S,(x) + Si(-x) = 0 by considering the coefficients of μa and μ2- Thus Vn 0.
Also. have S1 (x) (2(-x) = because the coefficient of μr in S1 is the same as the coefficient of μ2 in S2(-2) Thus Va C W
By Corollary C.3 when & > 10vd, we have
7 s2 Ez (wi(x)) = Ee-N((,.)) [00[(x)] - exp
8 8
Therefore, for such s,
L $ E.glwi( + w2(x))}² & 14s exp((pp
lim == +
As C for = is O, there exists >> 0 to maximize L. We choose such an s. Therefore the first
coordinate of Vn, C VugCi C is O. Therefore, Vn, C V2zCC C and they only have the first coordinate, Theorem 3.1 follows.
D.2 PROOF OF THROREM
Without loss of generality, assume that μ² O, and the initialization μ (0) satisfies 1|
Mpell S 10and nd( for i 1,.... We denote A Mo- For each 2 0, let
A = (1 A/)) Let (1,0, 0) E Rd. We define K 10000nd(((7)) We
define B(r) 10 exp 10" We use induction to prove that for each r. |1 A s
((r) maintained for 1,....
We first consider one iteration. We use μn to denote for simplicity. First we prove the following
two lemmas:
Lemma D.1. Let A > on K
10000ndA2, B dep' For any 1 S i S
||m Ac || < B.
Then for any E IRd such that ||^||| < K. we have that
4KB .igign n. <
(x((z)
Proof. For such x, we have
exp( x Mall 222)
exp(1
x Ae1 12/22
=exp
(-x (m Aei) 11|u11//2 11e.113/2)
= =exp
(-x (44 - -ee) ([|ææl - () + ||Ae; 1I)/2)
Sexp( K B + B(2A + B)/2).
Similarly, we have
exp( || 22112/22
> -B--BBB-24 B)/2).
exp exp(-1ee Aei 2/22)
37

---

Published as a conference paper at ICLR 2026
Therefore,
exp(- (-1|x - (2.l2/2)
(x) = 2==1 exp(- 1| m2;11//2
2
B(2A + B)),
and
exp( 1 || 1411//2)
(x(x)) Ej=1 exp(-ll My //112/2)
exp(2K B(2A B)).
Therefore, as 2KB B(2A+ + B) $ 0.1, we have that
[,)(()) -
1 - (3KB (3KB+1.B(24+ 1.5(22+ B)) < 4KB
n
n
Lemma D.2. Ler > 6n
K 10000nd
Assume that for any
1 isism. 11 Ae1 s B. Then for any we have that
1
6KBA
| L Ae S
Proof. Without loss generality, we assume that i here.
Let
S(x) (1)))=a)))) + (x: Σ wy ( 1n 2w1 20)(2)0(2)0(() in
2w0 (x: Σ (x)) (v(x)) T (5)) mj + 3 (x) (v(x) v(x)) v(x).
Then by Lemma B.I have Vu (E
Notice that ||l Aeill B for -.. n. have that [|v(z) Aeill I B v(x)
Σj=1 Wy ())5)))
Therefore, when ||س||| < K, A |S(x) =ell n
wi (x)o(()) - - Aee) wi(x)(())) (wi (m1- o(x)) + wi(x) Σ wi (r) (wi v(x))
n
w (z))((2))(()) (
Ensw,(t) (14 u(x)) -w())) " v(x))
4KB
1+ 4KB (A B)²
(A B + B 4.
n
n
5KBA
n
Also, when ||x||| > K, we have
A -e < (A B) + 8(A+ B)3 + 2A + 1043.
A
|S(x)
n
As the probability of || > K is at most exp(- - v2)2/2) < B², we have
[15(2)
A
6K
IV L eill < E E
eill x
n
n
38

---

Published as a conference paper at ICLR 2026
Now are ready to prove Theorem 3.2
Proof of Theorem 3.2 We prove that the fact 14 (4(7) - < B(7) is maintained for = 1, ... n by induction. We just consider one single iteration. We omit the superseript (r) for simplicity. In one step, we use A to denote A K denote K(T), and B to denote B(7), Notice that when A s Lemma B.4 has given the convergence guarantee, we only consider the case when A 2 6n
After one iteration, changed to A' = (1 n/n)nn A. is changed to 14 = F4g v uy L. We have that
(1 - (n/n)Aeell - || Aeill + m/l u. Ae1l || - Vn 114-n1p
n
6KBA
B + 1)
n
By the fact that
10 2
n
109r"dex((109m((1111)
10t d exp( (d.4)
2 B 1+ 6
n
|| V C (1 A
C. By induction, we know we have that the result holds for A' (1 n//n) A and n² = μi - u..
that || Aalll S B(²) for all 0.
Therefore, after O nloog Mo)
iterations, we have ( I 6n since A = (1-1//nAA()
Thus comes to the case as Lemma B.4
D.3 PROOF OF SECTION 3.3
Here, also consider that μx 0, and the initialization pi (0) is initialized by (MMee Ta).
We consider the case where there exists such that for any j # i, we have || 2 ||s/|| + M2/3 Also, all p4's satisfy ||m: Mull I M
Lemma D.3. Let Mo 2 10 10 Vd. The probability that:
1. all pa's satisfy [|p Moll S M0 /3
2. there exists i such that for any j # i, we have ||gl|| 2 ||x/|l + Mo
at least n²2 1/3
Proof. For each μs, the probability that I|m Moll 1//5 is at most exp ((M1)/3 33 -V by Lemma A.I
Now we consider the probability that:
1. -M, 1/3 -M, < ||/||| - || s MMM5/' 1/3
2. All p:'s satisfy ||; Moll < M
Actually, we consider the first coordinate of μ4, P22 where Mo - VTo S 11,00 S Mo + v Mo- Then
|| 2 ||| - ||
1
+31
2213
2
Mo 2(Mo
M2/1)
|2| - M²
||||ا||
>
2
(o + Mo /3)
(M + MM
2 2Mo + -M213)
39

---

Published as a conference paper at ICLR 2026
Therefore, if [20 222 2 2M,
1/3
have |||i||| || Mo 1/3
Now we just need to bound the probability that [x] x2 I 2M 1/3 Notice that when we fix
21 E [Mo - My MM + o the probability that [x] - x2l
S 2M
1/3
most
11 M1+
(Mo - x22)
2M0 1/3
exp
dr S
Mo 1/3
22T
2
V2T
The result is similar all (i.5) pairs (which means that for any two H4d Myj the probability that -My & |uul| - || _ M, 1/3 is at most
2M, 1/3
/2m
Therefore, in conclusion, the total probability of the two conditions is at least
- --nexp ((M6 vdd2/2)
n(n (11-11 1)
2M 1/3
2 Mo 1/3
22T
Moll S 2M/ for all 2, || 2 || + Mo Lemma D.4. Let n < ffr all ||ls then we have that:
1. |0 - Wus < 100M3 exp (-200
2. For any i 12 2, | C1l S 100 exp
1
M
2/3
200
Proof. Recall that in Lemma B.I have that
V Lr -EENN(0,1a) (x(x))(x)) + wx(xx) Σ Wy (z))) in (wx(z)x(z)00(x) m
-ww,(x)) Σ (x)( (u(x) p3) Ms + (w)(x) (o(z) v(x)) v(x)
(6)
Notice that when
||||||
f 1/3, have
1+ En exp(x T (u, - 11) -1,1111,,,,,1//3
1 + (n e)exp { MM133 . 4M1/3 (M 2M-1/3) M M-133)
1
1 + (n - 1) exp M2233
Thus wi I (n- exp (- M2133.
40

---

Published as a conference paper at ICLR 2026
Notice that have || - 2M for each i. Therefore, by eq. ( combining the contributions of
such that ||||| I |M/ and ||>||| > M have that
|| V n. +| S(n 1) exp M2/3 (2M +8 8.8)) + exp (M (2M + 8.
B B
1 e(x)+ Σ wy ( p1
-20(z)o(z) MI - Σ wi (v(x) us
+ 3 o(()) v(() - 11
1 M2/3
exp (-1 100M
<70M exp
+ 4(n exp
200
1
<100M exp
2/33
200
Moreover, as for any i 2 2, wi(x) $ - wi (3)). we have
|| C1I S 70M8.
(n- (aa-13 exp
MP/3 + exp (A/1///-vay)
100M exp
.12/33
200
Therefore, the lemma follows.
Proof of Theorems 3.5 and 3.7 Without loss of generality, we assume that the initialization satisfies
3.7
||m - Moll S Mo for = 1,.... nn. Also, for any i 2 2, we have |//||| 2 || + M, 1/3
Notice that the two conditions in Lemma D.4 hold, by the lemma we have that
|| v, 4Il s (1 1))len n 100M8 exp
M2/3 < Se
200
Therefore, in at most iterations, the condition must longer hold. However, during the iterations when the two conditions hold, for i > 2, we must have
(-1 M
1 2/3
1444 V n4 A 2 || || - 17 100M exp
(-200
200
Therefore, the second condition always hold in these iterations, and the first condition for i 2 2 is maintained because
(1 + 2) 100M exp (-
1
M S 2.
200
Therefore, in at most 1 iterations, we must have one time that || Moll 2 2M, we find that one iteration, we have
1/3 However,
|| - || - ,, C1 2 2 vlll|l - -1- 100M exp (- (--_ M2/3
200
2 7/ V #1 L 2n 100M3 exp
M223
200
2 M in + 1) 41 Cll - M - mIll - 2n 100M exp (-200 2/3
200
As in the at most + 2 iterations, || 1i || has increased by M!! we have that || || has decreased by at least M /3 2. Notice that for each i 2 2, the norm of μ4 has decreased by at most 2. Therefore, at this time (denoted time 7o), have the following conditions:
41

---

Published as a conference paper at ICLR 2026
1. for any 2 2. || 2 || Mo0 4.
2. for any 2 2. ||| 2 Mo 2.44' 1/3
Now we prove that we always have:
1. for any 2 2, || 2 | || + M 4.
2. for any 2 2. || 2 Mo -3M0
before we have | $ 1/(8))))
First, under this assumption, for any such that ||0||| S Ma.' we have that
W1 (ac)
1+ Σド exp(x' (14 M1) + (1 1|2 11-1111111 /2
(-l1luill + (1 ( -| + ||x||]
1 1 Σラ= exp
1 1 (n - -1)
--1 10,00 3220/8)
1
1+ (n - 1) exp (oo
Therefore, for any ||>:|| S the two assumptions, for any 2. we have
3 M we have that wi: wx(rr) S (n 11) exp(-Mop for 2 2. Thus under
|| ClI & (n 1m=1) Mo) + exp (-(3MG') -3- 11 vi
2
/2 12) -10M < 10exp(-0.05.,, )Mg.
On the other hand, if ||- 2 1/(8M))) have
(- MM ( - /2
va)
(-
1 2/3
|| vu, nn ClI <
exp
s 100 exp
20
Therefore,
1 1 L S |1 +
10M3 exp( exp(-0.05MM2/3
(1 -///)//mm11 - - mmll ||-
Therefore, in at most 1 + 10 log Ma iterations, we have || < 1/(8M3). In such time, the distance between 14 and Mo has increased by at most
(1 + 10 Mo/n) 10M exp(- M
Thus before | || < 1/(8M)), the two conditions always hold. Therefore, after O(log Mo/1) iterations. the condition becomes the same as Lemma C.I
Proof of Corollary 3.6 By Lemma D.3 the result is true.
D.4 PROOF FOR THEOREM 3.4
By Lemma B.I we know that the gradient for any l4 can only have a nonzero value on the first
coordinate
linear combination of all μý- For each we denote a; the first coordinate of H4. We denote u(r) as the first coordinate of v(x). Also, we denote L, as the first coordinate of
V C. During training, always denote μi (M e, 0, and μ4 (M, 0, .0) for
i = 2, ... n.
We need the following lemma:
42

---

Published as a conference paper at ICLR 2026
Lemma D.5. Let M > 10 vd no Assume that μ1 (M e, 0,0,, ... 0) and 14 (M,0,0, for i = 2. n. Here 0 < c < M/4. Then, we have
::
1n C 2 2E, [w) (z)u(x)
V nz C - 2B [w2 (x))((x))]
Proof. By
Vn, =2E, wx(x)e(z) + wx(x) Σ ws (1)D)up Ps - 2w)(2)((2)0(x)) p
- -w)(x) Σ (x) (u(z)) ps) as + (w)(x) (o(x))<(()) o(x)
we can find that
(2u(xx) a1) an)) (2u(o)2 - Emr(x)af - -m(a)))
L1 w (x))(()) + (x))
2 [wi (x))(()
+ (z) ((M- 2w1 (x) + (2(M ( M² + (2) (2Me +2)) - (M - (M--((x)ee) (x))))))
-2 (x)u(()) + w/ (2) - w)()) (M2 3u0 (x) Mr + 3 (x))²² w (x))
,2 (x))((x
-((o)()
L2 22
w2(x)u(z)+ + w((x)
(2u(z) - 02)
2u(x))² Σ w;(())af )
22
w2(x))(4) + (x() (2u(x) - a2) u(z)² Σ
(x(r)w,(I)(a - a))² -u(z)²
i.j
The last inequality is because u(x) a2 S 2u(x).
Now we prove Theorem 3.4
Proof of Theorem 3.4 In the following, we consider one step such that M > 10a 10%Vd no and
cxp(-M2 /100)
E 1. Notice that for any ||^||| M/3, we have
w1(x)= 2
+ (n exp (- + 2/2
1+ (n 1) exp (-eM/2)
As (2( wa wz(x)) (x)) have
exp (-eM/2)
w2(x) - 1 (n - exp (-eM/2)
Therefore, have
1 - e-xp(-(MM/2)
exp (cM/2) - > cM/2
W/ (I) w2( 2
1 (n exp (-cM/2) exp (cM/2) + (n-1) cM/2 +n"
Also, when ||,||| > M/3, we have (wi - w2 (2(x))((x)] < 2M, and the probability is at most exp -M//3- previous lemma, we have
I exp( (Mp(-M2/20), So when we define L,, Le as in the proof of the
L La > Ex [( (2) w2(x))u(x)1
eM
Ex [u(x)] - 3M 2xp(-M22/1
M /2
eM
(M - e).
M + 2n
43

---

Published as a conference paper at ICLR 2026
Therefore, consider update. We know that 2(M c) S L2 S 2M. Therefore, have M 2T + 2mp!!) 2 M(++1) 2 M(T) 27
Also,
(())
(r+1) 2 (t) c(T) 2m
(M - e
When < 1//M, > 100, we have (r+1) 2 (T) + (Mir)
3n
Notice that when 1/M, M > vd Vd 11 (10M). have
2 6 (.+11 eM2
7M2
exp(MM
exp(-2xg)M
3n
3n
nM2
2 exp(M))
1+
(1 -1- (1-2n1
3n
2 exp(M)c.
Therefore, exp(M/ doesn't decrease. As M doesn't 't increase, before becomes larger than 1 /M, the decrease of M in each iteration is at least y.. Therefore, there must be time such that e > 1/M or there must be a time such that M < Mo/2. However, if M < 2Mo/3, we must have
( (0) exp(Mo) (M)) 1.
exp(M(F)
So before M becomes smaller than Mo/2, 2, must be larger than 1/M.
When ( 1/M M 10 /d 10 have
M
(r+1) (r) +
3n
Therefore, Gre + M doesn't decrease. Therefore, there must time that > 1, M > Vd 10 Also, gradient step is less than 1/M, and each time can be increased by at most 21je, we must have e < this time. This satisfies the condition for Theorem 3.5 and the result follows.
E LLM USAGE DECLARATION
In preparing this paper, the role of large language models (LLMs) was limited to polishing the writing, checking grammar, improving readability, and assisting in drafting and debugging parts of the simulation code, all under the full supervision of the authors. All scientific content, research ideas, theoretical analysis, and contributions are entirely the work of the authors.
44


Reviews:
[{"review_id": "EIncGXoWTJ", "reviewer": "Reviewer_7qUQ", "summary": "The paper studies the convergence dynamics of score matching in an over-parameterized teacher-student setting, where the ground truth is a single Gaussian with identity covariance, and the score network is parameterized as a mixture of $n$ Gaussians with identity covariance. When the noise level $t$ is large, it is shown that gradient descent enjoys global convergence at a sub-linear rate. When the noise scale is small, it is shown that there are spurious stationary points, and thus the training is sensitive to initialization. When the parameters are initialized exponentially close to zero, global convergence is established, while under random Gaussian initialization, it is shown that the loss converges to zero at a linear rate, but only one parameter converges while the others diverge to infinity.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper makes a significant contribution towards understanding the optimization dynamics of score matching, moving beyond the typical student-teacher setting. The paper identifies a nice set-up which captures some key elements of over-parameterized deep learning while still being mathematically tractable. The results shed light on the qualitative effects of over-parameterization for score matching, in particular the sensitivity of initialization in the low-noise regime.\n\nAdditionally, it is a very well-written paper. I did not have time to read proofs in the appendix carefully, but the main paper does a nice job of illustrating the central ideas. Their analysis of the gradient dynamics in the small noise regime requires novel techniques.", "weaknesses": "The only weakness I can think of is that the paper analyzes the gradient dynamics of the score matching loss at a fixed time, whereas in practice, one usually trains a single, time-averaged score matching loss. I don't see this as a major flaw of the paper, but I am curious whether the results could be extended to this setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AvQd2EKtIQ", "reviewer": "Reviewer_px2W", "summary": "This paper studies the training dynamics of Gaussian mixture models of the score matching objective at a fixed point t. The analysis is conducted under the assumption that the underlying data follows a simple Gaussian distribution with an identity covariance matrix. The “overparametrization” comes from the fact that the class of models is that of Gaussian mixture models with $n$ modes while the underlying distribution is just a Gaussian distribution (i.e. Gaussian mixture with $m=1$ mode). In this work, the authors also show a surprising phenomenon that when the initialization of the weights $\\mu_i^{(0)}$ is far away from the target $\\mu^*$, the weights converge whilst a slight perturbation in the initialization may lead to divergence of the weight. It is worth pointing out that, although the individual parameters may not converge, the score function still converges to the target score function.", "questions": "What happens when two or more parameters are closer to the target $\\mu^*$ and the others are further? I would expect that the ones that are closer to converge to the target while the others diverge to infinity. A deeper discussion on this would be welcome.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper builds on and simplifies the model introduced in Buchanan et al., “On the Edge of Memorization in Diffusion Models”. Its main contribution lies in the analysis of the training dynamics of such models, which provides a complementary perspective to Buchanan et al.’s work. The paper also offers interesting insights into the sensitivity of these dynamics: it demonstrates that even small changes in initialization can shift the system from convergence to divergence, revealing a subtle transition phenomenon.", "weaknesses": "1. I am not completely convinced that this overparametrized setting, even if it can outline interesting phenomena, is the one to be considered. \n2. The authors claimed that it is important in practice to include the score matching in the large noise regime. However, the value of the loss function is scaled by $\\exp(-t)$, which should be negligible.\n3. It may be better to merge the two separate regimes (i.e. $t$ large and $t$ small), and discuss the effect of initialization, which in my opinion gives a clearer picture of the problem.\n4. The problem studied is actually equivalent to approximating the standard Gaussian from different initialization; the multiple reparametrizations (setting $\\mu=0$ and $t=0$) are a bit hard to parse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZmbOPJDUiA", "reviewer": "Reviewer_kQmq", "summary": "Diffusion models have become go-to approaches in generative modeling, achieving state-of-the-art performance across numerous visual computing tasks. Recent research has extensively analyzed their convergence behavior through metrics such as the Kullback–Leibler divergence and the Wasserstein distance under various assumptions on the true data distribution.\n\nThis paper focuses on a simplified theoretical setting in which the ground truth distribution is Gaussian and the score network is parameterized by n learnable parameters. The authors establish, in Theorem 2.1, the convergence of gradient descent applied to the score matching objective,  extending the known relationship between score-based diffusion models and the Expectation–Maximization (EM) algorithm. They prove that, under an appropriate choice of step size, the convergence rate scales with the square root of the number of iterations.\n\nMoreover, Theorem 3.1 demonstrates that for small diffusion times t, global convergence cannot generally be guaranteed; however, convergence to the ground truth can still be achieved under a suitable exponentially small initialization of the parameters. \n\nFinally, a concise empirical experiment is provided to validate and illustrate the theoretical findings in a controlled setting.", "questions": "- Would it be possible to extend the analysis in the strong logconcave setting ? And if not, the paper being already rather technical, could you discuss the most limiting points where strong log concavity is not enough and Gaussianity really matters to obtain the convergence results ?\n\n- Even if the paper's contribution is mostly theoretical, as the true data distribution is Gaussian, it would be interesting to see the empirical impact (i) of the dimension and (ii) of the structure of the covariance matrix of the true distribution. \n\n- The main results focus on the classical gradient descent algorithm.  However, score-based models are trained using stochastic gradient descent or adaptive gradient descent. Could you discuss the extension of your analysis for a stochastic gradient optimization ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper provides a detailed convergence analysis of the gradient descent optimization in the setting where the true data distribution is Gaussian. It allows to display settings in which convergence is guaranteed with a quantified rate (large t) and settings where convergence depends on the initialization of the parameters (small t). The authors also highlight a behavior where loss convergence may still lead to non converging behavior of the parameter.\n\n-  Even if the chosen true data distribution is really simple, the authors highlight a complex (and technical) analysis which has not been considered in the literature.\n\n- The experiments are simple but support well the theoretical guarantees.  They illustrate both behaviors, the case where all parameters are estimated correctly and the behavior under the other regime, where only one parameter converges and the others diverge.", "weaknesses": "- Of course restricting to a Gaussian case allows explicit computations which are interesting to understand deeply the behavior of the optimization process. Still, most theoretical papers have obtained theoretical guarantees in terms of KL or Wasserstein distance for strongly log concave distributions (even if this assumption has been relaxed recently). Extending to this setting or discussing the difficulty to extend the results, even if strong log concavity is still a strong assumption, would greatly improves the impact of the paper with respect to the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)