Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts
Authors: Ruofeng Yang
Yongcan Li
Bo Jiang
Cheng Chen
Shuai Li
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Estimation Error
Convergence Analysis
Mixture of Experts
Primary area: generative models

Abstract: Recent diffusion models demonstrate remarkable sample efficiency and fast optimization, contradicting standard estimation bounds that suffer from the curse of dimensionality $n^{-1/D}$ with the data dimension $D$. Since images are usually a union of low-dimensional manifolds, current works model the data as a union of linear subspaces with Gaussian latent and achieve a $1/\sqrt{n}$ bound. Though this modeling reflects the multi-manifold property, the Gaussian latent can not capture the multi-modal property of the latent manifold. To bridge this gap, we propose the mixture subspace of low-rank mixture of Gaussian (MoLR-MoG) modeling, which models the target data as a union of $K$ linear subspaces, and each subspace admits a mixture of Gaussian latent ($n_k$ modals with dimension $d_k$). With this modeling, the corresponding score function naturally has a mixture of expert (MoE) structure, captures the multi-modal information, and contains nonlinear property. Empirically, our MoE-latent MoG network significantly outperforms MoLRG Gaussian baselines and matches MoE-latent U-Net performance with $10\times$ fewer parameters, validating its practical suitability. Theoretically, we provide provable convergence guarantees for the optimization process and establish an estimation error bound of $R^4\sqrt{\sum_{k=1}^K n_k}\sqrt{\sum_{k=1}^K n_k d_k}/\sqrt{n}$, successfully escaping the dimensionality curse.  Collectively, with MoLR-MoG modeling, this work explains why diffusion models only require a small training sample and enjoy a fast optimization process. Furthermore, we also show the potential of MoE structure for diffusion models from the manifold perspective.

Published as a conference paper at ICLR 2026
MULTI-SUBSPACE MULTI-MODAL MODELING FOR DIF- FUSION MODELS: ESTIMATION, CONVERGENCE AND
MIXTURE OF EXPERTS
Ruofeng Yangl.3i, Yongcan Li Bo Jiang', Cheng Chen', Shuai Li
1 Shannhaii Jiao Tong University, (wanshuiyin, joseep bjiang, shuaili8) @sjtu.edu.cn "Eat China Normal University, chchen sei. sei.ccnu.edu.c
SShanghai Key Laboratory of Intelligent Information Processing
*Shanghai Innovation Institute
Eqqua Contribution Corresponding Author
Golf
Church
Dog
Parachute
Ttruck
Figure 1: ImageNet results with expert specific VAE and small latent 2-layer Softmax-type network.
ABSTRACT
Recent diffusion models demonstrate remarkable sample efficiency and fast op- timization, contradicting standard estimation bounds that suffer from the curse of dimensionality n -1/D with the data dimension D. Since images are usually a union of low-dimensional manifolds, current works model the data union of linear subspaces with Gaussian latent and achieve a 1/vn bound. Though this modeling reflects the manifodd property, the Gaussian latent can capture the multi-modal property of the latent manifold. To bridge this gap, propose the mixture subspace of low-rank mixture of Gaussian MMoLR-MMG) modeling, which models the target data as a union of K linear subspaces, and each subspace admits a mixture Gaussian latent (ng modals with dimension de). With this modeling, the corresponding score function naturally has mixture of expert (MoE) structure, captures the multi-modal information, and contains nonlinear property. Empiri- cally, our MoE-latent MoG network significantly outperforms MoLRG Gaussian baselines and matches MoF--latent U-Net performance with 10x fewer parameters, validating its practical suitability. Theoretically, we provide provable convergence guarantees for the optimization process and establish an estimation error bound
of Rd EK / Ek-1 Rada/ Vn. successfully escaping the dimensionality curse. Collectively, with MoLR-MoG modeling, this work explains why diffusion

---

Published as a conference paper at ICLR 2026
models only require small training sample and enjoy a fast optimization process. Furthermore, we also show the potential of MoE structure for diffusion models from the manifold perspective.
INTRODUCTION
Recently, diffusion models have achieved impressive performance in many areas, such as 2D, 3D,
and video generation (Rombach et al. 2022 Ho et al. 2022; Chen et al. 2023a Ma et al. 2024b
Liu et al.] 2024] Tan al. [024: 2025 Ma al.] 2024a] Feng et 2025 Yan et al. 2025). Due to
the score matching technique, diffusion models enjoy a more stable training process and can achieve great performance with a small training dataset.
Despite empirical successes, theoretical guarantees for score estimation in diffusion models often suffer from the curse of dimensionality. Standard bounds for architectures like deep ReLU networks and diffusion transformers yield a minimax rate of n = (s' is the smoothness parameter of the score function (Oko et al. 2023; Hu et al., [2024bla a Fu et al. 2024), which fails to explain their sample efficiency on high- sigh-dimensional data. To bridge this gap, recent works exploit specific structural assumptions about target distributions. One line of research adopts multi-modal modeling, such as Mixture of Gaussians (MoG), to better capture real-world data and improve estimation bounds (Shah) al. 2023 Cui et al. 2023 Chen et al. 2024b; Zhang al., (025). Another approach assumes data resides on a low-dimensional linear subspace (x = Az with bounded = E IRd,, successfully reducing the estimation error to n 2/d and eliminating the dependence on the ambient dimension D (Chen
et al.
2023b Yuan al. 2023 Guo al. 2024). However, as shown in Brown et al.) (2023) and Kamkari et al. (2024), though the image dataset admits low dimension, it is a union of manifolds instead of one manifold. Inspired by this observation, |Wang et al. (2024) model the image data as union of linear subspaces, assume each subspace admits a low-dimensional Gaussian (mixture of low-rank Gaussians (MoLRG)), and achieve a 1/v/ estimation error. Though the union of the linear subspace is closer to the real-world image dataset, the latent Gaussian assumption is far away from the dindeneniional multi- modal manifold (Brown et al. 2023). Hence, the following two natural questions remain open:
Can we propose a modeling that reflects the multi-manifold multi-modal property of real- data?
Can we escape the curse of dimensionality and enjoy a fast convergence rate based on this modeling?
In this work, for the first time, we propose and analyze the mixture of low-rank mixture of Gaussian MMLR-MMG) distribution, which is more realistic than MoLRG since it captures the multi-modal property of real-world distribution and has a nonlinear score function. Based on this modeling, we first induce a MoF-latent nonlinear score function and conduct experiments to show that Mol. RM- modeling is closer to the weal-world data. After that, we simultancously analyze the estimation and optimization error of diffusion models and explain why diffusion models achieve great performance.
1.1 OUR CONTRIBUTION
MoLR-MoG Modeling and MoE Structure Nonlinear Score. We propose the MoLR-MoG modeling for the target data, which captures the multi low-dimensional manifold and multi-modal property of real-world data and naturally introduces the MoE--latent MoG score. Through the real- world experiments, we show that with this score, diffusion models can generate images that is comparable with the deep neural network MoE-latent Unet and only has 10x smaller parameters. On the contrary, the MoEEllaten Gaussian score induced by previous MoLRG modeling can only generate blurry images, which indicates MoLR-MoG is a suitable modeling for the real-world data.
Take Advantage of MoLR-MoG to Escape the Curse of Dimensionality. For the estimation error, we show that by taking advantage of the union of a low-dimensional lincar subspace and the latent MoG property, diffusion models escape the curse of dimensionality. More specifically, we achieve the Rd "K_n ""__,,n /n estimation error, where R is the diameter of the target data, dg is the latent dimension and ngg is the number of the modal in the k-the subspace. This result clearly shows the dependence the number of linear subspaces, modal, and the latent dimensions R, de-
2

---

Published as a conference paper at ICLR 2026
Strongly Convex Property and Convergence Guarantee. After directly analyzing the estimation error, we study how to optimize the highly non-convex score- maathinn objective function. Facing nonlinear latent MoG scores, we use the gradient descent (GD) algorithm to optimize the objective function. To obtain the convergence guarantee, we take advantage of the closed form of nonlinear MoG score and show that the landscape around the ground truth parameter is strongly convex. Then, with great initialization area, we prove the convergence guarantee when considering MoLR-MoG.
2 RELATED WORK
Estimation Error Analysis for Difflusion Models. As shown in Section ! series of works Oko et al. (2023) study the general target data with deep NN and achieve the minimax -=x result. Then, some works analyze the general target data with a 2-layer wide NN and achieve n 22/5 estimation error with exp (n) NN size (Li et al., 2023 Han et al. 2024). For the multi-modal modeling, some works study MoG data and improve the estimation error (Shah et al.) [023 Cui et al. [2023 Chen eet al. 2024b). Except for the MoG modeling, Cole and Lu (2024) assume data is close to Gaussian and then prove the model escapes the curse of dimensionality, Mei and Wu (2023) analyze Ising models and prove that the term corresponds to n is 1/ /n. For the low- diw-dinensional modeling, some works assume the target data admits linear subspace (Chen al., 2023b Yuan al., (023). Chen al. (023b) assume data admit a linear subspace image is a union of low-dimensional manifolds, Wang et al. (2024) models the target data as a union of linear subspaces with Gaussian latent and achieve 1/ /n estimation error for each subspace.
A with
Rd and achieve
2/a As the
Optimization Analysis for Diffusion Models. Since the score is highly nonlinear (except for Gaussian), only few works analyze the optimization process, and most of them focus on the external
dimensional space (Bruno et al. 2023 Cui and Zdeborová 2023; Shah et al. 2023 Chen et al.
2024b Li al. 2023 Han et al. (2024) Since the score function of MoG has nonlinear closed-form
a series of works design algorithms for diffusion models to learn the MoG (Bruno et al. 2023 Cui
and Zdeborová 2023; Shah et al. 2023 Chen al. 2024b). For the general target data, Li al. (2023) and Han et al. (2024) adopt a wide 2-layer ReLU NN to simplify the problem to convex optimization. However, as discussed above, their NN has exp (n) size. For the latent space, Yang et al. (2024a) assume target data adopts linear subspace with Gaussian latent and provide the closed-form minimizer. Waan et al. (2024) analyze the optimization process of each linear subspace separately, which is also reduced to the optimization for the Gaussian.
PRELIMINARIES
First, we introduce the basic knowledge and notation of diffusion models. Let po be the data distribution. Given vo Po E RD, the forward process is defined by:
dæ, = [t)xx; dt + g(t) dBr,
where (Be is D-dimensional Brownian motion, f(t is the coefficient of the drift term and g({) is the coefficient of the diffusion term. Let p. be the density function of the forward process. After determining the forward process, the conditional distribution pt (x; (z)) has a closed-form
Pt (z (x)) = N (TT SYTo, s I
where 8, exp (Io (()ds)
o. VL ((E))>((EE)) To generate samples from po. diffusion models reverse the given forward process and obtain the following reverse process (Song et al. 2020):
dy: [f())) - ( log Pt (v:)) dt g())dB, Yo ~ po
where B, is a reverse-time Brownian motion. A conceptual way to approximate the score function is to minimize the score matching (SM) objective function:
1.¹ Exangg | log (z.) - s0 s0(
min LSM S6 ENN
(1)
where NN is given function class and 8 > is the early stopping parameter to avoid a blow-up score. Since the ground truth score V log pt unknown, this objective function can not be calculated. To avoid this problem, Vincent (2011) propose the denoised score matching (DSM) objective function:
min CDSM = / Ezong V log (Iz(zo) sa( t)ll dt.
SNENN
3

---

Published as a conference paper at ICLR 2026
As shown in Vincent (201T). the DSM and SM objective functions differ up constant independent of optimized parameters, which indicates these objective functions have the same landscape.
3.1 MIXTURE OF LOW-RANK MIXTURE OF GAUSSIAN (MoLR-MoG) MODELING
This part shows our MoLR-MoG modeling, which reflects the low-dimensional (Gong et al. and mudti-modal property (Brown et al. 2023] Kamkari et al.) 2024) of real-world data. More specifically, we assume the data distribution lives union of K linear subspaces rather than arbitrary manifolds. Concretely, for the k-th subspace (represented by matrix A E IR nxxx with orthonormal columns the k-th manifold), place ng-modal MoG within that subspace:
2019)
n1k
WK
(x(x))
TE, AlRkE AEEE,A
where covariance - U2 = mean of the 1-th modal of the k-th subspace. As shown in (Brown al. 2023), the different manifold has different d2 and we do not require that dy is exactly the same for each manifold. Then, the target distribution has the following form
ny with Ug xdu, (dg, S dy) and p , is the
K
PO
NEI A3ER,AA)
(2)
From the universal approximation perspective, by placing enough components and choosing parame- ters (TK, 1 Xg, a MoG can approximate any smooth density arbitrarily well, which is more general than the Gaussian latent of Yang et al. (2024a) and Wang et al. (2024).
Mixture of Experts NMoneeNonlinear MoG score. Let " s,0r, EkLLAA "AA -
yf and (z) z stpt, a2 AAUR,UT At (x: st AE) Under the MoLR-MoG modeling, the score function has the following form:
K
k T N.;
Ai A 85.1,1((())
K
V log)
k=1
77
NN, Ai A
This score function has a MoE structure, where each expert the latent nonlinear MoG score. The linear encoder A₁ first encodes images to the k-th manifold, and diffusion models run the denoising process. After that, the linear decoder A decodes the denoised latent to the full-dimensional images. Since the estimation error introduced by the linear encoder and decoder has the order Dd /n (Yang et al. 2024a) and is not the dominant term, we assume the linear encoder and decoder are perfectly learned and focus the more difficult latent MoG diffusion part in this work. From the empirical part, this operation is similar to using the pretrained stable diffusion VAE and only training the diffusion models in the latent space. For the k-th low-dimensional manifold, the score function is
kk. N sepee Ex.,t) (x
v log (o
=
(3)
78
k N N(; (1; Ek.)))
where I I.D. E Rh is a variable in the k-th low- lww-dimensional subspace, EaLt = S²U + 7
and (xLL) LLD s014 87t7 Ut UKT (g 8t1().)) Let
o?
(xLD s' where the parameters are 6* {ut, . Ui..A=I.....k In this work, we want to learn the pa- rameters of the ground truth score function. Hence, we construct a NN function class
(D V log lo LD),
1), -s) (((,"D,))........ $ (x ..),
(s1 (+.),, s2())), ...,,,skk (...)). according to the above closed-from of MoE-latent MoG score. Let 0 is the union of μa,1 and UR Since we mainly focus on the estimation and optimization in the latent subspace, we omit the superscript LD of the latent subspace when there no ambiguity.
4

---

Published as a conference paper at ICLR 2026
MNIST
7
7
7
7
7
7
9
9
9
9
9
CIFAR10
ImageNet
MoLR-Unet
MNIST
7
7
7
9
9
9
9
CIFAR10
ImageNet
MoLR-MoG
MNIST
CIFAR10
ImageNet
MoLR SauR-Ga
Figure 3: Results of Different Modeling on Real-world Data.
note that this modeling capture the information of each low-dimensional man- ifold and the multi-modal property of each latent dis- tribution. In the next sec- tion, through the real-world experiments, we show that the MoE-latent MoG score has better performance compared with the MoE-latent Gaussian score induced by MoLRG modeling and compatible with the results of the MoE-latent Unet. In Section 5 and 6 prove that by using the property of MoLR-MoG modeling, diffusion models can escape the curse dimensionality and enjoy a fast convergence rate.
Encoder
Latent Subspace
Decader
R
R4
A
R
MOG Expert1
M
Rd
Rd
X
MOG Expert2
Xe-1
R
My
Rd
Agr
A5
MOG Expert3
aa) MoLR-MoG Modeling
(b) MoF-nonlinear MoG Score
Figure 2: MoLR-MoG Modeling and Corresponding Nonlinear Score
Remark 3.1 (Comparison with MoLRG modeling). Wang et al. (2024) provide the first multi-subbsace modeling, which is important and meaningful step and Li al. (2025) further design noisy MoLRG to study the representation of diffusion models. However, they assume Gaussian latent with mean, which can not capture the multi-modal property real-world data. also note that the MoLR-MoG modeling can not be viewed as MoLRG with Σk= n4 subspace since this modeling assumes there 1= 7&g VAE, which is not reasonable in the real-world setting.
4 EXPERIMENTS FOR MOE-LATENT MoG SCORE
In this section, we empirically validate that our MoLR-MoG framework effectively models real-word data distributions. Specifically. demonstrate that the MoE-latent MoG score only generates semantically clear images that significantly outperform the MoL.G baseline, but also achieves performance comparable to a general R-U-e- while requiring 10x fewer parameters across MNIST, CIFAR-10, 10, and ImageNet 256 (Figure
Following Brown et al. (2023), we train VAEs for each number in the MNIST, which represents our K low-dimensional manifold. In this part, we adopt nonlinear VAEs to achieve a good perfor- mance in real-world datasets. However, we still note that a series of theoretical works adopt linear subspaces, and our MoLR-MoG modeling with linear VAEs makes step toward explaining the good performance of diffusion models. Subsequently, we train diffusion models comparing three parameterizations: latent U-Net, latent MoG NN (parameterized via Eq. 3 with n& E (4, (4,8, 40)

---

Published as a conference paper at ICLR 2026
for MNIST, CIFAR-10 10, and ImageNet 256, respectively, for & [K]), and latent Gaussian NN utilizing a flosed-form linear score (Wang et al. 2024).
Discussion. From qualitative perspective, as shown in Figure 3 the generation results with MoLRG model- ing are difficult to distinguish specific numbers. On the contrary, the MoE-latent MoG can generate clean im- ages comparable with the images generated by MoLR- Unet, which means this modeling captures the multi- modal property of each low-dimensional manifold. The training loss curve (Figure + shows that the loss of MoE-MoG NN significantly smaller than the MoE- Gaussian and close to MoF-Ueet, which indicates MoE- MoG NN efficiently approximates the ground-truth score and supports our theoretical results. From quan- titative perspective, we calculate the CLIP score for the parachute class of ImageNet with text prompts "a photo of parachute". The Clip score for MoLR with Unet, MoG, and Gaussian NN is 0.304, 0. 0929,, and 0.254, which indicates MoLR-MoG achieves almost comparable text-to-image alignment with MoE-Unet. Furthermore, the MoLR-Mod NN contains many fewer parameters compared to Unet since it uses the prior of latent MoG.
0.8
Single Gaussian (Baselin)
0.5
MoLR.Gaussian
U--et
MSE Loss (Log
0.33
0.1
20
60
100
120
140
Epochs
Figure 4: Loss Curve for CIFAR-I 10
Discussion Expert-Specifie VAE. As shown in the score of MoLR-MoG, different from latent diffusion models with a single VAF, there are K VAEs to encode the input to the corresponding manifold. We note that this operation is important for MoLR-MoG with small MoG experts. As shown in Figure with a unified VAE, the unified latent is complex, and a MoG expert can not learn a meaningful image with the target class. Hence, with unified VAE, latent diffusion models require a large latent Unet. However, with an expert-specific VAE (for example, we fine-tune the pretrained VAE with the parachute class dataset), the latent manifold becomes simple, and latent MoG experts are enough to generate clear models, which also supports our theoretical modeling.
We note that these experiments aim to show that the MoLR-MoG modeling is reasonable instead of achieving the SOTA performance. It is possible to achieve great performance with small-sized NN using MoLR- modeling in the application. For large-scab datasets without labels, we use clustering algorithm to divide the data into different clusters. Then, we can train VAE encoder, de- coder, and latent MoG score for each cluster. For the VAE training, we do not require training the VAE from a sketch. We can LoRA fine-tune a VAE pretrained on large-scale datasets (for example, DC- AE (Chen et al., 2024a) for our ImageNet experi- ments) for each expert, which shares a pretrained VAE backbone and has a smaller model size. When generating images, we activate different VAE LoRA according to the clustering weight, which matches the spirit of MoE. We leave it as interesting future work.
w/ Expert Specific VAE
w/ Unified VAE
Figure 5: MoLR-MoG with Different VAE
ESCAPE THE CURSE OF DIMENSIONALITY WITH MOLR-MOG MODELING
This section shows that diffusion models can escape the curse of dimensionality by using MoLR-MMoC properties. Before introducing our results, we first introduce the assumption on the target data.
Assumption 5.1. For x Po. we have that | S R.
The bounded-support assumption is widely used in theoretical works (Chen al. 2022 Yang et al. 2024b Bortoli 2022; Yang et al., 2025a b) and is naturally satisfied by image datasets. For a latent MoG, each component concentrates almost all mass within few standard deviations of its mean, so by taking the most component means and variances, one can choose R large enough that || | S R holds with high probability.

---

Published as a conference paper at ICLR 2026
Since Moe-latent MoG score has closed-form, we only need to learn the parameters Pk.1 and Uk at a fixed time t. As a result, we consider the estimation error at a fixed time t. Let E(8;z, xz t) = (1, t) s (z: l|2 be the per-sample squared error at time t. In this part, study the estimation error with limited training dataset { 1i=1
[())) - (0) with -x0000==>>00000000;;
= - ΣØ (00 T1
To obtain the estimation error, we first provide the Lipschitz constant for sq and the loss function by fully using the property of MoLR-MoG modeling and MoE-latent MoG score.
Lemma 5.2. [Lipschitz Continuity] Let Tep and Lu be the Lipschitz constant W.E. 80- With R-M modeling and Theorem
5.1 there is a constant
LS V EK (L,, - = =
such that for any 0,0', [1,) - sgr (z, t) S L 1 1112. where Cw (R+ (R+s,By) BB. max||ux,112- For se and s*, we have that 2 (z) s* t)l2 2(R stBw// Li.
21
Then, we obtain the Lipschitz constant L' L for the whole loss function. With this Lipschitz property, the next step is to argue that fitting the network on samples generalizes to the true population loss. We do so by controlling the Rademacher complexity the loss class and then using Bernstein concentration argument obtain the following theorem.
Theorem 5.3. Denote by (o) the empirical loss n i.i.d. samples and by [(0) its population counterpart. Then there exist constants C1, C2 such that with probability at least 1 6, for all 0 e O
[C()) - (0) s 0fc, (R (++88 s,B,))
EK k=
1og((10)
+ C2 V
?
where C1 = max 10, - 0,12, C2 = PEB
a log o² = sup ((0 X,t)
This result removes the exponential dependence on D with the number latent subspace K, the latent dimension de, and the number of modalities nı at each linear subspace, which reflects the key feature the real-world data and escape the curse dimensionality. The remaining question is why diffusion models enjoy fast and stable optimization process. In the next part, show that with MoLR-MoG modeling, the objective function is locally strongly convex and answer this question.
6 STRONGLY CONVEX PROPERTY AND CONVERGENCE GUARANTEE
In this part, by using the property of MoLR- modeling, we derive explicit expressions for the Jacobian and Hessian of the objective function for 2-modal MoG latent and general MoG latent. Then, we establish conditions under which the resulting score-matching loss is locally strongly convex for each setting. Finally, we provide the convergence guarantee for the optimization.
6.1 2-MODAL LATENT MOG HESSIAN ANALYSIS AND OPTIMIZATION
In this section, show that, under sufficient cluster separation, the Hessian matrix near 0* simplifies to a block-diagona form, yielding local strong convexity, which derives a linear convergence rate. following the real-world setting, consider the optimization dynamic in the k-th latent subspace. While our modeling contains K encoders and decoders, facing an input image x, we can first determine which cluster image æ belongs to, and then use the corresponding Aş to encode it into the corresponding latent space. Then, we only use data belonging to & clustering to train the k-th latent MoG score. This operation matches our experimental settings, and Wang et al.] (2024) also adopts this operation. When considering the optimization problem, to simplify the calculation of the Hessian matrix, we set de, =
As discussed in Section 3. 3.11
Similar to Shah et al. (2023), we start from latent 2-modal MoG with the same covariance matrix Ex and 1 = 1k: Mk,2 = -1t. which leads to the following score:
1 [ scp EE (x)x) + AN(s;; -spph Ei) En (x)
V (logP.a(z)
(4)
7 [x; StltEe Eg) + (z; - -AJU Ei)
7

---

Published as a conference paper at ICLR 2026
where (x()) = 8. - URUE (x-sa14), and & (x) = x+t. of UU (x+8)))) क,
Before providing the convergence guarantee, we make assumption on the 2-MoG latent distribution.
Assumption 6.1. [Separation within a cluster] Within each cluster k. the two symmetric peaks are well separated in the sense that |liss (=stpī] 2 Aintra, for some Aiutra > 7t- Consequently, if a sample æ is drawn from the peak then its responsibility under the " " peak satisfies
W - 8. ) I. (z sug 23 +
OiE - Ammmma///1r (121)) < 1,
FA (x)
Nxx + S. 21)
and symmetrically of (x) < 1 when ø: is drawn from the peak.
The above assumption means that the separation of the two modals sufficient. For each symmetric sub-peak, if the distance between them is relatively small, we can view them as having a mean of O. and Σ = U+UU
Since they are the same distribution (μ of how they mix, which indicates that we can assume r ~ 1 or R 1. Moreover, in practice, raw data do not exhibit such clear gaps, one can always apply simple linear embedding to magnify inter-mean distances relative to noise, thereby enforcing the same hard-assignment regime.
7/ T), they are the same regardless
Since the ground truth score function has a closed-for form under the MoLR-Mon modeling, we focus on the score matching objective function CSM( instead of (D) and abbreviate LSM (0) as (()). We note that (f) and CDSM(() (0) are equivalent up constant independent of 0 which indicates the optimization landscape is the same. Furthermore, when considering the convergence guarantee under 2-layer wide ReLU NN, Li al. (2023) also adopt score matching objective CsM instead of LDMM Though calculating the bound of Jacobian J (z) 0j 80, J (x) and the Hessian matrix w.r.t. L, we provide the local strongly convexity parameters for the objective function.
Lemma 6.2. [Local Strong Convexity] Combining Lemma B.4 with continuity of V L, there exist a > and neighborhood U of 6" such that V (()) ž= al, 70 E Of E R (x) = or ri (x) = are strictly satisfie,,
4(U pk)) + 1B 112
U Maalll/ 8(U p²)) + || 11
min
(o] + Ri
2
Theorem 6.3. [Local Linear Convergence) Under Assumptions 5.7 and and6..1 if we take 1/m 17 2/(n + L'), and K = L' ja/a, then there exists a neighborhood U of 6 such that
| - 00
||0(0) - 0-1|2,
where m is the number of gradient descent iterations.
This result gives a lower bound on the convergence rate near 0" Due its strongly convex property, the convergence rate is fast, which explains the fast and stable optimization process.
Proof Overview Assumption 6.11 justifies the Jacobian simplification (Lemma B.2),, which in turn yields the Hessian block structure (Lemma B.4). By Schur complement, this result gives local strong convexity (Lemma 6.2). Combining with the Lipschitz constant, we finish the proof.
6.2 GENERAL MOG LATENT HESSIAN ANALYSIS AND OPTIMIZATION
We now extend our analysis to the where each subspace k carries asymmetric Gaussian mixture (Equation 3. As before, we first state the key separation assumption and show that on each subspace, the individual Gaussian distributions in the mixture of Gaussian are highly separated from each other. Then, we simplify the Hessian and prove local convexity. Finally, we conclude linear convergence rate based on the strongly convex and smooth property.
Assumption 6.4. [Highly Separated Gaussian] Consider the Gaussian mixture
Th.t N (xx Pklt Et,)
(x) = Σ Th. (o: Ek.),
(x) =
Ene The N N(x; Ho,s, Ek.))
8

---

Published as a conference paper at ICLR 2026
There exist constants E < and 8 < 1 such that when pk have
Pr aopp
31 E ......,n] with (x) 2 1- 2 1
Justification. With MoLR-MoG modeling, after adding diffusion noise of scale тr. each point x remains within ((7e) of the subspace's moment-matched center jāg. Concretely, the subspace
structure (or a preliminary projection onto principal components) ensures || 1* l2l I A = Cx with high probability, for some moderate constant C. Hence, any third-order Taylor term & I| A | O which vanishes compared to the leading Hessian scale O In the following corollary. we further show the approximation effect of equivalent Gaussians.
Corollary 6.5. Assume that || Wk, ll2 S 1 UEgll2 S 6 and 1| - -1ill2 S A. We have
p(z) - log ((z) = O(c +CCC+5+++0 S
Remark 6.6 (Separated Gaussian Simplification). For simplicity of description, we assume the individual Gaussian distributions the mixture of Gaussians are highly separated. Actually, if there are ni Gaussians that are not separated from each other, we can employ clustering techniques to transform them into n& mutually independent Gaussian distributions. The error caused by such an operation can be calculated using corollary 6.5 The core intuition is that the modals should not have much influence on each other. Hence, we can also use the idea recursion to first cluster the general MoG into 2-modal MoG latent. Then, we can use the analysis of Section 6. with Theorem 6.1
Then, similar to the above section, we also calculate the Hessian matrix and show the local strong convex parameters. Finally, we provide the convergence guarantee for general MoLR-MoG modeling.
Lemma 6.7. (Eigenvalues of the Hessian] Assume Theorem 6.4 convex on a neighhorhood of fr If E R&&,, r (x) = or are strictly satisfied, we have
the Hessian at the k-th subspace is
TT. 18
Amin (H (Hp.ana))
(8] + 72)
and Amiu (Hu,, has the following form:
((G + ||U 11 |Unallallueella 8(UT, ||U,, 111.:111 /2.
Lemma 6.8. [Local Strong Convexity] Assume Theorem 6.4 in a neighhorhood of V2((0) = a'1, > 0, v0 O. If Rddy 31 (x,(() are strictly satisfied, a' min 22).
Ca.r where A min (a A2 min/=12,,....u Amin (Hu.,
Thus, even without symmetry, equivalent Gaussians and sufficient subspace separation recover the same local convexity and linear convergence guarantees in the asymmetric case. Similar to Theorem 6.3 under Theorem 6.4 we can obtain a convergence guarantee.
Remark 6.9 (Previous MoG Learning through Score Matching). [hahh et al. (2023) and Chen et al. (2024b) consider MoG data and analyze the optimization process of diffusion models the full space. However, these works aim to design specific algorithm to learn the MoG distribution instead of using standard optimization algorithm. On the contrary, by using the MoLR-MoG property to calculate the Hessian matrix, we adopt the GD algorithm and obtain the convergence guarantee.
Remark 6.10 (Initialization). Since the multi-modal GMM latent leads to highly non-convex landscape, Theorem 6.3 and the corresponding asymmetric variant require the initialization to be around 0* to guarantee local strong convexity and obtain a local convergence guarantee. As the MoLR-MoG is the first step to model the multi low-dimensional and multi-modal property, we leave the analysis of the global convergence guarantee as an interesting future work.
6.3 ANALYSIS WITHOUT HIGHILY SEPARATED CONDITION
In this part, we extend our analysis to latent MoG with overlap, which is closer to the real-world data. We define the pairwise overlap factor (1) between components i and at the k-th manifold
(x) 2 (r,(()))
9

---

Published as a conference paper at ICLR 2026
and the maximum expected overlap for the manifold as: Covertap max; EjfiBaan (z))
Without the high-separation assumption, our analysis proceeds in two steps. With the overlap factor Coverlaps we first examine the block-diagonal Hessian, deriving refined lower bound a. Second, we analyze the full Hessian by treating off-diagonal interference as perturbation bounded by the overlap factor. Applying Weyl's Inequality, we prove that the global matrix remains positive definite provided the perturbation (introduced by the overlap) is smaller than the effective diagonal curvature a, thus guaranteeing linear convergence.
Lemma 6.11 (Minimum Curvature for 2-Mode Mixture). Consider mixture of two Gaussian components. Let Coveriap = sup, T'k (z)rk (x) denote the maximum pointwise overlap factor. The minimum eigenvalue of the ideal Hessian matrix, denoted as 0r2-mode- bounded below by:
Ox2-mode
(1 4Coerrlpp) min (Aminl pansa ), Amin HU,.)))
and
Amia (H) a:2moode
C' Coverlaa 0,,
where C' is defined in D.1.3
Lemma 6.12 (Minimum Curvature for Multi-Modal). Let 'g, =Ei (1)z((=) represent the total probability mass leaking from the 1-th component due to overlap. The minimum eigenvalue of the biogo-diagonal Hessian, denoted as Middii-Modab is determined by the component with the minimum effective mass:
MMdlli-Modal
min 1e(
[[TE.]
sotal min (Amin (Amin (Huraaaa) Amin (iin(HHggau.)) Ck,l
and
Amin (H) Z OkMutiiMaial - C. Coveriap:
where Ö is defined in D.2.4 For the Hessian remain positive definite, the intrinsic weight of every cluster must exceed its total confusion with other clusters (i.e., Tus > Ex1 for all l).
total
CONCLUSION
In this work, we provide a mixture of low-rank mixture of Gaussian (MoLR-MoG) modeling for target data, which reflects the low-dimensional and multi-modal property of real-world data. Through the real-world experiments, we first show that the MoLR-MoG is a suitable modeling for the real- world data. Then, we analyze the estimation error and optimization process under the MoLR-MoG modeling and explain why diffusion models can achieve great performance with small training dataset and fast optimization process.
For the estimation error, we show that with the MoLR-MoG modeling, the estimation error is Rat E 1 > 1 nxda/ Vn, which means diffusion models can take fully use of the multi subspace, low-dimensional and multi-modal information escape the curse of dimensionality. For the optimization process, we conducted a detailed analysis of the score-matching loss landscape. By formulating the exact score in both symmetric and asymmetric mixture settings, we derived explicit expressions for the parameter Jacobians and identified the dominant components under standard separation assumptions. Then, we prove that the population loss becomes strongly convex in neighborhood of the ground truth score function, by estimating the Hessian and presenting lower bounds on both its minimal eigenvalue and the convergence rate. Then, we provide the local convergence guarantee for the score matching objective function, which explains the fast and stable training process of diffusion models.
Future work and limitation. Though we have extended the situation to multi-manifold MoG, how to extend the analysis to more general non-Gaussian sub-manifolds (e.g. heavy-tailed or multi-modal beyond second moments) by higher-order moment matching is still unknown. Meanwhile, we wish to design optimization algorithms or network architectures that explicitly leverage the block-diagonal Hessian structure for faster training. For example, can perform natural-gradien step separately in each block with a block-diagona Hessian with decomposed data, which will accelerate the optimization process.
10

---

Published as a conference paper at ICLR 2026
Ethics statement. Our work aims to deepen the understanding of the modeling of diffusion models and explain the success of diffusion models from a theoretical perspective. The MoLR Moo modeling has the potential to achieve a great performance with fewer parameters. Hence, this work can be viewed as an important step in understanding diffusion models, and the societal impact is similar to general generative models (Mirsky and Lee 2021).
Reproducibility statement. The detail and description of the real-world experiments are provided in Section E We detail the model, hyperparameters and data.
The corresponding author Shuai Li is supported by National Natural Science Foundation of China (925701 11) and Shanghai Key Laboratory of Intelligent Information Processing. Fudan University (Grant No. IIPL-2005-RDD1-04)
Acknowledgement.
REFERENCES
Valentin De Bortoli. Convergence of denoising diffusiin models under the manifold hypothesis. Truns. Mach. Learn. Res., 2022, 2022. URL https::/ppenreeeeer nette/ewweeeeeeeeeeeeeeeeeeeeeee
Bradley CA Brown, Anthony L. Caterini, Brendan Leigh Ross, Jesse C Cresswell, and Gabriel Loaiza-Ganem. Verifying the union of manifolds hypothesis for image data. In ICLR, 2023.
Stefano Bruno, Ying Zhang, Dong- Young Lim, Ömer Deniz Akyildiz, and Sotirios Sabanis. On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates. arXiv preprint arXiv:2311 1. 2023.
Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu. Video- dreamer: Customized multi-subject text-to-video generation with disen-mix finetuning. arXiv preprint arXiv:2311.000 2023a.
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410.10733 2024a.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models low-dimensional data. arXiv preprint arXiv:2302.07194 2023b.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning score: theory for diffussion models with minimal data assumptions. arXiv preprint 2rXiv:2209.11211 2022.
Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404./8880 2024b.
Frank Cole and Yulong Lu. Score-based generative models break the curse of dimensionality in learning family of sub- sub-gaussian probability distributions. arXiv preprint rrXi:22422.08082. 2024.
Hugo Cui and Lenka Zdeborová. High-dimensional asymptotics of denoising autoencoders. arXiv preprint arXiv:2305.I 1041, 2023.
Hugo Cui, Florent Krzakala, Fric Vanden- Fijeen-Fijnde, and Lenka Zdeborová. Analysis of learning a flow-based generative model from limited sample complexity. arXiv preprint 2:X0022310003555 2023.
Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Diffusson transformer for image editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2969-2977, 2025.
Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-fres guidance: A sharp statistical theory. arXiv preprint orXiv:2403. 1968, 2024.
Sixue Gong, Vishnu Naresh Boddeti, and Anil Jain. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3987-3999 2019.
Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models; An optimization perspective. arXiv preprint arXiv:2404.
240..4443.,, 14743, 2024.
11

---

Published as a conference paper at ICLR 2026
Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arXiv:2401.15600 2024.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Flect. Video diffusion models. arXiv preprint arXiv:2004.03558, 2022.
Jerry Yao-Chieh Hu, Weimin Wu, Yi-Chen Lee, Yu-Chao Huang, Minshuo Chen, and Han Liu. On statistical rates of conditional diffusion transformers: Approximation, estimation and minimax optimality, arXiv preprint arXiv:2411.17522 2024a.
Jerry Yao-Chich Hu, Weimin Wu, Zhuoru Li, Sophia Pi, Zhao Song, and Han Liu. On statistical rates and provably efficient criteria of latent diffusion transformers (dits). Advances in Neural Information Processing Systems, 37:31562-31625 2024b.
Hamid Kamkari, Brendan Ross, Rasa Hosseinzadeh, Jesse Cresswell, and Gabriel Loaiza-Ganem. A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. Advances in Neural Information Processing Systems, 37:38307- -333444473,4,.. 2024.
Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian, On the generalization propertics diffusion models. arXiv preprint arXiv:2311.0177 2023.
Xiao Li, Zekai Zhang, Xiang Li, Siyi Chen, Zhihui Zhu, Peng Wang, and Qing Qu. Understanding representation dynamics of diffusion models via low-dimensional modeling. arXiv preprint arXiv:2502.05743, 2025.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu. Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++; Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10072-10083, 2024.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li,, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4117-4125 2024a.
Yue Ma. Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Young Shum, Wei Liu, et al. Follow-your-liie:: Open-domain regional image animation via short prompts. 2024b.
arXiv preprint arXiv:2403.00
Song Mei and Yuchen Wu. Deep networks denoising algorithms: Sample-efficient learning of diffusion models in high- digeedimmnsional graphical models. arXiv preprint arXiv:2309.11420 2023.
Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey, ACM Computing Surveys (CSUR), 54(1):1-4 41, 2021.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303001861, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684 10695, 2022.
Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. arXiv preprint 1.XXv:2307.011788 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis. In European Conference on Computer Vision, pages 398-416 Springer, 2024.
Shuai Tan, Bill Gong, Bin Ji, and Ye Pan. Fixtalk: Taming identity leakage for high-quality talking head generation in extreme cases, arXiv preprint arXiv:2507.0139/ 2025.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neurul computation, 23(7): 1661-1674 2011.
Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409. 2024.
12

---

Published as a conference paper at ICLR 2026
Zexuan Yan, Yue Ma, Chang Zou, Wenteng Chen, Qifeng Chen, and Linfeng Zhang. Bedit: Rethinking the spatial and temporal redundancy for efficient image editing. arXiv preprint arXiv:2503. 10270, 2025.
Ruofeng Yang, Bo Jiang, Cheng Chen, Ruinan Jin, Baoxiang Wang, and Shuai Li. Few-shot diffusion models escape curse of dimensionality, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Ruofeng Yang, Zhijie Wang, Jiang, and Shuai Leveraging drift improve sample complexity of variance exploding diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Ruofeng Yang, Bo Jiang, and Shuai Li. The polynomial iteration complexity for variance exploding diffusion models: Elucidating sde and ode samplers. In The 28th International Conference on Artificial Intelligence and Statistics, 2025a.
Ruofeng Yang, Zhaoyu Zhu, Bo Jiang, Cheng Chen, and Shuai Li. Elucidating rectified flow with deter- ministic sampler: Polynomial discretization complexity for multi and one-step models. arXiv preprint arXiv:2508.0877 08735, 2025b.
Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. arXiv preprint arXiv:2307.07055. 2023.
Zekai Zhang, Xiao Li, Xiang Li, Lianghe Shi, Meng Wu, Molei Tao, and Qing Qu. Generalization of diffusion models arises with balanced representation space, arXiv preprint arXiv:25/2.2096 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A SCORE FUNCTION ERROR ESTIMATION
In this part, analyze the estimation error of diffusion models under the MoLR-MoG modeling and show why models can escape the curse of dimensionality under this modeling. start, we first calculate the closed-form of score function under MoLR-MoC modeling.
A.1 CALCULATE V log (a)) AND DECOMPOSITION
Consider the k-th subspace
nk
(c, = Σ
TN (Hu,) Ek.))
where Dx.1 = uLUR, + +2 We know that
Egll = 7/
1
I
UniUT!)
ot +
nkk
of (1- TK,NN 21,1)
(p)
18--2UUUl0 Unillh (x - mk.)),
2
o? ri
11
1=1
which indicates
Ei TR,N (Mn,,, Exsi
कोन (I-ZūAZUnA Un,Ui pkd
V)(,(())) (r,(()) 78
1
V (o)()
El Th (WN,NN( 2k.11)
Let
(z. t) = (s (xx t), $2 t)..... sx (x, t))
We want to learn the parameters of the score function for each subspace:
(x, t) V log Pt.A
where the parameters are (4t, : 12,). 1, 1,... K.
Define
((sx) [I| (x(,,))11 En(ss) 1 n n
Σ
|su(a, (2) - (x( (111.
We have the following decomposition:
[(3 - (n(.ko.)) = C( -(3,.0.))(s$$)
((s)) + ((s)) - (sx..)) + (n(sr)))) - Ln(3
Estimation
Approximation
optimization
We also obtain that
K C(s) c((s)
C(s)).
Since Estimation and Approximation reflect the fitting ability of the network, we analyze the first term first. Then, in the next section, we analyze the optimization dynamic.
A.2 ESTIMATION ERROR ANALYSIS
First, we show that of and loss function are Lipschitz. We will first prove that s3 is Lipschitz for Vk, then know that is Lipschitz.
14

---

Published as a conference paper at ICLR 2026
Lemma A.1. [Lipschitz Continuity] Let Ln and Lux be the Lipschitz constant 86- MoLR-MoG modeling and Theorem 5.1 there a constant
With
L L V EK 2 (T, I2e) O (
such that for any 0,0', [|,0(2,t) - so (x, 8)112 I L 110 8112, where Cr By) BB max(]uu,1122 For so and s", we have that 21180(0, 1) s² (a,t)): t)]l2 2(R BB ln-. kd
Proof. Since analyze the estimation error fixed time we ignore subscript for Ek,,,t Wk,ts WLk.t and 6k,1 and define by
ER.1 SUKKUK +
wi (ar) S
Ex.1),
wk. K TK Stplil Ek.i).
$t
(x)((2) s1Mk1
$t + UkK k, + s.pp.))
Assume that ||Ux,112 Bu, || Bus max For Eg,, we know that
BB,, By) C. and ||r||2 S R Var X.
Σx,== UK,U, 721 7/ Amin (Ex.)) 2 72 - |8 S
7t
To obtain the first L in this lemma, we need to bound
||,
and
Dok ((.0)) aU211
The bound o(..e Jpaz.1
For the latent score of the k-th subspace, we have that
sork 1=1
(2,
76
wk (x)
(2,
1
dx.1 + 06c,1(w) Oun.
Ouk. Wk.1 (x))(())
@w)(w) Oua.i
R2 Wke, ( (1x,,.)))
Opticl
7/
w
(x(x)
(x) +
Opiia Wk. (x))
Blml(a)) (Eir WK.
Opse.1
Ouki
8pinl
2
7
78
wx(
+
w (x)
To bound this term, we separately show that (1) has lower bound.
@wn,((x)
(0),(())
(2)w,,(x)) (x),
have upper bounds.
Bwg (at)
(x))
DE
05 (
(x))(
(w), SInE 1 (1), dps.
(3)
Jun,
have upper bounds.
Wk
wk (x:)
w((xx)
(1) We (2) has a lower bound.
(x)x) Tr,N Stf4,,1, Ek.)), which continuous.
Since continuous function has maximum and minimum in closed internal and ||>|||2 S R, assume that wx(x)= 2 mw. And for any x, (x(x) > 0, so m. > 0 holds.
(2) (x)) (x))
(6x)(z) (wx)(e) have upper bounds. Bisk
Oyk
15

---

Published as a conference paper at ICLR 2026
We already know that continuous function has maximum and minimum in closed internal and ||>||| I R. Thus, we can assume that (x(x)) I Mwn. We also have that
(x(x)) & Mw S = Tx, [xx,1
For the second term, we have that
(x) 1 YE
Un s(pta.)
( UniUks (x s(pLe)),
ot of + 7f
whose L norm bounded by
(
S || sepualll ||0|12 + 11s. S R s. st +7 Un.Uks (x supt.))
2
Then, for the third term, we know that
(6,,(()) -t + of
of UK = -8 (1-
Un.UUK!
88 + ni
For the last term, we have we have the following expression @wxa(a)
N( Ix.)) sepR.1) For term | (x sep4,,)) 112, have that
12 81/1,)))22 - 11 semu,1l22 l || 22 5 -
(R + ||5m)))122)
78
which indicates
(
< 8 (2; Ex,)) - (R + [1s.)a.112)) s (o) Ex.) En,)) (R (R+ 8: B,)
Öpuka
Du SwN St Ex,i) ? (R ss By).
2
(wy(=) (2,)(x) DPL,
26 ((ا) 1Pe.1 (x))
(ww(z) sith (2)6 Bme.l
2
(3)
have upper bounds.
(x)
(1)
For the first two term,
dwa (x) Oua.
(x)(())
St (R + Bp
5
Wk
2
and
(xx)
mik 06..(z)
0uni wk (x)
Constant
< St.
S St.
apx.1
wx(x)
For the third term, we know that
(mn(() ya
Ope.t L
(x.1(r)
2 (xx
# (R + st B,,)
22
<
(R + s,
w (x)
2
Combined with the above three, we obtain the bound for
2
Osk 6 (x, t)
m
(w))(x) 8uk.1
28.,(x) Opui
)on.(())
Own Tpe.1
(> Wk. (x,((x)6,,())))
+
Opci1
wx(xx)
w
2
z
2
R - sr BB,
+ 81 +
-
S1 (R + 8 B,)
=0
[s.(R + st st
?
16

---

Published as a conference paper at ICLR 2026
The bound we know that Ōsk,o OUk,
Ose o(z.)) DUn,
Now we compute the part about Un Through some simple algebra,
1 2=
M
(l)) (x. + B0L,
(frr) OUE, Wk wk, (x))xx))))
(wg(()
J0x
(> WK, ( (2)0,,
(x(x))
V/
Then, have the following inequality
0
1
En
òR,1 (2) + (6c,((
JUL, wk. (1),(()))x(()))
Dee Ow)(a)) a0i,
1 (2..
OUk. 7
(z(x))
Osk. (sk,o(z,t)
1
D
Own )0,( +
BUn.
dbm,(w) oUen Wke (x))
Btad (= (
+
OUK.
72
wx(x)
(x)
2
Osx (xx.t) Ojan.
Similar with
we need to provide:
(I) The upper bound of
Own.s JUk.
Dbn. and OUk,2
(2) The upper bound of M-
owk ((0)
ouk (a))+ 05p,(()) (x))(a))
002,1
and
aw (n) Juk
(x)
wf(x)
(I) The upper bound of
dure.d D04.
and
06 802,7
For the first term, we have the following form
Owk. ON (2 Ex,)
Tk.
aUk.1
OUk
- (Nx 8H14. Ex.i (x sxpua)) s. - - ET,))Uka
Then, we know that
Owad 00,,1 12
< (1; De R sell
mo
7t
71
(R + st Bp)z
S S(E Es
+
For the second term, we have that
061,(())
87 (U - S.pk,))1 + Un,(x sepke.)
2
0UK,
86
+ 71
which indicates
00).((x)
o
< 2 +
+ ||..))11122 S 2(R (|0.)e,1112)
OUki
1
< ( + s B,B,))
bc, (z)(
26 (m)
Dul we (x))
Owp (2 w2, (2)6,,) B0g
(2) The upper bound of
and
(x)
(x(
m
((m))(a) BUne
(2) +
05c,(m))
ate,
(x)(
(R +8+au
B,)3
st
+ + 2(R st
wx(x)
7t
%
1
We also have
Own (xr)
(y Wk., (Z))..
(R + St B,B, )
1
(R s,
w (z)
7
?
17

---

Published as a conference paper at ICLR 2026
(s,,a(x,t) U
(R (R+ 8y B,)?
1
(RR (R++ B,) {
S s/
+
+
+
+
+
+ 7
(R + st Bp)
7/2
7t
7
R + st B )
=0
Therefore, $0,6 is Lg- lipshiz, where
(
2 8 Br )
LA S
mallies + Lind -0 0
78
Furthermore, we know that
K [|s0(x)) s0(B)]12= Σ
K
k
(p,((=(())
L (z (1) - -y(00 5
L 1 y112.
==1
Thus,
k L
Σ (R + B,B,)
L3 = 0
After obtaining the Lipschitz constant for sg. we bound the gap between so and s*:
1 D TR, (1; 1 U + +nil) (s seMial उ, b
o7 U
URT (sc SAPPEL))
VlogPr.(<>)
M TK, Se/kE11 S. UUU + +7²1)
With the following bound
27 of + 1
URU (x: supeie
R + s By
we have that
1 (R
Voog <
B,B,) (R+sBB,) and [|s,,(=)]l2 S 2/ (R Bp
7/
7t
which indicates
[10),(()) logg) z (R + Bp).
Hence, we obtain that
L $ 211)),0(x) =( s,B)
Lemma A.2. [Rademacher Complexity] Let F = (E( 0 E }} and suppose θ has diameter
Re. Then the empirical Rademacher complexity satisfies
Rr(FF duv :)
Proof. Let function class be F
(se(z) 0 (llpa, UK θ) with uau
Rd UKA E Idd. We know that the number of parameters
p k (d + d) 2 ngde.
And the covering number of the parameter space is
1(, 112) S ( P
18

---

Published as a conference paper at ICLR 2026
If is L-lipschitz, we know that
v0, 30,, s.t.110 - 0,112 s e
v01,02 02 E θ life fea (e) - L||01 - 02ll2
and
L
+ l - So, [k<(p)) & 011 - 0jll2 e.
Thus, assume that 110, 0z 122 S C₁ for any 0,.0g E 0, N O, -l 112) v
C1 P
CiL P
N [ I
9, (1a) <
c
N (c, (1(a())) s SN (i O, - = (GL))) "
We also know that diam(()
L ((a
C1L. with Dudley integral, we have
12
(F
(R.(F) 2
log FF, 1lz2)))
Vn
J
12
Cih
CiL V
<
de
vn
12
TD PCLvtexp(-t)dt-
GL= GL- B. vn
<
Vn Jo
We take the squared loss function.
(C.LLLL p
R,(()) (F)
Theorem 5.3. Denote by En(0) the empirical loss n i.i.d. sumples and by L(0) its population counterpart. Then there exist constants Ci, C2 such that with probability at least 1 for all 0 O
|C(0) - (o) s 0 (R + 8t BB)
KK 1=
n
/ i
nxdk
+ C2 log(1/0)
7(
where Ci = max 10; 8 [l,, C2 a log 2, σ² = sup
leB
6EB
Proof.
Using Lemma
A.2
since
(F) OLGLLL P)
We have
A = sup - L(0) = L 9ee
m
Thus, by taking the expectation on both sides, we have:
A = O(CYLIL P.
n
By Bernstein inequality.let 0² = sup [(X: know that 8E6
ne²
nE²
Pr(sup]((0) L(0) E|A + e 2 2exp( 050
S exxp(
2( L,LCIe/3)
302
Let 2 eexp(-10 30d
n 6, we can obtain that
Pr(sup L(0) L(0) 2 CLLL 828
P +C2
log(1/0)
7
19

---

Published as a conference paper at ICLR 2026
A.3 APPROXIMATION
Since our network can represent V log p(r) strictly, we have
Approximation Error
B 2-MODE MoG OPTIMIZATION
In this section, we analyze the optimization process when the latent distribution at each subspace is 2-mode MoG. In the next part, we prove the extension to multi-mode MoG and show how to remove the highly separated Gassuian assumption.
ULU (e-Mape) ! M02 T+) x- of
ahd
Pp.. (x) (z,
1
+ +121) att UN (atsrpe)
VlogPe,s())
" SH,. +1 + Nax s U +1f1)
which can be reduced to
1 N sipk Ex) (x(x) + Ma StNk: Ex} (x(2)
V VlogPe,(2))
=
[1; Σx) + _ (z; 8.11R, Ex)
(5)
where (x(x) x 8ellk
URUET (x-8,112), and 8 (x) 8
at a URUE (x++)))))
B.1 OPTIMIZATION
Assumption B.1. [Separation within a cluster] Within each cluster k. the two symmetric peaks are well separated in the sense that [[ (-s( > Aintra, for some Aintra >> Tts Consequently, if a sample æ is drawn from the "_" peak then its responsibility under the "-" "-" peak satisfies
N(c; 8.14 Eg)
rh (x)
Oi- (1e-Amma/(221) < 1,
2 Mxx; s mg) + Mo; 88 E.)
and symmetrically of (z) << 1 when x is drawn from the "_" ppeak.
In the following discussion, we assume that x E k-th manifold, which means that wx(x)= 0 + Lemma B.2. [Jacobian Simplification] Under Assumption 6.1 in neighborhood of 0" the first derivatives simplify to their "self-cluster" terms: J (x) Sp s0 = st(I aPt)/ and
2s ( (2 + 7)
(r (1) (=tsspn)) 2n (U sstpee
Proof.
J
1
( (de (x) 0uk
& (x)+
(x)+
06f ane
mx (x)+
depiz) buz ux (ar) (o))
owx(s) x ong ( (o
one
w wx(xx) (x))
7t
W (x) 86 (x)
+ wx (z)
(e((z) Bua
ow, (2) (x() +
Sw (4)
Jue ex (x)
Ops
Ohe
72 (x)
2/2 (2)x(x)
Term
(w))(w) (w (2)))(x) + wt en (x))
Busk
78 F2
Term
We will now prove that term B can be ignored compared to term A under assumptions.
20

---

Published as a conference paper at ICLR 2026
For term B, we have
@w)(())
(w (x))%((2) + (x)(
Buz (1) (x) +
out()) cx(2))
Bp4
Ope
(2)((()
78 (2)
1
Owx
(w0 (z)) + wx (x))x(x)) - wx(xx( Oug (2)) (x()) + Bu cx(x)))
(22((())
0pk
0pk
duu
1
Owf (a)
oul (z)
w (x)) wg (x) owly (z) 8%. (x) -wi (x](x)
aw (a)
Opp w (2) (z) +
0pu Ex (1))
7/ (w)
Ouk
0pk
1
Ow Wk - Owk w (E<(x) - (z))
(x))(())
Ouk
2
Owk wik
Owh wit)
(1+
UnUx
Vi w (x)
apm
ot +
8t/lk
b7 W w 2z x (
UnUā {
ot
(
sellun lu ll 112l1//122 lo).
ue = 0
st
(
87 + 7
7t
And for term A, we have
"k (x)
(w) BpC
+ w (x)
(ex(()
( 82 (selluxel l12
Bue
=0
m w//
7 nuux ukkk (x)
7/
Thus,
re o
84
|a ///a|1/11z2 [12l2211222
x r wn wxll=rlaa
r wol wallll))) O. 0
7
0
Me 72
wK
?? Irt - -rk
77
Thus, J 20 1 (rx (x)
(x) Byn
+ r (x)
Den (en((l)) Bus
= - (r+ r (x)) (1 (I- अل -2-2/
We will analyze J now. Recall that
1
Ow, (2) (x)+ mi
(0) D0g oF (x)+ oix wt (
(2)
co (x))) - (wf (w)) (o))
7
wf(x))
1
(w)
00s w (x))
a0, w (2r)
7
wx(xx)
DuT (4)
80x 0% (x()))
(w)()) a0 (2)
(w)
(w (x)))(()) w ex ex(xx)
aU.
wx(xx)
w[(x)
By calculating, we have
Ow, (a)
(x(x)) + ow (e)
o0e (x(
(x) ab4 (w (2)))(()) + w (x)))
00k
wk (x(
wx(x))
Owo (x)
(x((x) + Ow (I)
@wx(x))
OUR en (x))
(wi (x) + (x)))))
(x)
OUk
OUk
dwā (I) +x)x+
+ (w)
@wx((z)
(m)((x))
(x(x))
(x)()))(())+ (x)(
wx(z)
0Uk
OUk
auk
Dut w oU,
dwh
w (ex()) - -x(())
(x)
OUr
2
NN(x; s.pk E) M (z) Us (I a ULU ) m
wx(x)
2 0 (xtr% 72 [s) + +17
21

---

Published as a conference paper at ICLR 2026
where (x) E- sepa)( s(mr) Σ- (x)(x) E-'(x + sepx)
- -E-1,a a u7+"?
We also know that
Σ k=1
(x) 808
Wk (x)) + OUR Wk (x)
(ee( +
8 5
$x 11 ll2
0
0
JR k=1 Wk (z)
$/ + 72
$f + 7
+ 0.
All 12
61 41+41
Thus,
J
2s 78 (58 Ht
(n (U) (x s(pu)) (x s(mm)UTE + (2) (z s.pu)) + Un(a s(px)
Before we provide the simplification of Hessian, first prove that for a, 6 E R" M = a bln ba" ..M positive-definite if and only if 6 a of O. At the same time, we provide the minimum eigenvalue of MM which will be used later.
Lemma B.3. Let a, b E Rn and M = bln + Ta MM is positive positive-definite if and only if 6 # O.
Moreover,
(MM Ami = =22 b)a + ||a 1 |a/|2 blle 8(a
6)2
a 6
Amin
2
Proof. Let M = a. bla + bu TT -= Tb. We know that Va E RD
x TM (M 2) (M x)
= 1IM Tx 2
Thus, MM is semi positive definite.
We can also have that
M |a bl, + ba T| cnI -ai" T 2c 2
where if and only if TT 0.
The last equation holds because
In + we _ 1 TT
Thus, |MM | 0, A is positive definite.
We can further get the eigenvalues of M. Expanding gives the convenient representation
MM (a 6)) In + aa 6(baa +ab +a TTab" abb
(6)
Vr E R2, Tf a= 0 and x bb = O, we have:
MM x: (a. TaaTby' 6)2x..
Thus, (a T)2 is an eigenvalue of M, and its eigenspace contains the orthogonal complement of span(a, b).lff.f.f if a and b are linearly independent then dim(span(a,, b}) = 2, so the multiplicity of the eigenvalue a is least 2.
22

---

Published as a conference paper at ICLR 2026
To find the remaining eigenvalues restrict M to the subspace S span(a, b). Assume first that a and b are linearly independent so that S is two-dimensional.
Using equation 6 we compute tr (M(MM) ) which is
tr(MM)) Ir((a b) + a b(baab ab ab") a abbb)
= n1aa 6)2 2(a 6)2 + ||112161122
= (n 2) ()2 + 1aal2/1b112
The second equation holds because of tr ) tr(y x) yT.
We set the other two eigenvalues are fa1 and μ22- Thus
(rMM E = (n - (a b) + μn + μ2 (n + +a(2)aaT 1)2 + ||a|211612
and
MM 1 = (a b) b))((=2)) 2) N1lt2 = -4aaa b)2m
So μr and μ1 are the two solutions
(4(a (T) - |a12118133 4(a T0) O.
Solving equation] 7 we have
4(a + + | 8(a 2)2 1|
M1 H2
2
Now we obtain all eigenvalues. we can calculate the minimum of eigenvalues.
Amin (MMT) p22 4(a + || |a |6/2/1612//800 6)2 1
2
Lemma B.4. [Eigenvauess of the Hessian blocks] Under the same conditions, H is convex. If Vr E Rdd t (x) or T (x) are strictly satisfied, the eigenvalues of the Hessian at 6 are
ot
Amin (pppp)) = (s² + 72)2
and
4(U% px)) 1U |U
S(U px))² + |0 P 11
Amin (HU,U,)
2
Proof. We first state the convexity the loss function near the true value 6*
Let f = 0 + Ad, we have
so(x,t) so (x, (Voso(2; + 0(JA0)1))
Thus,
L(0)- = (s0((2)) ( (sa(x, - V (z)
Ex-m(t (10- (2, ())
0(1A0))) v (x))
(so- (z. 1) + (Vos t)) ( | 0(1 [oo))(
TFen (2,)))))=)))))))) [A0)) [|(
(AA)T HA0.
23

---

Published as a conference paper at ICLR 2026
ØL(0)
2H.
O02
We then analyze the convexity of (x (Vosa( 10 ) (Voso 1)0 - H. We can divide into 4 parts:Hup pp: HoU, HA and Hups where HU (H₂v)²
Let Jh
Sps 0-
H = (e 1) (Voso ()oo))
= (2, (2, (x, t)
Term pp
We will show that H pape is a-convex, where > O.
Hazps (z) [
s (
st
Hppre R (o
7t
+
Let Pk ULU a = 17t37'
of
(I =[I 2aP + a (I aPx)²
77 ( )
We then prove that Amiu Amin((I - First, we calculate the eigenvalue of P.
p2 P \ A2
Then we take subspace Col(P) (8 iu Ker(P) = {u ; Pv = 0, E RD corresponding to X2.
Pz, x Rn} corresponding to Å1. and subspace
E Col(P), Pw w:
(I- 1-aPPw (1 a)w,
and
(I aP-aP (1 aw,
Thus,
X1 = - a1- Similar to the previous derivation, if w E Ker(P), Pw = 0:
(I a ylu- (I a2 2 w
Thus,
X/2 = 1.
Therefore, Amiu ((I (ppx)2)
Hence, we have
2
Xunn (Hpere) 2
(5) 2)
9t.
where ce (x (x))) 1.
24

---

Published as a conference paper at ICLR 2026
Term HUeU&
HUGU = Bz-peta) J
4st
(0) + 12) E Bampi (x + s.ps) + (z s ) (a + scpurl + (x + su ]
488
(s) + ??)² U pare URI
URI + si UR U + UUPN) + U&U UNM M(x),)
where (1) is semi-positive for (x) 0.
Using lemma
B.3 we can take Us and = pik. and obtain that
HUgU, is positive definite and
Amin (HinUHUU 4(U% 2x)) |
11 |1
S(UT pk))² + |1xx11 pm 113
2
Term Hp and Term HUmmE
Since HUPpE Haau. we just analyze Hu We want to analyze the Hessian block
Hurrz = Ez-p. (x) (x))
and show that under symmetric assumptions, this cross-term is zero.
The first-order derivative with respect to μn is approximately:
)
(+)) = 7(
8 sf + Y
The first-order derivative with respect to Us approximately:
(6)(z)
de((Zr OUk
(x) ~
rk (x)
+ + (2)
2
OU
with
(62(x)
$
dex(x)
2t
2
Un + st1R),
+spuk)
-2
Uslo sepa).
OUk
5t 7t
OUk
87
71
combining terms:
JU (x) - C. Un [r (z) s.pe) + rtt (x)( - sepp)]]
where C'
2 77 +12)
Assume that the underlying component distribution (x(x) is symmetrie:
(=(x)= p=(=)),
and the weights satisfy:
(-2) = (z),
(=--)==r) (x).
Then have:
(a) Jh (r) is an odd function:
Jk - -
-1 (rā (z) -Q4))))
8t
71
-x)
25

---

Published as a conference paper at ICLR 2026
(b) JL is an odd function:
(-xx CU (- sppn + + (-x)(->x spmmal
CU [ + (x) - Semua]
= -CUU +
Now compute:
Hu = f (J) (x(x) dr.
Using symmetry:
= 1 (-x) pa(-x) dx (-1
px(x) do Huuue
Thus,
287
of UnUh)
Hp = Exmpanca (T = Exepaace (5 +
(1 (x) - (x))
$7 + 7
(U) (x + s(px)1 I + Un(x + so/ts) + (U (x - S(HA) + U2 (x - semia) s()()
Exrpaata [I T [ λHOU Eppesara [(u T ,
AH., =
AHa Exmpasto [u J T JBM] S v AAHa
Analyze H
We have
Hperk HyuUk
H
Higu HUgu
If prove that Ha for Schur's Theorem.
HUm H U2u. Hom is positive-defiiite, then H is positive- definite
We know that
122 He AHOzUe = - )2
An 2 As 2 AHapp
2 (1 22)
>
0,
XHu
12
2)
u. T HTpuu
r
max |=|=||]|===|||| Vu H Harag u- o HU 0)
< 1
where 1 if and only if u J/ -co J.. + O, which is almost impossible to happen. More specially, if we assume that Vx E Rd .rt 1 or r 1, since
25
of URU
HpuU2 = Exmpanea (J(( Exupaara (s) + 7
(r (z) r (x))
of + 2
(T) (I + + Un(x + sc14k) + + (U) (x s(HA) + UK(x - -ss44) Till
2s²
_ UnU
=EvuNiaunn Mn)l
(1 (x) - (z)) 1
72 + ++)
(T) (I + sepu)] + Un(x 8.pb)) J) x (1) (x - s. Un (2 sepuk)
20²
1
UnUi + (U sppnaa1 + U2(2 selus) T))
7/ (0? + 7?)
3
0,
26

---

Published as a conference paper at ICLR 2026
We have O,
4(U px))² || 12
|Uullalleu 8(Uī / ux))² + ||UR 13 1s
of (s) +2)2²
a: = minf
2
Utill now, We have shown that H is a-convex and L-lipschiz, where (1 r² M And can know that L(0) is exponentially convergent.
Theorem B.5. If we take T)t =] TTI and K then
110 11 I
[|000] 1-112-
C MULTI-MODE MoG OPTIMIZATION
In this section, analyze the convergence guarantee of multi- MoG latent with highly separated Gaussain assumption. For the k-th subspace, we have that
(x)
VlogPe.(F)) = (t,)(1))
The 2E TR,N (1; & + +721) (s 81/k1 7,. Uk,UUUU ( -8am)a)
E FK (1; S U2l +)
C.I OPTIMIZATION
Assumption C.1. [Highly Separated Gaussian] Consider the Gaussian mixture
ni Te. (o: Ex,),
The (a; ph,is Ex.1)
Pc(x)=
[x,(())
Σ Th. (2: Ek,))
There exist constants & &< and < 1 such that when o ~ Pk we have
(al (1, (1...., n with (x) 2 1 - > 2 Pr pcpp
We assume that the gap between the subspaces is large, and the gap within the subspace is relatively small, and the equivalent Gaussian is used to replace the whole subspace.
Corollary C.2. Assume that ||mk Mk,lll S 6, | - Uk,lla S and ||x Pilla S A. We have
|| plr)(())) log p(z) O(c 11c+55+H0 A
Proof. For k-th subspace, wx(x)) TH. Seftk,1 Ek,i), we take (x(x) Nx; Tik Ex)
where
Eng [xx] = pbx = Eusy [x]] 2 18/15
Covin (x) = Cov (x) = Bi(x (x - (1)) |= En T -.. + & (.
Σg Enh (ER, + ST 5² (.P.).
We next show the order of the estimation under the condition that || Mk,ll2 I 8, |[ Ukilla and || ji la < A. Using Taylor's Theorem and take 2o A... can obtain that
1
log px)) log + (ar mn) V log
2 (xr xo) T T logp(zo) xa) O r. _r0l3)) (l3)
1
(xx) log (( + (xr _o) T V (og
+ (x
xo)
T log =x0) O(I xo roll3).
2
27

---

Published as a conference paper at ICLR 2026
We analyzed the results of their subtraction item by item. First,
D (R,:N(ao) pu..s En.))
logp(ro) - log ((r) = log
Nazi PAk: 2ix)
(1 -uee)) T "E - mmnll) ) + log [Ex]
= log
SP Th.
exp( 2
Σx,1l
log
Me 1=1 Thi
(1+0(69))
+
log EEs
Ek,
2
log
m T
+ +0(89)
Ex,1
Σx 7
0
you
1=1 Th.
-1
+ 0
|5 1
and
logo(r)) log(((zo)1: =
For the first derivative, we also have
v Poggp(a - V loo p(ro) = V log Eil
Th, Th (N Mkll Ek,1)
D I Tx (Noi 1kdd
-kx,)))
p(zo)
|| loggp(m)) V Fogg)(n))1== O())
For the second derivative,
V2p(zo)
Vp((a) ) vp((o))
(2p(2o)
2 logg(<o) - 2 (oz0(
p(ro)
p(ro)
p(
p(x0)
V'p(xo)
v'p(zo)) - [V(2o))
Vp(roo )T.
p(ro)
p(xo)
p(ro)
p(zo)
log 2 = +0).
Thus, [lgg((>) - logū(x))la = +
Lemma C.3. [Eigenvalues of the Hessian] Assume Theorem 6.4
the Hessian at the k-th subspace is
convex on a neighborhood of 6* If E TRds, rt (x) or are strictly satisfied, we have
TR
Amin (H (HuLio.))
[s} + 7
and Amin (HUe, has the following form:
(TLAA(UT + [ 1 - || /(U ((U)),))))))) w + | /2.
Proof. According the previous conclusion, we only need caleulate Jn and Jr. With these assumptions and simplifications, similar to the symmetry case, will prove that Jh, and Jú have
28

---

Published as a conference paper at ICLR 2026
dominant terms.
(2)
7/
2=11
(x,((z)
066,1(0)
Opeit wk. (x)) wx(x))
(ax((a)
sine (k.(02)0
Opane
Opn.t
78
W (x)
Dn a (2))())
Mn (6),)(a)
Opke (.,((x)
(rwn(ar)
2/= WR, (2r,
21=1
Ouka
78
Wkk (ar)
wx((r)
wf(xr)
Let's go ahead and do the calculation.
Ow),, (1x) (x)
(are(ar)
1 (x)6(x))
((2)
Opa.
Open
Ope.a
(6x.(2)- - ox(x))
(x(x)
(0r)
w (2r)
1== 8pa.
(d),((=)
(x,((x))
88
Di (2)
(1-
of
UniUk!)
wk(a1
78
2
ot 77
where (x,1(x) TL. (x: Therefore, obtain that
=n= N (,)
Re ((x),(()) 1=1 Open
S (x)
Omn,(z) ya Opeit -==1 (Wk, (I) 8k, (x))
-0(5(R+ s1B,) ?
Wk (z)
w (x)
((())
wkid (x)
1=1 Opai
= =0 =((s)).
wk (I)
where 8 < |1,
1kkll22 < 1.
Thus, we have
[h(
Oso
rx))(())
I
s?
UnaUks
Opk,
78
of + 7
We know that
H Hps.we. Exam (1) J [1)
2
2
of $t + Un.Uk,)
[[rR((x)7] (T
ot + +
For given x, since we focus on the equivalent Gaussian distribution for each cluster, we have Ueue diag Hp..m... M. EE 14rk.nmy4pp..... Me
We first show that H is positive-definite then we will further show that H Hpepe is dositive-definit.
For H H H we know that
Amia (Hee.ee.) Ck. (,, (J,,))
Ck..Amia ((I aPb) Ca.7
(s] + 22)2
29

---

Published as a conference paper at ICLR 2026
where
st Elrxal *h, Ck,l
2t
9t
78.
We know that for a block matrix A = diag( A1, A2....... AA).
X(A = -1
Therefore.
Amin H mmmm))
min
1=1 me (s) + ??)
Thus, we take
Ck,ze 7t
AHaguz
(5] + 72)2
Similar to previous situation, since
"1=1 dukd wn. w.. (=))(())))))
(6g,
(
(m) yru wA, (z)
1 - 0.
86g (e) Sin
(x)
wx(#))
we can obtain that
En (ww,)(0) (x) + (x)((2) 012,1
(1) )ux(x)=
(u))
) (k,((r)6.1(z))
(x)
aUv,
7/
wf(x)
d6x,(x)
-I=1 wK,
(k(r)
(x) [Ueilæ - (MA.) + (x m), URA1
70 of +
And
HURU2 = diag Ue. HU Elrb.n HURin Unim )
where
HU =
=BI( 2 (U+,(z - ( (k.)U + +U MM,)) mx.1) T)
+ E[( a ) (U (
+ (UKilE (k.1)))).
Similar to calculation in B.4
B.3 to calculate the minimum eigenvalue of HU
UU, is positive definite and
4(U A(UE,PR.))) ||U,,
|U |l ((U)) || 11. 12 mr.
Amin Cuin(Huu.se..
2
Recall that
HUGUE x dia, Eir Hu, Eir Hunnn Unime
and Elr Thi obtain the minimum eigenvalue of HUGUE. which is
((U)),H.,))² + ||U,,llall (SU (( + |1 p4,
min
1=1,2....mm
2
30

---

Published as a conference paper at ICLR 2026
Lemma C.A. [Local Strong Convexity Assume Theorem 6.4 in neighborhood of V2 > a'I,a' > 0, v θ Vf E Rd,, 31 E [nx], (k,(2)) = are strictly satisfied, = min 22).
where miny me (0++)]]= A2 minl 1.2..... Amin (.,))
Proof.
Hp = diag( uu Un, H Huppuups..... H,, Hpe Unime)
|H S VIllmaelll |Huuuull = O 79 (5) + 72)2
H [diag( H pe 1a. Han Hathin diag( H Un H Humme Unime? diag( H Ue,
H Hp Ukioy
diag Ue. HU Hu Ux
Let
S Han Huu Hou
have
22 AHuuR AHogUe (1 -r -r2))2
s
AH 2 As AHaupk
2 21 (1-
>
XHu
(st
72)
T H H
r max |u|==1|||==]| 111 Vu H Hunnn HUWU. 1
1 if and only u T J cv Jr. c + O, which almost impossible to happen.
More specifically, if we assume that Vr E RRde, E1 [mx], (x, 1, we have
H Un, Errpp (T)(() (JE,
22
Exnpe (x,(())(((x Uk.)) + (o - (x,) Unalt) (I-
Un
5t + 7t 7/
of +
$² BENRE,NNe (ra(())(((E UKI + (x pn,) Un
(1-
of + 76
7
The second equation holds because Vz, if x of Na Ek.)), ra,((x) = 0. And the third equation holds because if x NN... (Hr,,, Ex,)) Const C,
Exony NN. Hk.1)] = O.
Thus, let a' be the minimum eigenvalue of H.
a' min (i1, A2h
(8)
where
Ct.17 A min
1=1....
(+] + 7/
and
4UU A(UT.I4..1) + 111x,,11 |Uu..llellaa. B(U + || 1
Xa min
Tkd
1=1.2...
31

---

Published as a conference paper at ICLR 2026
D EXTENSION TO MoG LATENT WITHOUT SEPARATION ASSUMPTION
D.1 2-MODE ANALYSIS
In this section, we relax the high separation assumption (where r (z)r (a) 0). Instead, we treat the overlap between manifold components as bounded perturbation to the ideal system. We aim to prove that the Hessian remains positive definite provided the overlap factor is sufficiently small.
D.1.1 DEFINITION OF OVERLAP FACTOR
We define the pointwise overlap factor &A (x) as the product of the assignment probabilities for the positive and negative components of the k-thh manifold:
(x) - (x)r (z).
Since rx((( (x), rg (x) E [0, and (x) + r (x) = 1, the overlap factor naturally bounded:
0 s & ()) 1 0.25.
We denote the maximum expected overlap magnitude as Coverlap-
Coverlap
sup Esppp(pe)
(x(r)
(10)
D.1.2 JACOBIAN ANALYSIS
We revisit the derivation of the Jacobian Jr In the original derivation, J decomposed into Term A (dominant term) and Term B (previously ignored):
J (x)
-Jlbaa (x) +
Term
Term
When Ex(x) - 0, we can recover the ideal Jacobian derived previously:
31 (r ri (1
sf UnUK
Ideal (z)
of 7
Term B contains the cross-product of weights, which is exactly our overlap factor &u (x). Specifically:
4s² (x(x) En
E²(x)= = 7/ m/we (x((()
I+ of
UnU
pk.
We can bound the norm of this term. Since terms like within the support, there exists constant Cj such that:
(a] and projection matrices are bounded
(*) S C1 (x).
(11)
Similarly, for the Jacobian with respect to Up. we can decompose it into an ideal part and an error part proportional to the overlap:
(x) = J.i (x) E (z), where | (x)]lr S C2 (x)
D.1.3 HESSIAN ANALYSTS
The Hessian matrix H is defined the expected outer product of the Jacobians:
H F (e)))
32

---

Published as a conference paper at ICLR 2026
Let (x) Jideal (2) + Exx). Substituting this into the Hessian definition:
H- E [[itteal + (Jidell + E
E (dccall-eeel + E'Jaccal + E. + EE
Hita
ATl
Here, Hateal the Hessian matrix under the high separation assumption and AH is the perturbation matrix induced by the overlap.
From the previous proof, we established that Hated is block-diagonal (or has negligible off-diagonals due to symmetry) and positive definite. Let o >> 0 be its minimum eigenvalue:
Amin (Hideal) (I (x) - N (x))² min (Amin (Haape))) Amin (HUU
[[ 4fx (x))) min (Amin (H prpse) ) Amia (Hu,))) (1 4 min (Amin (Amin(Hpana (Hpann) Amin (HU) & a.
We apply the Triangle Inequality and
inequality to bound the spectral norm of AH:
||H||2 < 2 B B Jideal E' 1ll |EELEE" 1|
s 2 Ell Jidcal 1 E + E|[
Since | (x)]l
S C1 (x) and | (x)l S C2 (x)), the perturbation norm is dominated by the
overlap factor:
(x) = ( E (z), where | (x)lle C2 (z).
The Hessian perturbation matrix is given by AH E[Jacat + EJ To bound its spectral norm |A (2, we define the signal bounds
8t Sp - sup Jut (a))ll2
and
Su A sup M1 RR 7?
We define the composite perturbation constant C' as:
C' = 2(S, (u)
And thus,
||H||| S C'
Eoverlap
1.1.4 POSITIVE DEFINITENESS VIA WEYL'S INEQUALITY
We now use Matrix Perturbation Theory prove the convexity of the actual loss landscape. With Weyl's Inequality for Hermitian Matrices, we have: Let H Hideal + AH. The eigenvalues of H are bounded by:
Amin Amin(H) 2 Amin (Hlidea) - ||
(12)
Substituting our bounds:
Amin (H) 2 a C¹
Coverlap-
(13)
33

---

Published as a conference paper at ICLR 2026
Condition for Convexity: For the Hessian H to remain positive definite (ensuring strong convexity), we require:
C'
Coverlap 0
Coverlap
(14)
This physically implies that as long as the manifolds are not excessively overlapping, the loss function remains locally strongly convex.
D.1.5 CONVERGENCE ANALYSIS
Based on the perturbation analysis, we state the revised convergence theorem.
Theorem D.1 (Linear Convergence under Bounded Overlap). Let L()) be the loss function. Assume
the overlap factor satisfies Coveriap < Cr. Then, the Hessian H at 0 is positive definite with minimum eigenvalue:
Amin (H) 2 Okeff
C' Coverlap > 0.
Consequently, gradient descent with step size 1) converges linearty:
110t - 100-00112
Higff
110(0)
11.
Wief
where the effective condition number is degraded by the overlap:
Fief
C' Coverlap
Proof. The proof follows directly from the strong convexity of L(0) established by Weyl's inequality. As Covertap + 0, we recover the ideal convergence rate.
D.2 MULTI-MODE ANALYSIS
In this section, we analyze the convergence properties for the mutli-Mode Mixture of Gaussians model. We explicitly model the overlap between Gaussian components as perturbation.
D.2.1 THE OVERLAP FACTOR
formally define the Pairwise Overlap Factor (x)(x) between two components and j:
(x) 2 Tk (Z)
(15)
And define the Maximum Expected Overlap Coverlap for the manifold as:
Eoverlop mnax
Σ
Ee (x)).
(16)
This scalar Coverlap quantifies the deviation from the ideal high separation regime. If components perfectly separated, Eijj + and Coverlap + O.
D.2.2 JACORIAN DERIVATION
We need to compute the Jacobian of the score matching error vector s0(2, () V logpe(ar) with respect the parameter Pk.1. Let (1r) 8um. V Pog (x))
Similarly, we decompose the Jacobian for the I-th component into a Signal Term (Self) and Noise Term (Interference).
34

---

Published as a conference paper at ICLR 2026
J4 (x)
J Hiidea (x) + By
cerruss (z).
Signal
Noise
This term arises when we ignore the change in weights other clusters (i + 1). dominates when Tk x 1:
(T
sf Uk. sf + 7
J jdideal (x)
7
Un
This term captures the gradient leaking into other clusters due to overlap:
Th
Ey y.r.s (5) = Σ (xr
(ks(())
(17)
(g((s)
where (x) collects bounded vector terms. The norm of the error term is strictly bounded by the overlap:
E E. (x) S Cj Σ Eg((x).
For the Jacobian with respect to Us, we have Similar derivation.
cross)))))) S 0% > (n)(x).
D.2.3 HESSIAN BLOCK STRUCTURE
The Hessian H for the parameters μ [MR,1.... Mk is a block matrix composed of ny nkk blocks, where each block is D x D.
H1,
H1.2
Hine H2
H2
H2.2
pp
Hnxdl
Ho
Hnam
The (t. block is defined as:
For diagonal blocks = 1), the curvature is strictly determined by the expectation of the squared weights (x))]. Crucially, overlap causes signal attenuation, as the weight (z) drops below 1 in transition regions.
Using the identity T2,((x)² = (x) - EjSt Tkij (2)),, we derive the exact expectation:
E|ra,(2)7] = [x]r
Σ
B(ra.())rm.s()))
-Tkd Σ [[))
total =Tk.l
35

---

Published as a conference paper at ICLR 2026
Thus, lower-bound the diagonal curvature by accounting for the total overlap mass c leaking from cluster 1:
Hig ( [
Ading. 1,
where the effective base curvature is:
Ading.1 (TR. lstal) min
Amia (Hu....u.
Here, the term (#R. lotal) represents the effective probability mass contributing convexity. This FR
formulation explicitly shows that smaller clusters (small )T.)) are significantly more vulnerable to Jotal becomes comparable to the cluster size instability, as the effective mass can vanish if the overlap Ex, Tk.I.
For * j, the block Hi, represents the interference.
Hig Ez (Jjkaa) ((rrs()))
D.2.4 PERTURBATION ANALYSIS
We write the full Hessian as a sum of block-diagona matrix and a perturbation matrix:
Hap Haiugg + A Hoverlap-
For the minimum eigenvalue Hdiags
Amin (Hding)) min Amin (Hi,)) min Ading. Xhase.
For Spectral Norm of Aovertapps by Weyl's Inequality, the minimum eigenvalue of the full Hessian
is:
Amin H) 2 Amin (Haing)
|| H overtapl12
and
A Hoverlap S S..ElE,,
(18)
where
Ć 2 S2
Substituting the bounds:
Amin (H) 2 Abose Eoverlap
Therefore, H is positive definite if and only if:
Abuse
coverlap
Interpretation: The optimization landscape is locally strictly convex provided the overlap between clusters is smaller than the intrinsic curvature of the individual Gaussians.
D.2.5 FULL CONVERGENCE THEOREM
Combining the analysis of μ and the similar decoupling argument for U (using Schur complements handle Ha terms which are also O(e)), arrive the final result.
Theorem D.2. Let L(0) be the score matching loss. Assume the maximun expected overlap Coverlap satisfies the condition Eoverlap < T for some threshold x Abase- Then the Hessian H())) strictly positive defmite
36

---

Published as a conference paper at ICLR 2026
Linear Convergence: Gradient descent with step size v, converges as:
|1000)
0-1 p² p'110(0) 0* 0211,
where the convergence rate 1 is determined by the effective condition number:
Kelf
Abase
Coverrlap
This proves that the High Separation Assumption is not binary requirement, but rather a continuum. The algorithm is robust finite overlap, with the convergence rate degrading gracefully as the overlap increases.
Remark D.3. is important note that physically, Coverlap will not be arbitrarily large.
E THE DETAIL OF THE WORLD EXPERIMENTS
In the part, we provide the detail of the experiments, including dataset and training pipeline. We use MNIST and CIFAR-10 as the datasets, and we adopt the mixture Gaussian distribution as the prior distribution in both cases.
For MNIST, our model consists MLP-hased encoder and decoder networks, each with a single hidden layer of 256 dimensions. The model is trained with the AdamW optimizer at learning rate of 0.0005. We train 10 VAES with the numbers to 10 as the ten clusters.
On CIFAR-IO, we implement a 3-layer RNN encoder and decoder for CIFAR-IO. The encoder hidden dimensions are [64, 128, 256], and the decoder's are [256. 128, 64] we train 10 VAEs for each of the ten clusters based on the classification by category. Each layer in both networks stacks recurrent blocks. The model is trained with the A optimizer at learning rate of 0.0001.
Our experiment was conducted on RTX4090.
37


Reviews:
[{"review_id": "T0R9JXlzYP", "reviewer": "Reviewer_qmDg", "summary": "This paper introduces MoLR-MoG (Mixture of Low-Rank Mixture of Gaussians) modeling for diffusion models, aiming to better reflect the multi-subspace and multi-modal structure of real-world data. The authors derive a closed-form score function under this model, which naturally exhibits a Mixture-of-Experts (MoE) structure. They provide finite-sample estimation error bounds that avoid the curse of dimensionality by depending on latent dimensions and mixture counts rather than the ambient dimension. Additionally, they prove local strong convexity of the score-matching objective under MoLR-MoG, yielding gradient descent convergence guarantees.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. MoLR-MoG is a novel generative prior that generalizes prior single-subspace Gaussian latent assumptions. The nonlinear MoG score derivation and its MoE interpretation are new theoretical contributions.\n2. The paper delivers rigorous generalization bounds and optimization theory  that are tight up to log factors and explicit in all problem constants.", "weaknesses": "1. Experiments are small-scale (MNIST/CIFAR-10). No evaluation on high-resolution images (e.g., ImageNet) or complex datasets (e.g., text-to-image, multi-resolution). FID, LPIPS, or human evaluations are absent; only visual samples are shown. Additionally, the time comparison of the proposed new diffusion model with the original diffusion is lacking.\n2. Theoretical guarantees rely on Δ ≫ γ_t (Assumption 6.1) and highly separated Gaussians (Assumption 6.6). No discussion of relaxation regimes or failure modes when clusters overlap significantly, which is common in real data.\n3. MoLR-MoG needs K VAEs (one per subspace) and clustering at inference. Complexity scales linearly with K; no ablation on K vs. performance vs. compute, nor comparison with single-VAE baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lkyz99IRTG", "reviewer": "Reviewer_DPpX", "summary": "The paper introduces the Mixture of Low-Rank Mixture of Gaussians (MoLR-MoG) distribution, which is employed to model both the real data distribution and the neural network architecture. Empirically, the authors demonstrate that the MoLR-MoG distribution provides a reasonable approximation to the data distribution learned by a U-Net–based architecture. Theoretically, they prove that, under the MoLR-MoG assumption, diffusion model training can escape the curse of dimensionality. Furthermore, by assuming subspace separability, the paper establishes local strong convexity properties for the training dynamics.", "questions": "1. What exactly is the MoLR-UNet used in Section 4? Why does the MoLR-UNet produce worse results than the standard U-Net architecture on MNIST and CIFAR-10, as shown in Figure 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel MoLR-MoG modeling framework that is applied to both the training data distribution and the network architecture design.\n\n2. Empirically, the authors validate the effectiveness of this modeling by demonstrating performance comparable to that of the standard U-Net architecture.\n\n3. Theoretically, they show that the MoLR-MoG framework mitigates the curse of dimensionality and establishes local strong convexity in the loss landscape. These theoretical results contribute valuable insights to the broader understanding of diffusion models from a theoretical perspective.", "weaknesses": "1. As a theoretically tractable model, MoLR-MoG inevitably exhibits a gap from real image distributions. As shown in Figure 2, there remains a noticeable discrepancy between CIFAR-10 images generated by MoLR-MoG and those from the true CIFAR-10 dataset.\n\n2. The assumption of highly separated Gaussian components (Assumption 6.6) is unrealistic for real-world image data. In practice, different image classes often share overlapping or correlated semantic subspaces. Consequently, the condition required for local strong convexity in the loss landscape is unlikely to hold for real image distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykuF7hJGpG", "reviewer": "Reviewer_LLnu", "summary": "The paper introduces Mixture of Low-Rank Mixtures of Gaussians (MoLR-MoG), a novel latent modeling framework for diffusion models. In this approach, high-dimensional data, such as images, are represented as a union of low-dimensional subspaces, with each subspace modeled by a mixture-of-Gaussian latent distribution. This modeling naturally induces a mixture-of-experts (MoE) nonlinear score function, enabling the network to capture both multi-modal and nonlinear latent structures far more effectively than prior Gaussian-based latent models. By leveraging this structured latent representation, diffusion models can generate high-quality samples efficiently, even from relatively small datasets. Furthermore, the framework comes with theoretical guarantees on estimation error and convergence, providing a principled explanation for the observed sample efficiency and fast optimization in practical diffusion model training.", "questions": "**Q1.** Is it a good assumption that the latent vector is Gaussian?  The paper assumes that the latent vectors follow a Gaussian distribution, which is a common and convenient choice for theoretical analysis, particularly for deriving sampling complexity and convergence guarantees. While this assumption is understandable for establishing tractable results, it is unclear how realistic it is for modeling real-world data, which may exhibit strongly non-Gaussian or multi-modal latent structures.\n\n**Q2.** Is it possible to improve the sampling efficiency in the reverse process under this MoLR-MoG model?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) **Novel latent modeling framework**: The paper introduces MoLR-MoG, which combines low-dimensional subspace modeling with a mixture-of-Gaussian latent structure, enabling diffusion models to capture multi-modal and nonlinear latent structures more effectively than prior Gaussian-based approaches.\n\n(2) **Theoretical contributions**: It provides provable estimation error bounds that show how the model can escape the curse of dimensionality, and also establishes convergence guarantees for gradient descent on the nonlinear MoE-latent score, offering strong theoretical support for the framework.\n\n(3) **Empirical Efficiency**: Experiments demonstrate that the MoE-latent MoG network can generate high-quality samples comparable to a MoE-latent UNet while using 10× fewer parameters, highlighting both practical efficiency and effectiveness of the proposed approach.", "weaknesses": "Here are some concerns about this paper:\n\n(1) **Inconsistency between experiments and theoretical results**: In Section 3.1, the authors derive a score network architecture based on the MoLR-MoG model. However, in Section 4, they train 10 VAEs to serve as encoders and decoders before applying the network architecture. This experimental setup appears to deviate from the theoretical framework, and it is unclear how it aligns with the assumptions and analysis presented in Section 3.\n\n(2) **Comparison with the existing literature**: In Wang et al. 2024 (https://arxiv.org/pdf/2409.02426), the experiments on MNIST and CIFAR look much better than those implemented in this paper. Please carefully check it and make a fair comparison. \n\n (3) **Convergence analysis and empirical validation**: In Lemma 6.10, the authors establish linear convergence for their gradient descent procedure under the MoLR-MoG model. However, it appears that no empirical experiments are provided to support this theoretical claim. Including such validation, for example, by plotting convergence curves or reporting iteration counts for training on real datasets, would help demonstrate that the theoretical guarantees translate to practical performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZSNdEBZP9l", "reviewer": "Reviewer_fdxV", "summary": "This paper extends low-dimensional data modeling (MoLR-MoG) to analyze the training of latent diffusion models. The assumption is that different classes lie near low-dimensional subspaces and can be modeled as mixtures of Gaussians on those subspaces. The network is parameterized to approximate the optimal score, with learnable means and covariances.\n\nUnder this setup, the authors establish uniform concentration and Lipschitz continuity of the empirical DSM loss. Using a separability condition, they derive Hessian bounds and prove local strong convexity and linear convergence. The analysis is developed first for a two-Gaussian case and then extended to multiple clusters.", "questions": "1. Fig. 1 suggests linear encoders/decoders $A_k$ to project into low-dimensional subspaces, whereas Fig. 2 seems to imply a nonlinear VAE before training. Do you ultimately replace $A_k$ with a nonlinear VAE in the experiments? If so, this differs from the linear $A_k$ setup. Please clarify which setting is analyzed theoretically versus used empirically.\n2. Can you provide rates or Hessian eigenvalue bounds without turning soft assignments into hard assignments (i.e., without $r_k \\approx 1$)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper formalizes an intrinsic low-dimensional structure: after encoding (e.g., with a VAE), the latent distribution becomes MoLR-MoG. This is a natural refinement because VAEs encourage approximately Gaussian latents.\n2. The theoretical treatment under the MoLR-MoG assumption is comprehensive, providing concentration, smoothness, and local convergence guarantees.", "weaknesses": "1. The implications feel modest: much of the contribution is incremental over existing MoLRG-style assumptions. Moreover there are some possible overstatements:\n   1. Claims such as “escaping the curse of dimensionality” follow directly from the assumption rather than the learning procedure.\n   2. The “Mixture of Experts” terminology seems overstated; the parameterization behaves closer to a one-layer gated linear unit than an MoE in the LLM sense.\n2. **Theory setup and presentation issues.**\n   1. The framework appears to require learning $K$ VAEs (one per class/subspace) and then training a MoG score within each subspace. The linear maps $A_k$ are introduced but not further investigated (e.g., not showing up in the Hessian)\n   2. This means $A_k$ is used mainly to reduce the problem to learning within each subspace: first, this is not standard and is like training an independent diffusion model for each class; second, does this require the $A_k$ to be orthogonal and have similar dimensions? There seems to be a lot missing here.\n   3. To obtain strong convexity and convergence bounds, the analysis effectively turns softmax into hardmax ($r_k \\approx 1$), which reduces the problem to (piecewise) linear models. Moreover, the arguments are all local.\n3. There is no numerical verification of the convergence behavior or deeper experiments illustrating the intuition and implications of MoLR-MoG.\n4. **Lack of Clarity.** Several lemmas are straightforward calculations that could be moved to the appendix; the core setup only becomes clear around pp. 4–5. There are also typos (e.g., “$r_k^+(x)=1\\ \\text{or}\\ 1$”) that should be fixed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)