Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value
Authors: Yixian Xu
Shengjie Luo
Liwei Wang
Di He
Chang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Generative Modeling
Optimal Loss Values
Training Strategies
Scaling Laws
Primary area: generative models
TL;DR: We point out the optimal training loss of diffusion models is not zero and unknown, causing inconveniences, hence propose estimation methods enabling a clearer training diagnosis, a principled training schedule, and a corrected scaling law analysis.

Abstract: Diffusion models have achieved remarkable success in generative modeling. Despite more stable training, the loss of diffusion models is not indicative of absolute data-fitting quality, since its optimal value is typically not zero but unknown, leading to the confusion between large optimal loss and insufficient model capacity. In this work, we advocate the need to estimate the optimal loss value for diagnosing and improving diffusion models. We first derive the optimal loss in closed form under a unified formulation of diffusion models, and develop effective estimators for it, including a stochastic variant scalable to large datasets with proper control of variance and bias. With this tool, we unlock the inherent metric for diagnosing training quality of mainstream diffusion model variants, and develop a more performant training schedule based on the optimal loss. Moreover, using models with 120M to 1.5B parameters, we find that the power law is better demonstrated after subtracting the optimal loss from the actual training loss, suggesting a more principled setting for investigating the scaling law for diffusion models.

Published as a conference paper at ICLR 2026
DIAGNOSING AND IMPROVING DIFFUSION MODELS BY ESTIMATING THE OPTIMAL LOSS VALUE
Yixian Xul; Shengjie Luo'r, Liwei Wang', Di He'! Chang Liu
1 State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China
2 Zhongguaneun Academy
ABSTRACT
Diffusion models have achieved remarkable success in generative modeling. De- spite more stable training. the loss of diffusion models is not indicative of absolute data-fitting quality, since its optimal value is typically not zero but unknown, lead- ing to the confusion between large optimal loss and insufficient model capacity. In this work, we advocate the need to estimate the optimal loss value for diagnosing and improving diffusion models. We first derive the optimal loss in closed form under a unified formulation of diffusion models, and develop effective estimators for it, including a stochastic variant scalable to large datasets with proper control of variance and bias. With this tool, we unlock the inherent metric for diagnos- mainstream diffusion model variants, and develop more performant training schedule based on the optimal loss. Moreover, using models with 120M to 1.5B parameters, we find that the power law is better demonstrated after subtracting the optimal loss from the actual training loss, suggesting principled setting for investigating the scaling law for diffusion models.
ing training quality
INTRODUCTION
Diffusion-based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have shown unprecedented capability in modeling the dominant choice in various domains. The attractive potential has incentivized advances in mul- tiple dimensions, such as prediction targets (Kingma et al., 2021; Salimans and Ho, 2022; Lipman al., 2023), diffusion process design (Karras et al., 2022; Liu et al., 2023), and training schedule design (Nichol and Dhariwal, 2021; Kingma and Gao, 2023; Esser et al., 2024).
distribution and have become
The success is largely benefited from the more stable training process. Nevertheless, the diffusion loss only reflects the relative data-fitting quality for monitoring training process or comparing mod- els under the same setting, while remains obscure for measuring the absolute fit to the training data. It is due to that the optimal loss of diffusion model, i.e., the lowest possible loss value that can be attained by any model, is actually inconveniences. After the training converges, one still does not know whether the model is already close to oracle, or the remaining loss can be further reduced by tuning the model. Practitioners have to rely on generating samples to evaluate diffusion models, which requires significant computational cost, and sampler configurations introduce distracting factors. The unknown optimal loss also makes it obscured to analyze and compare learning quality at diffeernnt diffusion steps, impeding princi- pled design of training schedule. Moreover, as the actual loss value is not fully determined by model capacity but also the unknown optimal loss as the base value, it poses a question on using the actual loss value alone for monitoring the scaling law of diffusion models.
zero but unknown beforehand. This introduces series of
In this work, we highlight the importance of estimating the optimal loss value, and develop effective estimation methods applicable to large datasets. Using this tool, we unlock new observations of data-fitting quality of diffusion models under various formulation variants, and demonstrate how the optimal loss estimate leads to more principled analysis and performant designs. Specifically,
We reveal the indefiniteness of the optimal loss from its expression, then develop estimators for the optimal loss based on the expression. For large datasets, we design scalable estimator based on dataset sub-sampling, with a delicate design to properly balance variance and bias.
Equaa contribution.
Cooresponnence Di He <dihelpku.ede edu. cn>, Chang Liu <1 iuchang@bza. edu. en>.

---

Published as a conference paper at ICLR 2026
Using the estimator, reveal the patterns of the optimal loss across diffusion steps diverse datasets, and by comparison with the losses of mainstream diffusion models under a unified for- mulation, we find the characteristics of different diffusion formulation variants, and identify the diffusion-step region where the model still underfits compared to the optimal loss.
From the analysis, we designed principled training schedule for diffusion models, based on the gap between the actual loss and the optimal loss. Our training schedule improves the FID by 2%-14% (for EDM (Karras al., 2022) / FM (Lipman et al., 2023)) on CIFAR-10, 7%-25% (for EDM/ FM) on ImageNet-64, and 9% (for LightningDiT (Yao et al., 2025)) on ImageNet-256.
We challenge the conventional formulation to study neural scaling law for diffusion models. We propose using the loss gap as the measure for data-fitting quality. Using state-of-the-art diffu- sion models (Karras al., 2024) in various sizes from 120M 1.5B on both ImageNet-64 and ImageNet-511,, we find that our modification leads to better satisfaction of the power law.
1.1 RELATED WORK
Optimal loss and solution of diffusion model. A related work by Bao et al. (2022b;a) derived the optimal ELBO loss under discrete Gaussian reverse process, and used it to determine the optimal reverse Gaussian (co)variances and optimize the discrete diffusion steps. Gu et al. (2023) further studied the memorization behavior of diffusion models. In contrast, we consider general cases and develop effective training-free estimators for the optimal loss value, and emphasize its principled role with important real examples in monitoring and diagnosing model training, designing training schedules, and studying the scaling law.
There are also some other works that made efforts to estimate the optimal solution (Xu et al., 2023) using importance sampling. Although more scalable methods are proposed using fast KNN search (Niedoba et al., 2024), their viability for estimating the optimal loss on large datasets remains unverified, as the optimal loss requires estimating two nested expectations.
Training design of diffusion model. Due to the stochastic nature, intensive research efforts are paid to investigate diffusion model training in multiple directions such as noise schedules and loss weight. Karras et al. (2022) presented design space that clearly separates design choices, enabling targeted explorations on training configurations. Kingma and Gao (2023) analyzed different diffu- sion objectives in unified way and connect them via ELBO. Esser et al. (2024) conduct large-scale experiments to compare different training configurations and motivate scalable design choices for billion-scale models. Most works require large-scale compute for trial and error, due the lack of a principled guideline for training schedule design based on the absolute data- fittiiiitng process.
Scaling law study for diffusion model. Model scaling behaviors are of great interest in deep learning literature. In particular, the remarkable success of Large Language Models has been largely credited to the establishment of scaling laws (Kaplan et al., 2020; Henighan al., 2020; Hoffmann et al., 2022), which help to predict the performance of models as they scale in parameters and data. There also exist works that empirically investigate the scaling behavior of diffusion models (Peebles and Xie, 2023; Li et al., 2024: Mei et al., 2025; Esser et al., 2024), and make attempts to explicitly formulate scaling laws for diffusion transformers (Liang et al., 2024). However, training loss values are typically used as the metric in these works, which are not corrected by the optimal loss to reflect the true optimization gap, leading to biased analysis for scaling behaviors of diffusion models.
2 FORMULATION OF DIFFUSION MODEL
Diffusion models perform generative modeling by leveraging a step-by-step transformation from an arbitrary data distribution Pdata to Gaussian distribution. Sampling and density evaluation for the data distribution can be done by reversing this transformation process step by step from the Gaussian. In general, the transformation of distribution is constructed by:
X = ax5g + d.e. t E [0,
(1) p(c) == -o. I) is a Gaussian noise sample, and
where X ~ Pdata is taken as data sample, x; the constructed random variable that defines the intermediate distribution Pt. The coefficients Or and or satisfy ao 1, do 0, and ar < or, so that po = Pdata and pr / o yield the desired distributions. Eq. (1) gives p(x: Xo) = NNxx; a, of)). which corresponds to diffusion process expressed in the stochastic differential equation dx, axx, dt + 9r dw; starting from Xq Po- where a; = oz), 9t = o ((og of / and w, denotes the Wiener
2

---

Published as a conference paper at ICLR 2026
process. The blessing of the ceffusiinnproceess formulation is that the reverse process can be given explicitly (Anderson, 1982):
dx, = - x d8 + Vlog ds + 9r-s dw.
from x==0 ~ pr, where - denotes the reverse time. Alternatively, the deterministic process
dx, - x ds + - -s lxg
also recovers Pdata at s = T (Song et al., 2021b). The only obstacle to simulating the reverse process for generation is the unknown term V Vlog Pe (x,) called the score function. Noting that p: produced by perturbing data samples with Gaussian noise, diffusion models employ neural network model Se (x,, t) to learn the score function using the denoising score matching loss (Vincent, 2011; Song et al., 2021b): (())(()
yo (x))) No ||s (x,, Vx, log |x))|| Eq (1)
E (ox t) + c/o.1l (2)
To cover the whole diffusion process, loss weight w, and noise schedule p(t) are introduced to optimize over all diffusion steps using J(0) == Eptr (x) J{ (0).
Alternative prediction targets. Besides the above score prediction target, diffusion models also adopt other prediction targets. Eq. (2) motivates the noise prediction (c-prediction) target (Ho al., 2020) eo e 1) == -r (X:: l), which turns the loss into
J (0) (xo(xx) Pp(e)ll El (axX0 + 0zE,() ell².
If formally solving Xo from Eq. (1) and let xoo (X,, xx-0xEp(Xx.)) then we get the loss:
as
x(x0) (e) == Exo(x Epte) [xoo (a:Xo + o,e,t) xul1²
(4)
holds the semantics of clean-data prediction (Xo-prediction) (Kingma et al., 2021: Karras et al., 2022), and can be viewed as denoising auto-encoders (Vincent et al., 2008; Alain and Bengio, 2014) with multiple noise scales. From the equivalent deterministic process, one can also derive the vector- field prediction (v-prediction) target Va (X,,
0,8; 2 (x,,t) t) with loss function
(8)) = (xx Ep(e) ll (axxa t.c t) foxx+o
(5)
It coincides with velocity prediction (Salimans and Ho, 2022) and the flow matching formula- tion (Lipman et al., 2023; Liu al., 2023). Please refer to Appx. for details.
3 ESTIMATING THE OPTIMAL LOSS VALUE FOR DIFFUSION MODELS
The diffusion loss in various forms (Eqs. 2-5) allows effective and stable learning of intractable targets that would otherwise require diffusion simulation or posterior estimation. Nevertheless, as will show from the expression of the optimal solution and loss (Sec. 3.1), the optimal loss value is typically non-zero but unknown, obscuring the diagnosis and design of diffusion training. We then develop practical estimators of the optimal loss value, starting from standard one (Sec. 3.2) stochastic but scalable estimators applicable to large datasets (Sec. 3.3). Using these tools, we inves- tigate mainstream diffusion models against the optimal loss with a few new observations (Sec. 3.4).
3.1 OPTIMAL SOLUTION AND LOsS VALUE OF DIFFUSION MODELS
Despite the intuition, the names of the prediction targets of diffusion model introduced in Sec. might be misleading. Taking the clean-data prediction formulation as example, is information- ally impossible to predict the exact clean data from its noised version (Daras et al., 2023). From the appearance of the loss functions (Eq. (2-5)), the actual learning targets of the models are conditional expectations (De Bortoli et al., 2021; Bao et al., 2022b;a):
$g. (xxx () = Expxs xx.) [V) Vvx, logp(x; xo)];
(g(x;,,) = Eppe x.) x.)(E),
xie (xx, t) = Ep(xx//xs) [xo],
vē(xi. t) = Ex(xo, (a/xx + o'e),
where the conditional distributions are induced from p(Xo. Xt, E) = po((x) e) (0 For completeness, detail the derivation in Appx. D.
Looking back into the loss functions, the model learns the conditional expectations over random samples from the joint distribution. Hence even at optimality, the loss still holds a conditional
3

---

Published as a conference paper at ICLR 2026
variance value. Noting that the joint distribution hence the conditional variance depends on the data distribution, would be more direct to write down the optimal loss value in the clean-data prediction formulation, which we formally present below:
Theorem 1. The optimal loss value for clean-data prediction defined in Eq. (4) is:
J(
TE Poo(xo
[xo 2=
Bp(x.) (ooxx))lEpxxoooe
ppxxo(x)) xxo]| J = -E (x0) s(xx))
BB
See Appx. F.1 for proof. For other prediction targets, the optimal loss value can be calculated based on their relations in Eqs. (3, 4, 5) The expression is derived from
1( Ep(x.) [Ep(x./x.) Xo - Epcx, x.) [x]]]]
which is indeed an averaged conditional variance of p(xo xz). and takes positive value unless t or when (x())) concentrates only on single point. For sufficiently large t, X becomes dominated by the noise (see Eq. (1)) hence has diminishing correlation with Xo. This means p(xo xx) R (Xo), hence J. Rs E. Epanss(Xx) Xo Paaas((x;)) [x]]]| approaches the data variance. Note that this optimal loss only depends on dataset and diffusion settings, but not model architectures and parameterization.
3.2 EMPIRICAL ESTIMATOR FOR THE OPTIMAL Loss VALUE
To estimate the optimal loss value using Eq. (6) on dataset {xD (n)
JnE[a where [N]
{1,-.., ,N, the first term A == Epo(xo) ||x0l| can be directly estimated through one pass:
1
nE[N] ||x (n) 1|2
N
However, the second term Bg == Ep(x.)) [lxxi xx.) [xo]]] requires estimating two nested expec- tations that cannot be reduced. The inner expectation is taken under the posterior distribution p(xo X₂) which cannot be sampled directly. By expanding the distribution using tractable ones (Bayes rule), the term can be reformulated as
E Ep)xalx.)) [xo] I xxp(xxo x1) dxo
Ex(()) xop(xx xo)]
J p(xo: dx0
W. Ex(()) [p(x; xo)]
Using Eq. (1) further reduces it as:
Ep(xo xx) [xx]] = Ep(xo) ( xo)]
IIxe axx0
where K2(X,, xo) = exp
(8)
Ep(x,) (X,, xp)]
20
then be estimated the dataset. The outer expectation can be estimated by averaging over set of independent and identically distributed (IID) samples [x{"m))mE(A)) following Eq. (L), where each sample is produced by independently (i.e., with replacement) randomly selected data sample and randomly drawn noise sample ~ (N I). The estimator for the second term is then:
whose numerator and denominator
EnE(M (n) Kx(x( (m) x(m) X
B,
1 M
Σ
ME[M] EWE(N Ki(x!TT) (m) (n))
The outer expectation be conducted sequentially until the estimation converges. This typically takes up to two to three times of N. The estimator in Eq. (9) performs the most accurate analysis using the given dataset, thus its results serve as the ground truth value in our following develop- ments.See Appx. H.1 for details.
3.3 SCALABLE ESTIMATORS FOR LARGE DATASETS
Although asymptotically unbiased (Appx. G), the β estimator in Eq. (9) incurs a quadratic complex- ity in dataset size N, which is unaffordably costly for large datasets which are ubiquitous in modern machine learning tasks. For a scalable estimator, dataset sub-sampling is an effective strategy. This strategy inspires to estimate Eq. (9) using a subset of the dataset, which helps reduce the com- putation complexity.Inteeat of using independent random subsets to estimate the numerator and de- nominator separately, we adopt the self-normalized importance sampling (SNIS) estimator (Robert
4

---

Published as a conference paper at ICLR 2026
et al., 1999; Kroese and Rubinstein, 2012) (see Appx. G for background):
Eielc] (1) X
(x, (m)) x
(1)
BSNTS
Σ
M
Erew Kx (m)
x (1))
mE|M]
It uses the same randomly selected (with replacement) subset {x herz where L & N, for both the numerator and denominator, which leads more stable estimates. One can repeat drawing the random data subset (xo (1) hE(L)] and calculate the estimate until convergence.
A specialty for estimating the diffusion optimal loss is that, for given x(") sample, when oe is small, the weight term K2 (x(x,",, (x(")xx(" ) is dominated by the Xy sample closest to x(m)//x (see Eq. (8)), which could be missed in the randomly selected subset {x() Je(z] thus incurring a large variance. Fortunately, we know that by construction (Eq. (I)), each x!_) sample is produced from a data 02x6 + ore(m) and when o, is small, ar is also close to 1 (Sec. 2), indicating that x6"") is likely the most dominant Xo sample and should be included in the subset {xf²) he(z]- This can be simply implemented by constructing the {x(")) [e[] samples by independently (i.e., with replacement) drawing sample xf')) from the toje(m) with e( ~N(00,). We call it the Diffusion
xx6)
sample x6"m) and a noise sample e!n)) using x(m)
subset {xx²) he (r. and setting x²"")
Optimal Loss (DOL) estimator:
(1) EIELI X (x 2x
(m)) (1)
BPOL
Σ mE[Mr]
(10)
M
Σre( (x x
Nevertheless, this introduces an artificial correlation between x₃ and xy samples: it becomes more probable to calculate K, for (xt: Xo) pairs where x, constructed from Xp- Such pairs have larger K, values, hence over-estimating B, and under-estimating the optimal loss J/ This pair intro- duces bias for the DOL estimator and leads to poor practical performance. We introduce a simple correction by down-weighting such pairs with coefficient C, and call it the corrected DOL (cDOL) estimator:
EiELL (1) X (x
(m)) (1)
(1a))
Ke(x, (16))
(ma))
DOL
1
Σ mE[M]
X + x
:xo
(11)
M
ErEuLEAA (xx x
(m)) (2)) + (x
(m)) (lmm)
where Li indexes the sample {x() he(A) that is used to construct x (m) tiveness, we provide the following theoretical result on the cDOL estimator:
To formalize the effec-
Theorem 2. The BeDOL estimator with subset size L has the same expectation the BSNIS esti- mator with subset size L - 1 when M + oo, C - 00, hence is consistent estimator.
See Appx. F.2 for proof. Note that the first terms in the numerator and denominator are unbiased, but the second terms introduce biases due to the artificial correlation between and samples. The DOL estimator in Eq. (10) amounts to using C 1, which suffers from the biases. The bias can be reduced using C >>> 1 in the cDOL estimator. On the other hand, the second terms become the dominant components at small t for estimating the numerator and denominator, respectively. Always including them using finite C' hence reduces estimation variance. The complete process of the cDOL estimator is concluded Alg. in Appx. H.I.
3.4 ESTIMATION RESULTS OF OPTIMAL LOss VALUES
We now provide empirical results of diffusion optimal loss estimates on popular datasets. We first compare the scalable estimators on CIFAR-10 10 (Krizhevsky et al., 2009) and FFHQ-64 (Karras, 2019) (Fig. 1(a.b)). whose relatively small sizes allow the full-dataset estimate by Eqs. (7, 9) providing a reference for the scalable estimators. With the best scalable estimator identified (from Fig. 2), we apply it to the much larger ImageNet-64 (Krizhevsky et al., 2012) dataset, and analyze the optimal loss pattern (Fig. 1(c)).
As different prediction targets (Sec. 2) and diffusion processes (Sec. 4.1 below) can be converted to each other, we choose the clean-data prediction target and variance exploding (VE) process (a,, = 1) (Song and Ermon, 2019; Song et al., 2021b) to present the diffusion optimal loss. We plot the optimal loss for each diffusion step, which is marked by log a to decouple the arbitrariness in the time schedule o, (as advocated by (Karras al., 2022); the same a indicates the same distribution
5

---

Published as a conference paper at ICLR 2026
Optimel loss by verious estimators on CIFM-10
Oprimel loss by verious extimetors on FF110-6
Optimal loss on varfous dofasets
G54830
DCk Ri-2
C20L CHOR)
Pragelad
(20.
L-S0ES Coids
120, L-50C1 CwHO0
0.0
0.13
0.13
0.15
11:5030. Co000)
(DCL 1L-3525 C-00)
(3CL 1L-19200. C-201
.901 11 10300, C1400
Ingin
1gg
lngo
(b)
(c)
Figure 1: Estimation results of optimal loss value. (a,b) Stepwise optimal loss estimates by the DOL (Eqs. 7, 10) and the corrected DOL (cDOL) (Eqs. 7, 11) estimators, with the full-dataset estimate
(Eqs. 7, 9) as reference, on the (a) CIFAR-10 10 and (b) FFHQ-64 datasets. (c) Stepwise optimal loss on various datasets in different scales. Figures plotted for the xo prediction loss of the VE process.
at that step of diffusion). All the scalable estimators repeat data subset sampling until the estimate converges. See Appx. H.1 for settings and discussions on efficiency.
Comparison among the scalable estimators. From
Error and variance of cDOL estimators ot laga - 1.25
Fig. I(a,b), we can see that the DOL estimator indeed under-
Enor of the cDOL eatimator Stdd the cDCL estmator
estimates the optimal loss as pointed out, especially at intermediate diffusion steps. The cDOL estimator can ef- fectively mitigate the bias, and stays very close to the ref- erence under diverse choices of C. The insensitivity of the cDOL estimator w.r.t C' can be understood that, for small t (equivalently, d), both the numerator and denominator are dominated by the C-corrected terms, in which C' cancels out, and for large t, the K, (x; (M)) (x(" x ([x)) term is in the same scale other terms hence is overwhelmed when compared with the summation.
G-1000 cDOL Estimatios
C-100
To better analyze the behavior of the estimators, zoom in on their estimation error and standard deviation. Fig. 2 presents the results at an intermediate log where the es- timation is more challenging. The result confirms that the variance increases with C. Particularly, C = Do which corresponds to the SNIS estimator (Thm. 2), it is hard to sample the dominating cases for the estimate, leading estimation error. At the C' = 1 end which corresponds to the DOL estimator, although the variance smaller, its bias still leads it to a large estimation error. The cDOL estimator with C' in between achieves consistently low estimation error. Empirically, a preferred C is around 4N/L. The sub- set size L can be taken to fully utilize memory. We also compare our cDOL estimators with prior estimators for optimal solution, Appx. H.2.
Figure 2: Error and variance of cDOL estimates using various C' val- ues (including DOL and SNIS as ex- treme cases) for the optimal loss at log = 1.25 on CIFAR-10.
large variance, and significantly large
The pattern of optimal loss. From Fig. 1(C), we observe that the optimal loss j(xo)* increases all the three datasets. The optimal loss is close zero
monotonically with the noise scale only when the noise scale a is less than a critical point af, in which situation the noisy samples stay so close to their corresponding clean sources that they are unlikely to intersect with each other, hence preserve the information of the clean samples, allowing the model to perform a nearly perfect denoising. We can see that the critical point o² depends on the dataset. CIFAR-10 achieves the minimal o², since it has the lowest image resolution (32x x32), i..., the lowest data-space dimension, where the data samples appear less sparse hence easier to overlap after isotropic noise perturbation. Both FFHQ-64 and ImageNet-64 have 64 64 resolution, but ImageNet-64 is larger, hence data samples are easier to overlap, leading to smaller 0²
Beyond the critical point, the optimal loss takes off quickly. The positive value indicates the intrinsic difficulty of the denoising task, where even an oracle denoiser would be confused. The increase trend
converges for sufficiently large noise scale a, which meets analysis under Thm. 1 that )(x))*

---

Published as a conference paper at ICLR 2026
Table 1: Viewing mainstream diffusion models under the same formulation as x prediction un- der the VE process, following Eq. (12). Each diffusion model is labeled by "diffusion process"- "prediction target" ("common name").
Formulations
p(a)
14000 1),
VP-c (DDPM) (Ho ct 2020)
0
Vi403
999
je + 510mma Pant 22 log N Piu)
VE-F (EDM) (Karras et al., 2022)
loog
V
V
VE-e (NCSN) (Song al., 202la)
log
log ~ U(og log manx) 1~ (w, 1), T log ((
FM-v (FM) (Lipman et al., 2023)
FM-v (SD3) (Esser al., 2024)
140
(Lle
converges to the data variance. As ImageNet-64 contains more diverse samples (images from more classes), it has larger data variance, hence converges to a higher value than the other two.
ANALYZING AND IMPROVING DIFFUSION TRAINING SCHEDULE WITH OPTIMAL LOss
From the training losses in Sec. 2, the degree of freedom for the training strategy of diffusion models is the noise schedule p(t) and the loss weight wi, collectively called the training schedule. In the literature, extensive works (Ho et al., 2020; Song et al., 2021b; Karras et al., 2022; Kingma and Gao, 2023; Esser et al., 2024) have designed training schedules for various prediction targets and diffusion processes individually, based on the analysis on the loss scale over diffusion steps. Here, we argue that analyzing the gap between the loss and the optimal loss would be a more principled approach, since it is the gap but not the loss itself that reflects the data-fitting insufficiency and the potential for improvement. Under this view, we first analyze and compare the loss gap of mainstream diffusion works on the same ground (Sec. 4.1), identifying new patterns that are related to generation performance. We then develop new training schedule based on the observation (Sec. 4.2).
4.1 ANALYZING TRAINING SCHEDULES THROUGH OPTIMAL LOSS
Existing training schedules are developed under different diffusion processes and prediction targets. For unified comparison on the same ground, we start with the equivalence among the formulations and convert them to the same formulation. As explained in Sec. 3.4, we use the noise scale a in place of t to mark the diffusion step to decouple the choice of time schedule Tg.
Equivalent conversion among diffusion formulations. Sec. 2 has shown the equivalence and conversion among prediction targets. We note that different diffusion processes in the form of Eq. (1) can also be equivalently converted to each other. Particularly. the variance preserving (VP) process (a,, = /I - σ²) (Sohl-Dickstein et al., 2015; Ho et al., 2020) and the flow matching (FM) process (a,, 1- σ) (Lipman et al., 2023; Liu et al., 2023) can be converted to the variance exploding (VE) process (a, = 1) (Song and Ermon, 2019; Song et al., 2021b) by x := a. since x xu
by Eq. (1), and x YE The correspondence of diffusion step given by V __ With this o, fact, various diffusion models can be viewed as different parameterizations of the xo prediction under the VE process (Karras et al., 2022), where the parameterization is formulated by:
Xue (x, o) ckiip + F Fe(c"" Cenis)),
(12)
where Xoe, x, and a are the xu prediction model, the diffusion variable, and the noise scale under the VE process, and Fe(-.) represents the bare neural network used for the original prediction target and diffusion process. The precondition coefficients 6, are responsible for the conversion. Their instances for reproducing mainstream diffusion models are listed in Table 1, where the converted wi, and p(a)) from the original works are also listed. See Appx. E for details. For EDM (Karras et al., 2022), the precondition coefficients are not derived from a conversion but directly set to satisfy the ouputt unit variance principle. This leads to new prediction target call the F prediction.
Comparison between optimal loss gaps. Under the above convention, we convert the actual train- ing loss of various diffusion models the x. prediction loss under the VE process, which serves as a unified metric on the same ground. In the following development of the paper, we consider the x, prediction loss under the VE process of the diffusion model without loss of generality. We conduct
7

---

Published as a conference paper at ICLR 2026
Malritram oiTralon modda
Valnirram Valntream-diaake dityaleemmtttnnnmmmm model
lago
I0op
lage"
(a)
(b)
(c)
Figure 3: Actual and relative training loss gap across noise scales by various diffusion models on CIFAR-10. (a) Actual training loss gap of mainstream diffusion models. (b) Normalized training
loss gap, taken by dividing the value on each curve by the average value over the four curves, for clearer comparison over mainstream diffusion models ("VE--" is omitted for its salient differ- ence). (c) Normalized training loss gap for comparing existing training schedules and our schedule (Sec. 4.2) under the FM process with the prediction target. Curves of different diffusion models are plotted together by viewing them as parameterizations of the X prediction under the variance exploding (VE) process (Eq. (12); Table 1). See Appx. H.3 for detailed settings.
the comparison on the CIFAR-10 10 dataset, and compare the gap between their actual loss and the optimal loss, which has been estimated and presented in Fig. I(a; the full-dataset curve). The results are shown in Fig. 3(a), which reveals some new observations. The optimal loss gap across different diffusion steps is not even: most of the representative diffusion models leave large loss gap in the intermediate diffusion steps around log a E [-2, 2]. indicating room to improve. In addition, e prediction models incur large error for large a, revealing a difficulty in learning such models.
Loss gap vs. generation performance. We now use the gap to the optimal loss as the fundamen- tal data-fitting measure analyze which region is more critical for the generation performance, measured in Fréchet Inception Distance (FID) (Heusel et al., 2017) also marked in Fig. 3(a). All diffusion models use the same deterministic ODE sampler with NFE 35 following Karras et al. (2022). We can see that an erroneous fit at large noise scales of e prediction models leads to deficiency in generation quality, e.g., NCSN vs EDM Fig. 3(a). Among methods with a good fit for large a, the intermediate noise scale region log [-2.0, 2.0] becomes more relevant to the generation performance. We then zoom into the intermediate region and compare these methods using the normalized training loss gap, taken by dividing the value each curve by the average value over the four curves, as shown in Fig. 3(b) ("VE-e" is omitted due to its significant deviation from other curves). Counter-intuitiven optimal loss J becomes positive, although the gap is around the largest over noise scales, the loss gap turns out negatively correlate with FID. The correlation becomes positive only in the region further left to o²
around the critical point o" defined as the largest whose
o This reveals that there is a trade-off in learning a diffusion model at different noise scales, and sacrificing the fit in the region around a* for better fit in the noise scale interval further left to o* leads to better inference performance.
4.2 PRINCIPLED DESIGN OF TRAINING SCHEDULE
From the observation above, the training schedule plays crucial role in optimizing diffusion mod- els, due to the trade-off over different noise scales. We hence design a principled training schedule based conclusions from the representative optimal loss estimates.
The loss weight. wa calibrates the error resolution across different noise scales. For this, the optimal loss J provides perfect reference scale for the loss at each diffusion step 0,, so wa a/ with a scale factor a is a natural choice to align the loss at various a to the same scale. Although downweighs the loss for large noise scales, the above observation suggests that the model can still achieve good fit if using v-prediction and F- r--redictionn (Table I). For smaller noise scales, a cutoff' w" needed to avoid divergence, which stops the increase of wa before runs smaller than the critical point a+. As observed in Sec. 4.1, the interval where the loss gap has a positive correlation to inference performance is on the left of (vs. around) a*. We hence introduce an additional weight function faa N (log σ: μt, <²), whose parameters μ and are chosen to put the major weight over the left interval. The resulting loss weight is finally given by: W. = a min J/. w' f() Tocat.
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison between existing training schedules and ours on ImageNet-256 dataset.
Method
Generation w/o CFG
Generation CPG
FIX()
IS(1)
Pre.(1)
Rec.(1)
HD(())
1S(1)
Pre.())
Rec.(1)
Pisef-space Diffusion Models
ADM (Dhariwal and Nichol, 2021)
10.94
0.69
0.63
3.94
215.9
0.83
0.53
RIN (labri et 2022)
3.42
182.0
Simple Diffusion (Hoogehoom al., 2023)
2.77
2118
2.12
256.3
VDM++ (Kingma and Gao, 2023)
2.40
225,3
2.12
267.7
SID2 (Hoogeboom et al., 2024)
1.38
Latent Diffiston Models
MaskDiT (Zheng et al., 2023)
5.69
177.9
0.74
0.60
2.28
276.6
0.61
DiT (Pochles and Xie, 2023)
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
SIT (Ma et al., 2024)
8.61
131.7
0.68
0.67
2.06
270.3
0.82
0.59
FasterDiT (Yao ul.. 2024)
7.91
131.3
0.69
2.03
264.0
0.60
MITT (Ciao al., 2023a)
6.23
143.0
0.71
0.65
1.79
283.0
0.81
0.61
MDTv2 (Gao et al., 20233)
1.58
314.7
0.79
0.65
REPA (Yu al., 2024)
5,90
1.42
305.7
0.80
0.65
LightnnggDi (Yao et al., 2025)
2.17
205.6
0.77
0.65
1.35
295.3
0.79
0.65
reproduction our schedule
2.29
206.2
0.76
0.66
1.42
292.9
0.79
0.65
2.08
220.8
0.77
0.66
1.30
301.3
0.79
0.66
The noise schedule. p(o) allocates the optimization frequency to each noise level. A desired p(a) should favor noise steps on which the optimization task has not yet been done well, which can be measured by the difference from W))o(0) to J. This provides prin- cipled measure for optimization insuffi- ciency, which leads to an adaptive noise schedule: p(a) ()) J.)-
Table 2: Generation FID (U) by existing training sched- ules and ours on CIFAR-I0 and ImageNet-64.
CIFAR-10
ImageNet-64
Conditional Unconditional
Conditional
StyleGAN (Karras, 2019)
242
2.92
ScoreDDE (deep) (Song al., 2021b)
2,20
Improved DDPM (Nichol and Dhariwal, 2021)
2,94
3.54
2-Rectified Flow (Lin et al. 2023)
4.85
2.92
VDM (Kingma al., 2021)
2.49
3.40
EDM (Kamas et al., 2022)
1.79
1.98
2.44
EDME (Karas al., 2024) schedule + our schedule
1.9
2.09
1.75
1.94
2.25
Note that applying the loss weight and noise schedule in training model only requires estimating the optimal loss J on the training dataset, which does not require training a model beforehand.
FM (Lipman 2023)
6.35
14.45
EDM sampler
2.07
2.24
3.06
schedule
1.77
2,03
2.29
CIFAR-10 & ImageNet-64 Results. We evaluate the designed training schedule in training two advanced diffusion models EDM (Karras et al., 2022) and Flow Matching (FM) (Lipman et al., 2023) on the CIFAR-10 and ImageNet-64 datasets (details in Appx. H.5). As shown Table 2, our training schedule significantly improves generation performance upon the original for both the EDM and FM settings on both datasets, demonstrating the value of new insights from our analysis using the optimal loss. For closer look at how our schedule works, in Fig. 3(C) we plot the relative training loss gap across noise scales using our schedule, and compare with the schedule of original (Lipman al., 2023) and of StableDiffusion 3 (SD3) (Esser et al., 2024). We find that our schedule indeed further decreases the loss in the interval with positive correlation to performance, aligning with the insight from Sec. 4.1.
ImageNet-2566 Results. Finally, we evaluate our training schedule on the ImageNet-256 dataset and compare the results with existing approaches. We use VA-VAE (Yao et al., 2025) the to- kenizer and employ a modified LightningDiT (Yao et al., 2025) architecture enhanced with QK- Normalization (Dehghani et al., 2023) to improve training stability (details in Appx. H.5). As shown in Table 3, our training schedule further improves generation performance over the original Light- Din training schedule.
5 PRINCIPLED SCALING LAW STUDY FOR DIFFUSION MODELS
Neural scaling law (Kaplan et al., 2020) has been the driving motivation for pursuing large mod- els, which shows the consistent improvement of model performance with computational cost. The conventional version takes the form of power law (Kaplan et al., 2020; Henighan et al., 2020; Hoffmann al., 2022): J(F) F where F denotes floating point operations (FLOPs) measur- ing training budget, J(F)) denotes the minimal training loss attained by models in various sizes (the envelope in Fig. 4), and < 0 and B > are power law parameters. The specialty of scaling law

---

Published as a conference paper at ICLR 2026
Training at
Training Gep/e-/" 8: 4.38 (p 0.94)
Loss Gap)
65M2-25
S2M25
B2M0.5
147%
ID42
R1M23
8FMAFL
LS5522
|000000
EDN2 X5
LUMB5
EDR2 M4
EDM2-XL
2439
185
(c)
Figure 4: Scaling law study using optimal loss on ImageNet-64. Training curves at log σ = 4.38 using various model sizes and their envelope are plotted showing (a) the actual loss and (b) the gap between the actual and the optimal loss. Curves showing the gap for the total loss (covering all diffusion steps) are plotted in (c). The total loss gap is the training loss gap multiplied by the loss weight under the expectation of the noise schedule, i.e. the practical training loss, Correllcnl coefficients p for the envelopes are marked.
study for diffusion model that, as the optimal loss sets non-zero lower bound of the training loss, not all the loss value in (F)) can be reduced with the increase of F, questioning the form which converges to zero as - oc. Instead, the following modified power law is assumed:
J(F)
3F
(13)
where J denotes the optimal loss value. It can be rephrased that log(J(F) J*) linear in log F, so we can verify it through the linear correlation coefficient p. We conduct experiments using current state-of-the-art diffusion models EDM2 (Karras et al., 2024) with parameter size ranging from 120M to 1.5B.
We first compare the model training curves a large noise scale, for which Fig. 4(a) and (b) assume the original and the modified (Eq. (13)) scaling laws, respectively. We can observe that in the modified version, the envelope is indeed closer line, and the improved correlation coefficient 0.94 (vs. 0.82) validates this quantitatively. For the total loss, we use the optimized adaptive loss weight by EDM2 (Karras et al., 2024). The result shown in Fig. 4(c), which achieves 0.9917. 7, and the fitted scaling law is given by: (FF) 0.3675 F- 0.0014 + 0.015. Appx. H.6 provides more results on ImageNet-512. We hope this approach could lead to more profound future studies in the scaling law for diffusion models.
CONCLUSION
In this work, we emphasize the central importance of optimal loss estimation make the training loss value meaningful for diagnosing and improving diffusion model training. To the best of our knowledge, we are the first notice and work this issue. We develop analytical expressions and practical estimators for diffusion optimal loss, particularly scalable estimator applicable to large datasets and has proper variance and bias control. With this tool, we revisit the training behavior of mainstream diffusion models, and propose an cosnmallloss-basee approach for principled design of training schedules, which indeed improves performance in practice. Furthermore, we investigate the scaling behavior w.r.t model size, in which accounting for the optimal loss value offset better fulfills the power law.
Although the optimal loss estimation is not directly meant for analyzing inference performance, it introduces a scrious metric for data-fitting quality, which paves the way for future generalization study. We believe our approaches and insights could motivate future research on analyzing and improving diffusion models, and advance the progress of generative modeling research.
ACKNOWLEDGMENTS
This work is supported by Zhongguancun Academy (Grant No. C20250506). DH is supported by National Science Foundation of China (NSFC62376007), National Science Foundation of China (under Key Project No. 92570203), Beijing Natural Science Foundation (Z250001) and Beijing Major Science and Technology Project under Contract 7251 100088425044.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study does not involve human subjects, per- sonal data, sensitive demographic information. All experiments are conducted publicly avail- able benchmark datasets, which are widely used in the machine learning community. No new data collection human/animal experimentation was performed.
REPRODUCIBILITY STATEMENT
To facilitate the reproducibility of our research, we provide comprehensive details throughout the paper and its supplementary materials. We begin by establishing the necessary backgrounds in Sec. 2. For all theoretical claims in the main text, offer detailed derivations Appx. F. All our experiments are thoroughly documented; the datasets, training procedures, and settings are carefully described in Appx. H. Upon acceptance of this paper, we commit to making our full codebase and all model checkpoints publicly available to ensure that the community can fully reproduce our results.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
In the preparation of this manuscript, LLMs employed as writing assistant refine the language and improve the grammar. Following this process, all textual content was meticulously reviewed, revised, and validated by the authors, who assume full responsibility for the final work presented.
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In International Conference on Machine Learning, pages 1555-1584. PMLR, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the opti- mal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022b.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL https: http///ppeeeeeeeeetteteee / Forum?id=wBJBLe
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to basee-based generative modeling. Advances in Neural Information Processing Systems, 34:17695-17709, 2021.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pages 7480-7511 PMLR, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Ad- vances Neural Information Processing Systems, 34:8780-8794 2021.
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVE international conference on com- puter vision, pages 23164-23117 2023a.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion trans- former is a strong image synthesizer. arXiv preprint arXiv:2303. 14389, 2023b.
Xiangming Gu, Chao Du Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02666 2023.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 1840-6851, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hen- nigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neu- ral Information Processing Systems, 2022. URL https: /openreviei net worummmaii ?id= iBBcRULOAPR.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference Machine Learning, pages 13213-13232.
PMLR, 2023.
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Sali- mans. Simpler diffusion (sid2): 1.5 fid on imagenet5 12 with pixel-space diffusion, arXiv preprint arXiv:2410. 19324, 2024.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.11977 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Tero Karras. A btyle- generator architecture for generative adversarial networks. arXiv preprint arXiv: 2019.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 24174-24188 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21 2021.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives the ELBO with simple data augmentation. seirth-eventth Conference on Neural Information Processing Systems, 2023. URL https:tt////....... https://///eeeeee....e... forum?i d
12

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Dirk P Kroese and Reuven Y Rubinstein. Monte carlo methods. Wiley Interdisciplinary Reviews: Computational Statistics, 4(1):48-58, 2012.
Hao Li, Yang Zou, Ying Wang. Orchid Majumder, Yusheng Xie, R Manmatha, Ashwin Swami- nathan, Zhuowen Tu, Stefano Ermon, and Stefano Soatto. On the scalability of diffusion-based texi-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9400- 9109, 2024.
Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprint arXivv2410.08/84 2024.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: /ppp//peeeeiieeeeeee/eeeeeeeeeee net/ forum? P
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference Learning Repre- sentations, 2023. URL https: : / opp.reveee..e.e www.nreeeeweeeeeeeeeeeeeeee wtw.re/iwww.eett/oruriieeee.eee (oormmiiii//jfInnne
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.1 1081, 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Sain- ing Xic. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23-40. Springer, 2024.
Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M. Patel, and Peyman Milanfar. Bigger is not always better: Scaling properties of latent diffussio models. Transactions Machine Learning Research, 2025, ISSN 2835-8856. URL htttpp::/oppereeettte www.r/veeeeeeeeeeeeeeeeeeeee n
forum?id=Ou7? (p
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
Matthew Niedoba, Dylan Green, Saeid Naderiparizi, Vasileios Lioutas, Jonathan Wilder Lavington,
Xiaoxuan Liang, Yumpeng Liu, Ke Zhang, Setareh Dabiri, Adam Šcibior, et al. Nearest neighbour score estimators for diffusion generative models. arXiv preprint arXiv:2402.08011 08018 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods, volume 2. Springer, 1999.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL htte//oouumiiddTIdtX..s
Daniel Sanz-Alonso and Omar Al-Ghattas. A first course in monte carlo methods. arXiv preprint arXiv:2405.1635,, 2024.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Leamn ing, pages 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations, 2021a.
13

---

Published as a conference paper at ICLR 2026
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, volume 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 202 lb.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1677 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoins Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008.
Yilun Xu, Shangyuan Tong, and Tommi Jaakkola. Stable target field for reduced variance score estimation in diffusion models. arXiv preprint arXiv:2302.00677 2023.
Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers training without architecture modification. Advances in Neural Information Pro- cessing Systems, 37:56166-56188 2024.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2025.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09338 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Organization of the Appendix The supplementary material is organized as follows. We first review more related work in Appx. A and remark the relation between estimating optimal loss and achieving generalization in Appx. B. In Appx. D, we provide a detailed derivation of the optimal solution for diffusion models in various formulations. In Appx. E, we give the detailed derivation of the conversion shown in Table 1. In Appx. F, we give the proofs of all theorems. In Appx. G, we give brief introduction to the important sampling methods. In Appx. H, give the details of all our experiments.
A RELATED WORK
Optimal loss and solution of diffusion model. A related work by Bao et al. (2022b;a) derived the optimal ELBO loss under discrete Gaussian reverse process, and used it to determine the optimal reverse Gaussian (co)variances and optimize the discrete diffusion steps. Gu et al. (2023) further studied the memorization behavior of diffusion models. In contrast, we consider general cases and develop effective training-free estimators for the optimal loss value, and emphasize its principled role with important real examples in monitoring and diagnosing model training, designing training schedule, and studying scaling law. There also some other works that made efforts to estimate the optimal solution (Xu et al., 2023) using importance sampling. Although more scalable methods are proposed using fast KNN search (Niedoba et al., 2024), their viability for estimating the optimal loss on large datasets remains unverified, as the optimal loss requires estimating two nested expectations.
Training design of diffusion model. Due to the stochastic nature, intensive research efforts are paid to investigate diffusion model training in multiple directions such as noise schedules and loss weight. Karras al. (2022) presented design space that clearly separates design choices, enabling targeted explorations on training configurations. Kingma and Gao (2023) analyzed different diffu- sion objectives a unified way and connect them via ELBO. Esser et al. (2024) conduct large-scale experiments to compare different training configurations and motivate scalable design choices for billion-scale models. Most works require large-scale compute for trial and error, due to the lack of principled guideline for training schedule design based on the absolute data-fitting process.
Scaling law study for diffusion model. Model scaling behaviors are of great interest in deep learning literature. In particular, the remarkable success of Large Language Models has been largely credited to the establishment of scaling laws (Kaplan et al., 2020; Henighan et al., 2020; Hoffmann et al., 2022), which help to predict the performance of models they scale in parameters and data. There also exist works that empirically investigate the scaling behavior of diffusion models (Peebles and Xie, 2023; Li al., 2024; Mei et al., 2025; Esser et al., 2024), and make attempts to explicitly formulate scaling laws for diffusion transformers (Liang et al., 2024). However, training loss values are typically used as the metric in these works, which are not corrected by the optimal loss to reflect the true optimization gap, leading to biased analysis for scaling behaviors of diffusion models.
B DISCUSSIONS ON OPTIMAL LOss ESTIMATION AND GENERALIZATION IN DIFFUSION MODELS
We would like to emphasize that the motivation of estimating the optimal loss is not to optimize the model to achieve the optimal loss measuring the absolute fitness to a dataset. As an analogy, monitoring the supervision loss (e.g., the mean squared error (MSE) between model prediction and data labels) does not meant to optimize it to zero on the training set, but to evaluate the fitness to data. Particularly, one would use the test set to evaluate the performance, which essentially to evaluate the fitness to the test set. The specialty with diffusion loss is that its optimal value is unknown beforehand, so the loss value does not readily reflect the absolute fitness to a dataset. Estimating the diffusion optimal loss can hence provide a reference to interpret real training loss value. This even enables evaluating the generalization of a diffusion model using the loss value on a test set compared to the optimal value on the dataset. This evaluation does not require the costly and tricky generation process as typically adopted.
dataset, but to fulfill the more fundamental need of
supervision loss
For the design of the training schedule, we would like to mention that the design is not intended to reduce the loss gap for every time step. As is shown in Fig. 3, there a trade-off of loss optimization
15

---

Published as a conference paper at ICLR 2026
at different time steps, where the loss gap in some time-step intervals is positively correlated to in- ference performance, while on some other intervals negatively correlated to inference performance. Therefore, the design of the training schedule is rather putting more emphasis on the positively correlated regions, even if this would sacrifice the loss in some other regions.
C ALTERNATIVE FORMULATIONS OF DIFFUSION MODELS
Besides the score prediction target introduced in Sec. 2, diffusion models also adopt other prediction targets. The formulation of Eq. (2) motivates the noise prediction target (Ho et al., 2020). Let e (x,, 1) = -0,80 (x,, (), the the loss becomes
J(O)- = w Ep e Ex(e)) Eo + o,c,t) - ell where w = ne /o.-
wi()
This formulation poses a friendly, bounded-scale learning target and avoids the artifact at f = 0 of the denoising score matching loss. If formally solving xo from Eq. (1) and let
X₁ -x,0(xx,tt Xx oise (x..t) a2
Xxo(Xx;)
at
then we get the total loss becomes =(0)= (50) (x0) (0), where: J ((0))
(0) ((x0((0)
== ) Bptc) (Xot (a:xo + o,e,t) xo (o. (0), w/ = (a//o/)u(
holds the semantics of clean-data prediction (Kingma et al., 2021; Karras et al., 2022), and can be viewed denoising auto-encoders (Vincent et al., 2008; Alain and Bengio, 2014) with mul- tiple noise scales. From the equivalent deterministic process, one can also derive the vector-field prediction target by
ve (x(x == 0:8, - 2 2 sxt (x::)),
(v) an the loss function becomes J(0) = w/ v (0), where
(0) := Exxx (oo t) -
e)) "x' (4/
It coincides with velocity prediction (Salimans and Ho, 2022) and the flow matching formula- tion (Lipman et al., 2023; Liu et al., 2023): v := a/xo+oje is the conditional vector field given Xo and e (same distribution as xT). In particular, if we set Or
- I, oz 1, T 1, then - and we have v = c Xo. which corresponds to the Flow Matching (Lipman al., 2023; Liu al., 2023) formulation that is also widely used in generative modeling recently due to its simplicity.
a; 5
D OPTIMAL SOLUTION OF DIFFUSION MODELS
In this section, provide detailed derivation of the optimal solution for diffusion models across various formulations. We demonstrate that in every case the models' learning targets are conditional expectations, discussed in Sec. 3.1. To this end, we begin by introducing useful lemma that outlines a key property of conditional expectations (Durrett, 2019).
Lemma 3. Xetx,y be random vectors. Then the optimal approximation of based on x is f(x) E[y[x].
f' (x)
arg min Efyy F
Proof. We can compute Elyy f(x)11 directly by
Elyy f(x) = Elly - IEyy |x]
+ E6 ly xx] - f(x)11
E lly
Hy [x]|||| 2 + E [|]] - [ + 2E - E(ylx], Eiylx] - f(x)).
Since
By
Ely xx | (xx))
[Eay -
we have the following decomposition:
Elyy f(x) = Ely - Ely
E [ f(x)
Since [|]] - [( > 0, we have
Elyyy f() Fllyy
Fly x||
[|]
f(x)11
16

---

Published as a conference paper at ICLR 2026
E Ely x]||
The inequality becomes equality if and only if f(x) = Ely[x]. So the the optimal approximation of y based on is Ely[x],ia i.e.
f' [xx
mrg Elyy f(x) || E[y[x]. f-R--Rd
Score prediction target.
The score prediction target of diffusion model is given by
(0) :=) 2
E no (xo) Ep(x, ((a) ||se (x,, t) Vx, p(xe xx)]l
Then by Lemma 3, for any satisfying w (8) > _d is given by
O. the optimal solution of the network sa () R2
s/ (x,, t) Ep(xo x.) VV, log p(x; Xu)].
Noise prediction target. The noise prediction target of diffusion model is given by
()) == Ep(e) w/ (=) " (piex (e + o,e, t) - ell²
Then by Lemma 3. for any t satisfying w ({()) > 0, the optimal solution of the network Eo (...t) Rd I2d is given by
c cx,,,, 1) = [e))
Clean-data prediction target.
The clean-data prediction target of diffusion model is given by (0) ((xx)) Epte)) Xoo (oxX0 o t) xoll
((xx)(()
Then by Lemma 3, for any satisfying w. wexx)) > 0, the optimal solution of the network Xue (::() Rd + R$ is given by
Xño (x,, t) = Ep(xalx)) [xo]-
Vector-field prediction target.
The vector field prediction target of diffusion model is given by s( (0) =- ) Bptc (o:x0 t) (a/xo+o
Then by Lemma 3, for any t satisfying w! > O, the optimal solution of the network vo(:,) Rd IRd is given by
(x,, Eptxo + dfe..
E DETAILS ON THE GENERAL DIFFUSION FORMULATION
In this section, we introduce the detailed conversion between different diffusion formulations. As we have mentioned in Sec. 3.4, we can convert previous schedules to EDM formulation. The training objective in EDM formulation is given by
J(0) Ep(rr) w
E (x(x)) X (Xo + Te, xxo1
where the denoiser has precondition Xoo (x, a) skip x + cou Fo cholse) cholse).
E. CONVERT VP SCHEDULES EDM FORMULATION
DDPM (VP), c-pred training objective. 2020; Song et al., 2021b) is given by
The training objective in DDPM formulation (Ho al.,
E,pors (e ||Ea (axx + oee, (M 1)t) ell?
where M = 1000, p DDPM (t) = U(E,, 1), = 15 5, and
σ: V exp(-(3pl + ; [Bmaxx Bn)
0² = exp(-(3ot + (9max - Bb)
4
17

---

Published as a conference paper at ICLR 2026
The diffusion process satisfying aj + of 1 is also known as the variance preserving (VP) pro- cess. In order to convert the DDPM formulation to the EDM formulation, we should transform the diffusion process to VE. Dividing / of in both side of the equation x; = / of Xo + o,e, have
01
X0
E.
of
Let X1
X1
then & Xo + o i.. X, is a VE process. The inverse transform Under this transformation, the training obejective of DDPM becomes
Vi-at"
Vi-af
is given by 0z
61
Vi+o?
2
J(6)
Eypoee p((x))x((e)
colv1-070 - 07 + o, (M 1)t) c
V 1 x,x0
Dooo (xp(x))t)
cx(x;, (M - 1-))
Ea (1) (p)x
1-02
X;
co
- 1)t) (M- X0
10
v of
V of
1 Eppors ple)
x;
(M 1)t
x0
01
VI + 02
where we use the relations & objective with the EDM's general training objective
to get the last equation. Compare the training
Vi-oz
Vi+0?
_ = (a) u
DDPM (ā Ep(x))))e()) (()))))((ee))))) Xoo Xx + δ) X xol?2
where xoe (x, 8) Cskap (DPPM( (ō)x + Coul
DDPM (ō)E (D)M IDDPM (ō)).
we get
DDPM (o+) = 1, skip
DDPM (ōt) = -ôt;
Cout
DDPM (ā,)
DDPM DDPM(G.)
(ā.) = (M - -1)t.
v1 + o.
noise
And the training schedule is given by
1
DDPM w
g2
DDPM (8)
(
o
otgl((e,,
E.2 CONVERT VE SCHEDULES EDM FORMULATION
Review of EDM, F-pred training objective. EDM Karras et al. (2022) proposes the "unit variance principle" to derive the EDM precondition. Recall that the denoiser has precondition Xoo (x, r) <kipx cout Fo cgoise) where F, is the neural network, then the effective train-
ing objective is given by
J(F) (I) ( w EDM (o (ptx)))p)))) Xoo (Xo + oe,o) xpl12
Emn (a) FDM (a (o)com²
out? E p(ex)).(e)) F (c x,, c
Xy ea
skip X.
cout
where x, = Xo + oe. The unit variance principle is given by
Var(cin in x,) = 1,
skip cgut
ca
Var
1,
FDM (p) =
w
18

---

Published as a conference paper at ICLR 2026
Then get the explicit expression of the precondition follows:
o jata
EDM
EDM
(a) 00² + data
data
Skip
x² + daaa
Foul
1
EDM
noise (a) In
V0² + data
4
And the training schedule is given by
a² + data
EDM
(o ddata) )2
EDM (o) exp# V(Pmman P(a)
(Lu and Song, 2024) shows that when or sin( {t), the EDM training objective equivalent to v-pred of VP process.
NCSN (VE), e-pred training objective. al., 2021b) is given by
The training objective in NCSN(VE) formulatinn (Song
f(e) NCSN
Eypcss (o) Epixol /|eo co (xo + oe, In (2) +
where p NCSN (o) e (In min In mmx)): ). i.e In U(In m In m Then we can convert
to the EDM formulation:
co Xo + ae, In (2) + tell
NCSN = (o) p(e)
σ ()) + Xx X0 2
(p Ep(x)) p(e)
Eg xo + oe, In
Xa + oes (xo + oe, In
0 ( xo
(ppoo(o) Ep(x)).(())
Then we get the explicit expression of the precondition as follows:
NCSN
(a) = 1,
NCSN (0) =
skip
(O))N()) 1, Cin
NCSN
NCSN
0 (a) = In ?
Coise
And the training schedule is given by
NCSN
w 02¹
p
NCSN = exp4 (In miin In mmxx).
E.3 CONVERT FLOW MATCHING SCHEDULES TO EDM FORMULATION
Flow Matching, v-pred training objective. In original Flow Matching paperLipman et al. (2023), Px(x.) is the noise distribution when 0 and becomes the data distribution when 1. To align with the time line of diffusion models, we revert the original Flow Matching construction, iee (x;) the data distribution when 0 and becomes the noise distribution when Then the training objective in the Flow Matching formulation is given by
J/M (0) = E, (p(x)) V| (axx) + to,,,t) t) - (E 16-x0 x0
where ar = 1 o, t (t) 7(00 1). In order to convert the Flow Matching formulation to the EDM formulation, we should transform the flow matching diffusion process to the VE diffusion process. Dividing (1 ox) in both side of the equation X; (1 oz)x0 o,e, we have
Xo
az
Let x; = transform is given by or It Under this transformation, the training objective of Flow Matching becomes
x I- δ. =
I-o; then x; satisfies x; Xo + ô i.e. x, is a VE process. The inverse
J!Y) (0)
(oox (E =x
19

---

Published as a conference paper at ICLR 2026
Xx -x=0010
(o)) (ptaa
0:
(p(x)).p(e)
03
||x oxvo(xe; -xoll
1 1
2
&1
oi Vo 1 ò
= R) (x(x)))pe
X
Øt
1
0z
1 δz + oz
where we use the relations x objective with the EDM's general training objective
____ T- or
T+ get the last equation. Compare the training
(() Bo Xxue (Xo + oe,o) -xoll?
we get
1
Caip (ō) =
FM Got
(0) =
1 1+0 0'
1 + 8
PM (0) I+ 0'
FM noise
(6)
1 5
And the training schedule is given by
FM (1 + 6)2
FM (6) ( (0, 1).
Stable Diffusion 3, v-pred training objective. The Stable Diffusion 3 frameworkEsser et al. (2024) also uses the Flow Matching(Recttfied Flow) diffusion process and constructs v-pred training objective. The difference that SD3 proposes the logit-normal noise schedule, i.c.
1 Sv 2n t(1 - t)
1
(log - - mm)²
Pio (1; m, 8)
exp
2s²
(Esser et al., 2024) shows that m = 0, = consistently achieves good performance. Let p SD3 (t) = Pta(t; 0, 1). Then the SD3 training objective is given by
S (0) Eyso(()) (p(x))((e)) IIva (a,x0 - o,e,t) - (e - xx))1
It's obvious that the SD3 objective has the same precondition with FM, i.e.
1
SD3 (+)
SD3 (Ø)
1 1+6
1 +
SD3 (ō) 1+6"
SD3 Boise
(6)
1+
Since 8, 1-a; 1-t = logit(t). then SD3 (o,) = (0, 1). So the training schedule is given by
de
(1 + 0)²
SD3
"
8²
SD3 (6) = exp 1).
F PROOFS
F.I PROOF FOR THBOREM
Theorem 4. The optimal loss value for clean-data prediction defined in Eq. (4) is:
J (xx)* (pp(x))|xxo) Ex(x.)
| (x(xx xx)) [xo]
J
Ep (xx) (xx))
B:
Proof. According to Appx. D, the optimal solution of the network for alean-date, prediction given
by Xoe (Xe, f) = E[xo Xe], where x, a,Xo + o,e. And the objective is minimized if and only if
Xoo (x,, Ex xx), and the optimal loss value is given by (pp) |TExx xx,] xolla. So
20

---

Published as a conference paper at ICLR 2026
have
x(xa)*
E po(x [|xxx xxe xoll
Bp(xa) (xool Bpix.) Bxx xx0]
= pp(e ii (26) s(xo)"
and
F.2
PROOF FOR THEOREM 2
The following theorem is detailed version of Theorem
Theorem 5. For a given subset (x0'>>> the BoDOL estimator given in Eq. (11) converges a.s. the following expression as M- oc:
(
Ke(aixo (+) + tolexx)) + bx Ke(axxo ( tote,x0 (
Σ xD
BeDOL. Ep(e)
Σ
4
I.
Σ Ke(axx)" + to,exx0 + K +o,e,x1)
So BeDOL is consistent estimator, AC > 0. Furthermore, the BeDOL estimator with subset size has the same expectation as the SNIS estimator BANIS with subset size L when M - co, C - 00.
Proof. The cDOL estimator is given by
Σ
L
(1) X
Kx(x
(m)) xxx) xx Ki(x) Xx0 (((
(1a) (166 (ba))
M
[=
BeDOL.
X
where Xmn
(m))) x( + 2 (xx, (m) (1m)) K
,.0
F=l,
(1) Given subset {x0 11-1 since (1 m== and {e,, 1 are i..dd respectively, so {xa axx + oeem m==1 m== are i.i.d random vectors. Then { are iid. random variables. By the strong law of large number (Durrett, 2019), the estimator B BeDO converges almost surely to the following expression M oo:
(1) Ki(axxo 1 + torexx (1) + tx ( Kr(axx + -,E (8)
X
BeDOL M- EX = Bi
|
Ki(axx + x()xx("))) + (il 1 +
(
(1) X
Ki (0) + x (()) fx00) (4) Ki(ax (6) -o,ex (0)
1
=Ee
I
Ki(axp (0) to,ex + (K(aaxx ( to,exx0)) (1)
This completes the first statement. Since
(1) Kx (m))) (x())
1.-00, Ex(xo) (Xo Ke(,)" (1)) ;x)]],
X
L
E
L Kx )11) Loog Ep(xo) [x (1i)) xx)]. Vm,
(m)
L
21

---

(1') Published as a conference paper at ICLR 2026
C(L ((L-1) X ( K2 (1) + t,,exx0 Ltgg... 0,
C(L K2(axx0 + oze, xf) 1.000.00 0,
then Xmn a-s 7
p(xulx/!m)) [xu] = Bxley)lo (x) (1)) xo01
So we have
Ep(xo) [K, (xe) xx))
DOD M. Bp(x) |x
AC' 0.
Hence, BeDOL o | [xol| AC > i.e. BeDOL is consistent.
The SNIS estimator with subset size L - 1 given by
1 (1) (m)
Xg Kxx xx )
BSNIS
Ym where Ym
M
Ir=l K((x, (m) (())
Notice that {x, the strong law of large number, BSNIS converges to the following expressions almost surely:
( mm
are i.i.d. random vectors, (Ym J. are i.i.d. random variables. by
1-1 (1) Σ 1= X
K (x,, (2)
BSNIS M go EY Ex,
Σ/=1 Xx (2))
Next, we take the expectation of the subset for the cDOL and the SNIS estimator, respectively. Since {xD (1) 14 are i.i.d. random vectors, the expectation the cDOL can be simplified as:
x (1)
"K(aaz) ( x,,e,x K.(axxo"" to,,x (1)
Σ
1
E (x)))
B
BeDOL E
cx(x)))) Fia L
Σ
tfi
1=1
K.(oxx +ore,x!") Kelaxo + ta,exx! (1))
1'
L-1
Σ
(L) K((ax + xxc,x10 Kg(axx! (L) + x + ex
E
c,(x0
6-1 (L)
1m1
1-11
Σ Ki(aax( (L) +oTExx! + (Kilaxx!" (4) +0,E,x01 (D))
P=1
Notice that x, 0x o,e, when C + 0c:
X Ki( (L) + o (0)
Br DeDOL C-000
E
(x
(, ((x))
Kelaxx! (4) +o,e,x
Σ 1=1 Kr(x,, x(1))
E x. (x )
-= K (X1, x!² (11))
The expectation of the SNIS estimator is given by
ElELl-1) Xy (1) Ke(Xe, (())
E
1 [x
B
E
(xx)) Ex, Y
E
xx yim!
1-11
Eve(c-1) Ke (x x
So conclude that the ÊeDOL estimator with subset size L has the same expectation as the SNIS estimator βSNIS with subset size when M co, C 0o.
22

---

Published as a conference paper at ICLR 2026
G BACKGROUND ON IMPORTANCE SAMPLING
Assume is random variable, f R² + R is vector value function. Our goal is to estimate the expectation f(x) under a given probability density function x (x). that is,
I (xx)xxxx)xx
However, if ((x) is significant primarily in regions where T(x)) is low, the standard Monte Carlo estimator may provide poor accuracy due to infrequent sampling of these regions-aa an issue often referred as the rare event problem. Importance sampling is designed address this challenge.
Note that
f f(x)
((x
((x)dx = / ((xx)xxx) ((x)
I =
where w(x) x(x) is the weight function and ((x) is the probability density function of the pro- o(x) posal distribution. The importance sampling estimator is therefore given by
iss
1 N Σ f(xx)u((x;),
X;
iid.
g(x).
The estimator a powerful tool when both (x) and q(x) are known exactly. However, when the densities are only known up to normalizing constant (i.e., we can access only (xx) = Zz
x(x) and
s(x)) the standard importance sampling estimator cannot be applied directly. In this case, ((x) = Z
=(x). self-normalized importance sampling (SNIS) is used. Define w(x) [(x
then
(xxxx(x J (x())((x)dx
The SNIS estimator is thus constructed as Σn f(x;)x(x;)
iNIS
Ltd g(x).
Σi (x;)
We can see that the estimator unbiased:
1
Eis
B B[[(x))((xx))
N
N
f(x))w(xx)((x) dx;
fx)) dx =
Unfortunately, the SNIS estimator is biased, but it is asymptotically unbiased and remains consis- tent estimator Sanz-Alonso and AI-Ghattas (2024). The precise definitions are as follows.
Definition 6. Assume i, {I,} are estimators for Then we have the following definitions:
i is said to be unbiased if gi I.
{ is said to be asymptotically unbiased if Ei, -> I as n + co.
Definition 7. Assume i, is the estimator for I, Vna > 0. Then i,, is called consistent for I if i, P
as n Dc.
Proposition 8. Assume [f(x)]] < 1, (x(x)((x)) < co, Ex < oc, then the following holds:
1. E I 1||
Eetse (xc)
(2 (2) =
2. [I - I)]l <
2 [ N' (E,co) [x
23

---

Published as a conference paper at ICLR 2026
So can conclude that the SNIS estimator is asymptotically unbiased.
For completeness, give the proof of the proposition. The proof is modified from Sanz-Alons, and Al-Ghattas (2024).
Proof. To simplify our notation, let
JN Σ f(x:)((x))
PN Σ (xx),
x iid q(x).
Then iSNIs JN//NN PN.. Notice that
E (x( (x)) Bx(x)) (x))
sSNss I INIS
INIS Buts)) [x0x)) ] - Bx(x)) [[)))x(xx)
(0(cx)((((x))
ssIs (xx(x)(0(xx)) - PN)) - [(xxx0 -Nx) By((x) [xx(
Since
|x y11² & (||= + l1yl))² & 2(11x||2+11x/) + +1y1133,
then we can use the identity to bound the variance of isNIs:
E EsIIS
I||2
E [ ([[x(x))(((x))) PN) +E (B ( - -xx)
(zy( [xx(()) (x)))
(B [xx()) Pw]3] + (R)) Jx
E
(x)))) p (x)))2
Var + Var(3N))
(Lg(x) [20 (x)])²
Since Var(PN) Var((((x)), Var(JN) (xr(f(x))(xx))) then we have
2
E IsNIS I|
(Var(((x)) + (ar(f(x)((())))
(Exi))) (x))
2
(kxix) (x) Eg0 (x)(())(())))
(Bytx)) [1ū x
(x(()) ( (x)
N (x)))) ( (x))))²
where we use ||['] < 1 to get the last inequality. Hence, we have proved the first result. Similarly,
can prove the result for the bias. Since E JJ - (x00) [[x)x(x] 0. E PN [xx)]](( = we have:
[ - 1)]ll = 1
ISNIS (Bx(x)((((x)] -PN) - (Ex(s) J.N
[x(x) [x)]
(ISNUIS - 1) (1xxo)((((()) -Px)]]
IsnIs - -I 11]) E (E P:N) '7) 3/
1
E
(x(())(ū((())
1
ISNIs - 1| 11 (xx)))(((())))
E
[xxxx)((((x))
N
24

---

Published as a conference paper at ICLR 2026
By our first result, E EIiis - T|| < N (Bagor [x]]]] then we have
xx( [x)) 1
(xxt)) (x)))
E gx(x) أ (x)
[[[SNI I)ll [x)]
(Bg())) (0 (x))))
N
(Lm(x)) |اا {x))}}
So have prove the second result. When N oc, [[SNNI is asymptotically unbiased.
DIll - 0, then the SNIS estimator
Assume the dataset {x6 liml ~ (x)a(()). Then then estimator
N lid.
EnElN x0 (x,,,, x("))
ENEE[N] (x( x )
is the SNIS estimator of the posterior expectation E(xo xe). By Prop. 8, the SNIS estimator is
asymptotically unbiased. Then the estimator
EnEINi X (x,, xx )
(m)
B = 1 Σ M
Σn'E[N] (x (m) X (x²))
ME(M]
also asymptotically unbiased. For more details on importance sampling, see (Robert al., 1999).
H EXPERIMENTAL DETAILS
H.1 OPTIMAL LOSS ESTIMATION
In this subsection, we show more experiments results of our cDOL estimator, Our cDOL estimator concluded in Alg.
Algorithm The corrected Diffusion Optimal Loss (cDOL) estimator
input Diffusion schedule a and oe, diffusion step t, training dataset (x6 (m) Ieg number of re- peats R, data sample Xo subset size L, x sample size M, correction parameter C.
output Estimation of the diffusion optimal loss r(xo)*
1: for E [R] do
2: Sample data subset (x(==() JEEL
(r.() (1ee)) independently randomly from (x(") (x(")) IneiNji
3: for m [M] do
4: Sample an index Ln randomly from [L];
5: Construct x
(1.m)) axx)
(r.ba) o (m)) with (T)) NO,,
6: end for
7: Compute B3 (r using Eq. (11) (where K, is defined in Eq. (8)):
8: end for
9: Compute DEDOL 1 Ere(r (rDOL and À using Eq. (7);
10: Return À DEDOL
Convergence of cDOL estimator. Our cDOL estimator has four parameters (R, M, T,, C). As mentioned in Sec. 3.4, we empirically choose C = 4N/L, the subset size L can be taken to fully utilize memory. The parameters R, M should be large enough to ensure that the estimator converges. We perform a convergence analysis with respect to Rt MM in the CIFAR-10 10 dataset to justify our choice of M, R. The results are summarized estimator will converge when R is large enough and can approximate the ground truth optimal loss accurately. Empirically, we find that R 3N/ is enough for an accurate estimate. Next, we verify that our cDOL estimator will converge when M is large enough. As shown in Fig. 6, we can see that the cDOL estimator converges when M 4L.
Fig. and Fig. 6. As shown in Fig. 5, our cDOL
Efficiency. The computational complexity of the naive estimator B, (Eq. (9)) is O(N?), where N the size of the dataset. The eDOL estimator reduces this complexity to O x R). Based on the
25

---

Published as a conference paper at ICLR 2026
verifications the the previous paragraph, set the subset size L to fully utilize memory, and is often set to 2500 or 5000. We set L. M = 4L, then the total complexity becomes O ( With this setting, the total running time approximately 0.5 hour on CIFAR-10, 2.5 hours FFHQ, and about day for the ImageNet dataset when using 2400 8G CPU cores.
1o5s 1og 0.225
Optiina 80g0-2.5
Ustimaa ligembus
A.C
-0.00
Optimal lass for log dw 1.2.25
Optimnl oss for log pw1.5
6.10
0.90
0.00
4.02
4400
0.48
-341
Oppama loss pw
Ostimal uss fog 2-2525
Optimal log
0101
120 co
1000
(20)
Figure 5: Convergence of our estimator with respect to the number of subsets R on CIFAR-10. We plot the estimated optimal loss v.s. R for several choices of C among different noise scales. For fair comparison, we fix the subset size L = 5000.
1og.0-
Ustimal 2.5. L-5000
031114
03300
4.7110
131100
0159
4,000
0200
01.4
Optimmal lowe for Ing om 1-5000
Oprimm lows for Ing p=0.75, 1-5000
6.00
-200
0:28
Figure 6: Convergence our estimator with respect to M (the number of x; samples) on CIFAR- 10. We plot the estimated optimal loss v.s. M for fixed C 40, 5000 among different noise scales.
26

---

Published as a conference paper at ICLR 2026
H.2 DIRECT COMPARISONS WITH WORKS ON OPTIMAL SOLUTION
Xu et al. (2023) proposes an SNIS estimator for the inner expectation of the optimal loss in Eq. (6). If x, is sampled independently from batch separate from the Xy batch used for the inner expectation, then the estimator reduces to the SNIS estimator described in Sec. 3.3. By contrast, if the same Xo batch is used to sample X and compute the outer expectation, this corresponds to our DOL estimator. As shown in Fig. 1, the SNIS estimator suffers from high variance, leading to poor empirical performance. Meanwhile, the DOL estimator introduces extra bias and also does not achieve good performance.
Nicdoba et al. (2024) proposes a nearest neighbor estimator of the optimal solution. Given a noisy sample xt, the KNN estimator finds the K-nearest Xg samples in the dataset estimate the optimal solution. The KNN search method used in Niedoba et al. (2024) (Faiss with a flat index) has ((N) complexity per query x,, leading to overall complexity of (N2), which matches that of the naive estimator B, (Eq. (9)). Moreover, KNN search requires significantly more memory, as needs to generate an index of the entire dataset, which prevents effective multithreading parallelism.
For direct comparison, we report the error rate of each estimator, defined as
J Jesimated ground truth
J Jground truth
We consider the error rate more suitable than absolute error, since the scale of the optimal loss varies significantly across noise levels. For fair comparison, we tested L = 2500 for the cDOL estimator and n K 2500 for the KNN estimator, the results are shown in Table 4. We can see from the results that cDOI. achieves comparable accuracy and variance to KNN, but with significantly lower runtime: approximately 5x faster than KNN (n 2500, O(Lx N) (with It & N/L) versus ( for KNN. Moreover, cDOL benefits from straightforward multithreading parallelism since it only loads samples into memory at time, making it more scalable for large, high-resolution datasets such ImageNet.
2500) due to its lower complexity
Table 4: Comparison between different estimators on CIFAR-10 for an intermediate noise level
log σ = 1.25.
Methods
DOL
cDOL (L = 2500)
KNN (n 2500, K 2500)
SNIS
error rate
0.55
0.04
0.03
5.72 0.0210
variance (per dimension)
0.0170
0.0182
0.0183
run time
12 min
12 min
66 min
H.3 DETAILED SETTINGS FOR FIG. 3
Following EDM (Karras et al., 2022), we configure our training settings as follows. We train all models on CIFAR-10 until a total of 200 million images have been sampled from the training set. The batch size is set 512. For sampling. we employ the EDM deterministic sampler, consistently
setting the time steps according to o, (a -ーー 0.002, max 80, and the number of function evaluations (NFE) set to 35 for all models.
(a min = ),, where X Tmin
For training loss calculation, we evaluate the clean-data prediction loss for fair comparisons. In practice, we perform inference over three epochs estimate the training loss across noise levels and observe that these estimates exhibit good convergence. To justify the convergence of training loss estimation, we present additional results using a model trained with the DDPM schedule (VP- in Table 1) on CIFAR-10. We computed the mean, variance, and standard error of the mean from 150,000 independent evaluations (corresponding to epochs) of the training loss. The results (Table 5) demonstrate that although the variance depends on the noise level, accurate estimates can be obtained by increasing the number of model evaluations (the standard error of the mean is very small).
We also investigate the effects of sampling methods. For a fair comparison, we use the EDM sam- pler with identical parameters to evaluate different training methods in Fig. 3. Additionally, conducted experiments with the 250-step DDIM Song et al. (2021a) and Flow Matching (Lipman al., 2023) samplers, with the FID results presented inTable 6. These results show that while sam-
27

---

Published as a conference paper at ICLR 2026
Table Training loss statisties across noise levels using DDPM schedule on CIFAR-IO.
Noise levels
Mean
Variance
Standard Error of Mean
log a
0.230
0.015
3.95e-5 8.77e-6
log a = 2
0.130
0.0033
log a 0
0.025
2.23e-7 8.67e-5
log -2
0.0026
7.41e-7
1.91e-9
pling quality depends on the choice of sampler, a better-trained model consistently achieves higher sample quality across different samplers.
Samplers
EDM
DDPM
NCSN
FM
FM our schedule
EDM sampler
1.94
1.97
2.72
2.36
1.79
DDIM sampler
2.14
2.23
2.91
2.27
1.99
FM sampler
2.19
2.25
3.07
2.28
2.04
Table 6: FID results across different samplers and training methods with 250 sampling steps.
H.4 COMPLEMENTARY RESULTS FOR FIG. 3
As we primarily evaluate the model by the FID metric in Fig. 3, we give some complementary results for Fig. this subsection.
Precision and Recall metrics. As the FID metric is a mixture of the sample quality and diversity, cannot reflect the sample diversity and quality separately. The precision and recall metrics are designed to test the sample quality and diversity, respectively. We evaluate our models trained by different training schedule and formulations Fig. 3 under these two metrics. The results are shown in the following Table 7.
Training schedule
Precision
Recall
EDM (Karras al., 2022)
0.615
0.682
DDPM(Ho et al., 2020)
0,608
0,683
FM (Lipman et al., 2023)
0.615
0,677
SD3 (Esser al., 2024)
0.595
0.694
NCSN (Song et al., 2021b)
0,614
0.647
Ours schedule
0.626
0.667
Table 7: Precision and recall results for different training schedules.
Combining the results in Fig. 3(a,b) with the results shown in the table above, we observe that the precision metric also has stronger correlation to the training loss gap in the small noise regions. Thus, our training schedule outperforms all other training schedules under this metric. In contrast, the recall metric has stronger correlation the training performance in the larger noise levels around the critical point o*, thus the SD3 training schedule achieves the best performance. These results justify the intuition that the image quality related training performance small noise scales, while recall or diversity is related to larger noise scales.
Memorization metrics. As shown in Fig. 3(a), the training loss gap is large for all mainstream diffusion models. This implies that these diffusion models are still not overfit to the optimal solution. To study the memorization behavior, we follow Gu et al. (2023) for the metric and the experimental settings. We train model using Flow Matching precondition and our training schedule of CIFAR-10 10 with 5k data samples, and we train the same architecture with Flow Matching training baseline. We report the memorization rate, where sample is
subset
schedule the same dataset memorized its L distance to the nearest neighbor is smaller than 1/3 of that to the second nearest neighbor in the training data (Gu et al., 2023). Here the factor 1 1/3 is empirical threshold proposed
28

---

Published as a conference paper at ICLR 2026
in Gu al. (2023). The results are shown in the Table 8. We can observe that our schedule improves the generation performance without leading to severe memorization.
Training schedule/Training Epochs
0.5k
1k
1.5k
2k
2.5k
3k
3.5k
4k
Flow Matching schedule
0.0
0.0
0.0
0.0
0.0001
0.0024
0.0102
0.0224
Our training schedule
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 8: Memorization rate across different training epochs.
Human preference study. We also conduct the human preference study to further evaluate the generative performance of different methods. We randomly generated 24 image pairs (Baseline Ours) from the ImageNet-256 model. 19 independent evaluators were asked to select the image with better visual quality. The results are summarized in Table 9 and show that our method was preferred in 55.26% of the cases, while the baseline was preferred in 44.73%. This human evaluation aligns with our FID improvements, confirming that the reduction in the "Loss Gap" translates into perceptibly better image quality.
Training schedule
FID with guidance
Human preference
LightningDiT (Yao et al., 2025)
1.42
44.74%
+Ours schedule
1.30
55.26 %
Table 9: Human preference between baseline and our schedule.
H.5 IMAGE GENERATION
Following EDM (Karras et al., 2022), we configure our training settings as follows. We train all models on CIFAR-10 until a total of 200 million images have been sampled from the training set. The batch size is set to 512. Checkpoints are saved every 2.5 million images, and we report results based on the checkpoint with the lowest FID. We adopt the DDPM++ network architecture used in EDM, with our primary modifications being the incorporation of our loss weighting scheme and adaptive noise distribution. All models are trained on 8 NVIDIA A100 GPUs. For sampling, employ the EDM deterministic sampler, consistently setting the discretization steps according to
σ; = (omax + N- (a min mixx)) where و = 7, min 0.002, 0 80, and the number of function evaluations (NFE) is set to 35 for CIFAR-10 experiments.
For ImageNet-64, we follow similar setup EDM. We use the ADM architecture, which matches that of EDM (Karras et al., 2022). The batch size is set to 2048, and our loss weighting and adaptive noise distribution from the training set. Checkpoints are saved every 10 million images, and we report the checkpoint with the lowest FID. All ImageNet-64 models are trained 32 NVIDIA A100 GPUs. In sampling. we again use the EDM deterministic sampler with ò and NFE- = 79.
applied as well. Training proceeds until 2.5 billion images have been sampled
For ImageNet-256, we adopt a setup similar LightningDiT (Yao et al., 2025). Specifically, we uti- lize VA-VAE (Yao et al., 2025) as the tokenizer and implement a modified LightningDiT (Yao et al., 2025) architecture enhanced with Nommomnalinntion (Dehghani et al., 2023) to improve training sta- bility. The batch size is set to 2048, and we apply the same loss weighting and adaptive noise distri- bution strategies. Optimization performed using Adam W with parameters (31. (3)) (0.9, 0.95) and a learning rate of 2 x 10-1 The model is trained for 1600 epochs (approximately 10 million iterations), with checkpoints saved every 10,000 iterations. Again, we use 32 NVIDIA A A100 GPUs to train the model on ImageNet-256. Consistent with LightningDiT (Yao et al., 2025), we employ the FM Euler ODE sampler with 250 function evaluations (NFE 250).
Regarding our adaptive training schedule, we maintain bin that records the training loss gap for each noise scale. Inspired by the adaptive schedule proposed by Kingma and Gao (2023), we update this bin using an exponential moving average (EMA) during training. Specifically, the bin is updated every time 2 million images have been drawn, with a decay rate set to 0.9. The collected training loss gap statistics are then used construct a piecewise-linear probability density function, which
29

---

Published as a conference paper at ICLR 2026
Comparison between Noise Schedules
0.00
0.06
0.0
log 0
Figure 7: Plot of our proposed noise schedule on the CIFAR-10 dataset. We plot the noise schedule calculated by the model's loss gap the final optimization step of the training process.
Figure 8: Samples generated by our ImageNet-256 model.
serves the adaptive noise schedule. As mentioned, loss weight is given by
We = a minf J; w= + f(a)
Typically, w* is set to be 20 and 1/50. [(a) Nllooo: 5²) is an additional weighting function to let us put more weight on the region a < o², which is simply set as normal pdf. We set == -75, 2 for CIFAR-10, μ 7.75,, 2 for ImageNet-64 and /z -4.37.5 = 1.75 for ImageNet-256. In Fig. 7, we plot our noise schedule calculated by the model's loss gap in the final optimization step of the training process. We can see from the result that model allocates more optimization steps on the positive region a < a², as shown Fig. 3. We also show some samples generated by our model trained on the ImageNet-256 dataset in Fig. and Fig. 9.
H.6 SCALING LAW
In this subsection, we provide comprehensive account of our scaling law analysis. Our experi- ments employ the state-of-the-art diffusion model EDM2 (Karras et al., 2024), with parameter counts ranging from 120M 1.5B. In accordance with the training protocols outlined in EDM2, models trained in the RGB space for ImageNet-64 and in the latent space derived from a pretrained VAE for 1m1.eNet-512.
We begin by reporting the results on ImageNet-64. As described in Sec. 5, we apply our modified scaling law (Eq. (13)) to model performance. In Fig. 11, we present a detailed comparison between the original and modified scaling laws across various noise scales. Our findings indicate that the modified formulation yields a loss envelope that adheres more closely to a linear relationship. with corresponding improvements in the correlation coefficient, especially large noise scales. The
30

---

Published as a conference paper at ICLR 2026
Figure Samples generated by our ImageNet-256 model.
Total Loss Gap -Jp (p=
0.98)
5.2 x0-1
10
EDM2-5
EDM2-M
EDM2-L
EDM2-XL
1
EDM2-XXL
Envelope
4.85
0.95F-0014
x0
Flops
Figure 10: Scaling law fitting results using the modified power law in Eq. (13) for the total diffusion loss on ImageNet-512.
enhancements at small noise scales are less pronounced, largely due to the relatively minor optimal loss values compared to the models' actual training losses.
Subsequently, we extend our analysis to ImageNet-512. Mirroring the experimental setup used for ImageNet-64, adopt the optimized adaptive loss weighting from EDM2 (Karras et al., 2024) when calculating total loss. The results, shown in Fig. 10, achieve correlation coefficient of p 0.9857, with the fitted scaling law given by:
J(F) = 0.9493F 0.004 + 0.001.
thorough comparison between the original and modified scaling laws across multiple noise scales presented Fig. 12. These results are consistent with those observed ImageNet-64, once again demonstrating that the modified scaling law yields a loss envelope that is closer to linearity, with higher correlation coefficients, particularly at large noise scales. As before, improvements small noise scales remain limited due to the diminutive size of optimal loss relative to training loss.
31

---

Published as a conference paper at ICLR 2026
Training Loss at log a 3.75 (0 0.87)
/ap -/"
2.08
EDM2XX
EDM2XS
EDIMAS
EDM2-M
EUNZ-LL
EDM2
2x
LUMA-1
Bruedope
J= nc3
18x
J= 2.617F
2:35x121
x
10
2.3M x
1p12
1130
10
12
Plops
Flops
Training Loss Jo at log a 3.12 (p 0.9589)
Training lss Gap - at log 3. 12 (p 0.96144
2.000 12
20M
EDM2:L
EDM2-0
180
Ervelope
11
11
J 0
12
2,06
21
301H
1220
Flops
Training Loss o -1.25 (p 0.99)
Train Training Loss Gap at log
1.22 (o 0.99)
10M2-X5
783341
EDM2 14
MM0-5
7x
EDN2
ITNP-
EDM2
LDN2-M
59x
IDN24
froelope
14x
3000
66X12
65x10
15x30-3
Flops
Flops
Scaling law study on ImageNet-64. Each row corresponds to different noise scale. The left column shows the raw training loss values, while the right column displays the training loss gap relative to the optimal loss each noise scale. We observe that in the modified version, the envelope aligns more closely with straight line, particularly at larger noise scales. For smaller noise scales, the improvement less pronounced, since the optimal loss still very small compared to the training loss of the model.
Figure 11:
32

---

Published as a conference paper at ICLR 2026
Training Loss Jo at log a= 4.38 (p= 0.8271)
Training Loss Gap -) at o 4.38 ( 0.9560)
E3ND-5
25% 30
LBMZ-L
B3921
EDM2-XL
su8o2
14
Cnvelace
EDM2-X0
03003
1000
Etrange
J- 0.324F
24
30
23
109
Flops
Flops
Training Less je at = 3.12 (p 0.9355)
Training Loss Gap Jo at log a' 3.12 (p 0.9363)
EDM2-5
2.73 10
EDM24
EDM2-L
EDM2
ED/24
EDM230
237x1004
Ervelope
7264 16-1
/-02332--000
530 10
0.042
455 km
2.00
150x10"
2.702 10
2200 10
10tH
204
19
29
Flops
Flops
Training Loss f/ at lo -1.55 (p 0.9853)
Training Loss Gap -) at -1.25 (p= 0.9853)
2.
FRIM
ED2
EDM2
13x 30
EDM2 XXL
EDM2
Enowlope
25x 10
ED/4
10
J-
28x10
275
10
Flops
Hops
Scaling law study ImageNet-512. Each row corresponds different noise scale. The left column shows the raw training loss values, while the right column presents the training loss gap relative to the optimal loss each noise scale. We observe that in the modified version, the envelope aligns more closely with linear trend, especially for larger noise scales. For smaller noise scales, the improvement less significant, as the optimal loss still very small compared to training loss of the model.
Figure 12:
33


Reviews:
[{"review_id": "ULLMI21uaf", "reviewer": "Reviewer_AWVe", "summary": "This paper proposes to estimate the *optimal loss value* of diffusion models — a quantity that depends solely on the dataset and the diffusion setting, not on model capacity. The authors provide a theoretical derivation of this optimal loss under a unified formulation of diffusion models, and develop consistent and scalable estimators that remain stable and “not too greedy.” These estimators allow a principled analysis of model training quality, identifying where diffusion models underfit across noise scales. Leveraging these insights, the authors propose a new training schedule that equalizes performance across diffusion scales and leads to consistent improvements across various datasets (CIFAR-10, ImageNet-64, ImageNet-256). Finally, they show that subtracting the optimal loss from the observed training loss yields a more faithful scaling law, providing a better theoretical understanding of diffusion model scaling.", "questions": "- Intuitively, image *quality* tends to be related to predictions at small noise scales, while *recall/diversity* corresponds to large noise scales. It would be interesting to compare the loss gap across these regimes with standard Precision and Recall metrics — this could provide an even more interpretable connection between the optimal loss and generative performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The paper answers and analyzes questions that have been so far addressed only empirically — *finally!* The EDM sampling schedule has performed remarkably well for years, and this work explains *why*. Such principled analysis is much needed in a community increasingly driven by empirical results rather than theoretical foundations.  \n- Strong and complete theoretical analysis of the proposed estimators, including bias–variance tradeoffs and formal proofs of consistency.  \n- Extensive and well-structured experimental validation, showing clear and consistent improvements across multiple datasets and model variants.", "weaknesses": "- The presentation quality is poor. Example given: Figure 1 is hardly readable without zooming, and inline equations significantly hurt readability. Inconsistent line heights (e.g., lines 130–131, or the paragraph 189–200) make the paper visually dense and tiring to read.  \n- The alternative formulation of diffusion models in the main text feels unnecessary. A single sentence linking Theorem 1 to the alternative formulation in the appendix would suffice, leaving more space to improve layout and readability.  \n- While technically sound, the paper’s exposition could be more accessible, especially in highlighting intuition behind the estimators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wL9vPcSJG7", "reviewer": "Reviewer_LMDA", "summary": "The paper investigates how to estimate the optimal loss values in diffusion models’ training objectives. It first introduces several diffusion losses (corresponding to different prediction targets) and presents them under a unified formalism and notation. Since the optimal predictors for these objectives are given by conditional expectations, their minimal achievable losses still depend on the conditional variances. Based on this, Theorem 1 derives an analytical expression for the optimal loss of the clean-data prediction objective, showing that it decomposes into two terms: a simple dataset-average term and a second, computationally expensive nested expectation. One of the paper’s main contributions is to propose efficient approximations for this second term—the DOL and cDOL estimators—which make estimation feasible on large datasets. The authors validate these estimators on smaller datasets, demonstrating that the approximated optimal losses closely match their true values across diffusion times. They then use the estimated optimal losses to analyze how existing diffusion models approach these limits as noise varies, revealing distinct behaviors across diffusion steps. These observations motivate a combination of an improved training schedule and a loss-weighting scheme yielding better FID scores. Finally, the paper revisits scaling laws for diffusion models, showing that when losses are shifted by their corresponding optimal values, the resulting curves adhere more closely to power-law behavior.", "questions": "- Do the authors have an intuition behind the larger loss gap in the intermediate region? This gap appears to be consistent across various model variants.\n- Figure 1(c): in the paper the authors state: \"The result confirms that the variance increases with C.\" This does not seem apparent in the plot. could the authors please clarify?\n- Could the authors please clarify what is on the y-axis in Figure 2(b)?\n- From my understanding, the proposed loss weight tries to assign more importance to the interval to the left of $\\sigma*$ based on the results in Figure 2(b). This is because the loss-gap positively correlates with FID in that region. However, the same figure also shows a negative correlation in the proximity of $\\sigma*$. Does the proposed loss weight take that into account?\n- Figure 3(b): why the choice of $\\sigma = 4.38$?\n- Figure 3(c): what do you mean by total loss gap?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- From the best of the reviewer's knowledge, the characterisation of the diffusion optimal loss values is novel. \n- The paper's results are quite general as the manuscript is structured in such a way that several diffusion setups (i.e., different prediction targets and noising schemes) can be analysed under the same framework.\n- The use of optimal loss values as a diagnostic tool is interesting as it allows to identify the noise regimes where models are weaker and ultimately provide adjustment with improved nosing and loss-weighting schemes.", "weaknesses": "- Clarity of exposition and results presentation could be enhanced. The notation is quite heavy and the presentation of the results would benefit from more explanations/interpretation (see Questions). For example, it took me I while to understand what Figure 2(b) is plotting. \n- More fundamentally, it is not clear whether getting a smaller training loss value ultimately results in better performing/generalising models. For example, it would be interesting to understand to what extent the proposed scheduler and loss weight impact memorisation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqsrzrhHBF", "reviewer": "Reviewer_yn5v", "summary": "This paper introduce the optimal loss in the parameter designs of diffussion models. The authors propose an estimator of optimal loss. The loss gap idea is then used in the design of noise schedules and loss weights in training objectives. The scaling law of diffusion model is also discussed based on the loss gap other than just the loss value.", "questions": "- In section 2, sometimes the data distribution is $p_0(x_0)$, sometimes is $p(x_0)$. Are they different distributions?\n- Please specify the noise schedule $p(t)$ and weight $w_t$ in line 293 using the notation from Section 2 (especially which superscript should be used).\n- In Line 382, please specify what is v, F prediction.\n- In Line 386-387, is the loss weight discontinuous, which contradicts the continuity of actual loss gap? If so, then why this is a good choice?\n- In Line 396, why $p(\\sigma)$ is chosen to be proportional to $w_{\\sigma}(J-J*)$ instead of for example $(w_{\\sigma} (J-J^*))^a$ where $a\\ge 0$ ? Is there a theoretical/empirical reason for this? What precise noice schedule do you use in the experiments? Is this changing at each training iteration due to different $J_\\sigma(\\theta)$? If there is no explicit expression, please show some figures of $p(\\sigma)$.\n- The strategy of choosing weights and noise schedule is quite confusing to me. In loss weight, you would like to emphasize left side of the threshold $\\sigma*$. Btw, the authors state in the paper but I'm still confused why such pattern (the emphasis of positive correlation) is desired. In noise schedule, you would like to emphasize  $w_\\sigma (J-J*)$ and therefore in total, combining loss weight, noise schedule, and the l2 distance, the loss looks something like $w_\\sigma^2 (J-J*) J^2$ (or maybe $w_\\sigma^2 (J-J*) (J^2-J^{*2})$ since constant term does not affect the optimization). Can the author explain the overall intuition of the schedule?\n- What is the schedule (timestep, discretization) of $\\sigma$? Is this also adapted in some way?\n\n\nMinor:\n- font size in figures is too small", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose the use of optimal loss in training schedule design, and design practical estimation of the optimal loss. They also proposed a new schedule which seems to be some refined version of EDM schedules but focuses more about the nature of loss gap between the true loss and the optimal loss, and the practical performance of their schedule is validated.", "weaknesses": "The proposed schedule is still vague, both in writing and in the actual design (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rOeyj41Py6", "reviewer": "Reviewer_Bp46", "summary": "The paper targets a known limitation of diffusion training losses—namely, that the theoretical optimum is generally non-zero and unknown—making the raw loss an unreliable proxy for data-fitting quality. The authors (i) derive a closed-form expression for the optimal loss under a unified formulation of diffusion models, (ii) propose practical estimators, including a stochastic variant with variance/bias control scalable to large datasets, and (iii) use these tools to diagnose training quality and design an improved training schedule.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Theoretical derivations appear rigorous.\nWriting quality and notation are clear, consistent, and professionally presented.", "weaknesses": "1. Loss–performance alignment under matched settings. The paper argues it resolves the mismatch between diffusion loss (with unknown non-zero optimum) and generative performance. To substantiate this claim more convincingly, I suggest including a correlation analysis between training loss (after subtracting the optimal term) and FID under identical settings and budgets, compared to standard loss formulations. Concretely, report Pearson/Spearman correlations over training, rank consistency across checkpoints, and side-by-side training-curve plots for your method vs. baselines on the same architectures/datasets.\n\n2. Over-reliance on FID. FID alone cannot fully capture generative performance. Consider adding complementary metrics and qualitative evidence. A small human preference study (even limited-scale) would help validate perceptual quality beyond distributional statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)