Manifold Constraint Reduces Exposure Bias in Accelerated Diffusion Sampling
Authors: Yuzhe YAO
Jun Chen
Zeyi Huang
Haonan Lin
Mengmeng Wang
Guang Dai
Jingdong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models
Exposure Bias
Primary area: generative models

Abstract: Diffusion models have demonstrated significant potential for generating high-quality images, audio, and videos. However, their iterative inference process entails substantial computational costs, limiting practical applications. Recently, researchers have introduced accelerated sampling methods that enable diffusion models to generate samples with far fewer timesteps than those used during training. Nonetheless, as the number of sampling steps decreases, the prediction errors significantly degrade the quality of generated outputs. Additionally, the exposure bias in diffusion models further amplifies these errors. To address these challenges, we leverage a manifold hypothesis to explore the exposure bias problem in depth. Based on this geometric perspective, we propose a manifold constraint that effectively reduces exposure bias during accelerated sampling of diffusion models. Notably, our method involves no additional training and requires only minimal hyperparameter tuning. Extensive experiments demonstrate the effectiveness of our approach, achieving a FID score of 15.60 with 10-step SDXL on MS-COCO, surpassing the baseline by a reduction of 2.57 in FID.

Published as a conference paper at ICLR 2025.
MANIFOLD CONSTRAINT REDUCES EXPOSURE BIAS IN ACCELERATED DIFFUSION SAMPLING
Yuzhe Yao'; Jun Chen Zeyi Huang", Haonan Lin', Mengmeng Wang4.5t Guang Dai Jingdong Wang Xi'an Jiaotong University. ZZeeiaang Normal University, Huawei,
*Zhejiang University of Technology, SGIT AI Lab, State Grid Corporration of China,
Baidu
(yuzheyao.22) junc. chanc. mengmewang, guang gdai @mg il com muangzeyy22@uaaerr com, 1 linhaonan@stu. iataoonnnnstu.xttuuu xjtu. www.eedu.coom cn wangjingdongg out com
ABSTRACT
Diffusion models have demonstrated significant potential for generating high- quality images, audio, and videos. However, their iterative inference process entails substantial computational costs, limiting practical applications. Recently, researchers have introduced accelerated sampling methods that enable diffusion models to generate samples with far fewer timesteps than those used during train- ing. Nonetheless, the number of sampling steps decreases, the prediction errors significantly degrade the quality of generated outputs. Additionally, the exposure bias in diffusion models further amplifies these errors. To address these chal- lenges, we leverage a manifold hypothesis to explore the exposure bias problem in depth. Based on this geometric perspective, we propose a manifold constraint that effectively reduces exposure bias during accelerated sampling of diffusion models. Notably, our method involves no additional training and requires only minimal hyperparameter tuning. Extensive experiments demonstrate the effec- tiveness of our approach, achieving FID score of 15.60 with 10-step SDXL on MS-COCO, surpassing the baseline by reduction of 2.57 in FID.
INTRODUCTION
Diffusion models (DMs) Sohl-Dicksted al. (2015): Ho al. (2020); Song et al. 202T 2020);
Nichol & Dhariwal (2021) have emerged as powerful generative models for various applications, including image generation (Rombach al. 2022; Li al., (024b), video synthesis (Ho et al.)
2022), and audio generation (Kong et al. 22021) DMs are able to generate images of high fidelity and high diversity, when compared to generative adversarial networks (GANs) (Heusel et al. 2017 Goodfellow et al., 2014), variational autoencoders (VAEs) (Kingma & Welling 2022 Vahdat & [aauz]]2020 and flow models (Lipman et al. 2023 Papamakarios et 2017 Kingma et al. 2016). However, DMs necessitate an iterative process synthesize high-quality results (Ho et al. 2020).
In response to the computational demands of diffusion models, [Song et al. (2021) propose an accelerated sampling method. Given a pre- trained diffusion model, during generation, in- stead of sampling all latent variables on the sampling trajectory, [Song et al. (202T) sample along a trajectory shorter than that used during training, significantly accelerating the genera- tion. Nevertheless, KKim et al. (2024) find that during the accelerated generation, the error in- troduced to the sample at every timestep is proportional to the step size. Therefore, reducing the
Figure 1: Samples generated with 5, 10, 20, 50, and Ik steps (from left to right). Prompt: "A pro- paganda poster depicting a cat dressed as French emperor Napoleon holding a piece of cheese".
"This work was partly completed during the internship at SGIT Al Lab, State Grid Corporation of China. Corresponning Author.

---

Published as a conference paper at ICLR 2025.
Mr
Manilold
Manitaid or
Manitiil by
Mr-1
Accelentted Sumpling
Compise Sampling
By-k
22
Prefition %6"
Predkttin for Mtazifald Condirint
MI-
Data on Manifild
0000
Data ouside Manifoli
Dats cn Mazifedd cutote Marioii
Mr-3
(9)
2t-
0
Mr-4
M (8)
Mg
(t-k)
M
Mr
(a) Exposure hias in accelerated sampling
(b) Manifold Constraint on Denoising Ohservation
Figure 2: In both (a) and (b), the blue manifolds represent noisy data manifolds. In (a), small step sizes allow diffusion models transport a accurately between manifolds. However, with large step intervals (accelerated sampling), the noisy prediction transports data away from the manifolds, reducing accuracy in the next step. In (b), the green manifolds are manifolds of denoising observa- tions (Equation 6.. The yellow arrow represents the manifold constraint, which projects 20)) toward M allowing error correction in noise estimation and bringing æt-k closer to Mi-k-
number of sampling steps, while improving efficiency, can introduce substantial noise and degrade the sample quality (Rombach et al. 2022). To illustrate how the prediction errors affect sample qual- ity, we utilized Stable Diffusion XI. (Podell et al. 2024) to generate images with varying sampling steps. The results, shown in Figure demonstrate that as the number of sampling steps decreases, the generated images exhibit a noticeable loss of detail and increased distortion objects.
We propose that another issue contributes to the degradation of sample quality during accelerated generation: exposure bias (Schmidi) 2019). This training-inferent discrepancy, commonly ob- served in auto-regressive models, also affects diffusion models (Ning et al.) 2023 Li & der Schaar 2024 Li et al.) 2024a| [ing et al., [2024] Ren et al.] 2024). During training, diffusion models only exposed to ground truth inputs; however, when the input is significantly corrupted (e.g., during accelerated sampling), the models may generate increasingly inaccurate predictions. This cumulative effect is illustrated in Figure 2a
Li et al. (2024)) evaluate exposure bias us- ing the concept of pixel variance within sam- ple and propose a simple yet effective method based on it. Since pixel variance differs sig- nificantly from the sample variance used in
NFE-5
NFE
NFE-1000
Varix)
0.6.
prior works (Ho et al. 2020) Song et al. 2021
Nichol & Dhariwal] [ Dhariwal & Nichol
2021), we are particularly interested in its im- plications, especially in accelerated sampling.
0.4-
0.2
To explore this, we collected the pixel variance for different sampling step settings in the gen- cration Figure and visualized it in Figure The results show that with larger step sizes, the pixel variance deviates significantly from that observed with smaller step sizes. Inspired by
800
700
600
500
400
300
200
100
Denoising Steps
Figure 3: Variation in pixel variance for single sample across sampling settings: 5, 10, 20, 50, and 1000 steps.
prior works (Song et al. 2021 Chung et al.] 2022 Chen al. 2023 Yu al. 2023 Humayun al. 2024 Su et al. 2024 we aim explore the relation between exposure bias and pixel variance of sample under the manifold hypothesis.
In this work, focus on addressing exposure bias in the accelerated sampling process of diffusion models. Our contributions are summarized as follows:
We identify exposure bias as a key challenge in accelerated sampling for diffusson models and introduce the manifold hypothesis, offering geometric perspective this issue.
2

---

Published as a conference paper at ICLR 2025.
Based the manifold hypothesis, propose manifold constraint the denoising ob- servation in diffusion models. This method is training-fre frree nearly hyperparameter-free and effectively reduces exposure bias by correcting prediction errors during accelerated sampling-
Extensive experiments across various image generation tasks and base models demonstrate consistent and substantial improvements with our approach.
2 RELATED WORK
Diffusion Models represent significant advancement in generative modeling. Initially introduced by Sichssttcstttnn et al. (2015). these models iteratively refine data through denoising. Key enhance- ments were made by Ho et al. 2020). who incorporated weighted variational bound in training. Song et al. (202)) introduced a more efficient class of iterative implicit probabilistic models with the same training objective DDPMs. Further developments include Nichol & Dhariwal (2021), who optimized model architecture. Additionally, HH & Salimans (2022) offers method to jointly train conditional and unconditional diffusion model. Additionally, works accelerating sampling
methods (Song et al. 2023] Zhou et al., 2023 Chen et al., 2024] Lu et al. [202aa] b Zhang & Chen
2023
Liu et al., 2022) have contributed their practical use various domains.
Exposure Bias refers to the discrepancy between inputs during training and inference in sequential prediction models (Schmidt) 2019) Waang & Sennrich 2020) /Zhang al.) 2024). This issue also arises in diffusion models, where ground truth inputs are used during training. but previous predic- tions are fed into the model during inference (Li & van der Schaar 2024 Ning et al. 2024 2023] Ren et al. 2024 Yao et al. 2024 Everaert et al. 2024). This discrepancy can lead to cumulaativ errors in the generation process. Various strategies have been proposed to mitigate exposure bias in diffusion models. Ning al. (2023) suggest perturbing training data reflect potential input varia- tions during inference, while & van der Schaar (2024) analyze error propagation and recommend using the upper bound of cumulative error regularization term during training. Although these approaches enhance robustness, they necessitate model retraining. In contrast, [Li et al.) (2024a) observe that the pixel variance of noisy samples is not always aligned with the schedule during inference, prompting them to adjust the timestep accordingly. Additionally, Ning et al.] (2024) iden- tify noise prediction error as a key source of exposure bias and propose scaling noise estimation to address magnitude discrepancies, although direction errors remain unaddressed.
BACKGROUND
Diffusion Models In the forward diffusion process, Gaussian noise is added to samples from the data distribution ((x)), over series of time steps t {0, I...,, T), resulting in increasingly noisy versions of the data:
g( 11 = Na v 3,x, 3.1):
(1)
where Bg is variance schedule that controls the amount of noise added at each step. The overall forward process is described as:
(x12 II g(
In the reverse process, diffusion models denoise Er back to the original data distribution:
po( (10 1), E),
where 14g is the mean predicted by a neural network with parameters 8, and Σ is the predicted variance. Ho et al. (2020) found that in practice, using fixed variance schedule Σ, leads to simpler and more stable training. Diffusion models can be trained by minimizing the Kullback-Leibler divergence:
T
C = == Ey((O):))
KL 1l 20),
(4)
3

---

Published as a conference paper at ICLR 2025.
An alternative, often simpler, loss function involves predicting the added noise directly:
ssmple = Ber [|ec
co (xx;) (xx))
(5)
where 6o denotes the noise estimation network in diffusion model.
After training, diffusion models can generate data by sampling from the standard normal prior p(XT), iteratively refining these samples through the reverse process defined in Equation 3 When employing the DDIM sampler (Song et al. 2021), the model proceeds follows at each timestep t.
First, the model predicts the denoising observation using noise estimation:
Oit 60 (x:))
(1)
(0)
v 0r
where ai represents the estimated clean image at timestep t, æt is the noisy image at timestep t, and 6 19 (mx) is the model's prediction of noise.
Next, the latent variable for timestep t 1 is predicted:
1: Var-ia (1)
0 of (xx)
(x,) + ord. ~
where 0g 1 Br. ag II ag, and o; controls the stochasticity of the generation. The term ef is sampled from a standard normal distribution, introducing stochasticity into the generation process.
Prediction Error in Accelerated Sampling Given diffusion model trained with forward pro- cess Tf steps. during generation, instead of sampling all latent variables {Dr, Ir-1..... 21 itera- tively, Song al. (2021) propose to sample along trajectory of length TV: {I T In } where defined as an increasing subsequence of (0, :.....T).................. Since T' T, the proposed method significantly accelerates the sampling.
However, the prediction error tends to increase as the sampling interval grows. Kim et al. (2024) derive that the error introduced to the latent variable æ, during the transition from time to A/:
At e, O(A)),
where At denotes the timestep interval. Equation g shows an inherent trade-off between accuracy and efficiency in the sampling process in diffusson models.
Exposure Bias in Diffusion Models During training, diffusion models are trained noisy data sampled from the real data distribution æo:
q( (.)) NZx Vāyao, (1-aa)) a))
On the contrary, models are only exposed their previously prediction & during inference. The input discrepancy, 2n - a,, disrupts the model's prediction subsequent timesteps.
The exposure bias is exacerbated in accelerated sampling, where errors at each timestep are signif- icantly amplified, as shown in Equation g Therefore, we identify the combined effect of exposure bias and enlarged prediction error as the main challenge in accelerated diffusion model sampling.
4 METHOD
Inspired by previous work (Song Ermon 2019 Chung et al., 2022, Chen et al. 2024 He et al. 2024), we interested in understanding the exposure bias problem under the geometric view of diffusion models. In this section, we first provide the geometric view the exposure bias problem. Then, based on the geometrical analysis, we introduce Manifold Constraint Denoising Obser- vation (MCDO), a training-fre free and almost hyperparameter tuning- free method that mitigates the exposure bias in the accelerated sampling of diffusion models.
4

---

Published as a conference paper at ICLR 2025.
4.1 GEOMETRIC VIEW OF EXPOSURE BIAS
To better understand the training-inference discrepancy in diffusion models, we borrow a geometric view of the data manifold.
Assumption CChung al. 2022) Let Mo C. I" be the clean data manifold. The distribution of ( -im-emmnsooo manifold Mr.
the ground truth noisy data pl x+)) is concentrated on an (n
Under Assumption I for a diffusion model trained with sampling steps, ground truth noisy samples obtained with Equation form series of manifolds MM.Ma MM..... MT. The model is trained to smoothly map the data from M, to M,-1 (Chung et al., 2022). During the inference, pure noises from My are gradually mapped towards the target manifold Mu- The perfect generation process can be formulated as follows:
Mr Az,
As,
M2
A2 Mı
A1, Mo.
(10)
where A,, E (1,2,..., T3, denotes series smooth maps.
However, the perfect mapping is only applicable to data concentrated on the manifold.
Assumption 2 (Extended from He et al. (2024) Define the distance from a data point to given manifold as: d(x: v, M)) = in I V for O. For given timestep i, diffusion models only make accurate noise predictions for data concentrated on the manifold: Xx == {xx E IR" d(x,, M,) rt), where r, > the radius.
While prediction errors are inevitable, it possible that, for given timestep 1 1, data from MM4 is mapped to points away from Mr such that d (x, 1, M,) > With Assumption [ models' noise estimation for data क; X₁ will degrade. Based on Equation [8, we argue that the large deviation from the manifold is the main reason for exposure bias in the accelerated sampling of diffusion models. We illustrate this problem in Figure 2a
4.2 RELATION BETWEEN DISTANCE TO MANIFOLD AND THE PIXEL VARIANCE OF DATA
Under Assumptions and we relate the pixel variance of sample ž,, to its L distance to Mr.
For a prediction T; let ot argmin g. E. ||; - zzll. The distance between é and MM satisfies:
d(.,, 1, M.) ||a x; 2 |||=; ||
(11)
Given that æт No ((0,1) I). all pixels in æт follows normal distribution, such that E
According Equation 4 we have E [ 0. when the diffusion models are well-trained. As a consequence, the expectation of pixel mean of a single noisy sample approaches O well: E y m 0, according to Equations and 7 (more details Appendix A.1). Then the r
norm of 2t and its pixel variance satisfy:
Var(#,) = -ー (B., - - ~ ymhh 33,
--
-1
(12)
where is the data dimension. We show the evolution of L norm of sample as well as its pixel variance in Figure 4 It is shown that the curve of the L norm and pixel variance are highly overlapped for most of the time. Therefore, use Equation 12 approximate the L norm of
noisy data with its pixel variance.
Plugging Equations and 12 into Equation
have:
d(2, 1, M2)
Waaza + vI a,ell - V Var(a.
(13)
According to Wegner 22024), for data NN(0, I), TR". one has:
vnl
(14)
Vn
Given that the dimension n of the latent space is usually large for diffusion models (c.g., n = 12288 for LDM-4 trained on CelebA-HQ), with high probability, ||e vn.
5

---

Published as a conference paper at ICLR 2025.
In addition, since is random noise independent of zo, when is large, e, and æy almost orthogonal with high probability (Wegner 2024). Therefore, we have:
[I |]] -
o 1
(15)
Plugging Equations and 15 into Equation 13 the following inequality holds with high prob- ability:
d(0r, 1,
|vall x0 x0 n (+))
-
VnVar(())
(16)
where equality holds if and only if át, at and the origin are collinear.
Equation 16 indicates that the devia- tion in Var( could potentially make 1( bias problem under Assumption This is consistent with Figures and
Lg norm
Tt. resulting in exposure
110
-0.975
0,550
van vanincc
norm
From this perspective, the timestep-shifting strategy proposed by [i et al. (2024a), which determines timestep t, within range wt
0.925
106
0.900
argming (VI- áv /Vrr (æx) t' e {t-
104
w/2.... ......... ++ can interpreted as locat- ing the manifold M,, E {t w12,,,,....
00850
w/2), where the lower bound of samples-to- manifold distance is potentially reduced ac-
901
691 501
Denoising timestep
cording to Equation 16
Figure 4: Evolution of L norm and pixel variance of the noisy data. We plot the average statistics of 6,400 samples 100-steps generation on CelebA- HQ.
As this strategy provides valuable insights, the inaccessibility of the L norm of the clean data 2o in Equation [6 presents an opportunity for further refinement of the approach.
Inspired by prior studies (Bao et al., 2022 Chen et al., 2023 2024), we delve into the model's
predictions for noisy data and the corresponding denoising observations. Specifically, when the model given the ground truth input at + the analytical form of a and o, can be expressed as (see Appendix A.2):
1
vo d.e// 2. x. n.c (t)
(17)
Here, Zp denotes the clean data, the ground truth noisy data, eg the noise estimation er- ror. The scaling factors de and n, vary with f. We find that while n, remains within small range near O, de is hundreds or even thousands of times larger than n, when is large (see Fig- ure 5)
10:
Ratio
This substantial magnitude of d, suggests that (t) is predominantly influenced by the scaled 2
noise prediction eтror term d,eg timesteps. As consequence, we propose to (t) rectify the noise estimation error e by incor- porating information from the denoising obser- vation //
most
739
339
Denoising Timesteps
Figure 5: Comparison between error scale on de- noising observation (d,) and on noisy data (n,).

---

Published as a conference paper at ICLR 2025.
4.3 MANIFOLD
CONSTRAINT ON DENOISING OBSERVATION
Chung et al. 2022) view the diffusion process as an interpolation between Ma and Mr. Inspired by this, we substitute E/ (x) in Equation 7 with Equation 6
u - oivaL
20 VI- ---1 of v. C2 +1 (0, (18)
VI-ai at
Equation 18 suggests that To can be interpreted as interpolating between two noisy data. Based on this, we assume that the denoising observation a also concentrates on series of manifolds.
Assumption (Extended from Chung al. (022 Denote % as the set of denoising observations
of Lt. Xs. There exists series of manifolds: M .M (T-1)
.....Mo) .M(' where x!
(1))
for E (T, ,).... concentrates on: d(ab) 1, 1,M)") <ro Va E % Where ro is the radius.
(1) (t)
(t)
Under Assumption 3 when a noisy data é; is distant from Mr, its denoising observation, calculated
using Equation 6 could potentially be mapped away from M For clarity, denote denoising
observations close to mn! as (t)) and those distant from Ma as o
For a denoising observation #0" (e) & x(1) Let a
((()) argmin 1 eM
(00 ||
=
As to (+) linear combination of æ, and e according Equation the expectation of pixel
mean for both 26 and zo ((t) is O. We substitute &, and o in Equation 11 with s)" (())) and x x
d(a (() 1, 2 |0 | ||I vn Var(#
Var(#
(19)
Equation 19 indicates that the deviated a (t) can be constrained to approach M when its pixel variance is scaled to Var(#)())
Given that #n (+)) is inaccessible in generation process, we introduce the reference pixel variance:
04 = E Varl lar.g) where Y denotes the set of conditions. We approximate:
Var(# vt Vag +()) E MM! In practice, the pixel variance used for estimating is collected
during generation with large sampling steps (e.g., T' = 1000), such that the denoising observations are close to their corresponding manifolds. The pre-computation process detailed in Algorithm |
During accelerated sampling, we first align the pixel variance of 20 (()
to approach uy:
C Var($))))
(c)
(()
Mean
(0) Mean (2)
(20)
Ce
Then, we correct the noise estimation, by inverting Equation 6
(t)
(t) (x:) 23
(21)
By substituting Equations 20 and 21 into Equation 7 we obtain refined prediction for .;-1;
. va + / āi ofEl
(1) (xx) +
(22)
The correction is applied for i E (T....,44re))) where hhre is timestep threshold. The proposed method is illustrated in Figure[bbjand detailed in Algorithm 2
(thre))
We refer to the proposed method as Manifold Constraint on Denoising Observation (MCDO), as its
key idea is to constrain the denoising observation to approach its corresponding manifold.

---

Published as a conference paper at ICLR 2025.
A distinction between reducing the lower by scaling Var(ă;)
Algorithm 1 Pre-Computation for e2
bound of 1, 1, (t) 1,, by correcting and that of d(
1: Initialize n, list()
Var(z;))) (() is that the former modifies Ht. while the latter refines the noise estimation when æt.
2:
for = 1, ..2.....N .N do
3:
Fr (), ~ W
4:
for f. t= -T....,.. 2,
do 2/ 2.- 1.-vi-aae (e..v)
is preserved (see Appendix A.3)
5:
Estimating Us requires few samples (e.g., N 20 for SDXL on MS-COCO). Once computed,
vā. n4 naapeed (Var ($))
6:
Vt can be directly used to correct Var(#) (() during accelerated sampling without additional computational overhead. Since ii depends
क. = v1 04-1 o2 (zt.y)
(1)
7:
8:
c, ~ ~~,00,11
9:
Fh-1 V
(1)
+ xz + agd
only on the pre-trained model, when accurately estimated, the only tunable hyperparameter is ttrre which often works well tehre 0. Thus, DDIM-MCDO requires minimal hyper- parameter tuning.
10: end for
11: end for
12: for t = T...., 2, do
13: V/- = Mean(n,)
14: end for
EXPERIMENTS
5.1 EXPERIMENTS SETUP
We conduct experiments on high-resolution datasets across various tasks. Specifically. we evaluate MS-COCO (Lin et al. 2014) with
Algorithm 2 MCDO Sampling
1: ΦT ~ NN 1 Y
al., 2024), and CelebA- HQ (Karras et al. [018)) ImageNet-256 (Rus- sakovsky et al.] 2015). and L.SUN-Bedroon (256 x 256) with LDM-4 (Rombach et al. 2022].
SDXL (Podell
2: for T", T I.... 1 do
$0 (T)) R., -v-v &.e
3:
Vān,
4: if T, 2 tthre then
(TT))
5:
NT, = Mean
For text-to-image (T2I) generation, we gener- ate 30k samples using MS-COCO prompts to compute FID (Heusel et al. 2017) and CLIP scores (Radford et al., 2021). For other tasks, 50k images are generated to evaluate FID, with sFID and IS reported for ImageNet. We com- pare our method with state-of-the-art training- free approaches (Ning et al. 2024; Li et al. 2024a), following Li et al. (2024a) to imple- ment TS-DDIM and conducting hyperparam- eter search based on their range and qualitative results. For Ning et al. (2024), a coarse-to-fine search is used for the uniform schedule X. We visualize exposure bias reduction using Ning
(Tz))
6:
C, Var a
30 (7.) Uer
(Tt)
7:
1
Mr:)) + pr.
/a, =b (T))
2r.
8:
V-a,
9:
else
(7)) (T))
(T.)
(7d)
Co
E
(Z 1), 2
3/
11:
end if
x7
/1 :..-1 -022(11)
<~NN(0,1
14:
L. = Vān (0(11 + + + O,E`
end for
ctt al. (2024)'s metric (see Appendix A.4). Results with DPM-Solver++ (Lu ct al. 2022b) and Stable Diffusion 3 (Esser et al. 2024) are in Appendices A.12 and A.13
5.2 MAIN RESULTS ON TEXT-TO-IMACE GENERATION
Results of T2I generation (Table I Figure 6) show that with few sampling steps, image details and object shapes degrade significantly, suggesting deviation from the data manifold. The proposed manifold constraint improves image quality, indicating more accurate inference.
To implement MCDO, we use N = 20, T = 1000 for pre-computation. Samples are conditioned
on different prompts from MS-COCO. While larger N could improve e estimation, fixed for simplicity. For TS-DDIM (Li et al.) 2024a) we use t, 300, w = 4 for cutoff timestep and window size. A uniform epsilon scale: 1.008 is applied for DDIM-ES (Ning al. 2024).
keep it
8

---

Published as a conference paper at ICLR 2025,
Table Results Text-to-Image generation (1024
1024) on MS-COCO with SDXL.
Model
Method
Steps
FIDJ
CLIP Score
tehre
DDIM
18.17
31.58
DDIM- ES Ning et al. 2024
10
18.91
31.59
SDXL
IS-DDDM LI et al (024a)
10
23.82
31.31
DDIM-MCDO
10
15.60
31.75
"Three people on horse back at a rural wod intersection."
(b) "A buby next to stuffed bear of some sort."
Figure 6: Images generated using 10 steps SDXL with DDIM sampler. From left right: SDXL, TS-DDIM (Li et al. 2024a), LDM-ES (Ning et al. 2024), and DDIM-MCDO (ours).
5.3 RESULTS ON UNCONDITIONAL GENERATION
Considering that both LDM-ES (Ning al. 2024) and DDIM-MCDO aim to reduce noise estimation error, we investigate their orthog- onality. We conduct experiments combin- ing DDIM-MCDO with LDM-ES using the same hyperparameters, denoted as DDIM- MCDO+
Table 2: Results on CelebA-HQ 256 x 256 with DDIM Sampler. = 0.0. Bold indicates the best results, and underlined indicates the second-best.
Method
T
FID
À
DDIM
20
10.59
TS-DDIM (Li et al.
2024a
20
8.29
LDM-ES Ning ct al.
2024
20
18.39
1.001
We compare DDIM-MCDO and DDIM- MCDOtwith state-of-the-art methods (Ning al. 2024 Li et al., 2024a) the CelebA-
LDM-ES (Nig et al.
2024
20
11.73
0,999
DDIM-MCDO
20
9.49
DDIM-MCDO
20
10.59
1,001
HQ dataset Karras et al.) 2018) using LDM-
DDIM-MCDO:
20
10.53
0.999
4. Results under the recommended setting
DDIM
10
21.08
TS-DDIM Li al.
2024a
10
15.71
(17 = 0.0) (Rombach et al., 2022) are shown in Table 2 with À representing the epsilon scaling factor from Ning et al. (2024). For 100,
LDM-ES Ning
2024
10
18.56
1.002
DDIM-MCDO
10
15.63
DDIM-MCDO:
10
13.36
1.002
TS-DDIM (Li al. 2024a),
DDIM
54.99 5
60 for 20 and 10 steps sampling, and 30 for 5 steps. Qualitative results are in Appendix A.6
TS-DDIM al
2024a
5
50.69
to 200,
LDM-ES Ning et al. 2024
49.40
1.005
DDIM-MCDO
30.76
ADIM-MCDOS
5
27.39
1.005
DDIM-MCIO significantly enhances perfor- mance across all three accelerated generation settings, attributed to its correction of both magnitude
DDDIMMMCDO combined with I.MM-ES using the same hyperparameters.

---

Published as a conference paper at ICLR 2025.
and direction errors in noise estimation. It is worth noting that as T" decreases, the DDIM-MCDO yields more improvement, compared to other methods. Notably, DDDM-MCDOOooutperforo component method alone, both exceeding the baseline. Additionally, at T 20, LDM-ES (Ning 2024) shows sensitivity to hyperparameters (e.g., a large FID difference when À varies from 0.999 to 1.001), but performance stabilizes when combined with DDIM-MCIOO.
either
et al.
To maintain consistency with (Ning et al. 2024), we report results under the DDPM setting (1) 1.0) in Table (see Appendix A.8). Additionally, we provide quantitative re-
Table 3: Results LSUN-Bedroom 256 x 256, with LDM-4 and DDIM sampler, 7 0.0, Lhre 0.
sults for LSUN-Bedroom 256 256 (Yu al., 2015) in
Table 3 with corresponding qualitative results available in
Method
7
FIDJ
Appendix A.9
DDIM
4.26
DDIM-MCDO
20
4.20
For implementing DDIM-MCDO, variance statistics from 64 samples are collected during 500 steps of DDIM sam- pling- In all experiments on CelebA-HQ and LSUN- Bedroom, we set the threshold timestep lehve to O. Given that DDIM-MCDO consistently improves performance across all experiments, we consider it a hyperparameter tuning-free.
DDIM
10
9.33
DDIM-MCDO
10
6.25
DDIM
25,94
DDIM-MCDO
12.34
DDIM
44.91
DDIM-MCDO
5
19.31
DDIM
78.70
ADIM-MCDO
33.79
5.4 RESULTS ON CLASS- CONSDIOONTIL GENERATION
We conduct class-conditional image generation on ImageNet-256 (Russakovsky et al. 2015).
Table 4: Results on ImageNet-256, with LDM-
4 and DDIM sampler. 1) 0.0, guidance scale
To show the hyperparameter suning-free feature of DDIM-MCDO, we employ the same hyper- parameters in Section 5.3 N 64, hhhre 0. The pixel variance of 64 samples from different classes sample per class) is collected during 500-step DDIM sampling. Quantitative and qualitative results are shown Table and Ap- pendix A.I0 respectively.
s = 3.0, tahre O.
Method
T
FID4
sFID
ISt
DDIM
20
11.01
7.69
69.45
DDIM-MCDD
20
11.29
6.03
69.47
DDIM
10
9.78
12.35
66.30
DDIM-MCDO
10
10.22
7.13
67.95
DDIM
5
16.81
36.07
49.92
DDIM-MCDO
5
8.30
10.89
62.15
5.5 ABLATION STUDY
In this section, investigate how the number of constraints affects image quality by varying the hyperparameter letrre from 1000 0. We
Table 5: Result on ImageNet-256 with LDM-4.
T 5, 1) 0.0,
3.0
evaluate the FID, sFID, and IS scores of 50,000 images generated with each tehree on ImageNet- 256 (Russakovsky et al.] 2013) using LDM-4
Method
FID.
sFID.
ISt
lihre
16.81
36.07
49.92
1000
13.45
26.72
54.64
800
and 5-step DDIM samplers (Song et al. 2021).
DDIM-MCDO
10.67
19.14
58.29
600
9.38
15.32
60.29
400
Results in Table 5 show consistent improve- ments as tive decreases. While increasing dis- crepancy in (*) at lower t may introduce er- rors in pixel variance correction, the benefits of DDIM-MCDO in later stages outweigh this issue, leading to better results than its early-stopping version (e.g., 4thre = 200).
8.48
11.76
61.82
200
8.30
10.89
62.15
6 CONCLUSION
In this paper, we address exposure bias accelerated sampling of diffusion models. By analyzing this bias through the manifold hypothesis, we propose manifold constraint that aligns deviated de- noising observations with the manifold, enhancing both magnitude and direction accuracy in noise estimation. Our experiments several large-resolution datasets demonstrate significant effective- ness. Furthermore, the method training- free and requires minimal hyperparameter tuning, making it simple plug-and-play module for other sampling methods.
10

---

Published as a conference paper at ICLR 2025.
ACKNOWLEDGMENTS
This work is funded by the National Natural Science Foundation of China (62403429), Zhejiang Provincial Natural Science Foundation of China under Grant (LQN25F030018, LQN25F030008).
REFERENCES
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the opti- mal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022.
Defang Chen, Zhenyu Zhou, Jian-Ping Mei, Chunhua Shen, Chun Chen, and Can Wang. A geomet- ric perspective on diffusion models, 2023.
Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of
diffusion sampling. In Forty-first International Conference on Machine Learning, 2024. |RLtttps: /ppptt/ee//eeeeeeeve..e. wwweee/ooooidedeeeeeeeeeeee
ottppp/awww......ooooooooaaoaaaa wttp:/ormmm...---0.2....... 6W2
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net do-.umnddd-JJJ
attps://epenzeet
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. CFG++: classifier free guidance for diffusion models. In The Thirteenth Interna fional Conference on Learning Representations, 2025. URL httpss///peereeveew .nete
forum? d=E7 7ubbOTtp
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 07740 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, and Robin Rombach. Scaling rectified flow transformers for high-resolution image syn- thesis. In Forty-first International Conference on Machine Learming, 2024. URL https: //openreview..o..mmm http::/ee/ffffoooeooeoo wow.mmoddd--nuuuaaaooo
Martin Nicolas Everaert, Athanasios Fitsios, Marco Bocchio. Sami Arpa, Sabine Süsstrunk, and Radhakrishna Achanta. Exploiting the signal-leak bias in diffusion models. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision pp.
2024.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Yutong He, Naoki Murata, Chich-Hae Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji. J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference Learning Representa- fions, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL httph:t//tx/xii
org/ abs/ / 2207. 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, pp. 8633- 8646, 2022.
11

---

Published as a conference paper at ICLR 2025.
Ahmed Imtiaz Humayun, Ibtihel Amara, Candice Schumann, Golnoosh Farnadi, Negar Ros- tamzadeh, and Mohammad Havaci. Understanding the local geometry of generative model man- ifolds, 2024. URL https///awxw...o////////
08307
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
net/ forum? d=Hk 992CeAb
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning prob- ability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https: //arxiv.orq/aa 1111. 6114
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021. URL httLhttss//axxxy>.. http//awwww....//////////.. /tt./////.//////////////////////
Mingxiao Li, Tingyu Qu, Ruicong Yao. Wei Sun, and Marie-Francine Moens. Alleviating expo- sure bias in diffusion models through sampling with shifted time steps. The Twelfth Interna- tional Conference Learning Representations, 2024a. URL Forum?id 2SD3MKKKe6
net/
Yangming Li and Mihaela van der Schaar. On error propagation of diffusiin models. In The Twelfth International Conference on Learning Representations, 2024. URL URLhttts:://o/eeeiiiie
net hett/Errum2ideRtAcct..1..
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755, 2014.
Yaron Lipman, Ricky T. Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. /tRL/tttt///:peeeeeeet http://wwweeeeeeeeeeeeeraaaaaaaa Pa-
net/ Eorum?id
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representattons, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint TrXiv:2006.00927, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver+4: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 1r.10:2211.01095. 2022b.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input per- turbation reduces exposure bias in diffusion models. arXiv preprint arXiv:2301.11706 2023.
12

---

Published as a conference paper at ICLR 2025.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias diffusion models. In The Tweifth International Conference on Learning Repre- sentations, 2024.
xx
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp- PMLR, 2021.
Zhiyao Ren, Yibing Zhan, Liang Ding, Gaoang Wang, Chaoyue Wang, Zhongyi Fan, and Dacheng Tao. Multi-step denoising scheduled sampling: Towards alleviating exposure bias for diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4667- 4675, 2024.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)211-2522 2015.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Alexandra Birch, Andrew Finch, Hiroaki Hayashi, loannis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Sudoh (eds.). Proceedings of the 3rd Workshop on Neural Generation and Transla- tion, pp. 157-167, Hong Kong, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19/5/1/
org/Dl9 DD9 5616
Jascha Sohk-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Inter- national Conference Learning Representations, 2020. URL ob.p.orgg/eee//jorraaa.
corr/abs 200000002200222222
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11977, 2019.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference on Machine learning, pp. 3221 1-32252, 2023.
Xingzhe Su, Daixi Jia, Fengge Wu, Junsuo Zhao, Changwen Zheng, and Wenwen Qiang. Unbiased image synthesis via manifold guidance in diffusion models, 2024. URL htttt::tp//aiiii.
brg/abs/ 2t.bb//2237.. 08199
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667-19679, 2020.
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. arXiv preprint arXiv:2005.03364 2020.
Sven-Ake Wegner. Lecture notes on high- high-dimensional data, 2024.
orr/
abs/ / 2101. 05841
13

---

Published as a conference paper at ICLR 2025.
Yuzhe Yao, Feng Tian, Jun Chen, Haonan Lin, Guang Dai, Yong Liu, and Jingdong Wang. Timestep- aware correction for quantized diffusion models. arXiv preprint arXiv:2407.3391,, 2024.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 03365, 2015.
preprint
Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin David Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. In Thirty-sevnnth Confer- ence on Neural Information Processing Systems, 2023. URL htttps://ppenteview...er Forum?id=THE 8hdVxH.
Qinsheng Zhang and Yongxin Chen. Fast sampling diffusion models with exponential integrator. In International Conference on Learning Representations, 2023.
Yizhe Zhang, Jiatao Gu. Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly. Plan- ner: generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around steps. arXiv preprint arXiv:2112.0009, 2023.
14

---

Published as a conference paper at ICLR 2025.
A APPENDIX
A.1 ZERO MEAN ASSUMPTION AND ERROR ANALYSIS
The noise added clean data during the training phase follows normal distribution: E, (0, I),
which implies E(e,] O. According to Equation 5 assuming E[cğ] 0 is reasonable for well-
trained diffusion models. According to Equation 7
the predicted noisy data at timestep 1 can
be expressed as 8 = vā: R 1-a,s (m.)
VI 01-11-1- - of ( (a:) + ~N
Vai
Then as long as [
DD E 12 [. = O, one has R
- - 0, [1. 7)
We illustrate the evolution of the average absolute pixel mean (EE (EI |)) for the noisy data prediction, denoising observation, and noise estimation in Figure 9 The statistics were collected from 6,400 samples generated using a 100-step LDM on CelebA-HQQ
can be observed that the
(t) average absolute pixel mean values for å,, 2 and Co all remain close to zero throughout the generation process. This suggests that the pixel mean for all variables exhibits only minor deviations from 0 across timesteps.
In Equations and 19 the square root of the pixel variance for i; and #0 is used to approximate
| We present the evolution of the I norm alongside the pixel variance for the denoising observation in Figure 7 The relative error of this approximation is presented in Figure [8] The results demonstrate that throughout the entire
their 1,2 norm: V Var(
| V ra var(aa))))
generation process, the approximation error remains very small (less than 0.045). This indicates
that using the pixel variance of i or (t()) as an approximation for their L norm introduces only
minimal error, which acceptable for practical applications.
100
Lg norm
Pixel variance
90
0.7
0.66
Pixel variance
L norm
70
0.4
60
-0.3
50
-02
991
891
791
691
591
491
391
291
191
91
Denoising timestep
Figure 7: Evolution of the 2 norm and pixel variance for the denoising observation, based on average statistics from 6,400 samples generated using 100-step LDM on CelebA-HQ.
15

---

Published as a conference paper at ICLR 2025,
%
0.035
Relative error
0.025
0.015
0.005
891 791
591
491
391
291
191
Denoising timestep
Figure 8: Relative error of using square root of pixel variance to approximate the L norm of é and (())
2 Statistics are collected from 6,400 samples generated using 100-step LDM on CelebA-HQ.
0.10
Absulute pixel mean
0.08
0.06
0.04
0.02
2
991
891
791
691
591
491
391
291
191
91
Denoising timestep
Figure 9: Evolution of the average absolute pixel mean for the noisy data prediction, denoising observation, and noise estimation. Statistics are collected from 6,400 samples generated using 100- step LDM on CelebA-HQ.
16

---

Published as a conference paper at ICLR 2025.
A.2 DERIVATION OF NOISY RATIO SINGLE STEP FOR DDIM
In the forward process of DDIM, we have:
VāgTo + VI- dgC 3t, E (O,
(23)
In the reverse process, when given the ground truth input æ++1 the model's prediction of a is: V 04 c
(++1)
(++1) (m++1) + 01416 N (0,
(24)
When o; = 0 for all t, the sampling process become deterministic, and Equation 24 is simplified as:
a vār v1 ā (x.+1).
(t+1)
(25)
For timestep f, by denoting the prediction error on the noise Cg (t)
we have:
[ c + eg
(26)
We now derive the on denoising observation with respect the error on the noise estimation.
By plugging Eq 26 into Equation 6 we have:
(++1)
3++1 VI
a
(1+1)
(x1+1)
Var+i
B+41
ac+1E2+1
VT- a++1
(1+1)
e0
Vā++1
Vāx+1
V ac+1
(++1)
x0
Eg
(27)
Var+1
is worth noting that in the early stage of generation, e.g., when T, a, approaches 0. Therefore, according to Equation 27 the error coefficient on the denoising observation: d, /1-āc+1 is non- negligible.
va.
As a notably influenced by the amplified noise prediction error term d at most timesteps, it is possible to correct the noise estimation eπror by correcting the denoising observation.
With the same manner, we derive the error on the noisy data prediction with respect to the noise estimation error. By plugging X Equations 26 and 27 into Equation 25 we have:
④. =or
To
VI act1
(t+1) 6
+ +v1 - - ((+1) (xL-1)
Vatt1
= =Vāzz
V
V1 V1
aitl
eg
(1+1)
+ vI - 0 + vI - v (t+1)
Vār+1
VI - as+1
(Varto + VI - arE++1) +
V1- Vas
(1+1)
Eg
va++1
=x, (VI- vāl v -
((+1)
(28)
Vār+1
VI-au11
We denote the scaling factor for x; n4, R4
-
Vor
The value of both d, and
Vār+i
n, during denoising process are shown in Figures 10 and down for Ct, the same error is scaled up for (0)
It illustrated that while elf) scaled
Correcting the pixel variance of d; using a scale factor (directly correcting the pixel variance of noisy data prediction) will eventually scale the entire term æ; + = which leads to distortion of the H, component in कt. When correcting the pixel variance a using the proposed method, one corrects 2o de While the z (t) term is also influenced, it is important to note that by recal-
culating the noise estimation with the corrected denoising observation in Equation 6 the prediction
emor reduced while the original is preserved.
17

---

Published as a conference paper at ICLR 2025,
14
d:
12
10
Ratio
999
899
799
699
599
499
399
299
199
Denoising Timesteps
Figure 10: Error scale for denoising observation
-0.002
nt
0.004
Ratio
-0.008
-0.010
799
Denoising Timesteps
Figure 11: Error scale for noisy data
-500
Ratio
-1000
-1500
2000
-2500
999
899
799
699
599
499
399
299
199
99
Denoising Timesteps
Figure 12: Ratio of error scale
18

---

Published as a conference paper at ICLR 2025.
A.3 COMPARISON BETWEEN NOISY DATA AND DENOISING OBSERVATION CORRECTION
adjusting the pixel variance of noisy data & during the accelerated sampling process variance collected from sampling process with large enough T,, could potentially reduce the exposure bias and improve the generation results. For clarity, we refer to this method as
According to Equation. I6
Noisy Data Variance Adjustment (NDVA). We provide the results of NDVA in Tables O and
In 6 it can be observed that NDVA outperforms the baseline method, and exhibits performance comparable TS-DDIM when T 20. However, it notably worse than MCDO as the number of timesteps further decreases.
Table 6: Results on CelebA-Ho HQ dataset.
Method
Steps
FIDJ
DDIM
20
10.59
TS-DDIM
20
8.29
DDIM-NVDA
20
9.21
DDIM-MCDO
20
9.49
DDIM
10
21.08
TS-DDIM
10
15.71
DDIM-NVDA
16.45
DDIM-MCDO
10
15.63
DDIM
54,99
TS-DDIM
50.69
DDIM-NYDA
47.52
DDIM-MCDO
30.76
Table 7: Results on MS-COCO.
Method
Steps
FID.
DPM- (2M-Solver++((MM
15.10
(PM-Solver++((M) NVDA
15.76
DPM DPM-Solver+++(MM) + MCDO
14.88
DPM-Solver++(2M)
15.57
DPM- NVDA
15.76
DPM-Solver++((MD) + MCDO
15.32
DPM SoM-Sovee+++(2M
17.05
DPM-Solver++(2M) NVDA
6
17.13
DPM-Solver++ MCDO
16.39
Solverrrr++((MM
19.16
(PM-Solver+++(M) + NVDA
19.02
DPM- MCDO
18.51
A.4 EXPOSURE BIAS MEASUREMENT
In this section, employ the exposure bias metric proposed by Ning al. (2024) evaluate the single step variance discrepancy reduction when using our method. The statistics are collected during the accelerated diffusion sampling (5, 10, 20 steps) on Stable Diffusion XL. The variance deviation with respect to the fixed schedule 1 - ã₁, as well as the standard deviation are illustrated in Figure 13, Although our approach based on the manifold hypothesis and the concept of pixel variance, the results indicate that the proposed manifold constraint still reduces sample variance emror.
19

---

Published as a conference paper at ICLR 2025,
$20
DOIM
DOIM
devietton enter
Varance
1.24
2:00
2055
Dencising timestep
Deneising Eimesta
(a) steps variance error
(b) 5 steps error
1.2
DCIN-NCDO
DOIN
K12
DDIM
CITER
0.44
3.2
Dencising timestep
Denoising timestop
(c) 10 steps variance етror
(d) 10 steps std error
DDIM-ACDO DDIM
DDIM-MCD
DDIM
0.99
erTor
4224
$22-
Denolsing ümmssop
Denoising timestop
(c) 20 steps variance
(f) 20 steps std
4.30
DOIM MCDO
0.24
DDIMMCCO
DDM
DDIM
2322
6.22
1300
1.0
Denclsing Emeettp
Denaising timestpp
(g) 50 steps variance
(h) 50 steps std
Figure 13: Exposure bias measurement. We plot the variance error and standard deviation error in single-step samplings.
20

---

Published as a conference paper at ICLR 2025.
A.5 MORE QUALITATIVE RESULTS ON MS-COCO USING SDXL We provide more qualitative results on text-to-image generation in Figure 14
WAP
We
(a) "A bunch of sheep are standing in snowy field."
(b) "a woman walking along sidewalk while holding suitcase"
girl holding foot long hot dog restaurant"
(d) young man a hat holding a cup of coffee and pastry."
Figure 14: Images generated using 10 steps SDXL with DDIM sampler. From left to right: SDXL, TS-DDIM (Li al. 2024a), LDM-ES (Ning et 2024), DDIM-MCDO (ours).
A.6 QUALITATIVE RESULTS ON CELEBA-HQ
We present qualitative results for unconditional generation on CelebA-HQ using LDM-4 Fig- ures 15 16 and 17 with sampling steps of 5, 10, and 20, respectively. For the sampling settings, we use the recommended value of n = 0.0 when the number of steps is low (Rombach et al.) 2022)
21

---

Published as a conference paper at ICLR 2025.
S
Figure 15: Samples on CelebA-HO 256 x 256 using 5 steps LDM-4. From left to right: DDIM, TS-DDIM et al. 2024a), LDM-ES (Ning al. 2024) with 1.005, DDIM-MCDO, DDIM- MCDOfwith À = 1.005.
22

---

Published as a conference paper at ICLR 2025.
Figure 16: Samples CelebA-HQ 256 x 256 using 10 steps LDM-4. From left to right: DDIM, TS-DDIM (Li et al. 2024a), LDM-ES (Ning et al. (024) with 1.002, DDIM-MCDO, DDIM- MCDOfwith À = 1.022.
23

---

Published as a conference paper at ICLR 2025,
Figure 17: Samples CelebA-HQ 256 256 using 20 steps LDM-4. From left to right: DDIM, TS-
DDIM (Li et al. 2024a), LDM-ES (Ning et al., 2024) with À = 0.999, LDM-ES (Ning et al. 2024
with À 1.001. DCIM-MCDO, DDIM- 0.999, DDIM-MCDO with À L.001.
24

---

Published as a conference paper at ICLR 2025.
A.7 PIXEL-LEVEL CONSTRAINT FOR UNCONDITIONAL GENERATION
Considering that in unconditional generation, the similarity between samples within an inference
batch is higher than that in Text-to-lmage generation, we propose more fine-grained correction ap-
proach: to estimate the pixel-level variance of the denoising observation: = Var( o...k.
(t)
Ve E R.C.,.W (t) E BBB.C.W,WW Where B is the batch size, C' is the channel number, H and W are the latent's height and weight, respectively. We denote this variant as DDIM-MCDO When DDIM-MCDO* is coupled with LDM-ES (Ning et al. 2024)) it is denoted as DDIM-MCDO The batch size used for implementing DDIM-MCDO* 64 on CelebA-HQ.
A.8 QUANTITATIVE RESULTS ON CELEBA-HQ WITH DDPM SAMPLER
We provide results CelebA-HQ using LDM-4 and DDPM sampler, where 77 1.0, in Table We notice that contrast to LDM-ES (Ning et al. 2024), our method yields more performance improve- ment for DDIM than DDPM. Considering the DDIM setting recommended under accelerated generation (Rombach et al. 2022), results in Table g suggests that the proposed methods might be easily combined with other fast sampling method in scenarios where T" extremely low.
Table 8: Results on CelebA-HQ 256 256 with LDM-4 and DDPM sampler, rj 1.0.
Method
T
FID
DDPM
20
29.61
LDM-ES (Ning et al. 2024)
20
15.68
1.01
DDPM-MCDD
20
26.83
DDPM-MCDO'
20
23.84
1DPM-MCDO1
20
16.47
DDPM-MCDO
20
19.21
1.01
DDPM
10
56.22
LDM-ES Ning et al.
2024
10
33.36
1.03
DDPM-MCDO
10
47.29
DDPM-MCDO*
10
29.76
DDPM-MCDOF
10
19.87
1.03
DDPM-MCDO3
16.92 10
1.03
DDPM
5
102.1
LDM-ES (Ning et al.
2024
73.49
1.04
DDPM-MCDU
78.12
DDPM-MCDO*
5
54.56
DDPM-MCDO!
5
56.40
1.04
DCDM-MCDOF
51.76 5
1.04
2&DDIM-MCDO combined with L.M--ES using the hyperparameter.
25

---

Published as a conference paper at ICLR 2025.
A.9 QUALITATIVE RESULTS ON LSUN- LSUN-BEDROND -EEDROM
We present qualitative results for unconditional generation on LSUN-Bedroom using LDM-4 in Figures 18 19 and 20 with sampling steps of 4, and 10, respectively. For the sampling settings, we use the recommended value of r = 0.0 when the number of steps is low (Rombach et al. 2022).
DDIM
(b) DDIM-MCDO
Figure 18: Samples on L.SUN-bedroom 256 x 256 using 4 steps LDM-4.
(a) DDIM
(b) DDIM-MCDO
Figure 19: Samples on LeSUN-bedroom 256 x 256 using 5 steps LDM-4.
A. QUALITATIVE RESULTS ON IMAGENET-256
We present qualitative results of class-conditional generation on ImageNet-256 using LDM-4 in Figures 22, and 23 with sampling steps of 5, 10, and 20, respectively. For the sampling settings, we use the recommended value of = 0.0 when the number of steps is low (Rombach et al. 2022)
26

---

Published as a conference paper at ICLR 2025.
DDIM
(b) DDIM-MCDO
Figure 20: Samples on LSUN--berroo 256 x 256 using 10 steps LDM-4.
(a) DDIM
(b) DDIM-MCDO
Figure 21: Samples on ImageNet 256 256 using 5 steps LDM-4.
27

---

Published as a conference paper at ICLR 2025,
(a) DDIM
(b) DDIM-MCDO
Figure 22: Samples on ImageNet 256 x 256 using 10 steps LDM-4.
(a) DDIM
(b) DDIM-MCDO
Figure 23: Samples on ImageNet 256 x 256 using 20 steps LDM-4.
28

---

Published as a conference paper at ICLR 2025.
A.11 A MORE FINE-GRAINED MANIFOLD CONSTRAINT
In this section, based on the manifold analysis in Section 4.1 we provide a more fine-grained ver- sion of MCOD. Which denoted as DDIM-MCDO+: The statistics collection process of DDIM- MCDO+ and DDIM-MCDO are shared, so DDIM-MCDO+ can be implemented by adding one more line of code.
During inference, we first correct the pixel variance of the denosing observation ab" like in DDIM- MCDO:
V2.FTz = Vcee @0
Mean
(a)")
Va.2rr )] + Mean (ab6 loa::r))
(29)
Then, we calculate the transition coefficient di-x between Ma) and M²
Var(aj x(t 1) loa
d,
(30)
((t) (0 Var(11 (ya,)rr))
With di-k, we can further correct the denoising observation, which will be used in predicting æt-k-
20 (t-k)
12. = - 102.0r
112.0
(31)
The final expression of -
k = vā, u.
ār - -0 (2t)) + OC- G- ~ N (0,1).
(32)
The result on MS-COCO (Lin al., 2014) dataset with Stable Diffusson XL. (Podell et al. 2024) are provided in Figure 24 and Table 9 We find that although DDIM-MCDO+ does not outperform
DDIM-MCDO on metrics like FID and CLIP score, the quantitative results of DDIM-MCDO+ might be better.
Table 9: Results on text-to-Image generation on MS-COCO val2014 with SDXL and DDIM sam- pler. Image Resolution is 1024 x 1024.
Model
Method
Steps
FID
CLIP Score
Lehre
DDIM
10
18.17
31.58
DDIM-MCDO
10
15.60 (-2.75)
31.75 (+0.18)
0
SDXL
DDIM-MCDO+
15.79 (-2.38)
31.72 (+0.14)
DDIM-MCDO+
10
15.98 (-2.19)
31.71 (+0.13)
0
29

---

Published as a conference paper at ICLR 2025,
(a) "A beach area several chairs and umbrellas."
(b) "A dog laying on the beach in the sand."
Figure 24: Comparison between DDIM-MCDO+ (left) DDIM-MCDO (right) using 10 steps SDXL.
30

---

Published as a conference paper at ICLR 2025.
A.12 RESULTS WITH DPM-SOLVERS
To further investigate the compatibility of MCOD with high-order solvers, we evaluate the proposed method with DPM-Solver++ (Lu et al. 2022b). The pixel variance of 60 samples, collected from the 1000 steps SDEE-DPM-SSlver-<(MM sampling process, were used for v estimation. We generated 10k images for each evaluation Table
10
The corresponding qualitative results provided in
Figures 25 26 and 28
Table 10: Results on Text-to-Imen Image generation on MS-COCO val2014 with SDXL and DPM- Solver++. Image Resolution is 1024 x 1024
Method
Steps
FID.
CLIP Scoret
Ahre
(2M
15.10
31.74
DPM-Solver-x2M)-MMCDO
10
14.88
31.85
100
(PM-Solver++<2MM)
8
15.57
31.74
DPM- Solver++
8
15.32
31.79
100
DPM-Sole Solver++(2M)
17.05
31.65
DPM- Solver+<(M)-MMCDD
6
16.39
31.72
100
(PM-S))var+++(MM)
5
19.16
31.52
DPM--ooler++((M)MMMMO
18.51 5
31.62
100
31

---

Published as a conference paper at ICLR 2025,
(a) "A busy city street with many different vehicles."
(b) "Two men with glasses sitting at a table using laptops" "
Figure 25: Comparison between (2M) (Lu et al. 2022b) (left) DPM-Solver++(2M))
MCDO (right) using 10 steps SDXI.
32

---

Published as a conference paper at ICLR 2025.
(a) "A public transportation bus near curb with bicycle rack."
(b) "Two horses plowing the land as a man directs them."
Figure 26: Comparison between D2M-)olver+++(MM) (Lu et al. 2022b) (left) DPM-Solver++(2M))
MCDO (right) using steps SDXL.
33

---

Published as a conference paper at ICLR 2025.
""A person walking across the street with a truffic light above."
(b) A man in the ocean surfing holding to kite."
Figure 27: Comparison between (PM- (Lu et al. 2022b) (left) DPM-Solver+++(M))
MCDO (right) using 6 steps SDXL.
34

---

Published as a conference paper at ICLR 2025.
(a) "A bicyclist is riding past bus that is parked."
(b) "A picture of an alter holding bananas, skeletons, candles and flowers."
Figure 28: Comparison between DPM-Solver+++(MM) [Lu et al. 2022b) (left) DPM-Solver++(2M)
MCDO (right) using steps SDXL.
35

---

Published as a conference paper at ICLR 2025.
A.13 RESULTS ON STABLE DIFFUSION 3
We further evaluate the scalability on rectified flow model: Stable Diffusion 3 Esser et al. 2024 Quantitative results, computed using 10k images, are provided as follows.
Table 11: Results on Text-to-Image generation on MS-COCO val2014 with Stable Diffusion et al. 2024). Image Resolution is 1024 x 1024.
Esser
Model
Method
Steps
FI1
Score?
fehre
DDIM
33.31
30.56
DDIM-MCDO
27.57
31.18
0
Stable Diffusion 3
DDIM
22.71
31.44
DDIM-MCDO
22.26
31.70
0
A.14 RESULTS WITH DIFFERENT CLASSIFIER- GUIDANCE SCALE
Considering that large classifier-free guidance (CFG) scales might cause exposure bias (Chung al. 2025), we evaluate the proposed method, the CFG scale varies. Quantitative results, computed using 10k images, are provided Table
Table 12: Results on Text-to- text-to-Image generation on MS-COCO val2014 with SDXL (Podell et al. 2024 Image Resolution is 1024 1024.
Method
CFG Scale
FID.
CLIP Score
tahree
DDIM
8
18.51
31.81
DDIM-MCDO
17.38
32.03
100
DDIM
10
18.99
31.82
DDIM-MCDO
10
17.49
32.13
100
DDIM
12
20.62
31.70
DDIM-MCDO
12
17.80
32.18
100
The results show that incorporating the proposed method into sampling significantly enhances per- formance across varying CFG scales. Specifically, while higher CFG scales (e.g.. 12) typically exacerbate exposure bias, the DDIM-MCDO method effectively mitigates this issue, reducing the FID score by 2.82.
36

---

Published as a conference paper at ICLR 2025.
A.15 QUANTITATIVE COMPARISON WITH 1,000 STEPS RESULTS
We provide quantitative comparison between images generated using 10 steps DDIM, 10 steps DDIM-MCDO, and 1,000 steps DDIM in Figure 29
Figure 29: Comparison between 10 steps DDIM (left), DDIM-MCDO (middle) and 1000 steps DDIM (right) with SDXL. Prompts used (from top to bottom): "A street light that shows, horse crossing it", "There are people on the snow bank and snow lift chairs are above.", "Four people gather around a table and smile for the picture.", "A small child with a kite walking on a beach."
37


Reviews:
[{"review_id": "lF7taQpOxO", "reviewer": "Reviewer_rs44", "summary": "To narrow the gap between the training of the sampling phase of diffusion models, the authors analyze the diffusion processes from the view of the manifold. They propose to compute the statics of all intermediate diffusion variables and calibrate the sampling process based on the computed statics (variance, mean, \\etc). Experiments show that the proposed method can reduce the sampling steps while maintain the generation quality.", "questions": "See the weakness above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method improves sample quality without adding substantial computational overhead.\n2.Unlike some prior methods, this approach does not require model retraining or intensive hyperparameter tuning.\n3. The manifold constraint method shows improved performance across various high-resolution datasets, achieving better FID scores with fewer sampling steps.\n4. The paper provides both theoretical analysis and empirical evidence to support its method, including experiments on multiple tasks like image and video generation.", "weaknesses": "1. The proposed method needs to be verified on more diffusion schedulers, such as DPMSolver, PNDM.\n\n2. Some ODE-based diffusion models such as rectified flow and consistency models can reduce the sampling steps to two or even one. The proposed method focuses on accelerating the sampling process but is not compared with these fast-sampling diffusion models.  The authors are encouraged to apply their methods to more recent diffusion models (\\ie SD3, FLUX) to show their priority and general ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xv84t7FQkD", "reviewer": "Reviewer_JP98", "summary": "- This paper proposes a method for improving the performance in accelerated diffusion sampling algorithms\n- The paper identifies the exposure bias in accelerated diffusion sampling \n- The method applies manifold constraint for reducing the exposure bias that occurs in accelerated sampling.\n-  The paper presents evaluations of the methods showing improvements over the baseline.\n- A discussion on the geometric view of the exposure bias is presented.", "questions": "- Figure 3: the x-axis title is not clear, what do you mean by denoising steps if the sampling steps are given in the legend? and in general the figure needs to be explained properly\n- In section 4.2, you assume that $E[\\epsilon_\\theta^t]=0$, but this is not always the case.\n- Equation 15 does not make any sense without expectation value.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written\n- The method is evaluated on multiple diffusion models trained on different datasets\n- The method is simple yet effective. It does not require any further training and the additional computations are marginal.\n- The approach shows an improvement over the baselines in most cases", "weaknesses": "- The derivation of section 4.2 is relatively weak, many assumptions and loose steps need to be either refined or omitted.\n  - In section 4.2, you assume that $E[\\epsilon_\\theta^t]=0$, but this is not always the case.\n  - In Equation (12) the authors utilize the fact that $\\frac{1}{n}\\sum_{i=1}^n \\hat{x}_i \\approx 0$, which does not always hold.\n  - Equation (15) does not hold, an expectation value is required for it to be true\n- The evaluations include only comparison to DDIM, even though there are many accelerated samplers that achieve much better results [1,2,3], adding them to the tables is very important for evaluating the method.\n\n[1] GENIE: Higher-Order Denoising Diffusion Solvers\n\n[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps\n\n[3] DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s5XNVOBBKL", "reviewer": "Reviewer_oFBR", "summary": "This paper views the issue of the quality degradation during accelerating diffusion model’s inference on the perspective of the explore bias. The authors point out that explore bias is an important reason that contributes to the loss of image quality when reducing inference steps. The noisy prediction will direct to inaccurate manifolds and thus the errors would be accumulated and amplified. A manifold constraint is proposed to curate this bias and thus lead to better image quality when significantly reducing inference steps.", "questions": "1. In table4, why do fewer steps have lower FID?\n2. Can you add more details of how to get equation 11 and 16 in the appendix?\n3. I didn’t understand why \\epslon_t is equal to \\sqrt{n} when n is large in L261? Can you explain?\n4. Can you elaborate on L271? In my understanding, in fig3, var(x_t) decreases faster as reducing steps, thus potentially making d(.) larger than r_t in later steps. But figure 3 is only on one sample, how to generalize this observation?\n5. The qualitative examples look a bit over-saturated, it would be helpful if you can also show an oracle results (e.g., 1000 steps) on the side for comparison.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well supported by a series of proofs with some assumptions.\n2. The quantitative results look very promising, and largely outperforms the alternative.E.g.,  in Table 3, MCDO with 4 steps has better results than DDIM with 5 steps.\n3. The method is well motivated and the writing logically reasonable and flows well.", "weaknesses": "1. Assumption 3 is a very important part in deriving the manifold constraint. I think it is too strong. I understand it's motivated by Equation (17), however \\hat{x}_0 could have a different distribution of x_0 which is inaccessible during inference. Could you elaborate more?\n2. There are some details not explained well for some key equations/explanations. I added those in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sCZiWZi8V4", "reviewer": "Reviewer_fuNA", "summary": "This paper presents an approach to enhance the efficiency and accuracy of diffusion models by addressing the issue of exposure bias in accelerated sampling. Leveraging a manifold hypothesis, the authors introduce a manifold constraint that reduces error accumulation during sampling without requiring additional training or extensive tuning.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of the manifold constraint is an interesting idea, which addresses exposure bias without adding additional training costs during sampling.\n2. The use of well-chosen visualizations enhances the readability of the method section, conveying key information and clarifying the approach.\n3. The paper is technically clear and well-organized, with the proposed method thoroughly explained.", "weaknesses": "1. The term \"denoising observations\" requires a clear and precise definition, as its current use lacks specificity. A more rigorous description would help readers to understand and improve the technical clarity of the paper.\n2. The pre-computation process still requires a full denoising sequence (e.g., 1000 steps), which incurs substantial computational cost, especially when applying the proposed method to new datasets or domains. It is suggested that potential strategies for reducing this computational cost be discussed or that an analysis of the trade-offs between the number of steps in pre-computation and the method's performance be provided.\n3. The number of samples and their diversity will influence the resulting approach $v_t$. However, the experiments simply set the sample number to 20 without discussing the diversity of prompts or other characteristics of the samples. Including these details would be valuable for readers to have a better understanding, and provide insights for the community. It is recommended to conduct an ablation study on the impact of sample number and diversity on the performance of the proposed method, or to provide more details on how you selected the 20 samples used in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAAUuvdZSn", "reviewer": "Reviewer_Tp1G", "summary": "From the manifold hypothesis, the paper proposes a method to reduce Exposure Bias by adjusting the variance of $x_t$ at each step to match the variance of $q_t$ made by the forward diffusion process. The paper demonstrates that this approach yields FID gains across various model and datasets.", "questions": "**Q1**: Using the statistics of the data directly in generation—could this be seen as an FID \"hack\"? For example, I’m curious how the FID would change if the mean and variance of the generated data w/o MCDO were adjusted to match \\( q_0 \\).\n\n**Q2**: Does this method can improve other ODE solvers like DPM-solver++ [1] or PNDM [2]? I know that DDIM performs poorly when the NFE is below 50. I also want to see the results where NFE is around 50.\n\n[1]: DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models ([Arxiv](https://arxiv.org/abs/2211.01095))\n[2]: Pseudo Numerical Methods for Diffusion Models on Manifolds ([ICLR22](https://arxiv.org/abs/2202.09778))\n\n**Q3**: (Minor) When comparing performance, I recommend plotting FID on the y-axis and NFE on the x-axis for clarity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "They validated the method in various datasets and methods. \n\nIt is good to see that the method proposed from this work can be combined with the method from previous literature, i.e., DDIM-MCDO^\\dagger.", "weaknesses": "**1. strong assumption**\n\nAssuming that the manifold can be understood solely by considering variance is too strong an assumption. It might be better to tone it down to suggest that this approximation is sufficient for performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SfInZGKEke", "reviewer": "Reviewer_6dod", "summary": "The paper studies the exposure bias of accelerated diffusion model sampling from geometry perspective. The authors extend the previous manifold constraint theory with more detailed description of pixel variance, claiming that both exposure bias and truncation error account for the performance degradation. To this end, the authors propose to pre-calculate the reference pixel variance, which serves as a correction during inference. Such method achieves a training-free and easy-to-implement solution to performance degradation. Comprehensive experiments confirm the efficacy of the proposed algorithm.", "questions": "- It is intuitive that applying MCDO with larger NFEs or better sampler will achieve weaker improvements. I am curious about the comparison on better sampler like Heun or DPM-Solver. There is also no discussion about applicability on high-order samplers.\n\n- MCDO is proposed for manifold constraint to relieve exposure bias, how will the efficacy vary if different CFG scales are set? Larger CFG scale may also lead to severe exposure bias.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well structured and easy to follow. Most of the derivation is clear.\n\n- The paper extends the previous manifold constraint strategy with deeper study of the pixel variance and the consequent exposure bias, providing a novel perspective of geometric technique in diffusion models.\n\n- The proposed algorithm is overall both efficient and effective, inference time cost barely increases.\n\n- Quantitative experiments are convincing and extensive.", "weaknesses": "- There seem some theoretical flaws in the draft, harming the soundness:\n  1. Eq. (15) is wrong, when $x$ and $y$ are orthogonal, one can only deduce that $|x+y|^2=|x|^2+|y|^2$. Besides, what is the definition of $x_0$ and $\\epsilon_t$? $\\epsilon_t$ and $x_0$ may not be orthogonal if no further assumption are made.\n  2. I cannot understand the relation between Eq. (18) and analytical form of $Var(x^{(t)}_0)$ and $Var(x_t)$, $Var(\\cdot)$ is supposed to be the pixel variance as claimed in Eq. (12) which is a scalar.\n  3. Eq. (16) and Eq. (19) are similar, but the authors conclude differently. If minimizing right hand side in Eq. (19) could lead to the distance reduction, then so could minimizing right hand side in Eq. (16) be. They are all the **lower bounds** of distance of samples to manifold.\n  4. If the authors insist that nonzero $|x_0|$ affects the derivation, then (1) since $\\hat{x_t}\\in\\mathcal{M}_t$, one can simply choose $x_0=0$ (which is reasonable since the authors have already assumed zero mean in L346), or (2) move the term with $|x_0|$ outside the absolute value using $|a+b-c|\\geqslant|b-c|-|a|$, which is similar to the form of Eq. (19). The authors should make further clarification.\n  5. Why assume zero mean in L346? $\\hat{x}_0^{(t)}$ is the denoise observation at timestep $t$, somewhat a data sample with no noise. Then why is the case? The authors could calculate the mean to confirm the reasonability.\n\n- Figs. 4 and 6 only employ 64 samples, which seems inconvincing.\n\n- Visualization in Fig. 7 fails to be photorealistic with obvious color shift artifact, which is also the case in Fig. 12.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)