Elucidating the Exposure Bias in Diffusion Models
Authors: Mang Ning
Mingxiao Li
Jianlin Su
Albert Ali Salah
Itir Onal Ertugrul
Venue: ICLR 2024 poster (2024)
Keywords: Generative models
Diffusion models
Primary area: generative models

Abstract: Diffusion models have demonstrated impressive generative capabilities, but their exposure bias problem, described as the input mismatch between training and sampling, lacks in-depth exploration. In this paper, we investigate the exposure bias problem in diffusion models by first analytically modelling the sampling distribution, based on which we then attribute the prediction error at each sampling step as the root cause of the exposure bias issue. Furthermore, we discuss potential solutions to this issue and propose an intuitive metric for it. Along with the elucidation of exposure bias, we propose a simple, yet effective, training-free method called Epsilon Scaling to alleviate the exposure bias. We show that Epsilon Scaling explicitly moves the sampling trajectory closer to the vector field learned in the training phase by scaling down the network output, mitigating the input mismatch between training and sampling. Experiments on various diffusion frameworks (ADM, DDIM, EDM, LDM, DiT, PFGM++) verify the effectiveness of our method. Remarkably, our ADM-ES, as a state-of-the-art stochastic sampler, obtains 2.17 FID on CIFAR-10 under 100-step unconditional generation. The code is at https://github.com/forever208/ADM-ES

Published as a conference paper at ICLR 2024
ELUCIDATING THE EXPOSURE BIAS IN DIFFUSION MODELS
Mang Ning Utrecht University m. m.ning@uu. .nl
Mingxiao Li KU Leuven
mingxiao 1aliaao.11000
be
Jianlin Su Moonshot Al Ltd. bo jone@ spaces.
Albert Ali Salah Utrecht University
Itir Onal Ertugrul
Utrecht University
a.a. a. salaheuu........ n1
ABSTRACT
Diffusion models have demonstrated impressive generative capabilities, but their exposure bias problem, described as the input mismatch between training and sampling, lacks in-depth exploration. In this paper, we investigate the exposure bias problem in diffusion models by first analytically modelling the sampling dis- tribution, based step as the root cause of the exposure bias issue. Furthermore, we discuss potential solutions to this issue and propose intuitive metric for it. Along with the eluci- dation of exposure bias, we propose a simple, yet effective, training-free free method called Epsilon Scaling to alleviate the exposure bias. We show that Epsilon Scal- ing explicitly moves the sampling trajectory closer to the vector field learned the training phase by scaling down the network output, mitigating the input mis- match between training and sampling. Experiments on various diffusion frame- works (ADM, DDIM, EDM, LDM, DiT, PFGM++) verify the effectiveness our method. Remarkably, our ADM-ES, as a tofte--o-thearr stochastic sampler, ob- tains 2.17 FID on CIFAR-10 under 100-step unconditional generation. The code is at /
which we then attribute the prediction error at each sampling
com forever208 ADM-ES
INTRODUCTION
Due to the outstanding generation quality and diversity, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019) have achieved unprecedented success in image gen- eration (Dhariwal & Nichol, 2021; Nichol et al., 2022; Rombach et al., 2022; Saharia et al., 2022), audio synthesis (Kong et al., 2021; Chen et al., 2021) and video generation (Ho et al., 2022). Unlike generative adversarial networks (GANs) (Goodfellow et al., 2014), variational autoencoders (VAEs) (Kingma & Welling, 2014) and flow-based models (Dinh et al., 2014; 2017), diffusion models sta- bly learn the data distribution through a noise/score prediction objective and progressively removes noise from random initial vectors in the iterative sampling stage.
A key feature of diffusion models is that good sample quality requires a long iterative sampling chain since the Gaussian assumption of reverse diffusion only holds for small step sizes (Xiao al., 2022). However, Ning et al. (2023) claim that the iterative sampling chain also leads to the exposure bias problem (Ranzato et al., 2016; Schmidt, 2019). Concretely, given the noise prediction network (.(.), exposure bias refers to the input mismatch between training and inference, where the former always exposed to the ground truth training sample x; while the latter depends on the previously generated sample t., The difference between Z; and 2; causes the discrepancy between eo (xx) and eg (ăt), which leads to the error accumulation and the sampling drift (Li et al., 2023a).
We point out that the exposure bias problem in diffusion models lacks in-depth exploration. For ex- ample, there is no proper metric to quantify the exposure bias and no explicit error analysis for it. To shed light exposure bias, conduct a systematical investigation by first modelling the sampling distribution with prediction error. Based on our analysis, we find that the practical sampling distri- bution has a larger variance than the ground truth distribution every single step, demonstrating the

---

Published as a conference paper at ICLR 2024
analytic difference between 2; in training and b. in sampling. Along with the sampling distribution analysis, we propose a metric ó, to evaluate exposure bias by comparing the variance difference be- tween training and sampling. Finally, we discuss potential solutions to exposure bias, and propose simple yet effective Training-free and plug-in method called Epsilon Scaling to alleviate this issue.
We test approach extensive diffusion frameworks using deterministic and stochastic sampling, and on conditional and unconditional generation tasks. Without affecting the recall and precision (Kynkäääniemi al., 2019), our method yields dramatic Fréchet Inception Distance (FID) (Heusel et al., 2017) improvements. Also, we illustrate that Epsilon Scaling effectively reduces the exposure bias by moving the sampling trajectory towards the training trajectory. Overall, our contributions to diffusion models are:
We investigate the exposure bias problem depth and propose a metric for it.
We suggest potential solutions to the exposure bias issue and propose training-free -free plug- in method (Epsilon Scaling) which significantly improves the sample quality.
Our extensive experiments demonstrate the generality of Epsilon Scaling and its wide ap- plicability to different diffusion architectures.
2 RELATED WORK
Diffusion models were introduced by Sohl-Dickstein et al. (2015) and later improved by Song & Ermon (2019), Ho et al. (2020) and Nichol & Dhariwal (2021). Song et al. (2021b) unify score- based models and Denoising Diffusion Probabilistic Models (DDPMs) via stochastic differential equations. Furthermore, Karras al. (2022) disentangle the design space of diffusion models and introduce the EDM model to further boost the performance in image generation. With the advances in diffusion theory, conditional generation (Ho & Salimans, 2022; Choi et al., 2021) also flourishes in various scenarios, including text-to-image generation (Nichol et al., 2022; Ramesh et al., 2022; Rombach et al., 2022; Saharia et al., 2022), controllable image synthesis (Zhang & Agrawala, 2023; Liet al., 2023b; Zheng al., 2023), as well generating other modalities, for instance, audio (Chen et al., 2021; Kong et al., 2021), object shape (Zhou et al., 2021) and time series (Rasul et al., 2021). In the meantime, accelerating the time-consuming reverse diffusion sampling has been extensively investigated in many works (Song et al., 2021a; Lu et al., 2022; Liu et al., 2022). For example, distillation (Salimans & Ho, 2022), Restart sampler (Xu et al., 2023a) and fast ODE samplers (Zhao et al., 2023) have been proposed to speed up the sampling.
The exposure bias in diffusion models was first identified by Ning et al. (2023). They introduced training regularisation term to simulate the sampling prediction errors from the Lipschitz continuity perspective. Additionally. Li al. (2023a) alleviated exposure bias without retraining and their method involved the manipulation of the time step during the backward generation process. More recently, Li van der Schaar (2023) estimated the upper bound of cumulative error and optimized it during training. However, the exposure bias in diffusion models still lacks illuminating research in terms of the explicit sampling distribution, metric and root cause, which is the objective this paper. Besides, our solution to exposure bias is training- free and outperforms previous methods.
3 EXPOSURE BIAS IN DIFFUSION MODELS
3.1 SAMPLING DISTRIBUTION WITH PREDICTION ERROR
Given a sample Do from the data distribution ((zo) and a noise schedule B,, DDPM (Ho et al., 2020) defines the forward perturbation as g( (+,1) N N(z;; v1 - 3,1)). The Gaussian forward process allows us to directly sample x; conditioned on the input Zy:
(x: (zo) N(z;; Vonto. (1 -..)
2. Vin,00
v1 11e,
The reverse diffusion is approximated by neural network no (x [x)) (x 11 (10 t), a.I) and the optimisation objective is DxI (g(2: (x, 2o) || pe (x, 1 in which the posterior
(2 is tractable when conditioned on 2o using Bayes theorem:
1 (0 2o),
2

---

Published as a conference paper at ICLR 2024
1..11 Do 20) 1 αz
vou(1 (4-1))
B.
1 0x
Be
(4)
04 Zt
(3)
1 a2
Regarding the parametrization of (xte, (), Ho et al. (2020) found that using neural network to predict e (Eq. 6) worked better than predicting zo (Eq. 5) in practice:
n4 ( 1) = var-180
to (2,, 1) + var(1 ax-1)
It.
(5)
1 at
Or
3. VI c t)):
(Ie Vat
(6)
where zo (0;;)) 1) denotes the denoising model which predicts zo given Et. For simplicity, we use Tg the short notation of z (x, in the rest of this paper. Comparing Eq. 3 with Eq. 5, Song et al. (2021a) emphasise that the sampling distribution pe (0(2(-1|zt) is in fact parameterised as g ( 2g) where z means the predicted To given Ię. Therefore, the practical sampling paradigm is that we first predict e using eo (zt, t). Then we derive the estimation z for zo using Eq. 1. Finally, based on the posterior g (2,, Fo) It-1 is generated using q<tt-1/2 zb)) by replacing To with z/-
However, ( 2/)) q( (2,, holds only if xf Zo. this requires the network make no prediction error about 2o to ensure g( 12, x/)) share the same variance with g( [ on). However, zg zo practically non-zero and we claim that the prediction error of 2o needs to be considered to derive the real sampling distribution. Following Analytic-DPM (Bao et al., 2022), we model zá as po (Zolzz)) and approximate by a Gaussian distribution:
po (z (x)) N (x zo. e)
z
Zo + C&ED (Eo ~ ~((
Taking the prediction error into account, we now compute q( 1: ++1), ) which shares the same function format ( |z,,, mz)), ), by substituting with the index 1+1 and using &. to highlight that it generated in the sampling stage. Based on Eq. 2. we know ( x -# ns (#c+1, ++ 1), 3 I). Its mean and variance can be further derived according to Eq. 5 and Eq. 4, respectively. Thus, sample from the distribution is t. #0( 1)) BeT1eL namely:
VorBet1
s+1 1 +
Vor+T(1 aa))
-t+1 x + VBt4ee
(8)
Ø2+1
Plugging Eq. 7 and Eq. (using index 1+1 1) into Eq. 8, we derive the final analytical form of d; (see Appendix A.I for the full derivation):
f. = Varta + 11- ār + ( ( ext ext1) P
Varget1 0e+1
where, E1, 11,63~~N ((,,T). From Eq. 9, we can obtain the mean and variance of (z 26 For simplicity, we denote q 2, 1+1 as go (0t from now on. In Table 1, g(x (z)) shows the T, seen by the network during training while qo (#, (0++1) indicates the 2; exposed the network during sampling. The method of solving go (ž, (x+11) can be generalised to multi-step sampling (detailed in Appendix A.2). In the same spirit of modelling z as a Gaussian, we also derived the sampling distribution go (0( for DDIM (Song et al., 2021a) in Appendix A.3. Note that, one can also model æ/ as a Gaussian distribution whose mean is not Fo. but it does not affect the variance gap presented in Table 1 and the explosion of variance error (Section 3.2).
Table 1: The distribution ((x2) during training and go (R, x1++) during DDPM sampling.
Mean
Variance
( go (2, 11)+1)
Vazzo
(1 ã,)1
(1 āi + (/0, e
/ Va
1-00m
3

---

Published as a conference paper at ICLR 2024
3.2 EXPOSURE BIAS DUE TO PREDICTION ERROR
It is clear from Table I that the variance of the sampling distribution do (i, (+++1) is always larger than the variance of the training distribution (Z)) by the magnitude Varlt ((+1)² Note that, this variance gap between training and sampling is produced just in a single reverse diffusion step, given that the network Ea(-) can get access to the ground truth input Z What makes the situation worse is that the error of single-step sampling accumulates in the multi-step sampling, resulting in an explosion of sampling variance error. On CIFAR-10 (Krizhevsky et al., 2009), we designed an experiment to statistically measure both the single-step variance error of qo (#, xc+1) and multi-step variance error of go (#T()) using 20-step sampling (see Appendix A.4). The experimental results in Fig. indicate that the closer to = 1 (the end of sampling), the larger the variance error of multi- step sampling. The explosion of sampling variance error results in the sampling drift (exposure bias) problem and we attribute the prediction error z Do as the root cause of the exposure bias in diffusion models.
T-a211
Intuitively, a possible solution to exposure bias is us- ing sampling noise variance which is smaller than 3g, to counteract the extra variance term (Ya e+41)
multi-step
single step
0.4
errer enrrr
caused by the prediction error zi- Zo- Unfortunately, 3. is the lower bound of the sampling noise schedule 9, [&c, [r), where the lower bound and upper bound are the sampling variances given by ( being a delta func- tion and isotropic Gaussian function, respectively (Nichol & Dhariwal, 2021). Therefore, we can conclude that the exposure bias problem can not be alleviated by manipu- lating the sampling noise schedule 3..
veriance
0.1
0.0
6
8
10
12
14
16
18
timestep
Figure 1: Variance error in single-step and multi-step samplings.
Interestingly, Bao et al. (2022) analytically provide the optimal sampling noise schedule 8 which is larger than the lower bound B.. Based on what we discussed earlier, 87 would cause a more severe exposure bias issue than 3.. A strange phenomenon, but not explained by Bao et al. (2022) that 3; leads to a worse FID than using 3g under 1000 sampling steps. We believe the exposure bias is in the position to account for this phenomenon: under the long sampling, the negative impact of exposure bias exceeds the positive gain of the optimal variance GT.
3.3 METRIC FOR EXPOSURE BIAS
Although some literature has already discussed the exposure bias problem in diffusion models (Ning al., 2023; Li et al.. 2023a), we still lack a well-defined and straightforward metric for this concept. We propose to use the variance error of go ( to quantify the exposure bias at timestep / under multi-step sampling. Specifically, our metric & for exposure bias defined as 6, (V/3 - V where B, = ag denotes the variance of q( (m)) during training and 8 presents the variance of ( in the regular sampling process. The metric 6, is inspired by the Fréchet distance (Dowson where N the
& Landau, 1982) between (zo) and go ( which de (, dimensions of It, In Appendix A.6, we empirically find that 5, exhibits a strong correlation with FID given a trained model. Our method of measuring 5; is described in Algorithm 3 (see Appendix A.5). The key step of Algorithm 3 is that we subtract the mean / and the remaining term . - / /ā:=120 corresponds to the stochastic term of go (
3.4 SOLUTION DISCUSSION
We now discuss possible solutions to the exposure bias issue diffusion models based on the analysis throughout Section 3. Recall that the prediction error of x 2 is the root cause of exposure bias. Thus, the most straightforward way of reducing exposure bias is learning an accurate e or score function (Song& & Ermon, 2019) prediction network. For example, by delicately designing the network and dyyer--arameee EDM (Karras et al., 2022) significantly improves the FID from 3.01 to 2.51 on CIFAR-10. Secondly, believe that data augmentation can reduce the risk of learning
4

---

Published as a conference paper at ICLR 2024
inaccurate or score function for &, by learning denser vector field than vanilla diffusion models. For instance, Karras et al. (2022) has shown that the geometric augmentation (Karras et al., 2020) benefits the network training and sample quality. In the same spirit, Ning et al. (2023) augments each training sample ø, by a Gaussian term and achieves substantial improvements in FID. Additionally, Xu et al. (2022) claim that the Poisson Flow framework is more resistant prediction errors because high mass is allocated across a wide spectrum of the training sample, Our experiments verify the robustness of PGFM++ (Xu al., 2023b) and show that our solution exposure bias could further improve the generation quality of PGFM++ (more details in Appendix A.7).
worth pointing out that the mentineet methods require retraining the network and ex- pensive parameter searching during the training. This naturally drives us to the question: Can we alleviate the exposure bias the sampling stage, withou any retraining?
4
METHOD
4.1 EPSILON SCALING
In Section 3.2, we have concluded that the exposure bias issue can not be solved by reducing the sampling noise variance, thus another direction be explored in the sampling phase is the prediction of the network (-). Since we already know from Table 1 that 2, inputted to the network eg ee(-) in training differs from á; fed into the network (- in sampling, we are interested in understanding the difference in the output of (-(-) between training and inference.
For simplicity, we denote the output of (-) as eg in train- ing and éğ in sampling. Although the ground truth of Eg is not accessible during inference, we are still able to speculate the behaviour of e from the L2-norm perspec- tive. In Fig. 2. we plot the L2- L2-norm of eg and e at each timestep. In detail, given trained, frozen model and ground truth Fg, cg is collected by vg = eg (Ie, f). In this way, we simulate the training stage and analyse its prediction. In contrast, Eg is gathered in the real sampling process, namely eg = că, t). It clear from Fig. that the L2-norm of Eg is always larger than that of eg. Since & lies around æ, with larger variance (Section 3. we can know the network learns an inaccurate vector field eg ( I) for each à; in the vicinity of x, with the vector length longer than that of ( t).
51-
55.5
48
16
18
16
sampling direction
42
training
36-
sampling
2
8
12 timestep
14
16
18
20
Figure 2: Expectation of (|ea())llz dur- ing training and 20-step sampling on CIFAR-10. We report the 1.2noom ing 50k samples at each timestep.
One can infer that the prediction Eg could be improved if we can move the input (æ,, t) from the inaccurate vector field (green curve in Fig. 2) towards the reliable vector field (red curve in Fig. 2). To this end, we propose to scale down Eg by a factor X(t) at sampling timestep f. Our solution is based on the observation: ej and eg share the same input Fr ~ (O. I) timestep = T. but from timestep T - 1, & (input of GG) starts to diverge from o, (input of Eg) due to the 60(.) error made at previous timestep. This iterative process continues along the sampling chain and results in exposure bias. Therefore, we can push _ closer to I; by scaling down the over ver-predicted magnitude of ej. Compared with the regular sampling (Eq. 6). our sampling method only differs in the X term and co(st..t) ). Note that, our Epsilon Scaling serving as a is expressed as ei (z,, t) 1 (z; vI-a, Ax
yor plug-in method adds no computational load to the original sampling of diffusion models.
4.2 THE DESIGN OF SCALING SCHEDULE
Similar to the cumulative error analysed in Li & van der Schaar (2023), emphasise that the L2- norm quotient | e |- denoted as (t))), reflects the accumulated effect of the prediction
emors made at ancestral steps T 1,.... t +1. Suppose the 1.2-norm of e to be scaled at timestep t Ao, we have:
T-1 AN(() - 12 10 S. (DT l)dt+ + 1- (Ar-1 - 1)dt + +... 1.th 1+1
(D441 1)dt (10)
5

---

Published as a conference paper at ICLR 2024
given that the enror propagates linearly the sampling chain. The first term the right side of Eq. 10 corresponds to the error made at timestep T (the start of sampling) and propagated from T to t. Thus, after measuring (t),), one can derive the scaling schedule X(t) by solving Eq. 10.
As shown by Nichol Dhariwal (2021) and Benny & & 0 are very bad, with the loss larger than other timesteps by several orders of magnitude. Thereby, we can ignore the area close to / = to fit AN (L), because scaling a problematic better prediction. We plot the (f) curve in the cases of 20-step and 50-step sampling on CIFAR-10 Fig. 3 where (1) can be fitted by a quadratic function in the interval ~ (5. T)- Thus, the solution to Eq. 10 is a linear function A(t) where k, b are constants. Ideally, one should always first do simulations to measure || and || then solve out A(t) based on A (t). However, we propose to search for k, 6 because
Wolf (2022), the ee(.) predictions
steps
50 steps
1.14
1.12
1.10
(-(-) does not lead
1.08
1.06
1.02-
kt +
1.00
10
15
20
25
30
35
40
45
timestep
Figure 3: (N at each timestep.
find that the parameters do not change significantly in different networks. An added benefit of this proposal is that our approach becomes simulation-free. Moreover, Section 5.1, will that & would decay to 0 around 50-step sampling. Given this fact, decide uniform A(t) b for most experiments because of its effortless parameter searching and near- optimal performance.
RESULTS
In this section, we evaluate the performance of Epsilon Scaling using FID (Heusel et al., 2017). To demonstrate that Epsilon Scaling generic solution to exposure bias, we test the approach on various diffusion frameworks, samplers and conditional settings. Following the fast sampling paradigm (Karras al., 2022) in the diffusion community, we focus on T' < 100 sampling steps in this section and leave the FID results of T' > 100 in Appendix A.8. Our FID computation is consistent with (Dhariwal & Nichol, 2021) for equal comparison. All FIDs are reported using 50k generated samples and the full training set the reference batch. Lastly, Epsilon Scaling does not affect the precision and recall, and we report these results in Appendix A.9.
5.1
MAIN RESULTS ON ADM
Since Epsilon Scaling is a training-free method, we utilise the pre-trained ADM model as the baseline and compare it against our ADM-ES (ADM with Epsilon Scaling) datasets CIFAR- 10 (Krizhevsky et al., 2009), LSUN tower (Yu et al., 2015) and FFHQ (Karras et al., 2019) for unconditional generation and on datasets ImageNet 64 x4 and ImageNet 128 128 (Chrabaszez et al., 2017) for class- condiiionn generation. We employ the respacing sampling technique (Nichol & Dhariwal, 2021) to enable fast stochastic sampling.
Table 2: FID on ADM baseline. We compare ADM with our ADM-ES (uniform X(t)) and ADM-ES (lin- X()).. ImageNet 64x64 results are reported without classifier guidance and ImageNet 128 x 128 under clas- sifier guidance with 0c5
Table 3: We compare ADM-ES with recent stochastic diffusion (SDE) sam- plers regarding FID. We report their best
FID achieved under 7" sampling steps.
Unconditional
Unconditional
Conditional
Model
T
T"
Model
CIFAR-10
CIFAR-10 10
LSUN
FFHQ
ImageNet 64x64
ImageNet 128x128
32
32x
64x64
EDM (VP) (Karns et 2022)
511
2.27
100
ADM
14.52
2.71
3.55
(VE) (Karrss et al., 2022)
ADM-ES
2.17
2.91
6.77
2,39
3.37
Improved SDE (Kamss all, 2022)
1023
2.35
ADM
4.43
7.28
26.15
3.75
5.15
Restart (VP) IXu et 2023a)
115
2.21
50
ADM-ES
2.49
3.68
9.50
3.07
4.33
SA Solver (Kue
2.63
ADM-IP (Ning et 2023)
100
2.38
ADM
10.36
23.92
59.35
10.96
12.48
ADM (ours)
20
ADM-ES
5.15
8.22
26.14
7.52
9.95
ADM-ES EM (curs)
100
2.17
ADM-ES ES*
4.31
7.60
24.83
7.37
9.86

---

Published as a conference paper at ICLR 2024
Table 2 shows that independent of datasets and the number of sampling steps T", our ADM-ES outperforms ADM by a large margin in terms of FID. For instance, on FFHQ 128 128, ADM-ES exhibits less than half the FID of ADM, with 7.75, 16.65 and 34.52 FID improvement under 100, 50 and 20 sampling steps, respectively. Moreover, when compared with the previous best stochastic samplers, ADM-ES outperforms EDM (Karras et al., 2022), Improved SDE (Karras et al., 2022), Restart Sampler (Xu et al., 2023a) and SA-Solver (Xue et al., 2023), exhibiting state-of-the-ar stochastic sampler (SDE solver). For example, ADM-ES not only achieves better FID (2.17) than EDM and Improved SDE, but also accelerates the sampling speed by 5-fold to 20-fold (see Table 3). Even under 50-step sampling, Epsilon Scaling surpasses SA-Solver and obtains competitive FID against other samplers.
and ADM-ES applies the linear schedule X(t) = kt + in Table 2. We find that the slope k is approaching 0 as the sampling step T' increases. Therefore, we suggest uniform schedule X(t) for practical consideration. We present the complete parameters k, b used in all experiments and the details on the search of k, 6 in Appendix A. 10. A(t) is effortless and takes 6 to 10 trials. In Appendix A,11, also demonstrate that the FID gain can be achieved within wide range of X(t), which indicates the insensitivity of A(t).
Note that, ADM-ES uses uniform schedule A(t)
Overall, searching for the optimal uniform
5.2 EPSILON SCALING ALLEVIATES EXPOSURE BIAS
Apart from the FID improvements, we now show the exposure bias alleviated by our method using the proposed metric & and we also demonstrate the sampling trajectory corrected by Epsilon Scal- ing. Using Algorithm we measure ò, on the dataset CIFAR- under 20-step sampling for ADM and ADM-ES models. Fig. 4 shows that ADM-ES obtains a lower d; at the end of sampling / than the baseline ADM, exhibiting a smaller variance error and sampling drift (see Appendix A.12 for results other datasets).
Based on Fig. 2, we apply the same method to measure the L2-norm of (- ()) in the sampling phase with Epsilon Scaling. Fig. 5 indicates that our method explicitly moves the original sampling trajec- tory closer to the vector field learned in the training phase given the condition that the ||e (z((z))l )l2 is locally monotonic around Z₁. This condition is satisfied in denoising networks (Goodfellow al., 2016; Song & Ermon, 2019) because of the monotonic score vectors around the local maximal prob- ability density. We emphasise that Epsilon Scaling corrects the magnitude error of e (.). but not the direction error. Thus we can not completely eliminate the exposure bias to achieve 6g = 0 or push the sampling trajectory to the exact training vector field.
112
ADM ADM-ES
108
0.6
104
0.5
sampling direction
100
0,0.4
0.3
0.2
training sampling
0.1
88
0.0
84
sampling after Epsilon Scaling
2
10 timestep
16
18
0
3
6
9
12
15
18
21
timestep
Figure 4: Exposure bias measured by 8, on
Figure [e)2())l2 on LSUN 64x64. After apply-
LSUN 64x64, Epsilon Scaling achieves smaller o, at the end of sampling (1=1)
ing Epsilon Scaling, the sampling ||eall (blue) gets closer the training ||ea||2 (red).
5.3 RESULTS ON DDIM/DDPM
To show the generality of our proposed method, we conduct experiments on DDIM/DDPM frame- work across CIFAR-10 and CelebA 64x64 datasets (Liu et al., 2015). The results detailed in Table 4. wherein the designations 7) = 0 and = correspond to DDIM and DDPM samplers, respectively. The findings in Table 3 illustrate that our method can further boost the performance
7

---

Published as a conference paper at ICLR 2024
of both DDIM and DDPM samplers the CIFAR-10 10 and CelebA datasets. Specifically, our pro- posed Epsilon Scaling technique improves the performance of DDPM sampler on CelebA dataset by 47.7%, 63.1%, 60.7% with 20, 50, and 100 sampling steps, respectively. Similar performance improvement can also be observed on CIFAR-10 dataset. We also notice that our method brings less performance improvement for DDIM sampler. This could arise from the FID advantage of deter- ministic sampling under short sampling chain and the noise term in DDPM sampler can actively correct for errors made in earlier sampling steps Karras al. (2022).
Table 4: FID on DDIM baseline for unconditional generations.
Table 5: FID on EDM baseline and CIFAR-10 dateset (FID of EDM is reproduced).
CIFAR-10 32x32
CelebA 64X64
Unconditional
Conditional
T"
Model
T
Model
Heun
Fuler
Heun
Fuler
O
0
100
DDIM
4.06
6.73
5.67
11.33
35
EDM
1.97
3.81
1.82
3.74
EDM-ES (ours)
1.95
2.80
1.80
2.59
DDIM-ES (ours)
3,38
4.01
5.05
4.45
EDM
2.33
6.29
2.17
5.91
DDIM
4.82
10,29
6.88
15.09
21
EDM-ES
2.24
4.32
2.08
3.74
DDIM-ES
4.17
4.57
6.20
5.57
DDIM
8.21
20.15
10.43
22.61
13
EDM EDM-ES
7.16
12.28
6.69
10.66
20
6.54
8,39
6.16
6.59
DDIM-E'S
6.54
7.78
5.4 RESULTS ON EDM
We test the effectiveness of Epsilon Scaling on EDM (Karras et al., 2022) because it achieves state- of-the-art image generation under few sampling steps and provides unified framework for diffu- sion models. Since the main advantage of EDM is its Ordinary Differential Equation (ODE) solver, we evaluate our Epsilon Scaling using their Heun 2nd order ODE solver (Ascher & Petzold, 1998) and Euler 1st order ODE solver, respectively. Although the network output of EDM is not E, we still can extract the signal 6 at each sampling step and then apply Epsilon Sealing on E.
The experiments are implemented on CIFAR-10 dataset and we report the FID results in Table 5 using VP framework. The sampling step T' in Table 5 equivalent to the Neural Function Evalua- tions (NFE) used in EDM paper. Similar to the results on ADM and DDIM, Epsilon Scaling gains consistent FID improvement on EDM baseline regardless of the conditional settings and the ODE solver types. For instance, EDM-ES improves the FID from 3.81 to 2.80 and from 3.74 to 2.59 in the unconditional and conditional groups using the 35-step Euler sampler.
An interesting phenomenon in Table 5 is that the FID gain of Epsilon Scaling in the Euler sampler group is larger than that in the Heun sampler group. We believe there are two factors accounting for this phenomenon. On the one hand, higher-order ODE solvers (for example, Heun solvers) introduce less truncation error than Euler sf order solvers. On the other hand, the correction steps in the Heun solver reduce the exposure bias by pulling the drifted sampling trajectory back the accurate vector field. We illustrate these two factors through Fig. 6 which is plotted using the same method of Fig. 2. is apparent from Fig. that the Heun sampler exhibits a smaller gap between the training trajectory and sampling trajectory when compared with the Euler sampler. This corresponds to the truncation error factor in these two ODE solvers. Furthermore, in the Heun 2nd ODE sampler, the prediction error (cause of exposure bias) made in each Fuler step is corrected in the subsequent Correction step (Fig. 6(b)), resulting in a reduced exposure bias. This exposure bias perspective explains the superiority of the Heun solver in diffusion models beyond the truncation error viewpoint.
5.5 RESULTS ON LDM
To further verify the generality of Epsilon Scaling, we adopt Latent Diffusson Model (LDM) as the base model which introduces an Autoeoconder and performs the diffusion process in the latent space (Rombach et al., 2022). We test the performance of Epsilon Scaling (LDM-ES) on FFHQ 256 and CelebA-HQ 256 datasets using T' steps DDPM sampler. It is clear from Ta- ble 6 that Epsilon Scaling gains substantial FID improvements on the two high-resolution datasets, where LDM-ES achieves 15.68 FID under T = 20 on CelebA-HQ. almost half that of LDM.
8

---

Published as a conference paper at ICLR 2024
56
56
52
48
48
44
40
36
36
Eueer stop
32
32
28
training
24
Carrection step
training
sampling
sampling
24
10
timestep
timestep
(a) EDM: Euler 1e order sampler
(h) EDM: Heun 2nd order sampler
Figure 6: (-1l())l2 during training and sampling on CIFAR-10. We use 21-step sampling and report the L2-norm using 50k samples each timestep. The sampling is from right to left the figures.
Epsilon Scaling also yields better FID under 50 and 100 sampling steps CelebA-HQ with 7.36 FID at T" 100. Similar FID improvements are obtained on FFHQ dataset over different T".
Table 6: FID on LDM baseline using DDPM unconditional sampling.
CelebA-HQ
Model
256x 256
256 256
Finally, Epsilon Scaling is also effective on DiT (Peebles & Xie, 2023) which applies the ViT (Dosovitskiy et al., 2020) diffusion backbone in LDM. Please refer to Ap- pendix A.13 for the FID results on DiT baseline.
LDM
10.90
9.31
LDM ES (ours)
9.83
7.36
LDM
14.34
13.95
50
LDM
33.13
29.62
20
LDM ES
20.91
15.68
5.6 QUALITATIVE COMPARISON
In order to visually show the effect of Epsilon Scal- ing on image synthesis, we set the same random seed for the base model and Epsilon Scaling model in the sampling phase to ensure similar trajectory for both models. Fig. 7 displays the generated sam- ples using 100 steps on FFHQ 128 128 dataset. clear that ADM-ES effectively refines the sample issues of ADM, including overexposure, underexpo- sure, coarse background and detail defects from left to right in Fig. 7 (see Appendix A. 14 for more qual- itative comparisons). Besides, the qualitative com- parison also empirically confirms that Epsilon Scaling guides the sampling trajectory of the base model to an adjacent but better probability path because both models reach the same or similar modes given the common starting point æy and the same random seed at each sampling step.
Figure 7: Qualitative comparison between
ADM (first row) and ADM-ES (second row).
CONCLUSIONS
In this paper, we elucidate the exposure bias issue in diffusion models by analytically showing the difference between the training distribution and sampling distribution. Moreover, we propose training-free method to refine the deficient sampling trajectory by explicitly scaling the prediction vector. Through extensive experiments, we demonstrate that Epsilon Scaling a generic solution to exposure bias and its simplicity enables a wide range of diffusion applications.
REFERENCES
Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and allgeratiallaagbbraic equations, volume 61. Siam, 1998.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic- an analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICI.R, 2022.

---

Published as a conference paper at ICLR 2024
Yaniv Benny and Lior Wolf. Dynamic dual-output diffusion models. In CVPR, pp. 11482-11491, 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. ICLR, 2021.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llvr: Con- ditioning method for denoising diffusion probabilistic models. In ICCV, pp. 14367-14376, 2021.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. arXiv: (70700/77..88//// 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 34:8780-8794, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:/410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. ICLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
DC Dowson and B V666011 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 27, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //www www.wddeppeernnnngooo... org.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge local Nash equilibrium. NeurIPS, 30, 2017.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:
6840-6851, 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300. 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurIPS, 33:12104-12114, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine, Elucidating the design space of diffusion- based generative models. NeurlPS, 35:26565-26577 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Zhifeng Kong, Wei Ping Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. ICLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2024
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurlPS, 32, 2019.
Mingxiao Li, Tingyu Qu, Wei Sun, and Pranc-Fr Moens. Alleviating exposure bias in diffu- sion models through sampling with shifted time steps. arXiv preprini arXiv:2305. 15583, 2023a.
Yangming Li and Mihaela van der Schaar. On error propagation of diffusion models. In ICLR, 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, pp. 22511- 22521, 2023b.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In ICLR, 2022. URL https::/ httpp:://openrviee..o wtwpeenreeeeeeeee/eeeeeeeeeeeeee mertfform?id P1KWVd2yBKY.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurlPS, 35:5777--777 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In ICML, pp. 8162-8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, pp. 16784-16804. PMLR, 2022.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input per- turbation reduces exposure bias in diffusion models. /CML, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. ICCV, pp. 4195- 4205, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv:2204.06//5,, 2022.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. In ICLR, 2016.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assise assissee intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part 18, pp. 241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36499 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop Neural Generation and Translation @ EMNI.P 2019.

---

Published as a conference paper at ICLR 2024
Jascha Sohl-Dicksteii Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In JCML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In ICLR, 2022.
Yilun Xu, Ziming Liu, Max Tegmark, and Tommi Jaakkola. Poisson flow generative models. NeurIPS, 35: 35:12-82-,,795, 2022.
Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. arXiv preprint arXiv:2306. 14878, 2023a.
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi Jaakkola. Pfgm++: Unlocking the potential of physics-inspired generative models. In ICML, pp. 38566- 38591. PMLR, 2023b.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. arXiv preprint arXiv:2309.05019, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1X06::556.0335 2015.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv22302.055433 2023.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In CVPR, pp. 22490-22444 2023.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In ICCV, pp. 5826-5835 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I DERIVATION OF yo (#; 11+1) FOR DDPM
We show the full derivation of Eq. 3 below. From Eq. 11 to Eq. 12, we plug in zi
(++ 2o + ec+160
(Eq. 7) and T++1 v0u+120 + v1 ōet15 (Eq. 1), thus a sample from go (2: [(+1) is:
a. - uo 11 1)+ + V B14161
VonBr+1 1+
/1+ - (1.)
1 a+t1 I141 + V Be+1E1
(1I)
1 1+0
aet1
VorBr+1
(Eu + c0-1E0) + var+1(1 - ai)
(Vā1+130 + /1 a(t1c) + v Bet1c1
(12)
1 -+ā+11
act1
act1
VonB++1
Va(41(1 āi)
vayBet1
va(+1 - āz)
Va+ti20
V1 -0+1e++ v Boties
20
e4+16n
1 -(+1
1 a+t1
1 1-0441
a441
VarB++1 +
/1++1 aa)
at+1
Vārß++1
va++i(1 ā1)
/1 ās+1c+ + V Betia
Eu
C4+1ED
- act1
āct1
1- -i+1
Vor(1 0141) v0++1(1 Vá)
Vā
+ v - at)
- a + v BB41
20
+ 1 -a+11
1 1 -x+1
att1
Vār(1 - 0(+1) + ar+1(11 - -ar)vār
20 + vārbe41
C44160 + aa+1
ax)
v1 - - v Betiei
1-
1 -++1
Var(1 01+1 + a1411 Tut1)
VarBetz 141150 + Val+1(1 1011--p āi)
V 00+1e + Betier
20 +
1-0+11
1- 0++1
-a+11
6x4160 + Va++1(1 (it)
- ac+1e+ + v BeT1EL
= -a +
(13)
1-0++1 0:+1
1 a241
where, 60, 6, (O, From Eq. 13, we know that the mean of go ( (x++1) is Varzo. We now
focus on the variance by looking vā,le+2 1-0111 e--160 + (1--1())Ge)
a1t1E v Bettee
1-0111
Vānßit1
/0++1 -xt)
Var(
e(42)
)2
V
V1 ax+1) 2 *+ +B1+1
(14)
ac+1
1 --x+1
+ [a+++1 [1 at)
01+1)² + (1-āx)(1 a241)
e(41)² )
aut1
Cx+1
āc+1
SanBe41
22 a'41 αz)
(1 a,) -1+11
cx41)
1 1-4++1
1 x++1
+ 0/42(11 02)² + (1 51 a(41)
12+1)
64t1
04+1
Bot11
e2 ) + (1- a.)(a441 (1 - āa) + (1 - a(+1))
6ut1
1- -++1
(1 ae)(a
uz+1 a++1
12 +
C+4
C++1
1 04+1
22 + (1 - 0g ) [1 ar+1]
e(+1)² )2
a+t1
l(41) )2 +1 .t
(15)
x1+1
13

---

Published as a conference paper at ICLR 2024
A.2 DERIVATION OF go (# I(+1) AND MORE FOR DDPM
Since go (it-1 11)+1) contains two consecutive sampling steps: go (Ec xx++1 1) and qo ( (.x). we can
solve out so (#t-1 (0++1) by iterative plugging-in. According to go (#x [e+11 N 14 (2 + (0:
1), B++14) and Eq. 11, we know that 40
(0 (i:-1 (
1o 1), 3x1) and a sample from
4o (2: [2:) is:
21-1 1.. 3t
26 + var - )1-1)
2; + V Bre1.
(16)
at
1- at
From Table 1, we know that g0 ( [x++1 = N (Zxi Vā,zo. (1 - āi + T-a0+1 ec+1) 1)), so plug in
t. vārzo + V a. + vaiBit1
I-a.-1 ex41 E3 into Eq. 16, we know a sample from go (i; x(+1)
is:
1 at.
/0 (4-1)
( Varaot Varsoott11 1- ā, + ( Vān
- ex+1) P2z) + Bre. (17)
1
16 1 - a₁
Ox41
By denoting
(varde+ 111e11
2+1)) as ftt and plugging in zá-- = xo + Ce60 (Eq. 7), we have:
v
0
3r
[o e00p) + vo,(1 06-1))
Virtt + v σr + [(1)es) + BBa (18)
Φt-1
1 1 α7
22
3
(Do + +cEE))
+ Var(1 - Ō(-1)
(v Vatt0 vI - 2/1 a
1
f(t)Es) + V Be
1 āe
1 1-
(19)
/ 0 13
+ Vat-1
E160 + var(1 - -a--1)
Va,00
1 0t
1-
vayl ā(-1)
1
f(t)es) + VBee
(vI 0iE3 +
(20)
- āt
2/1 - ai
2 Va--120
+ vā
EGE
Varl (1 Ōi-1)
(V1 - at 2/1 αi
f(1))e - /.e.
1 1-0 0e
- ār
(21)
Taylor's theorem is used from Eq. 18 to Eq. 19. The process from Eq. 20 to Eq. 21 is similar to the simplification from Eq. 12 to Eq. 13. From Eq. 21, we know that the mean of go (#. [ is V 1Zg. We now focus the variance:
Var(a.-1)
vaa--18 e2) +
Varl a)-1)
VI σ4 +
1 1-a
/a a)-1)
+
[(1))
(22)
Oz
2 α1
0 31
a, ā, 1,-1)
1 - ā
+
e2) +
4(1 ā, )3 f(t)²
(23)
The above derivation similar to the progress from Eq. 14 to Eq. 15. Now we write the mean and variance of qo (((4)) in Table 7. In the same spirit of iterative plugging-in, could derive (Z)) which has the mean / Vā and variance larger than (1 - I.)
A.3 DERIVATION OF yo (z. 11++ FOR DDIM
We first review the derivation of the reverse diffusion P0 (z;)) for DDIM. To keep the symbols consistent in this paper, we continue to use the notations of DDPM in the derivation of DDIM.
14

---

Published as a conference paper at ICLR 2024
Table 7: The distribution g(zr-1(2)) during training and do (0 during DDPM sampling,
Mean
Variance
q ( FD)
140
(1 0
ạe ( - (1+1)
(1 at-1 + + 1-
var-
ex)²
or(1-81-13 f(t)²) 4(1-a,)
Recall that DDIM and DDPM have the same loss function because they share the same marginal distribution gz (o) (ax; y Va,to. (1 ae J.). But the posterior g( 1 [F, fo) of DDIM
obtained under Non-Markovian diffusion process and is given by Song et al. (2021a):
(( (111-1 200 (VVo VI ar-1 of
Ht V 0o
o.
(24)
VI āt
Similar to DDPM, the reverse distribution of DDIM is parameterized as pe pp(2z-1 (xp) (x where zg means the predicted Ia given Te. Based on Eq. 24, the reverse diffu- sion q x/)) is:
q a (2. 28)) N(Vov- / ar- - of Ft Varz
of
(25)
-T- ōg
z/)) ( xt. vn) holds only z = Zp. this requires the network make no prediction error about xo. Theoretically, need consider the uncertainty of the prediction zg and model it as a probabilistic distribution pe (Zo xx). Following Analytical-
Again, we point out that q
DPM (Bao et al., 2022), we approximate it by a Gaussian distribution po 10(2o/22) = (ari; e namely zg zn e,60- Thus, the practical reverse diffusion q( [ 2/))
((1--1 Dr. = ((VāL-1 = c,Eo) + V ā,-1 - oo It Vā, (Io + ceoo)
,T1). (26)
VI a4
Note that 0, 0 for DDIM sampler, so sample Iz- from g [2, x²) is:
It-1 v a (Io + etEn) + V δ,-1 -7
of.
F: v
0,E4
VI- ae
E₁ VaxD0
Votezen
v + Va-11 v1 -
v1 01-11-1
(27)
vI at
v1 1t
VānerED
Vā, V 1.-1 v1 a /1 ā, v1 Ct
(28)
Varez
10 + V σz +
+ (V0--14 - V1 04-1
)EO
(29)
-1-66 Ct.
From Eq. 27 to Eq. 28, we plug in x, Vagzo + - where ~ N/ T). We now
compute the sampling distribution q |1,1, o 1+1 which is the same distribution as g(z;-1 |x, x/) by replacing the index with + and using &, highlight is generated sample. According to Eq. 29, sample &, from q(t, (1411,55 x ++1
is:
Vor+lett1 )EO
f. = Vāzza VI 11-isss (Vāget+i VI aj
(30)
-- a++1
15

---

Published as a conference paper at ICLR 2024
From Eq. 30, we know the mean of q(i, 10+1, 26 is v Varoo.
We now calculate the variance by
looking at /1 aies + (vā /1 az Vautie JE::
1-ae+1
Var(t.) = (vI V āa)² (Vāre441 V1 - V0s+1e441
= ūp (Vāi v1 - VI a++1
/+++1
)'
vI di aivāi
= 1 āt + (Vā=
VI ac+1
6441
vI āi /or01
= 1 āi + (Vor(1
))²e2+1
a++1
= 1 āg + t,(1 Vot+1
0+++1 ) e
ar+1
1 a. + 0t+1
ac+1 )² 04
(31)
c++1
As a result, we can write the mean and variance of the sampling distribution g (2,1, 1111,, i.e. do (8. (T.-), and compare it with the training distribution (( in Table 8.
Table 8: The mean and variance of q( (z)) during training and go (i: xx+1) during DDIM sampling.
Mean
Variance
g( (zo)
Vanan
(1 āt.)
go (x141)
(1 - āi + (1
2111-1112
1-a
Since 04+1 < 1, ae+1 and (1 a++1 I-ae:1
hold for any t in Eq. 31. Similar to DDPM sampler, the variance of q(O, 12:-1 x1111 is always larger than that of ((x;/2o) by the magnitude (1 0011 1o411-a+11 ( Pā indicating the exposure bias issue in DDIM sampler.
1-0e-1
1-ac11
A.4 PRACTICAL VARIANCE ERROR OF do (0(2/(2441) AND qo (2,
We measure the single-step variance error of go (2, (141) and multi-step variance error of qo (0t (xr) using Algorithm and Algorithm 2, respectively. Note that, the multi-step variance error measure- ment is similar to the exposure bias &, evaluation and we denote the single-step variance error as A, and represent the multi-step variance error A, The experiments are implemented CIFAR-10 (Krizhevsky et al., 2009) dataset and ADM model (Dhariwal & Nichol, 2021). The key difference between A and A, measurement is that the former get access to the ground truth input each sampling step t, while the latter is only exposed to the predicted á, in the iterative sampling process.
16

---

Published as a conference paper at ICLR 2024
Algorithm Variance error under multi-step sampling
Algorithm 1 Variance error under single-step sampling
1: Initialize 8t = O, n = list() (vt e
1: Initialize A, 0, list() (Vt
{I,..., ....,--11])
2: fort :=:t=:1 T, T,... 1 do
2: repeat
3:
Zo q(zu), NOO,I)
3: repeat
4: 2T VūTID + vI Ü'Eé
4:
To q(u)), (O.I
for = T,.., ..., do
5:
I₁ (i,t0 + vI - dré
6:
if = Then $1 = IT
6:
21-1 = von (I) VI-a,
B1
ED ( (x..t)) +
I1-1 Vas ( ai (0 t)) +
BB2 x)),
B2 (z ~ ((,N
7:
R4-1 1.1..ppend( (p v (21)
8:
1a-1.append 20)
8: until 50k iterations
9: end for
9: end for
10: until 50k iterations
10: for := T ..... do
11: for / == T, T,... 1 do
3. numpy.var(ne)
12: 8r = -umppyvar(nn)
12: A, = 3.
13: A; = .. 3.
13: end for
14: end for
A.5 METRIC FOR EXPOSURE BIAS
The key step of Algorithm 3 is that we subtract the mean var 12 and the remaining term _; y 140 corresponds to the stochastic term of q( r/) In our experiments, we use N 50, 000 samples to compute the variance 3r.
Algorithm Measurement of Exposure Bias O
1: Initialize & = 0, list() (Vt E (1,..., 1})
2: repeat
3:
q(0)). ~ .VO0
4: compute FT using Eq.
5: for T ...., 1 do
6:
if T' then 8; = IT
7: 2, = Var (#: 8.
V1-a, E0 t)) + v Bi (2~ ((0,
8: Te-1.oppend(:1-1 120)
9: end for
until N iterations
11: for := T,..., 1..1o do
12: B. mumppy-vrr((y
6, = (v 3. 3
14: end for
A.6 CORRELATION BETWERN EXPOSURE BIAS METRIC AND FID
0.50
We define the exposure bias at timestep as 5, (VB, / 2 where 3
ADM-ES
0.48
- og denotes the variance of ((xx) during training and 8 presents the variance of go (á; (TT) in the regular sampling process. Although measures the discrepancy between network inputs and FID to evaluate the difference between training data and network outputs,
0.46
0.42
0.40
empirically find strong correlation between 8, and FID, which could arise from the benefit of defining 5, from the Fréchet distance Dowson & Landau
0.38
0.36
8
10
FID
Figure 8: Correlation between FID 51.
17

---

Published as a conference paper at ICLR 2024
(1982) perspective. In Fig. 8, present the FID-o1 relationships CIFAR-I0 and use 20-step sampling, wherein d1 represents the exposure bias the last sampling step = 1. Additionally, of has the advantage of indicating the network imput quality at any intermediate timestep t. Taking Fig. as an example, we can see that the input quality decreases dramatically near the end of sampling 1) as 6g increases significantly.
A.7 FID RESULTS ON PFGM+ BASELINE
In PFGM, Xu et al. (2022) claim that the strong norm-t correlation causes the sensitivity of prediction emors in diffusion frameworks, and PFGM is more resistant to prediction errors due to the greater range of training sample norms. Our experiments verify the robustness of Poisson Flow frameworks by observing that Epsilon Scaling enjoys wider range of X(() on PFGM+ baseline (Xu et al., 2023b) than EDM baseline. The rationale is simple: that require the network to be robust to counteract. However, we observe that the exposure bias issue still exists Poisson Flow frameworks even though they are less sensitive prediction errors than diffusion frameworks. Therefore, Epsilon Scaling is applicable in Poisson Flow models. Table and Table 10 show the significant improvements made by Epsilon Scaling (A(t) = b) on PFGM++
inappropriate X((1) introduces
Table 9: FID on CIFAR-10 using +FGM++ baseline (D=128, uncond).
T"
Model
9
13
21
35
PFGM+
37.82
7.55
2.34
1.92
17.91
4.51
2.31
1.91
PFGM+ with ES
(b=1.008)
(b=1.016)
(b=0.970)
(b=1.00045)
Table 10: FID on CIFAR- 10 using PFGM++ baseline (D=2048, uncond).
T"
Model
9
13
21
35
PFGM++
37.16
7.34
2.31
1.91
17.27
4.88
2,20
1.90
PFGM++ with ES
(b=1.007)
(b=1.015)
(b=0.996)
(b=1.0007)
A.8 FID RESULTS UNDER T" > 100
Although using sampling step T" = 100 often achieves the near-optimal FID in diffusion models, we still report the performance of our Epsilon Scaling in the large T" regions. We show the FID results Analytic-DDPM (Bao al., 2022) baseline (Table 11) and ADM baseline (Table 12), where we apply A(t) = for Epsilon Scaling.
Table 11: FID CIFAR- using Analytic-DDPM baseline (linear noise schedule).
Table 12: FID on CIFAR-10 using ADM baseline.
T'
7"
20
50
100
200
400
1000
200
300
1000
Analytic-DDPP DDMti
14.61
7.25
5,40
4.01
3.62
4.03
ADM
3.04
2.95
3.01
Analytic-DDPM with ES
11.02
5.03
4.09
3.39
3.14
3.42
ADM-ES
2.15
2.14
2.21
18

---

Published as a conference paper at ICLR 2024
A.9 RECALL AND PRECISION RESULTS
Our method Epsilon Scaling does not affect the recall and precision of the base model. We present the complete recall and precision (Kynkäänniemi al., 2019) results in Table 13 using the code provided by ADM (Dhariwal Nichol, 2021). ADM-ES achieve higher recalls and slightly lower previsions across the five datasets. But the overall differences are minor.
Table 13: Recall and precision ADM and ADM-ES using 100-step sampling.
CIFAR-10 32x32
LSUN tower
FFHQ 128 x 128
ImageNet 64x64
ImageNet 128 x 128
Model
64x64
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
ADM ADM-ES
0,591
0.691
0.605
0.645
0.497 0.545
0.696 0.683
0.621
0.738 0.632
0,586
0.771 0.592
0.613
0.684
0.606
0.641
0.726
0.771
A.
EPSILON SCALING PARAMETERS: b,b
We present the parameters k, b of Epsilon Scaling used in all of our experiments in Table 14, Table 15 and Table 16 for reprodueibility. Apart from that, we provide suggestions on finding the optimal parameters even though they are dependent on the dataset and how well the base model is trained. Our suggestions are:
Search for the optimal uniform schedule A(t) = in coarse-to-fine manner: use stride 0.001, 0.0005, 0.0001 progressively.
In general, the optimal b will decrease the sampling step T increases.
After getting the optimal uniform schedule A(t)" = b", we search for i of the linear sched- ule X(t) = kt + by keeping (A(()= = EX((),, thereby, in the linear schedule calculated rather than being searched.
Instead of generating 50k samples, using 10k samples to compute FID for searching A(t).
Table 14:
Epsilon Scaling schedule A(t) = kt + we used on ADM baseline. We keep the FID
results in the table for comparisons and remark k, b underneath FIDs
Unconditional
Conditional
rT"
Model
CIFAR-I0 32
L.UU tower 64
FFHQ x28x 128
TmageNNtt x4
ImageNet 128 128
100 ADM
3.37
3.59
14.52
2.71
3.55
ADM-ES
(b=1.017)
(b=1.006)
(b=1.005)
(b=1.006)
(b=1.004)
ADM
4.43
7.28
26.15
3,75
5.15
2.49
3.68
9.50
3.07
4,33
ADM ES
(b=1.017)
(b=1.007)
(b=1.007)
(b=1.06)
(b=1.004)
ADM
10.36
23.92
10.96
12.48
20
5,15
8.22
26.14
7.52
9.95
(b=1.017)
(b=1.011)
(b=1.008)
(b=1.066)
(b=1.005)
4.31
7.60
24.83
9.86
ADM-ES'
(k=0.0025, b=110)
(k=000008, b=1.0034)
(k=0.0004, b=1.002)
(k=0.0002, b=1.0041)
(k=0.00022, b=1.00291)
A.11 INSENSITIVITY OF A(I)
We empirically show that the FID gain can be achieved within a wide range of A(t). Table 17 and Table 18 demonstrate the insensitivity of our hyperparameters, which ease the search of X(t).
19

---

Published as a conference paper at ICLR 2024
Table 15: Epsilon Scaling schedule ((t) kt + (I: 0) we used on DDIM/DDPM and LDM baseline. We keep the FID results in the table for comparisons and remark underneath FIDs
T
Model
CIFAR-IO 32x32
CelobA 64x64
T
Model
FFHQ 256
ColebA-HQ 256
0
=
0
DDIM
4.06
6.73
5.67
11.33
LDM
10.90
9.31
3.38
4,01
5.05
4.45
9.83
7.36
DDIM-ES
(b=1.0014)
(b=1.03)
(b=1.003)
(b=1.04)
LDM-ES
(b=1,00015)
(b=1,0009)
LDM
4.17
4.57
6.20
5.57
11.57
9.16
DDIM-ES
(b=11.0030)
(b=1.04)
(b-1.004)
(b=1.08)
LDM-FS
(b=110016)
(b=1.003)
DDIM
8.21
20.15
10.43
22.61
LDM
33.13
20
20
6.54
7.78
10.38
11.83
20.91
15.68
LDM-ES
(b=1.0052)
(b=1.05)
(b=1.001)
(b=1.06)
(b=1.007)
(b=1.010)
Table 16: Epsilon Scaling schedule A(t) = kt + b, (I: = 0) we used on EDM baseline. We keep the FID results in the table for comparisons and remark underneath FIDs
Unconditional
Conditional
T"
Model
Heun
Euler
Heun
Euler
EDM
1.97
3.81
1.82
3.74
35
2.80
1.80
2.59
EDM-ES
b=1.0055
b=1.0034
b=1.0006
b=1.0035
EDM
2.33
6.29
2.17
5.91
21
2.24
4.32
2.08
3.74
EDM-ES ES
b=0.9985
b=1.0043
b=0.9983
b=1.0045
EDM
7.16
12.28
6.69
10.66
13
6.54
8.39
6.16
6.59
EDM-ES
b=1.0060
b=1.0048
b=1.0070
b=1.0051
Table 17: FID of ADM-ES achieved on CIFAR-10 under different A(I) (X(t) = bb) and unconditional sampling. b=1 represents ADM.
(baseline)
1.015
1.016
1.017
1.018
1.019
100
3.37
2.20
2.18
2.17
2.21
2.31
(baseline)
1.015
1.016
1.017
1.018
1.019
T' 50
4.43
2.53
2.51
2.49
2.53
2.55
Table 18: FID EDM-ES achieved on CIFAR-IO under different A(t) (A(t) = b) and unconditional Heun sampler, b=1 represents EDM.
b
(baseline)
1.0005
1.0006
1.0007
1.0008
T'
35
1.97
1.948
1.947
1.949
1.953
b
(baseline)
1.004
1.005
1.006
1.007
T'
= 13
7.16
6.60
6.55
6.54
6,55
20

---

Published as a conference paper at ICLR 2024
A.12 EPSILON SCALING ALLEVIATES EXPOSURE BIAS
In Section 5.2, we have explicitly shown that Epsilon Scaling reduces the exposure bias of diffusion models via refining the sampling trajectory and achieves lower 6, on CIFAR-10 dataset.
We now replicate these experiments on other datasets using the same base model ADM and 20-step sampling. Fig. 9 and Fig. 10 display the corresponding results CIFAR-10 10 and FFHQ 128x 128 datasets. Similar to the phenomenon on LSUN tower 64x64 (Fig. and Fig. 4), Epsilon Scaling consistently obtains smaller exposure bias ó and pushes the sampling trajectory to the vector field learned in the training stage.
ADM
0.4
ADM-ES
0.3
48
6.
45
training
sampling
36
sampling after Epsilon Scaling
2
4
8
10
12
14
16
18
12
21
timestep
timestep
(a) Exposure bias measured by on CIFAR-10
(b) L2-norm of c- CIFAR-10
Figure 9: Left: Epsilon Scaling achieves smaller ó at the end of sampling (! 1). Right: after applying Epsilon Scaling, the sampling |eall2 (blue) gets closer to the training ||el|] (red)
0.8
225
ADM ADM-ES
0.7
210
0.6
195
0.5
0.0.4
180
0.3
165
0.2
150
training
0.1
135
sampling
sampling after Scaling
120
2
4
8
10
12
14
16
18
0
6
12
15
18
21
timestep
timestep
(a) Exposure measured by 6, on FFHQ x28x 128
(b) L2- norm of co(-) on FFHQ 128x 128
Figure 10: Left: Epsilon Scaling achieves smaller at the end of sampling (t 1). Right: after applying Epsilon Scaling, the sampling |eal| (blue) gets closer to the training ||Eal|p (red).
1. FID RESULTS ON DIT BASELINE
In addition to UNet (Ronneberger et al., 2015) backbone, also test Epsilon Scaling diffusion models using Vision Transformers (Dosovitskiy et al., 2020). Table 19 presents the FID of DiT (Pee- bles & Xie, 2023) and our DiT-ES ImageNet 256x256 under uniform X(t) b. Again, Epsilon Scaling achieves consistent sample quality improvement over different sampling steps, indicating that the exposure bias exists in both UNet and Transformer backbones.
21

---

Published as a conference paper at ICLR 2024
Table 19: FID on ImageNet 256x256 using DiT baseline
T
Model
20
50
100
DiT
12.95
3.71
2.57
10.00
3.30
2.52
DiT-ES
(b=0.965)
(b=0.9899)
(b=0.995)
A.14 QUALITATIVE COMPARISON
In Section 5.6, we have presented the sample quality comparison between the base model sampling and Epsilon Scaling sampling on FFHQ 128x 128 dataset. Applying the same experimental settings, show more qualitative contrasts between ADM and ADM-ES on the dataset CIFAR-10 32 32 (Fig. 11), LSUN tower 64x 64 (Fig. 12), ImageNet 64 (Fig. 13) and ImageNet 128x 128 (Fig. 14). Also, we provide the qualitative comparison between LDM and LDM-ES on the dataset CelebA-HQ 256x 256 (Fig. 15). These sample comparisons clearly state that Epsilon Scaling effec- tively improves the sample quality from various perspectives, including illumination, colour, object coherence, background details and so on.
Figure 11: Qualitative comparison between ADM (first row) and ADM-ES (second row) on CIFAR- 10 32x32 using 100-step sampling.
Figure 12: Qualitative comparison between ADM (first row) and ADM-ES (second row) on LSUN tower 64 x 64 using 100-step sampling.
22

---

Published as a conference paper at ICLR 2024
Figure 13: Qualitative comparison between ADM (first row) and ADM-ES (sccond row) on Ima-
geNet 64 64x64 using 100-step sampling.
Figure 14: Qualitative comparison between ADM (first row) and ADM-ES (sccond row) Ima-
geNet 128x 128 using 100-step sampling.
Figure 15: Qualitative comparison between LDM (first row) and LDM-ES (second row) on CelebA-
HQ 256 x 256 using 100-step sampling.
23


Reviews:
[{"review_id": "2roaJSJf2c", "reviewer": "Reviewer_nv6a", "summary": "This paper take account into the exposure bias problem in the diffusion model, which is the distribution shift between the distribution derived by the forward diffusion process and the (learned) reverse diffusion process at the same time. The exposure bias phenomenon starts with the observation that the expected noise (or signal) from a noisy input, or a drawn sample within the trajectory, is not accurately evaluated. Precisely, the variance of the expected signal given the more-noisy signal is greater in the generative reverse process than in the forward process. The intuitive approach to overcome this is to directly downscale the noise variance term with respect to the variance ratio, but from existing works, this is infeasible because when doing this, the output variance term will be outbounded to the (theoretically available) noise schedule. However, these existing works does not explain the ill performance in the low-NFE regime. This leads to the hypothesis that the negative effect of the exposure bias exceeds the gain of the optimal variance.\n\nEven though there exist some works that deal with the exposure bias, they all consider re-training of the existing diffusion models, which is costly and sometimes require additional hyperparameter tuning. To alleviate this, this paper proposes a new calibration method called epsilon scaling. The method of epsilon scaling is derived simply by a data-driven approach: draw expected noise from the training set (noisy data as input) and from sampling trajectory (intermediate particle in the trajectory) and compare the expected noise term. The motivating experiment shows that the noise term of the sampling data is always greater than that of the training data, and can be intuitively scaled via the ratio between the expectation of the training and sampling estimated noise mean. The experimental session shows that in a mid-NFE regime (20~100), this shows superior performance compared to non-calibrated cases. Moreover, the paper validates that this method alleviates the exposure bias.", "questions": "* What about the case that the NFE is less than 20 or greater than 100?\n * At least for small NFE, I recommend adding some ablation studies on fully-searched $\\lambda_t$ with respect to all timesteps, rather than fixing this to the fixed value $b$ or taking a linear approximation.\n * It will more support the method, if the bias of the expected noise $\\epsilon_\\theta^s$ or $\\epsilon_\\theta^t$. This helps the reader to understand that by simply scaling down the expected noise calibrates the sampling steps, without taking bias (i.e. translating) the noise.\n\n======\n\nMiscellaneous\n * In the line below Equation 7, does $q ( x_t|x_{t+1},x_\\theta^{t+1})$ have the same distribution as $q(x_{t+1}|x_t,x_\\theta^t)$?\n * The exposure bias term $\\delta_t$ should be more precisely denoted in the main section. This proposed metric is only introduced in detail in the appendix section.\n * It will be helpful if the Analytic-DPM and the existing exposure bias methods are also considered as the benchmark, even though this does not use the baseline reverse diffusion process by modifying \n * It will be also helpful also to be mentioned if the Analytic-DPM requires the heavy clipping of the noise variance $\\beta_t$ in the near-signal part of the sampling, which causes", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The background and the motivating section is well clarified, by first notifying the necessity of calibrating the expected noise term in the diffusion sampling, and then compare the existing method to the newly proposed method.\n* To the best of our knowledge, this paper is the first training-free method that calibrates the distribution drift (e.g. exposure bias), by modifying the neural network output with some dataset statistics.\n* The experimental section showed that this training-free method works as a pipeline over a variety of existing diffusion model methods, including the early DDIM/ADM to the more recent LDM/EDM.", "weaknesses": "* Although this method is introduced as a simulation-free method, it is not completely simulation-free; in order to determine $\\lambda_t$ for each timestep, one can compute the dataset statistics with respect to all intermediate trajectory particle, which require some simulation. (But not heavy.)\n * The assumption that the model value $x_\\theta^t$ is averaged to the true $x_0$ should be more verified.\n * In the experimental section, only constant or linearized values are used as the scaling schedule $\\lambda_t$. This implies that constant reduction of the expected noise is helpful for the sampling process.\n * Analytic-DPM is also a simulation-free (only calculates the optimal variance over the timesteps) This paper proposed that the limit of the Analytic-DPM method is that this is not advantageous in the low-NFE regime, but this paper reproduces well even in low-NFE regime (NFE=10), if the variance clipping in high-SNR (last or second last sampling step) timesteps is held. The fair comparison in various NFE regime should be done, which can affect the scoring of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Klyh6hNhj", "reviewer": "Reviewer_FqVz", "summary": "The paper studies the \"exposure bias\" in diffusion models, which boils down to the statistical discrepancy between neural predictions at different time $t$. To alleviate the issue, the authors propose a simple scaling strategy to match the norm of neural predictions at training and sampling time. The idea is to calculate the related empirical statistics and then determine the scaling factor at different time $t$. Empirically, the proposed scaling method improve the baseline across architectures, noisy schedule and datasets.", "questions": "- I don't immediately see why the quotient of $\\Delta N(t) = |\\epsilon^s|/|\\epsilon^t|$ can be translated into the scaling factor at $\\epsilon(x_t,t)$ during sampling. Because $\\Delta N(t) = |\\epsilon^s|/|\\epsilon^t|$ and $\\Delta N(t)=\\int_t^T \\lambda_t$ only implies that $\\frac{d |\\epsilon^s(x_t,t)|/|\\epsilon^t(x_t,t)|}{dt}=\\lambda_t$. \n\nI tried the simplest data distribution I can come up with --- $p(x)=\\delta(x-0)$. The argument even failed in this case. (I use the notation in EDM and assume $t=\\sigma$). The true $\\epsilon$ at $x_t$ is $\\epsilon(x_t,t) = -\\frac{x_t}{t^2}$. By solving the diffusion ODE $dx/dt = -\\frac{x_t}{t}$ we get $x_t = \\frac{Tx_T}{t}$ (assume the start point at time $T$ is $x_T$). However, when we scale $\\epsilon$ by $\\lambda_t$, the resulting $x_t$ does not directly relate to $\\Delta N(t)=\\int_t^T \\lambda_t$, but a interval over $\\lambda_t/t$. I suggest rethinking the principled way of scaling by working out the toy example first.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper tries to address an important issue in diffusion models, where the initial error accumulation can negatively affect the quality of the generated samples.\n\n- It introduces a method that employs the empirical $\\ell_2$ ratio during both training and sampling phases to decide the appropriate scaling factor. This technique is straightforward yet proves to be effective.\n\n- Extensive experiments show that the proposed method can consistently improve the pre-trained models across datasets.", "weaknesses": "- Several prior works have observed and identified the \"exposure bias\" problem studied in the current paper. It would be helpful to discuss them in the paper: (1) Section 4 (practical considerations) and Fig 13 in EDM [1] points out that the neural network tends to remove slightly too much noise. Hence they use an inflated noise to counteract it. I think adding $S_{noise}$ into ODE or SDE samplers is a valid baseline for the current paper. (2) Section 4.2 in PFGM [2] / Section 5, Fig 4.b in PFGM++ [3] further dig into the exposure bias problem and show that the strong norm-t relation in the diffusion model is the cause. \n\n- The current paper demonstrates the discrepancy of the prediction norms (Fig 2) without further examining the reason. [2] seems to offer a plausible explanation for this phenomenon. If their argument is true, this \"exposure bias\" mainly occurs in large $t$ (norm-t is strongly correlated). I guess one simple experiment to validate the hypothesis is to use true score up to a certain time $t$ during sampling and study the effects (like current Fig.2, but the green and red curve would be the same in [t,T]).\n\n\n- It's encouraging to see that the method improves over EDM, especially in the small NFE regime. Could the authors try to apply the method on PFGM++, which is claimed to be more robust to the \"exposure bias\"? It could also verify if or not the exposure bias is due to the strong norm-t correlation pertaining to diffusion models.\n\n### Minors\n\n- I think the prediction error is not $x^t_\\theta - x_0$ but $x^t_\\theta - E_{x_0|x_t}[x_0]$.\n\n[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022\n\n[2] Poisson Flow Generative Models, NeurIPS 2022\n\n[3] PFGM++: Unlocking the Potential of Physics-Inspired Generative Models, ICML 2023\n\nOverall I'm impressed by the empirical performance of the proposed methods. However, there are many loose threads (the reasoning of exposure bias, the question below, comparing to related work (e.g., EDM inflating). I will raise my score if the authors address some of them.\n\n## Update\n\nI would like to thank the authors for providing the extra experiments. I raised my score to 6.\n\nFor Q4, sorry I made some mistakes in my previous question. What I mean is that if we deliberately scale the network prediction $\\epsilon_\\theta$ by $\\lambda_t$ (i.e., the network prediction error studied in the paper), the resulting solution seems not related to the $\\Delta N(t)=\\int_t^T \\lambda_t$. Concretely, the new ODE is $dx_t /dt = \\lambda_t /t x_t $, and the solution is related to the time integration of $t\\lambda_t$. So the argument in the paper appears to be inconsistent with this simple example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hOezNHNdq9", "reviewer": "Reviewer_ujfx", "summary": "This paper investigates the exposure bias problem in diffusion models by modeling sampling distribution, based on which they attribute the prediction error at each sampling step as the root cause of the exposure bias issue. They discuss potential solutions to this issue and propose a metric for it. Along with the elucidation of exposure bias, we propose a simple, yet effective, training-free method called Epsilon Scaling to alleviate the exposure bias. \nExperiments on various diffusion frameworks, unconditional and conditional settings, and deterministic vs. stochastic sampling verify the effectiveness of this method.", "questions": "Please refer to the questions in the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is interesting and insightful to take in-depth exploration on the exposure bias problem in diffusion models. This paper connects exposure bias with prediction error and gives the expressions of prediction error.\n\n2. To solve the exposure bias in a learning-free manner, this paper proposes to scale the noise prediction in the sampling process to match the noise prediction in the training process. \n\n3. Solid experiments. Extensive experiments demonstrate the generality of Epsilon Scaling and its applicability to different diffusion architectures.", "weaknesses": "1. The main concern lies in the assumption at the start of the derivation. In part 3.2, the authors assume that the reconstructed image x_{\\theta}^t at the sampling process follows the Gaussian distribution, where the mean is the GT image x_0, and the variance is the Gaussian noise. This conflicts with some intuitive observations. For example, the reconstructed image x_{\\theta}^t is often a degraded version of the GT image, and the mean of x_{\\theta}^t is different from x_0. \n\n2. Some formulations, notations and explanations in part 3.1 and 3.1 are redundant. It is highly recommended to improve the organization and the writing of this section.\n\n3. It is necessary to discuss the further meaning and defects of exposure bias correction. For instance, the necessity of matching the forward and sampling process. Reducing such bias forces the generated images to be more similar to the training images in distribution, i.e, more data repetition in the generated images. The authors are encouraged to give more discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwoxkdcHN7", "reviewer": "Reviewer_gcKq", "summary": "This work focuses on the `exposure` bias problem for diffusion models. The exposure bias is caused by error accumulation across the sampling trajectory since during inference only samples from the previous timestep step are used, while during training samples are exposed to ground truth training. The work then quantifies and analyzes the exposure bias during inference, during different timesteps showing the error is high by the end of sampling (for multi-step sampling) and shows a monotonic trend. Lastly, the paper proposes epsilon scaling as an inference time sampling strategy to mitigate this issue and shows higher quality generations compared to other samplers while (slightly) improving FID.", "questions": "1. Does the method perform worse than Heun solvers with suboptimal $k$ and $b$ selection?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The work does an overall good job of introducing and explaining exposure bias. The section itself was didactic and a valuable portion of the paper. Results are shown for simple single and multi-step samplings that quantify the variance error caused by exposure bias.\n2. The work then proposes a simple approach to reduce the exposure bias, by scaling the norm of the predicted noise term. The work also shows that the scaling factor needs to be handled differently when sampling for different number of steps. \n3. Results are shown for several different variants of diffusion models, and several baseline samplers are considered. The proposed method shows competitive FID scores with similar or fewer timesteps.", "weaknesses": "1. Results for the proposed epsilon-scaling mechanism with DDIM solver aren't impressive. There's only a marginal change in the FID score, compared to the DDIM sampler which is used as a popular framework for many diffusion models. Here, the gains are marginal even for cases with reasonable timesteps (~50 or more).\n2. There is little information provided regarding how the scaling factor $k$ and $b$ are selected, and how many hyperparameters were searched for optimal $k$ and $b$. It would also be good to quantify sensitivity to these hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owDCamiQ6I", "reviewer": "Reviewer_tyf7", "summary": "This paper studies the problem of exposure bias in diffusion models.  Exposure bias is defined as the input mismatch between $x_t$ during training and $\\hat{x}_t$ during sampling, which results error accumulation during sampling and eventual sampling drift. This in turn affects the quality of generated images. \n\nThe primary cause of exposure bias is ascribed to the difference between the growth truth posterior distribution $q(x_{t-1} | x_t, x_0)$ and the sampling distribution $q(x_{t-1} | x_t, \\hat{x}^t_\\theta)$. In practice, the difference between the prediction $\\hat{x}_\\theta^t$ and ground truth $x_0$ is non-zero. The paper analytically shows that this is due to increased variance at sampling time which results in accumulation of error during sampling. Inspired by the error in variance, a metric to measure exposure bias is also proposed by drawing inspiration from FID. \n\nTo alleviate exposure bias, this work proposes a method called Epsilon scaling which can be used at sampling and does not require re-training/finetuning of diffusion models. The core idea is to scale down the predicted $\\epsilon_\\theta^s$ at sampling so that it is closer to training $\\epsilon_\\theta^t$.  \n\nQualitative and quantitative results indicate that the proposed method results in improved FID scores and helps in alignment of sampling trajectory with training trajectory, thus reducing exposure bias.", "questions": "1. In Figure 5, Figure 9b and Figure 10b, what is the reason for smaller values of $\\| \\epsilon \\|_2$ of ADM-ES compared to its values at training time towards end of sampling (time steps 18-21)?\n2. Could the authors include the number of sampling steps in qualitative results in Figures 11-15?\n3. Could the authors indicate the amount of compute (number of GPUs as well as time) needed to find optimal hyperparameters?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written. The paper provides a clear explanation about the problem of exposure bias and its causes by deriving expressions for sampling probability distribution that shows increased variance. It also provides clear intuition for different choices while designing the proposed method. For instance, the intuition for correcting $\\epsilon_\\theta$ to reduce exposure bias instead of reducing variance error has been explained clearly. \n2. The method is relatively simple and easy to implement as it introduces a scaling factor in the sampling process of diffusion models and does not require re-training of diffusion models. Previous method by Ning et al. (2023) mitigates this issue by perturbing inputs during training which requires retraining diffusion models. \n3. The proposed method improves FID scores significantly for different families of diffusion models like DDPM, DDIM, ADM, EDM and LDM. \n4. The primary contribution of this work which is empirical demonstration of the fact that correction of exposure bias can be done without retraining diffusion models is valuable.", "weaknesses": "1. The proposed method of Epsilon scaling is not tuning-free as it is sensitive to the choice of hyper parameter of the schedule $\\lambda_t = kt+ b$ and thus requires extensive hyper parameter tuning. The choice of optimal hyperparameters varies for each dataset as well as the number of sampling steps T. As per Table 10, Table 11 and Table 12, values of k and b have been tuned up to 4th or even 5th decimal place. For instance, there are values like k=0.00022 , b=1.00291 in Table 10. This degree of hyperparameter tuning seems a bit extreme.  The paper is also missing relevant sensitivity analysis for hyperparameters k and b, and it would be useful to include it (both over large strides and small strides). \n2. The process of finding optimal hyperparameters for Epsilon scaling seems to be slow and can be computationally intensive.  Usually, generating 10K sampling to calculate FID takes several minutes for small sampling steps and several hours for large sampling steps. Tuning hyper-parameters for each dataset and each choice of sampling length T, can thus require lot of GPU compute and time. While the method itself is simple, tuning this method for optimal performance can be tricky.\n3. In Table 3, for CIFAR-10, values of EDM for VP and VE should be reported for NFE=35 which gets FID = 1.97 (VP) and 1.98 (VE) on CIFAR-10, respectively. Underreporting FID values of EDM is a bit misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)