Anti-Exposure Bias in Diffusion Models
Authors: Junyu Zhang
Daochang Liu
Eunbyung Park
Shichao Zhang
Chang Xu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Diffusion Models
Exposure Bias
Prompt Learning
Sampling Trajectory
Primary area: generative models

Abstract: Diffusion models (DMs) have achieved record-breaking performance in image generation tasks.
Nevertheless, in practice, the training-sampling discrepancy, caused by score estimation error and discretization error, limits the modeling ability of DMs, a phenomenon known as exposure bias.
To alleviate such exposure bias and further improve the generative performance, we put forward a prompt learning framework built upon a lightweight prompt prediction model.
Concretely, our model learns an anti-bias prompt for the generated sample at each sampling step, aiming to compensate for the exposure bias that arises.
Following this design philosophy, our framework rectifies the sampling trajectory to match the training trajectory, thereby reducing the divergence between the target data distribution and the modeling distribution.
To train the prompt prediction model, we simulate exposure bias by constructing training data and introduce a time-dependent weighting function for optimization.
Empirical results on various DMs demonstrate the superiority of our prompt learning framework across three benchmark datasets.
Importantly, the optimized prompt prediction model effectively improves image quality with only a 5\% increase in sampling overhead, which remains negligible.

Published as a conference paper at ICLR 2025
ANTI-EXPOSURE BIAS IN DIFFUSION MODELS
Junyu Zhang
Daochang Liu
School of Computer Science and Engineering Central South University
School of Physics, Mathematics and Computing University of Western Australia
Eunbyung Park
Department of Artificial Intelligence, Yonsei University
Shichao Zhang
Chang Xu"
School of Computer Science and Engineering Guangxi Normal University
School of Computer Science University of Sydney
ABSTRACT
Diffusion models (DMs) have achieved record-breaking performance in image generation tasks. Nevertheless, in practice, the training-sampling discrepancy, caused by score estimation error and discretization error, limits the modeling abil- ity of DMs, phenomenon known exposure bias. To alleviate such exposure bias and further improve the generative performance, we put forward a prompt learning framework built upon a lightweight prompt prediction model. Concretely, our model predicts an anti-bias prompt for the generated sample at each sampling step, aiming compensate for the exposure bias that arises. Following this design philosophy, our framework rectifies the sampling trajectory to match the training trajectory, thereby reducing the divergence between the target data distribution and the modeling distribution. To train the prompt prediction model, we simulate ex- posure bias by constructing training data and introduce time-dependent weight- ing function for optimization. Empirical results on various DMs demonstrate the superiority of our prompt learning framework across three benchmark datasets. Importantly, the optimized prompt prediction model effectively improves image quality with only a 5% increase in sampling overhead, which remains negligible. Our code is available at: Exposuree Bias.
INTRODUCTION
Diffusion models (DMs) (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) rep- resent a novel generative paradigm that has become de facto standard for image generation, and also showcasing impressive results in many downstream tasks (Luo al., 2023; Shue et al., 2023; Liu et al., 2023a; Mokady et al., 2023). In particular, the seminal work (Song et al., 2021b) unifies the design philosophy of DMs through continuous diffusion using stochastic differential equations (SDEs), boosting them for achieving start-of-the-art image quality (Kim et al., 2023a; Peebles & Xie, 2023) and improved mode coverage (Kingma al., 2021; Song et al., 2021a; Lu et al., 2022a; Kim et al., 2022). More recently, stable diffusion (Rombach et al., 2022) has bridged the gap in both text-to-image (Nichol et al., 2022; Ramesh et al., 2022: Saharia et al., 2022) and text-to-video generation (Blattmann et al., 2023; Khachatryan et al., 2023), further enhancing the modeling ca- pability for high fidelity, controllable content synthesis (Gao et al., 2023; Ruiz et al., 2023) and demonstrating great potential for practical applications (Xu et al., 2024: Sauer et al., 2023).
The core idea of DMs is to establish a diffusion path between the target data distribution and a prior distribution and simulate this path in the opposite direction for image generation, dubbed forward diffusion and reverse sampling, respectively SSohl-Dickstein et al., 2015; Song & Ermon, 2020). In the diffusion process, forward SDE is employed to formulate the diffusion path via perturbing the data distribution with a well-designed multilevel noise schedule (Song et al., 202 lb; Karras et al.,
"Correponning Author

---

Published as a conference paper at ICLR 2025
Training Trajectory
Sampling Trajectory
Learned Prompt
Exposure Bias
Anti-Bias Prompting Sumpling Trajectory (Ours)
idwoid
Blas
Xr
Xy-1
X
X3
X2 X1 X
Figure 1: Anti-Exposure Bias Prompt Learning. The exposure bias arises from the training- sampling discrepancy of model inputs. As clearly illustrated in this figure, the sampling trajectory gradually deviates from the training trajectory due to accumulated score estimation and discretiza- tion errors at each time step. leading to an increase in bias. To alleviate this bias and enhance modeling performance, our prompt learning framework employs lightweight parameterized model to generate an anti-bias prompt from the current time-step sample, compensating for exposure bias in the model input the next time step. addition to delivering excellent performance, the resulting extra sampling overhead is less than 5%, which can be considered negligible (Best viewed in color).
2022). Concretely, different noise scales represent distinct time steps in the diffusion path, with the transition between adjacent time steps characterized by Gaussian transition kernel. The reverse sampling can be achieved by traversing the diffusion path with the opposite direction (Kim & Ye, 2022: Lu et al., 2022b: Zhang& & Chen, 2022). Crucially, the reverse process satisfies a reverse-time SDE or a probability flow (PF) ordinary differential equation (ODE) (Song et al., 2021b). Both can be derived from the forward SDE by considering the score of the marginal probability densities as a function of time (Anderson, 1982; Luo, 2022). We can, therefore, approximate the reverse- time S/ODE by training time-dependent deep neural network to estimate the scores (Hyvärinen & Dayan, 2005; Song et al., 2020b; Song & Ermon, 2019), and generate new images using numerical S/ODE solvers (Song et al., 2021b; Lu et al., 2022b; Zhang & Chen, 2022: Zhang al., 2023).
However, in practice, a wraining-sampling discrepancy exists at each time step concerning the input to the time-dependent neural network model, resulting in an exposure bias problem (Ning et al., 2023b; Li et al., 2023a; Ning et al., 2023a) and, consequently, a degradation in image quality (Kim et al., 2023a). As illustrated in Figure 1, this issue arises because, during training, the model inputs for DMs are derived from ground truth samples, while during sampling, the inputs are predictions from the previous time step. Practically, the predictions cannot completely match the ground truth samples due to two fundamental errors: the score estimation error (Bao et al., 2022a) and the dis- cretization error (Zhang & Chen, 2022). The score estimation error primarily caused by the score conflict (Hang al., 2023), data sparsity (Kim et al., 2022) and model capacity (Karras al., 2022), as well as an imperfect diffussin schedule (Dhariwal & Nichol, 2021). Regarding the latter, since integration in high-dimensional spaces is intractable, we can only approximate reverse-time S/ODEs using numerical solvers to the best of our ability, which inevitably results in discretization error (Wang et al., 2021; Bao al., 2022c; Lu et al., 2022b). Due these two types of errors, ex- posure bias inherently arises during the sampling process in DMs. Furthermore, this bias becomes increasingly pronounced along the sampling trajectory, as each time step accumulates newly result- ing score estimation and discretization errors (De Bortoli et al., 2021; Xiao et al., 2021). As a result, exposure bias has a significant impact on the generative performance of DMs.
To illuminate exposure bias, we first thoroughly examine the training-samplinn discrepancy problem from an analytical perspective. Theoretically, the denoising distribution between adjacent time steps follows a naive Gaussian distribution (Ho et al., 2020; Sohl-Dickstein et al., 2015). We can, there- fore, approximate it by a Gaussian transition kernel. However, when considering the phenomenon of exposure bias, there is gap at each time step between the true Gaussian distribution and the transition kernel modeled by a pre-trained DM (Luo et al., 2024). Moreover, the magnitude this gap increases when fewer sampling time steps are used (Kim et al., 2023a), resulting in reduction in image quality, as the Gaussian assumption holds only in the infinitesimal limit of small denoising steps (Xiao et al., 2021). While the gap in each denoising step can be quantified using the Kullback-
2

---

Published as a conference paper at ICLR 2025
Leibler (KL) divergence between the true Gaussian kemel and the modeling transition kernel the current time step, accessing these latent distributions is not feasible (Kingma & Gao, 2024).
To remedy this, we propose novel exposure bias prompt learning framework that uses param- eterized prompt prediction model to rectify biases in the generated samples at each time step. To effectively optimize the prompt prediction model, construct training data that simulates exposure bias and introduce a time-dependent weighting function for stable training. During the sampling process, the optimized prompt prediction model predicts an exposure bias prompt based on the gen- erated sample at the current time step. This prompt is then used to correct the bias in the model input for the next step. The combination of the generated sample and its anti-exposure bias prompt creates an improved sample input for the subsequent time step. In this manner, we alleviate the exposure bias caused by the score estimation and discretization errors during sampling, effectively rectifying sampling trajectories through iterative execution of this process. Importantly, the prompt prediction model is lightweight backbone that requires only a 5% increase in sampling time, which can be considered negligible. Furthermore, our framework provides significant flexibility, enabling enhancements to guidance sampling (Dhariwal & Nichol, 2021), alleviating exposure bias in latent diffusion (Vahdat al., 2021; Rombach et al., 2022), and improving fast samplers (Song et al., 2020a; Bao et al., 2022b; Zhang & Chen, 2022; Lu et al., 2022b). Notably, compared to merely in- creasing DM parameters to alleviate exposure bias, our framework can serve as a plug-in to enhance various DMs parameterized by different model sizes and architectures. The former requires training the model from scratch, incurring significant computational costs and human effort, while we only necessitate training a lightweight backbone, demonstrating both flexibility and efficiency.
In a nutshell, our contributions can be summarized as follows: 1) We analyze the phenomenon of exposure bias in DMs caused by score estimation and discretization errors; 2) To alleviate exposure bias and enhance generative performance, we propose a prompt learning framework that employs a lightweight parameterized model to predict an anti-bias prompt for rectifying the next model input; 3) A novel training strategy is proposed to simulate exposure bias and ensure stable training; 4) Ex- tensive experiments demonstrate the effectiveness of our prompt learning framework across various datasets and different DMs, with only a negligible increase in sampling overhead.
BACKGROUD
Overview DMs (Song& & Ermon, 2019; Ho et al., 2020) are a new class of generative models that synthesize images by gradually denoising random points sampled from prior distribution. Specifi- cally, for given D-dimensional image xo, we assume it satisfies a distribution Zig ~ p(zo). Thus, the diffusion path leading to prior distribution can be constructed via the following forward SDE:
dx = F,rdl + Gidw,
where F. E DxD denotes the linear drift coefficient, G. E RDD denotes the diffusion coeffi cient, w is a standard Wiener process and t ~ U[0, 1]. Under some mild assumptions (Song et al., 2021b), the forward SDE in Eq. (1) is associated with reverse-time diffusion process:
dx F5r - G.G. v (x)) dt + G.døo
(2)
where @ denotes a standard Wiener process in the reverse-time direction, and V log (c represents the gradient of the log probability density with respect to the perturbed data at time step a.k.a. score (Hyvärinen& & Dayan, 2005; Vincent, 201 I). In theory, with a known prior distribution π, such as the normal distribution, one can generate new images via solving Eq. (2) using initial samples .T ~ π (Anderson, 1982).
Training In practice, log (xr) is inaccessible due to the high dimensionality of data, which leads to the analytical intractability of the probability density function (Hyvärinen & Dayan, 2005). To remedy this, prior works (Song et al., 2020b; Vincent, 2011; Song & Ermon, 2019) employ a time- dependent neural network so (zz, t) to approximate the score:
JsM (0: (-)) 1
Eng.,, (w(() | log por por so (a (xx,,)] e)]1 dt.
Here, V log Poe (1y (xo) has closed form expression as Poe (2, is a simple Gaussian distribution obtained from a given SDE (Song al., 2021b), and w(t) denotes time-dependent weighting
3

---

Published as a conference paper at ICLR 2025
function used for stable training (Kingma et al., 2021; Kim et al., 2022). When implementing advanced score matching techniques, Eq. (3) can be optimized using empirical samples via Monte Carlo methods (Hyvärinen & Dayan, 2005; Song & Ermon, 2019; 2020).
DISCUSSION
3.1
EXPOSURE BIAS PHENOMENON
Score Estimation Error Once 8e (o( 1) as llogpe (()) is matched for almost all x E RRD and ~ U10, II. one enables to generate images by solving Eq. (2) with V Pr)))( replaced by so(z,, t):
dx F1x - G,G, so t)] dt + G.du.
(4)
However, this process will results in score estimation error because of the discrepancy between v logg Pt (z: and sy t). This discrepancy is primarily attributed factors such data bias, model robustness, and training techniques (Kim et al., 2022; Hoogeboom et al., 2023), which cannot be easily resolved by merely increasing model parameters due to the intrinsic limitations DMs.
Discretization Error In practice, directly solving the integral in Eq. (4) is intractable. Instead, it approximated by discretizing it into T steps with step f + 1 to is governed by Gaussian kernel q( xx+11 1). For simplicity, we next investigate the discretization error via using the PF ODE, where 2y can be obtained via the following formulation:
intervals, where the transition from time
(1, 1) [ ((1, 7)
[- -
1 G. G. G.
S0..G, so( dr,
dr,
Je41
where 0.((L,++1 8t (t, 1), and (t + 1, + 1) = I represents the transition function from time t to time which can be derived from F, (Zhang & Chen, 2022). In this manner, new images 26 can be generated by iteratively solving Eq. (5) from the initial time step T to the final time step. Nevertheless, it is also intractable directly solve the integral part in Eq. (5) because of its ultra-high dimensional nature. In practice, one can utilize numerical solver (Lu et al., 2022b; Karras et al., 2022; Zhang & Chen, 2022: Li al., 2023b) approximate each integral part:
At GG+ GT+1G1110 (114+111+1)))) +1,,
. (1, L 100+11
where At is the integration interval between time step + and t. For simplicity, we demonstrate only the first-order Euler sampler for solving the integral part. Obviously, using Eq. (6) to solve each integral instead of Eq. (5) will cause the discretization error. This happens because linear solutions provide only rough approximation of the integral, particularly over large integration intervals.
Exposure Bias Phenomenon When accounting for score estimation and discretization errors, ex- posure bias occurs at each time step along the sampling trajectory, shown Figure 1. Formally, the model inputs during training are derived from the ground truth images, while the inputs during sampling are the model prediction outputs from previous steps. Due to these two errors, the model predictions cannot exactly match the ground truth value, leading to the exposure bias phenomenon. Based on the analysis, the modeling Gaussian kernel can be formulated as po xt+1). In this context, the exposure bias at each time step is actually the discrepancy between the ground truth output q(2, 2(41) and its predicted output pa (&g | It+1). Although we can express it as a KL divergence [g 2x+1) || pa (Ør Tz+1), directly minimizing this KL divergence to alleviate exposure bias is intractable, as we do not have access to the true Gaussian kernel.
3.2 DESIGN PRINCIPLE
To elucidate exposure bias, we conduct in-depth investigation into the gap between the true Gaus- sian kernel g( xx+1) and its modeling counterpart Pe (#, | xx+1) from an analytical perspective. Coneretely, in each sampling iteration, we can formulate the distinction between the ground truth sample and its biased sample as follows:
l' 1+ V(,,T) | G. dr G 1041111141,,1 1).
Al G
$ ( 2x) =
(7)
integral
linear term
4

---

Published as a conference paper at ICLR 2025
Here, ( 2z) represents the exposure bias in the sample &t compared to 24, stemming from dis- cretization and score estimation errors. Though we do not have access to integral term in Eq. (7), can employ & to present the ground truth sample as It L, + Φ(8,, In this manner, we can present this formulation as a transition kemel p(æt ..). Formally, we have the expres- sion ((xx)) = J p(az &z (a) (f).. However, dimensional property. In theory, the score of the integral term in Eq. (7) is based on T++1, thereby also forming the basis of (e)) Moreover, & is actually derived from x2+1 via Eq. (6). It is reasonable to reformulate the transition kernel p( (e)) as p(Iz [ Zt+1). We can deconstruct the true Gaussian kernel g(2;
cannot directly obtain p(a+) due to its high-
1t+1) into the combination of the modeling kernel and an extra transition kernel p(<;, (2, Fe+1) as follows:
/ p(zz (t,) 2(+1)dd;
(xx | x(+1)
In Eq. (8), po xx+1) is the modeling Gaussian kernel simulated by a pre-trained DM, which is fixed during sampling. Thus, we can mitigate exposure bias at each time step by using the newly introduced anti-bias kernel p(rr 22 ) contributing to smaller training-sampling discrepancy.
3.3 CONVERGENCE INVESTIGATION
From a theoretical perspective, our anti-bias transition kernel p(æ, it, (x+11 can help the sampling bound better align with the training KL divergence (KL(O(z)) || po())) To illustrate this advan- tage, we first review the diffusion training objective function, which serves as the theoretical bound for the optimization tasks. Specifically, DMs aim minimize DKL (p(xx) || (ao)) via optimizing the score matching loss with the weighting function g(-)² (Song et al., 2021 Lu et al., 2022a):
DrI.(p(zo) po (mo)) < (RI(P(ET) || x) + Jsm (0; (0:)()))
Here, g(-)² is the diffusion coefficient in forward SDE, and is a prior distribution. In this context, we can achieve an optimized KL divergence via minimizing JSM (0:g(-))). as (p((() lI T) constant. However, during sampling, this bound will be enlarged due to exposure bias phe- nomenon. Concretely, the score matching in Eq. (3) essentially optimizes the discrepancy between v log Por (ze (z0) and sg(r,, t). During training, the input for (2 t) consists of the training data perturbed by noise scale at time step i, whereas during sampling, the input for so (ā is the bias sample ăt. In this context, (o f) cannot exactly match V log Poe (2o) because of the discrep- ancy between d. and Tt, thus amplifying the upper bound of DKL (p(zo) framework can help so !) better approximate 8p (I,, f) since the anti-bias kernel p(z; Lt,, Te+1) can enable ft match Ty, resulting in a sampling bound closer to the the training bound, By ad- hering to this design philosophy, our framework enables the simulation of a more realistic transport path between the target data and prior distributions, thereby enhancing generative performance.
(o()) Ideally, our
PROMPT LEARNING FRAMEWORK
To alleviate exposure bias, we propose novel prompt learning framework that parameterizes lightweight model (-) to simulate the anti-bias kernel p(æ; (x+1) in Eq. (8). Concretely, we simulate the bias and introduce time-dependent weighting function to train the model. Once the model is optimized, prompt can be learned based on &,, thereby compensating for the bias ((0,, xx) in Eq. (7) for the next model input. By adopting this approach, we can rectify the sampling trajectory by mitigating exposure bias at each time step, resulting in enhanced image quality.
4.1 FRAMEWORK TRAINING
As previously mentioned, based on the modeling Gaussian kernel (o | 1x44)). we aim to construct
an anti-bias transition kernel p(æ; ft; 1x++1) to match the true kernel g(z, [244)). To accomplish this goal, we employ a parameterized model (-(-) to simulate p(,; fe, Ze) Below, we provide a detailed introduction on how to optimize (.).), with the conceptual framework shown in Figure 2.
Overview Based on the previous analysis, our prompt prediction model (-))) is designed to predict an anti-biar prompt that approximates the exposure bias defined in Eq. (7):
Vig (fx)) + ( w.)),
(9)
5

---

Published as a conference paper at ICLR 2025
Forward SDE
X
on + V (( (0)=
Z+41 20 + V 1) (0)
Xr+1
Data Construction
(G R2
Res1
At
2
G G.a.Ga
Diffision Model
Metric Funtion
U/ (tt)
gradients
Training Optimization
Prompts
Et
Learning Model
Figure 2: Prompt Learning Framework Training. For convenience, we employ the VE SDE and a first-order sampler to demonstrate the training process. 1) Data construction: obtain Ha and Tt-1 via a forward SDE with randomly selected noises og and c++1 and denoise 2141 é with a deterministic sampler (i.e., DDIM); 2) Training optimization: minimize the difference between the prompt vs (.x) and the exposure bias ( x4) via the training function Eq. (12), where ((;, (e) represents the discrepancy between the ground truth image and the modeling sample ft.
where t is the time index of z,. Since we do not adjust the parameters 0 in the pre-trained DM, the term ( T2+1) remains fixed and can be used to generate ât. For a given prompt prediction model (.(.,, we have us (ā,) (ft,; 24). where
(0, T). Suppose we have free-form deep neural network to represent our prompt prediction model, we can train it with our prediction loss:
((), E (fx), (()),
(10)
and d(:.)) :) is a metric function that satisfies yx,y y d(x, 3) 2 0 and y(x,y)= and only if x=xxy The metric function is designed minimize the difference between Us(3+) and (8), Inspired by this, there are several classic loss functions that satisfy our requirements, such as the squared l norm d(x,y) y) ||x - y1l2 and the 1 norm d(x.y) = ||x - yll Additionally, we also consider the contrastive loss (He et al., 2020; 2022) to maximize the similarity between the prompt and the bias. This loss has been successfully used in recent works on training or fine-tuning DMs, thanks to its theoretical guarantees (Daras et al., 2024; Zhang et al., 2024), with details provided in the Appendix.
Bias Simulation In practice, accessing is intractable because we cannot obtain the ground truth sampling trajectory. We only have the bias sample á. due to the accumulation of exposure bias. Therefore, to address this issue, we seek help from the deterministic solver (Song et al., 2020a; 2021b) to simulate the bias ((;,,,,) More concretely, we first utilize a forward SDE to perturb the target image zp. allowing to obtain the unbiased samples r++1 and I₁. For simplicity, suriance-cxploding (VE) SDE (Song et al., 2021b) to illustrate this process:
the
Tz+1 = xo + v o (L -) (0))=++
(1I)
[+++] 1) is the noise at time 1+11 and 1 is sampled from m, detailed in (Song et al., 2021b). Similarly, we can obtain I; by replacing (f++1) with (t) in Eq. (11). Subsequently, we employ the deterministic solver to denoise x++1 for just one time step. the detailed process is presented by Eq. (6). In this way, we have successfully simulated the exposure bias xx) = x, &, that arises at time step t. This is because the exposure bias problem is primarily caused by score estimation and discretization errors, both of which are simulated within one time step in Eq. (6). Although the deterministic solver may not fully denoise 7 to the final image, our goal to model this deviation to rectify the sampling trajectory. Thus, we simulate exposure bias in a reasonable manner.
where (1++
Optimization To optimize the prompt prediction model ug (Gt), we utilize Eq. (10) to train the pa- rameters é. However, in practice, the discrepancy between Vs and ((,,, 22)) can be substantial across different time steps, resulting in irregular fluctuations in the training loss. This is due to the fact that noisy samples at different noise scales contain entirely different structural information. For instance, samples with low noise levels may provide preferable detailed information (Lou & Ermon, 2023), while samples with high noise levels may only capture coarser shapes.
Motivated by this observation, we propose a time-dependent weighting schedule designed to en- hance training stability. To be specific, we utilize the signal-to-noise ratio (SNR) (Kingma et al.,

---

Published as a conference paper at ICLR 2025
Algorithm 1: Anti-Bias Sampling
Data: pre-trained DM sg, optimized prompt prediction model Vø-, default sampler S, pre-defined noise schedule I. = (o .. tt total sampling steps T
Result: New Images Zto - bius
sample batch of 2T from prior distribution .
Stemp 2T;
for 1 < IT to la do
4
tt, = (ig, ot, Xtemp);
anti bias Ug. (Re,) + De,5 This anti-hiss rectification only difference compared original sumpling schedale.
anti bius,
Ptemp Zt,
anti bius
2 to
Ttempi
2021: Choi et al., 2022) to formulate our weighting function, which is based on the coefficients of the forward SDE. The forward diffusion kernel can be represented g(æ;, mn) (ogaa, of and therefore, our weighting function can be expressed as SNR(1) = of 1//o/- In practice, for a given forward SDE, both a and o3 can be derived from the diffusion kernel g(2;; xo). For example, in the variance preserving (VP) SDE (Song et al., 2021 Ho al., 2020), 0; / of. while the variance exploding (VE) SDE (Song et al., 2021b; Song & Ermon, 2019), Or 1. With the newly proposed weighting function, our training loss can be expressed as:
C SNR( = E (SNR(t)d(v,(t)) (ff,,
(12)
During the training process, we employ stochastic gradient descent on the model parameters & via minimizing C( SNR(t)), and updating o with exponential moving average (EMA). We perform the following update with EMA after each training iteration: o <- stopgrad
+ (1 - 1))).
Here, μ is decay rate with 0 S μ < 1 (Song et al., 2023), with details provided in Appendix. When implementing these training techniques,
effectively optimize the prompt prediction model.
4.2 ANTI-BIAS SAMPLING
Once the prompt prediction model ug-() (-) is optimized, it can be used to improve sampling perfor- mance by reducing the exposure bias the input of the pre-trained DMs for future steps, based on the bias predicted for the output of the model at the current step. To be specific, we utilize Ua- (-) to predict an anti-bias prompt using the input dt, and the anti-biar image can thus be expressed as:
anti- bias = -ig-(ū;) + 24. zt
(13)
In the next time step, xt anati-bias serves as the input of pre-trained DMs, allowing us to obtain è via Eq. (6). Subsequently, our model ug- (#t-1) predicts the prompt using the input 1t-1. By iteratively implementing Eq. (6) and Eq. (13), æ anti bias can be generated with a high image quality, detailed shown in Figure 3 and Algorithm 1. Therefore, the prompting sampling trajectory can more closely match the training trajectory. Compared to the original diffusion sampling schedule, we retain the main procedure and only compensate prompt for the output of pre-trained DM each time step.
On the other hand, our method can also improve the guidance sampling mechanism (Dhariwal & Nichol, 2021), which is a milestone technique guide sample with a pre-trained classifier p(c | Lt. !). where e represents a class label. The classifier guidance provides auxiliary information on the sampling trajectory by evaluating whether the sample is correctly classified according to the class label c. This is equivalent to sampling from the joint distribution p(z,, c) because:
V Vlogp(I.. c) V log p(x,) + V p(c xx)) v sa t) + V log (..).
However, due to the presence of exposure bias, the biased image åt may lead inaccuracies in classification. This, in turn, results in a biased gradient V log | (z). Based on the previous analysis, our method enables further improvement in guided sampling via alleviating exposure bias:
Vlogp(zr, c) = s0 + vi- (.+), 1) + V logr(e) 4 + uo- (f:)).
(14)
is worth noting that visual prompting method is indeed beneficial for image classification task (Jia et al., 2022; Bahng et al., 2022). Therefore, using prompt prediction model to guide the score
7

---

Published as a conference paper at ICLR 2025
Table Performance on CIFAR-10. Here,
Connection Training-free Fast Samplers. Results obtained from experiments conducted on CIFAR-10. Here, we employ sev- eral classic samplers as our baselines, such as DDIM (Song al., 2020a), Analytie-DPMy (Bao et al., 2022b), DEIS (Zhang & Chen, 2022) and DPM-Solver (VP) (Lu al., 2022b). For fair- ness, we use the same DMs as those used by the samplers in their papers. We confirm that our framework reduces the exposure bias caused by fast samplers with large step sizes, and the re- sults tested across various NFEs using FID..
Table 2:
we select NSCNv2 (Song & Ermon, 2020),
DDPM (Ho et al., 2020), SDE (VE) (Song al., 2021b), SDE (deep. VE) (Song et al., 2021b),
ADM (Dhariwal 2023b) and ADM-IP (Ning et al., 2023b), well as EDM (Karras et al., 2022) to serve as baselines. When implementing our framework to them, the optimized prompt prediction models fa- cilitate significant improvements in image quality, as evidenced by lower FID scores and better IS re- sults. Notably, posed by the baselines, with the only difference being bias rectification, shown in Algorithm 1.
Nichol, 2021; Ning al.,
the original samplers pro-
NFEs
10
20
50
DDIM
13.36
6.84
Models
FID4
IS
NFEs|
DDIM+ours
12.94
6,71
4.59
NSCNv2
10.87
8.40
1000
Analytie-DPM
14.4
6.87
4.15
NSCNy2+ours
9.56
8.65
1000
Analytic-DPM+oun
13.98
6.76
4.10
DDPM
3.17
9.46
1000
DDPM+ourS
2.99
10.01
1000
DEIS (VP)
4.17
2.86
2.57
SDE (VE)
2.55
9.83
TO00
DEIS (VP)+ours
4.08
2.80
2.51
SDE (VE)+ours
2.41
9.91
1000
DEIS (VE)
20.89
16.59
16.31
SIE (deep, VEJ
2.20
9.89
TOO0
DEIS (VE)+ours
19.76
16.21
16.08
SDE (deep, VE)+ours
2.10
9.99
1000
NFEs
12
24
48
ADM
3.56
100
DPM-Solver-2
5.28
3.02
2.69
ADM+ourS
3.28
100
ADM-IP
3.12
100
OPM-Solver-2+ours
5.22
2.95
2.65
DPM-Solver-3
2.75
2.65
ADM-IP+ourS
3.06
100
EDM
2.04
9.84
35
DPM-Solver-3+our
5.93
2.69
2.61
EDM+ours
1.91
9.94
35
direction is reasonable approach, as can significantly enhance the guidance gradient. Moreover, framework has great potential for controllable generation (Ruiz et al., 2023; Nichol et al., 2022; Ramesh et al., 2022) via replacing e to text prompt, we leave this exploration for future work.
4.3 PROMPTING LATENT DIFFUSION
Recently, latent diffusion models (LDM) (Rombach et al., 2022; Peebles & Xie, 2023) have sig- nificantly enhanced the performance in image generation task. They employ an encoder to map training images into latent representations from the latent 2 with å = (x)). Given their promising future (He et al., 2023; Poole et al., 2022), is meaningful to further enhance their generative performance using our framework.
Exx), and the decoder D to reconstruct the image
Though their sampling trajectories traverse latent space, the exposure bias phenomenon still occurs due the discrepancy between 2; and ε, Here, 2, represents the ground truth latent and é, the latent simulated by the pre-trained LDM. To remedy this, we put forward variant of the prompt prediction model. Compared to diffusion in data space, the only difference that we predict the prompt in the latent space. Concretely, the prompt sampling in latent space can be written as:
anti - bias 2g + 2.. 2t
where ugg- (.) can be optimized via minimizing the metric function d(us(&)) (2x), ( 2:)) using a gra- dient descent algorithm. In this manner, our prompt learning framework effectively reduces the bias between & and 2,, thereby contributing to improved sampling trajectories in latent diffusion.
5 EXPERIMENTS
To evaluate the effectiveness our prompt learning framework in reducing exposure bias, we conduct experiments three benchmark datasets: CIFAR-10 (Krizhevsky al., 2009), CelebA
8

---

Published as a conference paper at ICLR 2025
Table Performance on ImageNet 256 x 256.
Table 4: Boosting Latent Diffusion. We use LDM (Rombach et al., 2022) as our baseline and test its performance on ImageNet resolution of 256 x 256. Both LDM-4 and LDM-8 show signif-
We select ADM (Dhariwal & Nichol, 2021) and
ADM-U (Dhariwal Nichol, 2021) as base- lines. After applying the optimized prompt model to the default sampler in ADM, both
icant improvements after applying the optimized
achieve improvements in image quality.
model to rectify bias in the default LDM sampler.
Models
FID4
1st
NFEs
Models
FID4
1ST
NFEs,
ADM
10.94
100.98
250
LDM-4
10.56
103.49
250
ADM +ours
10.37
112.00
250
LDM-4+ours
10.02
111.69
250
ADM-U
7.49
127.49
250
LDM-8
15.51
779.03"
200
ADM-U+ours 7.29
134.95
250
LDM-8+ours
14.03
91.02
200
64 x 64 (Liu et al., 2015), and ImageNet 256 x 256, utilizing various pre-trained DMs. Concretely, for CIFAR-10 10, we select NSCNv2 (Song & Ermon, 2020), DDPM (Ho et al., 2020), SDE (VE) and SDE (deep, VE) (Song et al., 202 (b), ADM (Ning et al., 2023b) and ADM-IP (Ning et al., 2023b), as well as EDM (Karras et al., 2022) to serve as the baseline models. For CelebA, we utilize ADM (Dhariwal & Nichol, 2021; Ning et al., 2023b) and ADM-IP (Ning al., 2023b) bascline models. In contrast, for ImageNet, we select ADM (Dhariwal & Nichol, 2021) and ADM-U (Dhari- wal & Nichol, 2021) serve as the baseline models. We then employ prompt models customized for different DMs to mitigate exposure bias at each step, aiming to enhance image quality. To quan- titatively evaluate the performance of our framework, Inception Distance (FID) (Heusel et al., 2017), Inception Score (IS) (Salimans et al.. 2016) and Spatial Fréchet Inception Distance (sFID) (Nash al., 2021), well neural function evaluations (NFEs) (Vahdat et al., 2021), to verify them on 50K newly generated samples.
utilize standard metrics, including Fréchet
As mentioned previously, our framework demonstrates good theoretical flexibility, such as enhanc- ing guidance mechanism, alleviating exposure bias in latent space, and improving image quality for training-free fast samplers. To verify this, we choose ADM-G (Dhariwal & Nichol, 2021) to assess the effectiveness of our framework on classifier guidance methods. Moreover, we also evaluate the performance of improving the latent diffusion model, thus choosing LDM-4 and LDM-8 (Rombach et al., 2022) as the baseline models. For evaluating the effectiveness on rectifying high-order solvers, we employ our framework to improve several classic training-free fast samplers.
To design the architecture of prompt prediction model, we employ a lightweight U-shaped network, with a backbone similar to that of EDM (Karras et al., 2022). We design the model architecture according to different data resolutions. Specifically, we set the model channels for resolutions 32, 64, and 256 as 32, 32, and 64, respectively. The corresponding model parameters are 3.2M, 3.2M, and 12.7M. We maintain these settings on all experiments, more training details refer to the Appendix.
5.1 PERFORMANCE EVALUATION
To evaluate performance in the data space, we conduct multi-group periments various datasets. In Table 1, we first present the evaluation of two well-known DMs in discrete diffusion, as well as the classic SDE DM, all of which demonstrate significant improve- ments. Importantly, we also demonstrate further improvements on the pioneering work of ADM- IP (Ning et al., 2023b), the ADM-IP, which first investigates the exposure bias problem. For per- formance evaluation on CelebA and ImageNet, we utilize ADM (Dhariwal & Nichol, 2021) and ADM-IP (Ning et al., 2023b) for verification, with results depicted in Table 3 and Table 8, respec- tively. On the other hand, our framework also possesses the capacity to reduce exposure bias in latent diffusion, demonstrating great flexibility. To be specific, our customized models for latent diffusion enable to improve LDM-4 and LDM-8 in terms of FID and IS, as detailed in Table 3. Moreover, we also test the effectiveness of our framework in enhancing guidance sampling, as shown in Table 4. This is because lower bias samples enable better classification accuracy, thus providing more effi- cient classifier gradients. In this context, our framework naturally improves generative performance when combined with guidance methods. As mentioned earlier, the exposure bias is notably larger in cases with fewer NFEs because the discretization error increases with larger sampling step size.
Quantitative Comparison
group ex-

---

Published as a conference paper at ICLR 2025
Table 5: Enhancing Guidance Mechanism. We evaluate performance in a guidance set- ting using unconditional ADM-G (Dhariwal & Nichol, 2021) with ImageNet at resolu- tion of 256 256. Here, "Scale" indicates the degree of guidance provided by the clas- sifier. After applying our prompt prediction model, they enable to achieve improvements in image quality, as reflected by better FID and higher IS scores. All results are based 50k samples generated using 250 NFEs.
Table 6:
Ablation Studies on Different Metric Functions. Here, we employ EDM (Karras et al., 2022) to test three different metric functions, includ- ing contrastive loss, L1 norm, and L2 norm, to train our prompt prediction model on CIFAR-10. For fairesss, we set the batch size to 1024 and the model channels to 32 in all experiments. Obviously, the contrastive loss presents enhanced performance due to its theoretical guarantees in the training bound.
Metric Functions
FID4
NFEs
Iterations
EDM (baseline)
2.04
35
Models
Scale
FID
ISt
L1 norm
1.96
35
100k
ADM-G
1.0
33.03
32.92
L2 norm
1.94
35
100k
ADM-G+ours
1.0
31.64
36.00
Contrastive loss
1.96
35
80k
ADM-G
10.0
12.00
95,41
Contrastive loss
1.93
35
100k
ADM-G+ours
10.0
10.64
106.13
Contrastive loss
1.91
35
150k
When implementing framework with fast samplers such as DDIM (Song al., 2020a), Analytic- DPM (Bao et al., 2022b) and DEIS (VP) (Zhang & Chen, 2022), as well as DPM-Solver (Lu et al., 2022b), the resulting sampling trajectory yields good results, detailed in Table 2. Moreover, Table 10 presents the efficiency of our model in comparison with the settings by using more NFEs.
Qualitative Comparison To demonstrate the effectiveness of approach from qualitative per- spective, we present some visualization results in Figure 7. In Figure 7, the generated images exhibit rich semantic information along with vivid visual effects. For thorough validation of performance in alleviating exposure bias, we present side-by-side visual comparisons in the Appendix.
A Ablation Study To evaluate the effectiveness with different metric functions, we conduct ablations on CIFAR-10 with EDM. For fairness, we train the prompt prediction model under the same settings. In this context, our model achieves great improvements on EDM, as shown in Table 6. In particu- lar, the contrastive loss even slashes the FID from 2.04 to 1.91, effectively increasing the anti-bias performance regarding image quality. Though L norm and L2 norm do not achieve such remark- able improvements on EDM, they effectively validate the performance of our framework. Thus, our framework validated in its ability to improve image quality by alleviating exposure bias.
CONCLUSION
In this paper, we conduct an in-depth investigation into the as exposure bias, which arises from score estimation and discretization errors. To alleviate exposure bias and thereby improve image quality, we put forward a prompt learning framework that employs a lightweight parameterized model to compensate for the bias. The optimized prompt prediction model can improve various pre-trained DMs on different beachmark datasets, with the additional sampling overhead being less than 5%. Moreover, our framework demonstrates great flexibility in adapting to various DM settings, including guidance mechanism, latent diffusion, and fast samplers.
discrepancy, referred to
Broader Impacts and Limitations While our method has achieved significant improvements, it still requires overhead for training and sampling. Besides, it is important to acknowledge that generating deepfake images using our model also entails the potential risk of negative misuse of this technology.
ACKNOWLEDGMENTS
This research was supported by the Ministry of Science and ICT (MSIT) of Korea, under the Na- tional Research Foundation (NRF) grant (RS-2024-003375499), This work was supported in part by the Australian Research Council under Projects DP240101845 18448 and FT230100549. This work was supported by the Natural Science Foundation of China under grant 61836016. The AI training platform supporting this work were provided by High-Flyer AI.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. /7274, 2022.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07300 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.066 065003, 2022b.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the opti- mal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022c.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusson mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22523222522722 2023.
Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11099 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chen-Hao Chao, Wei-Fang Sun, Bo- Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu- Lin Chang, Chia-Ping Chen, and Chun-Yi Lee. Denoising likelihood score matching for condi- tional score-based data generation. arXiv preprint arXiv:2203. 14206, 2022.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim. Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-11481, 2022.
Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Consistent diffusion models: Mitigating sampling drift by learning to be consistent. Advances in Neural Information Processing Systems, 36, 2024.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-based generative modeling. Advances in Neural Information Processing Systems. 34:17695-17709, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Ad- vances neural information processing systems, 0188780-8794 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 33164-231753 2023.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint arXiv:2303.09556. 2023.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrickk and Graham Neubig. Towards a unified view of parfmeter-efffiient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9723 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higheo-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-fre
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffuusion for high resolution images. In International Conference Machine Learning, pp. 13213-13232. PMLR, 2023.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. In European Conference Computer Vision, pp. 274-289. Springer, 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-15964 2023.
Beomsu Kim and Jong Chul Ye. Denoising MCMC for accelerating diffusion-based generative models. arXiv preprint 2rXiv:2209.145000 2022.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pp. 11201-11228 PMLR, 2022.
Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Refining gen- erative process with discriminator guidance in score-based diffusion models. In International Conference on Machine Learning, pp. 16567-16598, PMLR, 2023a.
Dongjun Kim, Chieh-Hsin Lai, Wei- Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. arXiv preprint arXiv:23 10.02279, 2023b.
12

---

Published as a conference paper at ICLR 2025
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 1921166--21701 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint AXXv:2104.0869), 2021.
Mingxiao Li, Tingyu Qu, Wei Sun, and Marie-Francine Moens. Alleviating exposure bias in diffu- sion models through sampling with shifted time steps. arXiv preprint arXiv:2305.1558; 2023a.
Shigui Li, Wei Chen, and Delu Zeng. Scire-solver: Efficient sampling of diffusion proba- bilistic models by score-integrand solver with recursive derivative estimation. arXiv preprint arXiv:2308. 07896, 2023b.
Xiang Lisa and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint XXiv:2201.001190, 2021.
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavio Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Texl-to-image diffusion model on mobile devices within two seconds. arXiv preprint arXiv:2306.0000 00980, 2023c.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404 5411, 2024.
Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10139-101499 2023a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P- tuming v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 2 2021.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.033: 00000, 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality liffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023c.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Deep learning face attributes in the wild.
Aaron Lou and Stefano Ermon. Reflected diffusion models. In International Conference on Machine Learning, pp. 22675-22700 PMLR, 2023.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pp. 14429-14466 PMLR, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfci Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787. 2022b.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint
arXiv:2208. 1970, 2022.
13

---

Published as a conference paper at ICLR 2025
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10209-10218, 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Bric Vanden-Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.065 087404000..... 2024.
Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14297-14306, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 6038-6047, 2023.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprini arXiv:2103.0384) 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob
Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- guided diffusion models. In International Conference on Machine Learning, pp- 16784-16804. PMLR, 2022.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint arXiv:2308. 15321, 2023a.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input per- turbation reduces exposure bias in diffusion models. arXiv preprint arXiv:2301.11706, 2023b.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807..3774 2018.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.149885 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.00 06125, 2022.
14

---

Published as a conference paper at ICLR 2025
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint drXiv:1511.06732,222 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005/1 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:2311. 2023.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1 15980, 2020.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20875-20886, 2023.
Jascha Sohl-Dicksttei, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33: 13:12438-12488 2020.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020b.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training score- based diffusion models. Advances in Neural Information Processing Systems, 14:1415-1428, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference Machine Learning, pp. 3221 1-32222. PMLR, 2023.
15

---

Published as a conference paper at ICLR 2025
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 84:11287-111 1130,, 2021.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1644 2011.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang. and Can Yang. Deep generative learning via Schrödinger bridge. In International Conference on Machine Learning, pp. 10794-10804. PMLR, 2021.
Zike Wu, Pan Zhou, Kenji Kawaguchi, and Hanwang Zhang. Fast diffusion model, 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv: 11.02/222.0044 2.07804, 2021.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196 8206, 2024.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision- laiguae models. arXiv preprint arXiv:2109 11797, 2021.
Junyu Zhang, Daochang Liu, Shichao Zhang, and Chang Xu. Contrastive sampling chains in diffu- sion models. Advances Neural Information Processing Systems, 36, 2024.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
Qinsheng Zhang, Jiaming Song, and Yongxin Chen. Improved order analysis and design of expo- nential integrator for diffusion models sampling. arXiv preprint arXiv:2308.02157, 2023.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.08667, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
A RELATED WORKS
Diffusion Models DMs (Dhariwal & Nichol, 2021; Ho et al., 2020; Kingma et al., 2021; Nichol & Dhariwal, 2021) are a new family of generative models with remarkable performance, particu- larly in the field of 2D image generation (Gao al., 2023; Vahdat al., 2021; Rombach et al., 2022), due to their ability to model complex data distributions. This is mainly because DM training directly models the target data distribution via minimizing the upper bound of the model log likeli- hood (Sohl-Dickstein et al., 2015; Luo, 2022). In this manner, DMs enable achieve comparable mode coverage (Kingma et al., 2021; Song et al., 2021a; Lu et al., 2022a; Kim et al., 2022), re- flected at lower negative log likelihood. Based the rigorous SDE framework (Song et al., 2021b; Anderson, 1982), some classic DMs contribute the image quality from different aspects, including deeper model structure (Song & Ermon, 2020; Song al., 2021b; Kingma et al., 2021; Peebles & Xie, 2023; Kim et al., 2023b), diffusion schedule (Lin et al., 2024), diffusion in latent space ( (Vahdat al., 2021; Rombach al., 2022; Jing et al., 2022), refined weighting schedules (Choi al., 2022; Kim et al., 2022: Song et al., 2021a) and well-designed training objectives (Kingma et al., 2021; Nichol & Dhariwal, 2021; Karras et al., 2022), as well as rational optimization strategies (Hang et al., 2023; Wu et al., 2023). Moreover, recent works utilize conditional information to guide the image generation (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), such as class label and text prompt (Ramesh et al., 2022; Ruiz et al., 2023; Saharia et al., 2022; Li et al., 2023c), which further improve the image quality. With the help of those techniques, DMs achieve new SoTA modeling ability (Kim al., 2023a; Peebles & Xie. 2023; Kim et al., 2023b) and better class diversity com- pared to previous SoTA generative models. Although they enable the generation of high-quality images, they cannot avoid the exposure bias problem, which does influence the image quality.
16

---

Published as a conference paper at ICLR 2025
Normal Sumpling Method
Baas Prompling Sampling
X++1
XF
X1
Xg
Kelor)
pp(k..)
Ausi-iaa
Prongt Nedd
Proept Madd
Figure 3: Anti-Bias Sampling. Our framework employs the prompt prediction model Ug- (-) to predict a prompt with the input of fz+1, and compensate the bias in the current time step L + 1. Better samples be obtained via va- (fz+1) ) 81+1, serving the input of DMs in next time step We can therefore enhance the image quality by iteratively running this process.
Exposure Bias Problem The exposure bias problem is originally mainly studied for language mod- els by the nature language process community (Ranzato et al., 2015), and the exposure bias of diffusion model is less explored (Ning et al., 2023b; Li et al., 2023a). Some classic works propose various methods to reduce the score estimation and discretization errors, i.c., the underlying source of the exposure bias, to handle the exposure bias problem indirectly. For instance, some methods make great efforts using high order numerical solvers (Zhang & Chen, 2022; Li et al., 2023b; Zhao et al., 2023; Zhang al., 2023). Stable diffusion (Rombach et al., 2022) matches the score in latent space (Vahdat et al., 2021), which naturally reduce the discretization error by solving the integral in a lower dimension. DMCMC (Kim & Ye, 2022) utilizes MCMC to obtain a good initialization points close to the modeling distribution, aiming to reduce the accumulation of errors. Besides, (Kim et al., 2023a; Chao et al., 2022) propose to adjust the matched score via robust discriminator. Beyond the usual treatment, certain works (Salimans & Ho. 2022; Song et al., 2023; Meng et al.. 2023; Kim et al., 2023b) mitigate the sampling errors in small time steps of model sampling by distilling knowl- edge from larger sampling steps. Specifically, Consistency Model (Song et al., 2023) even slashes the neural function evaluations (NFEs) to only two steps with improved score matching accuracy. Recently, rectified flow (Liu et al., 2022: 2023c; Esser et al., 2024; Ma et al., 2024) simulates the optimal transport between prior distribution and target data distribution via straightening the sam- pling trajectory. Orthogonal to them, present prompt learning framework, which employs transition function to learn an anti-bias prompt to compensate the next model input and handle the exposure bias in direct manner.
Prompt Learning Prompt learning is first proposed in natural language processing (NLP) (Liu et al., 2023b), which employs text to help pre-trained large models (LMs) "understand" the task. Subsequently, GPT-3 (Brown et al., 2020) demonstrates remarkable performance to downstream transfer learning tasks even in the shot or zero-shot settings (Radford et al., 2021; Ouyang et al., 2022). To improve the readability of prompting text for LMs, some methods (Jiang et al., 2020; Shin al., 2020) propose constructing more plentiful prompting texts. Recently, some heuristic approaches consider more efficient way, treating prompt as task-specific continuous vectors and fine-tuning them via gradient propagation, namely Prompt Tuning (Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021). Based on the great success on LMs, there are lots of vision LMs (He et al., 2021; Radford et al., 2021; Yao et al., 2021; Zhou et al., 2022) employ the text encoder to extract more information from the text prompt. More recently, Jia et al. (Jia et al., 2022) explored visual prompting in recognition tasks. As concurrent work, Bahng et al. (Bahng et al., 2022) demonstrated that visual prompting is effective for CLIP and distribution shift. In this paper, propose our anti-bias prompt learning, a novel variant of the visual prompting, aimed at alleviating the exposure bias problem in DMs.
B EXPERIMENTAL DETAILS
Architecture We follow the EDM framework (Karras et al., 2022), which adopts the NCSN+ model proposed by (Song al., 2021b) as the backbone of prompt model, as shown in Table
17

---

Published as a conference paper at ICLR 2025
8 of EDM paper. To be specific, NCSN++ is U-shaped architecture based on (Ho et al., 2020) that uses Finite Impulse Response (FIR) upsampling and downsampling, rescales skip connections, and employs four BigGAN (Brock, 2018) residual blocks at each resolution. Moreover, NCSN++ incorporates additional residual skip connections from the input image to each block in the encoder. The only difference between NCSN++ and our prompt model backbone is that we remove the time embedding and reduce the number of model channels for greater efficiency. Concretely, we set the model channels in the NCSN+ backbone for resolutions 32, 64, and 256 to 32, 32, and 64, respectively. As result, the corresponding model parameters for the different prompt models are 3.2M, 3. 3.2M, and 12.7M.
Training As mentioned before, we employ the EDM backbone to serve as our prompt prediction model and train our prompt model on the same datasets used to train the baseline models for a fair comparison. In practice, we exclude the time embedding setting from the architecture of EDM, thus reducing some model parameters. For instance, the original EDM backbone contains 56M parame- ters, whereas the backbone without time embedding contains 51M parameters. During training. we set the batch size to 1024 for all experiments and keep other hyperparameters the same as in EDM training. Detailed training settings be found in (Karras et al., 2022), and we maintain the default values. To train the model, we allocate A100 GPUs to optimize them and test the experimental results on just one A100 GPU. Specifically, we employ 8 A100 GPUs for training CIFAR-10 and CelebA, while we use 16 A 100 GPUs for training on ImageNet. Additionally, we allocate only 4 A100 GPUs for training the prompt prediction model for latent diffusion. For most experiments, the training iterations range from 100k to 150k across all datasets, which are considered acceptable training expenses. In practice, for the EMA selection, we maintain the same settings as those used in the consistency models (Song et al., 2023). Specifically, we set the EMA value to 0.9999 when training our model on CIFAR-10, and we set the EMA to 0.999943 for LSUN and ImageNet.
Sampling After completing the training process, we employ the optimized model to reduce exposure bias using the sampling process shown Figure 3 and Algorithm It is worth noting that the total sampling NFEs for the original sampler is T, our prompt prediction model will be employed T - times within the same sampler when aiming to mitigate exposure bias. For sampling computation, our prompt prediction model only increase the sampling time less than 5%. To verify this, we test it on an A 100 GPU via sampling 1k images with 35 NFEs. Concretely, EDM requires 33.9 seconds, while model increases the time only 36.3 seconds, with cost increase of 2.4 seconds. When sampling one image, the overhead can be almost ignored. For by-dee comparison, we present more results in Figure 4 to Figure 6. Other results are shown in Figure 7 and Figure 8, all randomly generated. Moreover, we present an additional ablation study to evaluate performance with different model parameters, as shown in Table 9. In this paper, we employ the backbone with 3.2M parameters to serve as our prompt model, to ensure the efficiency in sampling process.
Comparison on larger model or more sampling steps We also conduct experiments to test whether using more NFEs or additional parameters can achieve better results, detailed shown in Table 10. While employing a larger model has the potential to decrease score estimation errors, necessitates substantial training resources and considerable human effort. This is because the exposure bias phenomenon is caused by inherent factors within the diffusion modeling framework. Analogously, increasing NFEs will improve the image quality, but the performance marginally improves and does not increase indefinitely. Hence, our framework is meaningful for the diffusion community as it provides a special case for reducing exposure bias. Furthermore, we test more metrics to verify the effectiveness of our model, including sFID and NFEs, detailed shown in Table 7. Specifically, we achieve improvements both CIFAR-10 and ImageNet when using our model to enhance ADM-IP.
Diversity Testing Following the main design philosophy, our model can improve the image quality without affecting the image diversity. Because the diversity in diffusion modeling framework is mainly depends on the diffusion term G, in Eq. (1). On the contrary, our model only change the score term to sg(+++1 + U (fz+1), t + 1), thus our model will not affect diversity. To verify this, we also conduct experiments to test precision and recall on ADM, which are common metrics for evaluating diversity. Specifically, the precision and recall values are 0.69 and 0.63, respectively. These values are still 0.69 and 0.63 after employing our framework. Since precision and recall remain the same, our model has no negative effects on the diversity.
Comparison with training free anti-bias model We conducted experiments on the recent training- free method (Ning et al., 2023a), which is effective in reducing the exposure bias without any train-
18

---

Published as a conference paper at ICLR 2025
More Metrics for Evaluating Perfor- Table 8: Performance on CelebA. To evalu- mance in Comparison with ADM-IP. To better test ate effectiveness, we employ ADM (Dhariwal the effectiveness of our prompt prediction model, we & Nichol, 2021) and ADM-IP (Ning et al., employ more metrics, such as sFIDI and NFEs,, to 2023b) as baseline models, using the default test it on different datasets, including CIFAR-IO and samplers from the original papers.
Table 7:
ImageNet. Moreover,
employ the classic ADM-
Models
FIDI
NFEs|
sFID
IP (Ning et al., 2023b), the first work address the exposure bias issue, to verify these results. As shown
ADM
3.02
100
5.76
below, our prompt model further reduces the expo- sure bias for ADM-IP these datasets with different NFEs, despite ADM-IP being designed with train- ing strategy mitigate exposure bias, demonstrating the flexibility of our model.
ADM+ours
2.93
100
4.74
ADM-IP
2.21
100
4.33
ADM-IP+ours
2.15
100
4.19
Table 9: Ablation Study on Model Param- eters. We design various prompt models (PM) with different parameters to enhance
Models
Dataset
SFID
NFEs.
ADM-IP
CIFAR-10
3.86
100
EDM (Karras al., 2022) CIFAR-IO, the
ADM-IP+Ours
CIFAR-10 -00
3.80
100
detailed FIDJ results are shown as below.
ADM-1P
CIFAR-TO TO
3.89
80
Models
Parameters
FID
ADM-IP+Ours
CIFAR-10 10
3.84
80
ADM-IP
ImageNet
3.11
TO0
EDM
2.04
ADM-IP+Ours
ImageNet
3.04
100
EDM+PPM (((age)
12.7M
1.90
ADM-IP
ImageNet
3.36
80
EDM+PM (regular)
3.2M
1.91
ADM-IP+Ours
ImageNet
3.33
80
EDM+PM (small)
0.8M
2.02
Figure 4: Side-by-Side Comparison. We use ADM trained on ImageNet 256 as the baseline, with the sampler set to DDIM. The first row displays images generated by ADM, while the second row shows images improved by our prompt model, with both rows using the same initial noise. The difference is highlighted by red box, indicating that model can mitigate unrealistic features.
ing. The FID values for EDM and EDM-ES presented in (Ning et al., 2023a) are 1.97 and 1.95, respectively, because they were calculated using fixed seed. For a fair comparison, we recalcu- lated the FID using a random seed, resulting values 2.04 and 2.01. Compared to EDM-ES, prompt model significantly improves generative performance, the FID decreases from 2.04 to 1.91. Although our method achieves much better performance than EDM-ES, requires additional training computation, which unfair for comparison. Hence, do not present it in main paper.
C PROOFS OF CONTRASTIVE Loss
In our experiments, the best results are achieved by using contrastive loss as the metric function. This mainly because the contrastive loss theoretically assured that can further reduce the KL. divergence DKL (p(z, 1 (1p &t, I(+1)) between the true Gaussian kernel p(r; x(+1) and the modeling kernel Po Pr, 2.41). Below, we provide a detail proof for this bound.
19

---

Published as a conference paper at ICLR 2025
Table 10: Performance of EDM (Karras al., 2022) with More NFEs and Model Parameters CIFAR-10. To verify efficiency, we use more NFEs to test FID by sampling 50K images and report the corresponding time cost below.
Models NFES
FID
Time (min.))-
EDM 35
2.04
28.0
EDM X 35 + Ours x 34
1.91
30.3
EDM 37
2.03
30.1
EDM X 47
2.02
38.3
EDM 57
2.02
45.7
Figure 5: Side-by-Side Comparison. We use ADM trained ImageNet 256 the baseline, with the sampler set to DDIM. The first row displays images generated by ADM, while the second row shows images improved by our prompt model, with both rows using the same initial noise. The difference is highlighted by red box, indicating that our model can mitigate unrealistic features.
Theorem 1. Let p(x+) be the marginal probability density time step that locates in the diffusion path, po (ft) be the distribution from the reverse path that simulated by a pre-trained DM. Assume X and X4 represent two batches images of It and &n that sampled from p(xx) and po(iz) respectively. Then we can derive the upper bound of the gap between the true and the modeling transition kernels is actually the InfoNCE loss LintoNC((:s))
DEL (p(a 1++1))lP @ ((41)) S EinfonCE(X,,35)
(15)
Proof. Before derive the Eq. (15), first consider the mutual information (MT) Poole al. (2019) between batches of images X, and X, time step denoted [X2; X.). We can build a tractable variational upper bound by introducing the true distribution p(rz) in the diffu- sion path to the intractable marginal p(r, rx+1)
( Te+1 P (xr &e. T2+1) In theory, it is tractable to map I++1 to 2, with the state transition matrix. By multiplying and divid- ing the integrand in MI by p(æ)) and dropping negative KL term, we enable to get the tractable variational upper bound (Poole et al., 2019): I(X;; Xx) > DKL. (p(x, x1+1)llpo fr, x141)). Analogously, by optimizing LintNNC((s.),
connect the InfoNCE loss with MI (Oord al., 2018) via LIIGNCE 2 log (NV (Xx; X²). Here, N is the number of images in each training batch that containing one positive sample and N - 1 negative samples. Remarkably, large will make this bound tighter. In this context, we enable to derive the upper bound of DKL (p(ar) 1 (el11)lpp(( (x; de, 2141)) is actually a well-designed contrastive loss LIdoNCE(;)) via employing Jensen's 's inequality.
20

---

Published as a conference paper at ICLR 2025
Figure 6: Side-by-Side Comparison. We use ADM trained on ImageNet 256 the baseline, with the sampler set to DDIM. The first row displays images generated by ADM, while the second row shows images improved by our prompt model, with both rows using the same initial noise. The difference highlighted by red box, indicating that our model can mitigate unrealistic features.
Figure 7: Randomly selected 256 256 images improved by our prompt learning framework.
Figure 8: Randomly selected 256x256 images improved by our prompt learning framework.
21


Reviews:
[{"review_id": "GgkrIiS7hn", "reviewer": "Reviewer_xBN2", "summary": "This work reiterates the issue of exposure bias in diffusion models that is caused by score estimation and discretization errors. The work proposes a prompt prediction model, trained to simulate exposure bias, that can reduce the gap between the training trajectory and a typical sampling trajectory by introducing a reasonable overhead (5\\%). The authors show experiment results on a few benchmark datasets (CIFAR-10, CelebA and ImageNet), demonstrating improvements in FID and IS.", "questions": "I have made a suggestion as part of my weaknesses statement. It would be interesting to knopw the authors' thoughts on whether diversity of generations could be affected.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well motivated and well articulated and easy to follow. The method makes intuitive sense and the experiments on a few benchmark datasets further support the authors' claims. The method can be easily implemented and integrated with various diffusion model architectures.", "weaknesses": "While the experiments focus on improving image quality, is there a possibility of trade-offs such as loss of diversity that could be caused by applying the prompt adjustment? I think additional experiments that take into account possible trade-offs could further strengthen the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p16IQJTPOq", "reviewer": "Reviewer_TaPC", "summary": "In this paper, the authors begin by addressing the exposure bias that becomes increasingly pronounced along the sampling trajectory, as each time step accumulates newly generated score estimation and discretization errors. To tackle this issue, the authors propose a prompt learning module designed to eliminate the bias introduced by noised images from previous steps during the training and inference processes. The experiments demonstrate the effectiveness of the prompt learning across various model architectures and datasets.", "questions": "1.I understand that the authors propose the module's function is to bridge the gap between training and inference. However, I am curious why it is referred to as \"prompt.\" It seems that the results produced by the model are not intended to serve as a \"prompt\" and do not interact with information from DMs, which may lead to reader confusion.\n\n2.The readers will be a bit confused about Table 4. What do the numbers 10, 20, and 50 represent? What metrics are reflected in the results presented in the table? It is suggested to modify Table 4 or show more details more in caption.\n\n3.Please provide detailed responses, as I will reassess the proposed methods and results based on your answers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The insight is interesting.\n\n2.The method is simple and effective.\n\n3.The author presents some theoretical analysis which makes the paper more believable.", "weaknesses": "1.The architecture of the prompt learning network is unclear. It is recommended that the authors include the proposed module architecture or network detail description in the supplementary materials to aid readers in understanding the method.\n\n2.While the experiments indicate some improvement from the proposed module, the selection of baseline models is relatively simple. Will the issue of exposure bias become less significant in practical applications after updating the model architecture and expanding the training data scale?\n\n3.The improvements depicted in Figures 4, 5, and 6 (in the supplementary materials) do not appear to be particularly pronounced.\n\n4.There are missing details regarding the training data used for the prompt learning framework.\n\n5.The paper lacks references or calculation methods for NEFs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FL6fCeC67W", "reviewer": "Reviewer_Pxp6", "summary": "This paper addresses the exposure bias in diffusion models caused by score estimation errors and discretization errors, by proposing a novel prompt learning framework. This framework employs a prompt prediction model that learns anti-bias information in each timestep during the reverse process, thereby mitigating the impact of exposure bias during image generation. To train this prompt prediction model, the authors utilize the Variational Estimation Stochastic Differential Equation (VE SDE) to simulate exposure bias. Extensive experiments were conducted on several traditional benchmarks, including CIFAR, CelebA, and ImageNet 256x256, demonstrating significant improvements across various diffusion models.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-structured and presents its findings effectively. The authors provide substantial reasoning for each step, making it an enjoyable read that offers valuable insights.\n\n2. The paper introduces a novel prompt learning framework that addresses the gap between training and sampling in diffusion models by training an additional model. The $\\textbf{prompt prediction model}$ effectively mitigates exposure bias, and the authors conducted extensive experiments on several well-known benchmark datasets, yielding significant results. Furthermore, the model introduces minimal time delay, demonstrating practical value.", "weaknesses": "Overall, from my perspective, I think this paper is valuable and should be accepted. However, there still remains some weaknesses. If the authors can respond my puzzles, I will raise my scores.\n\nFor presentation:\n\n1. The title of this paper ``$\\textbf{Anti-Exposure Bias in Diffusion Models via Prompt Learning}$''. Through the whole paper, I think it is more like noise learning instead of prompt learning? What is the relationship between the proposed methods and prompt learning?\n\n2. This paper only provides the improved images in Fig.7. Can the authors provide more images for initutive comparsions?\n\nFor method:\n\n1. In eqn. 7, the authors claim $\\Phi(\\hat{x}_t,x_t)$ represents the exposure bias. However, based on my understanding, I only find the $\\textbf{discretization error}$, rather than both $\\textbf{discretization error}$ and $\\textbf{score estimation error}$.\n\n2. This paper utilizes VE SDE to get $x_{t+1}$, in order to simulate exposure bias. However, from my perspective, I think the $\\hat{x}_t$ from prompt prediction model is not the Gaussian noise. This approach can be considered as a way to mitigate the training-sampling discrepancy of diffusion model such as [1]. Any comparsions between these methods?\n\n[1] Common Diffusion Noise Schedules and Sample Steps are Flawed \n\nFor experiments:\n\n1. All of these experiments are conducted in traditional benchmark, such as CIFAR, CelebA, and ImageNet 256x256. Moreover, the DMs used in this paper are not the popular T2I diffusion models, such as SDXL. Although the authors leave this as the future work, I think without these experiments, the practial value of this paper is limited.\n\n2. Moreover, in this paper, the authors state the batch size of all experiments is $\\textbf{1024}$, and the iteration of EDM in Tab. 8 requires 150k. That means approximate 100M data samples. The  $\\textbf{prompt prediction model}$ is lightweight and the parameters of the DMs are fixed. Why such a huge amount of data samples are needed? \n\n3. Based on 2, the size of image in all experiments are not bigger than 256x256. How about generating the images with 1024x1024, should it require more data samples and computional resources?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lj6mFpWnYg", "reviewer": "Reviewer_3M1k", "summary": "This paper proposes to learn an anti-bias prompt for the generated sample at each sampling step, aiming to compensate for the exposure bias that arises. Following this design philosophy, our framework rectifies the sampling trajectory to match the training\ntrajectory, thereby reducing the divergence between the target data distribution and the modeling distribution.", "questions": "See the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method aims to solve a challenging problem known as exposure bias, which has a wide range of application scenarios.\n2. The proposed method is theoretically sound.", "weaknesses": "1. More visualization comparision should be provided.\n2. Based on the visual results, the improvement of the proposed method is not very significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RQCKIEPqxj", "reviewer": "Reviewer_2SrK", "summary": "This paper analyzes the gap between the true score function during training and the score function estimated during inference, highlighting that this gap can be narrowed, thereby improving model performance through the training of a prompt learning model. Experiments demonstrate that the algorithm performs well on both 64x64 and 256x256 diffusion models.", "questions": "The authors utilize the learning paradigm of the Prompt Learning Model and inquire whether they have attempted to minimize the difference in Eq. 12 using LoRA. If so, there may not be any additional computational overhead required?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work revisits the gap in the score function during training and inference, and proposes a fine-tuning framework to mitigate this gap.\n\n2. The idea of this work is novel, which suggests that it is difficult to achieve perfect estimation of the score function when diffusion models are trained.\n\n3. The presentation of the paper was excellent and easy to understand.\n\n4. The authors conduct sufficient experiments across various diffusion models, including those on CIFAR-10, ADM on ImageNet-256x256, and Latent Diffusion, to support the effectiveness of the algorithm.", "weaknesses": "1. while the authors use a lot of analysis to show that the gap between the true Gaussian kernel and its modeling counterpart does exist, there is a lack of visualization to further illustrate this point.\n\n2. It is suggested to move Table 9 in the appendix to the main paper, this is important to illustrate the efficiency of the algorithm.\n\n3. The authors need to conduct further analysis of the effect of the number of parameters of the Anti-Bias Prompt Model on inference performance.\n\n4. The authors do not provide code, which is important for the reproduction of the algorithm, even if it is simple code on CIFAR-10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)