Learn to Guide Your Diffusion Model
Authors: Alexandre Galashov
Ashwini Pokle
Arnaud Doucet
Arthur Gretton
Mauricio Delbracio
Valentin De Bortoli
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion models
Classifier-free Guidance
Conditional sampling
generative mode
Primary area: generative models
TL;DR: Learn Classifier-Free guidance (CFG) weights as a function of time and conditioning to better approximate the target distribution

Abstract: Classifier-free guidance (CFG) is a widely used technique for improving the perceptual quality of samples from conditional diffusion models. It operates by linearly combining conditional and unconditional score estimates using a *guidance weight* $\omega$. While a large, static weight can markedly improve visual results, this often comes at the cost of poorer distributional alignment.
In order to better approximate the target conditional distribution,
we instead learn *guidance weights* $\omega_{c,(s,t)}$, which are continuous functions of the conditioning $c$, the time $t$ from which we denoise, and the time $s$ towards which we denoise. 
We achieve this by minimizing the distributional mismatch between noised samples from the true conditional distribution and samples from the guided diffusion process.  We extend our framework to reward guided sampling, enabling the model to target distributions tilted by a reward function $R(x_0,c)$, defined on clean data and a conditioning $c$. We demonstrate the effectiveness of our methodology on low-dimensional toy examples and high-dimensional image settings, where we observe improvements in Fréchet inception distance (FID) for image generation. In text-to-image applications, we observe that employing a reward function given by the CLIP score leads to guidance weights that improve image-prompt alignment.

Published as a conference paper at ICLR 2026
LEARN TO GUIDE YOUR DIFFUSION MODEL
Alexandre Galashov
Ashwini Pokle
Arnaud Doucet Google DeepMind a arnauildoeet@@ooglee.cco
Google DeepMind agalashov@google. com com
Google apokle@google. com
Arthur Gretton
Mauricio Delbracio Google
Valentin De Bortoli Google DeepMind
Google DeepMind
gretton@googie.com com com
mde mndelbra@googee.cm
voebortoll@googee.com
ABSTRACT
Classifier-free guidance (CFG) a widely used technique for improving the percep- tual quality of samples from conditional diffusion models. It operates by linearly combining conditional and unconditional score estimates using a guidance weight w. While large, static weight can markedly improve visual results, this often comes at the cost of poorer distributional alignment. In order to better approximate the target conditional distribution, we instead learn guidance weights we,(s,f), which are continuous functions of the conditioning c, the time t from which we denoise, and the time towards which denoise. We achieve this by minimizing the dis- tributional mismatch between noised samples from the true conditional distribution and samples from the guided diffusion process. We extend our framework ward guided sampling, enabling the model to target distributions tilted by reward function R( c), defined on clean data and conditioning c. We demonstrate the effectiveness of our methodology on low-dimensional toy examples and high- dimensional image settings, where we observe improvements in Fréchet inception distance (FID) for image generation. In text-to-image applications, we observe that employing reward function given by the CLIP score leads to guidance weights that improve image-prompt alignment.
INTRODUCTION
Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2021b: Song and Ermon 2019) produce high-quality synthetic data in areas such as images (Saharia et al. 2022 Flacetorest Labs 2025). videos (Google 2025), and proteins (Watson et al. 2023 Abramson et al. 2024). These models proceed by gradually adding Gaussian noise through diffusion process, transforming the data distribution into a Gaussian distribution. The generative model is obtained by approximating the time-reversal of this noising process. Practically, this relies learned denoiser network which is obtained by minimizing a regression objective. A similar strategy can, in principle, be developed to sample from a conditional distribution by learning a conditional denoiser network.
Recently, Classifier-Free Guidance (CFG) (Ho and Salimans) 2022) has become a popular choice to address the task of conditional sampling. CFG modilies the generating process by incorporating guidance term, calculated as the difference between conditional and unconditional denoiser networks. This difference is scaled by guidance weight, w, such that
recovers unconditional generation whereas = 0 corresponds to the conditional one. Empirically, it was shown that CFG can drastically improve performance of diffusion models compared to conditional generation (which only uses the conditional denoiser), especially for very high guidance weights (Saharia al. 2022) Rombach et al.) 2022a). (Chidambaram et al. 2024 (Wu et al. (024) argued that such a regime pushes the samples to the edge of support such that they become easy to classify.
Classifier free guidance was initially justified as a method for sampling from a modified distribution (cc)))p(())))) which amplifies the conditioning term (Ho
Per (Zgle) p(z.) p((ule))** and Salimans, 2022). However, subsequent work has established that the standard CFG sampling
procedure does not, fact, produce samples from this target distribution (Du et al. 2023 Koulischer

---

Published as a conference paper at ICLR 2026
al. 2025a). Instead, CFG sampling shifts the generated samples towards the modes of the original conditional p(ag c). While this discrepancy can be corrected with computationally intensive methods
like Sequential Monte Carlo (Skreta al. [2025] He et al. 2025) or Markov Chain Monte Carlo
(Du et al. 2023: Moufad et al. 2025] Zhang et al.
2025), these approaches are often impractical for large-scale applications. More importantly, naive CFG already yields significant quality boost and
therefore such correction might not be required.
Many works, see e.g. [o and Salimans (2022); Kynkäänniemi et al. (2024) Wang al. (2024) Kim et al. (025), report improved FID scores when using some level of guidance (w > 0). This observation is at odds with the original motivation of CFG. Given that the learned denoiser ( c) is only an approximation of the true denoiser B|zo| Z2, e), this suggests that CFG acts as correction to this approximation which leads to a better modeling of the target conditional distribution p(ro (c).
Based on this observation, we propose to learn a guidance weight w in order to better approximate the conditional distribution p(Zo c). We generalize the CFG method and allow the guidance weight to be both conditioning and time dependent, i.e., We,(s.t) This weight is used to guide the denoising process from time to time 8. We learn We.(s.1) by matching the distribution of samples from the true diffusion process and the distribution of samples of the guided one. Empirically, we show that our method allows us to learn (s,(s.t) that better approximates the target distribution compared to the unguided model. For image generation, this leads to a consistently lower FID (Heusel et al. 2017) than the unguided model or a model with a constant guidance weight.
We further develop our method in setting where additional reward function R(xo, c) is defined on samples from the diffusion model and a conditioning c, and we want to bias the samples from the model to the regions of high rewards R( c). In text-to-image applications, we found that empowering our method with a reward function given by the CLIP score (Hessel et al. (2(2221)) l). leads guidance weights which improve both FID and umage-prompp alignment.
The paper is organized as follows. In Section 2
we provide a short background on the necessary
concepts. In Section 3, we describe our approach of learning guidance weights (*.(..)). In Section we present the related work on the topic. Finally, the experimental results are presented in Section
BACKGROUND
Notation. We write Pt for a probability distribution p(ar.), and Paje to denote a conditional distribu- tion pxx) for any s, We also use Poje denote p( (e). Pstt,o for p(x,|=r, m0), and Pslt,c for write N(x;; j,, Σ) to denote the Gaussian density of argument x, mean and covariance Σ and and.N(r (i,, Σ) for the distribution.
p(z,_l=e c). We denote by Po.c joint distribution p( c). Finally,
Diffusion models. The goal of conditional diffussinn models is to sample from target conditional distribution Puje on Rd where p(c) is conditioning signal (i.e. text prompt) sampled from conditioning distribution. We adopt here the Denoising Diffusion Implicit Models (DDIM) framework of Song et al. (2021a). Let It, Paje and define for 0 to < tN the process Ftyts = (x : Tix) by
p( (Tta)) PeN (FeN (tt)) I Psuleats oe (
where, for 0 <8615, have
(2s,,0(2,/1,,, 2o) (Zsi (o, rz), Es,()
Both the mean Ma.a (I0, .ri and covariance Σ,, depend on a churn parameter & [0.1], which controls the amount of stochasticity. Their full expressions are given in (3) in Appendix B Here, the mean and variance are selected so that for any t E [0, 1]. we recover the noising process
P)p((z//00) = (x,, a,4o, o ofld),
(3)
for ae and o, such that ag = =1,00- = 0 and a1 = 0, = i.c. p1o((11200) = Nax1; O, Id). More
precisely for any 0 < st 1, we have
Pajo (2)0(rr(11)) (ua) = J Pt o o 20)d
2

---

Published as a conference paper at ICLR 2026
Sampling with DDIM. At inference time, Tto Pole is generated by starting from Gaussian Fiy ~ (0, Id) and St, ~ p( : for k = N = 0, where for 0 I < /
1,..., Patt P I2,, c) Ps ax (a,, 2o Poje. 1,,, c)dxo.
(4)
where P (rt, c) is a posterior distribution, see also (Song et al. 2021a) for discussion. Since we do not have access to Pojt.es we approximate it by (20(2).0) for any t E [0. 1}, where (z c) Rs Bl c| is a neural network denoiser with parameters trained by minimizing the loss
( = Jo
|| (mr, c)
(5)
Here, X(1) 2 is a weighting function (see (Kingma al. 2021
This yields the following
procedure to sample approximately from Pole> by starting from Fity
No0 Id), we then follow
Tty N (Pte (Co(Fen))) c)2 ates)) Etedraa)
k = - 1,..,
(6)
Classifier-Free Guidance (CFG). Let (r,,, Ø) R B[ be an unconditional denoiser learned using a regression objective (5) (where conditioning is omitted). CFG replaces ie c) when simulating approximately from Paje by
Eo( c) + +w (o
where w is the guidance weight and An( c) ( c) to (It; Ø). In order sample the data with CFG, we first sample Tax ~ ~N(00, Id) and then follow
Ite (e (Z C; C.W), Fie+1) Eend N
(8)
When + 0, the sampling procedure K is equivalent to conditional sampling (6) while w corresponds to procedure using only the unconditional denoiser, allowing to sample approximately from Po- In practice, one sets w > 0 in order to emphasize conditioning.
In this paper, we generalize CFG and make guidance weight We,(s.() function of conditioning and of two time-steps s < t, where is the proposal timestep from which we denoise towards a target timestep in 4h. Denoising following (8) corresponds to setting. f4 and = th+11 We denote
w = (a.) (eo.t) (00.1))) Algorithm 2 we describe the CFG sampling method with DDIM.
Maximum Mean Discrepancy. To optimize the guidance weights w, we match the diffusion
distribution and the guided p see Section 3 We employ the Maximum Mean Discrepancy
(MMD) (Getton et al. 2012) with energy kernel Székely and Rizzo 2004 Sejdinovic et al. 2013
(MD( .p] Ep vll 3 (Ep 11 2'122 Epallll s/ 1/1
for independent æ, æ² pd and y.y' ~ p. with 3 E (0, 2) and À E [0, 1). We require À for valid MMD, however may be preferable for generative modeling (Bouchacourt al. 2016 De Bortoli et al.) 2025); other characteristic kernels may also be used (Sriperumbudur et al. 2010
3 LEARNING TO GUIDE YOUR DIFFUSION MODEL
In this section, we introduce our method for learning the guidance weights, w, assuming access to a pre-trained conditional denoiser, fo(x,, and an unconditional one, (x 2). Our approach is based on enforcing consistency conditions- that any valid diffusion process must satisfy. We first derive a theoretically sound objective from marginal consistency condition, which however impractical to optimize due to high variance. To overcome this, we introduce stronger, more practical condition, which we call self-consistency. Enforcing self-consistency is sufficient for achieving marginal consistency and, crucially, results in simple, low-variance objective. We formulate our complete approach for learning guidance weights based on this simplified approach. We present alternative approaches in Appendix C
3.1 CONSISTENCY CONDITIONS
Marginal consistency.
The process defined by admits the marginal distribution for E (0, 1] Pajo (2o0s////)))oo..oooooo (N, e)dxade.
(r(x.)
(10)
3

---

Published as a conference paper at ICLR 2026
This marginal distribution can also be obtained for any < t via
Ps (x.) = f (aje,c(zs Te, c)ptjo(2))) (o,c c)dzydzode,
(11)
with Palt.e given by DDIM 4 The equality (11) states that the marginal at time can be obtained by sampling (Io, c) Po,c, noising zig to di 22 Ptjo and then denoising it back to time with Patt.c
We now consider denoising mechanism relying on the guided denoiser approximation 7 which follows the construction in (II) to obtain a sample at time Again sample (zo, e) Po,e and It ~ (ejo(-(zo) However, the denoising to time s where 0 S < Σ is done with the guided denoiser approximation, so that the marginal distribution of the resulting sample at time is
1,10,0) (Ix) = [[
(8,w) Palt,c (xa c)pr (2, (ro))o,c(Too c)dx,d
ps
(12)
where
Pajt.c (ara ()), ( (re, Si (are, cww))
(13)
The distribution Ps 1,,0, 12 is typically not equal to ps (10) as we used a delta-function approxima-
tion (13) with model dig (3it, G; (:w) instead of sampling from polt,e as required by (4). Nevertheless,
we could attempt to find guidance weights w to satisfy marginal consistency. i.c. for all 0 < < t S
(0,w) (xxx) (x())))
Ps
(14)
This could be achieved by minimizing the MMD
(w) E (s.t)))((s.) [MD (8. (8,A) Pp (.).
(15)
where t(s,t) is distribution on 0 S < S 1. This distribution is crucial for good empirical performance, and we discuss detail in Section (.2 The gradient of (13) could suffer from high variance due to marginalization over (Io, c). however, which could make it challenging to optimize. We found that this approach did not work in practice. Below, we propose a simpler, lower variance method, albeit one that imposes more constraints the guided backward scheme compared to (I5).
Self-consistency (conditioning on (c, (o))) For a fixed (zo, c) ~ p(roo c), we denote by
(.(W.))) Pajo.c (Ir,/wo, = f Pajt,c (xal c)Pr (xx |ro )
(m:W)
(16)
which is the term under integral in depending on (Iro c). We consider self-coosissency condition
(,(0,w)) (I zo. c) ~ Ps (x c) = (=)))x)(
(17)
0.
where 1:(0.ww) (-12n,, c) is given by 160 and (o)o(-(20)) is noising process (3). We used notation Pejo,c(1i (ig c) = P=jo(=a//a) to highlight that Ty ~ Poje(-lc). Intuitively, this condition means that as we start from sole and go through the noising process 24 Ro and then denoise with guidance to time 8 < f, the distribution at time should be the same of the noising process.
The condition 117 is much stronger than 14). Indeed,
p satisfied for every (zg. Po.e- then by integrating it over (zo, c). we will get (14). The reverse is not true. Moreover, the lack of dependence on x; (and on wo) in guidance weights We.(s.t) makes very unlikely for this condition to hold. However, it provides learning signal for guidance weights and we will aim to satisfy it approximately.
For (Io, c) in the support of p(xo, c), could approximately satisfy 17 by minimizing wrt w. (.(0.w) MMD |s,o,,c (- (-Izo, c). ( (-12))). see Averaging over all (Io. c) ~ p(zo, c) leads to
La,((w)= = ((oo.e)-mm..ss..ppee... [MD (8. P
(.(ww) (-|ao, c). P-)),c(-I
(18)
This approach does not suffer from high variance compared to (15). since both and 2o are fixed. Furthermore, we found that using (8) works well in practice (see our experiments in Section
3.2 LEARNING TO GUIDE
We present here our approach for learning guidance weights w based the self-consistency loss (8). We also provide empirical evaluation of other approaches in Section
4

---

Published as a conference paper at ICLR 2026
Guidance learning objective. For (Iu, c) po.c(20. c), sample xa Ps o(1(20), i.e, D/ M(agIo oIId). We also sample (w) Pel0,c w) i.e. sample ãn Na IfI then
[o(w) ~
(e,,()))) Tt), Ex.t);
(19)
where Ma,t and E,,, given by DDIM (see 23 in Appendix B The objective (8) be written
as
(w) s...ps
[ 2. (w) x. - (ا
(w()) (w) 11211.
(20)
where we dropped terms not depending on w, and with independent (,(w)) T. (w) P=10,c
1,(6,w))
Simplified guidance learning objective. A special case of (20) with 3 = 2 and X 0, leads to T. (w) - x. 1
Le = Lg.o(w) =
(21)
We found that (2T) was very effective but more sensitive hyperparameters than (20) This approach is however cheaper than (20) since it avoids quadratic complexity ((m²) of computing interaction
terms. We refer the reader to Appendix [ and Algorithm for more details on the use of 21).
Guidance network. The guidance weights w (e,(e.)) w(s, I c; Φ) are given by a neural network with parameters &b We use ReI.U activation at the end to prevent negative guidance weights. For more details, see Appendix (g.x(w).
We denote w = (e. and employ (6) instead of
Distribution p(s, We choose target time s to be distributed as U[ 1- 6). We define At ~ U[0 1 - s] and we let t = 8 + At. Here, Smin controls the minimal time, and ( controls how close it gets to The parameter 5 controls the distance between time-steps and we found it to be very important, see Figure 2 Even though during inference it - s| = 1 is typically small (T' is number of steps), we found that using larger å As 0.1 during training worked better in practice.
T
Empirical objective. We sample (z)), c² Ji=1 po,e from the training set and we additionally sample noise levels {s LLd.
~ U[Smin, - Ç- 6), as well as time increments {At:)% i.d. U[6, 1 - si] and we let 4 = At;. For each (ri, st), we sample m "particles" {xd) 1 i..d Ps.jo(-|zz) from the noising process 3 which defines the target samples. We also produce m Pt.jo(-(=zh) from
"particles" {5gg ) i.d.d (..(0.w)) (-1zh, q) by first sampling { i.i.d
the noising process (3) and then denoising with guidance and DDIM using (19). This defines the proposal samples. We expand loss function (20) as function of guidance network parameters
defined on the empirical batches as follows (where [0, 1] and 3 E (0, 2), see Algorithm
=
m Em ||52 (w) z,k
mm Σj || (w ) - i (w 1
(22)
Learning to guide with rewards. CFG can be used to produce samples with high reward R(ao. C) set by practitioner. Denoising with guidance from to s as described by (19), gives us an approxi- mation (0 c; w (.(s.)) of clean data. We could use it to optimize guidance weights, by defining the
R ( e: w ((s..)).)) Directly minimiz- ing this loss may lead to reward hacking (Skalse et al. 2022). Thus, we regularize this objective using L.P.A. For reward weight 7R 2 0, we optimize Loor(9
loss CR(Ø)
(x) + R (see Algorithm 4h.
4 RELATED WORK
Classifier-Free Guidance. Using guidance in the sampling of diffusion models was first investi- gated by DDhriwal and Nichol (2021). They proposed a classifier guidance method which linearly combines the unconditional score estimate and the input gradient of the log-probability of (time- varying) classifier. To avoid training such classifier on noisy training data, Ho and Salimans (2022) proposed Classifier-Free Guidance (CFG), which linearly combines conditional and unconditional denoisers. By varying the guidance weight, one is able to obtain high-quality samples. This approach has become prominent in the literature, see Adaloglou and Kaiser (2024) for recent introduction.
5

---

Published as a conference paper at ICLR 2026
Algorithm Learning to Guide
1: Input: Init. guidance parameters d; (frozen) denoiser do; data distribution po; learning rate mj
< <> 0, Smin > 0, 6 0, b.s. n, n. of particles m, À E [0, 1],3 E [0, 2). DDIM churn & E [0, 1].
2: repeat
3: Sample batch of clean data and their conditionings
{at, c² c' Lid. ~ Po,c-
4: Sample {s: i.dd U[Smin 1- 1 - ( {At;)==1 L.d. U16, 16,1-55 s;]. let t, = 8; + At,
(True process) Sample m particles { lid. Pa.j((12z0) from noising process
5:
6:
(Guided process) Sample m particles (- Pt.,o(((a))) from noising process
7:
Compute guidance weights w/ = w (e)))) and in (7) Ga5 wi) using ?
8:
Sample N (w+) ~ Ps.le..ol .0 -12 @o (:ww))) from DDIM ( with churn parameter e (Loss) Compute loss
9:
10:
(a) n -== m 2j |ig (w²) 1/2 mmm- Ejft (1/,(w")) 3 (w))
11:
Update Φ c Φ 1/ (L)
12:
until convergence
13: Output: Optimized guidance network parameters &
CFG has been extended in many different directions, some of them proposing dynamic mixing strategies to control the guidance weight throughout the sampling process (e.g. Sadat et al.
(2024a);
Kynkäänniemi et al. (2024): Wang et al. (2024); [Shen et al. (2024): Malarz et al. (2025):
Li al.
(2024); Koulischer et al. (2025a b): Xia et al. (2025); Sadat et al. (2024)); 2Zheng and Lan (2024).
Another line of research focuses on removing the need for the unconditional score component by using
either an "inferior" version of the conditional model to provide a negative guidance signal (Karras)
ct al. 2024; Addloglouu et al. 2025), by leveraging time-step information (Sadat et al.] 2025), or by
enabling the model to act its own implicit classifier (Tang et al. (025). Other approaches focus
fixing some of the limitations of classifier-fn free guidance; e.g.. Chung et al. (2025) introduced CFG++, modification to the standard CFG address off-manifold issues, while Koulischer al. (2025b)
show that the true conditional distribution can be obtained by approximating term corresponding to the derivative of Rényi divergence. Very recently, Fan al. (2025) introduced CFG-Zero+, a CFG variant for flow-matching models. It leverages an optimized scale factor and zero-init technique (skipping initial ODE steps) correct for early velocity inaccuracies, resulting in improved text-to- image/video generation. We highlight that our approach could be seamlessly combined with any other guidance technique by simply replacing the definition of the guided denoiser (7). Finally,
there exists large literature correcting CFG with Sequential Monte Carlo (Skreta et al. 2025 He
jet al. 2025) or MCMC techniques (Du al. [2033] Moufad et al., 2025 /hhag et al.) (025). These
approaches are mostly orthogonal to other improvements of CFG, and can be combined with most of the dynamic mixing strategics (i.e. Malarz et al. (2025)).
CFG has also been reinterpreted prwdictor-cooreecteeceo
in Bradley and Nakkiran (0224) and analyzed
theoretically in a variety of works (Fu et al., 2024| Wu et al. [2244] Chidambaram et al. 2024|] Kong al.] 2024 Frans et al.; (2225). Pavasovic al. (2025) have shown that while CFG can overshoot the target distribution in low dimension, it can reproduce the target distribution in high dimensions.
Concurrently to our work, PPapalampidi et al. 2025) introduced dynamic classifier guidance by select- ing the guidance weight among a list of pre-determined guidance weights using online evaluators.
Correcting distribution mismatch. Like our method, the time- approach (Xia et al. 2024) corrects the mismatch between the sampled and target conditional distributions. However, time-tuner specifically addresses backward solver discretization errors in low-NFE regimes by adjusting the denoiser's 's noise level. In contrast, our approach addresses the distributional mismatch caused by CFG to improve alignment and quality, independent of NFE.
Learning CFG weight. Azangulov et al. (2025) introduced an algorithm based on stochastic control to optimize the guddance weights. However, this algorithm is not scalable it requires guided backwards trajectories to estimate the gradient. Concurrently to us, Yehezkel et al. (2025) introduced annealing guidance scheduler that dynamically adjusts the guidance weight based on

---

Published as a conference paper at ICLR 2026
the timestep and the magnitude of the conditional noise discrepancy. The method learns adaptive policy to better balance image quality and alignment with the text prompt throughout the generation process. One of their losses is similar to an alternative approach we explored guided score matching We found that optimizing such loss led to guidance weights equal to zero.
(see Appendix C.I..
Distillation. Our method shares similarities with distillation approaches such as Diff-Instruct Luo al., (202)), Variational Score Distillation (Wang al. 2023) and Moment Matching Distillation (MMD) (Salimans et al., 2024). which distill a diffusion model by approximately minimizing the KL divergence between the distilled generator and the pretrained teacher model. One could think about our method as a form of distillation where the KL divergence is replaced by scoring rule, the pretrained teacher model is replaced by the true distribution, and the student is replaced by a guided diffusion. Our methodology could be extended to the distillation setting.
Distributional approaches for diffusion models. Our objective function was motivated by (De Bor toli et al. 2025). , where (9) is used to learn p c) to obtain distributional diffuusoon models. Our method is also related to Inductive Moment Matching (IMM) (Zhou et al. 2025), where generative model is trained by enforcing marginal consistency (1). Instead of directly optimizing for (14), their objective minimizes the distance between y>s s.(0,.w) and p. (0(0--1.w) where n is an iteration number and s < r < t is an intermediate time between s and t. However, they do not marginalize over (wo, c) ~ p as in (1 instead, they sample a batch of (ro, c) and use it for both pa (.(9..W) and ps (0,)),,,,), This means they optimize an objective similar to self-consistency
17 where they use a
batch of (Io, c) instead of single points.
5 EXPERIMENTAL RESULTS
In this section, we present our core experimental results on learning guidance weights. In Section 5.1
we provide results for image generation benchmarks ImageNett 64 x 64 (Deng et al. 2009 and
CelebA (Liu et al., 2015) with resolution 64 x 64. We provide results on text-to-image (T2))
benchmark MS-COCO 2014 (Lin et al. 2014) (10K images) at 512 x 512 resolution in Section 5.2
We also refer the reader to Appendix G for discussion about training and inference cost of our
method.
Ablations. We refer the reader to Appendix H for ablations of our method. In Appendix H.I we
provide ablations over B and m on ImageNet 64 64 and in Appendix H..2 we provide ablation over conditioning in T2I. Finally, in Appendix H.3 we provide ablation over guidance network size and architecture on T2I.
Additional results. We refer the reader to Appendix [ for additional results and comparisons. In Appendix [.] we provide comparisons clamp-linent schedule (Wang et al. (0244) on ImageNet 64 x 64, and in Appendix [.2] we add comparisons to limited interval guidance (LIG) (Kynkäänniemi al. 2024) and clamp linear schedule on T2I. Moreover, Appendix [.3] we add results on Stable Diffusion-vll -iiusiioo-ll.5 (Rombach et al.] (022b). Furthermore, we report 2D Mixture of Gaussians (MoG) results in Appendix Finally, in Appendix we provide an analysis learned guidance weights in T2I and ImageNet 64 x 64.
5.1 IMAGE GENERATION
Experimental setting. We evaluate the performance of method ImageNet 64 x 64 and CelebA with resolution 64 x 64. As evaluation metrics, we use FID (Heusel et al. (017) and Inception Score (IS) (Salimans et al. 2016). First, pretrain diffusion models ImageNet and CelebA. We then freeze these models and train guidance network w/ (,(s,t) via Algorithm We also train a variant of our method where conditioning omitted, i.e. (o.t)' On top of that, we also train the simplified l2 method using Algorithm 3 jsing the same methodology. We report metrics based on 50% samples. Please refer Appendix F for more details.
7

---

Published as a conference paper at ICLR 2026
Baselines. We report performance of the unguided model well the model with constant guidance. We further report performance of limited interval guidance (LIG) (Kynkäääniemi et al.
2024). Guidance scale and intervals were selected via grid search for the lowest FID (see Appendix
Table 1: ImageNet 64x64. We report FID and IS for different methods (the best are in bold).
Method Name
Guidance Weight
FID +
Inception Score (IS)
Baselines
Unguided Constant guidance
0
4.46
43.52
W 0.25 (t) 0.95 for t [0.2,
2.40
66.72
Limited interval guidance
2.11
71.60
Learned guidance approaches
Self-consisteney
20
1.99
73.62
(a.e)
caff-onnsistnny
20
2.07
76.7
(a.t)
l objective 21
2.09
75.93
(a,1)
le objective 21
2.10
77.55
(e.t)
The results for ImageNet 64 x 64 are given in Table and for CelebA in Table In both cases, the LIG baseline outperforms the other baselines in terms of FID and Inception Score. Our self-consistency approach (2) leads the best results, especially when the conditioning information provided. This highlights the importance of adjusting guidance weights for different conditioning.
Results.
weight
34
Limited
CrllCanc-eteryy
0.tanchl
200 (tackawun)
sprune chiccen pimpeddd
Sergeted ed ctpettian
150 lo tasual
636 garrtnanl pompomppted..p.
Figure Learned guidance weights on ImageNet 64x64. Left, guidance weights w ((t-dr,() (conditioning-agnostie for baselines as well as for self-consistency 20) and &g (2I) objectives, where dt 1/100. X-axis is time. Right, guidance weights " c,(t-dtt1)) for specific ImageNet classes.
In Figure left, we visualize the learned conditioning-agnosic guidance weights w(- with dt = 1/100, for self-consistency 20 and & (21) approaches, well for the baselines. Learned guidance weights seem to be positive on a similar interval as LIG, but the shape of the weights is quite different. In Figure right, we visualize guidance weights w (tt-att. for different classes of ImageNet, learned by self-consistency (20). First, we arbitrary choose some classes 0 (tench), 10 (brambling), 100 (black swan) and 999 (toilet tissue). Then, we visualize the ones which achieve the largest and the lowest guidance weights, i.e. 83 (prairie chicken) = arg min, [max, _ (t,(t-)).)))
and 696 (paintbrush) = arg max, _ dt).)) For the class 83 (prairie chicken), the guidance weight is zero. For the class 696 (paintbrush) it has quite different behavior compared to others, being more aggressive and positive on larger interval. Overall, this, variability highlights importance of adjusting guidance weights per conditioning.
Ablations. In Figure 2 we study impact of 5 and Smin on ImageNet 64 x 64. We always use <= = 0.01 which introduces small "safety margin" over the original diffusion model (i.e. f I-] instead of / E [0,1]) 1)). The self-consistency approach (17) is less sensitive to the parameters compared to l 21 Overall, very small 5 = 0.01 leads to worse performance compared to larger ones, motivating to train with large gaps (8 z 0.1) between and t. This finding surprising because during sampling |s |s-t] ~ 0.01 for 100 sampling steps. We hypothesize that larger gap |s s--t], provides more stable and informative gradient signal for the guidance network, which then successfully
8

---

Published as a conference paper at ICLR 2026
Table 2: CelebA 64 64. We report FID and IS for different methods (the best are in bold).
Method Name
Guidance Weight Baselines
FID |
Inception Score (IS)
Unguided
2.44
2.94
Constant guidance
W 0.01
2.45
2.94
Limited interval guidance
(1)
= 0.7 for (0.0.088
2.37
2.96
Learned guidance approaches
coffisnssstencyy
(s,t)
2.10
2.98
Self-consistency
2.28
2.97
(s.1)
& objective (11
w ((18)))
2.36
2.95
l objective 21
w (s.t)
2.33
2.95
consistency sds (i
consistency uik cral
6s obective okk.lt
b objectine wich
5,m
5=0.01
5-0.2
0=0.3
Figure 2: Ablation over 8 and Smin on ImageNet 64 x 64. On the X-axis we report values of Smin and on Y-axis we show FID. Each column denotes a method while color corresponds to value of 6.
generalizes to the small-step intervals used during inference thanks to the smoothness of the network -,(-)) The performance not very sensitive to Smin- but we found Smin 0.2 worked the best.
5.2 TEXT-TO-IMAGE GENERATION
We evaluate our method on MS COCO 2014 dataset at 512 512 resolution for text-to-image (T2I) task. We pretrain 1.05B parameter flow matching model (Lipman et al.) (023) that uses a Multimodal Diffusion Transformer backbone (BlackForestt Labss 2025). We freeze and train guidance network w Wet.r (4.1)
via Algorithm where OCLIP and denote CL.IP (Radford et al.
2021) and T5 (Raffel et al. 2020) embeddings of the text prompt. As baseline, consider manually selected guidance weight W. Instead of an empty prompt Ø for the unconditional term 77. we replace with fixed negative prompt Cneg = "blurred, blurry, disfigured, ugly, tiling, poorly drawn". This "negative guidance' setup is for both our learned model and the baseline. We also consider a setting with reward function R( R(zo. c) given by CLIP score (Hessel al. 2021) computed between image To and prompt c. We train guidance network via Algorithm 4 with 7R = 10s, We report FID (Heusel al. 2017) and CLIP Score using 10K samples. For more experimental details, see Appendix
Results. The quantitative results are summarized in Table 3 Our method outperforms unguided and guided baselines in terms of FID, which is consistent to image experiments. However, achieves lower CLIP score than guided baseline. Adding CLIP score reward leads similar CLIP score as guided baseline, but achieves lower FID. We provide qualitative results Figures (4- see Appendix A Our method generates images that are more realistic and better aligned with the text prompts. The learned guidance weights are shown Figure on the prompt.
We observe high variability depending
6 DISCUSSION
In this paper, presented approach to learn CFG weights We,(s.t) function of conditioning and times s and f, using the self-consistency condition (17). This rather strong condition is motivated by weaker marginal consistency condition (14), which satisfied by the true backwards diffusion process. Our approach yields guidance weights that improve FID on image generation tasks Ima- geNet 64 x 64 and CelebA 64 64. Our analysis reveals that guidance weights vary significantly de-

---

Published as a conference paper at ICLR 2026
Table 3: MS COCO 512 512. FID and CLIP score for different methods (the best in bold). We
report the best qualitative results for the baselines: limited interval guidance (Kynkäänniemi et al.
2024) and clamp-linear schedule (Wang et al. 2024). We perform extensive hyperparameter tuning for each of these baselines shown in Table and Table
Method Name
Guidance Weight
FID
CLIP Score 1
Baselines
Unguided
0
24.74
0.278
Constant guidance
w = 7.5
31.2
0.306
Limited interval guidance
(t) 15.0 0 for t E 0.9
24.95
0.304
Clamp-linear guidance
w = 14.0
27.14
0.305
Our approaches
Self-consistency 20 Self-consistency
18.01
0.295
20
CLIP reward
28.37
0.306
(B,6.
Prompt performing stunt skatebourd on
Prompt 2. A street with cars and construction workers working.
Prompl 3, bird is sitting on brunch among unfocused trees.
Prompt 4. wooden table topped with four white howls.
Prompt 5. A light brown borse's 's face is shown at close range.
Prompt living room furniture, fireplace, and large
window.
Prompt lange
black feathers.
Prompt 8. A small dog TV behind the words; "What did do wrong?"
Figure 3: Learned guidance weights on MS COCO 512 x 512 trained with self-consistency (00)
and CLIP reward loss. Please refer to Figures 4 and 5 for the corresponding images.
pending on the conditioning information, implying that CFG with learnable, conditioning-dependeen weights can improve conditional sampling performance.
We extended methodology text-to-image tasks with reward function given by the CLIP score. We found that our approach leads to highly variable prompt-dependent guidance weights and visually provides better prompt alignment compared baselines. Quantitatively, however, we found that the performance was close to baseline with a manually selected guidance weight function.
Future work will focus on theoretical understanding of our objective function and its guidance solutions. Moreover, we will explore alternative reward functions for better prompt-image alignment, and investigate the impact of different guidance approaches. We hope our work motivates further research into how time- and cennttioninn-depeedeeee guidance weights affect the sampled distributions.
REFERENCES
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024). Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493-500.
Adaloglou, N. and Kaiser, T. (2024). An overview of dlasifier-free guidance for diffusion models. theaisummer.co..
Adalogiou. N.. Kaiser, T., lagudin, D.,, and Kollmann, M. (2025). Guiding diffusion model using sliding windows. arXiv preprint arXiv:2411.10257.
Azangulov, I., Potaptchik, P., Li, Q., Aamari, E., Deligiannidis, G., and Rousseau, J. (2025). Adaptive diffusion guidance via stochastic optimal control. arXiv preprini 250.
BlackForestLabs (2025). Flux.1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.1577 15742.
10

---

Published as a conference paper at ICLR 2026
Bouchacourt, D., Mudigonda, P. K., and Nowozin, S. (2016). DISCO nets: Dissimilarity coefficients networks. In Advances in Neural Information Processing Systems.
Bradley, A. and Nakkiran, P. (2024). Classifier-free guidance is predictor-corrector arXiv preprint arXiv:2408. 09000.
Chidambaram, M., Gatmiry, K., Chen, S., Lee, H., and Lu, J. (2024). What does guidance do? fine-grained analysis a simple setting. In Advances Neural Information Prucessing Systems.
Chung, H., Kim, J., Park, G. Y., Nam, H., and Ye, J. C. (2025). CFG++: Manifold-constrained classi- fier free guidance for diffusion models. In International Conference on Learning Representations.
De Bortoli, V., Galashov, A., Guntupalli, J. S., Zhou, G., Murphy, K., Gretton, A., and Doucet, A. (2025). Distributional diffusion models with scoring rules. In International Conference on Machine Learning.
Deng, J., Dong. W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat GANs image synthesis. In Advances in Neural information Processing Systems.
Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein. J., Doucet, A., and Grathwohl, W. (2023). Reduce, reuse, recycle: Compositional generation with energy- based diffusion models and MCMC. In International Conference Machine Learning-
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A. Boesel, F, Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., and Rombach, R. (2024). Scaling rectified flow transformers for high-resolution image synthesis. In International Conference Machine Learning.
Fan, W., Zheng, A. Y., Yeh, R. A., and Liu, Z. (2025). Cfg-zero*: Improved classifier-free guidance for flow matching models. arXiv preprint arXiv:2503. 18886.
Frans, K., Park, S., Abbeel, P., and Levine, S. (2025). Diffusion guidance controllable policy improvement operator. arXiv preprint arXiv:2505.23445
Fu, H., Yang, Z., Wang, M., and Chen, M. (2024). Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprint arXiv:2403./ /968.
Google (2024-2025). Veo AI video generator generative Al on vertex AI. Google Cloud Documen- tation.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773.
He, J.. Hernández-Lobato.] J. M., Du, Y., and Vargas, F. (2025). RNE: plug-and-pla- framework for diffusion density estimation and inference-time control. arXiv preprint arXiv:2506.05666
Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van 7ee, M. (2024). Flax: neural network library and ecosystem for JAX.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. (2021). Clipscore: A reference-free evaluation metric for image captioning. In Empirical Methods in Natural Language Processing.
Heusel, M., Ramsauer, H., Unterthiner, T.. Nessler, B., and Hochreiter, S. (2017). GANs trained by two time-scale update rule converge local Nash equilibrium. In Advances in Neural Information Processing Systems.
Ho, Jain, A., and Abbeel. P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems.
Ho, J. and Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.

---

Published as a conference paper at ICLR 2026
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine, S. (2024). Guiding a diffusion model with bad version of itself. In Advances in Neural Information Processing Systems.
Kim, B., Hsieh, Y.G.,, Klein, M., Cuturi, M., Ye, C., Kawar, B., and Thornton, (2025). Simple reflow: Improved techniques for fast flow models. In International Conference Learning Representations.
Kingma, D., Salimans, T., Poole, B., and Ho, J. (2021). Variational diffusion models. In Advances in Neural Information Processing Systems.
Kingma, D. P. and Gao, R. (2023). Understanding diffusion objectives as the ELBO with simple data augmentation. arXiv preprint arXiv:2303.0084..
Kong, L., Du, Y., Mu, W., Neklyudov, K., De Bortoli, V., Wu, D., Wang, H., Ferber, A., Ma, Y.-A., Gomes, C. P., al. (2024). Diffusion models constrained samplers for optimization with unknown constraints. arXiv preprint arXiv:2402. 18012.
Koulischer, F., Deleu, J., Raya, G., Demeester, T., and Ambrogioni. L. (2025a). Dynamic negative guidance diffusion models. International Conference on Learning Representations.
Koulischer, F., Handke, F., Deleu, J., Demeester, T., and Ambrogioni, L. (2025b). Feedback guidance of diffusion models. arXiv preprint arXiv:2506.06085 06085.
Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. (2024). Applying guidance limited interval improves sample and distribution quality in diffusion models. In Advances in Neural information Processing Systems.
Li, T., Luo, W., Chen, Z., Ma, L., and Qi, G.-. (2024). Self-guidance: Boosting flow and diffusion generation on their own. arXiv preprint arXiv:2412.05887.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer.
Lipman, Y., Chen, T., Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow matching for generative modeling. In International Conference Learning Representations.
Liu, Z., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV).
Luo, W. Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, (2023). Diff-instruct: universal approach for transferring knowledge from pre-trained diffusion models. In Advances in Neural Information Processing Systems.
Malarz, D., Kasymov, A., Zięba, M., Tabor, J., and Spurek, P. (2025). Classifier-free guidance with adaptive scaling. arXiv preprint arXiv:2502.1057.
Moufad, B., Janati, Y., Durmus, A., Ghorbel, A., Moulines, E., and Olsson, J. (2025). Conditional diffusion models with classffierfrree Gibbs-like guidance. arXiv preprint 1rX1v:2505.21001
Papalampidi, P., Wiles, O., Ktena, Shtedritski, A., Bugliarello, E., Kajic, I., A Albuquerque, and Nematzadeh, A. (2025). Dynamic classificr-free diffusion guidance via online feedback.
Pavasovic, K. L., Verbeek, Biroli, G., and Mezard, M. (2025). Understanding classifier free guid- ance: High-dimensional theory and non-linear generalizations. arXiv preprint 077842250..07849.
Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 4195-4205.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.. Sastry, G.. Askell, A.. Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
12

---

Published as a conference paper at ICLR 2026
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022a). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022b). High-resolution image synthesis with latent diffusion models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer.
Sadat, S., Buhmann, J., Bradley, D., Hilliges, O., and Weber, R. M. (2024a). CADS: Unleashing the diversity of diffusion models through condition-annedt sampling. In International Conference on Learning Representations.
Sadat, S., Hilliges, O., and Weber, R. M. (2024b). Eliminating oversaturation and artifacts high guidance scales in diffusion models. In International Conference on Learning Representations.
Sadat, S., Kansy, M., Hilliges, O., and Weber, R. M. (2025). No training, problem: Rethinking classifier-free guidance for diffusion models. In International Conference on Learning Representa- fions.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., and Salimans, T. (2022). Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems.
Salimans, T., Goodfellow, Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training GANs. In Advances in Neural Information Processing Systems.
Salimans, T., Mensink, T., Heek, J., and Hoogeboom, E. (2024). Multistep distillation of diffusion models via moment matching. In Advances in Neural Information Processing Systems.
Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. (2013). Equivalence of distance- based and RKHS-based statistics in hypothesis testing. The Annals of Statistics, pages 2263-2291.
Shen, D., Song, G., Xue, Z., Wang, F.-Y., and Liu, Y. (2024), Rethinking the spatial inconsistency in classifier-free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Skalse, J., Howe,
R., Krasheninnikov, D., and Krueger, (2022). Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems.
Skreta, M., Akhound-Saude Sadegh, T., Ohanesian, V., Bondesan, R., Aspuru-Guzik, A., Doucet, A., Brekelmans, R., Tong, A., and Neklyudov. K. (2025). Feynman- Kac correctors in diffusion: Annealing, guidance, and product of experts. In International Conference on Machine Learning.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning,
Song, J., Meng, C., and Ermon, S. (2021a). Denoising diffusion implicit models. In International Conference on Learning Representations.
Song, Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b). Score- based generative modeling through stochastic differential equations. In International Conference Learning Representations.
13

---

Published as a conference paper at ICLR 2026
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. R. (2010). Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11(Apr):151-1161.
Székely, G. and Rizzo, M. (2004). Testing for equal distributions in high dimension. InterStat, 5.
Tang, Z., Bao, J., Chen, D., and Guo, B. (2025). Diffusion models without classifier-free guidance. arXiv preprint 2rX1v22502./215:
Wang, X., Dufour, N., Andreou, N., Cani, M.-P., Abrevaya, V. Picard, D., and Kalogeiton, V. (2024). Analysis of classifier-free guidance weight schedulers. Transactions on Machine Learning Research.
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, (2023). Prolificdreamer: high- fidelity and diverse text-to-3d generation with variational score distillation. In Advances in Neural Information Processing Systems.
Watson, L., Juergens, D., Bennett, N. R., Trippe, L.,, Yim, J., Eisenach, E., Ahern, W., Borst, A. J., Ragotte, J., Milles, L. F., et al. (2023). De novo design of protein structure and function with REdiffusion. Nature, 620(7976); 1089-11100
Wu, Y., Chen, M., Li, Z., Wang, M., and Wei, Y. (2024). Theoretical insights for diffusion guidance: A case study for Gaussian mixture models. In International Conference Machine Learning.
Xia, M., Shen, Y., Lei, C., Zhou, Y., Zhao, D., Yi, R., Wang, W., and Liu, Y.-J. (2024). Towards more accurate diffusion model acceleration with a timestep tuner. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5736-5745.
Xia, M., Xue, N., Shen, Y., Yi, R., Gong, T., and Liu, Y.-. (2025). Rectified diffusion guidance for conditional generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13371-
Yehezkel, S., Dahary, O., Voynov, A., and Cohen-Or, D. (2025). Navigating with annealing guidance scale in diffusion space. arXiv preprint arXi::2506.2411
Zhang, X., Lin, H., Ye, H., Zou, J., Ma, J., Liang, Y., and Du, Y. (2025). Inference-time scaling of diffusion models through classical search. arXiv preprint arXiv:2505.23614
Zheng, C. and Lan, Y. (2024). Characteristic guidance: Non-linear correction for diffusion model at large guidance scale. In International Conference on Machine Learning.
Zhou, L., Ermon, S., and Song, J. (2025). Inductive moment matching. In International Conference Machine Learning
14

---

Published as a conference paper at ICLR 2026
ORGANIZATION OF THE APPENDIX
present the image samples for text-to-image experiment, see Section 5 Then, in Appendix B we briefly cover the DDIM framework. Following that, in Appendix C we discuss alternative approaches for learning guidance weights. In Appendix D we discuss the simple approach based on & objective (2I) and in Appendix [ we discuss the extension of our method to reward- guided setting. Appendix G we discuss training and inference costs for the guidance network. Moreover, in Appendix F we cover all the experimental details and in Appendix H we present additional experimental results. Finally, Appendix ! weights in T21 and ImageNet 64 x 64.
In Appendix A
we provide analysis of the learned guidance
A TEXT TO IMAGE SAMPLES
We provide qualitative results for the text-to-image experiments in Figure Figure and in Figure
B DDIM FRAMEWORK
Diffusion models. The goal of conditional diffusion models to sample from target conditional distribution Po (z/c) on Rd, which we denote as Poje- Here c is some user-specified conditioning signal (i.e., class label or a text prompt) and p(c) is conditioning distribution.
We adopt here the Denoising Diffusion Implicit Models (DDIM) framework of Song et (202 la Let 2t,- ~ Pole and define the process Ft,iy = (Zt, Zin) by
p (1to)= p( (3t)
II
p(ra, [Ttg Ttuns
x=1
with 0 = to ... ty 1. For 0 < we
p(z 21) Maxsi x1), Eg,e),
The mean and covariance of p( to, .1) are given by
(e,e xz) = (s2 t) =--=))ro..)a + a1 -0111-pp r222(8) - -11 (1- 11--- ()) ())t)
Σg,t = 01 (52
- (1)
(23)
with ri.j (s, (an/as)² (o3/o?)²; see Song et al. 2021a and Appendix F in (e Bortoli al. 2025). The parameter [0, 1] in (23 is churn parameter which interpolates between deterministic process (e 0) and a stochastic one (e 1). This ensures that for any / E [0. 1].
p(ze/zo) (xx 0, ofid).
for au, ae such that ag = 01 and a1 σo 0. In particular, this guarantees that p(z1 (2o) (211; O,ld).
The data Ito Puje is generated by sampling Ziw (d) and Tre p(-1 for N 1.... O, where for 0 S <<tl
p(x,111, o p(z,, ( p(z, wx)ppp/ll 2 24)d2
C ALTERNATIVE APPROACHES FOR LEARNING GUIDANCE
We describe the alternative approaches for learning guidance weights and discuss the their potential downfalls.
15

---

Published as a conference paper at ICLR 2026
(a) Man performing stunt on skateboard on road.
(b) street with cars and construction workers working.
(c) A bird is sitting on branch among unfocused trees.
(d) wooden table topped with four white bowls.
Figure T2I Results on MS-COCO. Part (left-to-right) We provide results of images generated from the given text prompt without CFG, with CFG w = 7.5, our method with self-consistency loss and our method with self-consistency loss and CL.P score reward.
C.I GUIDED SCORE MATCHING
The most straightforward approach for learning guidance weights is to simply use guided denoiser the score matching. This leads to the following objective
((Φ) - f. ((t) (mo)) [| to (xi, c; wl,
(24)
where omit the dependence of the guidance weights i.c, w (((a)) whe However, this approach is doomed to fail, because convergence, the solution to (44 satisfies the following property
(o G; w&) Re E[
(25)
16

---

Published as a conference paper at ICLR 2026
(a) A light brown horse's face is shown at close range.
(b) living room furniture, a fireplace, and large scenic window.
The large bird has red face and black feathers.
Vblión a Io m VAT WOT DOT WO WROIN W W'alhatto: Do hi What is Do to it Worthoig?
(d) A small dog TV behind the words What Did I Do Wrong?
(e) A walking under black open umbrella.
Figure 5: T2I Results on MS-COCO. Part (Randomly selected). (left-to-right) We provide results of images generated from the given text prompt without CFG, with CFG 7.5, our method with self-consistency loss and our method with self-consistency loss and CLIP score reward.
when 24 (x (z)) sample from noising process satisfied by the unguided score, when v, Proo(= (aa), i.e.
However, the condition 25 is also
fo(#,, c) Rs E[prolaee
(26)
17

---

Published as a conference paper at ICLR 2026
(a) A small cutting board and knife with a cut apple.
(b) hand holding for llama chew.
A with wrench turning off fire hydrant.
(d) An animal that in the themselves.
(c) A row of red and blue luggage sitting at an airport.
Figure 6: Additional T2I Results MS-COCO. Part 3. (left-to-right) We provide results of images generated from the given text prompt without CFG, with CFG w = 7.5, our method with self-consistency loss and method with self-consistency loss and CLIP score reward.
18

---

Published as a conference paper at ICLR 2026
(a) Two birds sit on top a parked car.
(b) wearing Santa suit including Santa hat.
Lilies in the foot of white canopy hed.
(d) A bathroom with marble counter under two giant mirrors.
(c) A cat laying top of shoes floor.
Figure 7: Additional T2I Results MS-COCO, Part 4. (left-to-right) We provide results of images generated from the given text prompt without CFG, with CFG w = 7.5, our method with self-consistency loss and our method with self-consistency loss and CLIP score reward.
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 CFG version of DDIM (sampling)
Require:
{4x [= with 40 = 0 ... < IN 1, churn parameter e, conditioning c, guiding weights
w ())) unconditional and conditional pretrained denoisers Zo (Tz, Ø), #0(
Sample Sty (O, Id)
for E {N 1..N-..... 1,....... do
Sample ~ I((OId))
Set current guidance weight w (e,(tx.le+1)
Compute Ωo =o((ee+1, Ft see n, (:w) using
Compute MaNNN (#o, Feety Eeetrrr using 23
Sample Ftz (H12 (0o: Binci Exe
Return: Z0
By equality of rhs of 260 and of 25 we have Va, ~ Pejo moo), e p(c), ~ po)o(- [0,
c (z (
èe (z c) = w (Be (2 c) 20 2)) =
We then assume that (a c) it (zz, c) (0 2) F 0. It is reasonable assumption since if Aa (2,, c) = O, the CFG would not have had any effect. This implies that
- 0, Vc p(c)p()),EE [0,
(27)
The only regime where 27 will not hold when
(o c) # B[zo| c]
In such case, one could still find guidance weights for (25) to hold. But this is regime of a very
under-trained model. In our image experiments (see Section 5}. we initially tried using this approach
and found that it was leading to zero guidance weights 27
Connection to backward KT., Another explanation for the behavior above that as we train 24 we only observe Dt p(z, æo). The form of (24) comes from optimizing backward KL
J. / | to c;
KL KL |
(28)
where P] is the distribution of all the trajectories obtained by running guided backwards diffusion, and Pl is the distribution of all the trajectories from the true diffusion process.
When use CFG during sampling, actually may observe noisy samples å which do not coincide with any It Pajo form the noising process (3. This hints that instead of backwards KL (8)). we
should actually aim optimize the forward KL
KL KL [Po.1]]
(29)
An approach based on optimizing (29 was considered in Azangulov et al. (2025). ). This approach, however, is very computationally expensive since it requires sampling full guided trajectory from the guided backwards process and then compute the gradients on it.
C.2 CONDITIONAL CONSISTENCY
Here, we describe an approach similar to marginul-commsisenncy [4), but which however could be more effective in practice in principle due to lower variance. We fix conditioning and we denote by
(.(w.) p Pajo.c
(xrg (rol c)
(9.w) Paje,e (ar (x)
which is the term under integral in 12 depending on (æo, e). We now consider conditional consistency condition where we fix c and marginalize over 20
(,(0,w)) Paj0.c (x (2ooe t.(0,w) (Ix(c) Rs (sle(Fslc) P pww.ottllooppoo.coolllllde
20

---

Published as a conference paper at ICLR 2026
The main difference with [4 is that fixed conditioning We could similar loss
(.(0w) Co(w) Bo=pp.(s.))pp(s.) [MMD (B.M (P (+lc),
which can be optimized by drawing conditional samples zo/c. While this approach is expected to have a lower variance than (15), since we do not need to sample it requires to be able to produce conditional samples which is not always available in practice during training (i.e., in text-to-image
applications). In practice, also found that it did not lead to good results. We hypothesize that it is due to still quite a high variance of the gradients of (W))).
D L2 APPROACH FOR LEARNING GUIDANCE
We describe here the algorithm based on the la objective function
21
(w) E (mo.o) (o..s)))
E [ xz x=112.1
(ro s..
[w]] 11
Trace Trace (C)(() Trace(Cov(z.)) (Tr,, (C,)(,,,,,,
So, by minimizing this loss function, we not only attempt to match on averages the means āي (w) while minimizing the expected sum of variances of the component of 2. (w) and maximizing the of the covariance terms between Ta and (w)).
Empirical objective. We sample (x) c' ) i..d.
Po,e from the training set and we additionally
sample noise levels {s:]i=1 iid. U[Smin, 1-{-8]. 6). as well time increments [ ii.d U(6 1-
< [s] and we let 4 = 8i + Ak. For each (ar si), we sample xs ~ Po.jo (o.1o(11a0) from the noising
process which defines the targer samples. We then also produce 3.. (w²) ~ Peel0,c. (+1rr ci)
ti,(0,w))
by first sampling ãt. 1.i.d
(-(1))) from the noising process (3) and then denoising with guidance and DDIM using (19). This defines the proposal samples. We expand loss function (21) function
of guidance network parameters defined on the empirical batches as follows
1
Les (x)
[1 (w xx,112]
The full algorithm is given in Algorithm
Algorithm 3 Learning to Guide with &2 loss
1: Input: Init. guidance parameters øi (frozen) denoiser beg data distribution Po; learning rate n); 0, Smin 0, 0, b.s. n, DDIM churn E [0.
2: repeat
3: Sample batch clean data and their conditionings {ah, of I
Lid.
Po.c-
4: Sample {s iid. 2[[Smin: 1- 5). {At)) iidd 2(55 C s;), let s; + At, (True process) Sample Is, Ps.jo(-lzn) from noising process
5:
6:
(Guided process) Sample ã., Pt.jo(+lazf from noising process
7:
Compute guidance weights W; = w (e,ee)) and to (ã, G3 w.) using
8:
Sample &.,(w") (w ~ Ps..lt.o(- (āt,, (0 ((ww)) from DDIM (2) with churn parameter E (Loss) Compute loss
9:
10:
Le, (Ø) = >n [[|@ (w)r Fes
Update & < Φ - v/V 2
12: until convergence
13: Output: Optimized guidance network parameters ø
21

---

Published as a conference paper at ICLR 2026
E LEARNING TO GUIDE WITH REWARDS
In this section, describe how our guidance learning approach is extended to a setting, where reward function R(zo, c) available, see Algorithm below.
Algorithm 4 Learning Guide with reward
1: Input: Init. guidance parameters d; (frozen) denoiser dog data distribution poi learning rate n)
C > 0, Smin > 0, 5 > O, b.s. n, n. of particles m, À E [0, 1],8 E 2), DDIM churn E E [0. 1]. Reward function R(zo: c), reward loss weight R/R 0.
2: repeat
fat e Lid. li=1
3: Sample batch of clean data and their conditionings
Po.c-
4: Sample {= L.dd U[Smin+ 1- Ç -- 6]. {A4))=
i..d. U16, 1 - s:). let 1, s; Al;
5: (True process) Sample m particles far pm iidd
Ps-1((122)) from noising process 3
6:
(Guided process) Sample m particles { J ( from noising process
7:
Compute guidance weights W, = wa (.), and žn (0), a; wi) using (7
Sample T (w ~ P...t..o (0,3%,,, ( o4 w:w) from DDIM ( with churn parameter
8:
9:
(Loss) Compute loss
10:
(o) = 1/4 Σi=1
مAأ= |ل (w 12, mmmm-) Esee 1 (w+) - 2tt (w))
(Reward loss) Compute reward loss
12:
CR(() = nm Σx Σj= Ri, ez wi
13:
(Total loss)
Compute total loss
14:
Eior (Ø) = (Ø) + *RC((0)
15:
Update o + Φ- V ( until convergence
17:
Output: Optimized guidance network parameters &
F EXPERIMENTAL DETAILS
Noise process. and 0; = f.
For all the experiments, we use recitifed flow noise process where de = (1 1)
Guidance network architecture for MoG, ImageNet and CelebA. For guidance network w(s, t, c; Φ) = w (c.(e.)) we use a layers MLP with hidden size of 64 followed by an output layer transforming it to a dimension 1. As an activation we use "GeLU". We also apply a "ReLU" acti- vation after the last layer of the guidance network to ensure that the gudiance weights are non-negative. For 2D experiments, we do not use ReLU activation and allow the weights to be negative.
The time which is processed by the weight network first converted to log SNR, i.e. s - log SNR(s) and log SNR (L). We do not use sinusoidal embedding typically is done with diffusion models, since we found that it worked significantly worse. Then the two time-steps are concatenated and passed through to 2-layer MLP with hidden dimension 256 and an output dimension 512 and GeL.U activation. We also use dropout with rate 0.3 in the middle of this MLP.
2D MoG. We use 4 layer MLP with hidden dimension 64 and GeLU activation for the denoiser network. The time for the backbone, is first embedded into log SNR and then we use sinusoidal embedding dimension 128. We use adam optimizer with learning rate 10-4 and batch size 128. We use maximum norm clipping for Adam to 1. When we do pretraining the well-trained regime, we use 10k iterations, and when we do pretraining in the under-trained regime, we use 250 iterations. We train guidance network for 1000 iterations, with learning rate 5e - 4. norm clipping and Adam optimizer. We use 32 particles. We use batch size 128. We use backwards DDIM (23) with E = 1 in order to sample x, with guidance (19). To produce Fig. 15 and Fig, [6] we sample data with velocity sampler with 10 steps. We 4096 samples in total.
22

---

Published as a conference paper at ICLR 2026
ImageNet. We pretrain the U-Net model (Ronneberger et al., 2015) similar to DDPM (Ho et al. 2020) which has channel dimension 192, 3 residual blocks, channel multiplier (1, 2, 3, 4). attention which is applied with (False, False, True, False) residual blocks (whenever it is True,
use attention), dropout which is applied at (False, True, True, True) residual blocks (whenever it is True, we use dropout). The dropout rate is 0.1. We also use sigmoid weighting (Kingma and Gao 2023) with bias b = 2. The model predicts the velocity and the loss is based on velocity. We train it with batch size 128 for 7M iterations. We use Adam optimizer with learning rate le- and norm clipping 1. We use EMA decay of 0.9999.
For training guidance we reload the pretrained diffusion model with EMA parameters and freeze it. We train the guidance network for 100K iterations with batch size B = 256. We use m = 4 number of particles and / 1.75, À = parameters. We use learning rate 1 - with adam optimizer and norm clipping 1. We use EMA decay of 0.9999. For self-consistency loss (20). in order to produce We sweep over
(,( in 119. we use DDIM sampler (33 with &
[0.01, 0.2, 0.3). Smin E [O.OI, 0.1, 0.2, 0.3] parameters. We use = le - 2 which is the same parameter as the safety parameter for the original diffusion model. This safety parameter in the original diffusion model just controls the time interval to be (C. 1 - () instead of (0, 1), for numerical stability. To sample data, we use DDIM (9) with E = for all the experiments. During training, we track FID on small subset of 2048 images. We observed that the performance over time could be unstable for some hyperparameters, therefore we use this training-time FID to keep the best checkpoint for every hyperparameter. We report the performance for the best hyperparameters based on FID, we use 50k: samples for FID computation. On top of that, we also train the simplified &g method using Algorithm 3 using the same methodology, except that in order produce ār, (w) in 19). we use DDIM sampler (3) but with e = 0, since it led to a better empirical performance.
CelebA. We pretrain the U-Net model (Ronneherger et al. 2015) similar to DDPM et
et al. 2020)
which has channel dimension 256. 2 residual blocks. channel multiplier (1, 2, 2, 2). attention which is applied with (False, True, False, False) residual blocks (whenever it is True, we use attention), dropout which is applied at (False, True, True, True) residual blocks (whenever it is True, we use dropout). The dropout rate is 0.2. We also use sigmoid weighting (Kingma and Gao 2023) with bias 6 = We do not transform time to log SNR for time embedding in the diffusion network. The model predicts the xp and the loss is based on velocity. We train it with batch size 128 for 300K iterations. We use Adam optimizer with learning rate le 4 and norm clipping 1. We use EMA decay of 0.9999.
For training guidance we reload the pretrained diffusion model with EMA parameters and freeze it. We train the guidance network for 100K iterations with batch size B 256. We use = 4 number of particles and / = 1.75, x = parameters. We use learning rate 1e - with adam optimizer and norm clipping 1. We use EMA decay of 0.9999. For ceff-consistency loss (20). in order to produce (.(W) in (19) we use DDIM sampler 23) with È = 1. We sweep over d [0.01, 0.1, 0.2, 0.3], Smin E (O.0T, 0.1, 0.2, 0.3) parameters. We use Ç = le 2 which is the same parameter as the safety parameter for the original diffusion model. This safety parameter in the original diffusion model just controls the time interval to be (ç.1- instead of (0, 1). for numerical stability. To sample data, we use DDIM with & 0 for all the experiments. During training, we track FID on a small subset of 2048 images. We observed that the performance over time could be unstable for some hyperparameters. therefore we use this training- time FID to keep the best checkpoint for every hyperparameter. We report the performance for the best hyperparameters based on FID, we use 50k samples for FID computation. On top of that, we also train the simplified &s method using Algorithm 3 Jusing the same methodology, except that in order to produce (w(w) in 19,, we use DDIM sampler (23) but with = 0, since it led to better empirical performance.
Guidance network architecture for MS-COCO. For guidance network w(s, t, CCLIP, Crs; d)
we (ccue:iss.c..)) we use a layers MLP with hidden size of 512 followed by an output layer transforming it to a dimension 1. As an activation we use "GeLU", We also apply a "ReLU" activation after the last layer of the guidance network to ensure that the guidance weights are non-negative.
The time steps t and s are each processed by separate networks. We do not use sinusoidal embedding to get the time embeddings. We also do not convert time to log SNR as we did for ImageNet and CelebA. Each of the two time-steps are passed through to a 2- MLP with hidden dimension and output dimension of 256 and SiLU activation. The two timestep embeddings are then concatenated.
23

---

Published as a conference paper at ICLR 2026
We also pass CLIP and T5 embeddings through similar MLP with output dimension of 256 and 512 respectively. Both time embedding and CLIP embedding are concatenated to a dimension 512 and then concatenated with the text embedding. This is passed through the MLP layers described above.
MS-COCO. We pretrain a 1.05B parameter Flux text-to-image backbone (BlackForestLaass 2025 which uses an architecture based on Esser et al. (2024) and consists of 30 Diffusion Transformes (DiT) blocks (Peebles and Xie )023) and 15 MMDiT blocks (Esser et al. 2024). We do not use any weighting, nor do we transform time with logSNR. The model is trained with the standard flow matching loss and predicts velocity. The CLIP embeddings used for training the model are extracted from the standard ViT-L/14 CLIP transformer architecture. The T5 embeddings are extracted from the standard T5-XXL architecture.
We load the pretrained flow matching model with EMA parameters and freeze it to train the guidance network. We train the guidance network for 50K iterations with a batch size of 256. We set the number of particles
m 4, B 1.75 and X 1.0. We use learning rate of Se 5 with Adam optimizer. We set the Adam parameters 3 = (0.9, 0.999), = 1e 8, and weight decay=0.01 For self-consistency loss (0)). in order produce .. (w) in (19
DDIM sampler (33 with 0. ee 2 while training the model. To sample
We set = 0.2, Smin = 0.01 and safety parameter
data from flow matching model, use the standard Euler ODE solve the time interval [0, and use 128 sampling steps. To select the best checkpoint, we tracked CLIP Score over 3K samples and then used the checkpoint corresponding the best CLIP Score on this subset to compute CLIP Score over 10K samples.
G TRAINING AND INFERENCE COST OF GUIDANCE NETWORK
Training the guidance network involves training a lightweight MLP with a few layers. The cost of training it is insignificant compared to the cost training a diffusion model. For example, in train U-Net with 260 M parameters for around 7M iterations with batch size of 128, while the guidance network with 600 K parameters trained for up to 100 with batch size 256. In text-to-image, the model has about 1B parameters and is trained for 1.6M steps with batch size of 2048, while guidance network has about 6M parameters which we train for 50K iterations with batch size 256.
ImageNet experiments,
At inference, we use a guidance network to produce a scalar w which involves evaluating the MLP on precomputed (t, s) and conditioning embeddings c. The computation cost is minimal compared to unrolling the diffusion model.
H ABLATIONS
In this section, we provide ablations of our method. We provide an ablation over 3 and m on ImageNet 64 x Appendix H.I In Appendix H.. we provide ablation of the conditioning signal in T2I signal, and in Appendix H.3 we demonstrate the impact of guidance network size and
architecture on the performance.
H.1 ABLATION OVER 3 AND m FOR SELF-CONSISTENCY LOSS
We ablation study over the exponent applied to the L2 norm in 20 and m, the number
of particles used for the MMD computation for the self-consistency loss in 22 The ablation was
conducted ImageNet 64 x 64. The results are given in Table and in Table We that the performance is not very sensitive to m, we see that m >= 4 works well. As for 3. we see that values [1.1.75] lead good results.
H.2 CHOICE OF MIP CONDITIONING ON T2I
We investigate the effects of the choice of MLP conditioning for T2I task with our method. We find that T5-only embeddings gets slightly lower CLIP score than the other two conditioning configuration which is CLIP-only, and both T5 and CLIP, but it gets slightly better FID score. We provide qualitative
24

---

Published as a conference paper at ICLR 2026
Table 4: Ablation over 3 on ImageNet 64
64
B
FID
IS
0.1
2.07
78.18 18
0.5
2.04
77.72
1.0
1.98
73.33
1.5
2.00
71.17
1.75
1.99
73.69
Table 5: Ablation over m on ImageNet 64 x
m
FID
IS
2
2.00
70.66
4
1.99
73.69
8
1.99
73.02
16
2.00
77.11
results in Fig. and quantitative results in Table All the experiments use the same hyperparameters for the model architecture and training. Qualitatively, we find that in most of the cases, the images look very similar, however, some cases, CLIP conditioning is helpful get better text-aligned images.
Table MS COCO 512 512. FID and CLIP score for different ML.P condtioning (the best are in bold).
Method Name
Guidance Weight
FID
CLIP Score f
Self-consistency
20
wh (ecureiss.t.. we Cccup,oss w (pur((a)))
18.01
0.295
Self-consistency
20
CLIP Score reward
28.37
0.306
Self-consistency
20
CLIP Score reward
28.63
0.306
Self-consistency
20
CLIP Score reward
w Ogs.iaa))
28.28
0.305
H.3 GUIDANCE NETWORK SIZE AND ARCHITECTURE
Using T2I testbed, we ran two ablations explore design choices of MLP. First, we increased the size of the Dense layers in the MLP from 128 to 1024. This larger MLP has 12M parameters and we refer this MIP MI.--Wis W-Wde. Second, we also introduced residual connections in the above MI.P, We refer to this MIP architecture Residual MLP-Wide We summarize the results below in Table do not observe significant improvements metrics upon increasing the number of parameters.
I ADDITIONAL RESULTS
In this section, we provide additional experimental results and comparisons. add comparison to clamp-linear schedule (Wang et al. 2024) on ImageNet 64 x 64 in Appendix [.] Moreover, in Appendix L.2] we compare our approach to limited interval guidance and clamp linear schedule in
T2I regime. Furthermore, add an additional experiment using Stable-Diffusion-v1. (Rombach
et al. 2022b model in Appendix
L.3 Finally, we report 2-d results in Appendix [4
1.1 COMPARISON TO CLAMP-LINEAR IMAGENET 64 64
We compare
method to the clamp-linear schedule Wang al.) 2024). Wt max(c, = ). on ImageNet 64 64. We sweep over parameter Wo (0.05, 0.1, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0)
25

---

Published as a conference paper at ICLR 2026
(a) A blue vase with yellow flowers sits on a window sill.
(b) A group of two dogs sitting in front of two plates of food.
(c) A giraffe eating grass over a rock wall.
(d) Stuffed teddy bear sitting next to garbage can on the side the road.
Figure 8: Choice of MLP conditioning, (left-to-right) Images generated with our method with (left)
both T5 and CLIP conditioning, (middle) CLIP conditioning, (right) T5 conditioning in the MLP. In some cases, we find that CLIP only conditioning slightly better than having both T5 and CLIP conditioning, T5-only conditioning.
26

---

Published as a conference paper at ICLR 2026
MLP
Dense layer size
Network size
FID 10K
CLIP 10K
MLP-Small
128
6M
28.37
0.306
MLP- P-dd
1024
12M
28.41
0.306
Residual MLP-Wide
1024
12M
28.68
0.306
Table 7: Comparison of MLP model architectures and performance.
and wi E {1.05, 1.2, 1.3, 1,5, 2.0, E {0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 1.0, 1.5, 2.5..
5.0. as well as over clamp parameter
We report the results in table below. Our method is the same as the one reported in Table i.c.. self consistency with all the conditioning information. We also report performance of constant guidance and limited-interval guidance. Results are given in Table We observe that clamp-linear works worse than our method and baselines.
Table 8: Comparison to different methods on ImageNet 64 64
Method
FID
IS
Ours
1.99
73.62
Clamp-linear
2.24
76.34
LIG
2.11
71.60
Constant guidance
2.40
66.72
1.2 COMPARISON AGAINST BASELINE METHODS ON MS COCO 512 512
We compare our method against classifier free guidance (Ho and Salimans, 2022). limited interval guidance (Kynkäänniemi et al. 2024). and clamp-linear schedule (Wang al. 2024). For each of these baseline methods, we perform extensive hyperparameter tuning as shown in as shown in TO and Table select images from the hyperparameter setting that results in the best metrics in terms of CLIP Score. We find that our method has comparable or better images compared to also plot the
these baselines in many cases shown in Figs. 9 to 13 Along with the images, guidance weight schedules obtained from our model and the baseline methods. We find that our guidance weight schedule can be similar to those of clamp-linear schedule in some cases such as Figs. | and 13 Quantitatively, we find that both the methods perform worse compared to our method and classifier-free guidance terms of CLIP score.
Table MS COCO 512 x 512. FID and CLIP score for limited interval guidance Kynkäääniemi et al. 2024) (the best are in bold).
Guidance Weight
Guidance Interval
FID +
CLIP Score f
7.5
(0.1, 0.9)
23.86
0.3033
7.5
(0.1, 0.2
24.09
0.3033
7.5
00.2, 0.9
18.89
0.2979
7.5
(0.2, 0.8
18.96
0.2979
15.0
0.1
24.95
0.3042
15.0
0.1,
25.10
0.3041
15.0
0.2, 0.9
20.58
0,2992
15.0
0.2, 0.8
20.57
0.2992
27

---

Published as a conference paper at ICLR 2026
Table 10; MS COCO 512 512. FID and CLIP score for clamp-linear guidance schedule wy 2024) (the best are in bold).
max (Co, 2wae
Wang et al.
Guidance Weight
Co
FID
CLIP Score f
7.5
1.0
21.99
0.3033
10
1.0
23.17
0.3039
14.0
1.0
24.15
0.3045
14.0
2.0
24.75
0.3049
14.0
4.0
27.14
0.3055
16.0
1.0
24.55
0.3047
CFG
(b) Clamp-lincar guidance
(c) Limited interval guidance
(d) Ours
Cuidence weight wis 3.Er Onc $
prompt Two b twn bears of somb macks their pen
000
02
066
68
(c) Plot of Guidance weights

---

Published as a conference paper at ICLR 2026
(a) CFG w == 7.5
(b) Clamp-linear guidance
(c) Limited interval guidance
(d) Ours
Cuidencce weight d1.
pranps An sid brick bu buiding along side o ruvor
20-
Clamp
Time
(e) Plot of Guidance weights
(f) Text prompt: An old brick building along side a river.
Figure 10: T21 Results on MS-COCO.. We provide results of images generated from the given text prompt with (first row left) CFG w 7.5, (first row right) clamp linear, (second row -left) limited interval, and (second row right) our method with self-consistency loss and CLIP score reward.
29

---

Published as a conference paper at ICLR 2026
(a) CFG w = 7.5
(b) Clamp-linear guidance
(c) Limited interval guidance
(d) Ours
Cuidencce weightt dt, prompt An elophant walks through the pross ficid.
Dur mechod
CTC 7.5
Incu
8-
Time
(e) Plot of Guidance weights
Text prompt: elephant walks through grass of field.
T21 Results on MS-COCO.. We provide results of images generated from the given text prompt with (first row left) CFG w 7.5, (first row right) clamp linear, (second row -left) limited interval, and (second row right) our method with self-consistency loss and CLIP score reward.
Figure 11:
30

---

Published as a conference paper at ICLR 2026
(a) CFG 7.5
(b) Clamp- guidance
72
(c) Limited interval guidance
(d) Ours
Cuidence weight #1.
aromot A troin travel and snow coverce track.
8-
Clamp
Time
(e) Plot of Guidance weights
(I) Text prompt: train traveling on snow covered track.
Figure 12: T21 Results on MS-COCO.. We provide results of images generated from the given text prompt with (first row left) CFG w 7.5, (first row right) clamp linear, (second row -left) limited interval, and (second row right) our method with self-consistency loss and CLIP score reward.
31

---

Published as a conference paper at ICLR 2026
(a) CFG 7.5
(b) Clamp-linear guidance
(c) Limited interval guidance
(d) Ours
Cuidence weightt dl,
prompt Man hald pape heort on teachbrush, back cround out focus
Clamp
Time
(e) Plot of Guidance weights
(I) Text prompt: Man hold paper heart on toothbrush, back ground out of focus.
Figure 13: T21 Results on MS-COCO.. We provide results of images generated from the given text prompt with (first row left) CFG w 7.5, (first row right) clamp linear, (second row lefft) limited interval, and (second row right) our method with self-consistency loss and CLIP score reward.
32

---

Published as a conference paper at ICLR 2026
Original data
Well-trained model
Under-trained model
Figure 14:
Mixture of Gaussians data. Red cross corresponds to the locations of the mean, while the numbers denote respective numbering of mixture components. The blue color corresponds to the data, while green color corresponds to the samples from diffusion model.
1.3 EXPERIMENTS WITH STABLE DIFFUSION-VI.5
We run experiment using Stable Diffusion-vl.55 55 (SD-v1.5) (Rombach et al. 2022b) using a checkpoint from Hugging Face with Flax (Heek al. 2024 backend. We use the (same guidance network architecture, same dataset, same batch size, same hyperparameters) as in T2I experiments reported in the main section of the paper, see Appendix [ for more details. However, we do not use 7'5 conditioning (since it is not used in SD-v1.5) we only use CLIP. Moreover, we use empty prompt instead of negative prompt, as we saw that constant guidance baseline led to better CLIP score with empty prompt (0.313) compared to negative prompt (0.304).
experimental setting
We train guidance network w w (a.t) via Algorithm where CCLPP is CLIP Radford et al. 202I) embedding and we additionally pass CCLIPseq which corresponds to non-pooled hidden state of CLIP, it has sequence dimension whereas CCLIP does not. Simiarly to the main paper, we also use reward function R(2o- c) given by CLIP score (Hessel et al., 2021 computed between image To and prompt We train guidance network via Algorithm 4 with YR 106 (higher than for T2I runs in the main section of the paper). We report FID (Heusel et al. 2017) and CL.IP Score using 10K samples. We train both variants for 40k iterations, and report the performance the end of the training. As baselines, we use constant guidance with w = 0 and w = 7.5.
The results are given in Table II that unlike for the results on MMDiT in the main section of the paper, we achieve a lower CLIP score compared to the constant guidance baseline. Drawing on our findings from ImageNet, we hypothesize that the performance of our method could be further improved by conducting hyperparameter sweep for the distribution p(s, t), rather than reusing the parameters from the MMDiT experiments.
We see that our method consistently leads lower FID. We found
Method name
CLIP Score
FID
Constant guidance w (
0.271
33.203
Constant guidance w
7.5
0.313
23.479
Self-consistency
20 acu (s.1)
0.294
21.670
Self-consistency
20
cam.-ausen CLIP reward
0.311
19.36
Table 11: Stable Diffusion-vl.55 results.
1.4 2D MIXTURE OF GAUSSIANS (MoG)
We consider a components MoG with the means (10, 10), (-10, 10), (10, 100) and (-10, 10), well the diagonal covariances "Id where the corresponding variances o² are equal to 5, 1, 1, The data is visualized in Figure [4] left.
We study two regimes, well-trained and under-trained. In well-trained regime, the diffusion model is trained for long enough in order to produce the samples close to the training distribution Poje- while in under-trained, it is trained for small amount of iterations and produces samples far away from the training distribution. See Figure [4] center and right for corresponding samples. Please refer to Appendix for all experimental details.
33

---

Published as a conference paper at ICLR 2026
We train w (s..)) with Algorithm 1 The results are provided in Figure 15 In Figure 15 A, visualize the MMD between samples from data distributions and samples produced by diffusion model with different constant guidance weights w. In the well-trained regime, as we increase the guidance weight w, it degrades the MMDD, because pushes the samples away from the distribution. What is intriguing is that in under-trained regime, MMD decreases as the guidance weight w increases, achieving the lowest value with the learned guidance weight w (c.(s.t)' This finding highlights one of main message CFG can be used to correct the mismatch of the sampled distribution and the data distribution. In Figure 15 B, we visualize the learned w dt) where dt time discretization. In the well-trained regime, the learned guidance weights tend to be close to O, while in under- trained regime, the learned guidance weights reach quite high values. In Figure 16 we visualize the corresponding samples. This finding highlights that in an under trained regime, high guidance weights are required.
B
HND We Berwied
HHC - lintectriieed
a 1.dd tranid
S So ned
stnn
Guidance weight 6
Guidance weight w
Time
Time
Figure 15: A: MMD between samples (4096) from data distribution and samples produced by diffusion model with different guidance weight. Blue dashed line denotes the performance of unguided model, while brown dashed line denotes the performance of a model guided with learned w d)(.-dr.
B: learned guidance weight w dt.t-dt. from Algorithm for different mixture components.
Under-trained
Well-trained model
Original Data
Guided wg s.0
Unguided
Figure 16: Samples with and without guidance. In blue, we show the original data, in green we show samples from unguided model while in purple show samples from the guided model with learned guidance with "c.(e.)) with Algorithm I First column corresponds under-trained regime while second column corresponds to well-trained one.
J ANALYSIS OF GUIDANCE WEIGHTS
For the analysis of guidance weights, evaluate guidance network wh from COCO or 1000 classes from ImageNet, using T 128 denoising steps for COCO, and T 100 steps for ImageNet. We then compute average guidance weight per conditioning, i.c.
(t, dr.;) all 10k prompts
We T
(t-ddt))
(30)
and also a sum of guidance weights per conditioning, i.e.,
We = Σ ct(t-dt,
(31)
34

---

Published as a conference paper at ICLR 2026
We then select top K (for K=10), median K (K after the median) and bottom K prompts / classes based on the values We-
J.O.1 ANALYSIS OF GUIDANCE WEIGHTS ON MS COCO
We visualize & (t,))dt.() for top, median and bottom prompts Figure 17 that the guidance weights behave quite differently for these categories. For bottom K, we see that guidance weights are high close the noise and low throughout. For the median K guidance weights, we see that they are increasing as approaches O. For top K guidance weights, we see that they generally high. This suggests that there is very different behavior depending the prompt. Moreover, in Figure T8] we show mean and standard deviation we across prompts for given prompt length. Overall, it seems that prompt length does affect the magnitude of guidance weights that much.
The results indicate
puldanes woights
gubdancs woight
12 pullancs soggts
62
EB datal
08
1t
data)
Times [T data)
Figure 17: _ dt,)) guidance weights we
for top, median and bottom K prompts based on the values of sum of per-prompt
31
10-
Man HAW
NM
Wc Constant guidance = 7.5
40
80
1.00
120
Prompt length
Figure 18: Average per-promps guidance weights We 30 for different prompt lengths, shaded region indicate standard deviation.
Below we show the prompts for top, median and bottom K categories. See Table 12 Table 13]
Table 14 It is hard to see if there is specific category of prompts which leads to high or low guidance, so we hypothesize that the discrepancy is due to how the original diffusion model was trained.
J.0.2 ANALYSIS OF GUIDANCE WEIGHTS ON IMAGENET
We visualize a (((t-at.t) for top, median and bottom K classes on ImageNet, see Figure
19
Unlike in T2I, we see that guidance weights seem to have two distinct behaviors for bottom K, they are generally small, while for median and top they are generally large. We also see that they look like form of Limited Interval Guidance (LIG) and clamp-lineal guidance. The discrepancy to T2I could be attributed to few things. First, T2I, we add clip score reward to the loss which encourages prompt alignment. Second, in T2I, the diffusion model was trained vast amount of text-image pairs, such that each prompt receives an unequal number of images. In ImageNet, each class is seen the same amount of time.
Below, we show the class labels and their corresponding W. for top, median and median classes based th See Table 15 Table 16 and Table 17
35

---

Published as a conference paper at ICLR 2026
Table 12: Top K Prompts with Highest average guidance weight (oe)
30
Prompt
We
an old rusty refrigerator is abandoned field.
12.975
An old truck with several years of paint schemes sits parked as artwork.
12.936
2
A sky filled with lots of colorful flying kites.
12.881
pile of junk sitring next to a curb on green grass. Street painted on the wall in an asian country.
12.756
12.733
OLD RUSTED BOXCARS STILL SITTING ON A SET OF TRACKS A field full of people flying kites in cloudy blue sky. crowded city filled with lots of people and traffic.
12.727
12.630
12.607
a large clock tower towering above city under cloudy blue sky.
12.544
Some roll down doors at building painted with graffiti.
12.494
Table 13: Median K Prompts by average guidance weight (w:.)
30
k Prompt
We
0 A group of men standing around table of food in wooden
6.477
room.
A room with toilet, door and shoes in it.
6.475
2 A boy in black shirt doing trick on skateboard.
6.474
man stands in front of flipped skate boarder
6,474
4 Oddly shaped homemade pizza about to be cut with pizza cutter
6.474
5
a silver toilet and some toilet paper and green switch
6.471
Two cakes sitting on a class table near a candle.
6.471
7 A counter top in a kitchen topped with fruits and vegeta-
6.470
bles.
8
A park with many trees and benches at night.
6.470
9 A duck floating on a lake with gray and black feathers.
6.470
Table 14: Bottom Prompts with Lowest average guidance weight (a.)
30
Prompt
We
0 A group of kids that are playing soccer on a field.
0.217
A group of men playing game of soccer on a field.
0.243
2 A group of people playing game of soccer on top of field.
0.280
3 A group of men who are playing basketball together.
0.335
Small boys are playing a softball game on grass.
0.347
5 A group of men on a field playing baseball.
0.358
Brown horse grazing alone in an open grassy field.
0.392
7 Men baseball diamond playing baseball, man just
0.393
swung
A group of soccer players scramble against each other
0.413
get the ball.
A group of children playing soccer on field.
0.415
36

---

Published as a conference paper at ICLR 2026
10
Median 16
10 quidiene
22-
k-768
k-788
k-88b
k-560
k=51
k-592
03
k=613
kw681
20
129
Time (t indicates data)
Time 1t indicates data)
Time (T Indicates data)
Figure 19: Guidance weights for bottom, median, top K on ImageNet.
Table 15: Top K Classes with Highest Mean Guidance Weight (mean w 30
k
Class id
Class name
mean
768
rugby_ball
0.727
1 650
microphone
0.735
2
788
shoe shoc_shop
0.735
3
886
vending_ machine
0.738
4
560
football
0.743
5
513
cornet
0.745
6
592
hard_disc
0.748
7
541
drum
0.751
8
613
joystick
0.752
681
notebook
0.777
Table 16: Median K Classes by Conditional Guidance Weight (5.) 30
k
Class id
Class name
We
345
ox
0.445
1
735
poncho
0.445
2
765
rocking chair
0.444
3
607
jack-o'-_lante
0.444
4
51
triceratops
0.447
5
660
mobile_home
0.447
6
565
freight_caa
0.444
7
108
sea an__nee
0.447
8
200
Tibetan terrier
0.443
255
Leonberg
0.443
Table 17: Bottom K Classes with Lowest Conditional Guidance Weight (w.) 30
k
Class id
Class name
We
640
manhole_cover
0.000
1
138
bustard
0.000
prairie_chicken
chicken
0.000
3
353
gazelle
0,009
974
geyser
0,012
5
44
alligator lizard
0.012
106
wombat
0.020
352
impala
0.021
8
276
hyena
0.024
209
Chesapeake Bay_ retriever
0.026
37


Reviews:
[{"review_id": "LDUPtVtTny", "reviewer": "Reviewer_6vA2", "summary": "This paper proposes a method to learn the classifier-free guidance (CFG) weight by setting it as the output of a neural network. While previous CFG approaches relied on heuristics or trial-and-error to find an optimal (often large) weight, which could lead to saturation and degraded sample quality, this work optimizes guidance weights as a continuous function of time and condition. The objective is to minimize the distributional mismatch between the generated guided samples and the noised true conditional samples, using a novel self-consistency loss. This loss successfully mitigates the high-variance problem associated with theoretical approaches, leading to stable training and demonstrated performance gains across various datasets, from toy examples to COCO T2I sampling.", "questions": "- How does this method compare to guidance Interval approaches? The guidance interval method empirically finds an optimal guidance schedule, assigning near-zero guidance weight in early steps and large weights in the final few steps. Since the proposed method can measure how well the estimated distribution matches the true one via its loss, it may be possible to evaluate the optimality of the interval-based scheduling using the same loss metric. If so, this could open new directions for systematic guidance interval optimization beyond the work of LIG.\n- Once the optimal guidance weights are obtained through this method, could they be further distilled into a single model to achieve both high performance and inference efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The idea of learning the optimal guidance weight continuously via a neural network, rather than relying on grid search for optimal guidance intervals or empirical adjustments, is highly novel.\n- Despite the inherent risk of the guidance weight collapsing to a trivial solution, the successful stabilization of learning via the newly proposed self-consistency loss is a significant achievement. The simplified L2 objective (Eq. 21) also appears surprisingly straightforward to implement.", "weaknesses": "The proposed method involves a large number of hyperparameters to determine the optimal guidance weight, and additionally requires performing $m$ rounds of noising and comparison at every denoising step, which introduces significant computational overhead. Given this, I wonder whether the computational cost difference compared to existing guidance distillation methods is actually substantial. It would be valuable to include a cost comparison table or discussion.\n\nMoreover, since the loss function in this paper formalizes the matching between the true sample distribution and the guided one, I suspect that a parameter-efficient fine-tuning (PEFT) approach such as LoRA could optimize a student model directly using the same loss formulation, without needing a separate guidance network at inference time. This would result in a more inference-efficient model, and I’m curious whether the authors have considered or experimented with this alternative.\n\nRegarding the design of the guidance network, the paper focuses heavily on the loss formulation for learning guidance weights but provides limited discussion or justification for the network’s architecture. The rationale behind its input-output design is unclear. Based on Table 1 and Fig. 1, 3, the conditioning input seems to have a stronger influence than expected, suggesting that incorporating the currently sampled image as an additional input could further improve weight estimation. Were such variants or conditioning combinations explored?\n\nIn the COCO experiments, the guidance network uses the text encoder output as a conditioning signal, and Fig. 3 shows that the learned guidance weights vary across prompts. Does this imply that the network has learned semantic understanding of the text embeddings? If so, how heavy must the model be to capture such semantics effectively? Additionally, from the qualitative results on COCO, the outputs often resemble those with weak or no guidance, which may indicate that the diffusion model’s original objective still struggles to fully capture the conditional distribution, similar to previous findings.\n\nFinally, in the loss formulation, the authors mention using $m=4$ particles for computing the MMD in the image domain. Even if self-consistency reduces variance, I question whether the MMD loss computed from only four samples can be sufficiently accurate. Was this number chosen primarily to reduce computational cost? If so, it would be helpful to report how performance varies with different particle counts, to justify the design choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsOdWrHTWh", "reviewer": "Reviewer_sh9D", "summary": "This paper puts forward learning a classifier-free guidance schedule that varies with both time and the input condition (class or text), enabling the sampler to better match the desired conditional distribution.\nCompare with handcrafting guidance strengths, the method optimizes a self-consistency objective: if one adds noise to a real image and then denoises it with guidance, the intermediate result should match what the forward noising process would produce at that timestep.\nFollowing this desin mechnism, this reframes guidance tuning as a distribution-matching problem and naturally extends to reward-guided generation by combining an external reward (e.g., CLIP-based) with the same self-consistency regularizer.\nImortantly, exntensive experiments on standard image datasets and text-to-image benchmarks show consistent quality gains over unguided sampling, fixed guidance, and simple time-limited guidance, with learned schedules that adapt across prompts and timesteps.", "questions": "1. How sensitive are results to the kernel/parameters in the MMD objective versus the simpler L2 variant?\n\n2. What are the training and inference costs attributable to guidance learning and self-consistency evaluation?\n\n3. Since the learned weights vary strongly with prompt, do certain semantic categories systematically demand higher guidance?\n\n4. Can practitioners easily inspect how guidance evolves with time and prompt?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It reframes classifier-free guidance tuning as a distribution-matching problem, replacing hand-crafted schedules with a learned, time- and prompt-aware policy—an original and practical angle.\nThe self-consistency objective is elegant and low-variance, making the method easy to implement on top of existing diffusion backbones without architectural changes.\nExperiments span multiple datasets and backbones and include sensible ablations, showing consistent gains over unguided, fixed-guidance, and limited-interval baselines.\nThe learned guidance schedules are interpretable (varying with prompt and timestep), which improves clarity and aids real-world debugging.\nOverall, the approach offers a broadly applicable and deployable improvement to conditional diffusion sampling with a favorable quality-to-complexity trade-off.", "weaknesses": "1. The paper argues that enforcing self-consistency is stronger and low-variance, but the conditions under which minimizing the proposed loss actually improves conditional sampling are not formalized.\n\n2. On text-to-image, self-consistency plus CLIP reward improves FID yet struggles to surpass strong guided baselines on CLIP score.\n\n3. Learning a guidance network and evaluating the self-consistency loss add additional overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z3O6qtCYR9", "reviewer": "Reviewer_h7dP", "summary": "The authors propose a method for learning the guidance schedule for diffusion model generation. In particular, they train a shallow network to minimize the distributional mismatch between a frozen denoiser and the true distribution. The authors provide solid theoretical motivation for their choice of objective function, and also experimental evidence for class and text-conditional models.", "questions": "- What do the authors mean by a high-variance objective?\n- What is the training cost of training the learned guidance network as a percentage of diffusion model training cost?\n- Why is the FID on COCO for the constant guidance model so high? FID 31 seems incredibly high for a 1B model. How was this model trained?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors replace a hyper-parameter (guidance strength) with a learned approach\n- The objective function is well theoretically motivated", "weaknesses": "- The authors train their own diffusion models for use in evaluations. It would greatly strengthen the paper if they also showed FID and reward improvements for existing pre-trained diffusion models from prior work. For example:\n     - EDM for cifar-10 (Karras et al, https://github.com/NVlabs/edm)\n     - EDM-2 for ImageNet (Karras et al, https://github.com/NVlabs/edm2)\n     - MicroDiffusion for text-to-image (Sehwag et al, https://github.com/SonyResearch/micro_diffusion)\n- The authors compare their learned approach to simple baselines such as constant or limited-interval guidance. It would strengthen the paper if they also showed improvements over stronger un-trained baselines such as in Wang et al (https://arxiv.org/abs/2404.13040).\n\nMy score recommends reject, but I would be happy to raise my score to accept if these additional experiments are included in the paper and my questions below are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ssRi73qZzv", "reviewer": "Reviewer_A5xC", "summary": "The paper proposes to introduce the Maximum Mean Discrepancy (MMD) to quantitatively evaluate the distance between the CFG-guided distribution and the ground-truth one. By doing so, it is enabled to design both time-dependent and condition-dependent guidance weight, which is more flexible and controllable. To realize such a pipeline, the authors employ an auxiliary network for guidance weight prediction, and further discuss the influences under different objectives. Both qualitative and quantitative results confirm the superiority of the proposed method.", "questions": "Beyond the Weaknesses part, I am curious about the general rule of the guidance weights. Current analyses in Fig. 3 is poor. Could the authors provide more quantitative analyses about the weights under different conditions and timesteps? For example, what if a text condition gradually becomes complex and a long caption? What about the mean or variance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well structured and easy to follow. All technical details are carefully discussed and clear.\n- The motivation is intuitive and effective, alleviation the discrepancy between CFG-guided and the ground-truth distributions is capable of improving the performance of guided sampling.\n- The further discussion about different objective is detailed and inspiring, encouraging future study for better training efficiency and synthesis performance.", "weaknesses": "- The whole pipeline of narrowing the distribution discrepancy by optimizing some scalars is not new. Beyond the main objective being directly motivated by prior works, similar idea of canceling mismatch (or \"self-consistency\") between forward diffusion and reverse denoising stages has also been proposed [1].\n\n  [1] Towards More Accurate Diffusion Model Acceleration with A Timestep Tuner. Xia et al., CVPR 2024.\n\n- The main concern about the proposed method is the optimality. There is no theoretical analysis about the optimality or convergence about the proposed objective. Then why the optimized guidance weight could guarantee the self-consistency? What further confirms my worries is in Appendix C.1. The most intuitive objective is the L2 norm in Eq. (23), which is consistent with the vanilla training loss of diffusion models, *i.e.*, the ELBO loss. However, as the authors themselves claim, such an objective leads to zero guidance weight. That is to say, **diffusion model converges well and both conditional and unconditional scores are accurate**, thus there is no need to employ CFG. Under this discussion, I am curious about the theoretical optimality of Eq. (20) or Eq. (21). Could the authors provide theoretical analyses under some trivial toy data to verify the correctness of the method? From my opinion, Eq. (21) is somewhat an upper bound of ELBO (*i.e.*, Eq. (23)). Given that ELBO itself is an upper bound of KL-divergence (by the native theory of diffusion models), the optimization of a looser bound may lead to meaningless results.\n\n- What further weakens the soundness of the paper is the employment of an auxiliary network. To be honest, such a setting could lead to more flexible guidance weight given different conditions. However, what if the network fails to converge well and only predicts sub-optimal guidance weight? The accuracy issue is more severe under open-vocabulary setting, *i.e.*, text-to-image or text-to-video. Considering that the authors employ a light-weight network, such a concern is also crucial. Could the author provide some closed-form expressions or some empirical solutions avoiding the employment of an auxiliary network?\n\n- The qualitative results of text-to-image is somewhat not convincing. The improvements in Figs. 4-7 are inconspicuous. This strengthens my concerns above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)