Learn to Guide Your Diffusion Model

## Abstract

Abstract Classifier-free guidance (CFG) is a widely used technique for improving the perceptual quality of samples from conditional diffusion models. It operates by linearly combining conditional and unconditional score estimates using a guidance weight ω \omega . While a large, static weight can markedly improve visual results, this often comes at the cost of poorer distributional alignment.
In order to better approximate the target conditional distribution,
we instead learn guidance weights ω c , ( s , t ) \omega_{c,(s,t)} , which are continuous functions of the conditioning c c , the time t t from which we denoise, and the time s s towards which we denoise.
We achieve this by minimizing the distributional mismatch between noised samples from the true conditional distribution and samples from the guided diffusion process. We extend our framework to reward guided sampling, enabling the model to target distributions tilted by a reward function R ​ ( x 0 , c ) R(x_{0},c) , defined on clean data and a conditioning c c . We demonstrate the effectiveness of our methodology on low-dimensional toy examples and high-dimensional image settings, where we observe improvements in Fréchet inception distance (FID) for image generation. In text-to-image applications, we observe that employing a reward function given by the CLIP score leads to guidance weights that improve image-prompt alignment.

## 1 Introduction

Diffusion models (Sohl-Dickstein et al.,, 2015; Ho et al.,, 2020; Song et al., 2021b,; Song and Ermon,, 2019) produce high-quality synthetic data in areas such as images (Saharia et al.,, 2022; BlackForestLabs,, 2025), videos (Google,, 2025), and proteins (Watson et al.,, 2023; Abramson et al.,, 2024). These models proceed by gradually adding Gaussian noise through a diffusion process, transforming the data distribution into a Gaussian distribution. The generative model is obtained by approximating the time-reversal of this noising process. Practically, this relies on a learned denoiser network which is obtained by minimizing a regression objective. A similar strategy can, in principle, be developed to sample from a conditional distribution by learning a conditional denoiser network.

Recently, Classifier-Free Guidance (CFG) (Ho and Salimans,, 2022) has become a popular choice to address the task of conditional sampling. CFG modifies the generating process by incorporating a *guidance* term, calculated as the difference between conditional and unconditional denoiser networks. This difference is scaled by a *guidance weight*, $\omega$, such that $\omega=-1$ recovers unconditional generation whereas $\omega=0$ corresponds to the conditional one. Empirically, it was shown that CFG can drastically improve performance of diffusion models compared to conditional generation (which only uses the conditional denoiser), especially for very high guidance weights (Saharia et al.,, 2022; Rombach et al.,, 2022). (Chidambaram et al.,, 2024; Wu et al.,, 2024) argued that such a regime pushes the samples to the edge of support such that they become easy to classify.

Classifier-free guidance was initially justified as a method for sampling from a modified distribution $p_{\omega}(x_{0}|c)\propto p(x_{0})^{-\omega}p(x_{0}|c)^{1+\omega}\propto p(x_{0})p(c|x_{0})^{\omega+1}$, which amplifies the conditioning term (Ho and Salimans,, 2022). However, subsequent work has established that the standard CFG sampling procedure does not, in fact, produce samples from this target distribution (Du et al.,, 2023; Koulischer et al., 2025a,). Instead, CFG sampling shifts the generated samples towards the modes of the original conditional $p(x_{0}|c)$. While this discrepancy can be corrected with computationally intensive methods like Sequential Monte Carlo (Skreta et al.,, 2025; He et al.,, 2025) or Markov Chain Monte Carlo (Du et al.,, 2023; Moufad et al.,, 2025; Zhang et al.,, 2025), these approaches are often impractical for large-scale applications. More importantly, naive CFG *already* yields a significant quality boost and therefore such correction might not be required.

Many works, see e.g. Ho and Salimans, (2022); Kynkäänniemi et al., (2024); Wang et al., (2024); Kim et al., (2025), report improved FID scores when using some level of guidance ($\omega>0$). This observation is at odds with the original motivation of CFG. Given that the learned denoiser $\hat{x}_{\theta}(x_{t},c)$ is only an approximation of the true denoiser $\mathbb{E}[x_{0}|x_{t},c]$, this suggests that CFG acts as a *correction to this approximation* which leads to a better modeling of the target conditional distribution $p(x_{0}|c)$.

Based on this observation, we propose to *learn* a guidance weight $\omega$ in order to better approximate the conditional distribution $p(x_{0}|c)$. We generalize the CFG method and allow the guidance weight to be both conditioning and time dependent, i.e., $\omega_{c,(s,t)}$. This weight is used to guide the denoising process from time $t$ to time $s$. We learn $\omega_{c,(s,t)}$ by matching the distribution of samples from the true diffusion process and the distribution of samples of the guided one.
Empirically, we show that our method allows us to learn $\omega_{c,(s,t)}$ that better approximates the target distribution compared to the unguided model. For image generation, this leads to a consistently lower FID (Heusel et al.,, 2017) than the unguided model or a model with a constant guidance weight.

We further develop our method in a setting where an additional reward function $R(x_{0},c)$ is defined on samples from the diffusion model and a conditioning $c$, and we want to bias the samples from the model to the regions of high rewards $R(x_{0},c)$.
In text-to-image applications, we found that empowering our method with a reward function given by the CLIP score (Hessel et al.,, 2021), leads to guidance weights which improve both FID and image-prompt alignment.

The paper is organized as follows. In [Section˜2](https://arxiv.org/html/2510.00815v1#S2), we provide a short background on the necessary concepts. In [Section˜3](https://arxiv.org/html/2510.00815v1#S3), we describe our approach of learning guidance weights $\omega_{s,(s,t)}$. In [Section˜4](https://arxiv.org/html/2510.00815v1#S4), we present the related work on the topic. Finally, the experimental results are presented in [Section˜5](https://arxiv.org/html/2510.00815v1#S5).

## 2 Background

#### Notation.

We write $p_{t}$ for a probability distribution $p(x_{t})$, and $p_{s|t}$ to denote a conditional distribution $p(x_{s}|x_{t})$ for any $s,t$. We also use $p_{0|c}$ to denote $p(x_{0}|c)$, $p_{s|t,0}$ for $p(x_{s}|x_{t},x_{0})$, and $p_{s|t,c}$ for $p(x_{s}|x_{t},c)$. We denote by $p_{0,c}$ a joint distribution $p(x_{0},c)$. Finally, we write $\mathcal{N}(x;\mu,\Sigma)$ to denote the Gaussian density of argument $x$, mean $\mu$ and covariance $\Sigma$ and $\mathcal{N}(\mu,\Sigma)$ for the distribution.

#### Diffusion models.

The goal of conditional diffusion models is to sample from a target conditional distribution $p_{0|c}$ on $\mathbb{R}^{d}$, where $c\sim p(c)$ is a conditioning signal (i.e. a text prompt) sampled from a conditioning distribution. We adopt here the Denoising Diffusion Implicit Models (DDIM) framework of Song et al., 2021a. Let $x_{t_{0}}\sim p_{0|c}$ and define for $0=t_{0}<\dots<t_{N}=1$ the process $x_{t_{1}:t_{N}}:=(x_{t_{1}},...,x_{t_{N}})$ by

$$ $\textstyle p(x_{t_{1}:t_{N}}|x_{t_{0}})=p_{t_{N}|t_{0}}(x_{t_{N}}|x_{t_{0}})\prod_{k=1}^{N-1}p_{t_{k}|t_{k+1},0}(x_{t_{k}}|x_{t_{k+1}},x_{t_{0}}),$ (2) $$

where, for $0\leq s<t\leq 1$, we have

$$ $\textstyle p_{s|t,0}(x_{s}|x_{t},x_{0})=\mathcal{N}(x_{s};\mu_{s,t}(x_{0},x_{t}),\Sigma_{s,t}).$ (3) $$

Both the mean $\mu_{s,t}(x_{0},x_{t})$ and covariance $\Sigma_{s,t}$ depend on a *churn* parameter $\varepsilon\in[0,1]$, which controls the amount of stochasticity. Their full expressions are given in ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) in [Appendix˜B](https://arxiv.org/html/2510.00815v1#A2).
Here, the mean and variance are selected so that for any $t\in[0,1]$, we recover the *noising process*

$$ $\textstyle p_{t|0}(x_{t}|x_{0})=\mathcal{N}(x_{t};\alpha_{t}x_{0},\sigma_{t}^{2}\mathrm{Id}),$ (4) $$

for $\alpha_{t}$ and $\sigma_{t}$ such that $\alpha_{0}=1,\sigma_{0}=0$ and $\alpha_{1}=0,\sigma_{1}=1$, i.e. $p_{1|0}(x_{1}|x_{0})=\mathcal{N}(x_{1};0,\mathrm{Id})$. More precisely for any $0\leq s\leq t\leq 1$, we have

$$ $\textstyle p_{s|0}(x_{s}|x_{0})=\int p_{t|0}(x_{t}|x_{0})p_{s|t,0}(x_{s}|x_{t},x_{0})\mathrm{d}x_{t}.$ (5) $$

#### Sampling with DDIM.

At inference time, $x_{t_{0}}\sim p_{0|c}$ is generated by starting from a Gaussian $x_{t_{N}}\sim\mathcal{N}(0,\mathrm{Id})$ and
$x_{t_{k}}\sim p(\cdot|x_{t_{k+1}},c)$ for $k=N-1,...,0$, where for $0\leq s<t\leq 1$

$$ $\textstyle p_{s|t,c}(x_{s}|x_{t},c)=\int p_{s|t,0}(x_{s}|x_{t},x_{0})p_{0|t,c}(x_{0}|x_{t},c)\mathrm{d}x_{0},$ (6) $$

where $p_{0|t,c}(x_{0}|x_{t},c)$ is a posterior distribution, see also (Song et al., 2021a,) for discussion.
Since we do not have access to $p_{0|t,c}$, we approximate it by $\updelta_{\hat{x}_{\theta}(x_{t},c)}$ for any $t\in[0,1]$, where $\hat{x}_{\theta}(x_{t},c)\approx\mathbb{E}[x_{0}|x_{t},c]$
is a neural network denoiser with parameters $\theta$ trained by minimizing the loss

$$ $\textstyle\mathcal{L}(\theta)=\int_{0}^{1}\lambda(t)\mathbb{E}_{(x_{0},c)\sim p_{0,c},x_{t}\sim p_{t|0}}[\|x_{0}-\hat{x}_{\theta}(x_{t},c)\|^{2}]\mathrm{d}t.$ (7) $$

Here, $\lambda(t)\geq 0$ is a weighting function (see (Kingma et al.,, 2021)). This yields the following procedure to sample approximately from $p_{0|c}$, by starting from $x_{t_{N}}\sim\mathcal{N}(0,\mathrm{Id})$, we then follow

$$ $\displaystyle\textstyle x_{t_{k}}\sim\mathcal{N}(\mu_{t_{k},t_{k+1}}(\hat{x}_{\theta}(x_{t_{k+1}},c),x_{t_{k+1}}),\Sigma_{t_{k},t_{k+1}}),\qquad k=N-1,...,0$ (8) $$

#### Classifier-Free Guidance (CFG).

Let $\hat{x}_{\theta}(x_{t},\varnothing)\approx\mathbb{E}[x_{0}|x_{t}]$ be an unconditional denoiser learned using a regression objective ([7](https://arxiv.org/html/2510.00815v1#S2.E7)) (where conditioning is omitted).
CFG replaces $\hat{x}_{\theta}(x_{t},c)$ when simulating approximately from $p_{0|c}$ by

$$ $\textstyle\hat{x}_{\theta}(x_{t},c;\omega)=\hat{x}_{\theta}(x_{t},c)+\omega\Delta_{\theta}(x_{t},c),$ (9) $$

where $\omega$ is the *guidance weight* and $\Delta_{\theta}(x_{t},c)=\hat{x}_{\theta}(x_{t},c)-\hat{x}_{\theta}(x_{t},\varnothing)$.
In order to sample the data with CFG, we first sample $x_{t_{N}}\sim\mathcal{N}(0,\mathrm{Id})$ and then follow

$$ $\displaystyle\textstyle x_{t_{k}}\sim\mathcal{N}(\mu_{t_{k},t_{k+1}}(\hat{x}_{\theta}(x_{t_{k+1}},c;\omega),x_{t_{k+1}}),\Sigma_{t_{k},t_{k+1}}),\quad k=N-1,...,0.$ (10) $$

When $\omega=0$, the sampling procedure ([10](https://arxiv.org/html/2510.00815v1#S2.E10)) is equivalent to *conditional* sampling ([8](https://arxiv.org/html/2510.00815v1#S2.E8)), while $\omega=-1$ corresponds to a procedure using only the unconditional denoiser, allowing to sample approximately from $p_{0}$. In practice, one sets $\omega>0$ in order to emphasize conditioning.

In this paper, we generalize CFG and make *guidance weight* $\omega_{c,(s,t)}$ a function of conditioning $c$ and of two time-steps $s<t$, where $t$ is the *proposal* timestep from which we denoise towards a *target* timestep $s$ as in ([6](https://arxiv.org/html/2510.00815v1#S2.E6)). Denoising following ([10](https://arxiv.org/html/2510.00815v1#S2.E10)) corresponds to setting, $s=t_{k}$ and $t=t_{k+1}$. We denote $\bm{\omega}=(\omega_{c,(s,t)})_{s,t\in[0,1],t>s}$. In [Algorithm˜2](https://arxiv.org/html/2510.00815v1#alg2), we describe the CFG sampling method with DDIM.

#### Maximum Mean Discrepancy.

To optimize the guidance weights $\bm{\omega}$, we match the true diffusion distribution $p$ and the guided $p^{\bm{\omega}}$, see [Section˜3](https://arxiv.org/html/2510.00815v1#S3). We employ the Maximum Mean Discrepancy (MMD) (Gretton et al.,, 2012) with energy kernel (Székely and Rizzo,, 2004; Sejdinovic et al.,, 2013),

$$ $\textstyle\mathrm{MMD}_{(\beta,\lambda)}[p^{\bm{\omega}},p]=\mathbb{E}_{p_{\bm{\omega}}\otimes p}[||x-y||^{\beta}_{2}]-\frac{\lambda}{2}(\mathbb{E}_{p^{\bm{\omega}}\otimes p^{\bm{\omega}}}[||x-x^{\prime}||^{\beta}_{2}]+\mathbb{E}_{p\otimes p}[||y-y^{\prime}||^{\beta}_{2}]),$ (11) $$

for independent $x,x^{\prime}\sim p^{\bm{\omega}}$ and $y,y^{\prime}\sim p$, with $\beta\in(0,2)$ and $\lambda\in[0,1].$ We require $\lambda=1$ for a valid MMD, however $\lambda<1$ may be preferable for generative modeling (Bouchacourt et al.,, 2016; De Bortoli et al.,, 2025); other characteristic kernels may also be used (Sriperumbudur et al.,, 2010).

## 3 Learning to guide your diffusion model

In this section, we introduce our method for learning the guidance weights, $\bm{\omega}$, assuming access to a pre-trained conditional denoiser, $\hat{x}_{\theta}(x_{t},c)$, and an unconditional one, $\hat{x}_{\theta}(x_{t},\varnothing)$. Our approach is based on enforcing *consistency conditions*— that any valid diffusion process must satisfy.
We first derive a theoretically sound objective from *marginal consistency* condition, which is however impractical to optimize due to high variance. To overcome this, we introduce a stronger, more practical condition, which we call *self-consistency*. Enforcing *self-consistency* is sufficient for achieving *marginal consistency* and, crucially, results in a simple, low-variance objective. We formulate our complete approach for learning guidance weights based on this simplified approach. We present alternative approaches in [Appendix˜C](https://arxiv.org/html/2510.00815v1#A3).

### 3.1 Consistency conditions

#### Marginal consistency.

The process defined by ([2](https://arxiv.org/html/2510.00815v1#S2.E2)) admits the marginal distribution for $s\in(0,1]$

$$ $\textstyle p_{s}(x_{s})=\int p_{s|0}(x_{s}|x_{0})p_{0,c}(x_{0},c)\mathrm{d}x_{0}\mathrm{d}c.$ (12) $$

This marginal distribution can also be obtained for any $0\leq s<t\leq 1$ via

$$ $\textstyle p_{s}(x_{s})=\int\int\int p_{s|t,c}(x_{s}|x_{t},c)p_{t|0}(x_{t}|x_{0})p_{0,c}(x_{0},c)\mathrm{d}x_{t}\mathrm{d}x_{0}\mathrm{d}c,$ (13) $$

with $p_{s|t,c}$ given by DDIM ([6](https://arxiv.org/html/2510.00815v1#S2.E6)). The equality ([13](https://arxiv.org/html/2510.00815v1#S3.E13)) states that the marginal at time $s$ can be obtained by sampling $(x_{0},c)\sim p_{0,c}$, noising $x_{0}$ to $x_{t}|x_{0}\sim p_{t|0}$ and then denoising it back to time $s$ with $p_{s|t,c}$.

We now consider a denoising mechanism relying on the guided denoiser approximation ([9](https://arxiv.org/html/2510.00815v1#S2.E9)) which follows the construction in ([13](https://arxiv.org/html/2510.00815v1#S3.E13)) to obtain a sample at time $s$. Again we sample $(x_{0},c)\sim p_{0,c}$ and $x_{t}\sim p_{t|0}(\cdot|x_{0})$. However, the denoising to time $s$ where $0\leq s<t\leq 1$ is done with the guided denoiser approximation, so that the marginal distribution of the resulting sample at time $s$ is

$$ $\textstyle p_{s}^{t,(\theta,\bm{\omega})}(x_{s})=\int\int\int p_{s|t,c}^{(\theta,\bm{\omega})}(x_{s}|x_{t},c)p_{t|0}(x_{t}|x_{0})p_{0,c}(x_{0},c)\mathrm{d}x_{t}\mathrm{d}x_{0}\mathrm{d}c,$ (14) $$

where

$$ $\textstyle p_{s|t,c}^{(\theta,\bm{\omega})}(x_{s}|x_{t},c)=p_{s|t,0}(x_{s}|x_{t},\hat{x}_{\theta}(x_{t},c;\bm{\omega})).$ (15) $$

The distribution $p_{s}^{t,(\theta,\bm{\omega})}$ ([14](https://arxiv.org/html/2510.00815v1#S3.E14)) is typically not equal to $p_{s}$ ([12](https://arxiv.org/html/2510.00815v1#S3.E12)) as we used a delta-function approximation ([15](https://arxiv.org/html/2510.00815v1#S3.E15)) with a model $\hat{x}_{\theta}(x_{t},c;\bm{\omega})$ instead of sampling from $p_{0|t,c}$ as required by ([6](https://arxiv.org/html/2510.00815v1#S2.E6)). Nevertheless, we could attempt to find guidance weights $\bm{\omega}$ to satisfy *marginal consistency*, i.e. for all $0\leq s<t\leq 1$

$$ $\textstyle p_{s}^{t,(\theta,\bm{\omega})}(x_{s})\approx p_{s}(x_{s}).$ (16) $$

This could be achieved by minimizing the MMD ([11](https://arxiv.org/html/2510.00815v1#S2.E11))

$$ $\textstyle\mathcal{L}_{m}(\bm{\omega})=\mathbb{E}_{(s,t)\sim p(s,t)}[\mathrm{MMD}_{(\beta,\lambda)}[p_{s}^{t,(\theta,\bm{\omega})}(\cdot),p_{s}(\cdot)]],$ (17) $$

where $p(s,t)$ is a distribution on $0\leq s<t\leq 1$. This distribution is crucial for good empirical performance, and we discuss it in detail in [Section˜3.2](https://arxiv.org/html/2510.00815v1#S3.SS2).
The gradient of ([17](https://arxiv.org/html/2510.00815v1#S3.E17)) could suffer from high variance due to marginalization over $(x_{0},c)$, however, which could make it challenging to optimize. We found that this approach did not work in practice. Below, we propose a simpler, lower variance method, albeit one that imposes more constraints on the guided backward scheme compared to ([17](https://arxiv.org/html/2510.00815v1#S3.E17)).

#### Self-consistency (conditioning on ( c , x 0 ) (c,x_{0}) ).

For a fixed $(x_{0},c)\sim p(x_{0},c)$, we denote by

$$ $\textstyle p_{s|0,c}^{t,(\theta,\bm{\omega})}(x_{s}|x_{0},c)=\int p_{s|t,c}^{(\theta,\bm{\omega})}(x_{s}|x_{t},c)p_{t|0}(x_{t}|x_{0})\mathrm{d}x_{t},$ (18) $$

which is the term under integral in ([14](https://arxiv.org/html/2510.00815v1#S3.E14)) depending on $(x_{0},c)$. We consider a *self-consistency* condition

$$ $\textstyle p_{s|0,c}^{t,(\theta,\bm{\omega})}(x_{s}|x_{0},c)\approx p_{s|0,c}(x_{s}|x_{0},c)=p_{s|0}(x_{s}|x_{0}),$ (19) $$

where $p_{s|0,c}^{t,(\theta,\bm{\omega})}(\cdot|x_{0},c)$ is given by ([18](https://arxiv.org/html/2510.00815v1#S3.E18)) and $p_{s|0}(\cdot|x_{0})$ is a *noising process* ([4](https://arxiv.org/html/2510.00815v1#S2.E4)). We used a notation $p_{s|0,c}(x_{s}|x_{0},c)=p_{s|0}(x_{s}|x_{0})$ to highlight that $x_{0}\sim p_{0|c}(\cdot|c)$. Intuitively, this condition means that as we start from $x_{0}|c$ and go through the *noising process* $x_{t}|x_{0}$ and then denoise with guidance to time $s<t$, the distribution at time $s$ should be the same as of the *noising process*.

The condition ([19](https://arxiv.org/html/2510.00815v1#S3.E19)) is much stronger than ([16](https://arxiv.org/html/2510.00815v1#S3.E16)). Indeed, if ([19](https://arxiv.org/html/2510.00815v1#S3.E19)) is satisfied for every $(x_{0},c)\sim p_{0,c}$, then by integrating it over $(x_{0},c)$, we will get ([16](https://arxiv.org/html/2510.00815v1#S3.E16)). The reverse is not true.
Moreover, the lack of dependence on $x_{t}$ (and on $x_{0})$ in guidance weights $\omega_{c,(s,t)}$ makes it very unlikely for this condition to hold. However, it provides a learning signal for guidance weights and we will aim to satisfy it approximately.

For $(x_{0},c)$ in the support of $p(x_{0},c)$, we could approximately satisfy ([19](https://arxiv.org/html/2510.00815v1#S3.E19)) by minimizing wrt $\bm{\omega}$, $\mathrm{MMD}_{(\beta,\lambda)}[p_{s|0,c}^{t,(\theta,\bm{\omega})}(\cdot|x_{0},c),p_{s|0,c}(\cdot|x_{0})]$, see ([11](https://arxiv.org/html/2510.00815v1#S2.E11)).
Averaging over all $(x_{0},c)\sim p(x_{0},c)$ leads to

$$ $\textstyle\mathcal{L}_{\beta,\lambda}(\bm{\omega})=\mathbb{E}_{(x_{0},c)\sim p_{0,c},s,t\sim p(s,t)}[\mathrm{MMD}_{(\beta,\lambda)}[p_{s|0,c}^{t,(\theta,\bm{\omega})}(\cdot|x_{0},c),p_{s|0,c}(\cdot|x_{0})]].$ (20) $$

This approach does not suffer from high variance compared to ([17](https://arxiv.org/html/2510.00815v1#S3.E17)),
since both $c$ and $x_{0}$ are fixed. Furthermore, we found that using ([20](https://arxiv.org/html/2510.00815v1#S3.E20)) works well in practice (see our experiments in [Section˜5](https://arxiv.org/html/2510.00815v1#S5)).

### 3.2 Learning to guide

We present here our approach for learning guidance weights $\bm{\omega}$ based on the *self-consistency* loss ([20](https://arxiv.org/html/2510.00815v1#S3.E20)). We also provide empirical evaluation of other approaches in [Section˜5](https://arxiv.org/html/2510.00815v1#S5).

#### Guidance learning objective.

For $(x_{0},c)\sim p_{0,c}(x_{0},c)$, we sample $x_{s}\sim p_{s|0}(\cdot|x_{0})$, i.e., $x_{s}\sim\mathcal{N}(\alpha_{s}x_{0},\sigma^{2}_{s}\mathrm{Id})$.
We also sample $\tilde{x}_{s}(\bm{\omega})\sim p_{s|0,c}^{t,(\theta,\bm{\omega})}$, i.e., sample $\tilde{x}_{t}\sim\mathcal{N}(\alpha_{t}x_{0},\sigma^{2}_{t}\mathrm{Id})$ then

$$ $\displaystyle\tilde{x}_{s}(\bm{\omega})\sim\mathcal{N}(\mu_{s,t}(\hat{x}_{\theta}(\tilde{x}_{t},c;\omega_{c,(s,t)}),\tilde{x}_{t}),\Sigma_{s,t}),$ (21) $$

where $\mu_{s,t}$ and $\Sigma_{s,t}$ are given by DDIM (see ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) in [Appendix˜B](https://arxiv.org/html/2510.00815v1#A2)).
The objective ([20](https://arxiv.org/html/2510.00815v1#S3.E20)) can be written as

$$ $\textstyle\mathcal{L}_{\beta,\lambda}(\bm{\omega})=\mathbb{E}_{(x_{0},c)\sim p_{0,c},s,t\sim p(s,t)}[\mathbb{E}[||\tilde{x}_{s}(\bm{\omega})-x_{s}||^{\beta}_{2}]-\frac{\lambda}{2}\mathbb{E}[||\tilde{x}_{s}(\bm{\omega})-\tilde{x}^{\prime}_{s}(\bm{\omega})||^{\beta}_{2}]],$ (22) $$

where we dropped terms not depending on $\bm{\omega}$, and with independent $\tilde{x}_{s}(\bm{\omega}),\tilde{x}^{\prime}_{s}(\bm{\omega})\sim p_{s|0,c}^{t,(\theta,\bm{\omega})}$.

#### Simplified guidance learning objective.

A special case of ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) with $\beta=2$ and $\lambda=0$, leads to

$$ $\textstyle\mathcal{L}_{\ell_{2}}(\bm{\omega})=\mathcal{L}_{\beta,0}(\bm{\omega})=\mathbb{E}_{(x_{0},c)\sim p_{0,c},s,t\sim p(s,t)}\left[\mathbb{E}[||\tilde{x}_{s}(\bm{\omega})-x_{s}||^{2}_{2}]\right].$ (23) $$

We found that ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) was very effective but more sensitive to hyperparameters than ([22](https://arxiv.org/html/2510.00815v1#S3.E22)). This approach is however cheaper than ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) since it avoids quadratic complexity $O(m^{2})$ of computing interaction terms. We refer the reader to [Appendix˜D](https://arxiv.org/html/2510.00815v1#A4) and to [Algorithm˜3](https://arxiv.org/html/2510.00815v1#alg3) for more details on the use of ([23](https://arxiv.org/html/2510.00815v1#S3.E23)).

#### Guidance network.

The guidance weights $\omega_{c,(s,t)}^{\phi}=\omega(s,t,c;\phi)$ are given by a neural network with parameters $\phi$ . We use ReLU activation at the end to prevent negative guidance weights. For more details, see [Appendix˜F](https://arxiv.org/html/2510.00815v1#A6). We denote $\bm{\omega}^{\phi}=(\omega^{\phi}_{c,(s,t)})_{s,t\in[0,1],t>s}$ and employ $\mathcal{L}_{\beta,\lambda}(\phi)$ instead of $\mathcal{L}_{\beta,\lambda}(\bm{\omega})$.

#### Distribution p ​ ( s , t ) p(s,t) .

We choose target time $s$ to be distributed as $s\sim\mathcal{U}[S_{\min},1-\zeta-\delta]$. We define $\Delta t\sim\mathcal{U}[\delta,1-\zeta-s]$ and we let $t=s+\Delta t$.
Here, $S_{\min}$ controls the minimal time, and $\zeta$ controls how close it gets to $1$.
The parameter $\delta$ controls the distance between time-steps and we found it to be very important, see [Figure˜2](https://arxiv.org/html/2510.00815v1#S5.F2). Even though during inference $|t-s|\approx\frac{1}{T}$ is typically small ($T$ is a number of steps), we found that using larger $\delta\approx 0.1$ during training worked better in practice.

#### Empirical objective.

We sample $\{x^{i}_{0},c^{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}p_{0,c}$ from the training set and we additionally sample noise levels $\{s_{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{U}[S_{\min},1-\zeta-\delta]$, as well as time increments $\{\Delta t_{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{U}[\delta,1-\zeta-s_{i}]$ and we let $t_{i}=s_{i}+\Delta t_{i}$. For each $(x^{i}_{0},s_{i})$, we sample $m$ “particles” $\{x_{s_{i}}^{j}\}_{j=1}^{m}\overset{\mathrm{i.i.d.}}{\sim}p_{s_{i}|0}(\cdot|x_{0}^{i})$ from the *noising process* ([4](https://arxiv.org/html/2510.00815v1#S2.E4)), which defines the target samples. We also produce $m$ “particles” $\{\tilde{x}_{s_{i}}^{j}(\bm{\omega}^{\phi})\}_{j=1}^{m}\overset{\mathrm{i.i.d.}}{\sim}p_{s_{i}|0,c_{i}}^{t_{i},(\theta,\bm{\omega}^{\phi})}(\cdot|x_{0}^{i},c_{i})$ by first sampling $\{\tilde{x}^{j}_{t_{i}}\}_{j=1}^{m}\overset{\mathrm{i.i.d.}}{\sim}p_{t_{i}|0}(\cdot|x_{0}^{i})$ from the *noising process* ([4](https://arxiv.org/html/2510.00815v1#S2.E4)) and then denoising with guidance and DDIM using ([21](https://arxiv.org/html/2510.00815v1#S3.E21)). This defines the proposal samples.
We expand loss function ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) as a function of guidance network parameters $\phi$ defined on the empirical batches as follows (where $\lambda\in[0,1]$ and $\beta\in(0,2)$, see [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1))

$$ $\textstyle\hat{\mathcal{L}}_{\beta,\lambda}(\phi)=\frac{1}{n}\sum_{i=1}^{n}\left[\frac{1}{m}\sum_{j=1}^{m}||\tilde{x}_{s_{i}}^{j}(\bm{\omega}^{\phi})-x_{s_{i}}^{j}||_{2}^{\beta}-\frac{\lambda}{2}\frac{1}{m(m-1)}\sum_{j\neq k}||\tilde{x}_{s_{i}}^{j}(\bm{\omega}^{\phi})-\tilde{x}_{s_{i}}^{k}(\bm{\omega}^{\phi})||_{2}^{\beta}\right]$ (24) $$

Figure: Algorithm 1 Learning to Guide

#### Learning to guide with rewards.

CFG can be used to produce samples with high reward $R(x_{0},c)$, set by a practitioner. Denoising with guidance from $t$ to $s$ as described by ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), gives us an approximation $\hat{x}_{\theta}(x_{t},c;\omega^{\phi}_{c,(s,t)})$ of clean data.
We could use it to optimize guidance weights, by defining the loss $\mathcal{L}_{R}(\phi)=-\mathbb{E}_{(s,t)\sim p(s,t),x_{0},c\sim p(x_{0},c),x_{t}\sim p_{t|0}(\cdot|x_{0})}\left[R\left(\hat{x}_{\theta}(x_{t},c;\omega^{\phi}_{c,(s,t)}),c\right)\right]$.
Directly minimizing this loss may lead to *reward hacking* (Skalse et al.,, 2022). Thus, we regularize this objective using $\hat{\mathcal{L}}_{\beta,\lambda}$. For reward weight $\gamma_{R}\geq 0$, we optimize $\mathcal{L}_{\text{tot}}(\phi)=\hat{\mathcal{L}}_{\beta,\lambda}(\phi)+\gamma_{R}\mathcal{L}_{R}(\phi)$ (see [Algorithm˜4](https://arxiv.org/html/2510.00815v1#alg4)).

## 4 Related work

#### Classifier-Free Guidance.

Using guidance in the sampling of diffusion models was first investigated by Dhariwal and Nichol, (2021). They proposed a classifier guidance method which linearly combines the unconditional score estimate and the input gradient of the log-probability of a (time-varying) classifier. To avoid training such classifier on noisy training data, Ho and Salimans, (2022) proposed Classifier-Free Guidance (CFG), which linearly combines a conditional and unconditional denoisers. By varying the guidance weight, one is able to obtain high-quality samples. This approach has become prominent in the literature, see Adaloglou and Kaiser, (2024) for a recent introduction.

CFG has been extended in many different directions, some of them proposing dynamic mixing strategies to control the guidance weight throughout the sampling process (e.g. Sadat et al., 2024a; Kynkäänniemi et al., (2024); Wang et al., (2024); Shen et al., (2024); Malarz et al., (2025); Li et al., (2024); Koulischer et al., 2025a; Koulischer et al., 2025b; Xia et al., (2025); Sadat et al., 2024b; Zheng and Lan, (2024)). Another line of research focuses on removing the need for the unconditional score component by using either an “inferior" version of the conditional model to provide a negative guidance signal (Karras et al.,, 2024; Adaloglou et al.,, 2025), by leveraging time-step information (Sadat et al.,, 2025), or by enabling the model to act as its own implicit classifier (Tang et al.,, 2025). Other approaches focus on fixing some of the limitations of classifier-free guidance; e.g., Chung et al., (2025) introduced CFG++, a modification to the standard CFG to address off-manifold issues, while Koulischer et al., 2025b show that the true conditional distribution can be obtained by approximating a term corresponding to the derivative of a Rényi divergence.
Very recently, Fan et al., (2025) introduced CFG-Zero$\ast$, a CFG variant for flow-matching models. It leverages an optimized scale factor and a zero-init technique (skipping initial ODE steps) to correct for early velocity inaccuracies, resulting in improved text-to-image/video generation.
*We highlight that our approach could be seamlessly combined with any other guidance technique by simply replacing the definition of the guided denoiser ([9](https://arxiv.org/html/2510.00815v1#S2.E9)).*
Finally, there exists a large literature on correcting CFG with Sequential Monte Carlo (Skreta et al.,, 2025; He et al.,, 2025) or MCMC techniques (Du et al.,, 2023; Moufad et al.,, 2025; Zhang et al.,, 2025). These approaches are mostly orthogonal to other improvements of CFG, and can be combined with most of the dynamic mixing strategies (i.e. Malarz et al., (2025)).

CFG has also been reinterpreted as a predictor-corrector in Bradley and Nakkiran, (2024) and analyzed theoretically in a variety of works (Fu et al.,, 2024; Wu et al.,, 2024; Chidambaram et al.,, 2024; Kong et al.,, 2024; Frans et al.,, 2025). Pavasovic et al., (2025) have shown that while CFG can overshoot the target distribution in low dimension, it can reproduce the target distribution in high dimensions.

Concurrently to our work, Papalampidi et al., (2025) introduced dynamic classifier guidance by selecting the guidance weight among a list of pre-determined guidance weights using online evaluators.

#### Learning CFG weight.

Azangulov et al., (2025) introduced an algorithm based on stochastic control to optimize the guidance weights. However, this algorithm is not scalable as it requires guided backwards trajectories to estimate the gradient.
Concurrently to us, Yehezkel et al., (2025) introduced an annealing guidance scheduler that dynamically adjusts the guidance weight based on the timestep and the magnitude of the conditional noise discrepancy. The method learns an adaptive policy to better balance image quality and alignment with the text prompt throughout the generation process. One of their losses is similar to an alternative approach we explored – guided score matching (see [Section˜C.1](https://arxiv.org/html/2510.00815v1#A3.SS1)). We found that optimizing such loss led to guidance weights equal to zero.

#### Distillation.

Our method shares similarities with distillation approaches such as Diff-Instruct (Luo et al.,, 2023), Variational Score Distillation (Wang et al.,, 2023) and Moment Matching Distillation (MMD) (Salimans et al.,, 2024), which distill a diffusion model by approximately minimizing the KL divergence between the distilled generator and the pretrained teacher model. One could think about our method as a form of distillation where the KL divergence is replaced by a scoring rule, the pretrained teacher model is replaced by the true distribution, and the student is replaced by a guided diffusion. Our methodology could be extended to the distillation setting.

#### Distributional approaches for diffusion models.

Our objective function was motivated by (De Bortoli et al.,, 2025), where ([11](https://arxiv.org/html/2510.00815v1#S2.E11)) is used to learn $p(x_{0}|x_{t},c)$ to obtain distributional diffusion models. Our method is also related to Inductive Moment Matching (IMM) (Zhou et al.,, 2025), where a generative model is trained by enforcing marginal consistency ([16](https://arxiv.org/html/2510.00815v1#S3.E16)).
Instead of directly optimizing for ([16](https://arxiv.org/html/2510.00815v1#S3.E16)), their objective minimizes the distance between $p_{s}^{t,(\theta_{n},\bm{\omega})}$ and $p_{s}^{r,(\theta_{n-1},\bm{\omega})}$, where $n$ is an iteration number and $s<r<t$ is an intermediate time between $s$ and $t$. However, they do not marginalize over $(x_{0},c)\sim p_{0,c}$ as in ([14](https://arxiv.org/html/2510.00815v1#S3.E14)); instead, they sample a batch of $(x_{0},c)$ and use it for both $p_{s}^{t,(\theta_{n},\bm{\omega})}$ and $p_{s}^{r,(\theta_{n-1},\bm{\omega})}$. This means they optimize an objective similar to self-consistency ([19](https://arxiv.org/html/2510.00815v1#S3.E19)), where they use a batch of $(x_{0},c)$ instead of single points.

## 5 Experimental results

In this section, we present experimental results on learning guidance weights. In [Section˜5.1](https://arxiv.org/html/2510.00815v1#S5.SS1), we provide results for image generation benchmarks – ImageNet $64\times 64$ (Deng et al.,, 2009) and CelebA (Liu et al.,, 2015) with resolution $64\times 64$. We provide results on text-to-image (T2I) benchmark - MS-COCO 2014 (Lin et al.,, 2014) ($10$K images) at $512\times 512$ resolution in [Section˜5.2](https://arxiv.org/html/2510.00815v1#S5.SS2). Finally, due to space constraints, we provide 2D Mixture of Gaussians (MoG) results in [Section˜G.1](https://arxiv.org/html/2510.00815v1#A7.SS1).

### 5.1 Image generation

#### Experimental setting.

We evaluate the performance of our method on ImageNet $64\times 64$ and CelebA with resolution $64\times 64$. As evaluation metrics, we use FID (Heusel et al.,, 2017) and Inception Score (IS) (Salimans et al.,, 2016). First, we pretrain diffusion models on ImageNet and CelebA. We then freeze these models and train guidance network $\omega^{\phi}_{c,(s,t)}$ via [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1). We also train a variant of our method where conditioning is omitted, i.e. $\omega^{\phi}_{(s,t)}$.
On top of that, we also train the simplified $\ell_{2}$ method using [Algorithm˜3](https://arxiv.org/html/2510.00815v1#alg3) using the same methodology. We report metrics based on $50k$ samples. Please refer to [Appendix˜F](https://arxiv.org/html/2510.00815v1#A6) for more details.

#### Baselines.

We report performance of the unguided model as well as the model with a constant guidance. We further report performance of limited interval guidance (LIG) (Kynkäänniemi et al.,, 2024). Guidance scale and intervals were selected via grid search for the lowest FID (see [Appendix˜F](https://arxiv.org/html/2510.00815v1#A6)).

**Table 1: ImageNet 64x64. We report FID and IS for different methods (the best are in bold).**
| Method Name | Guidance Weight | FID $\downarrow$ | Inception Score (IS) $\uparrow$ |
| --- | --- | --- | --- |
|  | Baselines |  |  |
| Unguided | $\omega=0$ | 4.46 | 43.52 |
| Constant guidance | $\omega=0.25$ | 2.40 | 66.72 |
| Limited interval guidance | $\omega(t)=0.95$ for $t\in[0.2,0.8]$ | 2.11 | 71.60 |
|  | Learned guidance approaches |  |  |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) | $\omega^{\phi}_{c,(s,t)}$ | 1.99 | $73.62$ |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) | $\omega^{\phi}_{(s,t)}$ | $2.07$ | $76.7$ |
| $\ell_{2}$ objective ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) | $\omega^{\phi}_{c,(s,t)}$ | 2.09 | 75.93 |
| $\ell_{2}$ objective ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) | $\omega^{\phi}_{(s,t)}$ | $2.10$ | 77.55 |

#### Results.

The results for ImageNet $64\times 64$ are given in [Table˜1](https://arxiv.org/html/2510.00815v1#S5.T1) and for CelebA in [Table˜2](https://arxiv.org/html/2510.00815v1#S5.T2). In both cases, the LIG baseline outperforms the other baselines in terms of FID and Inception Score. Our self-consistency approach ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) leads to the best results, especially when the conditioning information is provided. This highlights the importance of adjusting guidance weights for different conditioning.

Figure: Figure 1: Learned guidance weights on ImageNet 64x64. Left, guidance weights $\omega^{\phi}_{(t-dt,t)}$ (conditioning-agnostic) for baselines as well as for self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) and $\ell_{2}$ ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) objectives, where $dt=1/100$. X-axis is time. Right, guidance weights $\omega^{\phi}_{c,(t-dt,t)}$ for specific ImageNet classes.
Refer to caption: x1.png

In [Figure˜1](https://arxiv.org/html/2510.00815v1#S5.F1), left, we visualize the learned conditioning-agnostic guidance weights $\omega^{\phi}_{(t-dt,t)}$ with $dt=1/100$, for self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) and $\ell_{2}$ ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) approaches, as well as for the baselines. Learned guidance weights seem to be positive on a similar interval as LIG, but the shape of the weights is quite different. In [Figure˜1](https://arxiv.org/html/2510.00815v1#S5.F1), right, we visualize guidance weights $\omega^{\phi}_{c,(t-dt,t)}$ for different classes of ImageNet, learned by self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)). First, we arbitrary choose some classes – 0 (tench), 10 (brambling), 100 (black swan) and 999 (toilet tissue). Then, we visualize the ones which achieve the largest and the lowest guidance weights, i.e.
83 (prairie chicken) = $\arg\min_{c}[\max_{t}\omega^{\phi}_{c,(t-dt,t)}]$ and 696 (paintbrush) = $\arg\max_{c}[\max_{t}\omega^{\phi}_{c,(t-dt,t)}]$. For the class $83$ (prairie chicken), the guidance weight is zero. For the class $696$ (paintbrush) it has quite a different behavior compared to others, being more aggressive and positive on a larger interval. Overall, this, variability highlights importance of adjusting guidance weights per conditioning.

**Table 2: CelebA $64\times 64$. We report FID and IS for different methods (the best are in bold).**
| Method Name | Guidance Weight | FID $\downarrow$ | Inception Score (IS) $\uparrow$ |
| --- | --- | --- | --- |
|  | Baselines |  |  |
| Unguided | $\omega=0$ | 2.44 | 2.94 |
| Constant guidance | $\omega=0.01$ | 2.45 | 2.94 |
| Limited interval guidance | $\omega(t)=0.7$ for $t\in[0.0,0.8]$ | 2.37 | 2.96 |
|  | Learned guidance approaches |  |  |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) | $w^{\phi}_{c,(s,t)}$ | 2.10 | 2.98 |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) | $w^{\phi}_{(s,t)}$ | $2.28$ | $2.97$ |
| $\ell_{2}$ objective ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) | $w^{\phi}_{c,(s,t)}$ | 2.36 | 2.95 |
| $\ell_{2}$ objective ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) | $w^{\phi}_{(s,t)}$ | $2.33$ | $2.95$ |

Figure: Figure 2: Ablation over $\delta$ and $S_{\min}$ on ImageNet $64\times 64$. On the X-axis we report values of $S_{\min}$ and on Y-axis we show FID. Each column denotes a method while a color corresponds to a value of $\delta$.
Refer to caption: x2.png

#### Ablations.

In [Figure˜2](https://arxiv.org/html/2510.00815v1#S5.F2), we study impact of $\delta$ and $S_{\min}$ on ImageNet $64\times 64$. We always use $\zeta=0.01$ which introduces a small "safety margin" over the original diffusion model (i.e. $t\in[\zeta,1-\zeta]$ instead of $t\in[0,1]$). The self-consistency approach ([19](https://arxiv.org/html/2510.00815v1#S3.E19)) is less sensitive to the parameters compared to $\ell_{2}$ ([23](https://arxiv.org/html/2510.00815v1#S3.E23)). Overall, very small $\delta=0.01$ leads to worse performance compared to larger ones, motivating us to train with large gaps ($\delta\approx 0.1$) between $s$ and $t$. This finding is surprising because during sampling $|s-t|\sim 0.01$ for $100$ sampling steps. We hypothesize that a larger gap $|s-t|$ provides a more stable and informative gradient signal for the guidance network, which then successfully generalizes to the small-step intervals used during inference thanks to the smoothness of the network $\omega^{\phi}_{c,(\cdot,\cdot)}$.
The performance is not very sensitive to $S_{\min}$, but we found $S_{\min}=0.2$ worked the best.

### 5.2 Text-to-image generation

We evaluate our method on MS COCO 2014 dataset at $512\times 512$ resolution for text-to-image (T2I) task. We pretrain a 1.05B parameter flow matching model (Lipman et al.,, 2023) that uses a Multimodal Diffusion Transformer backbone (BlackForestLabs,, 2025).
We freeze it
and train a guidance network
$\omega^{\phi}_{c_{\text{CLIP}},c_{\text{T5},(s,t)}}$
via [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1), where $c_{\text{CLIP}}$ and $c_{\text{T5}}$ denote CLIP (Radford et al.,, 2021) and T5 (Raffel et al.,, 2020) embeddings of the text prompt. As baseline, we consider manually selected guidance weight $\omega$.
Instead of an empty prompt $\varnothing$ for the
unconditional term ([9](https://arxiv.org/html/2510.00815v1#S2.E9)), we replace it with a fixed negative prompt $c_{\text{neg}}=$ “blurred, blurry, disfigured, ugly, tiling, poorly drawn”. This ’negative guidance’ setup is for both our learned model and
the baseline.
We also consider a setting with a reward function $R(x_{0},c)$ given by CLIP score (Hessel et al.,, 2021) computed between image $x_{0}$ and prompt $c$. We
train guidance network via [Algorithm˜4](https://arxiv.org/html/2510.00815v1#alg4) with $\gamma_{R}=10^{5}$.
We report FID and CLIP Score using $10$K samples. For more experimental details, see [Appendix˜F](https://arxiv.org/html/2510.00815v1#A6).

**Table 3: MS COCO $512\times 512$. FID and CLIP score for different methods (the best are in bold).**
| Method Name | Guidance Weight | FID $\downarrow$ | CLIP Score $\uparrow$ |
| --- | --- | --- | --- |
|  | Baselines |  |  |
| Unguided | $\omega=0$ | 24.74 | 0.278 |
| Constant guidance | $\omega=7.5$ | 31.2 | 0.306 |
|  | Our approaches |  |  |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) | $\omega^{\phi}_{c_{\text{CLIP}},c_{\text{T5},(s,t)}}$ | 18.01 | $0.295$ |
| Self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) + CLIP Score reward | $\omega^{\phi}_{c_{\text{CLIP}},c_{\text{T5},(s,t)}}$ | 28.37 | 0.306 |

#### Results.

The quantitative results are summarized in [Table˜3](https://arxiv.org/html/2510.00815v1#S5.T3).
Our method outperforms unguided and guided baselines in terms of FID, which is consistent to image experiments. However, it achieves a lower CLIP score than a guided baseline. Adding CLIP score reward leads a similar CLIP score as guided baseline, but achieves lower FID.
We provide qualitative results in Figures ([4](https://arxiv.org/html/2510.00815v1#A1.F4))-([6](https://arxiv.org/html/2510.00815v1#A1.F6)), see [Appendix˜A](https://arxiv.org/html/2510.00815v1#A1). Our method generates images that are more realistic and better aligned with the text prompts. The learned guidance weights are shown in [Figure˜3](https://arxiv.org/html/2510.00815v1#S5.F3). We observe high variability depending on the prompt.

Figure: Figure 3: Learned guidance weights on MS COCO $512\times 512$ trained with self-consistency ([22](https://arxiv.org/html/2510.00815v1#S3.E22)) and CLIP reward loss. Please refer to [Figures˜4](https://arxiv.org/html/2510.00815v1#A1.F4) and [5](https://arxiv.org/html/2510.00815v1#A1.F5) for the corresponding images.
Refer to caption: figures/guidance_weights_coco_no_face.png

## 6 Discussion

In this paper, we presented an approach to learn CFG weights $\omega_{c,(s,t)}$ as a function of conditioning $c$ and times $s$ and $t$, using the self-consistency condition ([19](https://arxiv.org/html/2510.00815v1#S3.E19)). This rather strong condition is motivated by a weaker marginal consistency condition ([16](https://arxiv.org/html/2510.00815v1#S3.E16)), which is satisfied by the true backwards diffusion process. Our approach yields guidance weights that improve FID on image generation tasks – ImageNet $64\times 64$ and CelebA $64\times 64$. Our analysis reveals that guidance weights vary significantly depending on the conditioning information, implying that CFG with learnable, conditioning-dependent weights can improve conditional sampling performance.

We extended our methodology to text-to-image tasks with a reward function given by the CLIP score. We found that our approach leads to highly variable prompt-dependent guidance weights and visually provides better prompt alignment compared to baselines. Quantitatively, however, we found that the performance was close to a baseline with a manually selected guidance weight function.

Future work will focus on theoretical understanding of our objective function and its guidance solutions. Moreover, we will explore alternative reward functions for better prompt-image alignment, and investigate the impact of different guidance approaches. We hope our work motivates further research into how time- and conditioning-dependent guidance weights affect the sampled distributions.

## References

- Abramson et al., (2024)
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024).
Accurate structure prediction of biomolecular interactions with AlphaFold 3.
Nature, 630(8016):493–500.
- Adaloglou and Kaiser, (2024)
Adaloglou, N. and Kaiser, T. (2024).
An overview of classifier-free guidance for diffusion models.
theaisummer.com.
- Adaloglou et al., (2025)
Adaloglou, N., Kaiser, T., Iagudin, D., and Kollmann, M. (2025).
Guiding a diffusion model using sliding windows.
arXiv preprint arXiv:2411.10257.
- Azangulov et al., (2025)
Azangulov, I., Potaptchik, P., Li, Q., Aamari, E., Deligiannidis, G., and Rousseau, J. (2025).
Adaptive diffusion guidance via stochastic optimal control.
arXiv preprint arXiv:2505.19367.
- BlackForestLabs, (2025)
BlackForestLabs (2025).
Flux.1 kontext: Flow matching for in-context image generation and editing in latent space.
arXiv preprint arXiv:2506.15742.
- Bouchacourt et al., (2016)
Bouchacourt, D., Mudigonda, P. K., and Nowozin, S. (2016).
DISCO nets: Dissimilarity coefficients networks.
In Advances in Neural Information Processing Systems.
- Bradley and Nakkiran, (2024)
Bradley, A. and Nakkiran, P. (2024).
Classifier-free guidance is a predictor-corrector.
arXiv preprint arXiv:2408.09000.
- Chidambaram et al., (2024)
Chidambaram, M., Gatmiry, K., Chen, S., Lee, H., and Lu, J. (2024).
What does guidance do? a fine-grained analysis in a simple setting.
In Advances in Neural Information Processing Systems.
- Chung et al., (2025)
Chung, H., Kim, J., Park, G. Y., Nam, H., and Ye, J. C. (2025).
CFG++: Manifold-constrained classifier free guidance for diffusion models.
In International Conference on Learning Representations.
- De Bortoli et al., (2025)
De Bortoli, V., Galashov, A., Guntupalli, J. S., Zhou, G., Murphy, K., Gretton, A., and Doucet, A. (2025).
Distributional diffusion models with scoring rules.
In International Conference on Machine Learning.
- Deng et al., (2009)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009).
Imagenet: A large-scale hierarchical image database.
In IEEE Conference on Computer Vision and Pattern Recognition.
- Dhariwal and Nichol, (2021)
Dhariwal, P. and Nichol, A. (2021).
Diffusion models beat GANs on image synthesis.
In Advances in Neural information Processing Systems.
- Du et al., (2023)
Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S. (2023).
Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC.
In International Conference on Machine Learning.
- Esser et al., (2024)
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., and Rombach, R. (2024).
Scaling rectified flow transformers for high-resolution image synthesis.
In International Conference on Machine Learning.
- Fan et al., (2025)
Fan, W., Zheng, A. Y., Yeh, R. A., and Liu, Z. (2025).
Cfg-zero*: Improved classifier-free guidance for flow matching models.
arXiv preprint arXiv:2503.18886.
- Frans et al., (2025)
Frans, K., Park, S., Abbeel, P., and Levine, S. (2025).
Diffusion guidance is a controllable policy improvement operator.
arXiv preprint arXiv:2505.23458.
- Fu et al., (2024)
Fu, H., Yang, Z., Wang, M., and Chen, M. (2024).
Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory.
arXiv preprint arXiv:2403.11968.
- Google, (2025)
Google (2024-2025).
Veo | AI video generator | generative AI on vertex AI.
Google Cloud Documentation.
- Gretton et al., (2012)
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012).
A kernel two-sample test.
Journal of Machine Learning Research, 13(1):723–773.
- He et al., (2025)
He, J., Hernández-Lobato, J. M., Du, Y., and Vargas, F. (2025).
RNE: a plug-and-play framework for diffusion density estimation and inference-time control.
arXiv preprint arXiv:2506.05668.
- Hessel et al., (2021)
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. (2021).
Clipscore: A reference-free evaluation metric for image captioning.
In Empirical Methods in Natural Language Processing.
- Heusel et al., (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017).
GANs trained by a two time-scale update rule converge to a local Nash equilibrium.
In Advances in Neural Information Processing Systems.
- Ho et al., (2020)
Ho, J., Jain, A., and Abbeel, P. (2020).
Denoising diffusion probabilistic models.
In Advances in Neural Information Processing Systems.
- Ho and Salimans, (2022)
Ho, J. and Salimans, T. (2022).
Classifier-free diffusion guidance.
arXiv preprint arXiv:2207.12598.
- Karras et al., (2024)
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine, S. (2024).
Guiding a diffusion model with a bad version of itself.
In Advances in Neural Information Processing Systems.
- Kim et al., (2025)
Kim, B., Hsieh, Y.-G., Klein, M., Cuturi, M., Ye, J. C., Kawar, B., and Thornton, J. (2025).
Simple reflow: Improved techniques for fast flow models.
In International Conference on Learning Representations.
- Kingma et al., (2021)
Kingma, D., Salimans, T., Poole, B., and Ho, J. (2021).
Variational diffusion models.
In Advances in Neural Information Processing Systems.
- Kingma and Gao, (2023)
Kingma, D. P. and Gao, R. (2023).
Understanding diffusion objectives as the ELBO with simple data augmentation.
arXiv preprint arXiv:2303.00848.
- Kong et al., (2024)
Kong, L., Du, Y., Mu, W., Neklyudov, K., De Bortoli, V., Wu, D., Wang, H., Ferber, A., Ma, Y.-A., Gomes, C. P., et al. (2024).
Diffusion models as constrained samplers for optimization with unknown constraints.
arXiv preprint arXiv:2402.18012.
- (30)
Koulischer, F., Deleu, J., Raya, G., Demeester, T., and Ambrogioni, L. (2025a).
Dynamic negative guidance of diffusion models.
In International Conference on Learning Representations.
- (31)
Koulischer, F., Handke, F., Deleu, J., Demeester, T., and Ambrogioni, L. (2025b).
Feedback guidance of diffusion models.
arXiv preprint arXiv:2506.06085.
- Kynkäänniemi et al., (2024)
Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. (2024).
Applying guidance in a limited interval improves sample and distribution quality in diffusion models.
In Advances in Neural information Processing Systems.
- Li et al., (2024)
Li, T., Luo, W., Chen, Z., Ma, L., and Qi, G.-J. (2024).
Self-guidance: Boosting flow and diffusion generation on their own.
arXiv preprint arXiv:2412.05827.
- Lin et al., (2014)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014).
Microsoft COCO: Common objects in context.
In European Conference on Computer Vision, pages 740–755. Springer.
- Lipman et al., (2023)
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. (2023).
Flow matching for generative modeling.
In International Conference on Learning Representations.
- Liu et al., (2015)
Liu, Z., Luo, P., Wang, X., and Tang, X. (2015).
Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV).
- Luo et al., (2023)
Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z. (2023).
Diff-instruct: a universal approach for transferring knowledge from pre-trained diffusion models.
In Advances in Neural Information Processing Systems.
- Malarz et al., (2025)
Malarz, D., Kasymov, A., Zięba, M., Tabor, J., and Spurek, P. (2025).
Classifier-free guidance with adaptive scaling.
arXiv preprint arXiv:2502.10574.
- Moufad et al., (2025)
Moufad, B., Janati, Y., Durmus, A., Ghorbel, A., Moulines, E., and Olsson, J. (2025).
Conditional diffusion models with classifier-free Gibbs-like guidance.
arXiv preprint arXiv:2505.21101.
- Papalampidi et al., (2025)
Papalampidi, P., Wiles, O., Ktena, I., Shtedritski, A., Bugliarello, E., Kajic, I., Albuquerque, I., and Nematzadeh, A. (2025).
Dynamic classifier-free diffusion guidance via online feedback.
- Pavasovic et al., (2025)
Pavasovic, K. L., Verbeek, J., Biroli, G., and Mezard, M. (2025).
Understanding classifier-free guidance: High-dimensional theory and non-linear generalizations.
arXiv preprint arXiv:2502.07849.
- Peebles and Xie, (2023)
Peebles, W. and Xie, S. (2023).
Scalable diffusion models with transformers.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205.
- Radford et al., (2021)
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021).
Learning transferable visual models from natural language supervision.
In International Conference on Machine Learning.
- Raffel et al., (2020)
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020).
Exploring the limits of transfer learning with a unified text-to-text transformer.
Journal of Machine Learning Research, 21(140):1–67.
- Rombach et al., (2022)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition.
- Ronneberger et al., (2015)
Ronneberger, O., Fischer, P., and Brox, T. (2015).
U-net: Convolutional networks for biomedical image segmentation.
In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer.
- (47)
Sadat, S., Buhmann, J., Bradley, D., Hilliges, O., and Weber, R. M. (2024a).
CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.
In International Conference on Learning Representations.
- (48)
Sadat, S., Hilliges, O., and Weber, R. M. (2024b).
Eliminating oversaturation and artifacts of high guidance scales in diffusion models.
In International Conference on Learning Representations.
- Sadat et al., (2025)
Sadat, S., Kansy, M., Hilliges, O., and Weber, R. M. (2025).
No training, no problem: Rethinking classifier-free guidance for diffusion models.
In International Conference on Learning Representations.
- Saharia et al., (2022)
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., and Salimans, T. (2022).
Photorealistic text-to-image diffusion models with deep language understanding.
In Advances in Neural Information Processing Systems.
- Salimans et al., (2016)
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016).
Improved techniques for training GANs.
In Advances in Neural Information Processing Systems.
- Salimans et al., (2024)
Salimans, T., Mensink, T., Heek, J., and Hoogeboom, E. (2024).
Multistep distillation of diffusion models via moment matching.
In Advances in Neural Information Processing Systems.
- Sejdinovic et al., (2013)
Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. (2013).
Equivalence of distance-based and RKHS-based statistics in hypothesis testing.
The Annals of Statistics, pages 2263–2291.
- Shen et al., (2024)
Shen, D., Song, G., Xue, Z., Wang, F.-Y., and Liu, Y. (2024).
Rethinking the spatial inconsistency in classifier-free diffusion guidance.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
- Skalse et al., (2022)
Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. (2022).
Defining and characterizing reward hacking.
In Advances in Neural Information Processing Systems.
- Skreta et al., (2025)
Skreta, M., Akhound-Sadegh, T., Ohanesian, V., Bondesan, R., Aspuru-Guzik, A., Doucet, A., Brekelmans, R., Tong, A., and Neklyudov, K. (2025).
Feynman-Kac correctors in diffusion: Annealing, guidance, and product of experts.
In International Conference on Machine Learning.
- Sohl-Dickstein et al., (2015)
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015).
Deep unsupervised learning using nonequilibrium thermodynamics.
In International Conference on Machine Learning.
- (58)
Song, J., Meng, C., and Ermon, S. (2021a).
Denoising diffusion implicit models.
In International Conference on Learning Representations.
- Song and Ermon, (2019)
Song, Y. and Ermon, S. (2019).
Generative modeling by estimating gradients of the data distribution.
In Advances in Neural Information Processing Systems.
- (60)
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b).
Score-based generative modeling through stochastic differential equations.
In International Conference on Learning Representations.
- Sriperumbudur et al., (2010)
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. R. (2010).
Hilbert space embeddings and metrics on probability measures.
Journal of Machine Learning Research, 11(Apr):1517–1561.
- Székely and Rizzo, (2004)
Székely, G. and Rizzo, M. (2004).
Testing for equal distributions in high dimension.
InterStat, 5.
- Tang et al., (2025)
Tang, Z., Bao, J., Chen, D., and Guo, B. (2025).
Diffusion models without classifier-free guidance.
arXiv preprint arXiv:2502.12154.
- Wang et al., (2024)
Wang, X., Dufour, N., Andreou, N., Cani, M.-P., Abrevaya, V. F., Picard, D., and Kalogeiton, V. (2024).
Analysis of classifier-free guidance weight schedulers.
Transactions on Machine Learning Research.
- Wang et al., (2023)
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. (2023).
Prolificdreamer: high-fidelity and diverse text-to-3d generation with variational score distillation.
In Advances in Neural Information Processing Systems.
- Watson et al., (2023)
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. (2023).
De novo design of protein structure and function with RFdiffusion.
Nature, 620(7976):1089–1100.
- Wu et al., (2024)
Wu, Y., Chen, M., Li, Z., Wang, M., and Wei, Y. (2024).
Theoretical insights for diffusion guidance: A case study for Gaussian mixture models.
In International Conference on Machine Learning.
- Xia et al., (2025)
Xia, M., Xue, N., Shen, Y., Yi, R., Gong, T., and Liu, Y.-J. (2025).
Rectified diffusion guidance for conditional generation.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13371–13380.
- Yehezkel et al., (2025)
Yehezkel, S., Dahary, O., Voynov, A., and Cohen-Or, D. (2025).
Navigating with annealing guidance scale in diffusion space.
arXiv preprint arXiv:2506.24108.
- Zhang et al., (2025)
Zhang, X., Lin, H., Ye, H., Zou, J., Ma, J., Liang, Y., and Du, Y. (2025).
Inference-time scaling of diffusion models through classical search.
arXiv preprint arXiv:2505.23614.
- Zheng and Lan, (2024)
Zheng, C. and Lan, Y. (2024).
Characteristic guidance: Non-linear correction for diffusion model at large guidance scale.
In International Conference on Machine Learning.
- Zhou et al., (2025)
Zhou, L., Ermon, S., and Song, J. (2025).
Inductive moment matching.
In International Conference on Machine Learning.

## Organization of the appendix

In [Appendix˜A](https://arxiv.org/html/2510.00815v1#A1), we present the image samples for text-to-image experiment, see [Section˜5](https://arxiv.org/html/2510.00815v1#S5). Then, in [Appendix˜B](https://arxiv.org/html/2510.00815v1#A2), we briefly cover the DDIM framework. Following that, in [Appendix˜C](https://arxiv.org/html/2510.00815v1#A3), we discuss alternative approaches for learning guidance weights. In [Appendix˜D](https://arxiv.org/html/2510.00815v1#A4), we discuss the simple approach based on $\ell_{2}$ objective ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) and in [Appendix˜E](https://arxiv.org/html/2510.00815v1#A5), we discuss the extension of our method to reward-guided setting. Finally, in [Appendix˜F](https://arxiv.org/html/2510.00815v1#A6), we cover all the experimental details and in [Appendix˜G](https://arxiv.org/html/2510.00815v1#A7), we present additional experimental results.

## Appendix A Text to image samples

We provide qualitative results for the text-to-image experiments in Figure [4](https://arxiv.org/html/2510.00815v1#A1.F4), Figure [5](https://arxiv.org/html/2510.00815v1#A1.F5) and in Figure [6](https://arxiv.org/html/2510.00815v1#A1.F6).

Figure: (a) Man performing stunt on a skateboard on a road.
Refer to caption: images/no_cfg/00033.png

Figure: (a) A light brown horse’s face is shown at close range.
Refer to caption: images/no_cfg/00001.png

Figure: (a) A small cutting board and knife with a cut apple.
Refer to caption: images/no_cfg/00185.png

Figure: (a) Two birds sit on top of a parked car.
Refer to caption: images/no_cfg/00253.png

## Appendix B DDIM framework

#### Diffusion models.

The goal of conditional diffusion models is to sample from a target conditional distribution $p_{0}(x|c)$ on $\mathbb{R}^{d}$, which we denote as $p_{0|c}$. Here $c$ is some user-specified conditioning signal (i.e., a class label or a text prompt) and $p(c)$ is a conditioning distribution.

We adopt here the Denoising Diffusion Implicit Models (DDIM) framework of Song et al., 2021a. Let $x_{t_{0}}\sim p_{0|c}$ and define the process $x_{t_{1}:t_{N}}:=(x_{t_{1}},...,x_{t_{N}})$ by

$$ $p(x_{t_{1}:t_{N}}|x_{t_{0}})=p(x_{t_{N}}|x_{t_{0}})\prod_{k=1}^{N-1}p(x_{t_{k}}|x_{t_{0}},x_{t_{k+1}}),$ (25) $$

with $0=t_{0}<\dots<t_{N}=1$. For $0\leq s<t\leq 1$, we let

$$ $p(x_{s}|x_{0},x_{t})=\mathcal{N}(x_{s};\mu_{s,t}(x_{0},x_{t}),\Sigma_{s,t}),$ (26) $$

The mean and covariance of $p(x_{s}|x_{0},x_{t})$ are given by

$$ $\displaystyle\mu_{s,t}(x_{0},x_{t})=(\varepsilon^{2}r_{1,2}(s,t)+(1-\varepsilon^{2})r_{0,1})x_{t}+\alpha_{s}(1-\varepsilon^{2}r_{2,2}(s,t)-(1-\varepsilon^{2})r_{1,1}(s,t))x_{0},$ (27) $\displaystyle\Sigma_{s,t}=\sigma_{s}^{2}(1-(\varepsilon^{2}r_{1,1}(s,t)+(1-\varepsilon^{2}))^{2})\mathrm{Id},$ (28) $$

with $r_{i,j}(s,t)=\left(\alpha_{t}/\alpha_{s}\right)^{i}\left(\sigma_{s}^{2}/\sigma_{t}^{2}\right)^{j}$; see (Song et al., 2021a,) and Appendix F in (De Bortoli et al.,, 2025). The parameter $\varepsilon\in[0,1]$ in ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) is a *churn* parameter which interpolates between a *deterministic* process ($\varepsilon=0$) and a *stochastic* one ($\varepsilon=1$). This ensures that for any $t\in[0,1]$,

$$ $p(x_{t}|x_{0})=\mathcal{N}(x_{t};\alpha_{t}x_{0},\sigma_{t}^{2}\mathrm{Id}),$ (29) $$

for $\alpha_{t},\sigma_{t}$ such that $\alpha_{0}=\sigma_{1}=1$ and $\alpha_{1}=\sigma_{0}=0$. In particular, this guarantees that $p(x_{1}|x_{0})=\mathcal{N}(x_{1};0,\mathrm{Id})$.

The data $x_{t_{0}}\sim p_{0|c}$ is generated by sampling $x_{t_{N}}\sim\mathcal{N}(0,\mathrm{Id})$ and
$x_{t_{k}}\sim p(\cdot|x_{t_{k+1}})$ for $k=N-1,...,0$, where for $0\leq s<t\leq 1$

$$ $p(x_{s}|x_{t},c)=\int_{\mathbb{R}^{d}}p(x_{s}|x_{0},x_{t})p(x_{0}|c,x_{t})\mathrm{d}x_{0}.$ (30) $$

Figure: Algorithm 2 CFG version of DDIM (sampling)

## Appendix C Alternative approaches for learning guidance

We describe the alternative approaches for learning guidance weights and discuss the their potential downfalls.

### C.1 Guided score matching

The most straightforward approach for learning guidance weights is to simply use guided denoiser ([9](https://arxiv.org/html/2510.00815v1#S2.E9)) in the score matching. This leads to the following objective

$$ $\mathcal{L}(\phi)=\int_{0}^{1}\lambda(t)\mathbb{E}_{(x_{0},c)\sim p(x_{0},c),x_{t}\sim p_{t|0}(\cdot|x_{0})}[\|x_{0}-\hat{x}_{\theta}(x_{t},c;\omega^{\phi}_{c,t})\|^{2}]\mathrm{d}t,$ (31) $$

where we omit the dependence of the guidance weights on $s$, i.e., $\omega^{\phi}_{c,(s,t)}=\omega^{\phi}_{c,t}$. However, this approach is doomed to fail, because at convergence, the solution to ([31](https://arxiv.org/html/2510.00815v1#A3.E31)) satisfies the following property

$$ $\hat{x}_{\theta}(x_{t},c;\omega^{\phi}_{c,t})\approx\mathbb{E}[x_{0}|x_{t},c],$ (32) $$

when $x_{t}\sim p_{t|0}(x_{t}|x_{0})$ is a sample from noising process ([4](https://arxiv.org/html/2510.00815v1#S2.E4)). However, the condition ([32](https://arxiv.org/html/2510.00815v1#A3.E32)) is also satisfied by the unguided score, when $x_{t}\sim p_{t|0}(x_{t}|x_{0})$, i.e.

$$ $\hat{x}_{\theta}(x_{t},c)\approx\mathbb{E}[x_{0}|x_{t},c].$ (33) $$

By equality of rhs of ([33](https://arxiv.org/html/2510.00815v1#A3.E33)) and of ([32](https://arxiv.org/html/2510.00815v1#A3.E32)), we have $\forall x_{t}\sim p_{t|0}(x_{t}|x_{0}),c\sim p(c),x_{0}\sim p_{0|c}(\cdot|c),t\in[0,1]$,

$$ $\hat{x}_{\theta}(x_{t},c;\omega^{\phi}_{c,t})-\hat{x}_{\theta}(x_{t},c)=\omega^{\phi}_{c,t}\left(\hat{x}_{\theta}(x_{t},c)-\hat{x}_{\theta}(x_{t},\varnothing)\right)\approx 0$ (34) $$

We then assume that $\Delta_{\theta}(x_{t},c)=\hat{x}_{\theta}(x_{t},c)-\hat{x}_{\theta}(x_{t},\varnothing)\neq 0$. It is a reasonable assumption since if $\Delta_{\theta}(x_{t},c)=0$, the CFG would not have had any effect. This implies that

$$ $\omega^{\phi}_{c,t}=0,\quad\forall c\sim p(c),t\in[0,1]$ (35) $$

The only regime where ([35](https://arxiv.org/html/2510.00815v1#A3.E35)) will not hold is when

$$ $\hat{x}_{\theta}(x_{t},c)\not\approx\mathbb{E}[x_{0}|x_{t},c]$ (36) $$

In such a case, one could still find guidance weights for ([32](https://arxiv.org/html/2510.00815v1#A3.E32)) to hold. But this is a regime of a very under-trained model. In our image experiments (see [Section˜5](https://arxiv.org/html/2510.00815v1#S5)), we initially tried using this approach and found that it was leading to zero guidance weights ([35](https://arxiv.org/html/2510.00815v1#A3.E35)).

#### Connection to backward KL \mathrm{KL} .

Another explanation for the behavior above is that as we train ([31](https://arxiv.org/html/2510.00815v1#A3.E31)), we only observe $x_{t}\sim p(x_{t}|x_{0})$. The form of ([31](https://arxiv.org/html/2510.00815v1#A3.E31)) comes from optimizing backward $\mathrm{KL}$

$$ $\mathrm{KL}[p_{[0,1]}||p_{[0,1]}^{\theta,\bm{\omega}}]=\int_{0}^{1}\int\int||x_{0}-\hat{x}_{\theta}(x_{t},c;\bm{\omega})||^{2}p(x_{t}|x_{0})p(x_{0}|c)\mathrm{d}x_{0}\mathrm{d}x_{t}\mathrm{d}t,$ (37) $$

where $p_{[0,1]}^{\theta,\bm{\omega}}$ is the distribution of all the trajectories obtained by running guided backwards diffusion, and $p_{[0,1]}$ is the distribution of all the trajectories from the true diffusion process.

When we use CFG during sampling, we actually may observe noisy samples $\hat{x}_{t}$ which do not coincide with any $x_{t}\sim p_{t|0}$ form the noising process ([4](https://arxiv.org/html/2510.00815v1#S2.E4)). This hints that instead of backwards $\mathrm{KL}$ ([37](https://arxiv.org/html/2510.00815v1#A3.E37)), we should actually aim to optimize the forward $\mathrm{KL}$

$$ $\mathrm{KL}[p_{[0,1]}^{\theta,\bm{\omega}}||p_{[0,1]}].$ (38) $$

An approach based on optimizing ([38](https://arxiv.org/html/2510.00815v1#A3.E38)) was considered in Azangulov et al., (2025). This approach, however, is very computationally expensive since it requires sampling full guided trajectory from the guided backwards process and then compute the gradients on it.

### C.2 Conditional consistency

Here, we describe an approach similar to *marginal-consistency* ([16](https://arxiv.org/html/2510.00815v1#S3.E16)), but which however could be more effective in practice in principle due to lower variance. We fix conditioning $c$ and we denote by

$$ $p_{s|0,c}^{t,(\theta,\bm{\omega})}(x_{s}|x_{0},c)=\int p_{s|t,c}^{(\theta,\bm{\omega})}(x_{s}|x_{t},c)p_{t|0}(x_{t}|x_{0})\mathrm{d}x_{t},$ (39) $$

which is the term under integral in ([14](https://arxiv.org/html/2510.00815v1#S3.E14)) depending on $(x_{0},c)$. We now consider a *conditional* consistency condition where we fix $c$ and marginalize over $x_{0}$

$$ $\int p_{s|0,c}^{t,(\theta,\bm{\omega})}(x_{s}|x_{0},c)p_{0|c}(x_{0}|c)\mathrm{d}x_{0}=p_{s|c}^{t,(\theta,\bm{\omega})}(x_{s}|c)\approx p_{s|c}(x_{s}|c)=\int p_{s|0}(x_{s}|x_{0})p_{0|c}(x_{0}|c)\mathrm{d}x_{0}.$ (40) $$

The main difference with ([16](https://arxiv.org/html/2510.00815v1#S3.E16)) is that we fixed conditioning $c$. We could use a similar loss

$$ $\mathcal{L}_{c}(\bm{\omega})=\mathbb{E}_{c\sim p_{c},(s,t)\sim p(s,t)}[\mathrm{MMD}_{(\beta,\lambda)}(p_{s|c}^{t,(\theta,\bm{\omega})}(\cdot|c),p_{s|c}(\cdot|c))],$ (41) $$

which can be optimized by drawing conditional samples $x_{0}|c$. While this approach is expected to have a lower variance than ([17](https://arxiv.org/html/2510.00815v1#S3.E17)), since we do not need to sample $c$, it requires to be able to produce conditional samples which is not always available in practice during training (i.e., in text-to-image applications). In practice, we also found that it did not lead to good results. We hypothesize that it is due to still quite a high variance of the gradients of $\mathcal{L}_{c}(\bm{\omega})$.

## Appendix D L2 approach for learning guidance

We describe here the algorithm based on the $\ell_{2}$ objective function ([23](https://arxiv.org/html/2510.00815v1#S3.E23))

$$ $\displaystyle\mathcal{L}_{\ell_{2}}(\bm{\omega})$ $\displaystyle=\mathbb{E}_{(x_{0},c)\sim p_{0,c},s,t\sim p(s,t)}\left[\mathbb{E}[||\tilde{x}_{s}(\bm{\omega})-x_{s}||^{2}_{2}]\right]$ (42) $\displaystyle=\mathbb{E}_{(x_{0},c)\sim p_{0,c},s,t\sim p(s,t)}\left[\left\|\mathbb{E}[\tilde{x}_{s}(\bm{\omega})]-\mathbb{E}[x_{s}]\right\|^{2}+\textup{Trace}(\textup{Cov}(\tilde{x}_{s}(\bm{\omega})))+\textup{Trace}(\textup{Cov}(x_{s}))\right]$ (43) $$

So, by minimizing this loss function, we attempt to match on averages the means of $\tilde{x}_{s}(\bm{\omega})$ and $x_{s}$ while minimizing the expected sum of variances of the component of $\tilde{x}_{s}(\bm{\omega})$.

#### Empirical objective.

We sample $\{x^{i}_{0},c^{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}p_{0,c}$ from the training set and we additionally sample noise levels $\{s_{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{U}[S_{\min},1-\zeta-\delta]$, as well as time increments $\{\Delta t_{i}\}_{i=1}^{n}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{U}[\delta,1-\zeta-s_{i}]$ and we let $t_{i}=s_{i}+\Delta t_{i}$. For each $(x^{i}_{0},s_{i})$, we sample $x_{s_{i}}\sim p_{s_{i}|0}(\cdot|x_{0}^{i})$ from the *noising process* ([4](https://arxiv.org/html/2510.00815v1#S2.E4)), which defines the target samples. We then also produce $\tilde{x}_{s_{i}}(\bm{\omega}^{\phi})\sim p_{s_{i}|0,c_{i}}^{t_{i},(\theta,\bm{\omega}^{\phi})}(\cdot|x_{0}^{i},c_{i})$ by first sampling $\tilde{x}_{t_{i}}\overset{\mathrm{i.i.d.}}{\sim}p_{t_{i}|0}(\cdot|x_{0}^{i})$ from the *noising process* ([4](https://arxiv.org/html/2510.00815v1#S2.E4)) and then denoising with guidance and DDIM using ([21](https://arxiv.org/html/2510.00815v1#S3.E21)). This defines the proposal samples. We expand loss function ([23](https://arxiv.org/html/2510.00815v1#S3.E23)) as a function of guidance network parameters $\phi$ defined on the empirical batches as follows

$$ $\hat{\mathcal{L}}_{\ell_{2}}(\phi)=\frac{1}{n}\sum_{i=1}^{n}\left[||\tilde{x}_{s_{i}}(\bm{\omega}^{\phi})-x_{s_{i}}||_{2}^{2}\right],$ (44) $$

The full algorithm is given in [Algorithm˜3](https://arxiv.org/html/2510.00815v1#alg3).

Figure: Algorithm 3 Learning to Guide with $\ell_{2}$ loss

## Appendix E Learning to guide with rewards

In this section, we describe how our guidance learning approach is extended to a setting, where a reward function $R(x_{0},c)$ is available, see [Algorithm˜4](https://arxiv.org/html/2510.00815v1#alg4) below.

Figure: Algorithm 4 Learning to Guide with reward

## Appendix F Experimental details

#### Noise process.

For all the experiments, we use recitifed flow noise process ([2](https://arxiv.org/html/2510.00815v1#S2.E2)) where $\alpha_{t}=(1-t)$ and $\sigma_{t}=t$.

#### Guidance network architecture for MoG, ImageNet and CelebA.

For guidance network $\omega(s,t,c;\phi)=\omega^{\phi}_{(c,(s,t))}$ we use a $6$ layers MLP with hidden size of $64$ followed by an output layer transforming it to a dimension $1$. As an activation we use "GeLU". We also apply a "ReLU" activation after the last layer of the guidance network to ensure that the gudiance weights are non-negative. For 2D experiments, we do not use ReLU activation and allow the weights to be negative.

The time which is processed by the weight network is first converted to $\log\text{SNR}$, i.e. $s\rightarrow\log\text{SNR}(s)$ and $t\rightarrow\log\text{SNR}(t)$. We do not use sinusoidal embedding as typically is done with diffusion models, since we found that it worked significantly worse. Then the two time-steps are concatenated and passed through to a 2-layer MLP with hidden dimension 256 and an output dimension 512 and GeLU activation. We also use a dropout with rate $0.3$ in the middle of this MLP.

#### 2D MoG.

We use $4$ layer MLP with hidden dimension $64$ and GeLU activation for the denoiser network. The time $t$ for the backbone, is first embedded into $\log\text{SNR}$ and then we use sinusoidal embedding dimension 128. We use adam optimizer with learning rate $10^{-4}$ and batch size $128$. We use maximum norm clipping for Adam to be $1$. When we do pretraining in the *well-trained* regime, we use $10k$ iterations, and when we do pretraining in the *under-trained* regime, we use $250$ iterations. We train guidance network for 1000 iterations, with learning rate $5e-4$, norm clipping $1$ and Adam optimizer. We use $m=32$ particles. We use batch size $128$. We use backwards DDIM ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) with $\varepsilon=1$ in order to sample $x_{s}$ with guidance ([21](https://arxiv.org/html/2510.00815v1#S3.E21)). To produce [Fig.˜9](https://arxiv.org/html/2510.00815v1#A7.F9) and [Fig.˜10](https://arxiv.org/html/2510.00815v1#A7.F10), we sample data with velocity sampler with $10$ steps. We use $4096$ samples in total.

#### ImageNet.

We pretrain the U-Net model (Ronneberger et al.,, 2015) similar to DDPM (Ho et al.,, 2020) which has channel dimension 192, 3 residual blocks, channel multiplier (1, 2, 3, 4), attention which is applied with (False, False, True, False) residual blocks (whenever it is True, we use attention), dropout which is applied at (False, True, True, True) residual blocks (whenever it is True, we use dropout). The dropout rate is $0.1$. We also use sigmoid weighting (Kingma and Gao,, 2023) with bias $b=2$. The model predicts the velocity and the loss is based on velocity. We train it with batch size $128$ for $7M$ iterations. We use Adam optimizer with learning rate $1e-5$ and norm clipping $1$. We use EMA decay of $0.9999$.

For training guidance we reload the pretrained diffusion model with EMA parameters and freeze it. We train the guidance network for $100K$ iterations with batch size $B=256$. We use $m=4$ number of particles and $\beta=1.75,\lambda=1$ parameters. We use learning rate $1e-5$ with adam optimizer and norm clipping $1$. We use EMA decay of $0.9999$. For self-consistency loss ([22](https://arxiv.org/html/2510.00815v1#S3.E22)), in order to produce $\tilde{x}_{s}(\bm{\omega)}$ in ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), we use DDIM sampler ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) with $\varepsilon=1$. We sweep over $\delta\in[0.01,0.1,0.2,0.3]$, $S_{\min}\in[0.01,0.1,0.2,0.3]$ parameters. We use $\zeta=1e-2$ which is the same parameter as the safety parameter for the original diffusion model. This safety parameter in the original diffusion model just controls the time interval to be $(\zeta,1-\zeta)$ instead of $(0,1)$, for numerical stability. To sample data, we use DDIM ([21](https://arxiv.org/html/2510.00815v1#S3.E21)) with $\varepsilon=0$ for all the experiments. During training, we track FID on a small subset of $2048$ images. We observed that the performance over time could be unstable for some hyperparameters, therefore we use this training-time FID to keep the best checkpoint for every hyperparameter. We report the performance for the best hyperparameters based on FID, we use $50k$ samples for FID computation. On top of that, we also train the simplified $\ell_{2}$ method using [Algorithm˜3](https://arxiv.org/html/2510.00815v1#alg3) using the same methodology, except that in order to produce $\tilde{x}_{s}(\bm{\omega)}$ in ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), we use DDIM sampler ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) but with $\varepsilon=0$, since it led to a better empirical performance.

#### CelebA.

We pretrain the U-Net model (Ronneberger et al.,, 2015) similar to DDPM (Ho et al.,, 2020) which has channel dimension 256, 2 residual blocks, channel multiplier (1, 2, 2, 2), attention which is applied with (False, True, False, False) residual blocks (whenever it is True, we use attention), dropout which is applied at (False, True, True, True) residual blocks (whenever it is True, we use dropout). The dropout rate is $0.2$. We also use sigmoid weighting (Kingma and Gao,, 2023) with bias $b=1$. We do not transform time to $\log\text{SNR}$ for time embedding in the diffusion network. The model predicts the $x_{0}$ and the loss is based on velocity. We train it with batch size $128$ for $300K$ iterations. We use Adam optimizer with learning rate $1e-4$ and norm clipping $1$. We use EMA decay of $0.9999$.

For training guidance we reload the pretrained diffusion model with EMA parameters and freeze it. We train the guidance network for $100K$ iterations with batch size $B=256$. We use $m=4$ number of particles and $\beta=1.75,\lambda=1$ parameters. We use learning rate $1e-5$ with adam optimizer and norm clipping $1$. We use EMA decay of $0.9999$. For self-consistency loss ([22](https://arxiv.org/html/2510.00815v1#S3.E22)), in order to produce $\tilde{x}_{s}(\bm{\omega)}$ in ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), we use DDIM sampler ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) with $\varepsilon=1$. We sweep over $\delta\in[0.01,0.1,0.2,0.3]$, $S_{\min}\in[0.01,0.1,0.2,0.3]$ parameters. We use $\zeta=1e-2$ which is the same parameter as the safety parameter for the original diffusion model. This safety parameter in the original diffusion model just controls the time interval to be $(\zeta,1-\zeta)$ instead of $(0,1)$, for numerical stability. To sample data, we use DDIM with $\varepsilon=0$ for all the experiments. During training, we track FID on a small subset of $2048$ images. We observed that the performance over time could be unstable for some hyperparameters, therefore we use this training-time FID to keep the best checkpoint for every hyperparameter. We report the performance for the best hyperparameters based on FID, we use $50k$ samples for FID computation. On top of that, we also train the simplified $\ell_{2}$ method using [Algorithm˜3](https://arxiv.org/html/2510.00815v1#alg3) using the same methodology, except that in order to produce $\tilde{x}_{s}(\bm{\omega)}$ in ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), we use DDIM sampler ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) but with $\varepsilon=0$, since it led to a better empirical performance.

#### Guidance network architecture for MS-COCO.

For guidance network $\omega(s,t,c_{\text{CLIP}},c_{\text{T5}};\phi)=\omega^{\phi}_{c_{\text{CLIP}},c_{\text{T5},(s,t)}}$ we use a $6$ layers MLP with hidden size of $512$ followed by an output layer transforming it to a dimension $1$. As an activation we use "GeLU". We also apply a "ReLU" activation after the last layer of the guidance network to ensure that the guidance weights are non-negative.

The time steps $t$ and $s$ are each processed by separate networks. We do not use sinusoidal embedding to get the time embeddings. We also do not convert time to $\log\text{SNR}$ as we did for ImageNet and CelebA. Each of the two time-steps are passed through to a 2-layer MLP with hidden dimension and output dimension of $256$ and a SiLU activation. The two timestep embeddings are then concatenated. We also pass CLIP and T5 embeddings through a similar MLP with an output dimension of $256$ and $512$ respectively. Both time embedding and CLIP embedding are concatenated to a dimension $512$ and then concatenated with the text embedding. This is passed through the MLP layers described above.

#### MS-COCO.

We pretrain a 1.05B parameter Flux text-to-image backbone (BlackForestLabs,, 2025), which uses an architecture based on Esser et al., (2024) and consists of 30 Diffusion Transformer (DiT) blocks (Peebles and Xie,, 2023) and 15 MMDiT blocks (Esser et al.,, 2024).
We do not use any weighting, nor do we transform time with logSNR. The model is trained with the standard flow matching loss and predicts velocity. The CLIP embeddings used for training the model are extracted from the standard ViT-L/14 CLIP transformer architecture. The T5 embeddings are extracted from the standard T5-XXL architecture.

We load the pretrained flow matching model with EMA parameters and freeze it to train the guidance network. We train the guidance network for $50K$ iterations with a batch size of $256$.
We set the number of particles as $m=4$, $\beta=1.75$ and $\lambda=1.0$. We use learning rate of $8e-5$ with Adam optimizer. We set the Adam parameters $\beta=(0.9,0.999)$, $\epsilon=1e-8$, and weight decay=$0.01$.
For self-consistency loss ([22](https://arxiv.org/html/2510.00815v1#S3.E22)), in order to produce $\tilde{x}_{s}(\bm{\omega)}$ in ([21](https://arxiv.org/html/2510.00815v1#S3.E21)), we use DDIM sampler ([28](https://arxiv.org/html/2510.00815v1#A2.E28)) with $\varepsilon=0$.
We set $\delta=0.2$, $S_{\text{min}}=0.01$ and safety parameter $\zeta=1e-2$ while training the model.
To sample data from flow matching model, we use the standard Euler ODE solve in the time interval $[0,1]$ and use $128$ sampling steps. To select the best checkpoint, we tracked CLIP Score over $3K$ samples and then used the checkpoint corresponding to the best CLIP Score on this subset to compute CLIP Score over $10K$ samples.

## Appendix G Additional experiments and results

### G.1 2D Mixture of Gaussians (MoG)

We consider a 4 components MoG with the means $(10,10)$, $(-10,10)$, $(10,-10)$ and $(-10,-10)$, as well as the diagonal covariances $\sigma^{2}\mathrm{Id}$ where the corresponding variances $\sigma^{2}$ are equal to $5,1,1,1$. The data is visualized in Figure [8](https://arxiv.org/html/2510.00815v1#A7.F8), left.

Figure: Figure 8: Mixture of Gaussians data. Red cross corresponds to the locations of the mean, while the numbers denote respective numbering of mixture components. The blue color corresponds to the data, while green color corresponds to the samples from diffusion model.
Refer to caption: x3.png

We study two regimes, *well-trained* and *under-trained*. In a *well-trained* regime, the diffusion model is trained for long enough in order to produce the samples close to the training distribution $p_{0|c}$, while in *under-trained*, it is trained for a small amount of iterations and produces samples far away from the training distribution. See [Figure˜8](https://arxiv.org/html/2510.00815v1#A7.F8), center and right for corresponding samples. Please refer to Appendix [F](https://arxiv.org/html/2510.00815v1#A6) for all experimental details.

We train $\omega^{\phi}_{c,(s,t)}$ with [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1). The results are provided in [Figure˜9](https://arxiv.org/html/2510.00815v1#A7.F9). In [Figure˜9](https://arxiv.org/html/2510.00815v1#A7.F9), A, we visualize the $\mathrm{MMD}$ between samples from data distributions and samples produced by diffusion model with different constant guidance weights $\omega$. In the *well-trained* regime, as we increase the guidance weight $\omega$, it degrades the $\mathrm{MMD}$, because it pushes the samples away from the distribution. What is intriguing is that in *under-trained* regime, $\mathrm{MMD}$ decreases as the guidance weight $\omega$ increases, achieving the lowest value with the learned guidance weight $\omega^{\phi}_{c,(s,t)}$. This finding highlights one of our main message – CFG can be used to correct the mismatch of the sampled distribution and the data distribution. In [Figure˜9](https://arxiv.org/html/2510.00815v1#A7.F9), B, we visualize the learned $\omega^{\phi}_{c,(t-dt,t)}$, where $dt$ is a time discretization. In the *well-trained* regime, the learned guidance weights tend to be close to 0, while in *under-trained* regime, the learned guidance weights reach quite high values. In [Figure˜10](https://arxiv.org/html/2510.00815v1#A7.F10), we visualize the corresponding samples. This finding highlights that in an under trained regime, high guidance weights are required.

Figure: Figure 9: A: $\mathrm{MMD}$ between samples ($4096$) from data distribution and samples produced by a diffusion model with different guidance weight. Blue dashed line denotes the performance of unguided model, while brown dashed line denotes the performance of a model guided with learned $\omega^{\phi}_{c,(t-dt,t)}$. B: learned guidance weight $\omega^{\phi}_{c,(t-dt,t)}$ from [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1) for different mixture components.
Refer to caption: x4.png

Figure: Figure 10: Samples with and without guidance. In blue, we show the original data, in green we show samples from unguided model while in purple we show samples from the guided model with learned guidance $\omega^{\phi}_{c,(s,t)}$ with [Algorithm˜1](https://arxiv.org/html/2510.00815v1#alg1). First column corresponds to *under-trained* regime while second column corresponds to *well-trained* one.
Refer to caption: x6.png

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: - How does this method compare to guidance Interval approaches? The guidance interval method empirically finds an optimal guidance schedule, assigning near-zero guidance weight in early steps and large weights in the final few steps. Since the proposed method can measure how well the estimated distribution matches the true one via its loss, it may be possible to evaluate the optimality of the interval-based scheduling using the same loss metric. If so, this could open new directions for systematic guidance interval optimization beyond the work of LIG.
- Once the optimal guidance weights are obtained through this method, could they be further distilled into a single model to achieve both high performance and inference efficiency?
soundness: 3
strengths: - The idea of learning the optimal guidance weight continuously via a neural network, rather than relying on grid search for optimal guidance intervals or empirical adjustments, is highly novel.
- Despite the inherent risk of the guidance weight collapsing to a trivial solution, the successful stabilization of learning via the newly proposed self-consistency loss is a significant achievement. The simplified L2 objective (Eq. 21) also appears surprisingly straightforward to implement.
summary: This paper proposes a method to learn the classifier-free guidance (CFG) weight by setting it as the output of a neural network. While previous CFG approaches relied on heuristics or trial-and-error to find an optimal (often large) weight, which could lead to saturation and degraded sample quality, this work optimizes guidance weights as a continuous function of time and condition. The objective is to minimize the distributional mismatch between the generated guided samples and the noised true conditional samples, using a novel self-consistency loss. This loss successfully mitigates the high-variance problem associated with theoretical approaches, leading to stable training and demonstrated performance gains across various datasets, from toy examples to COCO T2I sampling.
weaknesses: The proposed method involves a large number of hyperparameters to determine the optimal guidance weight, and additionally requires performing $m$ rounds of noising and comparison at every denoising step, which introduces significant computational overhead. Given this, I wonder whether the computational cost difference compared to existing guidance distillation methods is actually substantial. It would be valuable to include a cost comparison table or discussion.

Moreover, since the loss function in this paper formalizes the matching between the true sample distribution and the guided one, I suspect that a parameter-efficient fine-tuning (PEFT) approach such as LoRA could optimize a student model directly using the same loss formulation, without needing a separate guidance network at inference time. This would result in a more inference-efficient model, and I’m curious whether the authors have considered or experimented with this alternative.

Regarding the design of the guidance network, the paper focuses heavily on the loss formulation for learning guidance weights but provides limited discussion or justification for the network’s architecture. The rationale behind its input-output design is unclear. Based on Table 1 and Fig. 1, 3, the conditioning input seems to have a stronger influence than expected, suggesting that incorporating the currently sampled image as an additional input could further improve weight estimation. Were such variants or conditioning combinations explored?

In the COCO experiments, the guidance network uses the text encoder output as a conditioning signal, and Fig. 3 shows that the learned guidance weights vary across prompts. Does this imply that the network has learned semantic understanding of the text embeddings? If so, how heavy must the model be to capture such semantics effectively? Additionally, from the qualitative results on COCO, the outputs often resemble those with weak or no guidance, which may indicate that the diffusion model’s original objective still struggles to fully capture the conditional distribution, similar to previous findings.

Finally, in the loss formulation, the authors mention using $m=4$ particles for computing the MMD in the image domain. Even if self-consistency reduces variance, I question whether the MMD loss computed from only four samples can be sufficiently accurate. Was this number chosen primarily to reduce computational cost? If so, it would be helpful to report how performance varies with different particle counts, to justify the design choice.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How sensitive are results to the kernel/parameters in the MMD objective versus the simpler L2 variant?

2. What are the training and inference costs attributable to guidance learning and self-consistency evaluation?

3. Since the learned weights vary strongly with prompt, do certain semantic categories systematically demand higher guidance?

4. Can practitioners easily inspect how guidance evolves with time and prompt?
soundness: 3
strengths: It reframes classifier-free guidance tuning as a distribution-matching problem, replacing hand-crafted schedules with a learned, time- and prompt-aware policy—an original and practical angle.
The self-consistency objective is elegant and low-variance, making the method easy to implement on top of existing diffusion backbones without architectural changes.
Experiments span multiple datasets and backbones and include sensible ablations, showing consistent gains over unguided, fixed-guidance, and limited-interval baselines.
The learned guidance schedules are interpretable (varying with prompt and timestep), which improves clarity and aids real-world debugging.
Overall, the approach offers a broadly applicable and deployable improvement to conditional diffusion sampling with a favorable quality-to-complexity trade-off.
summary: This paper puts forward learning a classifier-free guidance schedule that varies with both time and the input condition (class or text), enabling the sampler to better match the desired conditional distribution.
Compare with handcrafting guidance strengths, the method optimizes a self-consistency objective: if one adds noise to a real image and then denoises it with guidance, the intermediate result should match what the forward noising process would produce at that timestep.
Following this desin mechnism, this reframes guidance tuning as a distribution-matching problem and naturally extends to reward-guided generation by combining an external reward (e.g., CLIP-based) with the same self-consistency regularizer.
Imortantly, exntensive experiments on standard image datasets and text-to-image benchmarks show consistent quality gains over unguided sampling, fixed guidance, and simple time-limited guidance, with learned schedules that adapt across prompts and timesteps.
weaknesses: 1. The paper argues that enforcing self-consistency is stronger and low-variance, but the conditions under which minimizing the proposed loss actually improves conditional sampling are not formalized.

2. On text-to-image, self-consistency plus CLIP reward improves FID yet struggles to surpass strong guided baselines on CLIP score.

3. Learning a guidance network and evaluating the self-consistency loss add additional overhead.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: - What do the authors mean by a high-variance objective?
- What is the training cost of training the learned guidance network as a percentage of diffusion model training cost?
- Why is the FID on COCO for the constant guidance model so high? FID 31 seems incredibly high for a 1B model. How was this model trained?
soundness: 3
strengths: - The authors replace a hyper-parameter (guidance strength) with a learned approach
- The objective function is well theoretically motivated
summary: The authors propose a method for learning the guidance schedule for diffusion model generation. In particular, they train a shallow network to minimize the distributional mismatch between a frozen denoiser and the true distribution. The authors provide solid theoretical motivation for their choice of objective function, and also experimental evidence for class and text-conditional models.
weaknesses: - The authors train their own diffusion models for use in evaluations. It would greatly strengthen the paper if they also showed FID and reward improvements for existing pre-trained diffusion models from prior work. For example:
     - EDM for cifar-10 (Karras et al, https://github.com/NVlabs/edm)
     - EDM-2 for ImageNet (Karras et al, https://github.com/NVlabs/edm2)
     - MicroDiffusion for text-to-image (Sehwag et al, https://github.com/SonyResearch/micro_diffusion)
- The authors compare their learned approach to simple baselines such as constant or limited-interval guidance. It would strengthen the paper if they also showed improvements over stronger un-trained baselines such as in Wang et al (https://arxiv.org/abs/2404.13040).

My score recommends reject, but I would be happy to raise my score to accept if these additional experiments are included in the paper and my questions below are addressed.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: Beyond the Weaknesses part, I am curious about the general rule of the guidance weights. Current analyses in Fig. 3 is poor. Could the authors provide more quantitative analyses about the weights under different conditions and timesteps? For example, what if a text condition gradually becomes complex and a long caption? What about the mean or variance?
soundness: 2
strengths: - The paper is well structured and easy to follow. All technical details are carefully discussed and clear.
- The motivation is intuitive and effective, alleviation the discrepancy between CFG-guided and the ground-truth distributions is capable of improving the performance of guided sampling.
- The further discussion about different objective is detailed and inspiring, encouraging future study for better training efficiency and synthesis performance.
summary: The paper proposes to introduce the Maximum Mean Discrepancy (MMD) to quantitatively evaluate the distance between the CFG-guided distribution and the ground-truth one. By doing so, it is enabled to design both time-dependent and condition-dependent guidance weight, which is more flexible and controllable. To realize such a pipeline, the authors employ an auxiliary network for guidance weight prediction, and further discuss the influences under different objectives. Both qualitative and quantitative results confirm the superiority of the proposed method.
weaknesses: - The whole pipeline of narrowing the distribution discrepancy by optimizing some scalars is not new. Beyond the main objective being directly motivated by prior works, similar idea of canceling mismatch (or "self-consistency") between forward diffusion and reverse denoising stages has also been proposed [1].

  [1] Towards More Accurate Diffusion Model Acceleration with A Timestep Tuner. Xia et al., CVPR 2024.

- The main concern about the proposed method is the optimality. There is no theoretical analysis about the optimality or convergence about the proposed objective. Then why the optimized guidance weight could guarantee the self-consistency? What further confirms my worries is in Appendix C.1. The most intuitive objective is the L2 norm in Eq. (23), which is consistent with the vanilla training loss of diffusion models, *i.e.*, the ELBO loss. However, as the authors themselves claim, such an objective leads to zero guidance weight. That is to say, **diffusion model converges well and both conditional and unconditional scores are accurate**, thus there is no need to employ CFG. Under this discussion, I am curious about the theoretical optimality of Eq. (20) or Eq. (21). Could the authors provide theoretical analyses under some trivial toy data to verify the correctness of the method? From my opinion, Eq. (21) is somewhat an upper bound of ELBO (*i.e.*, Eq. (23)). Given that ELBO itself is an upper bound of KL-divergence (by the native theory of diffusion models), the optimization of a looser bound may lead to meaningless results.

- What further weakens the soundness of the paper is the employment of an auxiliary network. To be honest, such a setting could lead to more flexible guidance weight given different conditions. However, what if the network fails to converge well and only predicts sub-optimal guidance weight? The accuracy issue is more severe under open-vocabulary setting, *i.e.*, text-to-image or text-to-video. Considering that the authors employ a light-weight network, such a concern is also crucial. Could the author provide some closed-form expressions or some empirical solutions avoiding the employment of an auxiliary network?

- The qualitative results of text-to-image is somewhat not convincing. The improvements in Figs. 4-7 are inconspicuous. This strengthens my concerns above.


Decision: Accept (Poster)