Iterative Importance Fine-tuning of Diffusion Models

## Abstract

Abstract Diffusion models are an important tool for generative modelling, serving as effective priors in applications such as imaging and protein design. A key challenge in applying diffusion models for downstream tasks is efficiently sampling from resulting posterior distributions, which can be addressed using the h ℎ h italic_h -transform. This work introduces a self-supervised algorithm for fine-tuning diffusion models by estimating the h ℎ h italic_h -transform, enabling amortised conditional sampling. Our method iteratively refines the h ℎ h italic_h -transform using a synthetic dataset resampled with path-based importance weights. We demonstrate the effectiveness of this framework on class-conditional sampling, inverse problems and reward fine-tuning for text-to-image diffusion models.

## 1 Introduction

Diffusion models have emerged as a powerful tool for generative modelling (Ho et al.,, 2020; Dhariwal and Nichol,, 2021). As training these models is expensive and requires large amount of data, fine-tuning existing models for new tasks is of interest. In particular, given large pre-trained foundation models such as Stable Diffusion (Rombach et al.,, 2022) for images or RFDiffusion (Watson et al.,, 2023) for protein generation, the goal is to use the model as a prior for various downstream applications. For example, in imaging applications, the same diffusion model might be used as a prior for inpainting, deblurring or super-resolution tasks (Rout et al.,, 2024).

Sampling from the resulting conditional distribution can be interpreted as sampling from a tilted distribution (Domingo-Enrich et al.,, 2025), where the distribution defined by the pre-trained diffusion models $p_{\text{data}}({\bm{x}})$ is tilted by some reward or likelihood function $r:{\mathbb{R}}^{n}\to{\mathbb{R}}$. Then, we define the tilted distribution as

$$ $\displaystyle p_{\text{tilted}}({\bm{x}})\propto p_{\text{data}}({\bm{x}})\exp \left(\frac{r({\bm{x}})}{\lambda}\right),$ (1) $$

with a temperature $\lambda>0$. This general framework includes several applications, such as statistical inverse problems with $r({\bm{x}})=\ln p({\bm{y}}|{\bm{x}})$ as the log-likelihood given observed measurements ${\bm{y}}$; class conditional sampling with $r({\bm{x}})=\ln p(c|{\bm{x}})$ as the log-class probabilities for a class $c$ or reward fine-tuning where $r({\bm{x}})$ is learned explicitly as an image reward (Xu et al.,, 2024). The tilted distribution can also be expressed as the minimiser of an optimisation problem, i.e.,

$$ $\displaystyle p_{\text{tilted}}=\arg\min_{p}\left\{-\mathbb{E}_{{\bm{x}}\sim p }[r({\bm{x}})]+\lambda\mathrm{KL}(p,p_{\text{data}})\right\}.$ (2) $$

Here, the first term maximises the reward, and the second term acts as a regulariser with the temperature $\lambda$ as the regularisation strength. In the following, we directly incorporate $\lambda$ into the reward $r$ for an easier notation.

We can sample from the tilted distribution by adding an additional term, the generalised $h$-transform, to the drift function of the reverse SDE (Vargas et al., 2023b,). The $h$-transform is in general intractable in closed form and thus many works propose approximations, see for example (Jalal et al.,, 2021; Chung et al.,, 2023; Rout et al.,, 2024). As an alternative, Denker et al., (2024) propose a supervised framework to estimate the $h$-transform given a small dataset from the tilted distribution. However, this limits the method to domains where such a dataset is available.

### 1.1 Contributions

In order to sample from the tilted distribution, we propose a self-supervised framework for estimating the $h$-transform without access to samples from the tilted distribution.
Starting with an initial estimate of $h$, our method consists out of the following three steps.

- 1.
First, we *sample a batch of trajectories from the diffusion model* with our current estimate of the $h$-transform.
- 2.
Next, we use path-based importance weights to *filter the dataset* by rejecting samples which do not fit to the tilted distribution. Consequently, the filtered dataset describes the filtered distribution better than the original one. The corresponding importance weights can be computed on the fly during the first step and do not cause additional costs.
- 3.
Finally, we use the supervised fine-tuning loss from Denker et al., (2024) to *update our estimation of the $h$-transform*.

From a theoretical viewpoint, we prove in Theorem [5](https://arxiv.org/html/2502.04468v2#Thmtheorem5) that our iterative importance fine-tuning decreases a certain loss function in each iteration, where the global minimiser of this loss function is known to coincide with the $h$-transform.
Finally, we apply our method for class conditional sampling on MNIST, super-resolution, and reward-based fine-tuning of stable diffusion. We emphasize that our importance-based fine-tuning *does not require to differentiate the score function of the base-model and never considers more than one step of the generation process at once.* In particular, it can be applied for very large base-models where other methods run out of memory.

### 1.2 Related Work

#### Controlled Generation from Diffusion Models

Controlled generation for diffusion models can be achieved via inference-time or post-training methods, see the recent review by Uehara et al., (2025). Inference-time methods, such as classifier guidance (Dhariwal and Nichol,, 2021) or reconstruction guidance (Chung et al.,, 2023), guide the reverse diffusion process without additional training but typically increase computational cost and are often sensitive to hyperparameters (Song et al.,, 2024). Post-training techniques instead fine-tune models for a specific application. Fine-tuning comes with a higher initial computational cost but often results in reduced sampling time compared to inference-time methods (Denker et al.,, 2024). Supervised post-training methods require an additional task-specific dataset for fine-tuning (Ruiz et al.,, 2023; Zhang et al.,, 2023; Xu et al.,, 2024). In contrast, online post-training methods directly optimise some objective given by the reward function via reinforcement learning (Venkatraman et al.,, 2024; Clark et al.,, 2024; Fan et al.,, 2024; Black et al.,, 2024) or stochastic optimal control (Denker et al.,, 2024; Domingo-Enrich et al.,, 2025; Pidstrigach et al.,, 2025).
Denoising diffusion samplers for via optimal control losses were considered by Albergo et al., (2023); Albergo and Vanden-Eijnden, (2024); Berner et al., (2024); Blessing et al., (2025); Vargas et al., 2023a; Zhang and Chen, (2022).

#### Iterative retraining can cause model degradation

Iterative retraining of generative models on synthetic data has been shown to lead to performance degradation, including phenomena such as mode collapse (Alemohammad et al.,, 2024; Shumailov et al.,, 2023). Strategies to mitigate this issue have been proposed, such as mixing synthetic data with real data (Bertrand et al.,, 2024). Alternatively, training on curated synthetic datasets, i.e., choosing the synthetic data based on the reward $r$, has also been demonstrated to improve retraining stability and model performance (Dong et al.,, 2023; Ferbach et al.,, 2024). As we re-sample the synthetic dataset based on the importance weights, our approach also falls into the category of retraining using curated data. However, as opposed to Ferbach et al., (2024) our selection process incorporates both the reward and the path probability, which can mitigate risks associated with iterative retraining.

#### Self-supervised training for sampling

Recently, a number of self-supervised frameworks have been proposed for sampling from unnormalised densities, for example FAB (Midgley et al.,, 2023) or iDEM (Akhound-Sadegh et al.,, 2024). In iDEM, the authors propose an iterative framework, to train a diffusion model with a stochastic regression loss based on data sampled from the current estimation of the diffusion model. An importance-based acceptance-rejection scheme for improving the quality of generative models for sampling was proposed by Hertrich and Gruhlke, (2024).

### 1.3 Outline

The rest of the paper is structured as follows. In Section [2](https://arxiv.org/html/2502.04468v2#S2) we give the necessary background on diffusion models and supervised fine-tuning. The path-based importance weights and the resampling step are presented in Section [3](https://arxiv.org/html/2502.04468v2#S3). We present experiments on class conditional sampling for MNIST, super-resolution, and reward-based fine-tuning of Stable Diffusion (Rombach et al.,, 2022) in Section [4](https://arxiv.org/html/2502.04468v2#S4). Conclusions are drawn in Section [5](https://arxiv.org/html/2502.04468v2#S5).

## 2 Background

Let us first recap the score-based generative modelling framework of Song et al., (2021); we start with a forward SDE, which progressively transforms the target distribution $p_{\mathrm{data}}$

$$ $\displaystyle{\mathrm{d}}{\bm{X}}_{t}$ $\displaystyle=f_{t}({\bm{X}}_{t})\,{\mathrm{d}}t+\sigma_{t}\leavevmode\hbox to 15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 7.91669pt\lower-3. 47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.41669pt}{5.97221pt}\pgfsys@lineto{7.91669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{7.91669pt}{5.97221pt}\pgfsys@lineto{6.41669 pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{X}}_{0} \sim p_{\mathrm{data}},$ (3) $$

with drift $f_{t}$ and diffusion $\sigma_{t}$. As usual, we denote by $p_{t}$ the density of the solution ${\bm{X}}_{t}$ at time $t$ and by $\leavevmode\hbox to5.13pt{\vbox to8.85pt{\pgfpicture\makeatletter\hbox{\hskip 2
.51563pt\lower-3.12498pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-2.01563pt}{5.62498pt}\pgfsys@lineto{2.51563
pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{2.51563pt}{5.62498pt}\pgfsys@lineto{1.01563
pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t_{1}|t_{2}}({\bm{x}}
_{t_{1}}|{\bm{x}}_{t_{2}})$ the conditional densities of ${\bm{X}}_{t_{1}}$ given ${\bm{X}}_{t_{2}}$. Under some regularity assumptions, there exists by Anderson, (1982) a corresponding reverse SDE, that allows sampling from ${\mathbb{P}}_{T}$ (typically ${\mathcal{N}}(0,\mathbf{I}_{n})$) and denoising them to generate samples from $p_{\mathrm{data}}$. The reverse SDE is given by

$$ $\displaystyle{\mathrm{d}}{\bm{X}}_{t}$ $\displaystyle=\left(f_{t}({\bm{X}}_{t})-\sigma_{t}^{2}s_{t}({\bm{X}}_{t}) \right)\,{\mathrm{d}}t+\sigma_{t}\leavevmode\hbox to15.93pt{\vbox to9.54pt{ \pgfpicture\makeatletter\hbox{\hskip 8.01668pt\lower-3.47221pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{X}}_{T} \sim{\mathbb{P}}_{T},$ (4) $$

where the time flows backwards and $s_{t}({\bm{x}})=\nabla_{\bm{x}}\ln p_{t}({\bm{x}})$ is the score function.

To sample from the tilted distribution $p_{\mathrm{tilted}}$ instead of $p_{\mathrm{data}}$, we consider an additional guidiance term in the reverse SDE. More precisely, we consider the SDE

$$ $\displaystyle{\mathrm{d}}{\bm{H}}_{t}$ $\displaystyle=\left(f_{t}({\bm{H}}_{t})-\sigma_{t}^{2}\left(s_{t}({\bm{H}}_{t} )+h_{t}({\bm{H}}_{t})\right)\right)\,{\mathrm{d}}t+\sigma_{t}\leavevmode\hbox to 15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 8.01668pt\lower-3. 47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},$ (5) $$

where the time flows again backwards and we use ${\bm{H}}_{t}$ for the guided reverse SDE.
To ensure that ${\bm{H}}_{0}\sim p_{\mathrm{tilted}}$, Denker et al., (2024); Vargas et al., 2023b considered a generalisation of Doob’s $h$-transform. Note that Zhao et al., (2025) prove a similar results to Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (iii) from a reinforcement learning perspective.

###### Theorem 1 (Proposition 2.2 and Theorem 3.1 in Denker et al.,, 2024 ) .

Assume $Z_{r}\coloneqq\int\exp(r({\bm{x}}_{0}))\mathrm{d}{\bm{x}}_{0}<\infty$. Further, let $Q_{T}^{f_{t}}[p_{\mathrm{tilted}}]\coloneqq\int\leavevmode\hbox to5.13pt{\vbox
to
8.85pt{\pgfpicture\makeatletter\hbox{\hskip 2.51563pt\lower-3.12498pt\hbox to0
.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill
{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }
\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-2.01563pt}{5.62498pt}\pgfsys@lineto{2.51563
pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{2.51563pt}{5.62498pt}\pgfsys@lineto{1.01563
pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{T|0}({\bm{x}}|{\bm{x}
}_{0})p_{\mathrm{tilted}}({\bm{x}}_{0})\mathrm{d}{\bm{x}}_{0}$ and

$$ $h_{t}^{*}({\bm{x}})=\nabla_{\bm{x}}\ln p_{t}^{r}({\bm{x}}),\quad\text{where} \quad p^{r}_{t}({\bm{x}})=\int\frac{\exp(r({\bm{x}}_{0}))}{Z_{r}}\leavevmode \hbox to5.13pt{\vbox to8.85pt{\pgfpicture\makeatletter\hbox{\hskip 2.61562pt \lower-3.12498pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.51563pt}{5.62498pt}\pgfsys@lineto{2.01563 pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.51563pt}{5.62498pt}\pgfsys@lineto{-1.0156 3pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{0|t}({\bm{x}}_{0}|{ \bm{x}}){\mathrm{d}}{\bm{x}}_{0}.$ (6) $$

Then, the following holds true:

- (i)
Let $({\bm{H}}_{t})_{t}$ be the solution of the SDE ([5](https://arxiv.org/html/2502.04468v2#S2.E5)) with $h_{t}=h_{t}^{*}$ given in ([6](https://arxiv.org/html/2502.04468v2#S2.E6)) and ${\bm{H}}_{T}\sim Q_{T}^{f_{t}}[p_{\mathrm{tilted}}]$. Then, it holds ${\bm{H}}_{0}\sim p_{\mathrm{tilted}}$.
- (ii)
It holds that $h_{t}^{*}$ is the unique minimiser of the score-matching (SM) loss function
$\displaystyle\mathcal{L}_{SM}(h_{t})=\mathop{\mathbb{E}}_{\begin{subarray}{c}{
\bm{x}}_{0}\sim p_{\mathrm{tilted}}\\
t\sim\mathrm{U}(0,T),{\bm{x}}_{t}\sim\leavevmode\hbox to3.62pt{\vbox to6.22pt{
\pgfpicture\makeatletter\hbox{\hskip 1.76094pt\lower-2.18748pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-1.76094pt}{-0.8264pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}
{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-1.41093pt}{3.93748pt}\pgfsys@lineto{1.76094
pt}{3.93748pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{1.76094pt}{3.93748pt}\pgfsys@lineto{0.71094
pt}{3.23749pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}(\cdot|{\bm{x}}_{
0})\end{subarray}}\left[\left\|\left({h_{t}({\bm{x}}_{t})}\!+\!s_{t}({\bm{x}}_
{t})\right)\!-\!\nabla_{{\bm{x}}_{t}}\ln\leavevmode\hbox to5.13pt{\vbox to8.85
pt{\pgfpicture\makeatletter\hbox{\hskip 2.51563pt\lower-3.12498pt\hbox to0.0pt
{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-2.01563pt}{5.62498pt}\pgfsys@lineto{2.51563
pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{2.51563pt}{5.62498pt}\pgfsys@lineto{1.01563
pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}({\bm{x}}_{t}|{
\bm{x}}_{0})\right\|^{2}\right].$
(7)
- (iii)
It holds that
$h_{t}^{*}\in\operatorname*{arg\,min}_{h_{t}}\mathcal{F}({\mathbb{P}}_{h}),
\quad\text{where}\quad\mathcal{F}({\mathbb{P}})=\mathrm{KL}({\mathbb{P}},{
\mathbb{P}}_{\mathrm{data}})-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}}[r({
\bm{x}}_{0})]$
(8)
and where ${\mathbb{P}}_{h}$ and ${\mathbb{P}}_{\mathrm{data}}$ are the path measures of the corresponding SDEs ([5](https://arxiv.org/html/2502.04468v2#S2.E5)) and ([4](https://arxiv.org/html/2502.04468v2#S2.E4)).

###### Remark 2 .

In Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (i) the terminal distribution is given as $Q_{T}^{f_{t}}[p_{\mathrm{tilted}}]$ instead of ${\mathbb{P}}_{T}$ in Equation ([4](https://arxiv.org/html/2502.04468v2#S2.E4)) to remove the value function bias. In Domingo-Enrich et al., (2025), the authors deal with the value function bias by altering the noise of the controlled SDE. However, due to the mixing time in the commonly used VP-SDE the discrepancy, measured w.r.t. the total variation distance, decays exponentially (Denker et al.,, 2024, Proposition G.2) and we approximate $Q_{T}^{f_{t}}[p_{\mathrm{tilted}}]\approx{\mathbb{P}}_{T}\approx\mathcal{N}(0,
\mathbf{I}_{n})$ for all numerical experiments.

Part (i) of Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) states conditions on $h_{t}$ such that ${\bm{H}}_{0}\sim p_{\mathrm{tilted}}$ in ([5](https://arxiv.org/html/2502.04468v2#S2.E5)) and the (ii), (iii) provide loss functions to learn such a $h_{t}$. For the commonly used VP-SDE (Song et al.,, 2021), the loss ([7](https://arxiv.org/html/2502.04468v2#S2.E7)) in part (ii) reduces to

$$ $\displaystyle\mathcal{L}_{SM}(h_{t})=\mathop{\mathbb{E}}_{\begin{subarray}{c}{ \bm{x}}_{0}\sim p_{\mathrm{tilted}}\\ t\sim\mathrm{U}(0,T),{\bm{z}}\sim\mathcal{N}(0,I)\end{subarray}}\left[\left\| \left({h_{t}(\gamma_{t}{\bm{x}}_{0}+\nu_{t}{\bm{z}})}\!+\!s_{t}(\gamma_{t}{\bm {x}}_{0}+\nu_{t}{\bm{z}})\right)-{\bm{z}}/\nu_{t}\right\|^{2}\right],$ (9) $$

where the parameters $\gamma_{t},\nu_{t}$ are specified by the drift and diffusion coefficients. However, samples from the tilted distribution are often not available, making this approach limited in practice.
Moreover, the loss function in ([8](https://arxiv.org/html/2502.04468v2#S2.E8)) can be reformulated as a stochastic control (SC) loss, i.e.,

$$ $\displaystyle\mathcal{L}_{SC}(h_{t})=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{ P}}_{h}}\left[\frac{1}{2}\int_{0}^{T}\sigma_{t}^{2}\|h_{t}({\bm{x}}_{t})\|^{2} \mathrm{d}t-r({\bm{x}}_{0})\right].$ (10) $$

Differentiating $\mathcal{L}_{SC}$ requires differentiating through the sample generation of the SDE. Even though Denker et al., (2024) reduce the computational cost of this step by using the VarGrad (Richter et al.,, 2020) or trajectory balance loss (Malkin et al.,, 2022), the memory requirements for optimising $\mathcal{L}_{SC}$ still scale linearly with the number of steps in the SDE generation. Similar objectives have been considered from a reinforcement learning perspective, see e.g. Fan et al., (2024) using policy gradient methods (Schulman et al.,, 2017).

###### Remark 3 .

The connection of training diffusion models to stochastic optimal control have also been successfully exploited for sampling from unnormalised probability density function, see (Zhang and Chen,, 2022; Nüsken and Richter,, 2021; Berner et al.,, 2024). Further, the proof for Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (iii) relies on connections to Schrödinger half-bridges (Heng et al.,, 2021; Vargas et al.,, 2021).

###### Theorem 1 (Proposition 2.2 and Theorem 3.1 in Denker et al.,, 2024 ) .

###### Remark 2 .

###### Remark 3 .

## 3 Self-Supervised Importance Fine-tuning

To circumvent the limitations of the supervised loss function considered by Denker et al., (2024), we propose in this section a self-supervised method for fine-tuning diffusion models. To this end, we combine the recently proposed rejection sampling steps from Hertrich and Gruhlke, (2024) with the supervised fine-tuning from Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (ii) and prove that this defines a descent algorithm for the loss function $\mathcal{F}$ from Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (iii).

### 3.1 Importance-based Rejection

A basic tool for reweighting and fine-tuning generative models are importance weights. To this end, let ${\bm{x}}_{0}$ be generated from the backward SDE ([5](https://arxiv.org/html/2502.04468v2#S2.E5)) for some guidance term $h$. We assign to it the weight $\frac{p_{\mathrm{tilted}}({\bm{x}}_{0})}{p_{h}({\bm{x}}_{0})}$, where $p_{h}$ is the density of the generated distribution. Then, samples with a large importance weights are under-represented and samples with small importance weights are over-represented in $p_{h}$ for approximating $p_{\mathrm{tilted}}$.

Unfortunately, we do not have access to the explicit value of $p_{h}$. Therefore, we rewrite the importance weight with $p_{\text{tilted}}({\bm{x}})\propto p_{\text{data}}({\bm{x}})\exp(r({\bm{x}}))$ as

$$ $\displaystyle\frac{p_{\text{tilted}}({\bm{x}})}{p_{h}({\bm{x}})}=\frac{p_{ \text{tilted}}({\bm{x}})}{p_{\text{data}}({\bm{x}})}\frac{p_{\text{data}}({\bm {x}})}{p_{h}({\bm{x}})}\propto\exp(r({\bm{x}}))\frac{p_{\text{data}}({\bm{x}}) }{p_{h}({\bm{x}})}.$ (11) $$

Now we approximate the quotient $\frac{p_{\text{data}}({\bm{x}})}{p^{h}({\bm{x}})}$ by the RND, such that

$$ $\displaystyle\frac{p_{\text{tilted}}({\bm{x}})}{p_{h}({\bm{x}})}\propto\exp(r( {\bm{x}}))\frac{p_{\text{data}}({\bm{x}})}{p_{h}({\bm{x}})}\approx\exp(r({\bm{ x}}_{0}))\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{\mathrm{d}}{\mathbb{P} }_{h}}({\bm{x}}_{[0,T]})$ (12) $$

where ${\mathbb{P}}_{h}$ and ${\mathbb{P}}_{\text{data}}$ are the path measures for the SDEs corresponding to the prior and adjusted diffusion model and ${\bm{x}}_{[0:T]}=({\bm{x}}_{t})_{t\in[0,T]}$ is the whole generated trajectory. The Radon-Nikodym derivative $\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{\mathrm{d}}{\mathbb{P}}_{h}}({
\bm{x}}_{[0,T]})$ can now be computed based on the framework from Vargas et al., (2024). We summarise the result in the following lemma. The derivations are included in Appendix [A](https://arxiv.org/html/2502.04468v2#A1), both in continuous and in discrete time.

###### Lemma 4 .

The right side of ([12](https://arxiv.org/html/2502.04468v2#S3.E12)) can be rewritten as

$$ $\displaystyle\exp\left(r({\bm{x}}_{0})-\frac{1}{2}\int_{0}^{T}\sigma_{t}^{2}\| h_{t}({\bm{x}}_{t})\|_{2}^{2}dt+\int_{0}^{T}\sigma_{t}h_{t}({\bm{x}}_{t})^{ \top}dW_{t}\right),$ (13) $$

for $x_{[0:T]}\sim\mathbb{P}_{h}$.

These path-wise importance weights can be computed concurrently with sampling without any computational overhead. For numerical stability, we perform these computations in the log-space.

In practice, importance weights have the disadvantage that they often suffer from highly imbalanced weights.
As a remedy, Hertrich and Gruhlke, (2024) proposed a rejection sampling algorithm based on relaxed importance weights. Plugging in our approximation ([12](https://arxiv.org/html/2502.04468v2#S3.E12)), this rejection sampling step amounts to computing, for some generated trajectory ${\bm{x}}_{[0,T]}$, the acceptance probability

$$ $\alpha({\bm{x}}_{[0:T]})=\min\left(1,\frac{{\mathrm{d}}{\mathbb{P}}_{\text{ data}}}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{ 0}))}{c}\right),$ (14) $$

where $c$ is a hyper-parameter. In practice, we use an adaptive choice of $c$ such that a certain rate of samples is accepted, see (Hertrich and Gruhlke,, 2024, Rem 10). We will show in the next subsection that the distribution of accepted samples is closer to the tilted distribution than the distribution of the initially generated samples.

###### Lemma 4 .

### 3.2 Self-supervised Importance Fine-Tuning

Figure: Algorithm 1 Self-supervised estimation of the $h$-transform for DDPM

Next, we will combine the importance-based rejection steps with the supervised loss function from Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (ii) to obtain a tractable fine-tuning algorithm.
More precisely, starting with an initial guidance term $h^{0}$ we construct a sequence of guidance terms $h^{k}$ for $k=1,2,...$ by the following steps.

First, we sample a batch $\{{\bm{x}}_{0}^{(i)}\}_{i=1,\dots,N}$ from the reverse SDE ([5](https://arxiv.org/html/2502.04468v2#S2.E5)) with $h=h^{k}$ and compute the corresponding acceptance probabilities $\alpha_{i}=\alpha({\bm{x}}_{[0,T]}^{(i)})$ from ([14](https://arxiv.org/html/2502.04468v2#S3.E14)).
Second, we keep any sample from the batch with probability $\alpha_{i}$ and discard the rest. We denote the distribution of remaining samples by $\tilde{{\mathbb{P}}}_{h^{k}}^{0}$.
Finally, we update the guidance term $h^{k}$ by using the supervised loss function for the $h$-transform from Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) (ii). That is, we define $h^{k+1}\in\operatorname*{arg\,min}_{g}\mathcal{L}_{FT}(g)$, where

$$ $\displaystyle\mathcal{L}_{FT}(g)=\mathop{\mathbb{E}}_{\begin{subarray}{c}{\bm{ x}}_{0}\sim\tilde{{\mathbb{P}}}_{h^{k}}^{0}\\ t\sim\mathrm{U}(0,T),{\bm{x}}_{t}\sim\leavevmode\hbox to3.62pt{\vbox to6.22pt{ \pgfpicture\makeatletter\hbox{\hskip 1.76094pt\lower-2.18748pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-1.76094pt}{-0.8264pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-1.41093pt}{3.93748pt}\pgfsys@lineto{1.76094 pt}{3.93748pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{1.76094pt}{3.93748pt}\pgfsys@lineto{0.71094 pt}{3.23749pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}(\cdot|{\bm{x}}_{ 0})\end{subarray}}\left[\left\|\left({g_{t}({\bm{x}}_{t})}\!+\!s_{t}({\bm{x}}_ {t})\right)\!-\!\nabla_{{\bm{x}}_{t}}\ln\leavevmode\hbox to5.13pt{\vbox to8.85 pt{\pgfpicture\makeatletter\hbox{\hskip 2.51563pt\lower-3.12498pt\hbox to0.0pt {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.01563pt}{5.62498pt}\pgfsys@lineto{2.51563 pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{2.51563pt}{5.62498pt}\pgfsys@lineto{1.01563 pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}({\bm{x}}_{t}|{ \bm{x}}_{0})\right\|^{2}\right].$ (15) $$

We summarise our self-supervised importance fine-tuning in Algorithm [1](https://arxiv.org/html/2502.04468v2#alg1) for the discrete DDPM framework (Ho et al.,, 2020).

Recall that the generalised $h$-transform from Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1) minimises the function $\mathcal{F}({\mathbb{P}}_{h})=\mathrm{KL}({\mathbb{P}}_{h},{\mathbb{P}}_{
\mathrm{data}})-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}[r({\bm{x}}_{
0})]$. The following theorem proves that our self-supervised importance fine-tuning is a descent algorithm for $\mathcal{F}$. More precisely, it holds that $\mathcal{F}({\mathbb{P}}_{h^{k+1}})\leq\mathcal{F}({\mathbb{P}}_{h^{k}})$ for all $k=1,2,...$.
We include the proof in Appendix [B.1](https://arxiv.org/html/2502.04468v2#A2.SS1).

###### Theorem 5 .

Let $\alpha({\bm{x}}_{[0:T]})=\min\left(1,\frac{{\mathrm{d}}{\mathbb{P}}_{\text{
data}}}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{
0}))}{c}\right)$ be the acceptance probability of a sample ${\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}$ and denote by $\tilde{{\mathbb{P}}}_{h}$ the distribution of the accepted paths.
Then, the following holds true:

- (i)
$\frac{{\mathrm{d}}\tilde{{\mathbb{P}}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm
{x}}_{[0,T]})=\frac{\alpha({\bm{x}}_{[0,T]})}{\mathbb{E}_{{\bm{x}}_{[0,T]}\sim
{\mathbb{P}}_{h}}[\alpha({\bm{x}}_{[0,T]})]}$
- (ii)
$\mathcal{F}(\tilde{{\mathbb{P}}}_{h})\leq\mathcal{F}({\mathbb{P}}_{h})$, where $\mathcal{F}({\mathbb{P}})=\mathrm{KL}({\mathbb{P}},{\mathbb{P}}_{\mathrm{data}
})-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}}[r({\bm{x}}_{0})]\propto
\mathrm{KL}({\mathbb{P}},{\mathbb{P}}_{\mathrm{tilde}})$
- (iii)
Let $h_{t}^{*}\in\operatorname*{arg\,min}_{g_{t}}\mathcal{L}_{FT}(g_{t})$ with $\mathcal{L}_{FT}$ from ([15](https://arxiv.org/html/2502.04468v2#S3.E15))
and let ${\mathbb{P}}_{h^{*}}$ be the path measure of the corresponding SDE.
Then it holds
$\mathcal{F}({\mathbb{P}}_{h^{*}})\leq\mathcal{F}(\tilde{{\mathbb{P}}}_{h})\leq
\mathcal{F}({\mathbb{P}}_{h}).$

###### Theorem 5 .

### 3.3 Training and Network Parametrisation

#### Replay buffer

Sampling from the current model is the most computationally expensive part of the algorithm. Motivated by Midgley et al., (2023); Sendera et al., (2024), we make use of an un-prioritised replay buffer with a fixed length. We save the accepted samples from the current model and append them the buffer. Once the fixed length is reached, we discard the oldest samples. During training, we randomly sample batches from the buffer.

#### Network parametrisation

The parametrisation of the $h$-transform has a crucial effect on performance and convergence speed (He et al.,, 2025). Motivated by the network parametrisation in sampling applications with diffusion (Vargas et al., 2023a,; Zhang and Chen,, 2022) and fine-tuning approaches (Denker et al.,, 2024; Venkatraman et al.,, 2024), we make use of a reward-informed inductive bias given as

$$ $\displaystyle h^{\theta}_{t}({\bm{x}}_{t})=\text{NN}_{1}({\bm{x}}_{t},t)+\text {NN}_{2}(t)\nabla_{\hat{\bm{x}}_{0}}r(\hat{\bm{x}}_{0}),$ (16) $$

where $\hat{\bm{x}}_{0}$ is the Tweedie estimate given the pre-trained unconditional diffusion model, $\text{NN}_{1}$ is a vector-valued and $\text{NN}_{2}$ a scalar-valued neural network. We initialise the last layer of $\text{NN}_{1}$ to be zero and $\text{NN}_{2}$ to be constant. However, for text-to-image diffusion models we instead make use of prior work, see e.g. (Venkatraman et al.,, 2024; Fan et al.,, 2024), and use the parameter efficient LoRA method (Hu et al.,, 2022).

#### KL-Regularisation

Similar to Fan et al., (2024), we found that using a KL regulariser was useful to improve diversity in the supervised setting. The KL divergence between $p_{h}$ and $p_{\text{data}}$ can be bounded as

$$ $\displaystyle D_{\text{KL}}(p_{h},p_{\text{data}})\leq D_{\text{KL}}({\mathbb{ P}}_{h},{\mathbb{P}}_{\text{data}})=\mathbb{E}_{{\bm{H}}_{t}\sim{\mathbb{P}}_{ h}}\left[\int_{0}^{T}\sigma_{t}^{2}\|h_{t}({\bm{H}}_{t})\|_{2}^{2}dt\right],$ (17) $$

i.e., the norm of the $h$-transform over trajectories, see Appendix [A](https://arxiv.org/html/2502.04468v2#A1) for a derivation. The naive addition of this term to the supervised training loss is expensive, as one has to backpropagate through the full trajectory. Instead, we estimate the integral using a single random time step.

## 4 Experiments

In this section, we present a toy example on a 2D dataset, class conditional sampling for MNIST, super-resolution, and finally results for reward fine-tuning of text-to-image diffusion models. The code is available(^1^11[https://anonymous.4open.science/r/IterativeImportanceFinetuning-4606](https://anonymous.4open.science/r/IterativeImportanceFinetuning-4606))

Figure: Figure 1: 2D Toy example for fine-tuning diffusion models. We show the prior the log-probability of the tilted distribution and samples from the conditional diffusion model using classifier guidance (Dhariwal and Nichol,, 2021), online fine-tuning (Fan et al.,, 2024), our importance fine-tuning and a variation with only reward-based importance weights.
Refer to caption: x1.png

### 4.1 2D Toy example

As an initial example we make use of a diffusion trained on samples of a Gaussian mixture model (GMM) with $25$ modes, arranged on a grid. The goal is to sample from the tilted distribution $p_{\text{data}}({\bm{x}})\exp(r({\bm{x}}))$, where the reward $r$ is defined as the log-likelihood of a GMM with a reweighted subset of the modes, see Appendix [C.1](https://arxiv.org/html/2502.04468v2#A3.SS1) for the detailed setup. Figure [1](https://arxiv.org/html/2502.04468v2#S4.F1) illustrates both the prior and the density of the tilted distribution. We compare against Classifier Guidance (Dhariwal and Nichol,, 2021), an inference time method, where the $h$-transform is approximated by the gradient of the reward $h_{t}({\bm{x}}_{t})=\gamma\nabla_{{\bm{x}}_{t}}r({\bm{x}}_{t})$. Further, we compare against online fine-tuning, where we directly optimise Equation ([2](https://arxiv.org/html/2502.04468v2#S1.E2)). Instead of employing the policy gradient of DPOK (Fan et al.,, 2024), we backpropagate gradients through the entire trajectory for this low-dimensional toy example. Further, we evaluate our importance fine-tuning and a variation where importance weights are calculated solely based on the reward, referred to as reward-only fine-tuning.

In Figure [1](https://arxiv.org/html/2502.04468v2#S4.F1), we observe that classifier guidance is able to find all the modes of the posterior, fails to capture the correct weighting. In contrast, both online fine-tuning and importance fine-tuning yield samples that more accurately represent the target distribution. Lastly, the reward-only fine-tuning method collapses to the highest mode.

### 4.2 Posterior Sampling for Inverse Problems

We consider the task of posterior sampling in inverse problems. Here, the reward function $r$ is given by the likelihood $p^{\text{lkhd}}({\bm{y}}|{\bm{x}})$, i.e., the conditional distributions of measurements ${\bm{y}}$ given images ${\bm{x}}$. We focus on the linear inverse problem of super-resolution, where the measurements are given as ${\bm{y}}^{\delta}={\bm{A}}{\bm{x}}+\mathbf{\eta}$. Here, ${\bm{A}}$ is the downsampling operator, $\eta\sim\mathcal{N}(0,\sigma_{y}^{2}\mathbf{I})$ represents additive Gaussian noise, and the likelihood is given as $r({\bm{x}};{\bm{y}}^{\delta})=\ln p^{\text{lkhd}}({\bm{y}}^{\delta}|{\bm{x}})=
-\frac{1}{2\sigma_{y}^{2}}\|{\bm{A}}{\bm{x}}-{\bm{y}}^{\delta}\|_{2}^{2}$.

We train a diffusion model on the Flowers dataset (Nilsback and Zisserman,, 2008) and consider the $2\times$ super-resolution task with $5\%$ relative Gaussian noise. For the $h$-transform we make use of the reward-informed architecture ([16](https://arxiv.org/html/2502.04468v2#S3.E16)), see Appendix [C.2](https://arxiv.org/html/2502.04468v2#A3.SS2) for the detailed setup. Theorem [5](https://arxiv.org/html/2502.04468v2#Thmtheorem5) demonstrates that iterative importance fine-tuning functions as a descent algorithm for the SC loss $\mathcal{L}_{\text{SC}}$. Figure [2](https://arxiv.org/html/2502.04468v2#S4.F2) presents both the SC loss across iterations and the final model’s sample outputs, confirming that iterative fine-tuning operates as a descent algorithm for the SC loss in this scenario.

Figure: Figure 2: Posterior sampling for Flowers. Left: We show decay of the SC loss function $\mathcal{L}_{\text{SC}}(h_{k})$ over fine-tuning iterations $k$. We show the mean loss over $10$ independent training runs with $\pm 2$ std intervals. Right: The upper left image is the ground truth image and the remaining $8$ images are samples from the model.
Refer to caption: x2.png

### 4.3 Class-conditional sampling

As another application, we consider class conditional sampling. Let $c$ be the class and $p(c|{\bm{x}})$ the log class probabilities of a pre-trained classifier. The goal is to sample from the tilted distribution with the reward as $r({\bm{x}})=\ln p(c|{\bm{x}})$. We consider both the MNIST dataset and pre-train unconditional diffusion models using the U-Net architecture from Ho et al., (2020). We use a convolutional classifier with a $98.6\%$ accuracy. We use the reward-informed architecture as in Eqn. ([16](https://arxiv.org/html/2502.04468v2#S3.E16)).

Figure: Prior samples
Refer to caption: extracted/6605494/imgs/prior_samples_mnist.png

**Table 1: Expected reward and FID scores for class conditional sampling on MNIST.**
| Method | Single class | Even / Odd |  |  |
| --- | --- | --- | --- | --- |
|  | $\mathbb{E}[r({\bm{x}})]$ $(\uparrow)$ | FID $(\downarrow)$ | $\mathbb{E}[r({\bm{x}})]$ $(\uparrow)$ | FID $(\downarrow)$ |
| Classifier Guid. | $-1.495$ | $117.241$ | $-2.572$ | $193.984$ |
| Online FT | $-0.110$ | $31.994$ | $-0.065$ | $198.871$ |
| Importance FT | $-0.152$ | $30.814$ | $-0.878$ | $110.403$ |

We use importance fine-tuning to learn the posterior for different MNIST classes and for even/odd numbers. We show posterior prior samples in Figure [3](https://arxiv.org/html/2502.04468v2#S4.F3), see Figure [5](https://arxiv.org/html/2502.04468v2#A3.F5) for all classes. We compare both against classifier guidance and online fine-tuning. For online fine-tuning we optimise Equation ([10](https://arxiv.org/html/2502.04468v2#S2.E10)) using VarGrad (Richter et al.,, 2020) to reduce the memory cost. In Table [1](https://arxiv.org/html/2502.04468v2#S4.T1) we present the expectation of the reward and the FID (Heusel et al.,, 2017) based on the features of the penultimate layer of the pre-trained classifier. For single-class posteriors, online fine-tuning achieves a higher expected reward. A similar trend is observed for even/odd tasks. This indicates that online fine-tuning is more effective at maximising reward for this setting, although at the expense of reduced sample diversity, as reflected by the FID score.

### 4.4 Text-to-Image Reward Fine-tuning

Despite the progress in training text-to-image diffusion models, samples not always align with human preferences. In particular, the model can struggle with unnatural prompts such as “A green rabbit” (Venkatraman et al.,, 2024; Fan et al.,, 2024). To alleviate this problem, the diffusion model is fine-tuned to maximise some reward model, trained to imitate human preferences. We use the latent diffusion model Stable Diffusion-v1.5 (Rombach et al.,, 2022) and align it using the ImageReward-v1.0 (Xu et al.,, 2024) reward model. We refer to Appendix [C.4](https://arxiv.org/html/2502.04468v2#A3.SS4) for experimental details. We compare against Top-$K$ sampling ($K=6$) as a training-free baseline, where $K$ samples are generated, and the one with the highest reward is selected, increasing sampling time by a factor of $K$. Additionally, we evaluate against two state-of-the-art methods, i.e., DPOK (Fan et al.,, 2024) and adjoint matching (Domingo-Enrich et al.,, 2025). Both DPOK and our method use LoRA parametrization, whereas adjoint matching fine-tunes the full model. Consequently, DPOK runs on an RTX 4090 with $24$GB memory (using a tiny batch size and checkpointing), while adjoint matching demands a GPU with over $30$GB memory. Results are presented in Table [2](https://arxiv.org/html/2502.04468v2#S4.T2) and images are shown in Figure [4](https://arxiv.org/html/2502.04468v2#S4.F4), see Appendix [C.4](https://arxiv.org/html/2502.04468v2#A3.SS4) for additional results. We present both the mean reward and the diversity, measured as one minus the mean cosine similarity of CLIP embeddings. While our method exhibits slightly lower performance in reward scores compared to adjoint matching, it enables fine-tuning using only a single smaller GPU. This makes our approach more accessible for researchers with limited computational resources, and it could be scaled further with hardware upgrades.

Figure: Base Model
Refer to caption: extracted/6605494/imgs/A_green_colored_rabbit/base_model/img_0.png

**Table 2: Results for text-to-image. We compute the mean reward and diversity over $100$ samples.**
|  | “A green colored rabbit.“ | “Two roses in a vase.“ | “Two dogs in the park.“ | GPU hrs (train) | Peak GPU memory |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Reward $(\uparrow)$ | Diversity $(\uparrow)$ | Reward $(\uparrow)$ | Diversity $(\uparrow)$ | Reward $(\uparrow)$ | Diversity $(\uparrow)$ |  |  |
| Base Model | $-0.207$ | $0.154$ | $1.130$ | $0.108$ | $0.518$ | $0.193$ | N/A | 9GB |
| Top-K Sampling | $1.334$ | $0.149$ | $1.619$ | $0.121$ | $0.970$ | $0.161$ | N/A | 9GB |
| DPOK | $1.621$ | $0.065$ | $1.592$ | $0.106$ | $1.013$ | $0.144$ | 28 (A100) | 34 GB |
| Adjoint Matching | $1.706$ | $0.050$ | $1.504$ | $0.088$ | $1.325$ | $0.132$ | 4 (A100) | 36 GB |
| Importance FT (ours) | $1.462$ | $0.051$ | $1.534$ | $0.077$ | $1.010$ | $0.120$ | 7 (RTX 4090) | 16 GB |

## 5 Conclusion

We propose an iterative approach for fine-tuning diffusion models for conditional sampling tasks. As the naive iterative retraining of generative models often lead to performance degradation (Shumailov et al.,, 2023), we introduce an additional resampling step based on path-based importance weights. We show initial evaluations for class conditional sampling and reward fine-tuning of text-to-image diffusion models. Online fine-tuning methods often require additional tricks to increase training stability, e.g., Venkatraman et al., (2024) use loss clipping and disregard low reward trajectories or Fan et al., (2024) employ variance reduction techniques by additionally learning the value function. In contrast our importance fine-tuning method is trained using a score matching loss, see Theorem [1](https://arxiv.org/html/2502.04468v2#Thmtheorem1), leading to a more stable training.

#### Limitations

In our iterative refinement method, the model is fine-tuned using ”good“, i.e., having a high importance weight, samples from the pre-trained model. However, these samples necessarily lie in the support of the pre-trained model and we rely on the fact that such ”good“ samples exist. Thus, our refinement approach might struggle on domains where the distribution of the pre-trained model is sparse and high reward samples are are. One possibility to alleviate this problem is to make use of ”off-policy“ samples (Venkatraman et al.,, 2024), which are obtained by some other method. For off-policy samples, we loose the compact formulation of the RND from Lemma [4](https://arxiv.org/html/2502.04468v2#Thmtheorem4). However, we can still compute importance weights using the RND in Proposition [6](https://arxiv.org/html/2502.04468v2#Thmtheorem6), see Appendix [A](https://arxiv.org/html/2502.04468v2#A1).

#### Potential Societal Impacts

Diffusion models, as a class of generative models, have potential negative societal impacts. Fine-tuning diffusion models according to an external reward function, can for example potentially be used to generate deepfakes and spread disinformation.

## References

- Akhound-Sadegh et al., (2024)
Akhound-Sadegh, T., Rector-Brooks, J., Bose, J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., Malkin, N., and Tong, A. (2024).
Iterated denoising energy matching for sampling from boltzmann densities.
In Forty-first International Conference on Machine Learning.
- Albergo et al., (2023)
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. (2023).
Stochastic interpolants: A unifying framework for flows and diffusions.
arXiv preprint arXiv:2303.08797.
- Albergo and Vanden-Eijnden, (2024)
Albergo, M. S. and Vanden-Eijnden, E. (2024).
Nets: A non-equilibrium transport sampler.
arXiv preprint arXiv:2410.02711.
- Alemohammad et al., (2024)
Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. (2024).
Self-consuming generative models go MAD.
In The Twelfth International Conference on Learning Representations.
- Anderson, (1982)
Anderson, B. D. (1982).
Reverse-time diffusion equation models.
Stochastic Processes and their Applications, 12(3):313–326.
- Berner et al., (2024)
Berner, J., Richter, L., and Ullrich, K. (2024).
An optimal control perspective on diffusion-based generative modeling.
Transactions on Machine Learning Research.
- Bertrand et al., (2024)
Bertrand, Q., Bose, J., Duplessis, A., Jiralerspong, M., and Gidel, G. (2024).
On the stability of iterative retraining of generative models on their own data.
In The Twelfth International Conference on Learning Representations.
- Black et al., (2024)
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. (2024).
Training diffusion models with reinforcement learning.
In The Twelfth International Conference on Learning Representations.
- Blessing et al., (2025)
Blessing, D., Berner, J., Richter, L., and Neumann, G. (2025).
Underdamped diffusion bridges with applications to sampling.
In The Thirteenth International Conference on Learning Representations.
- Chung et al., (2023)
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. (2023).
Diffusion posterior sampling for general noisy inverse problems.
In The Eleventh International Conference on Learning Representations.
- Clark et al., (2024)
Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. (2024).
Directly fine-tuning diffusion models on differentiable rewards.
In The Twelfth International Conference on Learning Representations.
- Denker et al., (2024)
Denker, A., Vargas, F., Padhy, S., Didi, K., Mathis, S. V., Barbano, R., Dutordoir, V., Mathieu, E., Komorowska, U. J., and Lio, P. (2024).
DEFT: Efficient fine-tuning of diffusion models by learning the generalised $h$-transform.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Dhariwal and Nichol, (2021)
Dhariwal, P. and Nichol, A. (2021).
Diffusion models beat gans on image synthesis.
Advances in neural information processing systems, 34:8780–8794.
- Domingo-Enrich et al., (2025)
Domingo-Enrich, C., Drozdzal, M., Karrer, B., and Chen, R. T. Q. (2025).
Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.
In The Thirteenth International Conference on Learning Representations.
- Dong et al., (2023)
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. (2023).
RAFT: Reward ranked finetuning for generative foundation model alignment.
Transactions on Machine Learning Research.
- Fan et al., (2024)
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. (2024).
Reinforcement learning for fine-tuning text-to-image diffusion models.
Advances in Neural Information Processing Systems, 36.
- Ferbach et al., (2024)
Ferbach, D., Bertrand, Q., Bose, J., and Gidel, G. (2024).
Self-consuming generative models with curated data provably optimize human preferences.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- He et al., (2025)
He, J., Du, Y., Vargas, F., Zhang, D., Padhy, S., OuYang, R., Gomes, C., and Hernández-Lobato, J. M. (2025).
No trick, no treat: Pursuits and challenges towards simulation-free training of neural samplers.
arXiv preprint arXiv:2502.06685.
- Heng et al., (2021)
Heng, J., De Bortoli, V., Doucet, A., and Thornton, J. (2021).
Simulating diffusion bridges with score matching.
arXiv preprint arXiv:2111.07243.
- Hertrich and Gruhlke, (2024)
Hertrich, J. and Gruhlke, R. (2024).
Importance corrected neural JKO sampling.
arXiv preprint arXiv:2407.20444.
- Heusel et al., (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017).
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
Advances in neural information processing systems, 30.
- Ho et al., (2020)
Ho, J., Jain, A., and Abbeel, P. (2020).
Denoising diffusion probabilistic models.
Advances in neural information processing systems, 33:6840–6851.
- Ho and Salimans, (2022)
Ho, J. and Salimans, T. (2022).
Classifier-free diffusion guidance.
arXiv preprint arXiv:2207.12598.
- Hu et al., (2022)
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022).
LoRA: Low-rank adaptation of large language models.
In International Conference on Learning Representations.
- Jalal et al., (2021)
Jalal, A., Arvinte, M., Daras, G., Price, E., Dimakis, A. G., and Tamir, J. (2021).
Robust compressed sensing mri with deep generative priors.
Advances in Neural Information Processing Systems, 34:14938–14954.
- Léonard, (2014)
Léonard, C. (2014).
Some properties of path measures.
Séminaire de Probabilités XLVI, pages 207–230.
- Loshchilov and Hutter, (2019)
Loshchilov, I. and Hutter, F. (2019).
Decoupled weight decay regularization.
In International Conference on Learning Representations.
- Malkin et al., (2022)
Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y. (2022).
Trajectory balance: Improved credit assignment in gflownets.
Advances in Neural Information Processing Systems, 35:5955–5967.
- Midgley et al., (2023)
Midgley, L. I., Stimper, V., Simm, G. N. C., Schölkopf, B., and Hernández-Lobato, J. M. (2023).
Flow annealed importance sampling bootstrap.
In The Eleventh International Conference on Learning Representations.
- Nilsback and Zisserman, (2008)
Nilsback, M.-E. and Zisserman, A. (2008).
Automated flower classification over a large number of classes.
In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE.
- Nüsken and Richter, (2021)
Nüsken, N. and Richter, L. (2021).
Solving high-dimensional hamilton–jacobi–bellman pdes using neural networks: perspectives from the theory of controlled diffusions and measures on path space.
Partial differential equations and applications, 2(4):48.
- Pidstrigach et al., (2025)
Pidstrigach, J., Baker, E., Domingo-Enrich, C., Deligiannidis, G., and Nüsken, N. (2025).
Conditioning diffusions using malliavin calculus.
arXiv preprint arXiv:2504.03461.
- Richter et al., (2020)
Richter, L., Boustati, A., Nüsken, N., Ruiz, F., and Akyildiz, O. D. (2020).
Vargrad: a low-variance gradient estimator for variational inference.
Advances in Neural Information Processing Systems, 33:13481–13492.
- Rombach et al., (2022)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
- Rout et al., (2024)
Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A., and Shakkottai, S. (2024).
Solving linear inverse problems provably via posterior sampling with latent diffusion models.
Advances in Neural Information Processing Systems, 36.
- Ruiz et al., (2023)
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023).
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–22510.
- Schulman et al., (2017)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347.
- Sendera et al., (2024)
Sendera, M., Kim, M., Mittal, S., Lemos, P., Scimeca, L., Rector-Brooks, J., Adam, A., Bengio, Y., and Malkin, N. (2024).
Improved off-policy training of diffusion samplers.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Shumailov et al., (2023)
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. (2023).
The curse of recursion: Training on generated data makes models forget.
arXiv preprint arXiv:2305.17493.
- Song et al., (2024)
Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, L. (2024).
Solving inverse problems with latent diffusion models via hard data consistency.
In The Twelfth International Conference on Learning Representations.
- Song et al., (2021)
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021).
Score-based generative modeling through stochastic differential equations.
In ICLR.
- Uehara et al., (2025)
Uehara, M., Zhao, Y., Wang, C., Li, X., Regev, A., Levine, S., and Biancalani, T. (2025).
Reward-guided controlled generation for inference-time alignment in diffusion models: Tutorial and review.
arXiv preprint arXiv:2501.09685.
- (43)
Vargas, F., Grathwohl, W. S., and Doucet, A. (2023a).
Denoising diffusion samplers.
In The Eleventh International Conference on Learning Representations.
- (44)
Vargas, F., Ovsianas, A., Fernandes, D., Girolami, M., Lawrence, N. D., and Nüsken, N. (2023b).
Bayesian learning via neural schrödinger–föllmer flows.
Statistics and Computing, 33(1):3.
- Vargas et al., (2024)
Vargas, F., Padhy, S., Blessing, D., and Nüsken, N. (2024).
Transport meets variational inference: Controlled monte carlo diffusions.
In The Twelfth International Conference on Learning Representations.
- Vargas et al., (2021)
Vargas, F., Thodoroff, P., Lamacraft, A., and Lawrence, N. (2021).
Solving schrödinger bridges via maximum likelihood.
Entropy, 23(9):1134.
- Venkatraman et al., (2024)
Venkatraman, S., Jain, M., Scimeca, L., Kim, M., Sendera, M., Hasan, M., Rowe, L., Mittal, S., Lemos, P., Bengio, E., Adam, A., Rector-Brooks, J., Bengio, Y., Berseth, G., and Malkin, N. (2024).
Amortizing intractable inference in diffusion models for vision, language, and control.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Watson et al., (2023)
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. (2023).
De novo design of protein structure and function with rfdiffusion.
Nature, 620(7976):1089–1100.
- Xu et al., (2024)
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. (2024).
Imagereward: Learning and evaluating human preferences for text-to-image generation.
Advances in Neural Information Processing Systems, 36.
- Zhang et al., (2023)
Zhang, L., Rao, A., and Agrawala, M. (2023).
Adding conditional control to text-to-image diffusion models.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847.
- Zhang and Chen, (2022)
Zhang, Q. and Chen, Y. (2022).
Path integral sampler: A stochastic control approach for sampling.
In International Conference on Learning Representations.
- Zhao et al., (2025)
Zhao, Y., Uehara, M., Scalia, G., Kung, S., Biancalani, T., Levine, S., and Hajiramezanali, E. (2025).
Adding conditional control to diffusion models with reinforcement learning.
In The Thirteenth International Conference on Learning Representations.

## Appendix A Radon-Nikodym derivative betweens SDEs.

To calculate the approximated importance weights, we make use of the RND between SDEs. In particular, we use the result from (Nüsken and Richter,, 2021; Vargas et al.,, 2024).

###### Proposition 6 .

(RND between SDEs (Nüsken and Richter,, 2021; Vargas et al.,, 2024))
Given the following SDEs

$$ $\displaystyle{\mathrm{d}}{\bm{Y}}_{t}$ $\displaystyle=a_{t}({\bm{Y}}_{t})\,{\mathrm{d}}t+\sigma_{t}({\bm{Y}}_{t})\; \leavevmode\hbox to15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 7 .91669pt\lower-3.47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.41669pt}{5.97221pt}\pgfsys@lineto{7.91669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{7.91669pt}{5.97221pt}\pgfsys@lineto{6.41669 pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{Y}}_{0} \ \sim\mu,$ (18) $\displaystyle{\mathrm{d}}{\bm{X}}_{t}$ $\displaystyle=b_{t}({\bm{X}}_{t})\,{\mathrm{d}}t+\sigma_{t}({\bm{X}}_{t})\; \leavevmode\hbox to15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 7 .91669pt\lower-3.47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.41669pt}{5.97221pt}\pgfsys@lineto{7.91669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{7.91669pt}{5.97221pt}\pgfsys@lineto{6.41669 pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{X}}_{0} \ \sim\nu,$ (19) $$

with path probabilities $\leavevmode\hbox to6.91pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 3
.40279pt\lower-3.41666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279
pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279
pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{a}$ and $\leavevmode\hbox to6.91pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 3
.40279pt\lower-3.41666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{
{}{}}
{{}{{}}}{{}{}}{}{{}{}}
{
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1
.0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{
{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279
pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{}
{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt}
\pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279
pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }
\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{b}$, we get

$$ $\displaystyle\ln\left(\frac{d\leavevmode\hbox to6.91pt{\vbox to9.43pt{ \pgfpicture\makeatletter\hbox{\hskip 3.40279pt\lower-3.41666pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{a}}{d\leavevmode\hbox to 6.91pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 3.40279pt\lower-3.4 1666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{b}}\right)({\bm{Z}})= \ln\left(\frac{d\mu}{d\nu}\right)({\bm{Z}}_{0})+\int_{0}^{T}\sigma_{t}^{-2}(a_ {t}-b_{t})({\bm{Z}}_{t})d\leavevmode\hbox to7.64pt{\vbox to9.43pt{\pgfpicture \makeatletter\hbox{\hskip 3.77083pt\lower-3.41666pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-3.77083pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{Z}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-3.27083pt}{5.91666pt}\pgfsys@lineto{3.77083 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{3.77083pt}{5.91666pt}\pgfsys@lineto{2.27083 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}+\frac{1}{2}\int_{0 }^{T}\sigma_{t}^{-2}(b_{t}^{2}-a_{t}^{2})({\bm{Z}}_{t})dt,$ (20) $$

and in particular, when evaluated on ${\bm{Y}}$:

$$ $\displaystyle\ln\left(\frac{d\leavevmode\hbox to6.91pt{\vbox to9.43pt{ \pgfpicture\makeatletter\hbox{\hskip 3.40279pt\lower-3.41666pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{a}}{d\leavevmode\hbox to 6.91pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 3.40279pt\lower-3.4 1666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-3.40279pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathbb{P}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.90279pt}{5.91666pt}\pgfsys@lineto{3.40279 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{3.40279pt}{5.91666pt}\pgfsys@lineto{1.90279 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{b}}\right)({\bm{Y}})= \ln\left(\frac{d\mu}{d\nu}\right)({\bm{Y}}_{0})+\int_{0}^{T}\sigma_{t}^{-1}(a_ {t}-b_{t})({\bm{Y}}_{t})d\leavevmode\hbox to10.93pt{\vbox to9.43pt{\pgfpicture \makeatletter\hbox{\hskip 5.41667pt\lower-3.41666pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-5.41667pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{W}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-4.91667pt}{5.91666pt}\pgfsys@lineto{5.41667 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{5.41667pt}{5.91666pt}\pgfsys@lineto{3.91667 pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}+\frac{1}{2}\int_{0 }^{T}\sigma_{t}^{-2}\|b_{t}-a_{t}\|^{2}({\bm{Y}}_{t})dt.$ (21) $$

Let ${\mathbb{P}}_{\text{data}}$ be the path probability of

$$ $\displaystyle{\mathrm{d}}{\bm{X}}_{t}$ $\displaystyle=\left(f_{t}({\bm{X}}_{t})-\sigma_{t}^{2}\nabla_{{\bm{X}}_{t}}\ln p _{t}({\bm{X}}_{t})\right)\,{\mathrm{d}}t+\sigma_{t}\leavevmode\hbox to15.93pt{ \vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 8.01668pt\lower-3.47221pt \hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{X}}_{T} \sim\mathcal{P}_{T},$ (22) $$

and ${\mathbb{P}}_{h}$ the path probability of

$$ $\displaystyle{\mathrm{d}}{\bm{H}}_{t}$ $\displaystyle=\left(f_{t}({\bm{H}}_{t})-\sigma_{t}^{2}(\nabla_{{\bm{H}}_{t}} \ln p_{t}({\bm{H}}_{t})+h_{t}({\bm{H}}_{t}))\right)\,{\mathrm{d}}t+\sigma_{t} \leavevmode\hbox to15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 8 .01668pt\lower-3.47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},\quad{\bm{H}}_{T} \sim\mathcal{P}_{T}.$ (23) $$

For the approximated importance weights we require the RND of ${\mathbb{P}}_{h}$ with respect to ${\mathbb{P}}_{\text{data}}$ evaluated at trajectories from ${\mathbb{P}}_{h}$. Let the drift of ${\mathbb{P}}_{\text{data}}$ be given as $a_{t}({\bm{x}})=f_{t}({\bm{x}})-\sigma_{t}^{2}\nabla_{{\bm{x}}}\ln p_{t}({\bm{
x}})$ and the drift of ${\mathbb{P}}_{h}$ as $b_{t}({\bm{x}})=f_{t}({\bm{x}})-\sigma_{t}^{2}\nabla_{{\bm{x}}}\ln p_{t}({\bm{
x}})-\sigma_{t}^{2}h_{t}({\bm{x}})$. In particular, we have $a_{t}-b_{t}=\sigma_{t}^{2}h_{t}$. For readability, we do not omit the dependence on ${\bm{x}}$ for the drift in the following. Using Proposition [6](https://arxiv.org/html/2502.04468v2#Thmtheorem6) we get,

$$ $\displaystyle\ln\left(\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{\mathrm{d }}{\mathbb{P}}_{h}}\right)({\bm{H}})$ $\displaystyle=\int_{0}^{T}\sigma_{t}^{-2}(a_{t}-b_{t}){\mathrm{d}}\leavevmode \hbox to9.22pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 4.66248pt \lower-3.41666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-4.56248pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{H}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-4.56248pt}{5.91666pt}\pgfsys@lineto{4.06248 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-4.56248pt}{5.91666pt}\pgfsys@lineto{-3.0624 8pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}+\frac{1}{2}\int_{0 }^{T}\sigma_{t}^{-2}(b_{t}^{2}-a_{t}^{2}){\mathrm{d}}t$ (24) $\displaystyle=\int_{0}^{T}\sigma_{t}^{-2}(a_{t}-b_{t})b_{t}{\mathrm{d}}t+\int_ {0}^{T}\sigma_{t}^{-2}(a_{t}-b_{t})\sigma_{t}d\leavevmode\hbox to10.93pt{\vbox to 9.43pt{\pgfpicture\makeatletter\hbox{\hskip 5.51666pt\lower-3.41666pt\hbox to0 .0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-5.41667pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{W}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{4.91667 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{-3.9166 7pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}+\frac{1}{2}\int_{0 }^{T}\sigma_{t}^{-2}(b_{t}^{2}-a_{t}^{2}){\mathrm{d}}t$ (25) $\displaystyle=\frac{1}{2}\int_{0}^{T}\sigma_{t}^{-2}[2a_{t}b_{t}-2b_{t}^{2}+b_ {t}^{2}-a_{t}^{2}]{\mathrm{d}}t+\int_{0}^{T}\sigma_{t}h_{t}{\mathrm{d}} \leavevmode\hbox to10.93pt{\vbox to9.43pt{\pgfpicture\makeatletter\hbox{\hskip 5 .51666pt\lower-3.41666pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-5.41667pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{W}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{4.91667 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{-3.9166 7pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}$ (26) $\displaystyle=-\frac{1}{2}\int_{0}^{T}\sigma_{t}^{-2}(b_{t}-a_{t})^{2}{\mathrm {d}}t+\int_{0}^{T}\sigma_{t}h_{t}{\mathrm{d}}\leavevmode\hbox to10.93pt{\vbox to 9.43pt{\pgfpicture\makeatletter\hbox{\hskip 5.51666pt\lower-3.41666pt\hbox to0 .0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-5.41667pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{W}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{4.91667 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{-3.9166 7pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}$ (27) $\displaystyle=-\frac{1}{2}\int_{0}^{T}\sigma_{t}^{2}\|h_{t}\|_{2}^{2}{\mathrm{ d}}t+\int_{0}^{T}\sigma_{t}h_{t}{\mathrm{d}}\leavevmode\hbox to10.93pt{\vbox to 9.43pt{\pgfpicture\makeatletter\hbox{\hskip 5.51666pt\lower-3.41666pt\hbox to0 .0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-5.41667pt}{-3.41666pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\bm{W}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{4.91667 pt}{5.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-5.41667pt}{5.91666pt}\pgfsys@lineto{-3.9166 7pt}{4.91666pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},$ (28) $$

evaluated on a trajectory ${\bm{H}}$ from ${\mathbb{P}}_{h}$.

###### Proposition 6 .

### A.1 Discrete Version for DDPM

We can also express the RND in the discrete setting. For this derivation, we make use of the DDPM schedule (Ho et al.,, 2020). The generalisation to different schedules is straightforward. The path probability of the pre-trained model is given as

$$ $\displaystyle p_{\theta}^{\text{data}}({\bm{x}}_{0},\dots,{\bm{x}}_{T})=p_{T}( {\bm{x}}_{T})\prod_{t=1}^{T}p_{\theta}^{\text{data}}({\bm{x}}_{t-1}|{\bm{x}}_{ t}),\quad p_{\theta}^{\text{data}}({\bm{x}}_{t-1}|{\bm{x}}_{t})=\mathcal{N}({ \bm{x}}_{t-1}|\mu_{\theta}({\bm{x}}_{t},t);\tilde{\beta}_{t}^{2}I)$ (29) $$

where the mean is parametrised as $\mu_{\theta}({\bm{x}}_{t},t)=\frac{1}{\sqrt{\alpha}_{t}}({\bm{x}}_{t}-\frac{1-
\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}({\bm{x}}_{t},t))$. Similar, we have path probabilities for the fine-tuned model as

$$ $\displaystyle p_{\varphi}^{\text{h}}({\bm{x}}_{0},\dots,{\bm{x}}_{T})=p_{T}({ \bm{x}}_{T})\prod_{t=1}^{T}p_{\varphi}^{\text{h}}({\bm{x}}_{t-1}|{\bm{x}}_{t}) ,\quad p_{\varphi}^{h}({\bm{x}}_{t-1}|{\bm{x}}_{t})=\mathcal{N}({\bm{x}}_{t-1} |\mu_{h}({\bm{x}}_{t},t);\tilde{\beta}_{t}^{2}I),$ (30) $$

with mean $\mu_{h}({\bm{x}}_{t},t)=\mu_{\theta}({\bm{x}}_{t},t)+\Delta_{h}({\bm{x}}_{t},t)$ is the original mean plus a delta given by the $h$-transform. We use $\tilde{\beta}_{t}=\sqrt{\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{
t}}$ for the standard deviation of the reverse kernel. Using this setting, we can write the RND as

$$ $\displaystyle\ln\left(\frac{p_{\theta}^{\text{data}}({\bm{x}}_{0},\dots,{\bm{x }}_{T})}{p_{\varphi}^{\text{h}}({\bm{x}}_{0},\dots,{\bm{x}}_{T})}\right)=\sum_ {t=1}^{T}\operatorname{ln}\left(\frac{p_{\theta}^{\text{data}}({\bm{x}}_{t-1}| {\bm{x}}_{t})}{p_{\varphi}^{h}({\bm{x}}_{t-1}|{\bm{x}}_{t})}\right),$ (31) $$

where we assumed that the terminal distribution $p_{T}$ is the same for both diffusion models. The log ratio of two Gaussian reduces to

$$ $\displaystyle\operatorname{ln}\left(\frac{\mathcal{N}(x;\mu_{1},\Sigma_{1})}{ \mathcal{N}(x;\mu_{2},\Sigma_{2})}\right)=-\frac{1}{2}[(x-\mu_{1})^{T}\Sigma_{ 1}^{-1}(x-\mu_{1})-(x-\mu_{2})^{T}\Sigma_{2}^{-1}(x-\mu_{2})]+\frac{1}{2} \operatorname{ln}\frac{|\Sigma_{2}|}{|\Sigma_{1}|},$ (32) $$

which gets us

$$ $\displaystyle\operatorname{ln}\left(\frac{p_{\theta}^{\text{data}}({\bm{x}}_{t -1}|{\bm{x}}_{t})}{p_{\varphi}^{h}({\bm{x}}_{t-1}|{\bm{x}}_{t})}\right)=-\frac {1}{2\tilde{\beta}_{t}^{2}}\left[\|{\bm{x}}_{t-1}-\mu_{\theta}({\bm{x}}_{t},t) \|_{2}^{2}-\|{\bm{x}}_{t-1}-\mu_{h}({\bm{x}}_{t},t)\|_{2}^{2}\right].$ (33) $$

To further simplify these terms, we need some information about the trajectory ${\bm{x}}_{0},\dots,{\bm{x}}_{T}$. In particular, we assume that we have a trajectory sampled from the fine-tuned model, i.e.,

$$ $\displaystyle{\bm{x}}_{t-1}=\mu_{h}({\bm{x}}_{t},t)+\tilde{\beta}_{t}\epsilon, \quad\epsilon\sim\mathcal{N}(0,I).$ (34) $$

Given the parametrisation of the mean in the diffusion model

$$ $\displaystyle\mu_{\theta}({\bm{x}}_{t},t)$ $\displaystyle=\frac{1}{\sqrt{\alpha}_{t}}\left({\bm{x}}_{t}-\frac{1-\alpha_{t} }{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}({\bm{x}}_{t},t)\right),$ (35) $\displaystyle\mu_{h}({\bm{x}}_{t},t)$ $\displaystyle=\frac{1}{\sqrt{\alpha}_{t}}\left({\bm{x}}_{t}-\frac{1-\alpha_{t} }{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}({\bm{x}}_{t},t)-\frac{1-\alpha_{ t}}{\sqrt{1-\bar{\alpha}_{t}}}h_{\varphi}({\bm{x}}_{t},t)\right),$ (36) $$

we can reduce the terms on the RHS of Equation ([33](https://arxiv.org/html/2502.04468v2#A1.E33)) to

$$ $\displaystyle\|{\bm{x}}_{t-1}-\mu_{\theta}({\bm{x}}_{t},t)\|_{2}^{2}-\|{\bm{x} }_{t-1}-\mu_{h}({\bm{x}}_{t},t)\|_{2}^{2}$ (37) $\displaystyle=\|\mu_{h}({\bm{x}}_{t},t)+\tilde{\beta}_{t}\epsilon-\mu_{\theta} ({\bm{x}}_{t},t)\|_{2}^{2}-\|\mu_{h}({\bm{x}}_{t},t)+\tilde{\beta}_{t}\epsilon -\mu_{h}({\bm{x}}_{t},t)\|_{2}^{2}$ (38) $\displaystyle=\|\Delta_{h}({\bm{x}}_{t},t)+\tilde{\beta}_{t}\epsilon\|_{2}^{2} -\|\tilde{\beta}_{t}\epsilon\|_{2}^{2}.$ (39) $$

Combining Equation ([39](https://arxiv.org/html/2502.04468v2#A1.E39)) and Equation ([33](https://arxiv.org/html/2502.04468v2#A1.E33)), we obtain

$$ $\displaystyle\operatorname{ln}\left(\frac{p_{\theta}^{\text{data}}({\bm{x}}_{t -1}|{\bm{x}}_{t})}{p_{\varphi}^{h}({\bm{x}}_{t-1}|{\bm{x}}_{t})}\right)$ $\displaystyle=-\frac{1}{2\tilde{\beta}_{t}^{2}}\left[\|{\bm{x}}_{t-1}-\mu_{ \theta}({\bm{x}}_{t},t)\|_{2}^{2}-\|{\bm{x}}_{t-1}-\mu_{h}({\bm{x}}_{t},t)\|_{ 2}^{2}\right]$ (40) $\displaystyle=-\frac{1}{2\tilde{\beta}_{t}^{2}}\left[\|\Delta_{h}({\bm{x}}_{t} ,t)+\tilde{\beta}_{t}\epsilon\|_{2}^{2}-\|\tilde{\beta}_{t}\epsilon\|_{2}^{2}\right]$ (41) $\displaystyle=-\frac{1}{2\tilde{\beta}_{t}^{2}}\left[\|\Delta_{h}({\bm{x}}_{t} ,t)\|_{2}^{2}+2\Delta_{h}({\bm{x}}_{t},t)^{\top}\tilde{\beta}_{t}\epsilon+\| \tilde{\beta}_{t}\epsilon\|_{2}^{2}-\|\tilde{\beta}_{t}\epsilon\|_{2}^{2}\right]$ (42) $\displaystyle=-\frac{1}{2\tilde{\beta}_{t}^{2}}\|\Delta_{h}({\bm{x}}_{t},t)\|_ {2}^{2}-\frac{1}{\tilde{\beta}_{t}}\Delta_{h}({\bm{x}}_{t},t)^{\top}\epsilon.$ (43) $$

For the full RND we obtain

$$ $\displaystyle\sum_{t=1}^{T}\operatorname{ln}\left(\frac{p_{\theta}^{\text{data }}({\bm{x}}_{t-1}|{\bm{x}}_{t})}{p_{\varphi}^{h}({\bm{x}}_{t-1}|{\bm{x}}_{t})} \right)=\sum_{t=1}^{T}\left[-\frac{1}{2}\tilde{\beta}^{-2}\|\Delta_{h}({\bm{x} }_{t},t)\|_{2}^{2}+\tilde{\beta}^{-1}\Delta_{h}({\bm{x}}_{t},t)^{\top}\epsilon \right],$ (44) $$

which mimics a discretised version of the continuous RND in Equation ([28](https://arxiv.org/html/2502.04468v2#A1.E28)).

## Appendix B Resampling Step

### B.1 Proof to Theorem 5

###### Proof.

Part (i) is a Bayes theorem.

The proof of part (ii) follows similar ideas as (Hertrich and Gruhlke,, 2024, Prop 8 (ii)). To this end, let $Z=\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}[\alpha({\bm{x}}_{[0,T]})]$.
Then, we can estimate by Jensens’ inequality that

$$ $\displaystyle-\operatorname{ln}(Z)$ $\displaystyle=-\operatorname{ln}\left(\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb {P}}_{h}}\left[\min\left(\frac{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}{{ \mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))}{c}, 1\right)\right]\right)$ $\displaystyle\leq-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}\left[ \operatorname{ln}\left(\min\left(\frac{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data} }}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))} {c},1\right)\right)\right]$ $\displaystyle=-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}\left[\min \left(\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}{{ \mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))}{c} \right),0\right)\right]$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}\left[\max\left (\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))} \right),0\right)\right]$ $$

On the other side, we have by definition that

$$ $\displaystyle\mathrm{KL}(\tilde{{\mathbb{P}}}_{h},{\mathbb{P}}_{\mathrm{data}} )-\mathbb{E}_{{\bm{x}}_{[0:T]}\sim\tilde{{\mathbb{P}}}_{h}}[r({\bm{x}}_{0})]$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim\tilde{{\mathbb{P}}}_{h}}\left[ \operatorname{ln}\left(\frac{{\mathrm{d}}\tilde{{\mathbb{P}}}_{h}}{{\mathrm{d} }{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\right)\right]-\mathbb{E}_{{ \bm{x}}_{[0:T]}\sim\tilde{{\mathbb{P}}}_{h}}[r({\bm{x}}_{0})]$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim\tilde{{\mathbb{P}}}_{h}}\left[ \operatorname{ln}\left(\frac{cZ}{\exp(r({\bm{x}}_{0}))}\frac{{\mathrm{d}} \tilde{{\mathbb{P}}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{ [0,T]})\right)\right]-\operatorname{ln}(cZ).$ $$

By taking the expectation over ${\mathbb{P}}_{h}$ instead of $\tilde{{\mathbb{P}}}_{h}$ this is equal to

$$ $\displaystyle\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\frac{{ \mathrm{d}}\tilde{{\mathbb{P}}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[ 0,T]})\operatorname{ln}\left(\frac{cZ}{\exp(r({\bm{x}}_{0}))}\frac{{\mathrm{d} }\tilde{{\mathbb{P}}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]}) \frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}( {\bm{x}}_{[0,T]})\right)\right]-\operatorname{ln}(cZ).$ $$

Inserting the formula from part (i) and then the definition of $\alpha$, this is equal to

$$ $\displaystyle\qquad\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[ \frac{\alpha({\bm{x}}_{[0,T]})}{Z}\operatorname{ln}\left(\frac{c}{\exp(r(x))} \alpha({\bm{x}}_{[0,T]})\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\right)\right]-\operatorname{ln} (cZ)$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\frac{ \alpha({\bm{x}}_{[0,T]})}{Z}\operatorname{ln}\left(\frac{c}{\exp(r(x))}\min \left(1,\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{\mathrm{d}}{\mathbb{P}} _{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))}{c}\right)\frac{{\mathrm{d} }{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]}) \right)\right]-\operatorname{ln}(cZ)$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\frac{ \alpha({\bm{x}}_{[0,T]})}{Z}\operatorname{ln}\left(\min\left(\frac{{\mathrm{d} }{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]}) \frac{c}{\exp(r({\bm{x}}_{0}))},1\right)\right)\right]-\operatorname{ln}(cZ)$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\frac{ \alpha({\bm{x}}_{[0,T]})}{Z}\min\left(\operatorname{ln}\left(\frac{{\mathrm{d} }{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]}) \frac{c}{\exp(r({\bm{x}}_{0}))}\right),0\right)\right]-\operatorname{ln}(cZ)$ $\displaystyle=\frac{1}{Z}\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}} \left[\min\left(\alpha({\bm{x}}_{[0,T]})\operatorname{ln}\left(\frac{{\mathrm{ d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T] })\frac{c}{\exp(r({\bm{x}}_{0}))}\right),0\right)\right]-\operatorname{ln}(cZ)$ $\displaystyle=\frac{1}{Z}\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}} \left[\min\left(\min\left(1,\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{ \mathrm{d}}{\mathbb{P}}_{h}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))}{c} \right)\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}} {\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))} \right),0\right)\right]-\operatorname{ln}(cZ).$ $$

Since $1\leq\frac{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}{{\mathrm{d}}{\mathbb{P}}_{h
}}({\bm{x}}_{[0,T]})\frac{\exp(r({\bm{x}}_{0}))}{c}$ if and only if $\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{
\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))}
\right)\leq 0$ the minimum is attained either for both $\min$ in the above formula in the first argument or it is attained for both $\min$ in the second argument. Consequently the above formula is equal to

$$ $\displaystyle\frac{1}{Z}\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left [\min\left(\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm {d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0 }))}\right),0\right)\right]-\operatorname{ln}(cZ)$ $\displaystyle\leq\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\min \left(\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))} \right),0\right)\right]-\operatorname{ln}(Z)-\operatorname{ln}(c),$ $$

where the inequality comes from the fact that $Z\in(0,1]$ and that the expectation is non-positive (since the integrand is non-positive). Inserting the formula of $-\operatorname{ln}(Z)$ from the beginning of the proof, this is equal to

$$ $\displaystyle\quad\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[\min \left(\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))} \right),0\right)\right]$ $\displaystyle\qquad+\mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h}}\left[ \max\left(\operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{ d}}{\mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0} ))}\right),0\right)\right]-\operatorname{ln}(c)$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[ \operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\frac{c}{\exp(r({\bm{x}}_{0}))} \right)\right]-\operatorname{ln}(c)$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}\left[ \operatorname{ln}\left(\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{ \mathbb{P}}_{\mathrm{data}}}({\bm{x}}_{[0,T]})\right)\right]-\mathbb{E}_{{\bm{ x}}_{[0:T]}\sim{\mathbb{P}}_{h}}[r({\bm{x}}_{0})]$ $\displaystyle=\mathrm{KL}({\mathbb{P}}_{h},{\mathbb{P}}_{\mathrm{data}})- \mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h}}[r({\bm{x}}_{0})].$ $$

This concludes the proof of part (ii).

For part (iii), we observe that the loss in ([15](https://arxiv.org/html/2502.04468v2#S3.E15)) is learning the marginal score of a forward SDE (in our case a VP-SDE intiliased at $\tilde{{\mathbb{P}}}_{h}^{0}$, by construction the associated path measure of this process is given by (See appendix A (Vargas et al., 2023a,) for a similar sketch) (^2^22Note we slightly abuse notation here as these equalities are meant to be understood in an RND sense but we omit the full ratio notation for brevity.):

$$ $\displaystyle{\mathbb{P}}_{h^{*}}(\cdot)$ $\displaystyle={\mathbb{P}}_{\mathrm{forward}}(\cdot|{\bm{x}}_{0})\tilde{{ \mathbb{P}}}_{h}^{0}({\bm{x}}_{0})$ (45) $\displaystyle={\mathbb{P}}_{\mathrm{forward}}(\cdot|{\bm{x}}_{0}){\mathbb{P}}_ {\mathrm{data}}({\bm{x}}_{0})\cdot\frac{\tilde{{\mathbb{P}}}_{h}^{0}}{p_{ \mathrm{data}}}({\bm{x}}_{0})$ (46) $\displaystyle=\frac{\tilde{{\mathbb{P}}}_{h}^{0}}{p_{\mathrm{data}}}{\mathbb{P }}_{\mathrm{data}}(\cdot)$ (47) $$

Thus, the path measure minimising the score-matching loss satisfies

$$ $\displaystyle{\mathbb{P}}_{h^{*}}=\frac{\tilde{{\mathbb{P}}}_{h}^{0}}{p_{ \mathrm{data}}}{\mathbb{P}}_{\mathrm{data}}$ (48) $$

Then it follows that

$$ $\displaystyle\mathrm{KL}({\mathbb{P}}_{h^{*}}||{\mathbb{P}}_{\mathrm{data}})$ $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim{\mathbb{P}}_{h^{*}}}\left[\ln \left(\frac{{\mathrm{d}}{\mathbb{P}}_{\mathrm{data}}}{{\mathrm{d}}{\mathbb{P}} _{\mathrm{data}}}({\bm{x}}_{[0:T]})\frac{\tilde{{\mathbb{P}}}_{h}^{0}({\bm{x}} _{0})}{p_{\mathrm{data}}({\bm{x}}_{0})}\right)\right]$ (49) $\displaystyle=\mathbb{E}_{{\bm{x}}_{[0:T]}\sim\frac{\tilde{{\mathbb{P}}}_{h}^{ 0}}{p_{\mathrm{data}}}{\mathbb{P}}_{\mathrm{data}}}\left[\ln\left(\frac{\tilde {{\mathbb{P}}}_{h}^{0}({\bm{x}}_{0})}{p_{\mathrm{data}}({\bm{x}}_{0})}\right)\right]$ (50) $\displaystyle=\mathbb{E}_{{\bm{x}}_{0}\sim\tilde{{\mathbb{P}}}_{h}^{0}}\left[ \ln\left(\frac{\tilde{{\mathbb{P}}}_{h}^{0}({\bm{x}}_{0})}{p_{\mathrm{data}}({ \bm{x}}_{0})}\right)\right]=\mathrm{KL}(\tilde{{\mathbb{P}}}_{h}^{0},p_{ \mathrm{data}})$ (51) $$

Via the disintegration theorem (Léonard,, 2014, Theorem 1.6. and Theorem 2.4), the KL divergence can be decomposed as

$$ $\displaystyle\mathrm{KL}(\tilde{{\mathbb{P}}}_{h},{\mathbb{P}}_{\mathrm{data}} )=\mathbb{E}[\mathrm{KL}(\tilde{{\mathbb{P}}}_{h}(\cdot|{\bm{x}}_{0}),{\mathbb {P}}_{\mathrm{data}}(\cdot|{\bm{x}}_{0}))]+\mathrm{KL}(\tilde{{\mathbb{P}}}_{h }^{0},p_{\mathrm{data}})$ (52) $$

Where $\mathbb{E}[\mathrm{KL}(\tilde{{\mathbb{P}}}_{h}(\cdot|{\bm{x}}_{0}),{\mathbb{P
}}_{\mathrm{data}}(\cdot|{\bm{x}}_{0}))]\geq 0$ and thus

$$ $\mathrm{KL}({\mathbb{P}}_{h^{*}},{\mathbb{P}}_{\mathrm{data}})=\mathrm{KL}( \tilde{{\mathbb{P}}}_{h}^{0},p_{\mathrm{data}})\leq\mathrm{KL}(\tilde{{\mathbb {P}}}_{h},{\mathbb{P}}_{\mathrm{data}}).$ $$

Since the marginals of ${\mathbb{P}}_{h^{*}}$ and $\tilde{{\mathbb{P}}}_{h}$ at time 0 0 coincide, we obtain

$$ $\displaystyle\mathcal{F}({\mathbb{P}}_{h^{*}})$ $\displaystyle=\mathrm{KL}({\mathbb{P}}_{h^{*}},{\mathbb{P}}_{\text{data}})- \mathbb{E}_{{\bm{x}}_{[0,T]}\sim{\mathbb{P}}_{h^{*}}}[r({\bm{x}}_{0})]$ (53) $\displaystyle=\mathrm{KL}({\mathbb{P}}_{h^{*}},{\mathbb{P}}_{\text{data}})- \mathbb{E}_{{\bm{x}}_{[0,T]}\sim\tilde{{\mathbb{P}}}_{h}}[r({\bm{x}}_{0})]$ (54) $\displaystyle\leq\mathrm{KL}(\tilde{{\mathbb{P}}}_{h},{\mathbb{P}}_{\text{data }})-\mathbb{E}_{{\bm{x}}_{[0,T]}\sim\tilde{{\mathbb{P}}}_{h}}[r({\bm{x}}_{0})] =\mathcal{F}(\tilde{{\mathbb{P}}}_{h}),$ (55) $$

which shows the first inequality from the claim. The second inequality was proven in part (ii).
∎

###### Proof.

## Appendix C Experimental Details and Additional Results

### C.1 2D Toy Example

We train the initial diffusion model on a Gaussian mixture model with modes $25$ equally weighted modes where the set of means is defined by $\{-2.5,-1.25,0,1.25,2.5\}^{2}$ and the covariance matrix is given by $0.1I$. The reward function $r$ is defined as the negative log likelihood function of a GMM with four modes with means $(-2.5,1.25)$, $(-1.25,2.5)$, $(1.25,0)$ and $(2.5,-1.25)$, covariance matrix $0.1I$ and mode weights $\frac{1}{8}$, $\frac{1}{8}$, $\frac{5}{8}$ and $\frac{1}{8}$.
For classifier guidance, we choose the parameter $\gamma$ by maximising the expected reward, which is $\gamma=0.3$.
For the importance and reward-only fine-tuning we use a batch size of $4096$, a buffer size of $6000$ and a KL regularisation with $\alpha_{\text{KL}}=0.06$. We initialise the buffer with samples from the initial score-model.
Then we perform $20$ fine-tuning steps with $500$ gradient updates per iteration. We choose $c$ such that $40\%$ of the samples will be rejected.
The total training time is approximately $1$min on a single NVIDIA GeForce RTX 4090.

### C.2 Posterior Sampling for inverse problems

We train a unconditional score-based diffusion model on the Flowers dataset with the VP-SDE with $\beta_{\text{min}}=0.1$ and $\beta_{\text{max}}=20.0$. We parametrise the score model using a small Attention UNet (Dhariwal and Nichol,, 2021) with approx. $10$M parameters. We parametrise the $h$-transform as in Equation ([16](https://arxiv.org/html/2502.04468v2#S3.E16)) with approx. $0.8$M parameters. The scaling network $\text{NN}_{2}(t)$ was initialised as a linear function going from 0 0 to $0.001$. In this already at the start of training, we obtain trajectories which are similar to the ground truth leading to more stable optimisation. We used $50$ importance fine-tuning iterations with $20$ gradient updates per iteration. For every iteration we sampled $64$ new trajectories and used a resampling rate of $50\%$. We used a maximal buffer size of $256$ images and no KL-regularisation. The total training time is approximately $8$min on a single NVIDIA GeForce RTX 4090.

At the start of fine-tuning the end points ${\bm{x}}_{0}$ of most trajectories are not consistent with the measurements ${\bm{y}}^{\delta}$, leading to a low reward $r({\bm{x}}_{0};{\bm{y}}^{\delta})$. This can sometimes lead to instabilities at the start of training, if no valid high reward trajectory was sampled. We found that it was helpful to amortise over measurements, such that we consider the amortised loss function

$$ $\displaystyle\mathcal{L}_{FT}(g)=\mathop{\mathbb{E}}_{\begin{subarray}{c}{\bm{ x}}_{0}\sim\tilde{{\mathbb{P}}}_{h^{k}}^{0},{\bm{y}}\sim p^{\text{lhkd}}({\bm{ y}}|{\bm{x}}_{0})\\ t\sim\mathrm{U}(0,T),{\bm{x}}_{t}\sim\leavevmode\hbox to3.62pt{\vbox to6.22pt{ \pgfpicture\makeatletter\hbox{\hskip 1.76094pt\lower-2.18748pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-1.76094pt}{-0.8264pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-1.41093pt}{3.93748pt}\pgfsys@lineto{1.76094 pt}{3.93748pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{1.76094pt}{3.93748pt}\pgfsys@lineto{0.71094 pt}{3.23749pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}(\cdot|{\bm{x}}_{ 0})\end{subarray}}\left[\left\|\left({g_{t}({\bm{x}}_{t},{\bm{y}})}\!+\!s_{t}( {\bm{x}}_{t})\right)\!-\!\nabla_{{\bm{x}}_{t}}\ln\leavevmode\hbox to5.13pt{ \vbox to8.85pt{\pgfpicture\makeatletter\hbox{\hskip 2.51563pt\lower-3.12498pt \hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-2.51563pt}{-1.18056pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-2.01563pt}{5.62498pt}\pgfsys@lineto{2.51563 pt}{5.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{2.51563pt}{5.62498pt}\pgfsys@lineto{1.01563 pt}{4.62498pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t|0}({\bm{x}}_{t}|{ \bm{x}}_{0})\right\|^{2}\right],$ (56) $$

where the simulated measurements ${\bm{y}}\sim p^{\text{lhkd}}({\bm{y}}|{\bm{x}}_{0})$ are added as an additional input to the $h$-transform.

### C.3 Class conditional sampling

We pre-train a score-based diffusion model on the MNIST dataset with the VP-SDE with $\beta_{\text{min}}=0.1$ and $\beta_{\text{max}}=20.0$. We parametrise the score model using a small Attention UNet (Dhariwal and Nichol,, 2021) with approx. $1.1$M parameters. We parametrise the $h$-transform as in Equation ([16](https://arxiv.org/html/2502.04468v2#S3.E16)) with approx. $0.8$M parameters. For all experiments, we use a batch size of $256$, a buffer size of $2048$ and a reward scaling $\lambda=4.0$. We perform $50$ importance fine-tuning iterations with $50$ gradient updates per iterations. We use the KL regulariser with $\alpha_{\text{KL}}=0.001$. Lastly, we adaptively choose $c$ such that $10\%$ of the samples will be accepted for the first $10$ steps. After the initial $10$ steps, we accept $30\%$ of the samples. We start with $10\%$ as MNIST has $10$ classes with roughly similar class probabilities. The total training time is about $10$min on a single NVIDIA GeForce RTX 4090.

For the comparison methods in Table [1](https://arxiv.org/html/2502.04468v2#S4.T1), we use classifier guidance and online fine-tuning. For classifier guidance, we used a scaling $\gamma=4.5$ maximising sample quality. For online fine-tuning we directly minimise ([10](https://arxiv.org/html/2502.04468v2#S2.E10)) using VarGrad (Richter et al.,, 2020). VarGrad enables us to detach the trajectory from the backpropagation, thus saving memory cost and enabling us to train with a larger batch size. In particular, we use the formulation in Denker et al., (2024) as

$$ $\displaystyle D_{\mathrm{logvar}}({\mathbb{P}}_{h},{\mathbb{P}}_{\text{data}}; \mathbb{W})=\text{Var}_{{\bm{H}}^{g_{t}}_{0:T}\sim\mathbb{W}}\left[\ln\frac{{ \mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{\text{data}}}({\bm{H}}^ {g_{t}}_{0:T})\right],$ (57) $$

evaluated at a reference process $\mathbb{W}=\mathrm{Law}({\bm{H}}^{g_{t}}_{0:T})$, given by

$$ $\displaystyle{\bm{H}}_{T}$ $\displaystyle\sim Q_{T}^{f_{t}}[p_{\text{tilted}}]$ $\displaystyle{\mathrm{d}}{\bm{H}}_{t}$ $\displaystyle=\left(f_{t}({\bm{H}}_{t})-\sigma^{2}_{t}(s_{t}({\bm{H}}_{t})+g_{ t}({\bm{H}}_{t}))\right)\,{\mathrm{d}}t+\sigma_{t}\;\leavevmode\hbox to15.93pt {\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 8.01668pt\lower-3.47221pt \hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t}.$ (58) $$

In particular, if we choose $g_{t}=\mathrm{stop_grad}(h_{t})$ as a detached copy of the current estimation $h_{t}$, we obtain

$$ $\displaystyle\ln\frac{{\mathrm{d}}{\mathbb{P}}_{h}}{{\mathrm{d}}{\mathbb{P}}_{ \text{data}}}({\bm{H}}^{g_{t}}_{0:T})$ $\displaystyle=-\frac{1}{2}\int_{0}^{T}\sigma_{t}^{2}\|h_{t}({\bm{H}}_{t}^{g_{t }})\|^{2}{\mathrm{d}}t+\int_{0}^{T}\sigma_{t}^{2}(g_{t}^{\top}h_{t})({\bm{H}}_ {t}^{g_{t}}){\mathrm{d}}t-r({\bm{x}}_{0}^{g_{t}})$ $\displaystyle+\int_{0}^{T}\sigma_{t}h_{t}^{\top}({\bm{H}}_{t}^{g_{t}}) \leavevmode\hbox to15.93pt{\vbox to9.54pt{\pgfpicture\makeatletter\hbox{\hskip 8 .01668pt\lower-3.47221pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-7.91669pt}{-3.47221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{${\mathrm{d}}{\mathbf{W }}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{}{}{{}}{}{{}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{7.41669 pt}{5.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{}{}{{}}{}{ {}}{}{} {}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{0.2pt} \pgfsys@invoke{ }{}\pgfsys@moveto{-7.91669pt}{5.97221pt}\pgfsys@lineto{-6.4166 9pt}{4.97221pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{ {}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{t},$ (59) $$

using the RND for time reverse SDEs see Equation 64 in (Vargas et al.,, 2024). We use the same reward-informed network architecture to parametrise $h_{t}$.

In Figure [5](https://arxiv.org/html/2502.04468v2#A3.F5) we show the posterior for all $10$ classes. For all experiments, we chose the same hyperparameters as well as the same initial seed for the samples presented. We observe that these settings work for most classes. However, for class ”Four” we see a single digit ”Six” inthe image, giving evidence that this particular model has not fully converged.

Figure: Posterior: Zero
Refer to caption: extracted/6605494/imgs/cond_samples_zero.png

### C.4 Reward Fine-tuning

Diffusion models can be used for conditional generation via classifier free guidance (Ho and Salimans,, 2022). In classifier free guidance both an unconditional $\epsilon^{\theta}_{t}({\bm{x}}_{t})$ and a conditional $\epsilon^{\theta}_{t}({\bm{x}}_{t},c)$ noise-prediction model are learned by randomly masking out the text prompt $c$. During sampling the linear combination $\bar{\epsilon}^{\theta}_{t}({\bm{x}}_{t},t)=(1+w)\epsilon^{\theta}_{t}({\bm{x}
}_{t},c)-w\epsilon^{\theta}_{t}({\bm{x}}_{t})$, with a guidance scale $w$, is used. Despite the progress in training text-to-image diffusion models, the samples are not always aligned with human preferences. For the alignment we make use of ImageReward-v1.0, a human preference reward model (Xu et al.,, 2024). These reward models $r({\bm{x}};c)$ are trained to produce a reward from a given text prompt $c$ and an image ${\bm{x}}$, corresponding to human preferences. Here, we learn the tilted distribution for a given text prompt and directly fine-tune the classifier free model $\bar{\epsilon}^{\theta}_{t}({\bm{x}}_{t},t)$ using the LoRA parametrisation (Hu et al.,, 2022) instead of the reward-informed parametrisation. We use a LoRA for all attention layers with a rank of $10$. We used a buffer of $64$ images and sample $128$ new images at every iteration, the parameter $c$ was chosen such at $10\%$ of the images were accepted. We used a total of $25$ iteration, where we did $40$ gradient descent steps with an effective batch size of $64$ at each iteration (batch of $4$ and $16$ gradient accumulation steps). Fine-tuning took about $7$ hours on a single NVIDIA Geforce RTX 4090. We used the AdamW optimiser (Loshchilov and Hutter,, 2019) with a learning rate of $1\times 10^{-4}$ and a cosine decay to $1\times 10^{-5}$. For the final sampling evaluation we used the DDIM scheduler with $50$ time steps. We used a guidance scale of $5.0$ for Stable Diffusion.

We compare against Top-K sampling, DPOK (Fan et al.,, 2024) and Adjoint Matching (Domingo-Enrich et al.,, 2025). We give the experimental details in the following paragraphs.

Additional images are shown in Figure [7](https://arxiv.org/html/2502.04468v2#A3.F7) and Figure [8](https://arxiv.org/html/2502.04468v2#A3.F8). We also observe the reward and diversity during iterations, see Figure [6](https://arxiv.org/html/2502.04468v2#A3.F6).

Top-K Sampling is a widely known approach to increase the performance of text-to-image diffusion models without additional training. Here, we sample a batch of $K$ images and only return the image with the highest reward. In our implementation we use $K=6$. Top-K sampling is easy to implement and requires no additional training loss, but often decrease diversity and increased the sampling time by a factor of $K$.

DPOK was proposed by (Fan et al.,, 2024) to fine-tune diffusion models using a differentiable reward. We use the implementation by the authors using the provided hyperparameters(^3^33[https://github.com/google-research/google-research/tree/master/dpok](https://github.com/google-research/google-research/tree/master/dpok)). DPOK used LoRA for the fine-tuning parametrisation, we use the default parameters as suggested by the authors. Fine-tuning takes $28$h on a single A100.

Adjoint Matching reformulates fine-tuning as a stochastic optimal control problem and casting it as a regression problem. We use the implementation by the authors with the provided hyperparameters(^4^44[https://github.com/microsoft/soc-fine-tuning-sd](https://github.com/microsoft/soc-fine-tuning-sd)). Adjoint Matching fine-tunes the full model and does not use LoRA. This requires large GPUs and the experiments cannot be performed on a NVIDIA Geforce RTX 4090. We use the adjoint matching variant which uses CFG to sample trajectories, but not in the adjoint computation. This setting is not covered by theory, but leads to the best empirical results.

Figure: Figure 6: Reward and diversity for text-to-image fine-tuning on ”A green colored rabbit.” during training. As known for LoRA fine-tuning the diversity decreases over iterations.
Refer to caption: x3.png

Figure: Base Model
Refer to caption: extracted/6605494/imgs/Two_roses_in_a_vase/base_model/img_0.png

Figure: Base Model
Refer to caption: extracted/6605494/imgs/Two_dogs_in_the_park/base_model/img_0.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How crucial is the relaxed importance sampling step - eg. if you resampled directly based on the importance weights  - what would the effective sample size be? Or would it be infeasible?
2. Given that rejection sampling is used - do you sample a fixed number of trajectories in the outer loop, then accept a portion depending on the weights (to put in the buffer), or do you keep sampling until a fixed number of trajectories is accepted? 
3. The claim in 
    
    > We emphasize that our importance-based fine-tuning does not require to differentiate the score function of the base-model and never considers more **than one step of the generation process at once**. In particular, it can
    be applied for very large base-models where other methods run out of memory.
    > 
    
    is inaccurate (or unclear) since the method requires running the generation process to obtain trajectories (and importance weights). Did this mean to emphasize that the method doesn’t backpropagate at the level of the trajectory?
    

4. Experimental questions:
    - For the classifier guidance used for the 2D GMM, and MNIST - was DPS used, or was a time-conditioned classifier trained?
    - The main gap in performance for adjoint matching appears to be from moving to a LoRA parameterization. Is there any understanding for why this impacts the method so much?
    - Why is DPOK memory listed as O(B) instead of O(BT)? The (on-policy) gradient update in that paper appears to require backpropagation at each step of the generation trajectory.
        - This line (431) is inconsistent with the table “Importance fine-tuning reaches a competitive
        reward with similar diversity and the same O(B) memory footprint as DPOK or RTB”
soundness: 3
strengths: 1. The main important contribution of the paper is the use of relaxed importance sampling (from (Hertrich & Gruhlke, 2025)) to bypass the requirement of a fine-tuning dataset (as in DEFT), which is a useful and important improvement over that algorithm
    - Theorem 6, and the empirical validation in Fig 2 is useful for demonstrating that the relaxed importance sampling method doesn’t compromise theoretical validity of the method.
2. The method has a benefit over other methods in not storing (or backpropagating through) trajectories for training
3. The experimental settings chosen are relevant and useful for the proposed method (class conditional sampling, inverse problems, and prompt-alignment)
    - The method is demonstrated on a larger scale problem in prompt alignment for Stable Diffusion
4. The work is clearly written and has a logical presentation of ideas
summary: This work tackles the problem of fine-tuning a diffusion model $p_\theta(x_0)$ to sample from the reward-tilted distribution $p_{\mathrm{tilted}}(x) \propto p_\theta(x_0) \exp(r(x_0))$. It does so by obtaining approximate samples from $p_{\mathrm{tilted}}(x)$, and then minimizing the denoising score matching loss on these samples. The approximate samples are obtained by sampling from the model in its current iteration, and correcting them using a rejection sampling scheme with importance weights computed over trajectories. This optimization algorithm (with importance sampling) is shown to monotonically minimize the reverse KL ($KL(p_{\theta}(x) || p_{\mathrm{tilted}}(x))$ to the target distribution. The method is demonstrated by reward-tilted sampling for 2D Gaussian mixture models, class conditional sampling in MNIST, a linear-inverse problem in super-resolution, and improving prompt alignment in Stable Diffusion.
weaknesses: 1) The use of the replay buffer requires further explanation or justification, since it is critical to the efficiency of the algorithm:
    - Why is it reasonable to use a replay buffer? Shouldn’t trajectories be drawn on-policy (according to current $h$ ) to have the correct importance weights computed according to eq. 14 (I am assuming the accepted samples are placed in the buffer, as described in Algorithm 1)?
        - Notably in other works (eg. in Sendera et. al., 2024), the use of a replay buffer is justified by the training loss itself not needing on-policy samples, but such a justification doesn’t seem to hold for the proposed algorithm
        - Is there any degradation in performance for longer buffer lengths, it would be helpful to experimentally demonstrate the impact of the replay buffer.
        

2) For experiments - some more comprehensive evaluations are needed to demonstrate the performance of the method:
    - For the toy GMM task - from the figure or the reported metrics in Table 4 it is unclear whether the proposed method is less biased compared to alternatives (in particular classifier guidance). A metric based on reverse KL, or a comparison to a “ground truth” baseline (such as a diffusion model trained on ground truth samples from $p_{\mathrm{tilted}}$) would better illustrate the performance
    - For the linear inverse problem - the method should be compared to baselines such as VarGrad, or Adjoint Matching, to demonstrate how the method compares to alternatives
    - For prompt alignment fine-tuning - the images  produced by Importance FT seem overly bright and saturated, or with some artefacts, compared to the other fine-tuning methods (in Fig 4, as well as 8, 9, 10). Is there any degradation in performance due to the fine-tuning?

3) The advantages for the proposed method should be more clearly outlined, over the alternatives, in particular since the metrics in Table 3 doesn’t show a clear improvement for Importance FT over the alternatives. In my opinion this would be remedied by more empirical support for two claims made in the paper:
    - The conclusion implies the training method is more stable in terms of requiring fewer tricks (eg. loss clipping). Does this hold more broadly?  Eg. does the reward improve more smoothly along training iterations, or do the gradients have smaller variance compared to other baselines? Does the reward improve at a faster rate (either in terms of wall-clock time or the number of gradient updates made)? If so, the method would be more clearly useful.
    - The method has the same memory complexity as Adjoint matching (or other online RL methods using the adjoint method) (O(B)), so the claim of importance FT offering “a more efficient and scalable training pipeline” should be supported more clearly. If there is a time-overhead in adjoint matching compared to Importance FT, its significance to the prompt alignment experiment should be demonstrated clearly.

4) Minor grammar errors include:
   - Line 41 space between citation and “.Further, the tilted …”
   - Between Line 71, 73, “stable diffusion” should be capitalized for consistency
   - Line 179 “… sampling from unnormalised probability density function**s**” (s missing)
   - Line 205 “are in particular satisfied **if** the guidance $h$ …”
   - Line 277 (ii) - Should be “ $\mathbb{P}\_{tilted}$  rather than $\mathbb{P}_{tilde}$” - also proportion sign here doesn’t make sense, it should be off by a constant term
   - Line 400-401: “samples are **shown**”
   - Line 415: “samples **do** not always”
   - Line 473-474: missing word “We present **experiments** for class conditional sampling,”
   - Line 481 quotations around “good” are inverted
   - Line 483 “and high reward samples are are”

5) For related works, the paper is missing a reference and discussions to Particle Denoising Diffusion sampling (Phillips et. al., 2024), which uses a similar optimization scheme of sampling from the current iteration of the model, correcting with importance weights (in their case SMC), and then performing supervised training (albeit targeted at the sampling problem rather than reward fine-tuning).

For now I am recommending a weak accept. The primary concern being point 3 above. If more comprehensive evidence is provided which clearly shows the advantages of the proposed method, then I will strongly consider raising my score.

A. Phillips, H.-D. Dau, M. J. Hutchinson, V. De Bortoli, G. Deligiannidis, and A. Doucet, “Particle denoising diffusion sampler,” International Conference on Machine Learning (ICML), 2024.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: See weaknesses
soundness: 2
strengths: Unlike prior methods such as RAFT that make use of importance sampling with the current iteration as the proposal generator, the authors derive a principled approach for fine-tuning. The authors propose a tractable accept/reject probability for use in the diffusion fine-tuning setting.
summary: Given a diffusion model $p_\theta(x_0)$ and a reward model $r(x_0)$, the authors propose a method to sample from the reward tilted distribution. The authors introduce a fine-tuning method that builds on the h-transform learning framework introduced in Denker et al 2024, which learns the gradient of the intermediate reward $\nabla_{x_t} \log E_{p(x_0 | x_t)}[\exp(r(x_0))]$. 

- Unlike typical fine-tuning (or equivalently variational inference) methods that make use of the reverse KL, the authors propose using the supervised, forward KL, objective in eq 7.
- However, as samples from the tilted distribution are not available, the authors use the current iteration of the fine-tuned model to generate samples and then defining an accept/reject step using the Radon-Nikodym derivative (ie the importance weight) between the tilted path measure and the model defined path measure.
weaknesses: 1. The authors should engage with existing literature in a more thorough manner. For instance, iterative cross-entropy (De Boer et al 2005), reinforced self-training (Gulcehre et al 2023), and RAFT (Dong et al 2023), all use importance-sampling and rejection sampling for fine-tuning.
2. As baselines, the authors should include a comparison to sequential Monte Carlo methods, which in the low particle regime have been shown to outperform fine-tuned models. 
3. The authors also make use of reward gradient in parameterizing the h-transform, making it unclear to what extent the proposed fine-tuning method works.

De Boer, Pieter-Tjerk, et al. "A tutorial on the cross-entropy method." Annals of operations research 134.1 (2005): 19-67.

Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling." arXiv preprint arXiv:2308.08998 (2023).

Dong, Hanze, et al. "Raft: Reward ranked finetuning for generative foundation model alignment." arXiv preprint arXiv:2304.06767 (2023).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - Eq 16 states parameterizes the network with reward gradient conditioning. Is my understanding correct that is reward gradient conditioning is used across all experiments except the text-to-image ones? What would happen if you didn't use such conditioning for the remaining experiments? 

- Why is the Stochastic loss presented in Figure 2? Isn't the proposed method trained on $L_{FT}$ loss in eq 15?
soundness: 2
strengths: The paper in general is easy to follow. The proposed idea is simple and straightforward to implement.
summary: This paper presents a new method to finetune diffusion models to achieve high rewards. The main idea is an iterative approach, where the models are continuously trained on importance-weighted samples (or with a similar rejection sampling adjustment). These reweighted samples are supposed to align with the posterior distribution, induced by the prior distribution model and the exponentiated reward. The method is compared to a range of other methods in synthetic 2D, image super-resolution, class-conditional image generation and text-to-image generation tasks.
weaknesses: The main weakness of the current submission is that the efficiency and effectiveness of the proposed method remain unclear. The presentation lacks critical algorithmic details, making it difficult to understand how the method actually work. Moreover, the empirical comparisons to prior work are limited and not convincingly justified. See more details below, 

- Important algorithmic details are not included in the main paper. The choice of hyperparameter $c$ should be important for the efficiency of rejection sampler, however it is not discussed and only reference is mentioned to prior work. Similarly, while the computation of importance weights are built upon prior work, it should be explained in the main paper for the completeness of the method. The importance weights presented in Alg. 13 is not directly tractable.

- Empirical evaluation should include the algorithm efficiency or complexity aspects. For example, how many total training samples each method uses, and what is the sample acceptance rate for the proposed method. If we control for such compute factors, what is the margin of the proposed method to others?

- For each experiment subsection, too few benchmarks are considered. For example, inverse problem in sec 4.2 only includes one example, and text-to-image reward finetuning only includes 3 prompts. In addition,  I believe the paper can benefit from adding more baseline methods for comparison ,for example, for the image super-resolution task. 

- Empirical results did not include standard errors. 

- Finally, as the authors pointed out in the limitation section, such importance sampling or rejection sampling methods rely on the assumption that the proposal (the prior model) has sufficient support on the high reward region. This point is more about the assumption of the method rather than a critique.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 4
questions: How did you choose the set of prompts you use for the reward-based fine-tuning experiments?

Did you consider applying this to non-differentiable reward functions? Since you don't need to differentiate the reward function it may be useful to evaluating in this setting.
soundness: 2
strengths: The paper is well-written and presents a novel method for estimating the h-transform to improve conditional sampling from diffusion models, which draws on methods such as importance based fine-tuning as well as iterative fine-tuning procedures. Notably, this method does not require differentiating the score function of the base model and also is more memory efficient than other methods. The authors additionally prove that their iterative approach is a descent algorithm that minimizes the general loss function for the h-transform and that the distribution of samples at each step is closer to that of the tilted distribution.
summary: Oftentimes we want to use a pretrained diffusion model as a prior for conditional sampling for some downstream task. The posterior distribution in this case can be thought of as a tilted version of the pretrained diffusion model's distribution. One method of sampling from such tilted distributions involves using the Doob's h-transform. In this paper, the authors propose an iterative algorithm for estimating the h-transform. Specifically, they propose refining the estimate of the h-transform by sampling from the diffusion model using the current estimate of the h-transform, filtering samples based on how well they fit the tilted distribution, and using those samples for supervised fine-tuning. This procedure is applied iteratively many times, yielding a sequence of improved estimates for the h-transform. They present experiments applying this approach to class-conditional sampling, super-resolution, and reward-based finetuning.
weaknesses: The experiments in the paper could be improved. For class-conditional sampling, I would like to see additional results in a more difficult setting with more classes such as ImageNet. For super-resolution, I would like to see a few different things. First, I think there should be experiments with more than 2x super-resolution, such as 4x and 8x super-resolution. Additionally, I would like to see experiments on additional datasets, such as ImageNet and CelebA-HQ, which are more standard. For super-resolution, I also don't think you compare with other methods, such as DDRM [1] and SR3 [2]. For reward-based fine-tuning, you should use a larger set of prompts for evaluation. Finally, you should compare with additional baselines. SMC-based approaches also allow for sampling from the tilted distribution and you should compare with recent SMC-based methods for this problem, such as FK Steering [3]. I would also like to see a simple Best-of-N baseline for each setting.

[1] Kawar, Bahjat, Michael Elad, Stefano Ermon, and Jiaming Song. "Denoising diffusion restoration models." NeurIPS 2022. [https://proceedings.neurips.cc/paper_files/paper/2022/hash/95504595b6169131b6ed6cd72eb05616-Abstract-Conference.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/95504595b6169131b6ed6cd72eb05616-Abstract-Conference.html).

[2] Saharia, Chitwan, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. "Image super-resolution via iterative refinement." IEEE transactions on pattern analysis and machine intelligence 2022. [https://ieeexplore.ieee.org/document/9887996/](https://ieeexplore.ieee.org/document/9887996/).

[3] Singhal, Raghav, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. "A General Framework for Inference-time Scaling and Steering of Diffusion Models." ICML 2025. [https://arxiv.org/abs/2501.06848](https://arxiv.org/abs/2501.06848).


Decision: Reject