Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

## Abstract

Abstract Diffusion models excel at capturing the natural design spaces of images, molecules, and biological sequences of DNA, RNA, and proteins. However, for many applications from biological research to biotherapeutic discovery, rather than merely generating designs that are natural, we aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require “differentiable” proxy models ( e.g. , classifier guidance) or computationally-expensive fine-tuning of diffusion models ( e.g. , classifier-free guidance, RL-based fine-tuning). Here, we propose a new method, S oft V alue-based D ecoding in D iffusion models ( SVDD ), to address these challenges. SVDD is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, SVDD avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of SVDD across several domains, including image generation, molecule generation (optimization of docking scores, QED, SA), and DNA/RNA generation (optimization of activity levels).
The code is available at https://github.com/masa-ue/SVDD .

## 1 Introduction

Diffusion models have gained popularity as powerful generative models. Their applications extend beyond image generation to include natural language generation (Sahoo et al., ; Shi et al., ; Lou et al., ), molecule generation (Jo et al., ; Vignac et al., ), and biological (DNA, RNA, protein) sequence generation (Avdeyev et al., ; Stark et al., ). In each of these domains, diffusion models have been shown to be very effective at capturing complex natural distributions. However, in practice, we might not only want to generate *realistic* samples, but to produce samples that optimize specific downstream reward functions while preserving naturalness by leveraging pre-trained models. For example, in computer vision, we might aim to generate natural images with high aesthetic and alignment scores (Black et al., ; Fan et al., ). In drug discovery, we may seek to generate valid molecules with high QED/SA/docking scores (Lee et al., ; Jin et al., ) or RNAs (such as mRNA vaccines (Cheng et al., )) with high translational efficiency and stability (Castillo-Hair and Seelig, ; Asrani et al., ), and regulatory DNAs that drives high cell-specificity of expression (Gosai et al., ; Taskiran et al., ; Lal et al., ).

The optimization of downstream reward functions using pre-trained diffusion models has been approached in various ways. In our work, we focus on non-fine-tuning-based methods because fine-tuning generative models (e.g., when using classifier-free guidance (Ho et al., ) or RL-based fine-tuning (Black et al., ; Fan et al., ; Uehara et al., ; Clark et al., ; Prabhudesai et al., )) often becomes computationally intensive, especially as pre-trained generative models grow larger in the era of “foundation models”. Although classifier guidance and its variants (e.g., Dhariwal and Nichol (); Song et al. (); Chung et al. (); Bansal et al. (); Ho et al. ()) have shown some success as non-fine-tuning methods in these settings, they face significant challenges. First, as they would require constructing *differentiable* proxy models, they cannot directly incorporate useful *non-differentiable* features (e.g., molecular/protein descriptors (van Westen et al., ; Ghiringhelli et al., ; Gainza et al., )) or non-differentiable reward feedback (e.g., physics-based simulations such as Vina and Rosetta (Trott and Olson, ; Alhossary et al., ; Alford et al., )), which are particularly important in molecule design to optimize docking scores, stability, etc. This limitation also hinders the principled application of current classifier guidance methods to recently-developed discrete diffusion models (Austin et al., ; Campbell et al., ; Lou et al., ) (i.e., without transforming the discrete space into the Euclidean space).

Figure: Figure 1: Summary of SVDD. $v$ denotes value functions that predict reward $r(x_{0})$ (at time 0 0) from states at time $t-1$. SVDD involves two steps: (1) generating multiple noisy states from pre-trained models, and (2) selecting the state with the highest value according to the value function.
Refer to caption: x1.png

To tackle these challenges, we propose a novel method, Soft Value-based Decoding in Diffusion models (SVDD), for optimizing downstream reward functions in diffusion models (Figure 1). Inspired by recent literature on RL-based fine-tuning (Uehara et al., ), we first introduce soft value functions that serve as look-ahead functions, indicating how intermediate noisy samples lead to high rewards in the *future* of the diffusion denoising process.
After learning (or approximating) these value functions, we present a new inference-time technique, SVDD, which obtains multiple noisy states from the policy (i.e., denoising map) of pre-trained diffusion models and selects the sample with the highest value function at each time step. Specifically, we introduce two algorithms (SVDD-MC and SVDD-PM) depending on how we estimate value functions. Notably, the SVDD-PM approach does not require any additional learning as long as we have access to the reward feedback by utilizing the characteristics of diffusion models (i.e., the forward process in diffusion models to directly map $t$ to 0 0 in terms of expectation in Figure 1).

**Table 1: A comparison of SVDD to prior approaches. “Non-differentiable” refers to the method’s ability to operate without requiring differentiable proxy models. “No Training” means that no additional training of the diffusion model is required as long as we have access to the reward feedback. We compare to SMC-based methods in Section 6.**
|  | No fine-tuning | Non-differentiable | No Training |
| --- | --- | --- | --- |
| Classifier guidance | $\checkmark$ |  |  |
| DPS (Chung et al., ) | $\checkmark$ |  | $\checkmark$ |
| Classifier-free |  | $\checkmark$ |  |
| RL fine-tuning |  | $\checkmark$ |  |
| SVDD-MC (here) | $\checkmark$ | $\checkmark$ |  |
| SVDD-PM (here) | $\checkmark$ | $\checkmark$ | $\checkmark$ |

Our novel technique for optimizing downstream reward functions in pre-trained diffusion models makes two contributions (Table 1). First, it eliminates the need to construct differentiable proxy models. This allows for the use of non-differentiable reward feedback, which is common in many scientific fields, and makes our method applicable to recent discrete diffusion (Shi et al., ; Sahoo et al., ) models without any modification. Second, it avoids the need to fine-tune the generative model itself.
This addresses the high computational cost associated with fine-tuning diffusion models. We demonstrate the effectiveness of our methods across diverse domains, including image generation, molecule generation (optimization of docking scores, QED, and SA), and DNA/RNA generation (optimization of activity levels).

## 2 Related Works

To summarize related work, we first outline methods relevant to our goal, categorizing them based on whether they involve fine-tuning. We then discuss related directions, such as discrete diffusion models, where our method excels. We defer other relevant works (e.g., decoding in LLMs) to Section A due to space constraints.

##### Non-fine-tuning methods.

We discuss two main methods for optimizing rewards in diffusion models without fine-tuning. We further cover closely relevant methods based on sequential Monte Carlo (SMC) in Section 6 and Appendix B, after presenting our method.

- •
Classifier guidance (Dhariwal and Nichol, ; Song et al., ): It has been widely used to condition pre-trained diffusion models without fine-tuning. Although these methods do not originally focus on optimizing reward functions, they can be applied for this purpose (Uehara et al., , Section 6.2). In this approach, an additional derivative of a certain value function is incorporated into the drift term (mean) of pre-trained diffusion models during inference. Subsequent variants (e.g., Chung et al. (); Ho et al. (); Bansal et al. (); Guo et al. (); Wang et al. (); Yu et al. ()) have been proposed to simplify the learning of value functions. However, these methods require constructing *differentiable* models, which limits their applicability to non-differentiable features/reward feedbacks commonly encountered in scientific domains as mentioned in Section 1. Additionally, this approach cannot be directly extended to discrete diffusion models (e.g., (Lou et al., ; Shi et al., ; Sahoo et al., )) in a principle way.
Our approach aims to address these challenges.
Note a notable exception of classifier guidance tailored to discrete diffusion models has been recently proposed by Nisonoff et al. (). However, SVDD can be applied to both continuous and discrete diffusion models in a unified manner. Furthermore, their practical method requires the differentiability of proxy models, unlike SVDD. We compare its performance with our method in Section 7. We provide further details in Section C.
- •
Best-of-N: The naive way is to generate multiple samples and select the top samples based on the reward functions, known as best-of-N in the literature on (autoregressive) LLMs (Stiennon et al., ; Nakano et al., ; Touvron et al., ; Beirami et al., ; Gao et al., ). This approach is significantly less efficient than ours, as SVDD uses soft-value functions that predict how intermediate noisy samples lead to high rewards in the future. We validate this experimentally in Section 7.

##### Fine-tuning of diffusion models.

Several methods exist for fine-tuning generative models to optimize downstream reward functions, such as classifier-free guidance (Ho and Salimans, ) and RL-based fine-tuning (Fan et al., ; Black et al., ) or its variants (Dong et al., ; Wallace et al., ). However, these approaches often come with caveats, including high computational costs and the risk of easily forgetting pre-trained models. In our work, we propose an inference-time technique that eliminates the need for fine-tuning generative models.

##### Discrete diffusion models.

Based on seminal works Austin et al. (); Campbell et al. (), recent work on masked diffusion models (Lou et al., ; Shi et al., ; Sahoo et al., ) has demonstrated their strong performance in natural language generation. Additionally, they have been applied to biological sequence generation (e.g., DNA, protein sequences in Campbell et al. (); Sarkar et al. ()). In these cases, the use of diffusion models over autoregressive models is particularly apt, given that many biological sequences ultimately adopt complex three-dimensional structures. We also note that ESM3 (Hayes et al., ), a widely recognized foundation model in protein sequence generation, bears similarities to masked diffusion models. Despite its significance, it cannot be integrated with standard classifier guidance, because adding a continuous gradient to a discrete objective is not inherently valid. Unlike standard classifier guidance, SVDD can be seamlessly applied to discrete diffusion models.

## 3 Preliminaries and Goal

We describe the standard method for training diffusion models and outline the objective of our work: optimizing downstream reward functions given pre-trained diffusion models.

### 3.1 Diffusion Models

In diffusion models (Ho et al., ; Song et al., ), our goal is to learn a sampler $p(x)\in\Delta(\mathcal{X})$ given data consisting of $x\in\mathcal{X}$. The training process for a standard diffusion model is summarized as follows. First, we introduce a (fixed) forward process $q_{t}:\mathcal{X}\to\Delta(\mathcal{X})$. Next, we aim to learn a backward process: $\{p_{t}\}$ where each $p_{t}$ is $\mathcal{X}\to\Delta(\mathcal{X})$ so that the distributions induced by the forward process and backward process match marginally. For this purpose, by parametrizing the backward processes with $\theta\in\mathbb{R}^{d}$, we typically use the following loss function:

$$ $\displaystyle\mathbb{E}_{x_{1},\cdots,x_{T}\sim q(\cdot|x_{0})}\left[-\log p_{ 0}(x_{0}|x_{1})+\sum_{t=1}^{T-1}\mathrm{KL}(q_{t}(\cdot\mid x_{t-1},x_{0})\|p_ {t}(\cdot\mid x_{t+1};\theta))+\mathrm{KL}(q_{T}(\cdot)\|p_{T}(\cdot))\right],$ $$

which is derived from the variational lower bound of the negative log-likelihood (i.e., ELBO).

Here are two examples of concrete parameterizations. Let $\alpha_{t}\in\mathbb{R}$ be a noise schedule.

###### Example 1 (Continuous space) .

When $\mathcal{X}$ is Euclidean space, we typically use the Gaussian distribution $q_{t}(\cdot\mid x)=\mathcal{N}(\sqrt{\alpha_{t}}x,(1-\alpha_{t}))$. Then, the backward process is parameterized as

$$ $\displaystyle\mathcal{N}\left(\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})x_{ t}+\sqrt{\bar{\alpha}_{t-1}}(1-\alpha_{t})\hat{x}_{0}(x_{t};\theta)}{1-\bar{ \alpha}_{t}},\frac{(1-\alpha_{t})(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}} \right),$ $$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$. Here, $\hat{x}_{0}(x_{t};\theta)$ is a neural network that predicts $x_{0}$ from $x_{t}$ ($\mathbb{E}_{q}[x_{0}|x_{t}]$).

###### Example 2 (Discrete space in Sahoo et al. ( 2024 ); Shi et al. ( 2024 ) ) .

Let $\mathcal{X}$ be a space of one-hot column vectors $\{x\in\{0,1\}^{K}:\sum_{i=1}^{K}x_{i}=1\}$, and $\mathrm{Cat}(\pi)$ be the categorical distribution over $K$ classes with probabilities given by $\pi\in\Delta^{K}$ where $\Delta^{K}$ denotes the K-simplex. A typical choice is
$q_{t}(\cdot\mid x)=\mathrm{Cat}(\alpha_{t}x+(1-\alpha_{t})\mathbf{m})$ where $\mathbf{m}=[0,\cdots,0,\mathrm{Mask}]$. Then, the backward process is parameterized as

$$ $\displaystyle\begin{cases}\mathrm{Cat}(x_{t}),\quad\mathrm{if}\,x_{t}\neq \mathbf{m},\\ \mathrm{Cat}\left(\frac{(1-\bar{\alpha}_{t-1})\mathbf{m}+(\bar{\alpha}_{t-1}- \bar{\alpha}_{t})\hat{x}_{0}(x_{t};\theta)}{1-\bar{\alpha}_{t}}\right),\quad \mathrm{if}\,x_{t}=\mathbf{m},\end{cases}$ $$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$. Here, $\hat{x}_{0}(x_{t};\theta)$ is a neural network that predicts $x_{0}$ from $x_{t}$. Note that when considering a sequence of $L$ tokens ($x^{1:L})$, we use the direct product: $p_{t}(x^{1:L}_{t}|x^{1:L}_{t+1})=\prod_{l=1}^{L}p_{t}(x^{l}_{t}|x^{1:L}_{t+1})$.

After learning the backward process, we can sample from a distribution that emulates training data distribution (i.e., $p(x)$) by sequentially sampling $\{p_{t}\}_{t=T}^{0}$ from $t=T$ to $t=0$.

###### Example 1 (Continuous space) .

###### Example 2 (Discrete space in Sahoo et al. ( 2024 ); Shi et al. ( 2024 ) ) .

##### Notation.

The notation $\delta_{a}$ denotes a Dirac delta distribution centered at $a$. The notation $\propto$ indicates that the distribution is equal up to a normalizing constant. With slight abuse of notation, we often denote $p_{T}(\cdot|\cdot,\cdot)$ by $p_{T}(\cdot)$.

### 3.2 Objective: Generating Samples with High Rewards While Preserving Naturalness

We consider a scenario where we have a pre-trained diffusion model, which is trained using the loss function explained in Section 3.1. These pre-trained models are typically designed to excel at characterizing the natural design space (e.g., image space, biological sequence space, or chemical space) by emulating the extensive training dataset. Our work focuses on obtaining samples that also optimize downstream reward functions $r:\mathcal{X}\to\mathbb{R}$ (e.g., Quantitative Estimate of Druglikeness (QED) and Synthetic Accessibility (SA) in molecule generation), while maintaining the naturalness by leveraging pre-trained diffusion models. We formalize this goal as follows.

Given a pre-trained model $\{p^{\mathrm{pre}}_{t}\}_{t=T}^{0}$, we denote the induced distribution by $p^{\mathrm{pre}}\in\Delta(\mathcal{X})$ (i.e., $p^{\mathrm{pre}}(x_{0})=\int\{\prod_{t=T+1}^{1}p^{\mathrm{pre}}_{t-1}(x_{t-1}|
x_{t})\}dx_{1:T}$). We aim to sample from the following distribution:

$$ $\displaystyle p^{(\alpha)}(x)$ $\displaystyle:=\mathop{\mathrm{argmax}}_{p\in[\Delta(\mathcal{X})]}\underbrace {\mathbb{E}_{x\sim p(\cdot)}[r(x)]}_{\textbf{term (a)}}-\alpha\underbrace{ \mathrm{KL}(p(\cdot)\|p^{\mathrm{pre}}(\cdot))}_{\textbf{term(b)}}\propto\exp( r(x)/\alpha)p^{\mathrm{pre}}(x).$ $$

Here, term (a) is introduced to optimize the reward function, while term (b) is used to maintain the naturalness of the generated samples.

##### Existing methods.

Several existing approaches target this goal (or its variant), as discussed in Section 2, including classifier guidance, fine-tuning (RL-based or classifier-free), and Best-of-N. In our work, we focus on non-fine-tuning-based methods; specifically, we aim to address the limitations of these methods: the requirement for differentiable proxy models in classifier guidance and the inefficiency of Best-of-N.

Finally, we note that all results discussed in this paper can be easily extended to cases where the pre-trained model is a conditional diffusion model. For example, in our image experiments (Section 7), the pre-trained model is a conditional diffusion model conditioned on text (e.g., Stable Diffusion).

## 4 Soft Value-Based Decoding in Diffusion Models

First, we present the motivation behind developing our new algorithm. We then introduce SVDD, which satisfies our desired properties, i.e., the lack of need for fine-tuning or constructing differentiable models.

### 4.1 Key Observation

We introduce several key concepts. First, we define the *soft value function*:

$$ $\displaystyle\textstyle t\in[T+1,\cdots,1];v_{t-1}(\cdot):=\alpha\log\mathbb{E }_{x_{0}\sim p^{\mathrm{pre}}(x_{0}|x_{t-1})}\left[\exp\left(\frac{r(x_{0})}{ \alpha}\right)|x_{t-1}=\cdot\right],$ $$

where $\mathbb{E}_{\{p^{\mathrm{pre}}\}}[\cdot]$ is induced by $\{p^{\mathrm{pre}}_{t}(\cdot|x_{t+1})\}_{t=T}^{0}$. This value function represents the expected future reward at $t=0$ from the intermediate noisy state at $t-1$.

Next, we define the following *soft optimal policy* (denoising process) $p^{\star,\alpha}_{t-1}:\mathcal{X}\to\Delta(\mathcal{X})$ weighted by value functions $v_{t-1}:\mathcal{X}\to\mathbb{R}$:

$$ $\displaystyle\textstyle p^{\star,\alpha}_{t-1}(\cdot|x_{t})$ $\displaystyle=\frac{p^{\mathrm{pre}}_{t-1}(\cdot|x_{t})\exp(v_{t-1}(\cdot)/ \alpha)}{\int p^{\mathrm{pre}}_{t-1}(x|x_{t})\exp(v_{t-1}(x)/\alpha)dx}.$ $$

Here, $v_{t}$ are soft value functions and $p^{\star,\alpha}_{t}$ are soft optimal policies, because they literally correspond, respectively, to soft value functions and soft optimal policies, where we embed diffusion models into entropy-regularized MDPs (Geist et al., ), as demonstrated in Uehara et al. ().

With this preparation in mind, we utilize the following key observation:

###### Theorem 1 (From Theorem 1 in Uehara et al. ( 2024 ) ) .

The distribution induced by $\{p^{\star,\alpha}_{t}(\cdot|x_{t+1})\}_{t=T}^{0}$ is the target distribution $p^{(\alpha)}(x)$, i.e.,

$$ $\displaystyle\textstyle p^{(\alpha)}(x_{0})=\int\left\{\prod_{t=T+1}^{1}p^{ \star,\alpha}_{t-1}(x_{t-1}|x_{t})\right\}dx_{1:T}.$ $$

While Uehara et al. () presents this theorem, they use it primarily to interpret RL-based fine-tuning methods in Fan et al. (); Black et al. (). In contrast, our work explores how to convert this into a new fine-tuning-free optimization algorithm.

###### Theorem 1 (From Theorem 1 in Uehara et al. ( 2024 ) ) .

##### Our motivation for a new algorithm.

Theorem 1 states that if we can hypothetically sample from $\{p^{\star,\alpha}_{t}\}_{t=T}^{0}$, we can sample from the target distribution $p^{(\alpha)}$. However, there are two challenges in sampling from each $p^{\star,\alpha}_{t-1}$: (1) the soft-value function $v_{t-1}$ in $p^{\star,\alpha}_{t-1}$ is unknown, and (2) it is unnormalized (i.e., calculating the normalizing constant is hard).

We address the first challenge in Section 4.3. Assuming the first challenge is resolved, we consider how to tackle the second challenge. A natural approach is to use importance sampling (IS):

$$ $\displaystyle p^{\star,\alpha}_{t-1}(\cdot|x_{t},c)\approx\sum_{m=1}^{M}\frac{ w^{\langle m\rangle}_{t-1}}{\sum_{j=1}^{M}w^{\langle j\rangle}_{t-1}}\delta_{x ^{\langle m\rangle}_{t-1}},\quad\{x^{\langle m\rangle}_{t-1}\}_{m=1}^{M}\sim p ^{\mathrm{pre}}_{t-1}(\cdot\mid x_{t}),$ $$

where $w^{\langle m\rangle}_{t-1}:=\exp(v_{t}(x^{\langle m\rangle}_{t-1})/\alpha)$. Thus, we can approximately sample from $p^{\star,\alpha}_{t-1}(\cdot|x_{t})$ by obtaining multiple ($M$) samples from pre-trained diffusion models and selecting the sample based on an index, which is determined by sampling from the categorical distribution with mean $\{w^{\langle m\rangle}_{t-1}/\sum_{j}w^{\langle j\rangle}_{t-1}\}_{m=1}^{
\langle M\rangle}$.

Note that Best-of-N, which generates multiple samples and selects the highest reward sample, is technically considered IS, where the proposal distribution is the entire $p^{\mathrm{pre}}(x_{0})=\int\prod_{t}\{p^{\mathrm{pre}}_{t}(x_{t-1}\mid x_{t})
\}dx_{1:T}$. However, the use of importance sampling in our algorithm differs significantly, as we apply it at each time step to approximate each soft-optimal policy.

### 4.2 Inference-Time Algorithm

Figure: Algorithm 1 SVDD (Soft Value-Based Decoding in Diffusion Models)

Now, by leveraging the observation in Algorithm 1, we introduce our algorithm. Our algorithm is an iterative sampling method that integrates soft value functions into the standard inference procedure of pre-trained diffusion models. Each step is designed to approximately sample from a value-weighted policy $\{p^{\star,\alpha}_{t}\}_{t=T}^{0}$.

We note several key points.

- •
When $\alpha=0$, Line 4 corresponds to $\zeta_{t-1}=\mathop{\mathrm{argmax}}_{m\in[1,\cdots,M]}\hat{v}_{t-1}(x^{
\langle m\rangle}_{t-1}).$
In practice, we often recommend this choice. This is the default choice in Section 7.
- •
A typical choice we recommend for $M$ is from $5$ to $20$. The performance with varying $M$ values will be discussed in Section 7.
- •
Line 3 can be computed in parallel at the cost of additional memory (scaled by $M$). If Line 3 is not computed in parallel, the computational time in SVDD would be approximately $M$ times that of the standard inference procedure. We will check it in Section 7.
- •
In special cases where the normalizing constant can be calculated relatively easily (e.g., in discrete diffusion with small $K,L$), we can directly sample from $\{p^{\star,\alpha}_{t}\}_{t=T}^{0}$.
- •
A proposal distribution different from $p^{\mathrm{pre}}_{t-1}$ in line 3 can be applied (see Section D). For instance, classifier guidance or its variants may be used to obtain better proposal distributions than those from the pure pre-trained model.

The remaining question is how to obtain the soft value function, which we address in the next section.

### 4.3 Learning Soft Value Functions

Next, we describe how to obtain soft value functions $v_{t}(x)$ in practice. We propose two main approaches: a Monte Carlo regression approach and a posterior mean approximation approach.

##### Monte Carlo regression.

Here, we use the following approximation $v^{\prime}_{t}$ as $v_{t}$ where

$$ $\displaystyle v^{\prime}_{t}(\cdot):=\mathbb{E}_{x_{0}\sim p^{\mathrm{pre}}(x_ {0}|x_{t})}[r(x_{0})|(x_{t})=\cdot].$ $$

This is based on

$$ $\displaystyle v_{t}(x_{t})=\alpha\log\mathbb{E}_{x_{0}\sim p^{\mathrm{pre}}( \cdot|x_{t})}[\exp(r(x_{0})/{\alpha})|x_{t}]\approx\log(\exp(\mathbb{E}_{x_{0} \sim p^{\mathrm{pre}}(x_{t})}[r(x_{0})|x_{t}]))=v^{\prime}_{t}(x_{t}).$ (1) $$

By regressing $r(x_{0})$ onto $x_{t}$, we can learn $v^{\prime}_{t}$ as in Algorithm 2. Combining this with Algorithm 1, we refer to the entire optimization approach as SVDD-MC.

Figure: Algorithm 2 Value Function Estimation Using Monte Carlo Regression

Note that technically, without the approximation introduced in (1), we can estimate $v_{t}$ by regressing $\exp(r(x_{0})/\alpha)$ onto $x_{t}$ based on the original definition. This approach may work in many cases. However, when $\alpha$ is very small, the scaling of $\exp(r(\cdot)/\alpha)$ tends to be excessively large. Due to this concern, we generally recommend using Algorithm 2.

###### Remark 1 (Another way of learning value functions) .

Technically, another method for learning value functions is available such as soft-Q-learning (Section E), by leveraging soft-Bellman equations in diffusion models Uehara et al. (, Section 3)
However, since we find Monte Carlo approaches to be more stable, we recommend them over soft-Q-learning.

###### Remark 1 (Another way of learning value functions) .

##### Posterior mean approximation.

Here, recalling we use $\hat{x}_{0}(x_{t})$ (approximation of $\mathbb{E}_{x_{0}\sim p^{\mathrm{pre}}(x_{t})}[x_{0}|x_{t}]$) when training pre-trained diffusion models in Section 3.1, we perform the following approximation:

$$ $\displaystyle v_{t}(x):=\alpha\log\mathbb{E}_{x_{0}\sim p^{\mathrm{pre}}(x_{0} |x_{t})}[\exp(r(x_{0})/\alpha)|x_{t}]\approx\alpha\log(\exp(r(\hat{x}_{0}(x_{t }))/\alpha)=r(\hat{x}_{0}(x_{t})).$ $$

Then, we can use $r(\hat{x}_{0}(x_{t}))$ as the estimated value function.

The advantage of this approach is that no additional training is required as long as we have $r$. When combined with Algorithm 1, we refer to the entire approach as SVDD-PM.

Figure: Algorithm 3 Value Function Estimation using Posterior Mean Approximation

###### Remark 2 (Relation with DPS) .

In the context of classifier guidance, similar approximations have been employed (e.g., DPS in Chung et al. ()). However, the final inference-time algorithms differ significantly, as these methods compute gradients at the end.

###### Remark 2 (Relation with DPS) .

## 5 Advantages, Extensions, Limitations of SVDD

We discuss the advantages, extensions, and limitations of SVDD.

### 5.1 Advantages

##### No fine-tuning (or no training in SVDD -PM).

Unlike classifier-free guidance or RL-based fine-tuning, SVDD  does not require any fine-tuning of the generative models. In particular, when using SVDD-PM, no additional training is needed as long as we have $r$.

##### No need for constructing differentiable models.

Unlike classifier guidance, SVDD  does not require differentiable proxy models, as there is no need for derivative computations. For example, if $r$ is non-differentiable feedback (e.g., physically-based simulations for docking scores in molecule generation), our method SVDD-PM can directly utilize such feedback without constructing differentiable proxy models. In cases where non-differentiable computational feedback is costly to obtain, the usage of proxy reward models may still be preferred, but they do not need to be differentiable; thus, non-differentiable features or non-differentiable models based on scientific knowledge (e.g., molecule fingerprints, GNNs) can be leveraged. Similarly, when using SVDD-MC, while a value function model is required, it does not need to be differentiable, unlike classifier guidance. Additionally, compared to approaches that involve derivatives (like classifier guidance or DPS), SVDD can be directly applied to discrete diffusion models mentioned in Example 2.

### 5.2 Extensions

##### Using a likelihood/classifier as a reward.

While we primarily consider scenarios where reward models are regression models, by adopting a similar strategy to that in Zhao et al. (), they can be readily replaced with classifiers or likelihood functions in the context of solving inverse problems or conditioning (Chung et al., ; Bansal et al., ).

##### Fine-tuning by distilling SVDD .

The inference-time cost may become slow as $M$ increases in SVDD,. This issue can be mitigated by policy distillation, that is further fine-tuning diffusion models to align them closely with policies from SVDD (Salimans and Ho, ; Kim et al., ). We leave this aspect for future work.

### 5.3 Potential Limitations

##### Memory and computational complexity in inference time.

Our approach requires more computational resources (if not parallelized) or memory (if parallelized), approximately $M$ times more than standard inference methods, as noted in Section 4.2. Taking this aspect into account, we compare SVDD, with baselines such as best-of-N in our experimental section (Section 7). For gradient-based approaches like classifier guidance and DPS, while a direct comparison with SVDD is challenging, it is important to note that these methods also incur additional computational and memory complexity due to the backward pass, which SVDD avoids. Lastly, it is important to note that this additional inference-time burden can be alleviated through distillation, as discussed in Section 5.2.

##### Proximity to pre-trained models.

If significant changes to the pre-trained models are desired, we acknowledge that RL-based fine-tuning (Black et al., ; Fan et al., ) could be more effective than SVDD for this purpose in certain scenarios, such as image examples. However, this proximity to pre-trained models could also be advantageous in the sense that it is robust against reward optimization, which conventional fine-tuning methods often suffer from by exploiting these out-of-distribution regions (Uehara et al., ).
Lastly, in cases where reward backpropagation (Prabhudesai et al., ; Clark et al., ; Uehara et al., ) is not applicable, particularly in scientific domains for RL-based fine-tuning, we may need to rely on PPO. However, PPO is often mode-seeking and unstable, highlighting the challenges of RL-based fine-tuning in certain scenarios.

## 6 Comparison between SVDD and SMC-Based Methods

SMC-based methods for diffusion models are closely related to SVDD. These approaches (Wu et al., ; Trippe et al., ; Dou and Song, ; Phillips et al., ; Cardoso et al., ) use SMC (Del Moral and Doucet, ) for sampling from diffusion models. While they are originally designed for conditioning (by setting rewards as classifiers), they can also be applied to reward maximization. Notably, similar to our work, these methods do not require differentiable models.

However, these SMC methods are not tailored to reward maximization. Most importantly, they involve resampling across the “entire” batch, which complicates parallelization. Additionally, when batch sizes are small, as is often the case with recent large diffusion models, performance may be suboptimal, since the SMC theoretical guarantees hold primarily with large batch sizes. Even with larger batch sizes, using SMC for reward maximization can result in a loss of diversity across the entire batch, since the effective sample size based on weights, which is a standard diversity measure in SMC, does not ensure “real” diversity in the generated samples. In contrast, our method is highly parallelizable, performs well even with small batch sizes (as low as $1$), and maintains diversity with larger batch sizes, as sampling is conducted on a “per-sample basis” (Line 4). We empirically validate this in Section 7. We provide further details and experiments in Appendix B.

## 7 Experiments

We conduct experiments to assess the performance of our algorithm relative to baselines and its sensitivity to various hyperparameters. We start by outlining the experimental setup, including baselines and models, and then present the results.

### 7.1 Settings

##### Methods to compare.

We compare SVDD to several representative methods capable of performing reward maximization during inference, discussed in Section 2.

- •
Pre-trained models: We generate samples using pre-trained models.
- •
Best-of-N: We generate samples from pre-trained models and select the top $1/N$ samples. This selection is made to ensure that the computational time during inference is approximately equivalent to that of SVDD.
- •
DPS (Chung et al., ): It is a widely used training-free version of classifier guidance. For discrete diffusion, we combine it with the state-of-the-art approach (Nisonoff et al., ).
- •
SMC-Based Methods (SMC): Methods discussed in Section 6 and Appendix B, which do not require differentiable models, like SVDD.
- •
SVDD (Ours): We implement SVDD-MC and SVDD-PM. We generally set $M=20$ for images and $M=10$ for other domains, and $\alpha=0$. Recall $M$ is the duplication size in the IS part.

##### Datasets and reward models.

We provide details on the pre-trained diffusion models and downstream reward functions used. For further information, refer to Section F.

- •
Images: We use Stable Diffusion v1.5 as the pre-trained diffusion model ($T=50$). For downstream reward functions, we use compressibility and aesthetic scores (LAION Aesthetic Predictor V2 in Schuhmann ()), as employed by Black et al. (); Fan et al. (). Compressibility is a *non-differentiable reward feedback*.
- •
Molecules: We use GDSS (Jo et al., ), trained on ZINC-250k (Irwin and Shoichet, ), as the pre-trained diffusion model ($T=1000$). For downstream reward functions, we use drug-likeness (QED) and synthetic accessibility (SA) calculated by RDKit, as well as docking score (DS) calculated by QuickVina 2 (Alhossary et al., ), which are all *non-differentiable feedback*. Here, we renormalize SA to $(10-\mathrm{SA})/9$ and docking score to $\max(-\mathrm{DS},0)$, so that a higher value indicates better performance. The docking scores measure binding affinity regarding four target proteins: Parp1, 5ht1b, Jak2, and Braf following Yang et al. (). These tasks are critical for drug discovery.
- •
DNAs (enhancers) and RNAs (5’Untranslated regions (UTRs)): We use the discrete diffusion model (Sahoo et al., ), trained on datasets from Gosai et al. () for enhancers, and from Sample et al. () for 5’UTRs, as our pre-trained diffusion model ($T=128$). For the reward functions, we use an Enformer model (Avsec et al., ) to predict activity of enhancers in the HepG2 cell line, and a ConvGRU model that predicts the mean ribosomal load (MRL) of 5’UTRs measured by polysome profiling, respectively (Sample et al., ).
These tasks are highly relevant for cell and RNA therapies, respectively (Taskiran et al., ; Castillo-Hair and Seelig, ).

### 7.2 Results

**Table 2: Top 10 and 50 quantiles of the generated samples for each algorithm (with 95% confidence intervals). Higher is better. SVDD consistently outperforms the baseline methods.**
| Domain | Quantile | Pre-Train | Best-N | DPS | SMC | SVDD-MC | SVDD-PM |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Image: Compress | 50% | -101.4 $\pm$ 0.22 | -71.2 $\pm$ 0.46 | -60.1 $\pm$ 0.44 | -59.7 $\pm$ 0.4 | -54.3 $\pm$ 0.33 | -51.1 $\pm$ 0.38 |
|  | 10% | -78.6 $\pm$ 0.13 | -57.3 $\pm$ 0.28 | -61.2 $\pm$ 0.28 | -49.9 $\pm$ 0.24 | -40.4 $\pm$ 0.2 | -38.8 $\pm$ 0.23 |
| Image: Aesthetic | 50% | 5.62 $\pm$ 0.003 | 6.11 $\pm$ 0.007 | 5.61 $\pm$ 0.009 | 6.02 $\pm$ 0.004 | 5.70 $\pm$ 0.008 | 6.14 $\pm$ 0.007 |
|  | 10% | 5.98 $\pm$ 0.002 | 6.34 $\pm$ 0.004 | 6.00 $\pm$ 0.005 | 6.28 $\pm$ 0.003 | 6.05 $\pm$ 0.005 | 6.47 $\pm$ 0.004 |
| Molecule: QED | 50% | 0.656 $\pm$ 0.008 | 0.835 $\pm$ 0.009 | 0.679 $\pm$ 0.024 | 0.667 $\pm$ 0.016 | 0.852 $\pm$ 0.011 | 0.848 $\pm$ 0.014 |
|  | 10% | 0.812 $\pm$ 0.005 | 0.902 $\pm$ 0.006 | 0.842 $\pm$ 0.014 | 0.722 $\pm$ 0.009 | 0.925 $\pm$ 0.007 | 0.928 $\pm$ 0.008 |
| Molecule: SA | 50% | 0.652 $\pm$ 0.007 | 0.834 $\pm$ 0.014 | 0.693 $\pm$ 0.022 | 0.786 $\pm$ 0.004 | 0.935 $\pm$ 0.010 | 0.925 $\pm$ 0.016 |
|  | 10% | 0.803 $\pm$ 0.004 | 0.941 $\pm$ 0.008 | 0.844 $\pm$ 0.013 | 0.796 $\pm$ 0.003 | 1.000 $\pm$ 0.006 | 1.000 $\pm$ 0.010 |
| Molecule: Docking parp1 | 50% | 7.15 $\pm$ 0.52 | 10.00 $\pm$ 0.17 | 7.35 $\pm$ 0.43 | 6.90 $\pm$ 0.60 | 12.00 $\pm$ 0.26 | 11.40 $\pm$ 0.22 |
|  | 10% | 8.59 $\pm$ 0.31 | 10.67 $\pm$ 0.10 | 9.31 $\pm$ 0.26 | 9.37 $\pm$ 0.36 | 13.25 $\pm$ 0.16 | 12.41 $\pm$ 0.13 |
| Molecule: Docking 5ht1b | 50% | 7.20 $\pm$ 0.53 | 9.65 $\pm$ 0.17 | 7.30 $\pm$ 0.48 | 6.80 $\pm$ 0.35 | 10.50 $\pm$ 0.46 | 10.50 $\pm$ 0.53 |
|  | 10% | 8.69 $\pm$ 0.32 | 10.28 $\pm$ 0.10 | 9.21 $\pm$ 0.29 | 9.00 $\pm$ 0.21 | 12.87 $\pm$ 0.28 | 12.30 $\pm$ 0.32 |
| Molecule: Docking jak2 | 50% | 7.05 $\pm$ 0.45 | 8.85 $\pm$ 0.17 | 7.30 $\pm$ 0.43 | 6.80 $\pm$ 0.46 | 10.65 $\pm$ 0.35 | 10.30 $\pm$ 0.45 |
|  | 10% | 8.20 $\pm$ 0.27 | 9.59 $\pm$ 0.10 | 8.70 $\pm$ 0.26 | 10.00 $\pm$ 0.28 | 11.80 $\pm$ 0.21 | 11.91 $\pm$ 0.27 |
| Molecule: Docking braf | 50% | 7.20 $\pm$ 0.40 | 9.20 $\pm$ 0.11 | 7.50 $\pm$ 0.23 | 6.90 $\pm$ 0.46 | 10.00 $\pm$ 0.37 | 9.65 $\pm$ 0.33 |
|  | 10% | 8.59 $\pm$ 0.24 | 10.29 $\pm$ 0.07 | 9.20 $\pm$ 0.14 | 8.74 $\pm$ 0.28 | 11.30 $\pm$ 0.22 | 11.40 $\pm$ 0.20 |
| Enhancers | 50% | 0.121 $\pm$ 0.033 | 1.807 $\pm$ 0.214 | 3.782 $\pm$ 0.299 | $4.28\pm 0.02$ | 5.074 $\pm$ 0.096 | 5.353 $\pm$ 0.231 |
|  | 10% | 1.396 $\pm$ 0.020 | 3.449 $\pm$ 0.128 | 4.879 $\pm$ 0.179 | $5.95\pm 0.01$ | 5.639 $\pm$ 0.057 | 6.980 $\pm$ 0.138 |
| 5’UTR | 50% | 0.406 $\pm$ 0.028 | 0.912 $\pm$ 0.023 | 0.426 $\pm$ 0.073 | $0.76\pm 0.02$ | 1.042 $\pm$ 0.008 | 1.214 $\pm$ 0.016 |
|  | 10% | 0.869 $\pm$ 0.017 | 1.064 $\pm$ 0.014 | 0.981 $\pm$ 0.044 | $0.91\pm 0.01$ | 1.117 $\pm$ 0.005 | 1.383 $\pm$ 0.010 |

Figure: (a) Images: compressibility
Refer to caption: extracted/5950550/images/generated_images_comp.png

We compare the baselines with our two proposed methods regarding rewards $r$ (Table 2). To demonstrate the generated samples’ validity (i.e., naturalness), we present several examples in Figure 2. In Section F.3, we also include metrics for the validity of samples.

Overall, SVDD outperformed the baseline methods (Best-of-N, DPS, and SMC), as evidenced by higher quantiles. Furthermore, for both molecules and images, the samples generated by SVDD were valid. This suggests that SVDD can generate high-reward valid samples that Best-of-N, DPS, and SMC often struggle to generate or, in some cases, nearly fail to do.

The relative performance of our SVDD-MC and SVDD-PM appears to be domain-dependent. Generally, SVDD-PM may be more robust since it does not require additional learning (i.e., it directly utilizes reward feedback). The performance of SVDD-MC depends on the success of value function learning discussed in Section F.

Figure: (a) Performance of SVDD as $M$ varies for image generation while optimizing aesthetic score.
Refer to caption: extracted/5950550/images/change_K_asthetic.png

##### Ablation studies in terms of the duplication size M 𝑀 M italic_M .

We assessed the performance of SVDD-PM (when calculating value functions in Line 3 in a non-parallel manner) along with the computational and memory complexity as $M$ varies. First, across all domains, the performance gradually plateaus as $M$ increases (Figure 3(a) and 3(b)). Second, computational complexity increases linearly with $M$, while memory complexity remains nearly constant (Figure 3(c) and 3(d)). This behavior is expected, as previously noted in Section 4.2. The comparison with Best-of-N in Table 2 is made with this consideration in mind.

## 8 Conclusion

We propose a novel inference-time algorithm, SVDD, for optimizing downstream reward functions in pre-trained diffusion models that eliminate the need to construct differentiable proxy models. Future works include applications in other domains, such as protein sequence optimization (Gruver et al., ; Alamdari et al., ; Watson et al., ) and 3D molecule generation (Xu et al., ).

## Acknowledgement

We appreciate the feedback from Christian A. Naesseth. He pointed out the connection between our work and the twisted diffusion sampler, and noted our algorithm is close to an instantiation of nested-IS SMC in our context (Naesseth et al., , ).

## Appendix A Further Related Works

##### Decoding in autoregressive models with rewards.

The decoding strategy, which dictates how sentences are generated from the model, is a critical component of text generation in autoregressive language models (Wu et al., ; Chorowski and Jaitly, ; Leblond et al., ). Recent studies have explored inference-time techniques for optimizing downstream reward functions Dathathri et al. (); Yang and Klein (); Qin et al. (); Mudgal et al. (); Zhao et al. (); Han et al. (). While there are similarities between these works and ours, to the best of our knowledge, no prior work has extended such methodologies to diffusion models. Furthermore, our approach leverages characteristics unique to diffusion models that are not present in autoregressive models such as SVDD-PM.

## Appendix B Difference Between SVDD and “Standard” SMC-Methods

In this section, we compare our algorithm with the SMC-based methods (Wu et al., ; Trippe et al., ; Dou and Song, ; Phillips et al., ; Cardoso et al., ) for guidance. While they originally aim to solve a conditioning problem, which is different from reward maximization, they algorithms can be converted to reward maximization. First, we will explain this converted algorithm. We then elaborate on the differences between SVDD, and them in the context of reward maximization. Notably, our SVDD is an instantiation of nested-IS SMC (Naesseth et al., , Algorithm 5) in the literature on computational statistics, whereas these SMC-based Methods rely on standard sequential Monte Carlo.

### B.1 SMC-Based Methods

Figure: Algorithm 4 Guidance with “Standard” SMC (for reward maximization)

The complete algorithm of TDS in our setting is summarized in Algorithm 4. Since our notation and their notations are slightly different, we first provide a brief overview. It consists of two steps. Since our algorithm is iterative, at time point $t$, consider we have $N$ samples (particles) $\{x^{[i]}_{t}\}_{i=1}^{N}$.

##### IS step ( line 3 ).

We generate a set of samples $\{x^{[i]}_{t-1}\}_{i=1}^{N}$ following a policy from a pre-trained model $p^{\mathrm{pre}}_{t-1}(\cdot|\cdot)$. In other words,

$$ $\displaystyle\forall i\in[1,\cdots,N];x^{[i]}_{t-1}\sim p^{\mathrm{pre}}_{t-1} (\cdot|x^{[i]}_{t}).$ $$

Now, we denote the importance weight for the next particle $x_{t-1}$ given the current particle $x_{t}$ as $w(x_{t-1},x_{t})$, expressed as

$$ $\displaystyle w(x_{t-1},x_{t}):=\frac{\exp(v_{t-1}(x_{t-1})/\alpha)}{\int\exp( v_{t-1}(x_{t-1})/\alpha)p^{\mathrm{pre}}_{t-1}(x_{t-1}|x_{t})dx_{t-1}}=\frac{ \exp(v_{t-1}(x_{t-1})/\alpha)}{\exp(v_{t}(x_{t})/\alpha)},$ $$

and define

$$ $\forall i\in[1,\cdots,N];\quad w^{[i]}_{t-1}:=w(x^{[i]}_{t-1},x^{[i]}_{t}).$ $$

Note here we have used the soft Bellman equation:

$$ $\displaystyle\exp(v_{t}(x_{t})/\alpha)=\int\exp(v_{t-1}(x_{t-1})/\alpha)p^{ \mathrm{pre}}_{t-1}(x_{t-1}|x_{t})dx_{t-1}.$ $$

Hence, by denoting the target marginal distribution at $t-1$, we have the following approximation:

$$ $\displaystyle p^{\mathrm{tar}}_{t-1}\underbrace{\approx}_{\text{IS}}\sum_{i=1} ^{N}\frac{w^{[i]}_{t-1}}{\sum_{j=1}^{N}w^{[j]}_{t-1}}\delta_{x^{[i]}_{t-1}}.$ $$

##### Selection step ( line 5 ).

Finally, we consider a resampling step. The resampling indices are determined by the following:

$$ $\displaystyle\{\zeta^{[i]}_{t-1}\}_{i=1}^{N}\sim\mathrm{Cat}\left(\left\{\frac {w^{[i]}_{t-1}}{\sum_{j=1}^{N}w^{[j]}_{t-1}}\right\}_{i=1}^{N}\right).$ $$

To summarize, we conduct

$$ $\displaystyle p^{\mathrm{tar}}_{t-1}\underbrace{\approx}_{\text{IS}}\sum_{i=1} ^{N}\frac{w^{[i]}_{t-1}}{\sum_{j=1}^{N}w^{[j]}_{t-1}}\delta_{x^{[i]}_{t-1}} \underbrace{\approx}_{\text{Resampling}}\frac{1}{N}\sum_{i=1}^{N}\delta_{x^{ \zeta_{t-1}^{[i]}}_{t-1}}.$ $$

Finally, we give several important remarks.

- •
In SMC, resampling is performed across the *entire* batch. However, in the algorithm, sampling is done within a single batch. Therefore, the algorithms differ significantly. We will discuss the implications in the next section.
- •
All of existing works Wu et al. (); Cardoso et al. (); Phillips et al. (); Dou and Song () actually consider a scenario where the reward $r$ is a classifier. In Algorithm 4, we tailor the algorithm for reward maximization. Vice verisa, as we mentioned in Section 5.2, our SVDD can also operate effectively when $r$ is a classifier.
- •
In Wu et al. (); Cardoso et al. (); Phillips et al. (), the proposal distribution is not limited to the pre-trained model. Likewise, in our SVDD, we can select an arbitrary proposal distribution, as discussed in Section D.
- •
In the context of autoregressive (language) models, Zhao et al. (); Lew et al. () proposed a similar algorithm.

### B.2 Comparison of SVDD with “Standard” SMC-Based Methods (SSM) for Reward Maximization

We now compare our SVDD with “standard” SMC-Based Methods (SSM). Here, we write a batch size of SVDD in $G$. Importantly, we note that our implementation is analogous to nested-IS SMC in the literature in computational statistics; hence, many differences between nested-IS SMC (Naesseth et al., , ) and pure SMC in computational statistics are translated here.

We first reconsider the fundamental assumptions of each algorithm. SVDD’s performance, in terms of rewards, depends on the size of $M$ but is independent of the batch size $G$. In contrast, the performance of SSM depends on the batch size $N$. With this in mind, we compare the advantages of SVDD over SSM from various perspectives.

##### Tailored to optimization in SVDD.

SVDDis considered more suitable for optimization than SMC. This is because, when using SMC for reward maximization, we must set $\alpha$ very low, leading to a lack of diversity. This is expected, as when $\alpha$ approaches 0, the effective sample size reduces to 1. This effect is also evident in our image experiments, as shown in Figure 4. Although SMC performs well in terms of reward functions, there is a significant loss of diversity. Some readers might think this could be mitigated by calculating the effective sample size based on weights (i.e., value functions) and resampling when the effective size decreases; however, this is not the case, as the effective sample size does not directly translate into greater diversity in the generated samples. In contrast, SVDD, maintains much higher diversity.

Figure: (a) Samples from SMC
Refer to caption: extracted/5950550/images/SMC_result.png

##### Ease of parallelization in SVDD.

SVDD is significantly easier to parallelize across multiple nodes. In contrast, SSM requires interaction between nodes for parallelization. This advantage is also well-documented in the context of nested-IS SMC versus standard SMC (Naesseth et al., ).

##### High performance of SVDD under memory constraints.

Now, consider a scenario where the batch size is small, which often occurs due to memory constraints in large pre-trained diffusion models. In this case, while SSM may exhibit suboptimal performance, SVDD-PM can still achieve high performance by choosing a sufficiently large $M$.

##### In SSM, the “ratio” is approximated.

In SVDD, we approximate each
$\exp(v_{t-1}(x_{t-1})/\alpha)$
as a weight. However, in standard SMC, the ratio is approximated as a weight:

$$ $\displaystyle\frac{\exp(v_{t-1}(x_{t-1})/\alpha)}{\exp(v_{t}(x_{t})/\alpha)}.$ $$

The key difference is that in SSM, both the numerator and the denominator are approximated, which could lead to greater error propagation.

## Appendix C Comparison against DG (Nisonoff et al., 2024 ) and DiGress (Vignac et al., 2022 ) in Discrete Diffusion Models

Our method is closely related to guidance methods used in DG (Nisonoff et al., ) and DiGress (Vignac et al., ). However, we emphasize that our approach is more general, as it operates on any domain, including continuous spaces including Riemannian spaces and discrete spaces, in a unified manner. In this sense, a strict comparison is not feasible. With this in mind, we provide a comparison focusing on cases where all domains are discrete.

##### Comparison with Nisonoff et al. ( 2024 ) .

In this continuous framework, following the notation in (Lou et al., ), they propose the use of the following rate matrix:

$$ $\displaystyle Q^{\star}_{x,y}(t)=Q^{\mathrm{pre}}_{x,y}(t)\frac{\exp(v_{t}(y)/ \alpha)}{\exp(v_{t}(x)/\alpha)}.$ $$

where $Q^{\mathrm{pre}}_{x,y}(t)$ is a rate matrix in the pre-trained model. This suggests that, with standard discretization, the optimal policy at each time step is:

$$ $\displaystyle p(x_{t+\delta t}=y|x_{t}=x)=\mathrm{I}(x\neq y)+Q^{\star}_{x,y}( t)(\delta t)$ (2) $$

where $\delta t$ is step size. Asymptotically, this is equivalent to sampling from the optimal policy in Theorem 1, as we will show in Remark 3. However, as Nisonoff et al. () note, sampling from the optimal policy requires $O(KL)$ computation, where $K$ is the vocabulary size and $L$ is the sequence length, which is computationally expensive for large $K$ and $L$. To address this issue, they propose using a Taylor approximation by computing the gradient once. However, this is a heuristic in the sense that there is no theoretical guarantee for this approximation. In contrast, we avoid this computational overhead in a different manner, i.e., through importance sampling and resampling. Our algorithm has an asymptotic guarantee as $M$ goes to infinity. Empirically, we have compared two methods in Section 7.

###### Remark 3 (Asymptotic equivalence between formula in Nisonoff et al. ( 2024 ) and Theorem 1 ) .

The informal reasoning is as follows. Recall that the pre-trained policy can be written as

$$ $\displaystyle p(x_{t+\delta t}=y|x_{t}=x)=\mathrm{I}(x\neq y)+Q^{\mathrm{pre}} _{x,y}(t)(\delta t).$ $$

Then, Theorem 1 states that the optimal policy is

$$ $\displaystyle\frac{\{\mathrm{I}(x\neq y)+Q^{\mathrm{pre}}_{x,y}(t)(\delta t)\} \exp(v_{t}(y)/\alpha)}{\sum_{z}\{\mathrm{I}(x\neq z)+Q^{\mathrm{pre}}_{x,z}(t) (\delta t)\}\exp(v_{t}(z)/\alpha)}.$ $$

Now, we have

$$ $\displaystyle\frac{\{\mathrm{I}(x\neq y)+Q^{\mathrm{pre}}_{x,y}(t)(\delta t)\} \exp(v_{t}(y)/\alpha)}{\sum_{z}\{I(x\neq z)+Q^{\mathrm{pre}}_{x,z}(t)(\delta t )\}\exp(v_{t}(z)/\alpha)}$ $\displaystyle=\frac{\mathrm{I}(x\neq y)+Q^{\mathrm{pre}}_{x,y}(t)(\delta t)\} \exp(v_{t}(y)/\alpha)}{\exp(v_{t}(x)/\alpha)}\times\left\{1+O(\delta t)\right\}$ $\displaystyle\approx\frac{\{\mathrm{I}(x\neq y)+Q^{\mathrm{pre}}_{x,y}(t)( \delta t)\}\exp(v_{t}(y)/\alpha)}{\exp(v_{t}(x)/\alpha)}=\mathrm{I}(x\neq y)+ \frac{Q^{\mathrm{pre}}_{x,y}(t)(\delta t)\exp(v_{t}(y)/\alpha)}{\exp(v_{t}(x)/ \alpha)}.$ $$

Thus, this recovers the formula (2).

###### Remark 3 (Asymptotic equivalence between formula in Nisonoff et al. ( 2024 ) and Theorem 1 ) .

##### Comparison with DiGress in Vignac et al. ( 2022 ) .

Vignac et al. () proposed a diffusion model for graphs where each sampling and denoising step operates directly on the discrete structure, avoiding continuous relaxation. They discuss how to implement guidance by treating rewards as a classifier. To bypass the exponential computational cost of sampling from the optimal policy ($p^{(\alpha)}$ in Theorem 1), they employ a Taylor expansion. While it requires the calculation of gradients for value functions, then it mitigates the exponential blow-up in computational time. In contrast, we avoid this computational blow-up through importance sampling (IS) and resampling. A detailed empirical comparison between our method and theirs is left for future work.

###### Remark 4 .

Note that in our molecule generation experiment in Section 7, we use GDSS (Jo et al., ), which operates in continuous space and differs from DiGress.

###### Remark 4 .

## Appendix D Extension with Arbitrary Proposal Distribution

Here, we describe the algorithm where the proposal distribution is not necessarily derived from the policy of the pre-trained model, as summarized in Algorithm 5. Essentially, we only adjust the importance weight. In practice, we can use the gradient of a differentiable proxy model, such as DPS, as the proposal distribution $q_{t-1}$. Even if the differentiable proxy (value function) models are not highly accurate, our method will still perform effectively since other value function models $\hat{v}_{t-1}$ can be non-differentiable.

Figure: Algorithm 5 SVDD (Soft Value-Based Decoding in Diffusion Models)

## Appendix E Soft Q-learning

In this section, we explain soft value iteration to estimate soft value functions, which serves as an alternative to Monte Carlo regression.

##### Soft Bellman equation.

Here, we use the soft Bellman equation:

$$ $\displaystyle\exp(v_{t}(x_{t})/\alpha)=\int\exp(v_{t-1}(x_{t-1})/\alpha)p^{ \mathrm{pre}}_{t-1}(x_{t-1}|x_{t})dx_{t-1},$ $$

as proved in Section 4.1 in (Uehara et al., ). In other words,

$$ $\displaystyle v_{t}(x_{t})=\alpha\log\{\mathbb{E}_{x_{t-1}\sim p^{\mathrm{pre} }(\cdot|x_{t})}\left[\exp(v_{t-1}(x_{t-1})/\alpha)|x_{t}\right]\}.$ $$

##### Algorithm.

Based on the above, we can estimate soft value functions recursively by regressing $v_{t-1}(x_{t-1})$ onto $x_{t}$. This approach is often referred to as soft Q-learning in the reinforcement learning literature (Haarnoja et al., ; Levine, ).

Figure: Algorithm 6 Value Function Estimation Using Soft Q-learning

In our context, due to the concern of scaling of $\alpha$, as we have done in Algorithm 2, we had better use

$$ $\displaystyle v_{t}(x_{t})=\mathbb{E}_{x_{t-1}\sim p^{\mathrm{pre}}(\cdot|x_{t })}\left[v_{t-1}(x_{t-1})|x_{t}\right].$ $$

With the above recursive equation, we can estimate soft value functions as in Algorithm 6.

## Appendix F Additional Experimental Details

We further add additional experimental details.

### F.1 Additional Setups for Experiments

#### F.1.1 Settings

##### Images.

We define compressibility score as the negative file size
in kilobytes (kb) of the image after JPEG compression following (Black et al., ).
We define aesthetic scorer implemented as a linear MLP on top of the CLIP embeddings, which is trained on more than 400k human evaluations. As pre-trained models, we use Stable Diffusion, which is a common text-to-image diffusion model. As prompts to condition, we use animal prompts following (Black et al., ) such as [Dog, Cat, Panda, Rabbit, Horse,…].

##### Molecules.

We calculate QED and SA scores using the RDKit (Landrum et al., ) library. We use the docking program QuickVina 2 (Alhossary et al., ) to compute the docking scores following Yang et al. (), with exhaustiveness as 1. Note that the docking scores are initially negative values, while we reverse it to be positive and then clip the values to be above 0, i.e.. We compute DS regarding four proteins, parp1 (Poly [ADP-ribose] polymerase-1),
5ht1b (5-hydroxytryptamine receptor 1B), braf (Serine/threonine-protein kinase B-raf), and jak2 (Tyrosine-protein kinase JAK2), which are target proteins that have the highest AUROC scores of protein-ligand binding affinities for DUD-E ligands approximated with AutoDock Vina.

##### DNA, RNA sequences.

We examine two publicly available large datasets: enhancers ($n\approx 700k$) (Gosai et al., ) and UTRs ($n\approx 300k$) (Sample et al., ), with activity levels measured by massively parallel reporter assays (MPRA) (Inoue et al., ). These datasets have been widely used for sequence optimization in DNA and RNA engineering, particularly in advancing cell and RNA therapies (Castillo-Hair and Seelig, ; Lal et al., ; Ferreira DaSilva et al., ; Uehara et al., ). In the Enhancers dataset, each $x$ is a DNA sequence of length 200, while $y\in\mathbb{R}$ is the measured activity in the Hep cell line. In the 5’UTRs dataset, $x$ is a 5’UTR RNA sequence of length 50, and $y\in\mathbb{R}$ is the mean ribosomal load (MRL) measured by polysome profiling.

#### F.1.2 Baselines and Proposals

We will explain in more detail how to implement baselines and our proposal. We use A100 GPUs for all the tasks.

##### SVDD-MC.

In SVDD-MC, we require value function models. For images, we use standard CNNs for this purpose, with the same architecture as the reward model. For molecular tasks, we use a Graph Isomorphism Network (GIN) model (Xu et al., ) as the value function model. Notably, this model is not differentiable w.r.t. inputs. For GIN, we use mean global pooling and the RELU activation function, and the dimension of the hidden layer is 300. The number of convolutional layers in the GIN model is selected from the set {3, 5}; and we select the maximum number of iterations from {300, 500, 1000}, the initial learning rate from {1e-3, 3e-3, 5e-3, 1e-4}, and the batch size from {32, 64, 128}.
For the Enhancer task, we use the Enformer model (Avsec et al., ) as the value function model. The Enformer trunk has 7 convolutional layers, each having 1536 channels. as well as 11 transformer layers, with 8 attention heads and a key length of 64. Dropout regularization is applied across the attention mechanism, with an attention dropout rate of 0.05, positional dropout of 0.01, and feedforward dropout of 0.4. The convolutional head for final prediction has 2*1536 input channels and uses average pooling, without an activation function. The model is trained using a batch size selected from {32, 64, 128, 256}, the learning rate from {1e-4, 5e-4, 1e-3}, and the maximum number of iterations from {5k, 10k, 20k}.
For the 5’UTR task, we adopt the [ConvGRU](https://github.com/jacobkimmel/pytorch_convgru?tab=readme-ov-filea) model (Dey and Salem, ). The ConvGRU trunk has a stem input with 4 channels and a convolutional stem that outputs 64 channels using a kernel size of 15. The model contains 6 convolutional layers, each initialized with 64 channels and a kernel size of 5. The convolutional layers use ReLU as the activation function, and a residual connection is applied across layers. Batch normalization is applied to both the convolutional and GRU layers. A single GRU layer with dropout of 0.1 is added after the convolutional layers. The convolutional head for final prediction uses 64 input channels and average pooling, without batch normalization. For training, the batch size is selected from {16, 32, 64, 128}, the learning rate from {1e-4, 2e-4, 5e-4}, and the maximum number of iterations from {2k, 5k, 10k}.
All function models are trained to converge in the learning process using MSE loss.

##### SVDD-PM.

For this proposal, we directly use the reward feedback to evaluate. We remark when the reward feedback is also learned from offline data, technically, it would be better to use techniques mitigating over-optimization as discussed in Uehara et al. (). However, since this point is tangential in our work, we don’t do it.

##### DPS.

We require differentiable models. For this task, for images, enhancers, and 5’UTRs, we use the same method as SVDD-MC. For molecules, we follow the implementation in Lee et al. (), and we use the same GNN model as the reward model. Note that we cannot compute derivatives with respect to adjacency matrices when using the GNN model. Regarding $\alpha$, we choose several candidates and report the best one. For image tasks we select from [5.0, 10.0] and for bio-sequence tasks we select from [1.0, 2.0]. For molecule QED task we select from {0.2, 0.3, 0.4, 0.5}, for molecule SA task {0.1, 0.2, 0.3}, and for molecule docking tasks we select from {0.4, 0.5, 0.6}.

##### SMC.

For value function models, we use the same method as SVDD-PM. Regarding $\alpha$, we choose several candidates and report the best one. For image tasks we select from [10.0, 40.0]. For Enhancer and 5’UTR tasks as well as molecule QED and SA tasks we select from {0.1, 0.2, 0.3, 0.4}, while for molecule docking tasks we select from {1.5, 2.0, 2.5}.

### F.2 Software and Hardware

Our implementation is under the architecture of PyTorch (Paszke et al., ). The deployment environments are Ubuntu 20.04 with 48 Intel(R) Xeon(R) Silver, 4214R CPU @ 2.40GHz, 755GB RAM, and graphics cards NVIDIA RTX 2080Ti. Each of our experiments is conducted on a single NVIDIA RTX 2080Ti or RTX A6000 GPU.

### F.3 Additional Results

##### Histograms.

In the main text, we present several quantiles. Here, we plot the reward score distributions of generated samples as histograms in Figure 5.

Figure: (a) Images: compressibility
Refer to caption: extracted/5950550/images/Images_compress.png

##### Performance of value function training.

We report the performance of value function learning using Monte Carlo regression as follows in SVDD-MC. In Figure 6, we plot the Pearson correlation on the test dataset for the Enhancer and 5’UTR tasks, as well as the test MSE for the molecular task of parp1 docking score.

Figure: (a) 5’UTR MRL
Refer to caption: extracted/5950550/images/UTR_MRL.png

##### Validity metrics for molecule generation.

To evaluate the validity of our method in molecule generation, we report several key metrics that capture different aspects of molecule quality and diversity in Table 3 on page 3.

**Table 3: Comparison of the generated molecules of pre-trained GDSS model and SVDD applied on various metrics.**
| Method | Valid | Unique | Novelty | FCD | SNN | Frag/Test | Scaf | NSPDK MMD | Mol Stable | Atm Stable |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pre-trained | 1.0 | 1.0 | 1.0 | 22.2799 | 0.2992 | 0.8274 | 0.0033 | 0.0260 | 0.2903 | 0.9256 |
| SVDD | 1.0 | 0.9375 | 1.0 | 21.5671 | 0.3441 | 0.7803 | 0.0838 | 0.0772 | 0.4783 | 0.9095 |

The validity of a molecule indicates its adherence to chemical rules, defined by whether it can be successfully converted to SMILES strings by RDKit. Uniqueness refers to the proportion of generated molecules that are distinct by SMILES string. Novelty measures the percentage of the generated molecules that are not present in the training set. Fréchet ChemNet Distance (FCD) measures the similarity between the generated molecules and the test set. The Similarity to Nearest Neighbors (SNN) metric evaluates how similar the generated molecules are to their nearest neighbors in the test set. Fragment similarity measures the similarity of molecular fragments between generated molecules and the test set. Scaffold similarity assesses the resemblance of the molecular scaffolds in the generated set to those in the test set. The neighborhood subgraph pairwise distance kernel Maximum Mean Discrepancy (NSPDK MMD) quantifies the difference in the distribution of graph substructures between generated molecules and the test set considering node and edge features. Atom stability measures the percentage of atoms with correct bond valencies. Molecule stability measures the fraction of generated molecules that are chemically stable, i.e., whose all atoms have correct bond valencies. Specifically, atom and molecule stability are calculated using conformers generated by RDKit and optimized with UFF (Universal Force Field) and MMFF (Merck Molecular Force Field).

We compare the metrics using 512 molecules generated from the pre-trained GDSS model and from SVDD optimizing SA, as shown in Table 3. Overall, our method achieves comparable performances with the pre-trained model on all metrics, maintaining high validity, novelty, and uniqueness while outperforming on several metrics such as molecule stability, FCD, SNN, and scaffold similarity. These results indicate that our approach can generate a diverse set of novel molecules that are chemically plausible and relevant.

##### More Ablation Studies.

We provide several more ablation studies regarding $M$ on top of the results in the main text, as plotted in Figure 7. The results are consistent with what we have observed in Figure 3.

Figure: (a) Performance of SVDD as $M$ varies.
Refer to caption: extracted/5950550/images/change_K_compress.png

##### Visualization of more generated samples.

We provide additional generated samples in this section. Figure 8 and Figure 9 show comparisons of generated images from baseline methods and SVDD with different M values regarding compressibility and aesthetic score, respectively. Figure 10 and Figure 11 presents the comparisons of visualized molecules generated from the baseline model and SVDD regarding QED and SA, respectively. The visualizations validate the strong performances of SVDD. Given that SVDD can achieve optimal SA for many molecules, we also visualize some molecules with optimal SA generated by SVDD, as shown in Figure 12. In Figure 13, Figure 14, Figure 15, and Figure 16 we visualizes the docking of SVDD-generated molecular ligands to proteins parp1, 5ht1b, jak2, and braf, respectively. Docking scores presented above each column quantify the binding affinity of the ligand-protein interaction, while the figures include various representations and perspectives of the ligand-protein complexes.
We aim to provide a complete picture of how each ligand is situated within both the local binding environment and the larger structural framework of the protein.
First rows show close-up views of the ligand bound to the protein surface, displaying the topography and electrostatic properties of the protein’s binding pocket and providing insight into the complementarity between the ligand and the pocket’s surface. Second rows display distant views of the protein using the surface representation, offering a broader perspective on the ligand’s spatial orientation within the global protein structure. Third rows provide close-up views of the ligand interaction using a ribbon diagram, which represents the protein’s secondary structure, such as alpha-helices and beta-sheets, to highlight the specific regions of the protein involved in binding. Fourth rows show distant views of the entire protein structure in ribbon diagram, with ligands displayed within the context of the protein’s full tertiary structure.
Ligands generally fit snugly within the protein pocket, as evidenced by the close-up views in both the surface and ribbon diagrams, which show minimal steric clashes and strong surface complementarity.

Figure: Figure 8: Additional generated samples (Domain: images, Reward: Compressibility)
Refer to caption: extracted/5950550/images/Images_comp.jpg

Figure: Figure 9: Additional generated samples (Domain: Images, Reward: Aesthetic score)
Refer to caption: extracted/5950550/images/Images_asthetic.jpg

Figure: Figure 10: Additional generated samples (Domain: Molecules, Reward: QED score)
Refer to caption: extracted/5950550/images/qed.png

Figure: Figure 11: Additional generated samples (Domain: Molecules, Reward: SA score, normalized as $(10-SA)/9$)
Refer to caption: extracted/5950550/images/sa.png

Figure: Figure 12: Additional generated samples from SVDD-MC (Domain: Molecules, Reward: SA score = 1.0 (normalized as $(10-SA)/9$))
Refer to caption: extracted/5950550/images/molecule_SA1.0_grid_zinc_ours.png

Figure: Figure 13: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score - parp1 (normalized as $max(-DS,0)$))
Refer to caption: extracted/5950550/images/mol_vina_plot_pymol_grid.png

Figure: Figure 14: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score - 5ht1b (normalized as $max(-DS,0)$))
Refer to caption: extracted/5950550/images/mol_vina3_plot_pymol_grid.png

Figure: Figure 15: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score - jak2 (normalized as $max(-DS,0)$))
Refer to caption: extracted/5950550/images/mol_vina4_plot_pymol_grid.png

Figure: Figure 16: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score - braf (normalized as $max(-DS,0)$))
Refer to caption: extracted/5950550/images/mol_vina5_plot_pymol_grid.jpg

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weakness section.
soundness: 3
strengths: - The paper is generally well-written, and the motivation is also clear. It starts with an important problem and proposes a well-motivated solution that requires no finetuning or differential proxy model. 

- The paper is clear about how two critical challenges (the soft-value function is both unknown and unnormalized) are addressed by the proposed algorithm.
summary: This paper proposes a method for diffusion models to sample data that is both within target distribution and maximizing some downstream reward function. The problem the paper studies is of great importance, and the method shows empirical effectiveness in some downstream tasks.
weaknesses: - The soft value function seems to difficult to approximate in general. Is there any anlysis or justification to quantify the quality of the approximation? How does one know a good approximation is indeed attained? Moreover, how does the approximation quality matter for the generation? More ablation study can improve the paper further.

- Is there any additional computational overhead for the proposed method? Is the approximation to the soft value function costly?

- The performance gain does not seem to be very significant compared to simple baselines, say Best-of-N. From Table 2, Best-of-N baseline is only incrementally worse than the proposed method in molecule-related experiments.

- A minor question: Does the size of the diffusion model affect the performance of SVDD? I will be interested to see how this method works for diffusion models of different size.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. I did not see it mentioned in the manuscript -- how many seeds were used for experiments?
2. Could the authors provide a discussion of the relation between the consistency of nested SMC and the consequence of using more or less particles in their method?
3. How expensive is training the soft-value function estimate in SVDD-MC?  If it is reasonably long would it be worth adding a fine-tuning based method (e.g., relative trajectory balance https://arxiv.org/abs/2405.20971).  On the other hand, if training the soft-value estimate is especially cheap it would be worthwhile to emphasize this more in the manuscript as a benefit of this method compared to direct fine-tuning methods.
4. Since the goal of SVDD is to sample from the product distribution of reward and pretrained model could they add some metrics evaluating the diversity of their samples or their naturalness (e.g., likelihood under the pre-trained model when available)?
soundness: 1
strengths: The paper presents a number of strengths, such as

- Presenting a novel application of nested Sequential Monte Carlo to the difficult problem of sampling from the product distribution $p^*(x_0)$ given a pre-trained diffusion model.
- The method, especially SVDD-PM, provides a particularly efficient method to sample from the product distribution when no differentiable reward is available and the reward function is cheap.  The manuscript shows that SVDD can indeed increase the reward over the pre-trained model, offering a compelling option to sample from the target product distribution with little overhead.
- The problem of cheaply sampling from the product distribution in the presence of non-differentiable rewards is especially significant as existing methods typically require availability of gradients or expensive (typically simulation-based) fine-tuning algorithms.  Non-differentiable rewards are often seen in scientific discovery, a target area aptly pointed out by the authors.
summary: The paper introduces SVDD, a method which aims to sample from the product distribution $p^*(x_0) \propto p^{pre}_0(x_0) \exp(R(x_0)/\alpha)$ for some non-negative reward function $R(x)$, constant $\alpha \geq 0$ and pre-trained diffusion model $p^{pre}_t(x_t | x_{t + 1})$.

The method employs an SMC-inspired procedure for iteratively sampling and reweighting samples according to a soft value function and its corresponding optimal policy.  Providing two options to obtain the soft-value function (which is required for the method's importance sampling step), the authors show that SVDD can be used with a cheap approximation based on the diffusion model's denoiser or an amortized version based on regression to a Monte Carlo estimate.  The authors evaluate the performance of SVDD on a series of tasks -- images, molecule design, and DNA/RNA design.
weaknesses: Overall I had a some issues regarding clarity of the paper, concerns about sources of bias that are not discussed in the manuscript, and am worried that the experimental section does not paint a fair picture of SVDD's performance relative to baselines.  I will discuss each of these in turn



### Unclear focus of probabilistic inference vs reward maximization

Section 3.2 states that the objective of this paper is to perform probabilistic inference and sample from the target distribution $p^*(x_0) \propto p^{pre}(x_0)\exp(R(x_0) / \alpha)$.  However, towards the beginning of the experiment section and throughout the appendix the manuscript begins to say that SVDD is actually meant for reward maximization, not the problem of sampling from $p^*(x_0)$.  In particular, the manuscript states that in practice they set $\alpha = 0$ for all experiments, which corresponds to a constrained reward maximization where $p^*(x_0)$ is a Dirac centered at $x_0^* = \underset{x_0 \in Support(p_0^{pre}(x_0))}{\arg\max}R(x_0)$.  This is quite different from sampling from the $p^*(x_0)$ for any $\alpha$ and if this is the goal of SVDD it should be compared to baselines which try to do reward maximization.



### Missing discussion and investigation on bias of soft value function estimates

The manuscript defines the soft value function as $v_t(x_t) = \alpha \log \mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[\exp(R(x_0) / \alpha)]$.  Next, due to issues with numerical stability for small values of $\alpha$ the authors make use of an approximation

$v_t(x_t) = \alpha \log \mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[\exp(R(x_0) / \alpha)]$

$ \approx \alpha \log \exp(\mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[R(x_0)] / \alpha)$

$ = \mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[R(x_0)]$

The second step takes the $\exp$ function outside of the expectation and as such requires an application of Jensen's inequality, implying that $v_t(x_t) \geq \mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[R(x_0)]$.  This means that the Monte Carlo regression used for SVDD-MC is in fact biased (although consistent), a fact which is not mentioned in the paper.

The situation is more complicated for SVDD-PM which first applies Jensen's and then another approximation as 

$v_t(x_t) \geq \mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[R(x_0)]$

$ \approx R(\mathbb{E}_{x_0 \sim p^{pre}(x_0 | x_t)}[x_0])$

It is unclear to me whether the error of the posterior mean estimate can be shown to be bounded as the reward function is potentially non-convex, but would be happy if the authors had some insight into this.

Given that SVDD requires accurate estimates of the soft-value functions to sample from the target distribution $p^*(x_0)$ I would be more convinced of SVDD's abilities were there a more detailed (potentially including an empirical results) analysis of the bias of the Monte Carlo regression and posterior mean estimates.



### Issues with inconsistent setting of $\alpha$ for baselines

The stated goal of SVDD is to sample from the target distribution $p^*(x_0;\alpha) \propto p^{pre}(x_0)\exp(R(x_0) / \alpha)$, where the temperature parameter $\alpha$ controls how peaky $p^*(x_0;\alpha)$ becomes.  As discussed above, as $\alpha \rightarrow 0$ the target distribution becomes focused on the maximizer of the reward which is in the support of the pretrained model such that 

$\mathbb{E}_{x \sim p^* (x;0^+)}[R(x)] = \underset{x_0 \in Support(p^{pre}(x_0))}{\arg\max}R(x_0)$.  

In general, as the value of $\alpha$ is decreased the expected reward under the target distribution should increase.  As such, comparing the distribution of rewards of generated samples for methods using different values of $\alpha$ does not paint an accurate picture of each method's performance as one method having a higher quantile reward may simply be a consequence of the setting of $\alpha$. 

Unfortunately, the manuscript's experiments use significantly different values of $\alpha$ for its method and baselines while using the reward at different quantiles as the main performance metric.  This is more problematic as the value of $\alpha$ for their method is set to $0$ (where the true expected reward is the maximum reward value) and a larger value of $\alpha$ for baselines.  Because the value of $\alpha$ is not set to be equal for SVDD and all baselines I do not believe that the experimental results in Table 2 paint a fair picture of SVDD's performance.



### Overall comments

I generally have concerns with the settings of either the number of particles $M$ being too small or the bias of the soft-value function estimates being too large.  As far as I understand (and perhaps I am missing something!) by setting $\alpha=0$ for SVDD in the experiment section the method _should_ be suffering from mode collapse and generating very high reward samples as the target distribution $p^*(x_0)$ is a Dirac centered at $x_0^* = \underset{x \in Support(p_0^{pre}(x))}{\arg\max}R(x)$.  However, samples from SVDD do not exhibit this expected mode collapse, which seems to indicate that either many more particles $M$ need to be used or the bias from the value function estimation is preventing the algorithm from properly sampling from the target distribution.

I note that the main reason for my score mostly due to the issue with inconsistent setting of $\alpha$ for SVDD and baselines in the experiments section as well as the missing discussion.  A missing discussion/analysis of the bias of the value function estimates and their impact on SVDD's ability to sample from the target distribution also contributes significantly to my score.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - Is there a typo under the first equation in Section $4.1$, where the expectation is induced by $p_t^{pre}(\cdot | x_{t-1}) -$ note the negative instead of positive sign.
soundness: 3
strengths: - The authors provide a widely applicable method that can be applied both to discrete and continuous diffusion settings.
- The proposed method, unlike previous guidance algorithms, does not rely on an explicitly trained conditional diffusion model (eg. for classifier free guidance), or on differentiable reward terms (eg. classifier based guidance).
- Results on both image and scientific domains highlight the benefits of the approach towards controlled generation of objects with high downstream reward, as intended.
- The work also conducts experiments in a wide variety of domains, ranging from images, molecules and DNA.
summary: This work provides a unified framework of guidance in diffusion models, both discrete and continuous, with minimal additional training and applicability in domains where a downstream reward might not even be differentiable. The proposed method SVDD (MC and PM) is applicable in discrete diffusion where a continuous gradient of energy cannot be directly added to the discrete state space, as well as cases where the reward is non differentiable which is the case in a lot of scientific domains. The work tackles an important problem in the scientific domain and leads to controllable generation without having to fine-tune large scaled models. Their results show that generations from SVDD lead to higher downstream rewards than the baselines considered.
weaknesses: - The authors consider setting $\alpha=0$ in their experiments. However, prior work highlights that setting $\alpha=0$ leads to over-optimization and increasingly reduces the diversity and realistic nature of the samples. Could the authors provide clarity on why this is not a problem in their setup?
- The work relies on two major assumptions (one for SVDD-MC and the other for SVDD-PM), which are neither well motivated theoretically nor are there any details provided about it. 
- **Assumption about SVDD-MC**: The authors replace the logarithm of expectation with the expectation of logarithm in their quantity of interest, which in reality is only a bound on the actual quantity. Could the authors consider experimenting on some synthetic domain to describe the bias and variance caused by this approximation? When is this approximation reasonable and when would it be extremely incorrect?
- **Assumption about SVDD-PM**: This algorithm combines the above approximation with pushing the expectation inside the reward function $r(\cdot)$. As with above, could the authors conduct experiments on synthetic domains and highlight when and where such an assumption is reasonable, and when is it violated?
- While the approach leads to generation of samples with high reward, the authors do not provide any kind of metrics that test for diversity of the samples generated.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: I would appreciate it if the authors could address my theoretical concerns regarding 1.) what is the optimal distribution hit by SVDD 2.) what is the bias introduced in the MC estimate and 3.) the actual computational cost of SVDD-MC.

In addition, I would appreciate it if the authors could carry out the additional experiments I have suggested with added diversity quantification.
soundness: 2
strengths: The paper tackles a timely problem in considering fine-tuning diffusion models. Moreover, the suggested approach of SVDD-PM enjoys being computationally cheap to use as it does not require any further training while both SVDD-MC and SVDD-PM are applicable in settings where the reward function is non-differentiable. This is impactful because this unlocks a lot of potential application domains that have black-box rewards where learning a surrogate reward model is non-trivial. Finally, the paper considers a diverse set of experimental settings to showcase the universality of the proposed approach.
summary: This paper introduces SVDD which is a new method to fine-tune diffusion models in both continuous and discrete spaces. SVDD-MC learns a soft value function by regression to $x_t$ while SVDD-PM exploits the posterior mean parametrization of masked diffusion models to estimate the soft value function directly. Given a soft value function, SVDD can be applied to a general class of reward functions, including non-differentiable ones, at inference time without further fine-tuning. This can be seen as a variation of the famous Sequential Monte-Carlo algorithm but applied and modified for diffusion models. Experiments are done on images, molecules, and docking and show improvements in fine-tuning performance under a specified reward metric.
weaknesses: While the paper has some notable strengths there are a few lingering questions that point to potential weaknesses. I will try to list them below.

**Theoretical weaknesses**

Two main questions arise when looking at the setup. The first one is the actual target distribution and whether SVDD hits it. In the continuous setting, I have severe doubts about whether the correct terminal distribution is reached due to the initial value function bias problem as introduced in Adjoint Matching (Domingo-Enrich et. al 2024). Certainly, nothing in the current theory suggests the process during fine-tuning is memoryless. Moreover, it is unclear what ramifications and bias we introduce in the MC setup when the regression is done using $r(x_0) \to x_t$ as opposed to the more numerically unstable soft value function. For example, I believe when you remove the $\exp$ from your regression target, this is a based value function but there is no discussion on this point outside of the fact that it is less numerically stable. As a result, I am dubious about the claims made about hitting the correct target distribution.


Another question is the connection to Sequential Monte Carlo. There is a discussion on this in the paper but I think it's not accurate enough. I disagree with the statement made in the paper. The algorithm you propose is quite literally SMC but adapted to reward maximization, there is even a resampling step which is exactly what is done in SMC. The arguments that SMC is done over a batch are lukewarm. There is nothing wrong with demonstrating that SMC can be effectively applied to sampling from discrete diffusion---like analogously done for an autoregressive model by Zhao et. al 2024---and this is a valuable contribution. I suggest the authors be a bit more forthright with their claims as I would buy it a lot more. In fact, with the right framing, you achieve novelty by showing how SMC applies to a newer more interesting problem domain.

**Additional Technical weaknesses**

One of the main selling points of SVDD is the fact that it is supposed to be a cheap inference time algorithm. This I believe is not quite true because of the need to estimate the soft value function in SVDD-MC. Indeed, one must estimate the soft value function using rollouts which I believe adds a heavy pre-processing step. I also did not see SVDD-MC in the ablation studies about computational cost---likely because it's significantly more expensive than SVDD-PM. Thus, I believe the main claim for SVDD-MC being a lightweight method is a bit misleading. Of course, if you had the perfect estimated value function then inference scales as indicated in the plot for 3c,d but this is not the full picture.

**Experimental weaknesses**

A glaring missing baseline is Relative Trajectory Balance (Venkataraman et. al 2024) which does fine-tuning exactly like this paper considers for both discrete and continuous diffusion models. I kindly request the authors to consider adding this important baseline. Moreover, it is a bit surprising that there is no text experiment given the heavy emphasis on using Masked Diffusion Models which have primarily been introduced for text. I would be encouraged to see a text experiment---perhaps of a similar scale to Zhao et. al 2024---to highlight that SVDD can be applied in the text setting.

The current experimental findings in Table 2 are not complete as they do not show other important aspects of the generated sample. They simply show that reward is maximized but this could also happen through gamification of the reward function. For instance, I would appreciate the authors providing sample-based diversity metrics to quantify how bad the drop in diversity is among the baselines. At the very minimum, FID scores for images should be provided and I'll let the authors determine appropriate diversity metrics for the other domains to complement the findings in Table 2.

**Closing remarks**

Having said all of these weaknesses, I will note that I am open to significantly raising my score if **all of my concerns** are adequately addressed to my level of satisfaction. I will also state that I did not read the appendix so if I have missed something I would appreciate a pointer to the result there.

I encourage the authors in their rebuttal endeavors and I hope they can strengthen the paper which I would like to eventually recommend for acceptance but not in its current state.


**References**

Venkatraman, Siddarth, et al. "Amortizing intractable inference in diffusion models for vision, language, and control." arXiv preprint arXiv:2405.20971 (2024).

Zhao, Stephen, et al. "Probabilistic inference in language models via twisted sequential monte carlo." arXiv preprint arXiv:2404.17546 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: I do not have further questions.
soundness: 3
strengths: There are several advantages of SVDD over previous methods:
1. No need for differentiable proxy models
2. No fine-tuning required
3. Works with both continuous and discrete spaces
5. Maintains better sample diversity compared to other approaches

The writing of this paper is clear.
summary: This paper introduces a new method called Soft Value-based Decoding in Diffusion models (SVDD) for optimizing diffusion models to generate samples with desired properties while maintaining naturalness. The contributions include:
- SVDD is an inference-time technique that doesn't require fine-tuning the original diffusion model
- Can work with non-differentiable reward functions, unlike previous methods that required differentiable proxy models
- Applicable to both continuous and discrete diffusion models in a unified way

The algorithm works in the following way:
1. Uses "soft value functions" that predict future rewards from intermediate noisy states
2. At each denoising step:
   - Generates multiple samples using the pre-trained model
   - Selects samples based on their predicted value
There are two variants:
   - SVDD-MC: Uses Monte Carlo regression to learn value functions
   - SVDD-PM: Directly uses reward feedback without additional training

Experimental Results span across multiple domains: image generation, molecule generation, DNA/RNA sequence generation. The proposed method consistently outperformed baseline methods while maintaining sample validity. 
The paper demonstrates that SVDD provides an effective way to guide diffusion models toward desired properties while preserving the natural characteristics learned during pre-training.
weaknesses: The main weakness is about novelty. To be more specific, I can not see significant difference with twisted SMC methods (e.g., the papers mentioned in Sec 6 and App B). In the writing I see two differences claimed by the authors:
1. In previous works such as Wu et al., the reward is a classifier; while here it is "reward maximization" setting. 

First, I think the setting in this work should not be called "reward maximization" but be called "alignment" or "reward sampling" or similar names, due to the reasons in Sec 3.2 "HIGH REWARDS WHILE PRESERVING NATURALNESS". Second, whether the reward function is a classifier is not critical, as even for an usual reward r(x), we can understand it as an unnormalized probability prob(optimality | x).

2. "SMC methods involve resampling across the “entire” batch, which complicates parallelization. Additionally, when batch sizes are small, as is often the case with recent large diffusion model"

I do not quite understand this part. I may miss the difference between SVDD and twisted SMC methods. Does the batch size mean the number of particles in SMC? It will be good if there could be a clarification.

Meta-review: This work presents a unified framework for guidance in diffusion models, encompassing both discrete and continuous settings, with minimal additional training. It extends applicability to domains where downstream rewards may not be differentiable. The proposed method, SVDD (MC and PM), addresses discrete diffusion scenarios where continuous energy gradients cannot be directly applied to the discrete state space. Additionally, it is well-suited for cases involving non-differentiable rewards, which frequently arise in scientific domains.

However, reviewers have raised concerns regarding the novelty of the approach, noting limited differentiation from existing twisted Sequential Monte Carlo (SMC) methods. The authors also acknowledge that the term "reward maximization" may be misleading in this context. Unlike SVDD, SMC methods require resampling across the entire batch, complicating parallelization.

Decision: Reject