Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding
Authors: Xiner Li
Yulai Zhao
Chenyu Wang
Gabriele Scalia
Gökcen Eraslan
Surag Nair
Tommaso Biancalani
Shuiwang Ji
Aviv Regev
Sergey Levine
Masatoshi Uehara
Venue: NeurIPS 2025 poster ()
Keywords: Biomolecular design
Reward-guided generation
Diffusion models
Primary area: machine_learning_for_sciences

Abstract: Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require differentiable proxy models (e.g., classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (e.g., classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation.

Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding
Xiner Li 1.4 Yulai Zhao Chenyu Wang Gabriele Scalia 4
Gokeen Eraslan Surag Nair Tommaso Biancalani Shuiwang Ji
Aviv Regev 4
Sergey Levine 5x
Masatoshi Uchara I
Texas A&M University Princeton University MIT *Genentech
UC Berkeley "EvolutionaryScale
Abstract
Diffusion models excel at capturing the natural design spaces of images, molecules, and biological sequences. However, for many applications. rather than merely generating designs that are natural, we aim to optimize downstream reward fune- tions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require "differentiable" proxy models (e.g., classi- fier guidance) or computationally-expennee classifier-free guidance, RL-based fine-tuning). Here, propose new method, Soft Value-based Decoding in Diffusion models (SVDD), to address these chal- lenges. SVDD an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the fu- ture, into the standard inference procedure of pre-trained diffusion models. Notably, SVDD avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us (1) directly use con-differentiable fea- tures/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of SVDD across several domains, including image generation, molecule generation (optimization of docking scores, QED, SA), and DNA/RNA generation (optimization of activity levels). The code is available at httts://gitub.coo/mmsssee///
fine-tuning of diffusion models (e.g.,
1 Introduction
Diffusion models have gained popularity as powerful generative models. Their applications extend beyond image generation include natural language generation (Sahoo al. 2024 Shi al.
2024 Lou et al. 2023), molecule generation (Jo et al.] 2022 Vignac et al. (022) ). and biological (DNA, RNA, protein) sequence generation (Avdeyev et al. 2023 Stark al.) 2024). In each of these domains, diffusion models have been shown to be very effective at capturing complex natural distributions. However, in practice, we might not only want to generate realistic samples, but to produce samples that optimize specific downstream reward functions while preserving naturalness by leveraging pre-trained models. For example, computer vision, we might aim to generate natural images with high aesthetic and alignment scores (Black et al., 2023 Fan et al.) 2023). In drug discovery, we may seek to generate valid molecules with high QED/S LLED/SA/doce A/oocing scores (Lee et al. 2023 Jin et al. 2018) or RNAs (such mRNA vaccines (Cheng et al. 2023)) with high translational efficiency and stability HCastillo-Hair & Seelig 2021 Asrani et al. 2018), 8), and regulatory DNAs that drives high cell-specificity of expression GGosa et al. 2023 Taskiran et al. 2024 Lal et al. 2024).
*Corresponding authors: regev. wcgev.avivégene. avivêgeno con, svlevinc@eoce hww.:eeccss.bckkey.......... borkeley.edae
ucharamasatosh oshiiiststtssstsss 11l.mmmmmmm..mmmmmmmmm.......m.m
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The optimization of downstream reward functions using pre-trained diffusion models has been approached in various ways, In our work, we focus on non- non-fine-tune tine-tuning-bes based methods because fine-tuning generative models (e.g.. when using classifier- free guidance (Ho et al. 20200 or RI RI-baaedd
fine-tuning (Black et al. 2023 Fan et al. 2023] Clark al., 2023] Prabhudesai et al., 2023 often becomes computationally intensive, especially as pre-trained generative models grow larger in the era of "foundation models", Although classifier guidance and its variants (e.g.. Dhariwal & Nichol (202I); Song et al. (2020b); Chung et al. (2022); Bansal et al. (2023); Ho et al. (2022) ) have shown some success as non-fine-tuning methods in these settings, they face significant challenges. First, as they would require constructing differentiable proxy models, they cannot directly incorporate useful non- differentiable features (e.g., molecular/ soolecular/protein descriptors (van Westen et al. 2013], Ghiringhelli et al.,
Gainza et al. 2020h) non con-differentiable reward feedback (e.g.. physics based simulations such as Vina and Rosetta (Trott& & Olson, 2010 Alhossary et al., 2015 Alford et al. 2017)). which are particularly important in molecule design optimize docking scores, stability, etc. This limitation also hinders the principled application of current classifier guidance methods to recently-
discrete diffusion models (Austin al.) 202T: Campbell et al. [022 Lou et al. 2023) (i.e., without
transforming the discrete space into the Euclidean space).
To tackle these challenges, novel method, Soft Value-based Decoding in Diffusion models (SVDD), for optimizing down- stream reward functions in diffusion models Figure 1). Inspired by recent literature on RL- based fine-tuning, we first introduce soft value functions that serve look-ahead functions, in- dicating how intermediate noisy samples lead to high rewards in the future of the diffusion denoising process. After learning (or approxi- mating) these value functions, we present new inference-ting technique, SVDD, which obtains multiple noisy states from the policy (i.e. de- noising map) of pre-trained diffusion models and selects the sample with the highest value function at each time step. Notably, SVDD does not require any additional learning as long as we have access to the reward feedback by utilizing the characteristics to 0 in terms of expectation in Figure Tp. Finally, to fully leverage test-time compute, we introduce a more computationally efficient variant, SVDD-R, which halts unpromising samples early and reallocates inference-time resources to more promising ones.
propose
X
35
RMA
Translatonal efficiency
Figure 1: Summary of SVDD. v denotes value functions that predict reward r(20) (at time 0) from states at time 1. SVDD involves two steps: (1) generating multiple noisy states from pre- pre-trained models, and (2) selecting the state with the highest value according to the value function.
diffusion models (i.e., the forward process in diffusion models to directly map t
Our novel technique for optimizing reward functions in pre-trained diffusion models makes two contributions. First, it eliminates the need to construct differentiable proxy models. This allows for the use of non-differentiable reward feedback, which is common in scientific fields, and makes our method applicable to recent discrete diffusion (Shi et al.] 2024 Sahoo et al.) 2024) models in a principled manner. Second, it avoids the need to fine-tune the generative model itself. This addresses the high computational cost associated with fine-tuning diffusson models. We demonstrate the effectiveness of SVDD across diverse domains, including image generation, molecule generation (optimization of docking scores, QED), and DNA/RNA generation (optimization of activity levels).
2
Related Works
We outline methods relevant to our goal. We defer other relevant works (e.g., decoding in LLMs, discrete diffusion models) to Section B due to space constraints.
Classifier guidance (Dhariwal & Nichol, 2021; Song et al., 2020a). It has been widely used to condition pre-trained diffusion models without fine-tuning. Although these methods do not originally focus on optimizing reward functions, they can be applied for this purpose (Uehara et al. 2024a Section 6.2). In this approach, an additional derivative of certain value function is incorporated into the drift term (mean) of preitrdined diffusion models during inference. Subsequent variants (e.g..

---

Chung et al. )2022); Ho et al. (2022); Guo et al. 2024); Yu et al. (2023)) have been proposed to
simplity the learning of value functions. However, these methods require constructing differentiable models, which limits their applicabiliity non-differentiable features/reward feedbacks commonly encountered in scientific domains as mentioned in [Sectionn 1] Additionally, this approach cannot be directly extended to discrete diffusion models (e.g., (Lou et al.) 2023) (hh et al. 2024 Sahoo et al. 2 in a principle way. Our approach aims to address these challenges.
Note a notable exception of classifier guidance tailored to discrete diffusion models has been recently proposed by Nisonoff et al. (202)): Schiff et al. (024). However, SVDD can be applied both continuous and discrete diffusion models in a unitied manner. Furthermore, their practical method requires the differentiability of proxy models, unlike SVDD. We compare its performance with our method in SSection 6) We provide further details in Section
SMC-Based Methods. Several SMC-based (sequential Monte-Carlo) guidance methods, which also do not need hon-differentiable reward feedback, have been proposed in recent work (Wu et al.
2024| Trripee al. 2022 Dou & Song 2024; Phillips al. 2024 Cardoso al. 2023), primarily
largeting conditioning problems. In contrast, our focus is on reward maximization. While there are
concurrent efforts exploring reward-guided generation (Kim al. 2025 Singhal et al. 2025 Ma
iet al. 2025 instantiation of nested importance sampling (nested-IS) SMC (Naesseth et al.) (2019 Algorithm 13), whereas other concurrent works typically employ standard sequential Monte Carlo. Additionally, we introduce novel technique for reward maximization in Section 5 that integrates ideas from both
Guo et al. 2025), our approach differs in key ways. Most notably, our al algorithm is an
SMC and nested-IS SMC organically.
in diffusion models. For more details, refer Section B In essence, we do not intend to claim that our approach is superior, as both methods have their respective advantages.
3 Preliminaries and Goal
We describe the standard method for training diffusion models and outline the objective of our work: optimizing downstream reward functions given pre-trained diffusion models.
3.1 Diffusion Models
In diffusion models (Ho et al. 2020 Song et al. 2020a), our goal is to learn sampler p(x) E ((X) given data consisting of x E &. The training process for standard diffusion model is summarized as follows. First, we introduce a (fixed) forward process g: X' + A(X). Next, we aim to learn backward process: {pr} where each Pt is X' + ( that the distributions induced by the forward process and backward process match marginally. For this purpose, by parametrizing the backward processes with Rd. we typically use the loss derived from the variational lower bound of the negative log- ligelikel (i.e., ELBO).
Here two examples of concrete parameterizations. Let a E R be noise schedule. Example (Continuous space). When X Euclidean space, we typically use the Gaussian distri- bution qe(- x) = M(yaz, (1 ax)). Then, the backward process can be parameterized as the Gaussian distribution whose mean is
Var(1 - 1 Jas + Vor-i(1 -aao--or)aaao (1))
where on - a, Here, (11; 8) is neural network that predicts zo from (Eglao) (z))).
Example (Diserete space in (Shhoo al. (2024); [Shi al. (2024)). Let X be a space of one-hot column vectors {x {0, 13K K over K classes with probabilities given by E A where A denotes the K-simplex. typical choice is gel. | x) = Categorical(a.æ: + (1 0z)m) where 00.... 0, Mask]. Then, the backward process can be parameterized as
E x; = 1). and Categorical(s) be the categorical distribution
1( xt), if +
Categorical (1-
(ār- (d0o((x+0))
if m,
1-a.
where -IIi_ ar Here, Lo(x 0) is neural network that predicts zo from Tt-
3

---

After learning the backward process, we can sample from distribution that emulates training data (i.e., p(x)) by sequentially sampling {pel: Fe+1)liz from T
Notation. The notation 6, denotes a Dirac delta distribution centered at a. The notation & indicates that the distribution is equal up normalizing constant. With slight abuse of notation, we often denote pr(+)-) by pr(-).
3.2 Objective: Generating Samples with High Rewards While Preserving Naturalness
We consider scenario where we have pre-trained diffusion model, which is trained using the loss function explained in [Secction 3.1 These pre-trained models are typically designed to excel at characterizing the natural design space (e.g., image space, biological sequence space, chemical space) by emulating the extensive training dataset. Our work focuses on obtaining samples that also optimize downstream reward functions X' + IR (e.g.. Quantitative Estimate of Druglikeness (QED) and Synthetic Accessibility (SA) in molecule generation), while maintaining the naturalness by leveraging pre- trainrd diffusion models. We formalize this goal follows.
J1=7" we denote the induced distribution by pPre A(X) (i.e. ppre (pre = | [IT==++1 pp- (Te (d)rr)) We aim to sample from the following distribution:
Given a pre-trained model {pF (ppre)o
pla) (x) = argmax E Bx [r(x)] -a (L(()))ppeT))))) x exp(r(#)/a)p**((() pE(A(X)
term
term(()
Here, term (a) introduced to optimize the reward function, while term (b) used to maintain the naturalness of the generated samples. Note these targets are widely employed in generative models, such as RLHF LLMs (Ziegler et al. 2019) Ouyang et al., 2022). Notably, we generally focus on regimes where is small, as primary objective is to generate high-reward samples. In the extreme scenario, when a - 0, this reduces to argmax,esupptrrr r(z).
Existing methods. Several existing approaches target this goal (or its variant), including classifier guidance, fine-tuning (RL-based or (assiffer-free), and Best-of-N, In our work, we focus non- fine-tuning- methods; specifically, we aim address the limitations of these methods: the requirement for differentiable proxy models in classifier guidance and the inefficiency of Best-of-N.
Finally, note that all results discussed in this paper be easily extended to cases where the pre- trained model is a conditional diffusion model. For example, in our image experiments (Section 6), the pre-trained model is conditional diffusion model conditioned text (e.g.. Stable Diffusion).
4 Soft Value-Based Decoding in Diffusion Models
We present the motivation behind developing our new algorithm. We then introduce SVDD, which satisfies desired properties, i.e., the lack of need for fine-tuning constructing differentiable models.
4.1 Key Observation
We introduce several key concepts. First, we define the soft value function 1 [T + 1, 1....
0z (-) = a log Ezo 1)
cxp
r(se)
|
where E (parc) [-] is induced by (-1x241) 12 This value function represents the expected future reward at from the intermediate noisy state t
Next, we define the following soft optimal policy (denoising process) p: : 2' - A(X) weighted by value functions ut-1 X x R:
p (1)) (xp(:))xxppp.)1111 I J.p (wl exp(et- (x))
(11a)
Here, are soft value functions and p: are soft optimal policies, because they literally correspond, respectively, to soft value functions and soft optimal policies, where we embed diffusion models into
catropy-regularin MDPs (Geist et al. 2019), demonstrated in Uchara al. (202cc).

---

With this preparation in mind, we utilize the following key observation:
Theorem 1 (From Theorem in Uehara et al. (2024c)). The distribution induced by ((s++1))==r "
the larget distribution pla) (x), i.e.
(ro) = {TTi-r+a pi (x)-(()))1)) dri:r.
While Uehara al. (2024c) presents this theorem, they use it primarily to interpret RL-based fine- tuning methods Fan al. ) Black et al. (2023). In contrast, our work explores how to convert this into tine-tuning--e free optimization algorithm.
Our motivation for a new algorithm. Theorem states that if we can hypothetically sample from [p,-0 [+++1)]=r> we can sample from the target distribution p(o). However, there are two t.a challenges in sampling from each Pt-1 *.a (I) the soft-value function 1 in Pi-1 is unknown, and (2)
is unnormalized (i.e., caleulating the normalizing constant is hard).
We address the first challenge in Section C
Assuming the first challenge is resolved, we consider how to tackle the second challenge. natural approach to use importance sampling (IS):
M
(m)
Pr (-
(m)) 1mm= ~ pi xx),
m=
j=1
where w,-1 = exp(1)-1(z1") (x; /))). Thus, we can approximately sample from p (- by
obtaining multiple (M) samples from pre-trained diffusion models and selecting the sample based on an index, which is determined by sampling from the categorical distribution with mean
(m) (w(m)/
E3 w wi-15m=1
()) (M)
Note that Best-of-N, which generates multiple samples and selects the highest reward sample, is technically considered IS, where proposal distribution the entire px** = II (x,
(zx-11
an) Jdr..-- However, the use of importance sampling in our algorithm differs significantly, as we apply it at each time step to approximate each soft-optimal policy.
Inference-Timee Algorithm
Algorithm SVDD (Soft Value-Based Decoding in Diffusion Models)
1: Require: Estimated soft value function (úr )- (refer to Algorithm or Algorithm 4 in Sec- tion C), pre-trained diffusion models (ppee t=r hyperparameter E IR, Batch size N IS size M.
2: for E [T' + 1,...... do
3: For each E [1.... NV, get M samples from pre-trained polices (1.)
{x( (k.m) JM ~ pp (1122
and for each m, calculate w = exp(6t- (B.m)) )a)
(i- after selecting an index: G-1 Catogorical 4: x + x1
up (t.m) AM uc m=1
a
5: end for
6: Output: Z0
Now, by leveraging the observation Algorithm I, we introduce our algorithm. Our algorithm an iterative sampling method that integrates soft value functions into the standard inference procedure of pre-trained diffusion models. Each step is designed to approximately sample from a value- weighted policy {p, ((
Finally, we note several key points.
In practice, we typically recommend low value for a, our focus is optimizing the reward.
Performance improves with increasing M, albeit at the cost of computational time. This re- flects a test-time scaling behavior, where improved performance achieved through increased computational resources. We present empirical results for varying values of M in Section 6
5

---

I can be computed in parallel at the cost of additional memory (scaled by M). If Line is not computed in parallel, the computational time in SVDD would be approximately M times that of the standard inference procedure. We will empirically demonstrate it in (Section 6
Line
A proposal distribution different from pp-1 in line I can be applied (sce Section F). For instance, classifier guidance or its variants may be used to obtain better proposal distributions than those
from the pure pre-trained model. For further details, refer to (Section H
4.3 Approximation of Soft Value Functions
A key remaining question is how to obtain the soft value function. Based on the definition of (1) we propose two general approaches: the Monte Carlo regression method and the posterior mean approximation method.
Monte Carlo approach. We approximate a value function via supervised learning based on
Posterior mean approximation approach. We use r()o(11)). where (o())) is predictor in pre-trained diffusion models from 2; to zip- This approach motivated by the observation that the expectation in (1) can be approximated using Dirac delta distribution centered at its mean. Similar approximations have been widely adopted in the context of classifier guidance, such as
DPS (Chung et al. 2022), universal guidance (Bansal et al. 2023) success.
with substantial empirical
Notably, the latter requires additional training and involves only a single forward pass through the pre-trained diffusion model; thus, we adopt it as our default choice. In practice, we observe that both approaches yield comparable performance. For more detailed discussion, see Secction C
4.4 Relation with SMC
SVDD differs from direct instantiation of standard SMC like Wu al. (2024), SVDD more closely resembles the nested SMC algorithm. This distinction between standard and nested SMC has also been recognized in the computational statisties literature (Naesseth et 2019 Algorithms and 13). Notably, while standard SMC requires computing the ratio of approximated value functions (i.e., exp(0;-1 //a) exp( ég (z))/a)), which often unstable in practice, our method avoids this step, leading to improved robustness. For more details, refer to Seection D
4.5 Potential Limitation
Our approach requires more computational resources (if not parallelized) or memory (if parallelized), approximately M times more than standard inference methods, noted in /eection 4.2 Taking this aspect into account, we compare SVDD, with baselines such as best-of-N in our experimental section (Section 6). For gradient-based approaches like classifier guidance and DPS, while direct comparison with SVDD is challenging. it is important to note that these methods also incur additional computational and memory complexity due the backward pass, which SVDD avoids.
A More Efficient SVDD via Replacement of Non-Promising Samples
As mentioned in Section 4.2 computational efficiency is crucial in our context, as the performance of inference-time techniques expected to scale with the amount of computation inference. Here, to further improve reward maximization in a computationally efficient manner, we introduce an enhanced variant of SVDD.
The core idea as follows. In Algorithm each sample I!(
1....,N) N) propa- gated from f = T to f. 0. However, during intermediate steps, it often becomes ap- parent that some samples are unlikely yield high rewards. In such cases, continuing to sample them is inefficient. Instead, propose reallocating computation by terminating un- promising samples early and redistributing resources to more promising ones within the same batch. This intuition is implemented through resampling step based on value functions, which serve as look-ahead estimators, as illustrated in Figure 2, The detailed procedure pre- sented in Algorithm 2] where resampling is guided by exponentially weighted value functions.

---

Notably, this step is computationally inexpen- sive, requires no additional forward passes through neural networks. In Section 6, we em- pirically evaluate the effectiveness this com- ponent.
Sample win 111
Evaluate with with.
mompling
Note that this algorithm is an improved ver- sion of Algorrihh I incorporating ideas from the standard SMC implementation discussed in Remark 4.4 In standard SMC, global resam- pling is also used select promising samples based. Similarly, our method introduces inter- action among samples within batch. However, unlike standard SMC, we first significantly improve each sample individually- the primary step of SVDD Algorithm T
Figure 2: Illustration of Algorithm 2 where M = 3 and N 3. After conducting weighted local sampling from pre-trained models, we perform global resampling step.
before performing global resampling.
Algorithm SVDD with Replacement of Non- Pon-Promising Samples
1: Require: Estimated soft value function { (refer to Algorithm 3 or Algorithm 4 pre- trained diffusion models [p] prp 10 JI hyperparameter
RR Batch size N, size
2: for t E [T + .... do
3: Obtain H; (.4 1) following Line in Algorithm and set as a:41
4: Global resampling: choose
(6221) NN 1 + 1-
explo(E 1/aa
Categorical
(2)
Σ, exp(((22))))
5: end for
6:
Output: 20
Experiments
We conduct experiments sensitivity to various hyperparameters. We start by outlining the experimental setup, and then present the results. The code is in /ecciin G
the performance of algorithm relative to baselines and its
6.1 Settings
Methods to compare. We compare SVDD to several representative methods capable of performing reward maximization during inference, discussed in [Section 2 Note for all methods relevant to a, generally the same set of a. For example, we use 0.01 for biological sequences and images and & = 0.05 for molecules.
Pre-trained models: We generate samples using pre-trained models.
We generate samples from pre-trained models and select the top 1/N samples. This selection is made ensure that the computational time during inference is approximately equivalent to that SVDD.
Best-of-N:
DPS (Chung et al., 2022): It is widely used training-fii version of classifier guidance. For discrete diffusion, we combine with the state-of-the-ar approach (Nisonoff al. 2024).
SMC-Based Methods (e.g., Wu et al. (2024)): Methods, which do not require differrntiable models, like SVDD. Further details are in Appendix D
SVDD (Ours): We implement Daneemeen//ggortthhee Algorithm I with the posterior mean approximation of soft-value functions. We generally set M 20 for images and M = 10 for other domains. Recall M the duplication size the IS part.
SVDD-R (Ours): We implement Algorithm which an improved version of SVDD with replacement of non- con-promising samples.

---

Table 1: Top 10 and 50 quantiles of the generated samples (512) in terms of rewards (with 95%
confidence intervals) and metrics quantifying naturalness (CLIP score and LL). SVDD consistently outperforms the baselines in terms of rewards, while reasonably maintaining naturalness. The top two methods are highlighted in bold.
Domain
Metric
Pre-Tmiin
Best-N
DPS
SMC
SVDD
SVDD-R
Compress
50% Quantile
-1014 a: 0.23
-71.2 a 0.46
-58.5 1: 0.39
-57.1 1:
-49.7 0.21
-234 1 0.05 61.8 4: 2.8
10% Quantile
-57.3
-36.5
CLIP scoce
27.5 4: 0.2
29.5
27.2 4: 0.0
28.3 0.1
28.9 4 0.1
Image: Acsthetic
50% Quantile
5.658 + 0.003
6.11 0,007
5.60 0.01
5.99 0.004
6.12 0.005
6.49 0.10
10% Quanile
4:
0.004
5,99
0.003
CLIP score
22
27.2
27.4
2.17:
29.0 2: 0.3
11.5 : 2.7
Molecule: QED
50% Quantile
0.566
0008
D835 + 0.90
0.679 0.024
0.667
0.016
0,848 + 0.014
0.924 0.008
10% Quantile
0.812 = 0.005
0,902 + 0,906
0,812 0.014
0.722 = 0.009
0,928 1: 0.008
0.961 =: 0.005
LL
-961 5:52
-944 + 28
-972 =
-1092 19
-1032
-548
Molecule:
50%
Quantile
0.007
0.925
0,992
10% Quantile
0.803 + 0.004
0,941 + 0.008
0.844 0.133
0.796 0.0015
1,000 + 0.I1D
1.000 0.000
LL
-970 1
-917 1 54
-916 =
-1053
-1040
-938 32
Molecule: Docking pappl
50%
Quantile
7.15
10.00
6.90
10%
Quantile
8.59 4: 0.31
10.67 0.10
9.31 0.26
9.37 0.36
12.41 0.13
13.98 :: 0.08
1.1.
.973 + 30
-954 +
-949 56
.92
-987 28
-643 33
Enhanoers
50%
Quantile
0.121 = 0.033
1.007 : 0.214
3.782 1 0.299
1.28 = 0.02
5,353 :: 0.231
6.782
=: 0.205
10% Quantile
1,396 0.020
3.449 1: 0,128
4,879 0.179
5,96 0.01
6,980 1: 0.138
8,020 0.167
-2598
0.7
-257.2
-287.1
-347.3 4.5
S'UTR
50% Quantile
0.406 2: 0.028
0,912 :: 0.023
0,426 1: 0.073
0.76 0.02
1.214 1: 0.016
1.549 1: 0.008
10% Quantile
0,869 = 0.017
1.064 + 0.014
0.981 0,044
0.00
1,383 + 0.010
1.688 0.006
.67.9 0.3
-6.22
-71.6 0.3
-68.7
-67.0 0.8
Datasets and reward models. We provide details on the pre-trained diffusion models and downstream reward functions used. For further information, refer to (Sectionn G
Images: We Stable Diffusion v1.5 as the pre-trained diffusion model (T = 50). For downstream reward functions, we use compressibility and aesthetic scores (LAION Aesthetic Predictor V2 in Schuhmann (2022)) ) employed by Black et al. (2023): Fan al. 2023). Compressibility is
non-differentiable reward feedback. feedback....
Molecules: We use GDSS (Jo et al. 2022). trained on ZINC-250K (Irwin & Shoichet 2005), as the pre-tr diffusion model (T 1000). For downstream reward functions, we use drug- likeness (QED) and synthetic accessibility (SA) calculated by RDKit, as well as docking score (DS) calculated by Quick 2 (Alhossary et al. 2015), which are all non-differentiable feedback. Here, we renormalize SA to (10 SA)/9 and docking score to max(-DS, 0), so that higher value indicates better performance. The docking scores measure binding affinity regarding four target proteins: Parp1, b,tlb, braf, and jak2 following Yang al. (2021). These tasks critical for drug discovery. We add results for further targets in Section
DNAs (enhancers) and RNAS (Stnaraanllttdd regions (UTRs)):: We use the discrete diffusion model (Sahoo al. 2024), trained on datasets from Gosai al. (2023) for enhancers, and from Sample et al. ) for 5'UTRS, as our pre-trained diffusion model (T' = 128). For the reward functions, we use an Enformer model (Avsec al. 2021 to predict activity of enhancers in the HepG2 cell line, and a Conv model that predicts the mean ribosomal load (MRL) of 5' S'URRs
measured by polysome profiling, respectively (Sample et al. 2019) These tasks are highly relevant for cell and RNA therapies, respectively (Taskiran et al. 2024 LastilloHHin & Seelig 2021
Metrics. We report the top 10% quantile and median of rewards from the generated designs, as these are the primary metrics in our context. As secondary metrics, we also report measures of the "naturalness" of the samples: CLIP score (Hessel et al. 2021) for images and (estimated) log-likelihood (LL) based on the ELBO of the pre-trained diffusion model in other domains. For additional metrics (e.g., diversity), see SSccioo H.I
6.2 Results
In this section, we first compare our methods with the baselines. Then, we present several ablation studies within our algorithm.
Firstly, we compare the baselines with our two proposed methods regarding rewards and naturalness (Table I.. Overall, SVDD outperforms the baseline methods (Best-of-N, DPS, and SMC) in terms
8

---

Pre trainedd
Pre-trained model
model
-103
-108
-96
-97
5.3
5.4
5.6
SVDD-FM
-29
-35
-37
-30
72
6.7
6.6
6.7
(a) Images: compressibility
(b) Images: aesthetic scores
Prwitaled model
gho n
12.4
12.5
12.3
13.6
12.1
11.3
12.4
SVDD-VC
(c) Molecules: QED scores
(d) Molecules (by SVDD): high (c) Molecules (by SVDD): high docking scores Jak2
docking scores to Parpl
Figure 3: Generated samples from SVDD. For more samples, refer to Section H.6 Note that the surfaces and ribbons in (e)-(f) (such as the green objects in (e)) are representations of the target proteins, while the generated molecules are displayed in the center.
1.0
GPI Tiress
1800 Mat PePeer
Meeryy
2680
150
(a)
Performance of (b) Performance of (c) GPU time and mem-(d) GPU time and SVDD M varies for SVDD M varies for ory SVDD as varies ory SVDD M varies for image generation while molecule generation image generation while opti- molecule generation while op- timizing the QED score.
optimizing aesthetic while optimizing the mizing aesthetic score.. score.
QED score.
Figure 4: Ablation studies with respect to M for SVDD. Figures (a) and (b) indicate that the performance increases M increases. Figure (c) and (d) indicate that the computational time does not scale linearly with M, whereas memory usage seales linearly.
of rewards, as evidenced by higher quantiles. Additionally, method maintains reasonably high naturalness metrics, which can also be visually verified in the images and molecular domains shown in Figure 3) This suggests that SVDD generate high-reward, natural-like samples that Best-of-N, DPS, and SMC often struggle to generate or, in some cases, nearly fail to generate.
Remark (Additional metrics). For performance on additional metrics, such as diversity, refer to Section H.I For more quantitative metrics that describe naturalness in molecules, refer to Section H.3
Ablation studies in terms of the duplication size M (test-time scaling). We assessed the perfor- mance of Swhenwhen calculating value functions in Line [ non-parallel manner) along with the computational and memory complexity as M varies. First, across all domains, the performance gradually improves as M increases (Figure 4a and 4b). Second, computational complexity increases linearly with M, while memory complexity remains nearly constant (Figure 4c|and 4d). This behavior
expected, as previously noted in Section 4.2 The comparison with Best-of-N in Table ] is made with this consideration in mind.
Ablation studies terms of a. We assess the performance of our algorithms by varying a in Figure 5 Here, expected, observe clear trend where reducing increases the rewards.

---

Note choosing a low a generally reduces diversity, as high- reward samples are more limited compared low-reward samples. However, even with low a, we find that diversity reasonably retained without mode collapse, as evidenced both visually in Figure 3 and quantitatively in Section H.1
Effectiveness of replacement of non-promising sam-- ples. Finally, we compare the performance of SVDD and SVDD-R (SVDD augmented with the global resam- pling module). As shown, SVDD-R consistently achieves higher reward performance across all evaluated tasks, These results demonstrate that SVDD-R, by replacing non- promising samples, more effectively maximizes rewards than SVDD, without incurring additional computational costs.
ataa
apha D.3
SVDC-
Figure 5: The performance in terms of rewards as varies for S'UTR design while optimizing MRL. For results in more domains, refer to Section H.5
Conclusion
We propose novel inference-time algorithm, SVDD, for optimizing downstream reward functions in pre-trained diffusion models that eliminate the need to construct differentiable proxy models. Future works include more applications, such as protein sequence optimization (Watson al. 2023) and 3D
molecule generation Xu et al. 2023).
Acknowledgement
We appreciate the feedback from Christian Naesseth. He pointed out the connection between our work and the twisted diffusion sampler, and noted our algorithm is close to an instantiation of nested-IS SMC in our context (Naesseth et al. 2019 2015).
References
Alford, R. F., Leaver- Fay, A., Jeliazkov, J. R., O'Meara, M. J., DiMaio, F. P., Park, H., Shapovalov, M. V., Renfrew, P. D.. Mulligan, V. K., Kappel, K., et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6): 3031-3048, 2017.
Alhossary, A., Handoko, S. D., Mu, Y., and Kwoh, C.-K. Fast, accurate, and reliable molecular docking with quickvina 2. Bioinformatics, 31(13):2214-22 2216, 2015.
Asrani, K. H., Farelli, J. D., Stahley, M. R., Miller, R. L., Cheng, J., Subramanian, R. R., and Brown, J. M. Optimization of mrna untranslated regions for improved expression of therapeutic mrna. RNA biology, 156))775--72, 2018.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete saaeess Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
Avdeyev, P., Shi, C., Tan, Y.. Dudnyk, K., and Zhou, J. Dirichlet diffusion score model for biological sequence generation. arXiv preprint arXiv:2305. 10699, 2023.
Avsec, Ż., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska- Barwinskw. A., Taylor, K. R., Assacl, Y., Jumper, J., Kohli, P., and Kelley, D. R. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 196- 2021.
Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein.
T. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 252, 2023.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, Training diffusion models with reinforce- ment learning. arXiv preprint 13000, 2023.
10

---

Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-282777 2022.
Camphell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997 2024.
Cardoso, G., Idrissi, Y. J. E.. Corff, S. L., and Moulines, E. Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308.0798.. 2023.
Castillo-Hair Hair, S. M. and Scelig, G. Machine learning for designing sett-enneaation mrna therapeutics. Accounts of Chemical Research, 55(1):24-34, 2021.
Cheng, F., Wang, Y., Bai, Y., Liang, Z., Mao, Q., Liu, D., Wu, X., and Xu, M. Research advances the stability of mrna vaccines. Viruses, 15(3);668, 2023.
Chorowski, J. and Jaitly, N. Towards better decoding and language model integration in sequence sequence models. arXiv preprint arXiv:
2016.
Chung, H., Kim, J., Mccann, M. T., Klasky, M. I.., and Ye, C. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint vrXiv:2209.16877 2022.
Clark, K., Vicol, P., Swersky, K., and Fleet, D. Directly fine-tuning diffusion models on differen- tiable rewards. arXiv preprint arXiv:2309 174000 2023.
Dathathri, S., Madotto, A., Lan, J., Hung. J., Frank, E.., Molino, P., Yosinski, J., and Liu, Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: /912. 164, 2019.
Dey, R. and Salem, F. M. Varianariants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp. 1597-1600. IEEE, 2017.
Dhariwal, and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang. J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
Dou, Z. and Song, Y. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. The Twelfth International Conference on Learning Representations, 2024.
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305.1638) 2023.
Ferreira DaSilva, L., Senan, S., Patel, Z. M., Reddy, A. Gabbita, S., Nussbaum, Z., Cordova, C. M. V. Wenteler, A., Weber, N., Tunjic, T. M., et al. Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. bioRxiv, pp. 2024-02, 2024.
Gainza, P., Sverrisson, F., Monti, F., Rodola, E., Boscaini, D., Bronstein, M. M., and Correia. B. E. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2): (((2):884418) 192, 2020.
Geist, M., Scherrer, B., and Pietquin, A theory of regularized markov decision processes. In International Conference Machine Learning, pp. 2160-2169. PMLR, 2019.
Ghiringhelli, L.. M., Vybiral, J., Levchenko, V., Draxl, C., and Scheffler, M. Big data of materials science: critical role of the descriptor. Physical review letters, 114(10):105503, 2015.

---

Gosai, S. J., Castro, R. I,, Fuentes, N., Butts, J. C., Kales, S., Noche, R. R., Mouri, K., Sabeti, P. C., Reilly, S. K., and Tewhey, R. Machine-gen gechine-guiiedd design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023.
Guo, Y.. Yuan, H., Yang, Y., Chen, M., and Wang, M. Gradient guidance for diffusion models: An optimization perspective. arXiv preprint arXiv:2404.1474. 2024.
Guo, Y., Yang, Y., Yuan, H., and Wang, M. Training- free guidance beyond differentiability: Scalable path steering with tree search in diffusion and flow models. arXiv preprint arXiv:2502.11420, 2025.
Han, S., Shenfeld, I., Srivastava, A., Kim, Y., and Agrawal, P. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.0mm 06639, 2024.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.,, and Choi, Y. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2104.000 08.00848888.. 2021.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12538 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems,
2022.
Inoue, F, Kreimer, A., Ashuach, T., Ahituv, N., and Yosef, N. Identification and massively parallel characterization of regulatory elements driving neural induction. Cell stem cell, 25(5):713-727, 2019.
Irwin, J. J. and Shoichet, B. K. ZINC- a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 4577):177-182, 2005.
Jin, W., Barzilay, R., and Jaakkola, T. Junetion tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling of graphs via the system of stochastic differential equations. International Conference on Machine Learning, pp. 10362- 10383. PMLR, 2022.
Kim, S.. Kim, M., and Park, D. Test-time alignment of diffusion models without reward over- optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Lal, A., Garfield, D., Biancalani, T., and Eraslan, G. regim: Designing realistic regulatory dna with autoregressive language models. bioRxiv, pp. 2024-02, 2024.
Landrum, G. et al. Rdkit: Open-source cheminformatics software, 2016. URL htp:://ww. rdkit, org/, http::/ggthhbb com/rakki/rdkii, 2016
Leblond, R., Alayrac, J.-B., Sifre, L., Pislar, M., Lespiau, J.-B., Antonoglou, I., Simonyan, K., and Vinyals, O. Machine translation decoding beyond beam search. arXiv preprint arXiv:2104.05330 2021.
Lee, S., Jo, J., and Hwang, S. J. Exploring chemical space with score-based out-s--distiibution generation. In International Conference on Machine Learning, pp. 18872-18892. PMLR, 2023.
Lew, A. K., Zhi-Xuan, T., Grand, G., and Mansinghka, V. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint arXiv:2306.0308/ 2023.
Lou, A., Meng, C., and Ermon, S. Diserete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310 10. 16834, 2023.
Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.C.,, Zhang, M., Yang. X., Li, Y., Jaakkola, T., Jia, X., et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732. 2025.
12

---

Mudgal, S., Lee, J., Ganapathy, H., Li, Y., Wang, T., Huang, Y., Chen, Z., Cheng, H.-T., Collins, M.. Strohman, T., et al. Controlled decoding from language models. arXiv preprint arXiv:2310.1702.. 2023.
Naesseth, C., Lindsten, F., and Schon, T. Nested sequential monte carlo methods. In International
Conference on Machine Learning, pp. 1292-1300 PMLR, 2015.
Naesseth, C. A., Lindsten, F., Schön, T. B., et al. Elements of sequential monte carlo. Foundations and Trends@ in Machine Learning, 12(3):307-392, 2019.
Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten. J. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2400.0//572, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Phillips, A., Dau, H.-D., Hutchinson, M. J., De Bortoli, V., Deligiannidis, G., and Doucet, A. Particle denoising diffusion sampler. arXiv preprint arXiv:2402.06320) 2024.
Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.037339 2023.
Qin, L., Welleck, S., Khashabi, D., and Choi, Y. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35: 9538-9551, 2022.
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. arXiv preprint 47X:542406.07524 2024.
Sample, J., Wang, B., Reid, D. W., Presnyak, V., McFadyen, J., Morris, D. R., and Seelig, G. Human 5' utr design and variant effect prediction from massively parallel translation assay. Nature biotechnology, 37(7)8003809, 2019.
Sarkar, A., Tang, Z., Zhao, C., and Koo, P. Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, pp. 2024-05, 2024.
Schiff, Y., Sahoo, S. S., Phung, H., Wang. G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A., Pierrot, T., and Kuleshov, V. Simple guidance mechanisms for discrete diffusion models. arXiv 2024.
preprint ar arXiv:2412.
Schuhmann, C. LAION aesthetics, Aug 2022.
URL Bttps://laion.al/bl
Laion- aesthetics/
Shi, J., Han, K., Wang, 7., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. arXiv preprint 2rXiv:2406.0432) 2024.
Singhal, R., Horvitz, Z., Teehan, R., Ren, M., Yu, Z., McKeown, K., and Ranganath, R. A gen- eral framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06844, 2025.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint 2xXi:2000000202. 2020a.
Song. Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020b.
Stark, H., Jing, B., Wang, C., Corso, G., Berger, B., Barzilay, R., and Jaakkola, Dirichlet flow 2024.
matching with applications to dna sequence design. arXiv preprint
13

---

Taskiran, I. L., Spanier, K. I., Dickmänken, H., Kempynck, N., Pančíková, A., Ekşi, E. C., Hulselmans, G., Ismail, N.. Theunis, K., Vandepoel, R., al. Cell-type-directed design synthetic enhancers. Nature, 626(7997):212-2230 2024.
Trippe, B. L., Yim, J., Tischer, D., Baker, D., Broderick, T., Barzilay, R., and Jaakkola, T. Diffusion probabilistic modeling of protein backbones in for the motif-scaffolding problem. arXiv preprint arXiv:2206.041 19, 2022.
Trott, O. and Olson, A. J. Autodock vina: improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455 461, 2010.
Uehara, M., Zhao, Y., Biancalani, T., and Levine, S. Understanding reinforcement learning-base baasdd fine-tuning of diffusion models: A tutorial and review. arXiv preprint arXiv:2407. 2024a.
Uchara, M., Zhao, Y., Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L., Tseng, A. M., Biancalani, T., and Levine, S. Fine-tuning of continuous-time diffusion models as entropy- regularized control. arXiv preprint urXiv:2402.15199 2024b.
Uchara, M.. Zhao, Y., Hajiramezanali, E., Scalia, G., Eraslan, G., Lal, A., Levine, S., and Biancalani, T. Bridging model-based optimization and generative modeling via conservative fine-tuning of diffusion models. arXiv preprint arXiv:2405. 00.405..9660 19673, 2024c.
van Westen, G. J., Swier, R. F., Cortes-Ciriand I., Wegner, J. K., Overington, J. P., IJzerman, A. P., van Vlijmen, H. W., and Bender, A. Benchmarking of protein descriptor sets proteochemo- metric modeling (part 2): modeling performance of 13 amino acid descriptor sets. Journal of cheminformatics, 5:1-20, 2013.
Venkatraman, S., Jain, M., Scimeca, L., Kim, M., Sendera, M., Hasan. M., Rowe, L., Mittal, S., Lemos, P., Bengio, E., et al. Amortizing intractable inference in diffusion models for vision, language, and control. arXiv preprint arXiv:2405.2097,, 2024.
Vignac, C., Krawezuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209. 2tt.:22999.4444... 2022.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffussion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-8228 2024.
Watson, L.,, Juergens, D., Bennett, N. R., Trippe, L.., Yim, J., Eisenach, E., Ahern, W., Borst, A. J.. Ragotte, R. J., Milles, I.. F., et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976): 1009-1100 2023.
Wu, L., Trippe, B., Naesseth, C., Blei, D., and Cunningham, J. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Wu, Y.. Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 10X.::66699.844400 2016.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? arXiv preprint 1800vv:8100000
2018.
Xu, M., Powers, A. S., Dror, O., Ermon, S., and Leskovee, J. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pp. 38592-38610. PMLR, 2023.
Yang, K. and Klein, D. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.052/88 2021.
Yang, S., Hwang, D., Lee, S., Ryu, S., and Hwang, S. J. Hit and lead discovery with explorative rl and fragment-base based molecule generation. Advances in Neural Information Processing Systems, 34: 7924-7936, 2021.
14

---

Yu, J., Wang, Y., Zhao, C., Ghanem, B., and Zhang, J. Freedom: Training- free energy energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 23174-23184 2023.
Zhao, S., Brekellans, R., Makhzani, A., and Grosse, R. Probabilistic inference in language models via twisted sequential monte carlo. arXiv preprint arXiv:2404. 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving. G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 2019.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, in the abstract and introduction accurately claim contributions and scope, and we support this with both empirical results and theoretical insights.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, we provide the limitations in Section 4.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
Yes, theoretical assumptions and proofs are clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide code through anonymous link as well as implementation details in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We provide code through anonymous link in Appendix G
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
public ceslic/guiiese CodesSimm ss onPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. w/nppsccppubbll wttppcccppppppptttetcccccccccccp cesii///uies///eeeeeeeeeoiesssee ies CodeeSumm ss cioooooliio ooooicyyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide implementation details in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Standard deviations across tasks and metrics provided in Section 6 and Appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details for hardware and computational requirements Ap-
pendix G.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httlil/tttt::::///ptpte http://c/..ooolooectccccccoooea.
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any new data or pre-train models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, citations included and existing licences discussed in Appendix G.3
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetatttt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: (https:// htt.eeeriips..coooeeeeeeeeeeeeee 2025/ LLM) for what should should not described.
22

---

A Broader Impact
This paper presents work whose goal is to advance the field of Deep Learning, particularly diffusion models. While this research primarily contributes to technical advancements in generative modeling, has potential implications in domains such as drug discovery and biomolecular engineering. We acknowledge that generative models, particularly those optimized for specific reward functions, could be misused if not carefully applied. However, our work is intended for general applications, and emphasize the importance of responsible deployment and alignment with ethical guidelines in generative AI. Overall, our contributions align with the broader goal of machine learning methodolo- gies, and do not foresee any immediate ethical concerns beyond those generally associated with generative models.
B Further Related Works
Decoding in autoregressive models with rewards. The decoding strategy, which dictates how sentences generated from the model, is critical component of text generation in autoregressive language models (Wu et al., [016 Chorowski & Jaitly [0166] Leblond et al.) 2021). Recent studies have explored inference-time time techniques for optimizing downstream reward functions Dathathri et al. (2019); Yang & Klein (2021) :: Qin et al. (2022); Mudgal et al. (2023); Zhao et al. (2024); Han et al. (0024 While there are similarities between these works and ours, to the best of know lloweegge, no prior work has extended such methodologies to diffusion models. Furthermore, our approach leverages characteristics unique diffusion models that are not present in autoregressive models such as SVDD-PM.
Discrete diffusion models. Based on seminal works Austin et al. (2021) 1; Campbell et al. (0022), recent work on masked diffusion models (Lou et al. 2023 Shi et al. 2024 Sahoo et al.) 2024 has demonstrated their strong performance in natural language generation. Additionally, they have been applied to biological sequence generation (e.g.. DNA, protein sequences in Campbell al. (2024): (0arka et al. (2024)). In these cases, the use of diffusion models over autoregressive models particularly apt, given that many biological sequences ultimately adopt complex ciree-dimensional structures. Despite its significance, it cannot be integrated with standard classifier guidance, because adding continuous gradient to a discrete objective is not inherently valid. Unlike standard classifier guidance, SVDD can be seamlessly applied to discrete diffusion models.
Fine-tuning of diffusion models. Several methods exist for fine-tuning generative models to optimize downstream reward functions, such as classifier-free guidance (Ho & Salimans, 2022) and
RL-based fine-tune fnn-tuning (Fan et al. 2023 Black et al. (023) or its variants (Dong al., | Wallace
al. 2024 Venkatraman et al. 2024). However, these approaches often come with caveats, including high computational costs and the risk of easily forgetting pre-trained models. In our work, we propose an inference-time technique that eliminates the need for fine-tuning generative models.
We acknowledge that if significant changes to the pre-trained models are desired, we acknowledge that RL-based fine-tuning [Black et al. 2023 Fan et al. 2023) could be more effective than SVDD for this purpose in certain scenarios, such image examples. However, this proximity to pre-trained models could also be advantageous in the sense that it is robust against reward optimization, which conventional tun-ttning methods often suffer from by exploiting these out-of distriiiiin regions (Uehara et al.] (002c).. Lastly, in cases where reward backpropagation (Prabhudesai et al.] [2023] Clark et al., 2023] Uehara et al. 2024b) not applicable, particularly in scientific domains for RL-based fine-tuning, we may need to rely on PPO. However, PPO is often mode-seeking and unstable, highlighting the challenges of RL-based fine-tuning certain scenarios.
C Approximation of Soft Value Functions
Next, we describe how to get soft value functions (x(x:) in practice. We propose two main approaches: a Monte Carlo regression and a posterior mean approximation approach.
C.1 Monte Carlo regression
23

---

Based on the definition of soft value functions in ①I. we can formulate it as regression problem, as described Algorithm 3 Combining this with Algorithm approach as SVDD-MC.
we refer the entire optimization
Algorithm 3 Value Function Estimation Using Monte Carlo Regression
1: Require: Pre-trained diffusion models, reward X + R, function class O X [0, + R.
1 2: Collect datasets {a1p ... "0 ) by rolling-out {py 10:
ler from to 3: &' argmin Ei E ( exp(f(x) .t)/a)}²
4: Output: 6/
C.2 Posterior Mean Approximation
Here, perform the following approximation:
(x) == ar log E /eae
z a (exp((exx(((()))) = r(fo(=1)),
where we replace To ~ ppre (zx) with the Dirac delta distribution with its estimated mean (o(xz) obtained during pre-training
Section 3.1 Here, recall that ( is the approximation of
Exg (zg=pe<o(rr)) 22). In this way, we can use (x)) as the estimated value function (Algorithm 4). The advantage of this approach is that additional training is required as long as we have When combined with Allgorithm I we refer to the entire approach as SVDD PM.. Note that this is our default choice.
is worthwhile to note in the context classifier guidance, essentially equivalent approximations have been used (e.g., DPS in Chung al. (2022). reconstruction guidance (Ho et al. (022). and universal guidance (Bansal et al. ). In our context, we similarly empirically demonstrate that even with this approximation, examine the empirical quality of this approximation in Section H.2
can effectively optimize the reward, shown in Section 6 We also
Algorithm Value Function Estimation using Posterior Mean Approximation
1: Require: Pre-trained diffusion models, reward r X + R
2: Set (',t) r(fo(r, = =), t)
3: Output: 6
Quality of value function approximation in practice. We visualize the performance of two algorithms for value function estimation (MC and PM, as described in Algorithmm 3 and [l Algorithm 4) in Figure 6 We make two observations. First, the performance (measured by Pearson's 's correlation) generally similar, regardless of whether we use MC or PM. Second, the approximation quality improves as the time step in approaches 0. This is expected, as the sample becomes progressively less noisy. For results more domains, refer Section H.2
C.3 Comparison Performance Between SYDD-MC and SVDD-PM
The relative performance of our SVDD-MC and SVDD-PM appears to be domain-dependent. Gen- erally, SVDD-PM may be more robust since it does not require additional learning (i.e., it directly utilizes reward feedback). The performance of SVDD-MC depends on the success of value function learning. Regarding its quality, refer to Section H.2
D Difference Between SVDD and "Standard" SMC-Methods
In this section, we compare our algorithm with the SMC-based methods (Wu et al. 2024 Trippe et al. 2022 Dou & Song 2024 Phillips et al. 2024 Cardoso et al. 2023) for guidance. While they originally aim to solve conditioning problem, which different from reward maximization. their algorithms can be converted to reward maximization. First, will explain this converted algorithm.
24

---

Pearson's 0.381
Pearson's S
Pearson's 0.717
16-
14-
Estinated Roward
Estimated Reward
Estiaaed Resard
3.-
2.3
3.3
12-
13-
1.2
13
0.8
1.0
1.2
1.4
0.8
Espected Reward
Expected
Expected
(a) PM when 96
(b) PM when t 64
(c) PM when t 16
Pearson's 0.461
Pearson's 0.197
Pearson's r 0.005
1.2
Estintedd hewardd
Estimateed Roward
1.3
Eotmaaed beawed
1.3
0.5
0.-
0.9
0.25
0.50
0.75
1.00
125
1.50
0.25
0.50
0.75
1.00
125
1.50
0.5
0.50
3.75
1.00
3.5
1.50
Expected Rwward
Expectidd Roward
Expected Remord
(d) MC when t = 96
(e) MC when = 64
(f) MC when f 16
Figure 6: The performance of PM and MC for value function approximations at each time step (Y-axis: predicted, X-axis: actual) for 5' 5'UTR design is shown. Here, the total time step size 128. As the time step approaches O, the performance improves.
Table 2: Comparison between SVDD-MC and SVDD-PM f indicates higher values correspond to better performance while indicates lower for better.
Image Compresibility Compressibilitty
Image Aestheiic Score Acsthetic
Enhuncer HepG2
S'UTR MRI.
Method
Qualityl
Qualityl
HepG21
NLLJ
MRL1
NLL
SVDD-PM
0.21
6.12 0.01
29.0
5,353
-6588 -67.0 0.8
SVDD-MC
56.3 + 0.29
11.4 7.4
6.49 + 0.10
11.5 2.7
6.782 + 0.208
-249.6 6 + 3.4
1.549 0.006
Molecule QED QED
Molecule SA
Molecule Docking Purpl Docking Score
Method
NLL.
NLL.
NLL.
SVDD-PM
0.848 0.014
1032 + 22
0,925 +: 0.016
1040 + 27
11.40 = 0.22
-987 :: 28 -643 35
SVDD-MC
0.924 0.008
-548
0.992 1 0.003
-938 32
13.61 0.20
We then elaborate on the differences between SVDD, and them the context reward maximization. Notably, our SVDD is an instantiation of nested-IS SMC (Naesseth et al. [209 Algorithm 5) in the literature on computational statistics, whereas these SMC-based Methods rely on standard sequential Monte Carlo.
D.1 SMC-Based Methods
The complete algorithm of TDS in our setting summarized in Algorithm 5 their notations are slightly different, we first provide a brief overview. It consists of two steps. Since
Since our notation and
algorithm is iterative, at time point I, consider have N samples (particles) {xr)") [4 Ji=1
IS step (line 5 We generate a set of samples {-] 1N { following policy from a pre-trained
model p (FP). In other words,
Vi E [1, M p. (-1a2(1).
Now, we denote the importance weight for the next particle xT-1 given the current particle F, as w(2;-1,2;). 2:), expressed
exp(v.-1 (xt-1)//
exp(vi-1 (Ix- 1)(x)
w 10 =
exp(et=1 1) pp. ( ( (x)d
exp(v)(xe)/om
25

---

Algorithm 5 Guidance with "Standard" SMC (for reward maximization)
1: Require: Estimated value functions (+r(z)) pre-trained diffusion models {p: Tt= hyper- parameter a E R, Batch size N
2: for E [ 1, do
3: IS step:
4:
exp(e,_ (zi exp( (z. 3/a)
19 [L. NN ~ p (-1x,)) -
5: Selection step: select new indices with replacement
N
6: (-
|N - 12
(s--11 IN ~ Cat
Σ
i=1
7: end for
8: Output: 20
and define
Vi E [1, [1,.... N]; - == wxx
Note here we have used the soft Bellman equation:
exp(ve(x))/a)
Hence, by denoting the target marginal distribution at have the following approximation:
[
tar
1S
1=1
1=1
W
Selection step (line 5) determined by the following:
Finally, we consider resampling step. The resampling indices
N
(c
(5-11111
L.
Cat
N
[7]
Wi-1
1=1
To summarize, conduct
N
[
P
H 4--1
M
18
i=1
w
Resampling
i=1
Finally, we give several important remarks.
In SMC, resampling is performed across the entire batch. However, in the algorithm. sampling is done within a single batch. Therefore, the algorithms differ significantly. We will discuss the implications in the next section.
All of existing works Wu al, (2024 Cardoso al. (2023 ; Phillips al. (2024) Dou Song 2024) actually consider a scenario where the reward classifier. In Algorithm 5
we tailor the algorithm for reward maximization. Vice verisa, our SVDD can also operate
effectively when is classifier.
In Wu et al. 2024);; Cardoso al. 2023 Phillips et al. (2024), the proposal distribution is not limited to the pre-trained model. Likewise, in our SVDD, we can select an arbitrary proposal distribution, discussed in /ecciin
In the context autoregressive (language) models, Zhha et al. (2024) Lew al. 2023
proposed similar algorithm.
26

---

D.2 Comparison of SVDD with "Standard" SMC-Based Methods (SSM) for Reward
Maximization
We now compare our SVDD with "standard" SMC-Based Methods (SSM). Here, we write a batch size of SVDD in G. Importantly, we note that our implementation analogous nested-IS SMC in the literature in computational statistics; hence, many differences between rested-IS SMC (Naesseth
al. 2019 2015) and pure SMC computational statistics are translated here.
We first reconsider the fundamental assumptions of each algorithm. SVDD's performance, in terms of rewards, depends the size of M but is independent of the batch size G. In contrast, the performance of SSM depends on the batch size N. With this in mind, we compare the advantages of SVDD over SSM from various perspectives.
In SSM, the "ratio" is approximated. In SVDD, approximate each exp(tt-1(7-11//a) as weight. However, in standard SMC, the ratio is approximated as weight:
exp(0t-((x-11/a))
exp(v((ze)/a)
The key difference is that in SSM, both the numerator and the denominator are approximated, which could lead to greater error propagation.
Ease of parallelization SVDD. SVDD significantly easier to parallelize across multiple nodes. In contrast, SSM requires interaction between nodes for parallelization. This advantage is also well-documented in the context of nested-IS SMC versus standard SMC (Naesseth et al., 2019).
Relation between Blocked SMC and SVDD. When using SMC (M K) and SVDD (N M = K) with a = 0, the two algorithms become theoretically equivalent. However, in practice, we avoid setting a preserve diversity among samples. As a result, the two algorithms exhibit different behaviors.
E Comparison against DG (Nisonoff et al., 2024) and DiGress (Vignac et al. 2022) in Discrete Diffusion Models
Our method is closely related to guidance methods used in DG (Nisonoff et al.) 2024) and DiGress (Vignac et al. 2022). However, we emphasize that our approach is more general, as it operates on any domain, including continuous spaces including Riemannian spaces and discrete spaces, in unified manner. In this sense, strict comparison is not feasible. With this in mind, we provide a comparison focusing on cases where all domains discrete.
Comparison with Nisonoff et al. (024). In this continuous framework, following the notation in (Lou al. 2023). they propose the use of the following rate matrix:
Qiyy (t) (t) exp(vx(u)/a)
exp(vxxx/aa)
where (1) is rate matrix in the pre-trained model. This suggests that, with standard discretization, the optimal policy at each time step is:
plae+5t yla x) = I(x + y) + (t)
(3)
where öt is step size. Asymptotically, this is equivalent to sampling from the optimal policy in Theorem as we will show in Remark 2] However, as Nisonoff al. (2024) note, sampling from the optimal policy requires O(KL, L) computation, where K is the vocabulary size and L is the sequence length, which is computationally expensive for large K and L. To address this issue, they propose using Taylor approximation by computing the gradient once. However, this is a heuristic in the sense that there is no theoretical guarantee for this approximation. In contrast, we avoid this computational overhead in different manner, i.e., through importance sampling and resampling. Our algorithm has an asymptotic guarantee as M goes to infinity. Empirically, we have compared two methods in Section 6
27

---

Remark 2 (Asymptotic equivalence between formula in Nisonoff et al. (2024) and Theorem The informal reasoning is as follows. Recall that the pre-trained policy can be written as
p(I++5t = u/ - x) - 1(x + u) +
Then, Theorem
states that the optimal policy is
[T(x + v) (000)))))))))))
Now, have
(Tx + +) O1
1(x # v) + (xx](00)))))xxxxxx( x + O()t)}
exp(vx(x)/aa)
[I(x u) + ( (xp(
(t)( cxp(u()))xxpp))))) I(x # y) exp(vi(0)/o)
(xp(v((x)/o)
Thus, this recovers the the formula
Comparison with DiGress Vignac et al. (2022). Vignac et al. (2022) proposed a diffusion model for graphs where each sampling and denoising step operates directly on the discrete structure, avoiding continuous relaxation. They discuss how to implement guidance by treating rewards as a classifier. To bypass the exponential computational cost of sampling from the optimal policy (p(a) in Theorem 1). they employ Taylor expansion. While it requires the calculation of gradients for value functions, then it mitigates the exponential blow-up in computational time. In contrast, we avoid this computational blow-up through importance sampling comparison between our method and theirs is left for future work.
(IS) and resampling. A detailed empirical
Remark 3, Note that in our molecule generation experiment in Section 6 we use GDSS (Jo et al. 2022), which operates in continuous space and differs from DiGress.
F Extension with Arbitrary Proposal Distribution
Here, we describe the algorithm where the proposal distribution is not necessarily derived from the policy of the pre-trained model, summarized in AlAlloiithmm Essentially, only adjust the importance weight. In practice, we can use the gradient of differentiable proxy model, such as DPS, as the proposal distribution qt-1- Even the differentiable proxy (value function) models are not highly accurate, our method will still perform effectively since other value function models ê can be non-differentiable.
Algorithm 6 SVDD (Soft Value-Basee Decoding in Diffusion Models)
Require: Estimated soft value function { (refer to Algorithm or Algorithm 4), pre- trained diffusion models {pf 1pp }o hyperparameter E R, proposal distribution { J
2: for t E [T' + 1,... 1] do
3: Get M samples from pre-trained polices { Jmmi 11
9t-1 (--1(=)22). and for each and
calculate
pre
(m) (xr)
(m) wim = exp 0:- (x (m))
(m)
a (x [x
(G--) after selecting index: G--1 Cat 4: Pt-1 x-11
En w mm
5: end for
6: Output: 200
28

---

G Additional Experimental Details
We further add additional experimental details. To ensure the reproducibility of this work, we provide details for the experiments in this section. We also provide anonymous code link containing the implementation of our method and baselines: https:://annnnnet 4open. science///
SVDD/]
G.1
Additional Setups for Experiments
G.1.1
Settings
Images. We define compressibility score as the negative file size in kilobytes (kb) of the image after JPEG compression following (Black et al. (023). We define aesthetic scorer implemented as a linear MLP on top of the CLIP embeddings, which is trained on more than 400k human evaluations. As pre-trained models, we use Stable Diffusion, which is a common text-to-image diffusion model. As prompts to condition, we use animal prompts following (Black et al. 2023) such as [Dog, Cat, Panda, Rabbit, Horse....]
Molecules. We calculate QED and SA scores using the RDKit (Landrum et al. 2016) library. We use the docking program Quick Vina 2 (Alhossary et al. 2015) to compute the docking scores follow- ing Yang et al. (202T), with exhaustiveness values, while we reverse it to be positive and then clip the values to be above 0, i.e. We compute DS regarding four proteins, parpl (Poly [ADP-ribose] polymerase-l) 1), 5ht1b 33-hydroxyrryptamine receptor IB), braf hetpee//trrenineprrtteeee kinase B-raf), and jak2 (Tyrosine-proite kinase JAK2), which are target proteins that have the highest AUROC scores of protein-ligand binding affinities for DUD-E ligands approximated with AutoDock Vina.
I. Note that the docking scores are initially negative
DNA, RNA sequences. We examine two publicly available large datasets: enhancers (n = 700k) (Gosai et al.) (023) and UTRs (n Rs 300k) (Sample et al. 2019). with activity levels measured by
massively parallel reporter assays (MPRA) (Inoue et al. 2019). These datasets have been widely
used for sequence optimization in DNA and RNA engineering, particularly in advancing cell and RNA therapies (Castillo-Haan & Scelig 2021 Lal et al. al. 2024c). In the Enhancers dataset, each is DNA sequence of length 200, while y R is the measured activity in the Hep cell line. In the S'UTRs dataset, x is a 5'UTR RNA sequence of length 50, and y R is the mean ribosomal load (MRL) measured by polysome profiling.
2024 Ferreira DaSilva et al. 2024 Uchara
G.1.2 Baselines and Proposals
We will explain in more detail how to implement baselines and our proposal. We use A100 GPUs for all the tasks.
SVDD-MC. In SVDD-MC, we require value function models. For images, we use standard CNNs for this purpose, with the same architecture as the reward model. For molecular tasks, we use a Graph Isomorphism Network (GIN) model (Xu et al. 2018) as the value function model. Notably, this model is not differentiable w.r.t. inputs. For GIN, we use mean global pooling and the RELÚ activation function, and the dimension of the hidden layer is 300. The number of convolutional layers in the GIN model is selected from the set {3. 5): and we select the maximum number of iterations from (300, 500, 1000). the initial learning rate from ( (1e-3, 3e-3, 5e-3, le-4), and the batch size from (32, 64, 128). For the Enhancer task, we use the Enformer model (Avsec et al. 2021) as the value function model. The Enformer trunk has 7 convolutional layers, each having 1536 channels. as well as 11 transformer layers, with 8 attention heads and a key length of 64. Dropout regularization is applied across the attention mechanism, with an attention dropout rate of 0.05, positional dropout of 0.01, and feedforward dropout of 0.4. The convolutional head for final prediction has ** 1536 input channels and uses average pooling, without an activation function. The model trained using a batch size selected from {32, 64, 128, 256}. the learning rate from [1e-4, 5e-4, le-3), and the maximum number of iterations from {5k, 10k, 20k). For the 5'UTR task, we adopt the ConvGRU model (Dey & Salem] 2017). The ConvGRU trunk has a stem input with 4 channels and a convolutional stem that outputs 64 channels using a kernel size of 15. The model contains 6 convolutional layers, each initialized with 64 channels and kernel size of 5. The convolutional layers use ReLU as the
29

---

activation function, and residual connection is applied across layers. Batch normalization is applied both the convolutional and GRU layers. A single GRU layer with dropout of 0.1 added after the convolutional layers. The convolutional head for final prediction uses 64 input channels and average pooling. without batch normalization. For training, the batch size selected from {16, 32, 64, 128], the learning rate from {1e-4, 2e-4, 5e-4), and the maximum number of iterations from (2k, 5k, 10k). All function models are trained to converge in the learning process using MSE loss.
SVDD-PM (default version of SVDD). For this proposal, we directly use the reward feedback to evaluate. We remark when the reward feedback is also learned from offine data, technically, would be better to use techniques mitigating over-optimization discussed in Uchara al. (2024c). However, since this point is tangential in our work, we don't do it.
DPS. We require differentiable models. For this task, for images, enhancers, and 5'UTRs, we use the same method SVDD-ME MC. For molecules, follow the implementation in Lee et al. (2023), and we use the same GNN model as the reward model. Note that we cannot compute derivatives with respect to adjacency matrices when using the GNN model.
SMC. For value function models, we use the same method as SVDD-PM.
SVDD-R. In 22. we discard suboptimal samples of a certain percentage and resample from high- quality samples We sample by probability of exponential tiling. The percentage of samples replaced a hyperparameter selected from {0.03, 0.04, 0.05] in (2)
G.2 Software and Hardware
Our implementation is under the architecture of PyTorch (Paszke al.) 2019). The deployment environments are Ubuntu 20.04 with 48 Intel(R) Xeon(R) Silver, 4214R CPU @ 2.40GHz, 755GB RAM, and graphics cards NVIDIA RTX 2080Ti. Each our experiments is conducted on a single NVIDIA RTX 2080Ti or RTX A6000 GPU.
G.3 Licenses
The pretrained models and datasets for image tasks are under MIT license tasks is under Database Contents License (DbCL) v1.0, The dataset for DNA task is covered under AGPL-3.0 license The dataset for RNA tasks is under GPL-3.0 license We follow the regulations
The dataset for molecular
for all licenses.
H Additional Experimental Results
H.1 Analysis in terms of Diversity
We measure the target reward as well as naturalness and diversity metrics for comprehensive evalua- tion. A higher diversity score indicates greater variability in generated samples, ensuring broader exploration of the data space.
For images, we measure the diversity using cosine similarities of CLIP embeddings.
For biological sequences, we measure diversity using the pairwise cosine distance one-hot encoded representation subtracted by 1 to capture structural variations.
For molecular diversity, employ the Tanimoto similarity molecular Morgan fin- gerprints (ECFP), where diversity quantified as the average pairwise similarity among generated molecules, subtracted by 1. This captures structural diversity by assessing the uniqueness of molecular substructures.
We report the diversity performances of different methods in Table on page 31 As shown, baselines such as SMC, which rely on resampling strategies, show marked drop in diversity, reinforcing the theoretical analyses. In contrast, SVDD exhibits balance, ensuring reasonable level of diversity while significantly enhancing reward.
30

---

Table 3: Diversity performance of different methods. Higher values correspond to better performance.
Task
Pre-Trained
Best-N
DPS
SMC
SVDD-MC
SVDD-PM
SVDD-R
Images: compressibility
0.30
0.33
0.34
0.12
0.36
0.31
0.17
Images: aesthetics
0.34
0.30
0.28
0.11
0.28
0.32
0.20
Molecule: QED
0.874
0.880
0.899
0.252
0.868
0,860
0.616
Molecule: SA
0.840
0.845
0.892
0,689
0.866
0,860
0,446
Molecule: Docking Parp1
0.780
0.795
0.888
0.510
0.902
0,895
0.674
DNA Enhancers HepG2
0.722
0.710
0.748
0.579
0.724
0.715
0.570
5'UTR MRL
0.740
0.730
0.745
0.556
0.638
0.634
0.523
H.2 Analysis Approximation Quality in Value Function Estimation
To show the quality of our value functions, we evaluate the effectiveness of our proposed value estimation methods for predicting the final reward of diffusion -eeeiooed samples at intermediate time steps. We assess the accuracy of our intermediate state value predictions by computing the Pearson correlation coefficient between the estimated reward and the actual reward obtained at zo- We visualize this relationship using seatter density plots across several time steps. For sampled trajectories, estimate rewards various intermediate diffusion steps and compare them against the ground-trun truth reward from the oracle. Higher values of Pearson correlation indicate better alignment between the estimated and actual rewards.
Here are our observations.
Figures and 8 show the results using the posterior mean approximation (PM). During early diffusion steps, the estimated rewards show weaker correlation with the final reward, which is expected. later diffusion steps the correlation improves noticeably, and at late diffusion steps, the estimated rewards achieve high correlation with the final reward of nearly perfect, confirming that as the diffusion progresses, our value estimation method accurately predicts the final reward.
Figures and TO show the results using the Monte Carlo regression (MC). The correlation trends are generally similar with PM, while the MC estimation is more stable, especially in mid diffusion steps. The density of points along the red diagonal line suggests that the estimated values are well-calibrated. The correlation supports the effectiveness of using these value functions for intermediate state selection.
Time Pesmor's = 0.000
Pesmoons e = 0,000
Poomton's = 0.140
Pearson's 033
Dilnated Fowent
Time Stop 116
Time Stos 120
Time Step 124
"The Stop
Capacted hahadd
Expeciec Samard
Lxpested Roward
Oppcted
Figure 7: Scatter density plots between estimated reward (PM) and ground truth reward for DNA Enhancer task.
31

---

Step Nearsarss 0.351
Plersttiss 0.577
Peanuat's 0.71
16.
13
1.0
04
1.0
0.3
03
133
10
Time Step 336
Time Step 120
"ime Step 124
Time Step 127
Pearsons
Poarsarrs
1.7
10
11
supected rovard
oupacted
ixpected Rouard
Figure 8: Scatter density plots between estimated reward (PM) and ground truth reward for S'UTR MRL task.
Time Step 37
Time S-ep 64 Pramsrr'r 0.155
Time Step R8 Prorsor'st
Time Step 122 Teanon's
Pesmor's
5.0
Camsedd foreet
Calraeed
6.7
Pipested REANEE
expected
supacted
Paseciad sear Nanaa's 0.b0
Nearsorrs C.S0
Paanaraa 0.431
Pourao 0.122
5.4
40
Figure 9: Scatter density plots between estimated reward (MC) and ground truth reward for DNA
Enhancer task.
H.3 Validity Metrics for Molecule Generation
To evaluate the validity of our method in molecule generation, report several key metrics that capture different aspects of molecule quality and diversity in Table 4 on page 32
Table 4: Comparison of the generated molecules across various metrics. The best values for each
metric are highlighted in bold.
Method
Valid
Unique?
Novelly?
FCD
SNN
Frag
Scaf 1
NSPDK MMD |
Mol Stable
Alm Stable 1
Pre trained
1.000
I,000
1.000
12.979
0.414
0.513
1.000
0
0.320
0.917
DPS
1.000
1.000
1.000
13.230
0,389
0.388
1.000
0.040
0.310
0.878
SMC
1.000
0.406
1.000
22.710
0.225
0.068
1.000
0.285
0.000
0.968
SVDD-PM
1.000
12.278
0.428
0.052
SVDD-MC MC.
1.000
1,000
1.000
14.765
0.349
0.478
1,0000
0.063
0.375
0.932
32

---

Step
Step
Step
Nearorrrs C.197
Plersctiss 0.051
Poerso 0.522
15-
15-
1--
06-
1.23 020 0.1 LAS Lo Li00
C2> 633 WMS
100 120
-30
1.0
153
6.5
140 125
150
154 0.0
1.3 8/7 103 125
Rowels
Time Step 336
Time Step 120
"ime Step 124
"ime Step 128
Pearsons
Poarsaris 0.657
Peuras 0.607
Nearson's
09-
655 oupacted Revidd
0.55
8.55 380 158
supected ravard
-speced Rovard
Figure 10: Scatter density plots between estimated reward (MC) and ground truth reward for 5UUTR MRL task.
The validity of a molecule indicates its adherence to chemical rules, defined by whether it can be successfully converted to SMILES strings by RDKit. Uniqueness refers the proportion of generated molecules that are distinct by SMILES string. Novelty measures the percentage of the generated molecules that are not present in the training set. Fréchet ChemNet Distance (FCD) measures the similarity between the generated molecules and the test set. The Similarity to Nearest Neighbors (SNN) metric evaluates how similar the generated molecules are to their nearest neighbors in the test set. Fragment similarity measures the similarity of molecular fragments between generated molecules and the test set. Scaffold similarity assesses the resemblance of the molecular scaffolds in the generated set to those in the test set. The neighborhood subgraph pairwise distance kernel Maximum Mean Discrepancy (NSPDK MMD) quantifies the difference in the distribution of graph substructures between generated molecules and the test set considering node and edge features. Atom stability measures the percentage of atoms with correct bond valencies. Molecule stability measures the fraction of generated molecules that are chemically stable, i.e., whose all atoms have correct bond valencies. Specifically, atom and molecule stability are calculated using conformers generated by RDKit and optimized with UFF (Universal Force Field) and MMFF (Merck Molecular Force Field).
We compare the metrics using 512 molecules generated from the pre-trained GDSS model and from different methods optimizing QED, as shown in Table 4 on page 32] Overall, our method achieves comparable performances with the pre-trained model on all metrics, maintaining high validity, novelty, and uniqueness while outperforming on several metrics such as FCD, SNN, fragment similarity and molecule stability. Pre-trained performs consistently well across all metrics, particularly in SNN and atomic stability. However, it does not optimize specific molecular properties as effectively as the other methods. SMC performs poorly in fragment similarity, NSPDK MMD, and molccular stability, indicating that it generates unrealistic molecules. These results indicate that our approach can generate a diverse set of novel molccules that are chemically plausible and relevant.
H.4 Further Ablation Studies in terms of M
We provide several more ablation studies regarding M on top of the results in the main text, plotted The results are consistent with what we have observed in Figure 4]
in Figure II]
H.S Further Ablation Studies in terms of a
The hyperprameter a is a crucial hyperparameter that controls the balance between exploitation and exploration in state selection based on value estimation. A smaller a biases the selection towards higher-reward samples, whereas a larger a allows for broader exploration of potential high-reward candidates. In the extreme case of a O, only the highest-reward trajectory is selected
33

---

GPU Time vs M
Max Memory M
1600
10000
-00
1400
Max Memory (MBB
8000
1200
Roward
-100
1000
6000
800
-150
GPU
600
4000
2000
200
-200
0
20
40
60
80
20
40
60
80
Pre-trained
M
(a) Performance of SVDD M varies.
GPU time and memory of SVDD as M varies.
Figure 11: Abeltation Studies (for image generation while optimizing the compressibility)
24
CIOSEN
(a) DPS
(b) SMC
(c) SVDD-MC
(d) SVDD-PM
Figure 12: Reward performances different methods as a varies for Enhancer design while optimiz- ing cell- cell-specifiiity.
-22
DPC
WCh 3J
40No XLL
DPS
(b) SMC
SVDD-MC
SVDD-PM
Figure 13: Reward performances of different methods varies for S'UTR design while optimizing MRI.
deterministically. We analyze the effect of varying across multiple tasks. We conduct experiments with multiple a values, ranging from more explorative setting (a 0.2) to more exploitative regime ( (a 0.01 for biological sequences and 0.05 for molecules). Note that we study relatively small value cases, since our major target is reward optimization. Results are shown in Figures
We observe that reducing leads increase in the mean reward for all methods. This is expected, as smaller a prioritizes states with high estimated rewards. However, some variances the reward distributions also increase, particularly for DPS and SVDD-PM, indicating that extreme exploitation results in more volatile sample quality. SMC struggles to maintain performance across all a values, indicating that its value estimation less robust in this setting. Notably, SVDD-PM achieves relatively high rewards while maintaining consistency, highlighting the feature of a more balanced value function.
H.6 Further Visualization of Generated Samples
We provide additional generated samples this section. Figure 16 and 17]show comparisons of generated images from baseline methods and SVDD with different M values regarding compressibility and aesthetic score, respectively. Figure and Figure 19 present the comparisons of visualized molecules generated from the baseline model and SVDD regarding QED and SA, respectively. The visualizations validate the strong performances of SVDD. Given that SVDD can achieve optimal SA for many molecules, also visualize some molecules with optimal SA generated by SVDD, as
34

---

gea
65
040
64
DPS
(b) SMC
SVDD-MC
SVDD-PM
Figure 14: Reward performances of different methods as a varies for molecule design while optimiz- ing QED.
41402
0F7
CRC
500
ALA
(a) DPS
(b) SMC
(c) SVDD-MC
(d) SVDD-PM
Figure 15: Reward performances different methods as a varies for molecule design while optimiz- ing SA.
visualizes the docking of SVDD-generated molecular ligands to proteins parpl. 5ht jak2, and braf, respectively. Docking scores presented above each column quantify the binding affinity of the ligand-prote interaction, while the figures include various representations and perspectives of the ligand-protein complexes.
shown in Figure 20 In Figure 21
Figure 22
Figure 23 and Figure 24
We aim to provide a complete picture of how each ligand is situated within both the local binding environment and the larger structural framework the protein. First rows show close-up views of the ligand bound to the protein surface, displaying the topography and electrostatic properties of the protein's 's binding pocket and providing insight into the complementarity between the ligand and the pocket's surface. The second rows display distant views of the protein using the surface representation, offering a broader perspective on the ligand's s spatial orientation within the global protein structure. The third rows provide close-up views of the ligand interaction using ribbon diagram, which represents the protein's 's secondary structure, such as alpha-helices and beta-sheets, to highlight the specific regions of the protein involved in binding. The fourth rows show distant views of the entire protein structure in the ribbon diagram, with ligands displayed within the context of the protein's 's full tertiary structure. Ligands generally fit snugly within the protein pocket, evidenced by the close-up views in both the surface and ribbon diagrams, which show minimal steric clashes and strong surface complementarity.
35

---

Monkey
Panda
Rabbit
Cat
Dog
Horse
Pre-trained
-103
-112
-155
-97
-103
-130
Best-of-N
-79
-74
-81
-61
-68
-73
SVDD-PM
(M=20)
-37
-39
-20
-39
-31
-43
SVDD-PM
(M=80)
-32
-26
-16
-22
-27
-29
Figure 16: Additional generated samples (Domain: images, Reward: Compressibility)
Monkey
Panda
Rabbit
Cat
Dog
Horse
Pre-trained
5.7
5.5
5.5
5.6
5.3
5.6
Best-of-N
6.1
6.0
6.1
5.8
5.7
5.9
SVDD-PM
(M=20)
6.5
6.4
6.5
6.6
6.9
6.5
SVDD-PM
(M=80)
6.7
6.4
7.0
7.3
7.1
6.7
Figure 17: Additional generated samples (Domain: Images, Reward: Aesthetic score)
36

---

mag mnoo botun
Pre-trained model
SVDD-MC
Figure 18: Additional generated samples (Domain: Molecules, Reward: QED score)
Pre-trnined model
ASU
frim tomo Ieme
Figure 19: Additional generated samples (Domain: Molecules, Reward: SA score, normalized as
(10 SA)/9)
37

---

Figure 20: Additional generated samples from SVDD-MC (Domain: Molecules, Reward: SA score 1.0 (normalized as (10 SA)/9))
11.5
11.3
12.4
12,5
12.3
13.6
13.1
12.3
Figure 21: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score
parpl (normalized mar(- (mar(-D33 DS, 0)))
Figure 22: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score
5 (normalized as max( DDS, 0)))
38

---

11.4
Figure 23: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score
jak2 (normalized as max max(-DS. (DDS, 0)))
Figure 24: Additional generated samples from SVDD (Domain: Molecules, Reward: Docking score
braf (normalized as max( DD, 0)))
39


Reviews:
[{"review_id": "pebTnBsWoK", "reviewer": "Reviewer_Kr5s", "summary": "The paper introduces a method called SVDD to address the challenges of current diffusion models on optimizing downstream reward functions while preserving the naturalness of the design spaces. SVDD does so by integrating value functions to enable a look-ahead into the intermediate noisy states and the rewards that can be achieved. Experiments on different domains show improved performance over the baselines.", "questions": "- How many seeds have been used when generating the results and comparisons with the baselines? \n- The LL metrics for the proposed method and the baselines seems pretty close. Is there a reason for the proposed method not bringing too much benefits as compared to the baselines?\n- How is the quality quality of the learnt value function evaluated and how is its dependence on the number of sample trajectories needed for the training analyzed?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The work targets an important problem, especially related to sampling in the cases on non-differentiable rewards which can be common in scientific discovery scenarios.\n- The paper overall is written well along with a good motivation of the method.\n\nWeaknesses:\n- The method relies on learning a good value function, which can be difficult to do properly in general. A discussion on this in the manuscript would be useful.\n- The additional cost of sampling good trajectories to learn such a value function would be useful to analyze. Even though the method avoids the cost of expensive fine-tuning, it is not very clear how much is the additional cost of sampling trajectories and learning a value function is.\n- The method seems an extension of SMC, and in terms of that, the novelty of the method is not too clear.", "was_revised": true, "final_justification": "Update:\nI would like to thank the authors for their rebuttal which has been useful for me to understand their work better. I have also looked at the reviews and discussions from other reviewers that has also clarified a bunch of my concerns. Since the authors have clarified most of my concerns, I am updating my scores accordingly.", "rebuttal": "Thank you for the thoughtful review! The reviewer primarily raised two points: (1) our method appears to rely on learned value functions, and (2) the analysis may be insufficient. \n\nRegarding point (1), we would like to clarify that **our proposed method does not rely on a learned value function**. Instead, we focus on a learning-free variant, which serves as the default throughout our main experiments. For point (2), we provide a detailed analysis of the learning-free approach (SVDD-PM) in **Appendix C and Appendix H.2, including its empirical performance**. In this rebuttal, we also offer **additional details regarding its sample and computational efficiency**. We would be happy to elaborate further! \n\n> W1: Reliance on Learning Value Function \n\nWhile we agree that learning a good value function could be challenging sometimes, in our method, importantly, we propose a posterior mean approximation (Section 4.3), **which avoids learning entirely, and use this approach as a default version in our experimental section, Section 6**. In contrast, learned value function is merely an alternative ablation we study in Appendix C and Appendix H.2, and not the primary proposal of this paper.\n\n* More specifically, our posterior mean estimation approach approximates the expected reward from a noisy input using a single forward pass of the pre-trained diffusion model—an idea inspired by DPS and universal guidance. This method avoids the sample inefficiency with learned value functions, while remaining effective empirically, as demonstrated in Section 6.\n\n* We further show that this training-free approach, when combined with SVDD, performs comparably to the Monte Carlo variant that requires a learned value function (Appendix C.3, Table 2). Additionally, we show that the posterior mean approximation itself serves as a strong value function estimator (Appendix C.3, Figure 6, and Section H.2), achieving competitive performance in practice.\n\n> W2: Cost of Sampling/ Learning Value Functions + Q3: Evaluation of Value Function Quality and Sample Efficiency\n\nWe appreciate the reviewer’s suggestion (but we would like to reiterate—as noted in our response to W1—that our default method is SVDD with a posterior mean approach, which **does not require learning value functions**). \n\nBelow, we address your specific concerns:\n\n### (1) Value function quality \n\n**We discussed the quality of value function approximations in detail in Appendix H.2**. Several plots are provided to illustrate its performance. As shown there, **the Pearson correlation between predicted and true rewards is relatively high**. Notably, as the diffusion process progresses, our value estimation method increasingly aligns with the final reward, indicating strong predictive accuracy.\n\n### (2) Sample Efficiency \n\nWe provide additional details regarding computational and sample efficiency in the table below for clarity.\n\n| Task| Num of samples | Runtime|Fine-tuning diffusion model runtime (with DDPO)|\n|-|-|-|-|\n| Enhancer HepG2|704 * 128 | ~25min|~3.8h|\n| Molecule QED| 576 * 1000| ~2.5h|~46h|\n| 5’UTR MRL|736 * 128 | ~20min|--|\n\n* The reported runtime includes the total time for sampling trajectories, value function learning, and evaluation\n* For comparison, we also experimented with fine-tuning diffusion models using reward signals via DDPO (Black et al., 2023), and we report the runtime to convergence across several tasks. Compared to fine-tuning the diffusion model, this computational cost remains negligible.\n\nWe will incorporate these details into the final version of the paper. Due to the rebuttal format restrictions, we are unable to include learning curve plots of the value function experiments here, but we will ensure they are included in the final camera-ready version.\n\n> W3: Novelty Compared to SMC\n\nWe understand that this point may be somewhat confusing, but we have explicitly emphasized the distinction in our paper—particularly in Section 2 (Lines 102–107). \n\n*While there are concurrent efforts exploring reward-guided generation (Kim et al., 2025, etc), our approach differs in key ways. Most notably, (1) our algorithm is an instantiation of **nested importance sampling (nested-IS) SMC** (Naesseth et al., 2019, Algorithm 5, 105 13), whereas other concurrent works typically employ standard sequential Monte Carlo.  (2) Additionally, we introduce **a novel technique for reward maximization in Section 5** that integrates ideas from both SMC and nested-IS SMC organically.*\n\nThe algorithmic distinction between our approach and standard SMC-based methods is also highlighted in Section 4.4. We would be happy to elaborate further if it is still unclear. \n\n> Q1: Number of Seeds\n\nThank you for the comment. While the appropriate methodology may vary depending on the domain, we would like to clarify that the confidence intervals reported in our tables are constructed with this in mind. For example, in the image domain, we generate 300 samples using different random seeds to compute these intervals.\n\n> Q2: Close LL Scores Across Methods\n\nThank you for the observation. This result is in fact expected. We would like to clarify that the objective of SVDD, as described in Line 170, is to generate samples with high reward while preserving naturalness. Importantly, the log-likelihood (LL) serves as a proxy for naturalness, but it is not the primary optimization objective. Given this, the observed result aligns well with the intended behavior of our method."}, {"review_id": "oAoNrmtdJz", "reviewer": "Reviewer_kjpL", "summary": "This work presents an algorithm called Soft Value-based Decoding in Diffusion Models (SVDD), which is a conditioned generation method for diffusion model which does not requires a differentiable classifier for guidance. The key idea is to define a \"soft value function\" (Eq. 1), and which re-weights the sampling probability of Monte-Carlo generation of diffusion models. Furthermore, the paper proposes SVDD-R, a more efficient sampling method (similar to beam search). The proposed algorithms have been empirically evaluated on image, molecular and DNA generation tasks. Ablation studies have also been performed. Overall, the current work suggests an effective method for diffusion model's generation from non-differentiable feedback signals.", "questions": "1. Can you train a guidance model, which computes an additional gradient $\\nabla(\\exp(r(x)/\\alpha)))$ like that in a classifier-free guidance? While the reward function may not be differentiable, this guidance model could be trained using policy gradient. This may decrease inference cost in case large amount of generation is needed.\n\n2. In Table 1, why SVDD-R sometimes showed a much higher CLIP score (Image:Compress) whereas sometimes a much lower CLIP score (Image:Aesthetic) than SVDD?\n\n3. Also in Table 1, why SVDD-R can show a much higher log-likelihood even than the pretrained model (Molecule: QED, Docking parp1)?\n\n4. Before RL was used on diffusion models, the concept of soft value and related studies have been fruitful. What is the relationship between the paper's soft value and those proposed in classic RL literature such as [1,2,3] ?\n\n[1] Haarnoja T, Tang H, Abbeel P, et al. Reinforcement learning with deep energy-based policies[C]//International conference on machine learning. PMLR, 2017: 1352-1361.\n[2] Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International conference on machine learning. Pmlr, 2018: 1861-1870.\n[3] Levine S. Reinforcement learning and control as probabilistic inference: Tutorial and review[J]. arXiv preprint arXiv:1805.00909, 2018.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengthes**\n\n1. The paper aims at solving derivative-free guidance for diffusion models, which is a practical and important challenge in AI applications.\n2. Comprehensive empirical evaluations are presented in the main texts and the appendix, showing the effectiveness of SVDD and its features.\n3. The idea of using soft value function is well-motivated.\n\n\n**Weaknesses**\n\n1. SVDD requires multiple evaluations during inference, which limits its potential to leverage human feedback, since asking humans to provide reward signals for so many times during generation is not practical.\n\n2. The manuscript structure is a bit top-heavy. Space is too much occupied by methods, and many results are extruded to the Appendix. Some key results in the main texts are not sufficiently explained (see my questions). Conclusion is too short.\n\n3. The choice of $\\alpha$ is crucial, as the paper mentioned -- when $\\alpha$ is small, the \"soft\" value becomes \"hard\" value, and lead to a higher reward, while may loss diversity. How to select a proper $\\alpha$ in principle remains unsovled.", "was_revised": true, "final_justification": "My questions are well addressed by the authors. I think that the this work, after incorporating the suggestions from other reviewers and me, would be a good contribution to the venue.", "rebuttal": "Thank you for your thoughtful and positive feedback!\n\n> W1: Inference Cost Limits Use of Human Feedback\n\nWe appreciate the reviewer raising this important point. While SVDD does require multiple reward evaluations during inference, we would like to emphasize the following:\n\n* In human-in-the-loop settings, **it is possible to leverage surrogate reward functions trained on offline data. In such cases, the reward may be differentiable; however, as shown in Experiment 6, our method outperforms SOTA approaches that directly optimize differentiable rewards, such as DPS**.\n\n* When differentiable proxy models are available, SVDD can be seamlessly integrated with gradient-based approaches, as shown in Algorithm 6 in Appendix F. \n\n* Importantly, SVDD remains effective even with a small number of candidate samples. As demonstrated in Figure 4, using just $M = 5$ already yields substantial improvements in reward.\n\nWe will clarify these points further in the revised version in our conclusion section.\n\n* Our SVDD-R variant reduces repeated evaluations by promoting top candidates using global resampling, improving efficiency.\n\nIn the revised version, we will expand the discussion on how SVDD can be adapted for human-in-the-loop reward optimization, especially via bootstrapped human preferences, offline reward modeling, and hybrid schemes where SVDD is only used in early iterations and replaced by faster sampling later.\n\n> W2: Manuscript is Top-Heavy; Results in Appendix\n\nThank you for the suggestion. We acknowledge this structural issue and sincerely appreciate the reviewer’s feedback. Due to space constraints, we prioritized theoretical exposition and algorithmic clarity in the main text. However, we agree that some experimental results warrant more prominent placement. To address this, we plan to revise the paper as follows:\n\n* We will move key ablations and plots from Appendices D/E/F into the main paper, including results on value approximation vs. reward correlation, the effect of importance sample size $M$, and value function quality.\n\n* We will expand the Conclusion section to highlight better practical implications, methodological trade-offs, and future directions.\n\nWe believe these changes will improve the balance between theory and empirical results and enhance the overall accessibility of the paper.\n\n> W3: Selection of $\\alpha$\n\nIndeed, the parameter $\\alpha$ governs the reward–diversity trade-off, analogous to the temperature in softmax policies or the guidance scale in classifier-free guidance. While we empirically investigate its effect in Figure 5, we agree that developing a principled method for selecting $\\alpha$ remains an open question—particularly because the optimal value is often highly context-dependent.\n\nAs a practical guideline (which we will include in the Discussion section), we propose the following strategy:\n\n*  Offline settings: Choose $\\alpha$ to maximize a weighted combination of reward and diversity (e.g., entropy, coverage metrics.... specific metrics are context-dependent) on a small validation set.\n\n* Active learning setting: $\\alpha$ can be annealed during inference to balance exploration and exploitation. \n\nWe will highlight this insight and include a discussion in Section 7.\n\n> Q1: Can you train a guidance model, which computes an additional gradient like that in a classifier-free guidance? While the reward function may not be differentiable, this guidance model could be trained using policy gradient. \n\nThank you for the suggestion. We assume you are referring to reinforcement learning (RL)-based fine-tuning. Indeed, Black et al. (DDPO) introduced such an approach, which has been shown to outperform classifier-free guidance enve. While we greatly appreciate these contributions, our work focuses on inference-time techniques (without fine-tuning), which are also highly relevant and practical in real-world applications. We will clarify this distinction in the revised manuscript. Please feel free to let us know if we have misunderstood your intention—we would be happy to elaborate further.\n\n> Q2: SVDD-R CLIP Score Discrepancy in Image Tasks\n\nThat's an insightful observation. We conjecture that SVDD-R tends to favor high-reward outliers, which can either increase or decrease auxiliary metrics such as CLIP scores, depending on the correlation between the optimized reward and the auxiliary metric. We will clarify this phenomenon in the discussion accompanying Table 1. \n\n\n> Q3: SVDD-R Shows Higher Log-Likelihood than Pretrained Model\n\nThis effect may be attributed to the resampling dynamics. Since SVDD-R prioritizes trajectories with both high reward and high model confidence (i.e., high log-likelihood), the resulting resampled batch can exhibit a higher average log-likelihood than samples drawn directly from the pretrained model. We emphasize that this does not violate the generative nature of the model; rather, it reflects a biased selection within the model’s existing sample space. We will clarify this explanation in the revised manuscript, specifically under Table 1.\n\n> Q4: Comparison with related RL papers\n\nThat’s a great point. This indeed corresponds to soft value functions in the classical RL literature, particularly when embedding diffusion models into entropy-regularized MDPs. This connection has been noted in several related works—for example, see Section 3 of the reference below, which discusses this concept in the context of fine-tuning diffusion models via reward signals.\n\n(However, it is worth noting that while the connection is discussed, these soft value functions are not directly used in the algorithms proposed in that work. Hence, our proposed algorithm is novel. )\n\nReference:\nUehara, M., Zhao, Y., Biancalani, T., & Levine, S. (2024). Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review. arXiv preprint arXiv:2407.13734."}, {"review_id": "zFIvImEm3u", "reviewer": "Reviewer_ETLn", "summary": "This paper introduces SVDD, an iterative sampling method using soft value functions for diffusion models. Without the need of fine-tuning and differentiable model, SVDD enables direct use of non-differentiable feedback, applicable to both continuous and discrete diffusion models. Approaches based on classifier guidance require differentiable proxy models, and cannot utilize non-differentiable features which limits its usage. To address this issue, SVDD leverage soft value functions which works as look-ahead functions that gives information of how intermediate samples lead to rewards in the future. SVDD selects the intermediate states with the highest reward, resulting in desired sample. Experiments on image, molecule, and DNA/RNA generation shows that SVDD outperforms previous baselines such as DPS and SMC-based methods.", "questions": "Please address the questions in the weakness section.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n- Writing is easy to follow with clear motivation and problem setup.\n\n- Optimizing the reward without need of fine-tuning and enabling non-differentiable feedback is advantageous for diverse field and realistic settings. Previous approaches based on classifier guidance cannot use non-differentiable features.\n\n- To the best of my knowledge, integrating soft value function for reward optimization of diffusion models is a novel approach.\n\n- Validation on several domains shows the wide applicability of the proposed method, supporting both continuous and discrete diffusion models.\n\n- Conducted through ablation studies, including study on the duplication size M for performance and time, and study on hyperparameter alpha in the appendix.\n\n\nWeakness\n\n- While the experimental setup is sound using SD 1.5 for image generation and GDSS for molecule generation, using larger and more recent models, such as SDXL for image generation and DiGress, a discrete diffusion model for molecule generation, could strengthen the experimental results. I want to clarify that these are not required experiments, but a suggestion.\n\n- How is the generation time of SVDD compared to the baselines, DPS and Best-of-N? It seems that look ahead with soft value function would require additional inference time compared to the baselines. Also, how much additional memory is required compared to the baselines?", "was_revised": true, "final_justification": "Authors have adequately addressed my concerns and thereby I recommend acceptance.", "rebuttal": "Thank you for your thoughtful and positive feedback!\n\n> w1. Use of Larger, More Recent Models (e.g., SDXL, DiGress)\n\nWe appreciate the suggestion. Due to the limited timeframe of the review period, we were not able to include these additional results. However, we selected Stable Diffusion and GDSS as baselines because they are well-established models widely used in the community. In particular, Stable Diffusion has been frequently used in prior work on fine-tuning diffusion models with reward signals, including recent studies such as Clark et al. (ICLR 2024).\n\nReference:\nClark, K., Vicol, P., Swersky, K., & Fleet, D. J. Directly fine-tuning diffusion models on differentiable rewards. ICLR, 2024.\n\n> w2. Generation Time and Memory Overhead vs DPS and Best-of-N\n\nThis is a valuable question. Here is an answer.  \n\n* The total computational cost scales approximately linearly with the importance sample size $M$. For instance, using $M = 4$ incurs roughly 4× the cost of a single sampling step—comparable to Best-of-4—yet SVDD achieves significantly higher rewards.\n\n* As shown in Figure 4 (Section 6.2), we vary the value of $M$ and analyze the trade-off between computational cost and reward performance.\n\n* Technically, the additional cost can manifest as either increased runtime or memory usage, depending on the implementation. For example, **if samples are generated sequentially in a loop, the runtime increases linearly with $M$ while memory usage remains nearly constant,** as shown in our paper. Alternatively, if samples are generated in parallel, **memory usage may increase with $M$, while runtime remains roughly constant**.\n\nWe further include a table summarizing wall-clock time and GPU memory for SVDD, DPS, and Best-of-N under comparable settings, where we run a loop in code implementation for all methods.\n\n| Task| Image Aesthetic(M=20)| Enhancer HepG2(M=10)| 5’UTR MRL(M=10)|\n|-|-|-|-|\n|Best of N|200s 9878M|27.56s 2102M|10.68s 1872M|\n|SVDD|203s 12355M|27.78s 2173M|13.27s 1967M|\n|DPS|38s 35086M|8.19s 8076M|7.77s 8068M|\n\nNote that we set $N=M$ in Best-of-N, and DPS is much **slower** than regular sampling by comparing \"Best of M time/M\".  We will incorporate it in our final version."}, {"review_id": "nSrQM6duoA", "reviewer": "Reviewer_eWG2", "summary": "The authors propose a novel algorithm for sampling from pretrained diffusion models while optimising for a riven downstream reward function. The proposed method relies on first training a value function, which assigns values not only to final samples but also to noise samples. It then uses BoN sampling at each diffusion step, picking the sample with the highest value under the learned value function. The authors claim that their method significantly improves on relevant image generation and molecule synthesis benchmarks.", "questions": "See weaknesses", "limitations": "no", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- developing significantly better optimisation methods for broadly applicable models (like pretrained diffusion models) is of broad utility\n- the proposed method is simple and easy to understand\n- the paper writing is clear\n\nWeaknesses:\n- \"eliminates the need to construct differentiable models\" -- while this is true, the proposed method itself relies on training an additional value function. Why is this much better? e.g. using derivatives of proxy models should be more sample efficient?\n- how much more compute intensive is the BoN sampling? Could we e.g. annotate this for each row in Table 1?\n- Do the Table 1 results somehow account for reward hacking? shouldn't we also evaluate the distance to the original model, to see which methods achieve the highest reward while having the smallest distance tot he original model?\n- no limitations addressed\n- Why does SVVD-R often outperform SVDD? shouldn't it be similar? (In Table 1)\n- Comparison to prior work does not seem sufficient, given that the paper rarely is based on improved empirical performance. E.g., no reward hacking is addressed in Table 1 (and not discussed?), the comparisons in figure 3 are only with the pretrained model and do not take other baselines into account (apart from the BoN baseline in the appendix?)\n\nMinor things/ Comments:\n- \"naturalness of these design spaces\" -- would be great if this could be expressed a bit more formally (if possible)", "was_revised": true, "final_justification": "authors addressed concerns largely.", "rebuttal": "We sincerely appreciate the reviewer’s time and thoughtful feedback. The reviewer primarily raised two key concerns: (1) why we don't use differentiable proxy models, and (2) the potential risk of reward hacking.\n\nRegarding point (1), we note that (a) constructing accurate differentiable proxies is often non-trivial in many practical scenarios, and (b) **even in such cases, our method demonstrates greater reward optimization performance** compared to gradient-based methods such as DPS. Furthermore, as shown in Algorithm 6 (Appendix H), **our method (SVDD) can be seamlessly integrated with gradient-based techniques when suitable proxies are available**. \n\nRegarding point (2), we provide multiple pieces of empirical evidence in the paper indicating that **our method is robust to reward hacking (Likelihood, Clip scores as shown in Section 6 already, and additional evidence with LaVa model)**. \n\nWe would be happy to offer further clarification if needed.\n\n> W1The proposed method itself relies on training an additional value function.\n\nIn our method, importantly, we propose a posterior mean approximation (Section 4.3), **which avoids learning entirely, and use this approach as a default version in our experimental section, Section 6**. In contrast, learned value function is merely an alternative ablation we study in Appendix C and Appendix H.2, and not the primary proposal of this paper.\n\n* More specifically, our posterior mean estimation approach approximates the expected reward from a noisy input using a single forward pass of the pre-trained diffusion model—an idea inspired by DPS and universal guidance. This method avoids the sample inefficiency with learned value functions, while remaining effective empirically, as demonstrated in Section 6.\n\n* We further show that this training-free approach, when combined with SVDD, performs comparably to the Monte Carlo variant that requires a learned value function (Appendix C.3, Table 2). Additionally, we show that the posterior mean approximation itself serves as a strong value function estimator (Appendix C.3, Figure 6, and Section H.2), achieving competitive performance in practice.\n\n> W1: Why is SVDD better than Using Differentiable Proxy Models?\n\nThank you for raising this critical point. Here is our response. \n\n* While we acknowledge that in some scenarios it is feasible to construct accurate differentiable proxy models—and doing so may lead to sample-efficient optimization—we would like to emphasize that **many real-world rewards are difficult to model with accurate differentiable proxies**, for the following reasons:\n   * **Non-differentiable domain-specific features**: Some rewards are based on discrete or dictionary-based features (e.g., molecular or protein descriptors, or outputs from tools like AlphaFold3), which are inherently non-differentiable.\n   * **Black-box feedback**: Many important objectives in molecular and protein design rely on black-box simulations, such as physics-based tools like Vina or Rosetta, where gradients are unavailable but the reward signals (e.g., docking score, folding stability) are crucial.\n\n* Even in settings where differentiable rewards are easily available—such as “Enhancers” and “5′ UTR”—**our method still outperforms gradient-based approaches like DPS**, as shown empirically in Table 1 in Section 6.\n\n* **Moreover, SVDD can be seamlessly combined with differentiable proxy models when they are available**, even if they are potentially suboptimal. As described in **Algorithm 6 (Appendix F)**, SVDD can incorporate classifier guidance to construct proposal distributions, allowing it to leverage differentiable signals in place of a pretrained diffusion model. \n\n> W2: Compute Overhead of Best-of-N Sampling – Can We Annotate It in Table 1?\n\nIn general, we set up experiments where the computational complexity for querying diffusion models is the same. More specifically, with this regard, the computational complexity is O(N’M) (Batch-size: N’, M: search width) while in Best-of-Ns computational complexity is O(N). We set N’M=N in Table 1.   \n\nWe further include a table summarizing **wall-clock time and GPU memory** for SVDD and Best-of-N under comparable settings, where we run a loop in code implementation for all methods. \n\n| Task| Image Aesthetic(M=20)| Enhancer HepG2(M=10)| 5’UTR MRL(M=10)|\n|-|-|-|-|\n|Best of N|200s 9878M|27.56s 2102M|10.68s 1872M|\n|SVDD|203s 12355M|27.78s 2173M|13.27s 1967M|\n\nWe will expend Table 1 to annotate the computation complexity in the paper final revision.\n\n> W3: Reward Hacking and Evaluation of Distance to Original Model\n\nThank you for highlighting this important point. While we do not explicitly use the term “reward hacking” in the paper, we would like to emphasize that **we have included multiple empirical and conceptual evidence demonstrating that our method is roubst to reward hacking** in our submission. We have also conducted **additional experiments** to further support this claim.\n\n### Empirical Evidence 1:\n\nIn Table 1, we report the **log-likelihood (LL)** of generated samples under the pretrained model. **LL serves as a proxy for how well the samples align with the model’s original distribution (distance)** —a natural indicator of whether the generation has drifted too far (i.e., potential reward hacking). **We observe that the LL of SVDD is comparable to that of Best-of-N, suggesting that SVDD maintains high fidelity to the pretrained model's distribution**.\n\n### Empirical Evidence 2:\n\nIn the image domain, we present both **visualizations and CLIP scores** to assess alignment with prompts in Section 6. As seen in our results (Figure 3 in our paper)—and in contrast to prior representative work such as DRaFT [Deng et al., 2024]—there is no visual indication of reward hacking. In DRaFT, **when reward hacking occurs, the deviation is typically apparent upon visual inspection** (Figure 3 in the DRaFT paper).\n\n### Empirical Evidence 3 (**Additional Analysis**):\n\nTo more systematically assess semantic fidelity, we conducted a new experiment using LLaVA-1.5-7B, an image-to-text model. Given an animal prompt (e.g., “cheetah”), we evaluate **the object mention accuracy**—the proportion of generated images where the model correctly identifies the target prompt. This directly checks whether the reward optimization process sacrifices semantic consistency, which would be indicative of reward hacking.  Note that this experiment is inspired by two related works (DDPO in Black et al., 2023, and BRAID), which aim to optimize rewards in pre-trained image diffusion models.  \n\n The results show that SVDD maintains high object mention accuracy, further supporting its robustness against reward hacking. We note that achieving **100% accuracy is non-trivial**—in fact, in scenarios where models are fine-tuned using DDPO or DRaFT, reward hacking can occur and lead to near 0% accuracy in the end, **as illustrated in Figure 3 of the DRaFT paper and Figure 3 of the BRAID paper**.\n\n​​| Object Mention Accuracy(%) in Aesthetic|\n|-|-|\n| Best-of-N|100\n| DPS|100\n| SMC|100\n| SVDD|100\n| SVDD-R|100\n\n### Conceptual Evidence:\n\nBy design, SVDD samples from a search tree rooted in the pretrained model, ensuring that generation remains close to the original distribution. This constraint acts as a natural safeguard against reward hacking, as it prevents excessive divergence from the pretrained model's behavior.\n\nWe will incorporate these clarifications and the additional experiment in the final version of the manuscript.\n\nReference:\n\nDRaFT: https://openreview.net/pdf?id=1vmSEVL19f (ICLR 2024)\n\nBRAID: https://proceedings.neurips.cc/paper_files/paper/2024/file/e68274fc4f158dbcbd4dddc672f7ee9c-Paper-Conference.pdf  (NeurIPS 2024)\n\n> W4 no limitations addressed\n\nThank you for pointing this out. We would like to clarify that the limitations of our approach are discussed in Section 4.5. If the reviewer was referring to additional concerns not addressed there, we would be happy to elaborate further and incorporate any necessary clarifications in the revised manuscript.\n\n> W5: Why Does SVDD-R Often Outperform SVDD?\n\nThis is an insightful question. While SVDD-R is based on SVDD, its resampling strategy can amplify high-reward trajectories more aggressively, which leads to higher average rewards, particularly when:\n*  The value approximation is imperfect, and resampling acts as a second-stage correction;\n* Sampling budget is limited, and SVDD-R focuses computation on promising samples.\n\n> W6: The comparisons in Figure 3 are only with the pretrained model and do not take other baselines into account \n\nThank you for your suggestion. In Figure 3, we did not include results from all proposal methods, primarily due to space constraints. Additionally, for tasks such as generating aesthetic images or high-QED molecules, it is often difficult to evaluate quality purely through visual inspection. That said, we will include visualizations for all remaining baselines in the revised version of the paper! Due to the conference's strict rebuttal guidelines, we are unable to provide additional figures at this stage, but we will ensure they are included in the final submission. (If there are other concerns regarding further comparisons to prior work, we would be happy to provide additional information. ) \n\n> Minor Comments “Naturalness of These Design Spaces” – Could Be More Formal\n\nThank you for your suggestion. We agree that the term \"naturalness\" may sound informal, as its meaning can be context-dependent (e.g., natural images, natural chemical space, etc.). Mathematically, we assume that the pre-trained diffusion model captures this notion. We will clarify this point further in the revised version."}]

Author rebuttal:

Meta-review: This work presents an algorithm called Soft Value-based Decoding in Diffusion Models (SVDD), which is a conditioned generation method for diffusion model that does not requires a differentiable classifier for guidance. The work is targeting an important problem, especially when considering non-differentiable reward functions and will be of interest to the NeurIPS audience. All of the reviewers concerns were adequately addressed during the rebuttal stage.
Decision: Accept (poster)