A General Framework for Inference-time Scaling and Steering of Diffusion Models
Authors: Raghav Singhal
Zachary Horvitz
Ryan Teehan
Mengye Ren
Zhou Yu
Kathleen McKeown
Rajesh Ranganath
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
steering
fine-tuning
particle-based sampling
sequential Monte Carlo
Primary area: probabilistic_methods->monte_carlo_and_sampling_methods

Abstract: Diffusion models have demonstrated remarkable performance in generative modeling, but generating samples with specific desiderata remains challenging. Existing solutions --- such as fine-tuning,  best-of-n sampling, and gradient-based guidance --- are expensive, inefficient, or limited in applicability. In this work, we propose FK steering, a framework for inference-time steering diffusion models with reward functions. In this work, we introduce FK steering, which applies Feynman-Kac interacting particle systems to the inference-time steering of diffusion models with arbitrary reward functions. FK steering works by generating multiple trajectories, called particles, and resampling particles at intermediate steps based on scores computed using functions called potentials. Potentials are defined using rewards for intermediate states and are chosen such that a high score indicates the particle will yield a high-reward sample. We explore various choices of potentials, rewards, and samplers. Steering text-to-image models with a human preference reward, we find that FK steering outperforms fine-tuned models with just 2 particles. Moreover, FK steering a 0.8B parameter model outperforms a 2.6B model, achieving state-of-the-art performance on prompt fidelity. We also steer text diffusion models with rewards for text quality and rare attributes such as toxicity, and find that FK steering generates lower perplexity text and enables gradient-free control.  Overall, inference-time scaling and steering of diffusion models, even training-free, provides significant quality and controllability benefits. Code available [here](https://github.com/zacharyhorvitz/FK-Diffusion-Steering).

A General Framework for Inference-Tinee Scaling and Steering of Diffusion Models
Raghav Singhal"! Zachary Horvitz. *2 Ryan Teehan *1
Mengye Ren Zhou Yu 23
Kathleen McKeown2 2
Rajesh Ranganath" 14
Abstract
1. Introduction
Diffusion models have demonstrated remarkable performance in generative modeling, but gener- ating samples with specific desiderata remains challenging. Existing solutions - such as fine- tuning, best-of-n sampling, and gradient-based guidance are expensive, inefficient, or lim- ited in applicabillity. In this work, we intro- duce Feynman-Kac (FK) steering, which applies Feynman- Kac interacting particle systems to the inference-time steering of diffusion models with arbitrary reward functions. FK steering works by generating multiple trajectories, called particles. and resampling particles at intermediate steps based scores computed using functions called potentials. Potentials are defined using rewards for intermediate states and are chosen such that a high score indicates the particle will yield high- reward sample. We explore various choices of potentials, rewards, and samplers. Steering text- to-image models with human preference re- ward, we find that FK steering outperforms fine- tuned models with just 2 particles. Moreover, FK steering a 0.8B parameter model outperforms a 2.6B model, achieving state-of-the-art perfor- mance on prompt fidelity. We also steer text dif- fusion models with rewards for text quality and rare attributes such as toxicity, and find that FK steering generates lower perplexity text and ables gradient-free control. Overall, inference- time scaling and steering of diffusion models, even training-free, provides significant quality and controllability benefits. Code available here.
Diffusion-based generative models (Sohl-Dickstein al., 2015) have led to advances in modeling images (Ho et al., 2020; Song et al., 2020b), videos (Ho et al., 2022), and proteins (Gruver et al., 2023), as well as promising results for text generation (Li et al., 2022; Han et al., 2023; Gong et al., 2023; Gulrajani & Hashimoto, 2023; Horvitz et al., 2024). Despite these advances, diffusion models have fail- ure modes. For example, text-to-image models often fail to adhere to text prompts (Ghosh et al., 2024). Addition- ally, adapting models to produce samples that conform to specific user preferences remains a challenge.
One approach for making generative models po(x) adhere to user preferences to encode preferences a reward r(xo) and sample from the rilted distribution Px)ree(X) x po (x) exp(r(x)) (Korbak et al., 2022), where r(x) can be human preference models (Xu al., 2024; Wu et al., 2023b), vision-language models (Liu et al., 2024a), or like- lihoods p(y x) (Wu et al., 2023a). Sampling from this tilted distribution favors high-reward samples. Current approaches for sampling from the tilted distribution can be categorized into (a) fine-tuning and (b) inference-time steering methods.
Black et al. (2023), Fan al. (2024), Domingo- -nnrihh et al. (2024), and Wallace et al. (2024) fine-tune diffusion models with reward functions. However, fine-tuning re- quires expensive training and ties the model to the reward used while training. Alternatively, two common inference- time approaches are maudient-based guidance (Song al., 2020b; Bansal et al., 2023) and best-of-n sampling. Best- of-n sampling can be used for any diffusion model and reward function, however, it wastes computation on low- reward samples (Chatterjee & Diaconis, 2018). Gradient- based guidance presents an efficient alternative, but it is limited to differentiable reward functions and continuous- state diffusion models.
'Equal contribution CCourant Institute of Mathematical Sci- ences, New York University *Columbia University 'Arxlex.ai "Center for Data Science, New York University. Correspondence to: Raghav Singhal <rsinghal@nyu.edin @nyu.eduu, Zachary Horvitz cthh2000@couumbiaacduu.. Ryan Techan <rst306@ ndu.duuu
In this work, we present FK steering, flexible framework for steering diffusion- based generative models with arbi- trary rewards that uses FK interacting particle system meth- ods (Moral, 2004; Vestal et al., 2008). We generalize previ- ous works that define Feynman- Kac Kac measures to condition-
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Provides an alternative to fine-tuning and gradient guid- ance. steering text-to-image diffusion models with A General Framework for Ceffrrence-Tiee Scaling and Steering of Diffusion Models
Inference
Best-of-N
FK Steering
2
Reward
1.
3.
Denoising
Denozing
Denoiing
Iteratively de-neise ør
Prompt: "a green stop sign in a red field"
Generate multiple samplea (particles).
(3) Resample promising particles intermediate steps.
Figure I. Feynman-Kac steering is particle-bas based sampler which produces consistent approximations of the target distribution, pa (Xo) exp (Ar(x.)). At intermediate steps, FK stecring scores partieles using functions called potentials and and then resamples based on potential scores. Potentials defined using intermediate rewards and selected such that paths yielding high-reward samples up-weighted.
ally sample diffusion models (Trippe al., 2022; Wu al., 2023a; Chung et al., 2022; Janati et al., 2024). FK steering enables guidance with arbitrary reward functions, differen- tiable or otherwise, for both discrete and continuous-state models. The approach makes use of rare-event simulation method, Feynman- Kac interacting particle system (FK-IPS) (Moral, 2004; Del Moral & Garnier, 2005; Hairer & Weare, 2014; Vestal al., 2008). FK-IPS enables the generation of samples with high-rewards, which may be rare events un- der the original model po
Stable Diffusion v2.1
Stable Diffusion XL-base
GenEVAL scoe ue
0.55
Stecrin
3.55
Steering
3.50
0.45
3.45
k03 2- 2-18-
2d4 2=
Stable Diffusion
Stable Diffusion
GenEAL score
FK Steering
3.05
Steering
0:30
3.30
Applying FK steering has two components: defining se- quence of tilted distributions over the diffusion trajectory using potential functions, and then sampling from these tilted distributions. To sample from these tilted distri- butions, FK steering (1) samples multiple diffusion pro- cesses, called particles, (2) scores particles using the po- tential functions, and (3) resamples the particles based potential scores at intermediate steps during generation, see fig. 1. Potential functions are defined using intermediate rewards and are selected such that resampling high-scoring particles yield high-reward samples Xo.
0.43
3.40
1=2 1w6
16
Sane DOPO
1 K led 1m8
Figure 2. FK stecring small models outperforms bigger models with less compute. We measure the prompt fidelity of samples from text-to-image models using the GenEval benchmark (Ghosh et al., 2024). We compare the highest-reward sample from FK steering the base models against the base models and their fine- tuned versions. As the reward, we use ImageReward (Xu et al., 2024). FK steering. for all models, outperforming fine-tuning with & 2. Moreover, FK steering SDv2. (0.8B) outperforms fine-tuned SDXL (2.6B) model, with fewer FLOPS and faster sampling.
no extra training, improves performance
We show that diffusion models enable many choices of in- termediate rewards, samplers, and potentials. We then pirically demonstrate that these new choices improve traditional choices (Wu et al., 2023a). Remarkably, for number of tasks, we see significant performance benefits for both image and text diffusion models with FK steering with as few as particles (see fig. 2).
for both continuous and discrete diffusion models, and for arbitrary rewards.
We show that particle-based methods such as twisted dif- fusion sampler (TDS) (Wu al., 2023a) and Li et al. (2024), are instances of FK interacting particle systems. Expanding the set of potentials, samplers, and reward models improves performance across many tasks.
Contributions. Our methodological contributions are the following:
Empirically, we demonstrate that FK steering:
We present Feynman-Kan steering, a flexible and effec- tive framework for building particle-base approxima- tions of the tilted distribution po (x exp(Ax(xxxe) c)),
2

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
plpurrr the
(2024) propose direct preference optimization (Rafailov et al., 2024) for diffusion models. However, fine-tuning requires allocating training resources and coupling a model to a specific reward. Moreover, we show FK steering, with just 2 particles, outperforms fine-tuning in several settings without any additional training.
Statie Fiffurion NT
FK strering
Stabie Tifforio NL.
Inference-time steering. Gradient-basek methods such as classifier guidance (Song et al., 2020b; Bansal et al., 2023) enable steering diffusion models at inference-time. Reward gradients are used to tilt the diffusion model's score, so (Xt, Vx. Txx)) where sy is the marginal score. However, gradient-based guidance is limited to differen- tiable rewards and continuous-state models.
PK steering
Suilee Difloise
Figure 3. FK steering improves prompt fidelity and sample quality. First row: random sample from SDXL. Middle and bottom rows: the highest reward sample using gradient-free FK steering with SDXL and SDv2.1, with k 4. FK steering SDXL and 1 improves prompt fidelity compared to a random sam- ple from the base model. Prompts selected from the GenEval benchmark set.
FK steering builds on top of recent works that sample from Feynman-Ka path distributions for conditional sampling with diffusion models, either using partiele-based sampling (Trippe et al., 2022; Wu et al., 2023a; Cardoso et al., 2023; Dou & Song, 2024; Zhao et al., 2024b) or gradient-based sampling (Chung et al., 2022: Janati et al., 2024). In ap- pendix F.2, we show how TDS (Wu et al., 2023a) and SVDD (Li et al., 2024) are examples of FK interacting par- ticle systems (Moral, 2004). Our experiments demonstrate the effectiveness of these methods for new settings, and the value of expanding the choice of potentials, rewards, and samplers.
human preference rewards outperforms fine-tuned mod- els and gradient-guidance on a prompt fidelity bench- mark with just two particles, see fig. Moreover, FK steering combined with fine-tuned models or gradient guidance unlocks even further improvements. We also steer text diffusion models to generate higher quality samples with improved linguistic acceptability and per- plexity.
Enables smaller models to beat larger models (Ghosh al., 2024), with faster sampling and less compute (see the right panel in fig. 2).
3. Keynman-Kac Steering of Diffusion Models In this section, we present details of the FK steering frame- work for inference-time steering of diffusion models.
Generates samples with (rare) specified attributes, such as toxicity, useful attribute for red-teaming (Zhao et al., 2024a). FK steering text diffusion model, without gra- dient guidance, increases the toxicity rate from 0.3% to 64.7%, and outperforms best-of-n.
3.1. Diffussin Models
Diffusion models "Sohl-Dickstein et al., 2015) are stochas- tic processes that are learned by reversing a forward nois- ing process, ((x;). ). The noising process takes data x ~ qdata and produces noisy state x, 9 X, = x) such that at terminal-time T,, q(xr) Tprior where Tprior is the model prior. The noising process can be defined as a continuous- time Markov process (Song et al., 2020b; Kingma et al., 2021; Singhal et al., 2023; 2024) or discrete-ting time Markov chain (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Campbell et al., 2022). For exposition, we focus on discrete-time models, though the techniques to continuous-time noted below. A discrete-ting time diffu- sion model, given context c, is defined as:
Overall, in all settings we consider, FK steering always im- proves performance, highlighting the benefits of inference- time scaling and steering of diffusion models.
2. Related Work
Current approaches to generate samples from the tilted dis- tribution po (x)) exp()r(xp) can be categorized into two types: (I) fine-tuning and (2) inference-time steering ap- proaches, such as universal guidance (Song et al., 2020b; Bansal et al., 2023) and purticle-based approaches such best-of-n and TDS (Wu et al., 2023a).
applicable
pa P*( Xo c) = Tprior (XT)
Pe(xi Xx+1, t=T-1
II
Fine-tuning. Recent work (Black et al., 2023; Xu et al., 2024) proposes fine-tuning a diffusion model to maximize the reward without a Kullback-Leibler (KL) penalty. Fan et al. (2024): Domingo-Ennn -nniihh et al. (2024) propose KL- regularized fine-tuning, and more recently Wallace et al.
Sampling involves iteratively generating a path (X7, x1. Xo). where Xg is the model sample. The model pe can be trained by maximizing a lower bound on the model log- likeliholood log px = xt).
3

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
Most uses of generative models require samples with user- specified properties. In the next section, describe generic formulation for steering diffusion models towards such samples.
Algorithm 1 Feynman-Kac Diffusion Steering
Input: Diffusion model po | c), reward r(xo, c), proposals rxx, X(+1: c), potentials Gr, intermediate re- wards ro (x+, c), number of particles k.
Sample x (xT | c)) for [K]
3.2. Steering Diffusion Models
Score G Gr(xp: c) for i E [K]
One way to steer diffusion models is to encode user pref- erences in reward model r(xo) and sample from dis- tribution that tilts the diffusion model's generations po (Xo) towards an exponential of the reward function r(xn):
for E (I. do
Resample: Sample of ~ Multinomial(x,,, G and let
x x for i E [K]
Propuse: Sample x ~ - xi....,,iy
x f c) for
i E [K]
1 7 poo(o exp (Ar(xo, c)).
Pharget (xo c)
(1)
Re-weight: Compute weight Gi-1 for i E [K]: G Po Po X c)
The reward can be any arbitrary function, such as a human preference reward (Xu et al., 2024; Wu et al., 2023b), a non non-differentiable constraint, or a likelihood pyy xo).
G (1)
-(x:_1 xi c)
end for Output: return samples {x]
The target distribution favors high-reward samples, which may be rare under the model pe. This suggests the use of simulation methods that better tilt towards rare events. One broad class of rare-event simulation methods are FK-IPS approaches (Moral, 2004; Hairer & Weare, 2014) that filt the transition kernels of the diffusion process to up-weight paths that have higher-reward samples.
pe (XT, .......... .o c) exp(Ar(xx c)). Potential functions that satisfy this constraint are not unique.
Sampling from PFK,O- Direct sampling from the FK mea- sure, Prk.0, is intractable. However, targeting the interme- diate distributions PFx. supports sampling of the distribu- tion Prk,0 with particle-based methods, such as sequential Monte Carlo (SMC) (Moral, 2004; Doucet & Lee, 2018), nested IS (Naesseth al., 2019), and diffusion Monte Carlo (DMC) (Hairer & Weare, 2014). SMC generates k particles using proposal generator T(x, Xt+1, Xr, c) and at each transition step scores the particles using the potential and the transition kernel importance weights:
Next, develop FK steering, framework for inference- time steering of diffusion models using FK-IPS.
3.3. Feynman-Kac diffusion steering
We use FK-IPS produce paths (xT, ---1, Xo) with high-reward Xy samples. FK-IPS requires defining se- quence of FK distributions, P XT- xt). by tilting the base distribution Xy-............. xx) using potentials G, (Moral, 2004; Chopin et al., 2020). The sequence of distributions PvK, is built iteratively by tilt- ing the transition kernels Po (x; xx+1) with poten- tial Gr (xT, ,,- Xt). We start with (xT. x po pxr c)Gr(xx,, c) and then define the subsequent dis- tributions as:
pe (x x
= G X x x c)
x r(x:
x7, c)
Next, the particles x; are resampled based on the scores Gj. See algorithm for details. Particle approximations are consistent, that is the weighted empirical distribution defined by ((x),..., x)), Gi) converges to PrK,t, the- orem 3.19 in Del Moral & Miclo (2000). For a proof that the weighted empirical distribution, (x), Gi), converges to Ptarget. see appendix C.
Pex,t (x.,(xXT....... lc)
(2)
1 Z, po -
(c) G. [
where Z, = Ep. IT's Gz] is the normalization con- stant. The potentials G2 selected up-weight paths (XT, (Xr,...,X; x.) that yield high-rewards r(xp)- We require that the product of the potentials G, matches the exponen- tial tilt of Purget
Choosing the proposal generator т. For the proposal generator т, the simplest choice is to sample from the dif- fusion model's transition kernel pa(x; Xr- c). Alter- natively, another choice is to tilt the transition kernels to- wards high-reward samples, for instance, by using reward- gradient guidance (Song et al., 2020b; Bansal et al., 2023). We discuss some choices in appendix D.I.
(.T. X,, exp (Ar(xo, c)).
(3)
T=
Choosing the potential G,, One choice of potentials is G, = for 2 1 and Gg = exp(A(r(xo c))), this leads to importance sampling. However, importance sampling
This choice ensures that sampling X from Pex.0 is equivalent to sampling (xo c). since Pek,O x
4

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
can require many partieles to generate high-reward sam- ple (Chatterjee & Diaconis, 2018). Instead, FK steering uses potentials to up-weight paths that yield high-reward samples. We consider the following potentials that satisfy eq. (3), defined using intermediate rewards ri (x),
fd ~ po po x c) and then x we sample N samples xo use ro ro c) = log N Σj= exp(r(x)) / .c) to summa- rize the empirical distribution of rewards.
Learned rg. When sampling from pe(xo xt, c) is ex- pensive, we can use the fact that pe is trained to approx- imate the noise process q (Sohl-Dickstein et al., 2015; Song et al., 2020b). Therefore, we can use data sam- ples to train Té. For instance, when r(xo) is a classi- fier (y xo). then Nichol et al. (2021) train a classifier Po(y xx). For more general rewards, we can use:
Gx, Xx+1, c) = exp(A(ro(X c) rx(xx+1.c)) c))) and GT 1. similar to (Wu et al., 2023a), prefers partieles that have increasing rewards.
DIFFERENCE:
MAX: G.(XT,.... Xz, c) = exp(A max c)) and Go = exp(Ar(Xo, e) <T G.) prefers particles that have the highest rewards.
E [~ ue (Xo)g xx)
|xas(xe; c) exp(r(xo, c)
SUM: G.(XT..... xx) exp(A Ei=rr To (xx,,)) c)) and Go = exx(Xr(xx. () G,) selects particles that have the highest accumulated rewards.
and define no log ag. When po g. the reward To log (xo | x,,e) (exp(r(xx, c)]] can be used to de- fine potentials G, that leads to the local transitions which minimize the variance of the potential at each step, see theorem 10.1 in (Chopin et al., 2020).
Any choice of potentials that satisfy eq. (3) produce consis- tent approximations of (Xo)) However, the rewards of the particle approximation depend on the choice of poten- tials. For instance, if rxx)) is bounded, then using the dif- ference potential assigns low scores to particles that reach the maximum reward early in generation. In this setting, alternatives like the MAX potential may be apt.
We note that as long as the potentials satisfy eq. (3), any choice of Ts allows for consistent approximations. See fig. 5 for how different choices of ra correlate with r(xo)-
Continuous-time diffusions. While the presentation above is for discrete-time models, FK steering can also be used for continuous-time models (Song et al., 2020b; Kingma et al., 2021; Singhal et al., 2023). Continuous-time models are sampled using numerical methods, such as Euler-Maruyama (Särkkä & Solin, 2019), which involve defining a discrete grid {1,1 At...., 0) and then sampling from the transition kernel po (x(x; x)+a. Therefore, similar to discrete-time mod- els, FK steering can tilt the transition kernels with potentials G (x1, XI-
Interval Resampling. For a typical diffusion process, the states X2 and Xx+1 do not differ significantly. As result, we propose interval resampling. We resample at selected steps, specified by resampling schedule It = (4r.....0) O}. Fort E R, G2 is a non-uniform potential, such as the max potential, otherwise G, = 1. Interval resampling encour- ages exploration and reduces sampling time and compute. See fig. 8 and appendix E for its effect on samples.
Choosing intermediate rewards re(xx, The ideal re- wards for the intermediate state x, requires knowledge of the distribution of terminal rewards given an interme- diate step, po(o(x)) xx, c). With this distribution, re- wards ra can be chosen to ensure high-expected rewards or
4. Experiments
We evaluate FK steering with the following experiments:
FK steering for sample quality: This experiment steers text-to- -otato--mage diffusion models and text diffusion models with rewards that measure sample quality.
good worst-case quality by using the 10th percentile. Pro- ducing this distribution of rewards requires training with model samples, which can be expensive. Alternatively, we demonstrate that diffusson models offer many options with different trade-offss between compute versus the quality of the reward estimate r(xo):
For text-to-image models, we use a human prefer- ence score, ImageReward, as the reward function. We evaluate on the heldout GenEval benchmark, a prompt fidelity benchmark.
Rewards at expected Xg- Similar to Song et al. (2020b); Bansal et al. (2023); Wu et al. (2023a); Li al. (2024), intermediate rewards can be defined by evaluating the reward function at the diffusion model's approximation of the expected sample Xo: & R Epo(xo | xx Xxo Xx, c). With this choice, the intermedi- ate rewards are T (xx, c) = r(xo = x),
For text diffusion models, we explore three choices of rewards: the perplexity computed using either GPT2 (Radford et al., 2019) or a trigram language model (Liu et al., 2024b), and a linguistic accept- ability classifier (Morris et al., 2020).
Studying potential choices in FK steering: Here we study the effect of the choices of potential on the rewards r(x)).
Manyl rs- Diffusion models provide a means to sample po x,, c). During inference, for each partiele
5

---

A General Framework for Inference- Scaling and Steering of Diffusion Models
Studying different choices of intermediate rewards:
S0UL
50NGA5000
We examine the effect of using different intermediate re- wards with FK steering.
Gottial Score
For text diffusion models, we consider control of text toxicity, which occurs in around 1% of base model samples.
For image diffusion models, we do class- conditional generation on ImageNet. In this experiment, we incorporate reward gradients to tilt the proposal generator.
Nawber paricke
Namber of pertides
Imagalfnord
Figure 4. Effect of scaling the number particles. Left: GENEVAL scores for FK steering using IMAGEREWARD, aver- age particle performance. Dashed lines indicate performance of fine-tuned baselines. Middle: Corresponding IMAGEREWARD scores. Right: Distribution of IMAGEREWARD scores for sam- ples from SDv2.1 (0.8B) with and without FK steering, compared with SDXL (2.6B).
4.1. FK steering for sample quality
Text-to-Image Diffusion Models. Here we use stable diffusion (Rombach et al., 2022; Podell et al., 2023; von Platen al., 2022) text-to-image models (x c), where is the text prompt. These models include both contin- uous and discrete-time processes. As the reward, the Rewae preference model (Xu et al., 2024). In- termediate rewards are defined by evaluating the reward model on the denoised state, ro (x:) = r(xo- = 2,) where & A Ep. Xo| x1]-
use
We observe:
FK steering the base model beats fine-tuning, FK steer- ing with k particles outperforms fine-tuned mod- els on both prompt fidelity and human preference align- ment. Moreover, Figure 2 shows that FK steering with just k: = 2 has higher GenEval score than the DPO and DDPO fine-tuned models. Additionally, we show that in Table 4, FK steering outperforms steering with gradient guidance (Bansal et al., 2023) using the ImageReward model.
For the proposal generator т, we use the base model itself. For sampling from the base model, we use classifier-free guidance (Ho& & Salimans, 2022) with guidance scale set to 7.5| alongside the DDIM sampler (Song et al., 2020a) with = 1 and T = 100 time-steps. We use À = 10 and re- sampling schedule [0, 20, 40, 60, 80] with the max potential exp(A max n (xx)), see table for score model param- eter counts and sampling time.
FK steering smaller models outperforms larger mod- els. With h = 4, FK steering SDv2.1 outperforms SDXL and its DPO (Wallace et al., 2024) fine-tuned version, on GenEval scores and aesthetic quality with less sampling time: 11.5s versus 9.1s, see fig. 3 for samples.
We measure prompt fidelity using the GenEval benchmark (Ghosh et al., 2024) and we also report ImageReward and HPS (Wu et al., 2023b) scores. See appendix A for results with different sampling choices. As a benchmark, we com- pare against best-of-n (BoN) sampling and gradient guid- ance. Additionally, we also benchmark against publicly available models, fine-tuned for prompt alignment and aes- thetic quality. We use models fine-tuned using DPO (Wal- lace et al., 2024) and DDPO (Black et al., 2023),, an RL- based method. Additionally, we also evaluate FK steering fine-tuned models.
Steering fine-tuned models. table 1, we observe that FK steering fine-tuned models further improves perfor- mance.
Effect of scaling the number of particles. Figure 4 shows that scaling the number of particles improves the average prompt fidelity and human preference alignment scores of all particles for all models.
In table we report the prompt fidelity and aesthetic quality scores of the highest-reward particle generated by FK steer- ing, and in fig. 4, we report average particle performance.
Text Diffusion Models. Next, we investigate steering to improve the sample quality of text diffusion models (Li et al., 2022; Guirajani & Hashimoto, 2023; Horvitz et al., 2024). We consider two base text diffusion mod- els: SSD-LM (Han et al., 2023) and MDLM (Sahoo al., 2024) and use these models as the proposal generator т. SSD-LM is a continuous space diffusion model trained on noised word logits, while MDLM is a discrete diffusion model. We consider three reward functions for improv- ing text quality: perplexity computed with a trigram lan-
I Default choice from Hugging Face, see https:// huggi htt.aa/c.c././////eeeeeeeeeo//ee wttp////////ttlleeeeooe// ffusicn
Proompts from https://githut, htt.hut...cco///hohhh.so geneval seneval/treee htt///teee/mmeeeee tttree/mam//rrotttt....
'Prompts
from https://github. comTT/DDO.
/htress//angginggfeee..///// 12908
ddpo-alignrent.
6

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
Model
Sampler
GenEval
IR²
HPS
Model + Sampler(r)
PPL. (GPT-XL)
CoLA t
SDv1.4
k= 1
0.44
0.23
0.245
GPT2-medium
14.1
87.6
S.v1.4
BoN(k 4)
0.54
0.80
0.256
SSD-LM
1
23.2
68.3
SDv1.400ro
k
0.43
0.26
0.241
SSD-LMTXX00
18.8
76.6
SDv1.4
FK(
0.54
0.92
0.26
FK(GPT2)
11.0
80.0
S.v
k 1
0.44
0.18
0.245
FK(Trigram)
14.1
77.4
S.v1.5
V(k
0.45
0.66
0.245
FK(CoLA)
4
17.4
95.7
SDv1.5
BoN(k: = 4)
0.52
0.73
0.265
BoN(GGT22
4
13.6
75.6
SDv
k: =
0,46
0.34
0.255
BoN(Trigram)
4
15.9
71.9
SDv1.5
FK( l: 1)
0.54
0.89
0.263
BoN(CoLA)
19.2
93.8
SDv1.5
FK(k 1)
0.57
0.88
0.276
BoN(GGT2)
8
11.2
80.3
SDv2.1
k: =
0.51
0.37
0.253
BoN(Trigram)
13.9
76.8
SDv2.1
BoN(k 4)
0.61
0.88
0.263
BoN(CoLA)
8
18.4
97.2
SDv2.1
FK(A: = 3)
0.59
0.86
0.265
SDv2.1
FK(A = 4)
0.62
1.01
0.268
MDLM
85.3
28.9
SDXL
=
0.55
0.87
0.289
FK(GPT2)
49.0
39.8
SDXL
BoN(k = 4)
0.63
1.23
0.296
FK(Trigram)
40.3
37.0
SDXLam k 1
0.58
0.85
0.296
FK(CoLA)
4
73.6
69.8
SDXL
FK(k 4)
0.64
1.29
0.302
BoN(GGPT2)
4
55.5
32.9
SDXLoro
FK(k =
0.67
1.19
0.317
BoN(Trigram)
52.1
30.1
BoN(CoLA)
71.4
59.4
Table Effect of FK steering on prompt fidelity and human preference scores: For all models, FK steering improves per- formance, outperforming best-of-n, gradient guidance (V), and fine-tuning. Interestingly, even best-of-n outperforms fine-tuning, showing the effectiveness inference-time scaling. For all met- rics, higher value is better.
BoN(GPT2)
8
46.9
37.2
BoN(Trigram)
45.9
35.4
BoN(CoLA)
68.2
73.1
Table 2. Text sample quality results metrics. We sample texts of length 50 from all models and score perplexity with GPT2-XL and CoLA accoptability. Results are averaged over three seeds, Both SSD-LM and GPT-medium have 355 million parameters. MDLM is smaller model with million parameters.
guage model", a classifier? (Morris al., 2020) trained on the Corpus of Linguistic Acceptability (CoLA) dataset (Warstadt et al., 201 (), and perplexity computed by GPT2. For all choices of reward models, define the intermedi- ate rewards using To (x,) r(xo- x,) and the potential G2 exxp(A(r
steering dramatically improves perplexity (40.3 vs 85.3), but is less effective at improving CoLA (37.0 vs 28.9).
- T (x(+1))).
FK steering outperforms best-of-n. For all settings, FK steering outperforms best-of-n for the same number of particles. Notably, in many cases steering outper- forms best-of-n with twice as many particles. Addition- ally, FK steering SSD-LM with T 500 outperforms SSD-LM with T 5000 for all metrics.
For both models, we resample 50 times, every steps for SSD-LM (T 500) and every 20 MDLM (T 1000). We use 10.0 and return the highest reward sample at 0. Following Han et al. (2023), generate 20 continuations of length 50 using their 15 prompts. In addi- tion. we evaluate base model performance, best-of-n, and GPT2-Medium performance. As baseline, we also in- clude results for SSD-LM with more sampling time-steps, 5000 versus T 500 for steering. We evaluate perplexity using GPT2-XL and CoLA acceptability. Addi- tional details are included in appendix B.
Overall, our results demonstrate that FK steering with off- the-shelf rewards can enable sampling lower pwwer-perplexity. more linguistically acceptable text from diffusion models.
4.2. Studying different choices potentials
In the previous section, we use two different poten- tials: the max potential, cxp(A (xx( for the text-to-image experiments and the difference potential, exp(A(rs
Table 2 contains the evaluation results. We observe:
FK steering improves the perplexity and CoLA scores of both models. For all reward functions, FK steering with à outperforms best-of-4 on the corresponding target metric (perplexity or CoLA). For MDLM, trigram
ra (x++1))) for the text quality experi- ment. However, discussed in section the choice of potential is not unique. In this experiment, we steer text-to- image diffusion models with different choices of potentials, including the sum, max and difference potentials.
We compute trigram probabilities using oc-gram (Liu et al., 2024h).
In table 3, for all models, using the max potential yielded higher prompt fidelity scores. Since ImageReward
roberta Saberta-base-aee base-Cola
7

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
Potential
k
SDv1.4
SDv1.5
SDv2.1
SDXL
Model Sampler
Toxic
Toxic (H) f
PPI.
Max
0.540
0.540
0.616
0.633
SSD-LM
0.4%
1.2%
23.2
SSD-LM (V guidance)
22.3%
22.6%
40.3
Sum
4
0.496
0.499
0.569
0.613
MDLM
0.3%
1.9%
85.3
Difference
0.525 4
0.526
0.578
0.603
SSD-LM (no gradients)
Max
8
0.569
0.561
0.635
0.648
BoN(4)
1.6%
4.8%
21.9
Sum
8
0,532
0.517
0,588
0.634
BoN(8)
5.0%
8.1%
23.0
Difference
8
0.566
0.553
0.615
0.640
FK(h: 1)
8.4%
14.0%
22.5
FKík 4, learned rg)
15.2%
19.6%
26.3
Table 3. Effeet of different potentials on GenEval scores. Here we have GenEval prompt fidelity score, averaged over all par- ticles. Using the max potential outperforms the difference poten- tial and sum potential.
FK(E: 8)
25.0%
29.7%
23.9
FK(E 8, learned ra)
39.0%
38.0%
26.9
MDLM (no gradients)
BoN(4)
2.2%
6.7%
83.8
BoN(8)
3.7%
10.8%
84.6
FK(1: 4)
23.0%
29.0%
81.0
Model
Sampler
GenEval
IR
HPS
Time
FK(1: 4, many T&))
37.0%
40.2%
83.0
SDv1.5
k:
0.44
0.187
0.245
2.4s
FKíE 8)
53.4%
48.3%
74.3
FK(E many rg))
64.7%
51.7%
82.9
SDv1.5
V(k = 1)
0.45
0.668
0.245
20s
SDv1.5
FK(k: = 4)
0.54
0.898
0.263
8.1s
SDv1.5
FK(V, k 4)
0.56
1.290
0.268
55s
Table S. Toxicity results. We evaluate the toxicity of the gen- erated samples with (a) the classifier used for steering and (b) separate holdout (H) classifier, we also report GPT2-XI, perplex- ity.
Table 4. Comparison against gradient guidance. Here we note FK steering with the model as the proposal generator outper- forms gradient guidance, with faster sampling. We also note that FK steering can benefit from gradient guidance, albeit at the cost of more compute and sampling time.
classifier (Logacheva et al., 2022).
In this experiment, we explore the effect of different choices of intermediate rewards:
bounded between [-2, 2), using the difference of inter- mediate rewards can assign lower scores to particles that achieve the maximum reward early in generation. How- ever, for the same À and k. the max potential favors higher scoring particles more so than the difference potential. This can lead to lower particle diversity. See appendix E for samples.
For SSD-LM, we consider two choices: (1) the reward evaluated at the denoised state and (2) the reward re learned with real data.
For MDLM, we use N samples x po po(x)) | xx) to compute the reward rg = log 1 N Σ exp(r(x)) with N = 4, 16 samples.
For evaluation, we also include results from an additional holdout toxicity classifier, trained on multilingual mix- ture of toxicity datasets (Dementieva et al., 2024). Details are included in appendix B. In Table 5, we observe the fol- lowing:
4.3. Studying different choices of intermediate rewards In this experiment, we study the effect of using different choices of intermediate rewards on FK steering. Here we generate samples with rare attributes, such as (a) toxicity text diffusion models and (b) coas-cooniiional image generation with 1000 classes in the dataset.
Using sampls r improves controllability: FK steering MDLM with
achieves an accuracy of 53.4%. Using more samples for intermediate rewards improves performance even further to 64. 7%. steer- ing outperforms best-of-n sampling with both 4 and 8 particles.
Controlling Text Toxicity. We consider the task of red- teaming toxicity, a rare attribute identified only 1% of base SSD-LM samples and 0.3% of MDLM samples. Here, we examine whether FK steering enables testing rare but dangerous model behavior, a critical factor considered be- fore deploying systems (Zhao al., 2024a). The text dif- fusion models, SSD-LM and MDLM, the sampling param- eters, and prompts are identical to section 4.1. We use the base models as the proposal generators. As baseline, we compare against gradient guidance for SSD-LM and best- of-n for both models. For reward, we use popular toxicity
FK steering can outperform gradient guidance and preserves fluency: With particles, FK steering SSD- LM outperforms gradient guidance on holdout toxicity accuracy (29. 29.7% 22.6%), and improves on perplexity
htttpes//oggingfffe..ccctttteee xi amne-large-toxicctty-llaas-i---
8

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
(23.9 40.3). Using learned intermediate rewards im- proves performance further, increasing toxicity to 39.0%.
(x(())
Ge
ply xo) Mean (Max)
r(xo- = =x,) r(xo- &c)
Diff.
0.59 (0.72)
Max
0.65 (0.70)
Image Reward Correlation (zy .t)
Text Toxicity Correlation (At
1.0
SED
Learned
Diff.
0.88 (0.94)
3304M ULarer Posertie
Learned
Max
0.88 (0.96)
0.5
0.2
0.8
Table 6. ImageNet class-conditional probabilities with differ- ent choices of rewards and potentials. In this experiment, we explore the effect of two choices of rewards, learned and the re- ward evaluated at the denoised state (Wu et al., 2023a). We also explore the effect of different choices of potentials, the difference and the max potential. We observe that learning the reward im- proves performance significantly.
0.44
0.2
0.4
0.0
heme (3M
012
0.6
0.4
0.5
Inference Progress
Inference Progress
Figure 5. Correlation between (x())) and final state r(xo): Left: Correlations between r(xo) and r(xo = &,) for sev- eral text-to-image models, where is the ImageReward model. Right: Correlation between a text toxicity classifier r(Xu) and (a) r(xo xx) and (b) learned (x:),, using on SSD-LM. Learning the intermediate rewards with a regression objective improves the correlation between r(x)) and r(xo).
FK-IPS (Moral. 2004). Our experiments demonstrate that FK steering can improve sample quality and controllabil- ity of image and text diffusion models, outperforming fine- tuning and other inference-time approaches.
Better Rewards vs. More Particles. We observed that using better intermediate rewards, either learned or using multiple samples, improves performance. For instance, FK steering SSD-LM for = achieves 15.2% accuracy with learned rewards, compared to 8.4% when using the reward evaluated at the denoised state, however, with k 8 accu- racy increases to 25%, without the learned rewards. There- fore, FK steering offers two ways for scaling compute to improve performance: allocating additional resources to better estimate rewards r(xo), or by scaling the number of particles.
FK steering can be used in a "plug-and-play" fashion, with no extra training. For instance, using the difference poten- tial with intermediate rewards defined using the denoised state and the base model the proposal generator improves performance significantly, outperforms fine-tuned models, and enables small models to outperform larger models, with less compute. Additionally, by exploring different choices of potentials, intermediate rewards, and samplers, users can optimize performance for their tasks.
Our experiments show that scaling the number of particles is a natural mechanism for improving diffusion models. Notably, in our text-to-image experiments, even of ouperforms fine-tuned models. FK steering improves on best-of-n by resampling using intermediate rewards during generation, resulting in efficient inference-tine scaling.
Caas-Condittonal Image Generation. In this experi- ment, we steer marginal diffusion model (x() to pro- duce samples from one of 1000 different classes. Similar to Wu et al. (2023a), the reward is r(xu, y) = log Po xu) and we also use gradient guidance for the proposal distri- bution 7 rxx: x(+1.C). c).
We compare two potentials, the max potential and the dif- ference potentials, along with two different reward mod- els: one that uses the denoised state r(xo R, y) and one that is trained on noisy states X, q(x, xo) where Xg ~ Adata (Nichol et al., 2021). This experiment uses pre-trained marginal diffusion model and classifiers from Nichol & Dhariwal (2021) and generates 256 x 256 reso- lution images. In table 6, we observe that learning ra. for both gradient guidance and potential computation, provides significant improvements over the reward evaluated at the denoised state.
Acknowledgments
The authors would like to acknowledge Stefan Andreas Baumann, Yunfan Zhang, Anshuk Uppal, Mark Goldstein, and Eric Horvitz for their valuable feedback.
This work was partly supported by the NIHNHLBII Award R01HL148248. NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science. NSF CAREER Award 2145542, ONR N00014-23-1-2634, and Apple. Additional support was provided by a Fellowship from the Columbia Center of AI Technology. This work was also supported by IITP with a grant funded by the MSIT of the Republic of Korea in con- nection with the Global AI Frontier Lab International Col- laborative Research (No. 2--222000469499 & RS-2024- 00509279).
5. Conclusion
We present Feynman-Kaa Kac steering, a framework for inference-time steering of diffusion modeling, based on

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
Impact Statement
Pierre Del Moral and Josselin Garnier. Genealogical parti- cle analysis of rare events. 2005.
Controllable generation methods such as FK steering can be applied to align language models with human preferences, including to improve their personalization or safety. Ad- ditionally, we show that FK steering can be used for auto- mated red-teaming which can inform model deployment. We recognize that any such method for controllable gener- ation can be used to generate harmful samples by malicious actors. However, FK steering enables the research commu- nity to better understand properties of generative models and make them safer, which we believe will ultimately out- weigh these harms.
Pierre Del Moral and Laurent Miclo. Branching and in- teracting particle systems approximations of Feynman- Kac formulae with applications to non-linear filtering. Springer, 2000.
Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Frolian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Eli- sei Stakovskii, Alisa Smirnova, Ashraf Elnagar, Ani- mesh Mukherjee, and Alexander Panchenko. Overview of the multilingual text detoxification task at pan 2024. In Guglielmo Faggioli, Nicola Ferro, Petra Galuščáková, and Alba Gareía Seco de Herrera (eds.), Working Notes of CLEF 2024 Conference and Labs of the Evaluation Forum. CEUR-WS.org 2024.
References
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tar- low, and Rianne Van Den Berg. Structured denoising dif- fusion models in discrete state-spaces. Advances in Neu- ral Information Processing Systems, 34:17981-17993, 2021.
Carles Domingo-Enrich, Michal Drozdzal, Brian Kar- rer, and Ricky TQ Chen. Adjoint matching: Fine- tuning flow and diffusion generative models with mem- oryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspec- tive. In The Twelfth International Conference on Learn- ing Representations, 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with re- inforcement learning. arXiv preprint 133101 2023.
Arnaud Doucet and Anthony Lee. Sequential monte carlo methods. In Handbook of graphical models, pp. 165- 188. CRC Press, 2018.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to- image diffusion models. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete de- noising models. Advances in Neural Information Pro- cessing Systems, 35:28266-28279, 2022.
Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308.07983. 2023.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Sourav Chatterjee and Persi Diaconis. The sample size re- quired in importance sampling. The Annals of Applied Probability, 28(2):1099-1133: 2018
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http: ion00..... g htt.bb//oooopmmeeeeeetcooooopop
2019.
Nicolas Chopin, Omiros Papaspiliopoulos, et al. An intro- duction sequential Monte Carlo, volume 4. Springer, 2020.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models, 2023, URL https: / 1/arxiv.org// abs abs/22l0 2210. 08933.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion poste- rior sampling for general noisy inverse problems. arXiv preprint 2rvi9 14687, 2022.
Nate Gruver, Samuel Stanton, Nathan C. Frey, Tim G. J. Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson.
10

---

A General Framework for Inference-Timee Scaling and Steering of Diffusion Models
Protein design with guided discrete diffusion, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural in- formation processing systems, 36, 2024a.
URL www.thtpp///....... WRLLhttss://axiive http:///axivvvoooo org/
20009.
Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood- based diffusion language models, 2023. URL https:
Jiacheng Liu, Sewon Min, Luke Zeutlemoyer Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling un- bounded n-gram language models to trillion tokens. arXiv preprint arXiv:2401. 2024b.
www. aas./235. 18
Martin Hairer and Jonathan Weare. Improved diffusion monte carlo. Communications on Pure and Applied Mathematics, 67(12):1995 2021, 2014.
Varvara Logacheva, Daryna Dementieva, Sergey Ustyant- sev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6804 6818, Dublin, Ireland, May 2022. Associa- tion for Computational Linguistics. URL https:// aclanthology. org/ org/ www.0222.c1-1ooo.........
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd- Im: Semi-autoregressive simplex-based diffusion lan- guage model for text generation and modular control, 2023. URL https:///rxtt htttp::/aarxxv.orrr////////1 17432.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.125995 2022.
Jonathan Ho Ajay Jain, and Pieter Abbeel. Denois- ing diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Pierre Moral. Feynman-Kad formulae: genealogical and interacting particle systems with applications. Springer, 2004.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffiusion models. Advances in Neural Information Pro- cessing Systems, 35:8633-8646, 2022.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 1 119-126, 2020.
Zachary Horvitz, Ajay Patel, Chris Callison- Burch, Zhou Yu, and Kathleen McKeown. Paraguide: Guided diffu- sion paraphrasers for plug-and-play textual style trans- fer, 2024. URLhtttst //tp///aavvvv//////e//eee/.aaeve wwwp:vvv..oovv.ooooooooooooo 15459.
Christian A Naesseth, Fredrik Lindsten, Thomas B Schön, et al. Elements of sequential monte carlo. Founda- tions and Trends@ in Machine Learning, 12(3):307-392, 2019.
Yazid Janati, Badr Moufad, Alain Durmus, Eric Moulines, and Jimmy Olsson. Divide-and-conueer posterior sam- pling for denoising diffusion priors. Advances in Neu- ral Information Processing Systems, 17:97408994447 2024.
Alex Nichol and Prafulla Dhariwal. ing diffusion probabilistic models. arXiv preprint vrXiv:2102.0962. 2021.
Improved denois-
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion mod- els. arXiv preprint arXiv:2112.10741 2021.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630 2021.
Tomasz Korbak, Ethan Perez, and Christopher I. Buckley. RI with kl penalties is better viewed as bayesian infer- ence. arXiv preprint arXiv:2205. 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.0199: 2023.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im im- proves controllable text generation, 2022. URL https::: / / //arxiv.. orgg abs /2205.20.2 1 14217.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. 2019.
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokeen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Masatoshi Uchara. Derivative-fie guidance in continuous and discrete dif- fusion models with soft value- decoding, 2024. URL attpps://are org/ abs/2408 2408. 08252.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al.

---

A General Framework for Inference-Tine Scaling and Steering of Diffusion Models
Learning transferable visual models from natural lan- guage supervision. In International conference chine learning, pp. 8748-8763. PMLR, 2021.
tein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.044 04/ 2022.
Douglas Vestal, René Carmona, and Jean-Pierre Fouque. Interacting partiele systems for the computation of cdo
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is se- cretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
tranche spreads with rare defaults. 2008.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pe- dro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https: // /ithhbb. com/ http//uggingffcc/eee/ di ffusers, diffeeeeedfffffffeeeeeeaee
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
2022.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238, 2024.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:/803 /2471, 2018.
Simo Särkkä and Amo Solin. Applied stochastic differen- tial equations, volume 10. Cambridge University Press, 2019.
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. Ad- vances in Neural Information Processing Systems, 36, 2023a.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint Siii:22006043329 2024.
Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where diffuse, how to diffuse, and how to get back: Automated learning for multivariate diffusions. In Inter- national conference learning representations, 2023.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human prefer- ence score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.092: 09341, 2023b.
Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. What's 's the score? automated denoising score matching for nonlinear diffusions. In International conference on machine learning, 2024.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Con- ference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language mod- els via twisted sequential monte carlo, 2024a. URL https: /t///axivvve wtg/ haarg//bbs//44/ 240.. 17546.
Jamingg Song, Chenlin Meng, and Stefano Ermon. De- noising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Zheng Zhao, Ziwei Luo, Jens Sjölund, and Thomas Schön. Conditional sampling within generative diffusion models. arXiv preprint urXiv:2409.090 2024b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differen- tial equations. arXiv preprini arXiv:2011.1344 2020b.
Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of pro-
12

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
A. Text to Image Experiments
Model
Params
Base(k = 1)
Base(k = 4)
FK(A: = 4)
FK(k = 4, parallel)
SD v1.4/v1.5
860M
2.4s
7.3s
8.1s
5.0s
SD v2.1
865M
4.6s
15.6s
17.4s
9.1s
SDXL
2.6B
11.5s
42.3s
43.5s
21.7s
Table 7. Parameter counts and timing. In this table, we provide inference timing for text-to-image diffusion models with FK steering. We include results for FK steering single NVIDIA-AIOM GPU and two-device parallel implementation. FK steering incurs only a minimal increase in time compared to independently generating particles. This gap shrinks as the diffusion model parameter count increases.
In this section, we explore the effect of and the resampling schedule on particle diversity for text-to-image generation. Similar to Bomingo-Enrich et al. (2024), we measure the diversity of generations using the CLIP (Radford et al., 2021) encoder fe. given {xf) particles, we measure:
2 5=i k:kk - 1) [o(xb)) fo (xxx))
CLIP-Div ((x))EE)
(4)
Similar section 4.1, we use the stable diffusion text-to-image models (Rombach et al., 2022) with the ImageReward human preference score (Xu et al., 2024) the reward function. Here use the difference potential.
We evaluate FK steering with different values of X and different resampling schedules, [0,20, 20, 40, 60, 80] and [0, 70, 75, 80, 85, 90]. In table 8, we observe that for all values of À and the resampling schedule, the GenEval score of FK steering outperforms the base model. However, for lower values of X, the CLJP diversity score is significantly higher, implying higher particle diversity. Similarly, in table 9, we observe that for higher values of X, the human preference scores are higher, while the particle diversity is lower.
B. Text Experiments
For all text experiments, we use publicly available SSD-LM' MDLM", and GPT2-Medium checkpoints. For both text experiments, we generate sequences of length 50, conditioned on the prompts used by Han et al. (2023) to evaluate controllable text generation. We generate 20 continuations for each of the 15 prompts.
B.1. Baselines
Following Han et al. (2023), for SSD-LM we iteratively generate these continuations in blocks of 25. Except for our T = 5000 quality experiment, we default to T = 500 for all SSD-LM experiments, and follow the multi-hot sampling procedure, with a top-p = 0.20 (Han et al., 2023). For toxicity gradient guidance, we set the learning rate = 2000. For MDLM, we condition on each prompt by prefilling the prompt tokens at inference time. The model trained to generate tokens in blocks of 1024. For consistency, we only consider the first 50 tokens of each generated sample, after re-tokenizing with the SSD-LM tokenizer. We use 1000 steps for all MDLM experiments. For the GPT2-Medium baseline, we generate all samples with top-p- = 0.95 and temperature- 1.0.
B.2. FK steering Details
For all FK steering text experiments, for each inference: at every 10 steps for SSD-LM and every 20 steps for MDLM. To convert intermediate SSD-LM states to text, we sample tokens from the logit estimate, 8, with top-p 0.20. To convert intermediate MDLM states to text, we sample the masked tokens from the multinomial distribution given by &, By default, we sample one intermediate text for SSD-LM, and four texts for MDLM. Rewards are averaged over these samples. For Improved FK steering with MDLM, we sample and evaluate 16 intermediate texts, rather than 4.
À 10.0 and use the difference of rewards potential. We resample 50 times
For Improved FK steering with SSD-LM. we take the more involved approach of fine-tuning the off-the-shelf toxicity classifier on intermediate states, 2, To build a training dataset, used reward toxicity classifier to identify 26K non-toxic
wwteee...oo//eeeeeeeeee wwwkuulsshvveeee grtpe/-rroo//d/ee m- owt
"https://hugginggfff copoona commun 2ttp:/pp2222// medi um
13

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
Model
Sampler
Schedule
CLIP Div.
GenEval Score
SD v1.4
FK(k 4, A 10)
5-30-5
0.1437
0,4814
SD v1.4
FK(k 4, 10)
20-80-20
0.1050
0.5258
SD v1.4
FK(k -4,
5-30-5
0.2321
0,4975
SD v1.4
FK(k 4, 2)
20-80-20
0.2239
0,4910
SD v1.4
base = 4)
0.3158
0.4408
SD v1.5
FK(k 4, À = 10)
5-30-5
0.1459
0,4861
SD v1.5
FK(k = 4, À = 10)
20-80-20
0,1038
0,5224
SD v1.5
FK(A: = 4, À = 2)
5-30-5
0,2330
0.4854
SD v1.5
FK(A = 4, 2)
20-80-20
0,2252
0,5114
SD v1.5
base (k == 4)
0.3115
0,4483
SD v2.1
FK(A = 4, À = 10)
5-30-5
0,1259
0,5523
SD v2.1
FK(k 4. À 10)
20-80-20
0.1061
0.5783
SD v2.1
FK(K 4, À
5-30-5
0.2051
0.5607
SD v2.1
FK(k 4.
20-80-20
0,2213
0.5587
SD v2.1
base (k =
0.2948
0,5104
SDXL
FK(k 4. X 10)
5-30-5
0.1182
0.6056
SDXL
FK(K: 4, À = 10)
20-80-20
0.1055
0.6034
SDXL
FK(A = 4, À 2)
5-30-5
0.1816
0,5863
SDXL
FK(A: = 4, A 2)
20-80-20
0,2111
0.5857
SDXL
base (A:
0.2859
0.5571
Table 8. Effect of A and resampling schedule on diversity. Here we report average GenEval scores of all particles generation by FK steering show that prompt fidelity increases for all particles. Moreover, notice that lower values of À can also be used to generate diverse particles.
Model
Sampler
Schedule
IR (Mean Max)
HPS (Mean / Max)
CLIP Div.
SD v1.4
base (k 4)
0.234 (0.800)
0.245 (0.256)
0.348
SD v1.4
FK (k 4,2 10.0)
5-30-5
0.506 (0.783)
0.251 (0.255)
0.193
SD v1.4
FK (k 4.A 10.0)
20-80-20
0.811 (0.927)
0.258 (0.259)
0.091
SD v1.4
FK (k: 4, 1.0)
20-80-20
0.502 (0.763)
0.252 (0.256)
0.173
SD v1.4
FK (k = 4, = L.0)
5-30-5
0.368 (0.723)
0,248 (0.254)
0.236
SD v2.1
base (k 4)
0.372 (0.888)
0.253 (0.263)
0.318
SD v2.1
FK (k: 4,İ 1.0)
5-30-5
0.582 (0.835)
0.258 (0.261)
0.180
SD v2.1
FK (k 4,A = 10.0)
20-80-20
0.891 (1.006)
0.264 (0.266)
0.087
SD v2.1
FK (k: 4, = 1.0)
20-80-20
0.579 (0.826)
0.257 (0.261)
0.164
SDXL
base (k 4)
0.871 (1.236)
0.289 (0.296)
0.248
SDXL
FK (k: 4, 10.0)
5-30-5
1.032 (1. (1.88)
0.293 (0.295)
0.123
SDXL
FK (k: 4, = 10.0)
20-80-20
1.211 (1.298)
0.296 (0.297)
0.071
Table 9. Effect of X and resampling schedule on diversity. Here we report the average ImageReward and HPS scores of all particles generation by steering to show that sample quality increases for all particles.
and 26K toxic texts from the Open WebTTee corpus (Gokaslan et al., 2019). We then applied the SSD-LM forward process q to noise the text to random timestep f, and then use the base model to infer ãt. We then fine-tune the off-the-shelf reward classifier to estimate the toxicity probability of the original text given the intermediate text.
We fine-tune three reward models for different SSD-LM time-step ranges:
[500, 300), [300, 200), [200, 100)
14

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
We train with batch size 16 and learning rate 5e 7, using a constant learning rate with 50 warm-up steps. We train with cross entropy loss, and weighting (0.99 non-toxic, 0.01 toxic), due to the rarity of toxicity in the original data distribution. For the gradient-based guidance baseline for SSD-LM, we use the default guidance scale from Han et al. (2023)²3
C. Consistency of Particle Approximations In this section, we prove that using SMC with multinomial resampling leads to a consistent approximation of the target distribution, is, suppose we have & particles x and potentials (o. xi). : then the weighted empirical distribution converges the target Puugga(Xo) x po (xx) exp(Ar(xo)). Let wi denote the normalized potential scores
1
w/
Gi(z))
(5)
Σj=1 Gr(z))
where of = -x/,,.... x for E (1,...,A) k.) denotes the path sampled till time L. Then we show that as - oot
Σ woong z po (xT.X....... xo) exx((((xx))))
(6)
which implies that Σ W66xt, Ptarger (Xo).
The proof of consistency relies the following two facts:
The process on the extended space is also Markov, that is (z Zr41...., .2........)).... = po(z: Z1+1).
For each E (T (7-1) 1,....0) (,.,,...,0))
the particle-based approximation consistent, so
w/8+ (rx,,(x)....,x//
Note that, since II G. exp(Ar(xo)), eq. (7) for / = 0 implies (6).
To prove consistency we rely on lemma 11.1 in Chopin et al. (2020) which proves weak convergence of the SMC particle approximations.
Lemma (Lemma 11.1 in Chopin al. (2020)). Suppose the potential functions (Gi 10 are upper- upper-bounded bounded and x, Rd, then for all I (T...., 0}, there exists constant e, O such that for all continuous and bounded functions Raxt + R. for all k we have:
E
wlo(z))
E [o(z)]] Pos.
ct 10
(8)
where w|
Gx(z)) Σ5- (x() are the normalized resampling weights.
Lemma implies that the weighted empirical distribution for all are consistent, proving eq. (7). Now, note that eq. (8) implies that the weighted empirical distribution Σ i= wo converges to Piarget Ptarge((X)). (xo), since for all continuous and bounded
functions Rdd - R. eq. (8) implies that
E
[xx()))
co
|
Pro.o
therefore, for t the approximtptooiiitioooonn converges to the target distribution:
w 7 Px(o)) exp((((x))
(10)
Prrovided in private communication by authors Han et al. (2023).
15

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
D. Feynman-Kac IPS Discussion
D.1. Choice of proposal distribution
Here we discuss various choices for twisting the transition kernel towards high reward samples:
Gradient-basee guidance: For continuous-state models and differentiable rewards, we can use gradient's from the reward (Sohl-Dickstein et al., 2015; Song al., 2020b; Bansal et al., 2023; Wu al., 2023a) to guide the sampling process. Suppose po (x, X c) = ((oo c), TaT, then can swist the transition kernel using reward gradients:
N (Ho(X,, c) + V x, T (x,, c),
(11)
where rá is the intermediate reward, either learned or evaluated at the reward on the denoised state r(xo = X,).
Discrete normalization: For discrete diffusion models, such as masked diffusion language model (MDLM) (Sahoo et al., 2024; Shi al., 2024), we can also estimate the normalization constant:
Σ po(x, ()G(x.) X,,
(12)
and sample from (xx,((x; X(+1. xy) x po (x1 Xx+1)G xt).
However, such methods for twisting the transition kernel can lead increased sampling time compared sampling from the base model pa.
D.2. How existing work fits into FK steering TDS (Wu et al., 2023a) uses SMC do conditional sampling with a marginally trained model and a differentiable reward. They make the choices:
Potential. (x(x,Xx+11) = exp(A(r(x.) r(x++1))), where the reward is computed on the denoised state r(x:)
r(Xo = x)).
Proposal generator. They use classfferr-uiidance to approximate the conditional score model se (xt,,,))) t, y)
(x( + Vx, log Xo = =x,(x, t)) and use the following proposal generator T(x, xx+1):
r(x: x = N(Ar(s -99 so (x, v)), ((1))))
(13)
FK steering allows for more flexible use of potentials G,, as well as proposal generators. For instance, Nichol et al. (2021) show that conditionally trained scores outperform classifier-guidance even when the classifier is trained on noisy states Xx. However, as shown by Ghosh et al. (2024), conditionally trained models still have failure modes. Therefore, we demonstrate how particle based methods can be used to improve the performance of conditionally trained models as well. Furthermore, FK steering allows these methods to be applied to discrete-space diffusions well non-differentiable rewards.
Soft value- based decoding in diffusion models (SVDD) is another particle-basee based method. Instead of using SMC, SVDD utilizes a nested importance sampling algorithm (see algorithm 5 of Naesseth et al. (2019)) for the proposal generator with a single particle. SVDD makes the following choices:
Potential. Similar to TDS, they use the potential G. = exp()(((x;)- - -()++)))) where r(x:) can be off-the-shelf like TDS or learned from model samples.
Sampler. SVDD the base model as the proposal generator and generates is samples each step, selects single sample using importance sampling and makes & copies of it for the next step.
With A = co, SVDD is equivalent to doing best-of-n at each step, since the authors recommend sampling from Ptarget (Xo) x lim Po (x,) (x.))) We note that similar to SVDD, Prx.u can be sampled using nested importance sampling.
D.3. Adaptive Resampling
Following Naesseth et al. (2019): Wu et al. (2023a), we can use adaptive resampling to increase diversity of samples. Given particles x; and their potentials GJ, we define the effective sample size (ESS):
1
ESS,
(14)
Σ x==
16

---

A General Framework for Inference-Time Scaling and Steering of Diffusion Models
where G refers to the normalized potentials and ESS, E [] If ESS, l then skip the resampling step. This encourages particle diversity.
E. FK steering samples
In this section, we show the effect of various sampling parameters, such as potentials, the temperature parameter X, number of sampling steps, etc. on the diversity of samples. We use the stable diffusion XL-base (SDXL) as the base model and proposal generator and the ImageReward (Xu et al., 2024) human preference score model as the reward function. We also use adaptive resampling introduced in appendix D.3. We compare FK steering against generating k independent samples, using the same seed for generation, thus providing counterfactual generation.
Effect of A: The parameter used define the target distribution:
Ptarget (xo) 7 (x()) exp(Ar(xo)),
(15)
therefore, higher values of À upweight higher reward samples Xo- Similarly, the potentials also use À which affects resampling. We generate = samples from the SDXL using FK steering as well as A = independent samples using the max potential. fig. 7, we observe that using FK steering improves prompt fidelity, and higher values improve fidelity the cost particle diversity.
Effect of potential: In fig. 6, observe that steering with the potential reduces diversity compared to the difference potential. Here we use À and generate k 8 samples using the max and difference potential.
Effect of sampling steps. In fig. 6, we observe that diversity can be increased by increasing the number of sampling steps from 100 to 200. Here we use [180, 160, 140, 120, 0] and [80, 60, 40, 20, 0] as the resampling interval. We note that even the samples xo share the same particle as parent, there is diversity in the final samples.
Effect of interval resampling: In fig. 8, show that using interval resampling even with 100 sampling steps produces diversity in samples. For comparison, fig. for the independent versus steering generations.
STOP
STOP
STOP
STOP
Diff
STOP
STOP
STOP
STOP
STOP
STOP
Max
STOP
STOP
STOP
STOP
STOP
STOP
STOP
STOP
Figure 6. Max versus Difference potential: the top row, we plot independent samples from hase model and in bottom two have the steering partieles for the max and difference potentials. Using the potential reduces diversity compared to the difference potential. However, we note that by increasing the number of sampling steps, the diversity of the samples can be increased. Caption: green stop sign in red field
17

---

A General Framework for Inference- Time Scaling and Steering of Diffusion Models
STOP
STOP
STOP
STOP
STOP
STOP
A = 2
STOP
STOP
PTOP
STOP
==4 4
STOP
STOP
STOP
STOP
A 10
Figure 7. Effeet on on diversity: the top panel, we plot independent samples from the base model the bottom panels, we have the steering partieles for varying values of x. We observe that inereasing À leads decrease in diversity, the cost of higher prompt fidelity and improved aesthetic quality, compared to the first row which has 4 independent samples. Caption: a green stop sign red field
Figure 8. Increased prompt fidelity: In this generation, compare k 8 independent samples (top panel) versus t samples
from FK stecring (bottom pancl). FK steering selects samples which follow the prompt. Caption: a photo of a frishee above truck
18


Reviews:
[{"review_id": "eEOEpTtV8E", "reviewer": "Reviewer_xvFF", "summary": "The paper presents Feynman-Kac (FK) steering, a general particle-based framework for inference-time steering of diffusion models to generate outputs aligned with user-defined reward functions without requiring additional training or fine-tuning. FK steering generates multiple parallel sample trajectories from diffusion models and iteratively resamples these particles based on their scores computed using the potential function, which measures the user-interested property. Empirically, the method demonstrates substantial improvements in prompt fidelity and sample quality for both text-to-image and text diffusion models, outperforming fine-tuned baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments, and for highlighting that FK steering is a practical, effective approach for efficient, controllable generation with diffusion models. We also appreciate your assessment of our paper's clarity and strong theoretical grounding.\n\nWe address each of your concerns below. Let us know if there is any additional information we can include that would be helpful for your review.\n\n_Missing gradient guidance baseline in image experiments._\n\n1. **Requested gradient guidance results**.\n    - FK steering (with 4 particles, gradient-free) **outperforms gradient guidance** on all metrics and **is significantly faster** (8.1 vs 20 seconds for SDv1.5)! \n    - FK steering combined with ImageReward gradient guidance performs even better but requires a significant increase in sampling time. \n\n### Gradient-guidance Results\n| Model | GenEval | IR | HPS   | Sampling Time  |\n|-|-|-|-|-|\n| SDv1.5   | 0.44    | 0.187 | 0.245 | 2.4s  |\n| SDv1.5 + IR guidance   | 0.450    | 0.668 | 0.245 | 20s   |\n| SD v1.5 + FK (k = 4)   | **0.54**    | **0.898** | **0.263** | 8.1s  |\n| SD v1.5 + IR guidance + FK (k = 4)   | **0.56**    | **1.290** | **0.268** | 55s |\n\nThank you for encouraging us to run these experiments. These results strengthen our paper and we will include them in our final draft.\n\n2. **Additional advantages over gradient guidance**. Gradient-free steering has the following additional benefits: \n    - **Enables steering with non-differentiable rewards**. Gradient-free steering methods enable the use of non-differentiable rewards such as perplexity from an autoregressive model or a trigram model, see table 2 in the paper for examples. Additionally, gradient-free steering can make use of closed-source reward models hosted via APIs, or non-differentiable constraints. \n    - **Enables steering discrete models**. Gradient-free steering of masked diffusion models can enable attribute control (tables 2 and 4). More recently, LLaDA [Nie et al 2025] showed that discrete-state text diffusion models are competitive with auto-regressive models, making the development of gradient-free steering even more pertinent. \n\n_FK steering involves additional computation per particle due to resampling steps...this raises questions about its computational efficiency and whether its performance gains justify the added cost._\n\n1.  **Computational  efficiency**. This is an excellent question. In the text-to-image generation experiments, using interval re-sampling (calling the reward model every 20 of 100 total steps) results in minimal compute overhead. We include timing results in the appendix (table 6) which show that FK steering results in a **minimal increase in sampling time compared to best-of-n**. For our most performant model, SDXL, the increase in sampling time is only 3%, see table below. We also note that as model size increases, the gap between FK and best-of-n reduces further. \n\n| Model         | Params |   BoN (*k = 4*) | FK (*k = 4*) | \n|-|-|-|-|\n| SDXL          | 2.6B             | 42.3s          | 43.5s        | 21.7s                  |\n\n2. **Improved results justify computational overhead**. In our controllable text generation experiments (see table 4) with the masked diffusion language model, FK steering with **k=4** achieves attribute accuracy of **22.0%**, significantly outperforming the **best-of-8** attribute accuracy of **3.7%**. Best-of-8 takes 19.3 seconds versus 20.7 seconds for FK steering(k=4) on a single A100 GPU. We have added the timing results to the revised draft.\n\n_Diversity is an important factor...incorporating it as an additional evaluation metric would provide a more comprehensive assessment of FK steering’s effectiveness and potential trade-offs._\n\n- **Diversity Results**. Currently, we include CLIP-diversity scores in table 7, and provide 4 and 8 particle run samples, and highlight different components of the algorithm that can affect diversity. In the revised draft, we will include global diversity metrics as well. \n\n_...a direct comparison to recent adaptive guidance techniques and a more thorough analysis of computational efficiency trade-offs would further contextualize its impact within the broader literature._\n\n - **Adaptive guidance techniques**. Other forms of conditioning, beyond the classifier and classifier-free guidance approaches we evaluate, could be used for the proposal generator in FK steering. We are very interested in exploring building on top of other recent methods in future work.\n \n - **Computational efficiency trade-offs**. In table 6, we include the sampling time of FK steering compared to best-of-n. We discuss additional timing results above.\n\n_For gradient guidance, the paper should cite \"Diffusion Posterior Sampling for General Noisy Inverse Problems\"_\n\nThank you. We will update our related works section to include this work in our final draft.\n\n\n### References\nNie et al. \"Large Language Diffusion Models.\" arXiv preprint arXiv:2502.09992 (2025).", "claims_and_evidence": "The paper claims that FK steering outperforms best-of-n sampling and gradient guidance, but there are some issues that need further clarification. \n\n(1) Missing gradient guidance baseline in image experiments: While the paper includes gradient guidance comparisons in text diffusion tasks, it does not evaluate gradient guidance in image generation tasks, which use continuous-state diffusion models. Since gradient guidance is known to perform well in continuous diffusion settings, a direct comparison in image experiments would provide a fairer assessment of FK steering’s advantages. \n\n(2) FK steering involves additional computation per particle due to resampling steps, making a direct comparison to best-of-n with the same number of particles k somewhat unfair. Since FK steering does not significantly outperform best-of-n in Table 1 and 2 (especially Table 1), this raises questions about its computational efficiency and whether its performance gains justify the added cost. Including more balanced comparisons that account for computational overhead would strengthen the evaluation.", "methods_and_evaluation_criteria": "Selecting the best sample from multiple trajectories can reduce the diversity of generated outputs. Since diversity is an important factor in many generative tasks, incorporating it as an additional evaluation metric would provide a more comprehensive assessment of FK steering’s effectiveness and potential trade-offs.", "theoretical_claims": "The paper's theoretical claims, particularly the consistency of FK steering’s particle-based approximation to the target distribution, are well-grounded in sequential Monte Carlo theory and FK interacting particle systems.", "experimental_designs_or_analyses": "As mentioned before, the lack of gradient guidance baselines in image tasks, unfair compute comparison with best-of-n, missing diversity analysis, and limited resampling strategy evaluation weaken the rigor of the results. Addressing these would improve fairness and clarity.", "relation_to_broader_scientific_literature": "The paper integrates ideas from diffusion model guidance, sequential Monte Carlo, rare-event sampling, and reinforcement learning-based fine-tuning, offering a unified inference-time steering framework. By demonstrating that FK steering can match or outperform fine-tuning approaches with lower computational cost, it contributes to the growing trend of efficient and controllable generative modeling. However, a direct comparison to recent adaptive guidance techniques and a more thorough analysis of computational efficiency trade-offs would further contextualize its impact within the broader literature.", "essential_references_not_discussed": "For gradient guidance, the paper should cite \"Diffusion Posterior Sampling for General Noisy Inverse Problems, Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Marc L. Klasky, and Jong Chul Ye\", which originally proposed the concept of training-free gradient guidance.", "other_strengths_and_weaknesses": "The proposed framework is easy to implement, plug-and-play, and gradient-free, making it highly practical for real-world applications. Additionally, the paper is well-written and easy to follow, with clear explanations that effectively communicate the method and its contributions.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to the comments above. I am open to increasing the rating if all concerns are adequately addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bEUuhzKunw", "reviewer": "Reviewer_49Va", "summary": "This paper proposes a general framework for inference-time scaling and steering of diffusion models using Feynman-Kac (FK) particle resampling. It claims to unify various existing steering methods by casting them into a single FK-IPS (Feynman-Kac Interacting Particle System) framework. The framework is validated empirically with relatively small numbers of particles.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our paper, and for noting the clarity of our writing. We discuss your concerns below.\n\n\n_The main claim—unifying inference-time steering methods into a single FK-based framework— might lack sufficient novelty_\n\n-  We show that Feynman-Kac interacting particle systems, a well-known tool for rare-event sampling, are an effective and flexible framework for sampling from reward tilted diffusion models, where the reward can be any arbitrary scalar-valued function and the diffusion model can be continuous or discrete state. \n    -  In section F of the paper, we do show that prior works such as TDS and SVDD are specific instances of FK-IPS. However, we generalize beyond these choices and show the benefit of this generalization. We provide a detailed list of contributions below [[also see response to r3 below](https://openreview.net/forum?id=Jp988ELppQ&noteId=bBpOWtgFwj)].\n    -  Notably, these new choices for inference-time steering yield significant improvements.\n\n- **New choices yield higher-reward samples**. The product of potential condition we identify in equation 3 unlocks novel choices (of potentials, samplers, and rewards).\n    - *Potential choices*. For example, tables 3 & 5 show that new potential choices outperform traditional ones, such as the difference potential. \n    - *Intermediate rewards*. Furthermore, we show that diffusion models enable many choices for estimating intermediate rewards. In tables 4 & 5, we show that these new choices improve performance. \n\n_Why did you choose simplistic methods Best-of-N, DPO, DDPO?_\n\n- **Choice of DPO, DDPO**. We select DPO and DDPO as benchmarks as they provide public checkpoints and code, as opposed to DRAFT or ELEGANT. DPO is used in prior work, such as [Esser et al. 2024]. We are happy to add DRAFT or ELEGANT as benchmarks in the revised draft.\n\n\n- **FK steering can boost fine-tuning**. In Table 1, DPO does improve performance over the base model, but combining it with FK steering is even more performant. This indicates that fine-tuning and inference scaling are two complementary axes, and **any improvements provided by DRAFT or ELEGANT could be further increased by using FK steering.** \n\n- **Comparison to classifier-guidance**. We have also added results comparing FK steering to gradient guidance. FK steering outperforms both DDPO, DPO, best-of-n, and gradient guidance.\n\n| Model | GenEval | IR | HPS | Time  |\n|-|-|-|-|-|\n| SDv1.5                 | 0.44    | 0.187 | 0.245 | 2.4s  |\n| SDv1.5 + IR guidance   | 0.450    | 0.668 | 0.245 | 20s   |\n| SD v1.5 + FK (k = 4)   | **0.54**    | **0.898** | **0.263** | 8.1s  |\n\n- **Best-of-N is simple yet effective**. One exciting result in our paper is that best-of-n can even beat fine-tuning approaches (that require significant training compute) despite requiring no training.\n\n_Could you elaborate on the novelty provided by the MAX and SUM potentials?_\n\n- **Theoretical justification.** Based on eq. 3, many choices of potential provide a consistent approximation of the target distribution (see equation 3). However, when steering with arbitrary rewards, the DIFFERENCE potential does not necessarily yield high-reward samples.\n\n    - **Bounded rewards.**  Several choices of rewards are bounded. If a particle achieves maximum reward during sampling, the difference score will have to be negative and paradoxically will downweight this particle even though it has achieved a high reward. ImageReward is an example of such a bounded reward function.\n\n- **Empirical benefits.**  In tables 3 and 5, the MAX potential outperforms the DIFFERENCE and SUM potentials across model classes as well as different numbers of particles. \n\n_What distinguishes your intermediate rewards?_\n\nPrior works such as SVDD and TDS use the intermediate rewards $\\log E_{p_\\theta(x_0 \\mid x_t)} \\exp(r(x_0))$, either by approximating with the denoised expectation or by learning from model samples. In this work, we show that many choices of intermediate rewards can be used to consistently approximate the target distribution. \n\n1. Unlike prior work that use intermediate rewards defined using model expectations, we show that the noise process and real data can be used to learn intermediate distribution via a regression objective. \n2. Many-sample intermediate rewards: To the best of our knowledge, using samples from $p_\\theta(x_0 | x_t)$ to define intermediate rewards $\\log \\sum_{i=1}^N \\frac{1}{N} \\exp(r(x_0^i))$ is a novel contribution of our work and was not proposed in prior works. The many-sample intermediate reward is a consistent estimator of $\\log E_{p_\\theta(x_0 | x_t)}\\exp(r(x_0))$. \n\n### References\n[Esser et al. 2024] Esser, Patrick, et al. \"Scaling rectified flow transformers for high-resolution image synthesis.\" *Forty-first international conference on machine learning*. 2024.", "claims_and_evidence": "The main claim—unifying inference-time steering methods into a single FK-based framework— might lack sufficient novelty. Specifically, the DIFFERENCE potential is essentially the common weighting strategy previously utilized by inference-time steering approaches such as SMC; the other two variants are also not too impressive. Even Figure 1 which shows the high-level principles does not appear to be significantly different as compared to current inference-time diffusion model guidance/alignment papers.", "methods_and_evaluation_criteria": "Evaluation selection is appropriate.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "My biggest concern is in the authors' selection of baselines. For example, in section 4.1, where text-to-image models are aligned with prompt alignment and aesthetic quality, for inference-time technique, this work compares against best-of-n (BoN). This work chooses DPO and DDPO to represent finetuning-based methods. However, these baselines are not representative of the current literature on this topic, especially in their convergence speeds. In terms of fine-tuning based approaches, direct back-propagation-based methods such as DRAFT [1] and ELEGANT [2] are very effective in aligning diffusion models. And I am certain that tasks like prompt alignment and aesthetic quality permit employing these methods because the reward functions are differentiable. \n\nOn the other hand, BoN is also a pretty naive strategy. Why does this work not compare with more recent SMC-based methods? If the rationale is that these methods are all encapsulated by the 'general form' raised in this work, it may not be too convincing.\n\n[1] https://arxiv.org/pdf/2309.17400\n[2] https://arxiv.org/pdf/2402.15194", "relation_to_broader_scientific_literature": "This work situates itself within recent efforts for inference-time steering of generative models, claiming to unify existing methods into a single theoretical framework", "essential_references_not_discussed": "See suggested baseline selections above", "other_strengths_and_weaknesses": "1. I appreciate the writing flow of this work. It's easy to follow.", "other_comments_or_suggestions": "In page 5, the authors discuss variants of intermediate reward functions. However, the results do not appear to be new. Many papers have proposed using (1) DDIM or (2) trained value functions to serve as intermediate rewards.", "questions_for_authors": "1. Why did you choose simplistic inference-time methods (Best-of-N) and fine-tuning methods (DPO, DDPO) instead of more recent methods that are more representative? Please justify or include these additional comparisons.\n\n2. Could you elaborate on the concrete novelty provided by the MAX and SUM potentials? How are these variants practically advantageous over the standard DIFFERENCE potential?\n\n3. What exactly distinguishes your intermediate reward strategy from those already extensively explored in the literature using DDIM and trained value functions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WkZ5Za4nt", "reviewer": "Reviewer_9NEc", "summary": "This paper proposes Feynman-Kac steering for diffusion models using pretrained reward functions like ImageReward.\n\n**Update after rebuttal**\n\nMy main concern was that earlier works proposing FK for diffusion were not clearly acknowledged and discussed. During the rebuttal phase the authors proposed specific revisions to remedy this. Therefore I have increased my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the quality of our writing and experiments, and the practical applications of our approach.\n\n_Concerns regarding novelty claims in this paper._\n\nVarious fields such as statistics, posterior inference [Naesseth et al 2019], have used Feynman-Kac interacting particle systems for rare-event sampling and posterior inference with state-space models. We evaluate using FK-IPS to sample from target distributions defined by arbitrary rewards and diffusion models. \n\nFK-IPS requires two things, (1) potential functions used to up-weight sample paths that yield high rewards, see eq 2, and (2) a method for sampling from the FK distributions. In this work, we provide:\n - **New potentials**.\n     1. We identify a simple condition in equation 3  that potentials must satisfy to consistently estimate the target distribution, and show that various new choices of potentials are possible.\n     2. The traditionally used DIFFERENCE potential minimizes the variance of the potentials (see thm 10.1 in [Chopin et al., 2020]), not necessarily generating high-reward samples. \n        - For bounded rewards (e.g. ImageReward), using the DIFFERENCE potential can paradoxically eliminate particles that achieve the maximum reward early in generation.\n        - In tables 3 & 5, we show that a potential we develop that satisfies the product conditions, yields higher reward samples than the difference potential.\n    3. We show that diffusion models offer many choices of intermediate rewards with different trade-offs between compute and knowledge of the terminal sample's reward. We propose two novel choices:\n        - *Learned from data*. Using data and the noising process, we train intermediate rewards using regression. This objective generalizes learning a classifier trained on noisy states in Dhariwal et al., 2023 to arbitrary reward functions.\n        - *Many-sample intermediate reward*. Using samples from $p_\\theta(x_0 \\mid x_t)$ we define intermediate rewards of the form: $\\log \\frac{1}{N} \\sum_{i=1}^N \\exp(r(x_0^i))$. \n- **Sampling**.\n   1. We show the effectiveness of the base model as the proposal generator, which is faster than techniques like gradient guidance and expands the use of SMC to discrete-state diffusion models and non-differentiable rewards.\n\nWith this framework, we show **that inference-time steering of diffusion models is remarkably effective.**  FK steering:\n- **Outperforms fine-tuning**. With only $2$ particles, FK steering outperforms DPO, DDPO fine-tuning approaches on prompt fidelity metrics for all the text-to-image models considered. \n- **Outperforms gradient-guidance**. FK steering outperforms gradient guidance and is **significantly faster** (8.1s vs 20s for SDv1.5). \n- **Gradient-free control of discrete models**: FK steering can be gradient-free and enables plug-and-play control of discrete-space diffusion models.\n- **Overcomes parameter counts**. FK steering enables smaller models (0.8B) to outperform much larger models (2.6B) with less total compute.\n\n_This paper says that SVDD only selects a single sample, but that is only the case for the choice alpha=0, whereas for alpha > 0 SVDD seems very similar (if not identical?) to FK_\n\n**Comparison with SVDD**. SVDD selects a **single proposal** by sampling from a categorical distribution over the proposals $x_t^i$, with probability proportional to $\\exp(r(x_t^i)/ \\alpha)$  (see line 4 in algorithm 1). SVDD is a specific instantiation of nested importance sampling (algorithm 5 in Naesseth et al. 2019). FK steering uses SMC, which selects $k$ proposals instead of one, However, as noted in lines 198-204, it can use other particle-based samplers.\n\n**Prior works**.\n 1. **Nested diffusion sampling**. Elata et al 2023 uses an inner denoising loop to sample a single $x_0$ for each state $x_t$, rather than using $E_{\\theta}[x_0 | x_t]$, in the transition kernel. This approach samples from the diffusion many times, but does not retain multiple samples for each step $x_t$. \n     \n  2. **Conditional Sampling within Generative Diffusion Models** We discuss the relevant method from this review paper, Wu et al 2023, which proposes a particle-based sampler for conditional sampling with continuous diffusion models with $\\log p(y | x)$ as reward. In contrast, we steer with arbitrary rewards and present various choices of potentials.\n\nThank you for highlighting these references, we will add include them in the revised draft.\n\nThe other related works cited were posted after the ICML submission deadline. Due to limited space, we will add a discussion on these works in either the final draft or in the discussion period. \n    \n### References\n\n[Wu et al 2023] Wu, Luhuan, et al. \"Practical and asymptotically exact conditional sampling in diffusion models.\" (2023)\n\n[Chopin et al., 2020] Chopin, Nicloas et al. \"An Introduction to Sequential Monte Carlo\" (2020).\n\n[Naesseth et al. 2019] Naesseth, Christian et al. \"Elements of Sequential Monte Carlo.\" (2019).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I believe there was only proof in Appendix E, which is a direct application of a lemma of Chopin et al. It appears correct to me.", "experimental_designs_or_analyses": "Did not check.", "relation_to_broader_scientific_literature": "I have some concerns about the novelty claims in this paper in relation to prior work. It is certainly not the first to propose FK for steering diffusion models. I feel that is it too strong to say things like \"we propose FK steering\" as on L17 of abstract. To me, the main contribution of the work is in making a clear and experimentally-demonstrated connection to modern text-to-image settings and the idea of using pretrained reward functions.\n\n\"Conditional sampling within generative diffusion models\" (https://arxiv.org/abs/2409.09650v1). An earlier work discussing FK steering of diffusion models. I think this one should be cited and discussed.\n\nThis paper does cite and compare to the paper \"Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding\" (SVDD) (https://arxiv.org/abs/2408.08252). However, on L868 this paper says that SVDD only selects a single sample, but that is only the case for the choice alpha=0, whereas for alpha > 0 SVDD seems very similar (if not identical?) to FK (please clarify if this is incorrect).\n\nSome other concurrent related works (just an FYI):\n\"Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo\" (https://arxiv.org/abs/2502.12786). Also uses FK steering but requires an energy-based model (EBM) (which they distill from a diffusion model) to build reward functions that depend on the density p_t (for example, temperature control and composition). It might be nice to highlight how pretrained reward functions as in you suggest help avoid the need for EBMs.\n\n\"Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo\" (https://arxiv.org/abs/2502.06079). Similar approach for discrete temperature control.", "essential_references_not_discussed": "\"Conditional sampling within generative diffusion models\" (https://arxiv.org/abs/2409.09650v1) (see Relation To Broader Scientific Literature)\n\n\"Nested Diffusion Processes for Anytime Image Generation\" (https://arxiv.org/abs/2305.19066) -- a similar idea to the \"many-sample\" approach mentioned on L246 was discussed in Section 5 of this reference.", "other_strengths_and_weaknesses": "The writing is clear and the experiments are nice. Although I feel that the technical contributions is somewhat limited (see comments on prior work and missing references) I do appreciate the clarity of the exposition compared to earlier papers which are harder to digest. Using pretrained rewards is a good idea, and the experiments showcase clear practical applications of the method. However, I feel that earlier works proposing FK steering of diffusion models should be more clearly acknowledged.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see Relation To Broader Scientific Literature and Essential References Not Discussed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oTQ0lsgbLd", "reviewer": "Reviewer_FaZn", "summary": "This study proposes FK steering, a scalable inference method for both image and text diffusion models. FK steering consists of a proposal generator, a potential function, and an intermediate reward. In each denoising interval, the diffusion model generates multiple proposals, and the potential function uses one of three functions—difference, max, or sum—to reweight the particles for resampling in the next denoising step. The intermediate reward employed by the potential function is chosen from among expected x_0, many sample reward, or learned reward. FK steering is a training-free method that outperforms fine-tuning methods like DPO. It even surpasses SDXL (with 2.6B parameters) when using SDv2.1 (with 0.8B parameters), and its superiority is evaluated on text-to-image tasks using metrics such as GenEval, image reward, and HPS. Additionally, the method demonstrates effectiveness in text diffusion models, proving that it is a general framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the strength of our results. We also deeply appreciate your comments about the high-quality execution of our paper, the rigor/strength of our evaluations, and the clarity of our exposition. \n\n_Can FK steering make a Stable Diffusion 3 medium (2B) [A] model outperform the large (8B) [B] model?_\n\n**SD3 2B + FK steering indeed outperforms SD3 8B.** Stable Diffusion 3 2B with FK steering using just $k=3$ particles achieves a higher prompt fidelity score than the 8B parameter model. \n\n| Model                  | Param. Count | GenEval score    |\n|-|-|-|\n| SD3 medium                 | 2B  | 0.72 |\n| SD3 medium + FK(k=3)  | 2B    | **0.761** |\n| SD3 large   | 8B   | 0.74 |\n\n_Can FK steering make a timestep-distilled model like Flux-Schnell outperform Flux-dev?_ \n\n**Distillation + FK steering**. Improving timestep distillation is an interesting future direction and a natural extension of FK steering. \n\n_Given that classifier guidance by Nichol & Dhariwal (2021) significantly improves these metrics, it is unclear if FK steering can achieve similar improvements._\n\n**Combining gradients with FK steering**. Our ImageNet experiment show that FK steering can improve on classifier-guidance. We have also added ImageReward gradient-guidance results:\n\n| Model                  | GenEval | IR    | HPS   | Sampling Time  |\n|-|-|-|-|-|\n| SDv1.5                 | 0.44    | 0.187 | 0.245 | 2.4s  |\n| SDv1.5 + IR guidance   | 0.450    | 0.668 | 0.245 | 20s   |\n| SD v1.5 + FK (k = 4)   | **0.54**    | **0.898** | **0.263** | 8.1s  |\n| SD v1.5 + IR guidance + FK (k = 4)   | **0.56**    | **1.290** | **0.268** | 55s |\n\nHere, FK steering **both outperforms and improves gradient guidance**.\n\n_I am curious as to why the GenEval performance does not match between Table 1 and Table 3. Additionally, for text-to-image tasks, it would be helpful to include more qualitative comparisons using multiple seeds._\n\n- **Average vs best particle performance**. Table 1 has results from the best particle out of $k$ while Table 3 has results averaged over all $k$ particles. We do this to highlight that FK steering improves sample quality and prompt alignment metrics for all the particles. We will make this distinction more clear in the text and captions for the tables in the revised draft. \n\n_I'm interested in knowing whether the proposed method offers an orthogonal contribution to classifier-free guidance, or if it only proves effective at a specific classifier-free guidance scale._\n\n- Thanks for this question. We ran FK steering with $k=4$ with a lower guidance scale in the table below. However,  we note that higher guidance scales lead to better performance.\n\n| Model | Scale |  GenEval   | HPS   |IR   |\n|-|-|-|-|-|\n| SDv1.5  + FK | 4    | 0.52 | 0.795 | 0.256  |\n| SDv1.5  + FK   | 7.5   | 0.54 | 0.898 | 0.263  |\n| SDv2.1  + FK  | 4 | 0.59 | 0.901 | 0.262 |\n| SDv2.1  + FK  | 7.5 | 0.62 | 1.006 | 0.268  |\n| SDXL  + FK| 4    | 0.62 | 1.264 | 0.299  |\n| SDXL  + FK| 7.5    | 0.64 | 1.298 | 0.302  |\n\n_The evaluation could be improved by incorporating experiments on more up-to-date models and enhancing the metrics in the ImageNet evaluations._\n\n- **ImageNet**. We will add these numbers in the final paper revision. \n- **Up-to-date models**. We have included additional SD3 results above.\n\n_It would be beneficial to include a table or plot comparing the increase in inference time and memory cost_\n\n- **Memory and sampling time increase**. In table 6, we show the increase in sampling time from FK steering with $k=4$ particles. We include sample times for a single GPU run as well as parallel generation with 2 GPUs. We will update our draft to contain FLOP/memory details.\n\n_Showing whether performance continues to improve when increasing the number of particles beyond k=8 would help demonstrate the scalability of the proposed method._\n\n- **Beyond k=8**. In figure 4, we include the GenEval and ImageReward scores for increasing values of $k$, from 2 to 16. The figure indicates that increasing particle count improves both scores. For our text experiments, we will include additional results in the revised version of the paper. \n\n_In the early denoising steps, the expected x_0 will be blurry as the diffusion models predict the “expectation” at each denoising step...How does the reward model play a meaningful role in the earlier denoising steps?_\n\nThank you for this question. In figure 5 in the appendix, we plot the correlation of the reward $r_\\phi(x_t) = r(x_0 = E_\\theta[x_0 | x_t])$ with the reward at the terminal step $r(x_0)$. For ImageReward, we see a high correlation early in the generation. We demonstrate that learning the rewards will improve this correlation even further and can lead to improved performance. For instance, for toxicity we see a low correlation, prompting us to learn the intermediate rewards, which improves the generation's attribute accuracy as shown in table 4, see SSD-LM with learned $r_\\phi$.", "claims_and_evidence": "Overall, the experiments are well-executed and provide strong support for most of the claims. However, a few questions remain, and additional evidence would benefit the paper. It would be beneficial to strengthen experiments in the text-to-image domain with up-to-date open-weight models. Can FK steering make a Stable Diffusion 3 medium (2B) [A] model outperform the large (8B) [B] model? Or if it can surpass advanced models like Flux-dev [C]. Can FK steering make a timestep-distilled model like Flux-Schnell [D] outperform Flux-dev? In addition, experiments on ImageNet in Appendix A would be more convincing if they included additional metrics like FID and IS. Given that classifier guidance by Nichol & Dhariwal (2021) significantly improves these metrics, it is unclear if FK steering can achieve similar improvements.\n\n[A] Stability AI, https://huggingface.co/stabilityai/stable-diffusion-3.5-medium\n\n[B] Stability AI, https://huggingface.co/stabilityai/stable-diffusion-3.5-large\n\n[C] Black Forest Labs, https://huggingface.co/black-forest-labs/FLUX.1-dev\n\n[D] Black Forest Labs, https://huggingface.co/black-forest-labs/FLUX.1-schnell", "methods_and_evaluation_criteria": "The proposed methods make sense for the inference time scaling. The paper clearly explains its components—a proposal generator, potential function, and intermediate rewards—along with all the possible choices for each, which are well-motivated. The evaluation criteria are also appropriate; for text-to-image tasks, metrics like GenEval and HPS, which are commonly used in the field, are employed. However, I have less expertise in text diffusion model experiments, so insights from other experts would be valuable.", "theoretical_claims": "I reviewed the equations and theoretical claims presented in the main manuscript. The notations are meticulously defined, and the derivations appear to be correct.", "experimental_designs_or_analyses": "Overall, the experiments effectively demonstrate the benefits of FK steering. However, I am curious as to why the GenEval performance does not match between Table 1 and Table 3. Additionally, for text-to-image tasks, it would be helpful to include more qualitative comparisons using multiple seeds.", "relation_to_broader_scientific_literature": "Previous work aimed at improving diffusion model quality, text alignment, and human preference has largely relied on fine-tuning methods or inference steering techniques that employ gradient guidance from reward models. In contrast, this paper is novel and effective in that it is training-free and bypasses the need for gradient guidance.", "essential_references_not_discussed": "No critical issues were found.", "other_strengths_and_weaknesses": "As mentioned previously, the key strength of this paper is that it achieves effective results without relying on finetuning or gradient guidance. It explains each option and validates their effectiveness through ablation studies. However, the evaluation could be improved by incorporating experiments on more up-to-date models and enhancing the metrics in the ImageNet evaluations.", "other_comments_or_suggestions": "It would be beneficial to include a table or plot comparing the increase in inference time and memory cost. Additionally, showing whether performance continues to improve when increasing the number of particles beyond k=8 would help demonstrate the scalability of the proposed method.", "questions_for_authors": "In the early denoising steps, the expected x_0 will be blurry as the diffusion models predict the “expectation” at each denoising step. So, the reward model may not be able to provide a reliable signal at that point. How does the reward model play a meaningful role in the earlier denoising steps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for steering diffusion models without retraining them. The method is based on the Feynman-Kac (FK) steering. Multiple trajectories are generated from a diffusion model and iteratively resampled based on a score provided by a potential function that measures how well the user requested property is being satisfied. Empirically, the method is shown to improve prompt fidelity and sample quality for both text-to-image and text diffusion models, outperforming fine-tuned baselines.

By and large reviewers recognized the clarity of the manuscript and the experimental evidence.  Reviewers have raised some concerns about novelty and comparison with other baselines.

During the rebuttal the authors have provided more empirical evidence and comparisons with the requested baselines and clarified 
some of the phrasing regarding the novelty of the work which resulted in some scores being increased.
Decision: Accept (poster)