CoDe: Blockwise Control for Denoising Diffusion Models
Authors: Anuj Singh
Sayak Mukherjee
Ahmad Beirami
Hadi J. Rad
Venue: Accepted by TMLR ()

Abstract: Aligning diffusion models to downstream tasks often requires finetuning new models or gradient-based guidance at inference time to enable sampling from the reward-tilted posterior. In this work, we explore a simple inference-time gradient-free guidance approach, called controlled denoising (CoDe), that circumvents the need for differentiable guidance functions and model finetuning. CoDe is a blockwise sampling method applied during intermediate denoising steps, allowing for alignment with downstream rewards. Our experiments demonstrate that, despite its simplicity, CoDe offers a favorable trade-off between reward alignment, prompt instruction following, and inference cost, achieving a competitive performance against the state-of-the-art baselines}. Our code is available at https://github.com/anujinho/code.

Published in Transactions on Machine Learning Research (04/2025)
CoDe: Blockwise Control for Denoising Diffusion Models
Anuj Singh*
Sayak Mukherjee
Ahmad Beirami
Hadi Jamali-Rad
DDlfft University of Technology, The Netherlands
Shell Global Solutions International B.V., Amsterdam, The Netherlands "Massachusetts Institute of Technology, Cambridge MA, USA
(a.r.singh, h. a.jamaliradd@@tudllft.t
Reviewed on OpenReview: http:///ppaarriiisw.aa//oorrrr/p
Abstract
Aligning diffusion models to downstream tasks often requires finetuning new models or gradient-based guidance at inference time to enable sampling from the reward-tilted poste- rior. In this work, we explore a simple inference-time gradient-froe guidance approach, called controlled denoising (CoDe), that circumvents the need for differentiable guidance functions and model finetaning. CoDe a blockwise sampling method applied during intermediate denoising steps, allowing for alignment with downstream rewards. Our experiments demon- strate that, despite its simplicity, CoDe offers favorable trade-off between reward alignment, prompt instruction following, and inference cost, achieving competitive performance against the stato-of-tho-art basclines. Our code is available at: https:/ http:///gtttuuboon// hom.iiiiiooa///oooo/oaoooo aanjinho///////////a code
1 Introduction
Diffusion models have emerged as a pow- erful tool for generating high-fidelity real- istic images, videos, natural language con- tent and even molecular data (Ho et al., 2020; Song et al., 2020; Bar-Tal et al., 2024; Wu et al., 2022). While diffusion models have proved to be effective at mod- cling complex and realistic data distri- butions, their successful application of- ten hinges on following user-specific in- structions in the form of images, text, bounding- boxes or other rewards. A com- mon approach to the alignment of diffu- sion models to user preferences involves finetuning them on preference data, which is typically done through reinforcement learning (RL), to generate samples with a higher reward while main- taining low KI. divergence from the base diffusion model (Fan et al., 2023; Uchara et al., 2024a).
Compression
Style
Face
Stroke
Headshot a woman made of marble
colorful photo of an Eiffel lower
photo church
Reforence
Ilama
Coowin
Figure 1: CoDe generates high quality compression (non- differentiable reward), style, face and stroke (differentiable re- wards) guided images.
Guidance-based approaches keep the base diffusion model frozen and control its output by aligning its generative process to reward function at inference time. Gradient-bass guidance methods utilize gradients of the reward at each denoising step to align the generated samples with the downstream task (Chung et al., 2023; Yu el al., 2023; Bansal et. al., 2024b; He el al., 2024). In addition to requiring access lo a differentiable reward signal, these approaches require memory-intensive gradient computations. On the other hand, gradient-free guidance methods such as Best-of-N (Beirami al., 2024) circumvent the need for differentiable rewards but can potentially be computationally intractable they sometimes need a large number of samples, N, satisfy the alignment goal.

---

Published in Transactions on Machine Learning Research (04/2025)
In this paper, consider simple gradient-free guidance approach that aims at remedying the intractability of best-of-N. Drawing inspiration from blockwise controlled decoding in language models (Mudgal et al., 2024), we propose controlled denoising (CoDe), which exerts best-of-N control over N blocks of B denoising steps rather than waiting for the fully denoised images. Our key contributions can be summarized as follows:
We propose CoDe - an inference- time blockwise guidance approach which samples from an optimal KT-regularized objective. We study the interplay between the sample size (N) and block-size (B) and demonstrate that CoDe effective improving the reward at the cost of the least amount of KL divergence from the base model.
II. We assess the performance the aligned diffusion models structurally for two case studies (Gaussian Mixture Model (GMM), and image generation), four scenarios under image generation: style, face, stroke, and compressibiiity guidance. Our extensive (qualitative and quantitative) experimental results demonstrate that CoDe achieves competitive performance against the state-of-the-art baselines, while offering balanced trade-off between reward alignment, prompt instruction following, and inference cost.
2 Preliminaries
2.1 Diffusion Models
A diffusion model provides an efficient procedure sample from a probability density ((x) by learning to invert forward diffusion process. The forward process is Markov chain iteratively adding small amount of random noise to "clean" data point E X over T steps. The noisy sample at step / is given by Zt Vā + v1 - ōgSt, where ε (N 1), = - 8,, ã. = I at, and (Ae is variance schedule (Ho et al., 2020; Nichol & Dhariwal, 2021). The forward process can then be expressed as:
((<1r1100) = II (+)))a))) (( = -Nam; / 8,84-1,
Now, to estimate q(x), the diffusion model pa learns the conditional probabilities g(xx-11xx; to reverse the diffusion process starting from fully noisy sample or ~ ~N(0. 1) as:
T
(o(0))= = p(or) II (o (a), po(xe-1l0:) =
1). 3.).
(2)
where the variance is fixed at B.I, and only (o( is learned as:
1 (m 1 ai Vas 1
(
no(a,, 1) =
VI ai (
(3)
Here, Sy neural network which altempts to predict the noise added to xz-1 in the forward process as:
t0(x,,t) Ft Varso
(4)
Furthermore, using a conditioning signal c, diffusion models can be extended to sample from po (z]c). The conditioning signal, c, can take diverse forms, from text prompts and categorical information semantic maps (Zhang al., 2023; Mo et al., 2023). Our work focuses on text-conditioned model, Stable Diffusion (Rombach et al., 2021), which has been trained on large corpus consisting of M image-text pairs D = ((xt, M using a reweighed version of the variational lower bound (Ho et al., 2020) as optimization loss function
= arg min E.-11,7, 20.5 [[ler sp(Vanz0 + vI asst. e, t)
(5)
2

---

Published in Transactions on Machine Learning Research (04/2025)
2.2 KL- KL-Regularized Objective
Consider we have access to reference diffussin model pl-), which we refer to the base model. Note that here we drop 0 (from pe) for the ease of notation, also because base diffusion model parameters are kept intact throughout the inference-time guidance. Our goal is to obtain samples from the base model that optimize a downstream reward function r(.): : X + R, while ensuring that the sampled data points do not deviate significantly from p to prevent degeneration in terms of image fidelity and diversity of the output samples (Ruiz al., 2023). Thus, we aim to sample from a reward aligned diffusion model (T) that optimizes for a KL-regularized objective to satisfy both requirements. Let us start by defining some key concepts.
Value function. The expected reward when decoding continues from partially decoded sample xy:
V( = Ezg Pooo-paaaleee (x) ( (xo)))
(6)
Advantage function. We can define a one-step advantage of using another diffusion model ı for optimizing the downstteam reward as:
A A == Pee-ina(a--1 2i) (V - Egg ( (V(x-1;;)))
(7)
It is important to note that the advantage of the base model (when π = p) is O. Thus, we aim to choose an aligned model π to achieve a positive advantage over the base model.
Divergence. We further denote the KL divergence (KT(-II-), also known as relative entropy) between the aligned model and the base model p at each intermediate step 24
D(EG))= = (L(T((x-11xe) II p(=x-1221) = / π (x (1))) log T T(# (2t)) dxz-1-
(8)
p( (1tz)
Objective. Using Eq. (7) and Eq. (8), we can now formulate the KL-rogularized objective as:
オ = arg max [AA(z; -()x:))
(9)
where À E 20 trades off reward for drift from the base diffusion model p. Theorem 2.1. The optimal model nx for the objective formulated in Eq. (9) is given by:
" (x (2x) p(xx-1 (x))) e (x)-)))))
(10)
As we shall discuss in Section 3, our proposed approach builds on Theorem 2.1 to approximately sample from this reward aligned model using a Monte Carlo sampling strategy. An extension of the result in a conditional diffusion setting can be found in Appendix A. Notably, this is a step-wise variant of the more widely known similar objective (Korbak et al., 2022), which has been used in the context language models (Beirami et al. (2024); Mudgal et al. (2024)), and in some learning- basedd methods (Prabhudesai al., 2023; Fan et al., 2023; Wallace et al., 2023; Black et al., 2023; Gu et al., 2024; Lee et al., 2024) discussed in Scction for finc-tuning diffusion model. However, contrary to the prior art, we use our objective directly for guidance- based alignment, where as the end-to-end objective would be intractable. We also remark that this advantage is similar to controlled decoding (Mudgal et al., 2024) and how it enables efficient sampling from reward guided distributions in language models. In Appendix B, we demonstrate that sampling can be achieved using Langevin dynamics (Welling& Teh, 2011), resulting in a generalized form of classifier guidance (Dhariwal & Nichol, 2021). However, a key limitation of gradient-based approaches is the need for differentiable reward function. To alleviate this, we explore a sampling- based method for model aligmment allowing us to handle both differentiable and non-differentiable downstream rewards.
3 CoDe: Blockwise Controlled Denoising
Inspired by recent RI-basod alignment strategies for L.MSs through process rewards or valuo-guided decod- ing (Mudgal et al., 2024), we propose a sampling- based guidance method to align a conditional pretrained diffusion model, p(:lc), following the optimal solution, nx, described in Theorem 2.1. In the following, we
3

---

Published in Transactions on Machine Learning Research (04/2025)
outline an approach to approximate the value function for intermediate noisy samples followed by introducing our campling-basel aligmment strategy. Our proposed approach, coined as CoDe, is summarized in Algorithm 1.
Approximation of the value function. To compute the value function in Eq. (6) for an intermediate noisy sample 24, it is necessary to compute the expectation over 2g ~ p(zo/2)). Note that for diffusion models
such as DDPMs (Ho et al., 2020); the predicted clean sample do can be estimated given an intermediate sample 2t using Tweedie's formula (Efron, 2011) as follows:
c [[[ro|=s] Z4 v1 CreolTe,
(1)
Var
By plugging Eq. (11) into Eq. (6), the value function can be approximated as:
(zz;p, c) = (re R r(E|zo (rr]) = r(fo)
(12)
The benefit of such an approximation is that it circumvents the need for training a separate model to learn the value function, as is for instance adopted by DPS (Chung et al., 2023) and Universal Guidance (Bansal et al., 2024b). However, in certain scenarios, it also possible to use a pre-trained detection or segmentation model to extract reward-aligned features and then use Tweedie's formula to oblain dn- According to the Tweedie's s formula,, the approximation of the conditional expectation, Ezg [r(z)),, is tight when the base diffusion model parameters 0 perfectly optimize Eq. 5. For example, this approximation is expected to be more accurate towards the end of the denoising process (Ye et al., 2024).
Our objective is to achieve an improved alignment vs. di- vergence trade-off by sampling from the optimal solution presented in Theorem 2.1, Therefore, by taking advantage
Algorithm 1: CoDe
Require: p. T. N, B,
1. Sample initial noise: 27 ~ ()
of the approximation in Eq. (12), we present block-
2 Initialise counter: =
a for t E [T' 1,.......00 then
wise extension of Best-of-N (BoN) for diffusion models,
if nod(s, Sample N times over B steps:
termed as Controlled Denoising (CoDe) and outlined in Algorithm 1. Note that the timesteps during diffusion denoising are indexed from T to 0 (in descending order), and not 0 to (line 3). CoDe integrates BoN sampling into the standard inference procedure of a pretrained dif- fusion model. Unlike BoN, instead of rolling out the full denoising N times and selecting the best resulting sample, opt for performing blockwise BoN. Specifically, for each block of B steps, we unroll the diffusion model N times independently (Algorithm 1, line 5). Then, based on the value function estimation (line 6) using Eq. (12), select the best sample (line 7) to continue the reverse process until we obtain a clean image at = 0. A key advantage of CoDe its ability to achieve similar alignment-divergenee trade-offs while using a significantly lower value of N, as is demonstrated in Section 5. A more detailed discussion the optimal reward-tilted posterior in Eq. (10) and our blockwise controlled denoising procedure can be found in Appendix C.
5
(= 1i.. II p(u; 1|2;)
Compute values of all N samples:
(V(((=))) = {r(B)an ")"))) Select the sample with maximum value:
20-1 argmax V(=[") {4 (m) p
end
a s ++11
10 end Return:
Best-of-N (BoN) sampling for diffusion models. A strong baseline for inference-time alignment is Best-of-N (BoN). Empirical evidence from the realm of large language models (LLMs) (Gao et al., 2022; Mudgal et al., 2024; Gui et al., 2024) suggests that BoN closely approximates sampling from the optimal solution presented in Thcorem 2.1, which theoretically corroboratod by Beirami et al. (2024); Yang et al. (2024). More recently, BoN has emerged a strong baseline for scaling inference-time compute (Snell el al., 2024; Brown al., 2024). In BoN, N samples are obtained from the diffusion model by completely unrolling it out over T denoising steps. Then, the most favorable image is selected based on reward. This renders BoN sampling equivalent to CoDe with B = T. For other intermediate values of B, CoDe could be seen as a blockwise generalization of BoN.
Soft Valuo-Based Decoding (SVDD) for diffusion models. Concurrently to our work, Li et al. (2024) proposed an iterative sampling method to integrate soft value canction-based reward guidance into the standard inference procedure of pre-trained diffusion models. The soft value function helps look-ahead into
4

---

Published in Transactions on Machine Learning Research (04/2025)
how intermediate noisy states lead to high rewards in the future. Specifically, this method involves first sampling N samples from the base diffusion model, and then selecting the sample corresponding to the highest reward across the entire set. This highest-reward sample is used for the next denoising step in the reverse-difrusin process. This renders SVDD-PM sampling as a special case of CoDe, operating specifically on a step block size B 1.
Why blockwise BoN is almost optimal CoDe's technique for sampling from the reward-tilted posterior and its theoretical optimality in terms of reward vs divergence tradcoffs has also been used in related works such as (Li et al., 2024; Beirami et al., 2024; Gui et al., 2024; Yang et al., 2024). Specifically on the optimality of this sampling technique, we would like to mention that several recent works have shown that BoN sampling is almost optimal in terms of reward vs divergence tradeoffs (Beirami et al., 2024; Gui et al., 2024; Yang et al., 2024; Mudgal et al., 2024). In particular, Theorem 1 from (Yang et al., 2024) shows that the samples obtained from BoN follow the same distribution as the optimal CD from Eq. (10). This is the reason (Mudgal et al., 2024) reported the most favorable reward vs divergence tradcoffs using blockwise language model decoding as blockwise decoding is also optimal in terms of reward vs divergence given that it interpolates two (almost) optimal decoding schemes.
4 Experimental Setup
We assess the performance of CoDe by comparing it against a suite of cxisting statc-of-the-art guidance methods, in Texl-to-Image (T21) and Text--and-Image-o--nne and non-differentiable reward models. Unless otherwise mentioned, for all experiments, we use a pretrained Stable Diffusion version 1.5 (Rombach et al., 2021) as our base model, which is trained on the LATON-400M dataset (Schuhmann et al., 2021). As highlighted carlier, we strive to present meaningful comparative (both qualitative and quantitative) results across variety of scenarios. For quantitative evaluations, we generate 50 images per setting (i.e., prompt-reference image pair) with 500 DDPM steps. To achieve this, have used NVIDIA A100 GPUs with 80GB of RAM. Through extensive experiments, we aim to answer the following questions: Does CoDe offer a complelilive adigumenl-divergene trade-off compared lo other baselines? How does CoDe perform across guidance tasks qualitatively and quantitatively!
((T+I)21) scenarios, across both differentiable
Baselines. We select a set of widely adopted baselines from the literature. Recall that our goal to sample from the optimal value of the KL-regularized objective, as outlined in Theorem 2.1. One approach to achieve this, detailed in Appendix B, is using gradient-based method with an approximated value function, in DPS (Chung et al., 2023), which serves as our first baseline. Further, Universal Guidance (UG) (Bansal et al., 2024b), MPGD (He et al., 2024) and Freedom (Yu et al., 2023), improve upon DPS by offering better gradient estimation. Another way to sample from Theorem 2.1 is by using sampling- based approach such as in CoDe. In this direction, we consider Best-of-N (BoN) (Beirami et al., 2024) and SVDD-PM (Li et al., 2024) as our third and fourth baselines, which are also special cases of CoDe as explained earlier. For the sake of completeness, we also consider SDEdit (Meng et al., 2021) as a relevant (T+1)21 approach, for which all basclines could build on.
Extensions with Noise Conditioning. When the reward distribution deviates significantly from the base distribution sampling- baase approaches would require a relatively larger value of N to achieve alignment. To tackle this, a reference input sample, e.g. an image with the desired paining stroke or style, denoted as Tref. is provided as an additional conditioning input. Next to that, inspired by image editing techniques using diffusion (Meng et al., 2021; Koohpayegani et al., 2023), we add partial noise corresponding to only T = r7 x T (with 7 E (0, 1J) steps of the forward diffusion process, instead of the full noise corresponding to T steps. Then, starting from this noisy version of the reference image 2,, CoDe progressively denoises the sample for only steps to generate the clean, reference-aligee aligned image To. By conditioning the initial noisy sample I, the reference image Tref, we can generate images Zo that better incorporate the characteristics and semantics of the reference image while adhering to the text prompl c. An exlended version of Algorithhm 1 with noise-conditioning, denoted as CoDe(n) is discussed in detail in Appendix C (see lines 1 - 3 in Algorithm 2). For the sake of fair comparison, we apply this enhancement also to other (T+I)2I basclines denoting them as BoN( SVDD-PM(n). UG (7) and DPS(r). As we demonstrate in our experimentation, threshold n provides extra knob built. in CoDe allowing the user to efficiently trade off divergence for reward. Note
5

---

Published in Transactions on Machine Learning Research (04/2025)
Prion: pix)
Reward: p(rjs)
Posterion on pixir i p(xtpt/(k)
BoN
-B- UG (Bansal, 2024)
- CoDe (Ours)
[Chung, 2023)
[U, 2024)
1.0
0-0000
0.9
0.0
Boss
0.8
0.5
0.0
2.5
5.0
75
10.0
12.5
15.0
KL.
Divergence
Setup (left, middle) and reward vs. divergence trade-off (right) for Case Study I. CoDe offers highest, reward at lowest divergence with much lower N than BoN.
Figure 2:
that the reward-conditioning of the generated image inversely proportional to the value of 1j. Setting 1/ = results in r T and fully deactivates the input-image conditioning. A byproduct of this conditioning is compute efficiency, as is discussed in Section 8.
Evaluation Settings and Metrics. We consider two case studies. Case Study I: a prototypical 2D Gaussian Mixture Models (GMMs) in Section 5, as is also studied in (Ho et al., 2021; Wu et al., 2024); Case Study II: widely adopted T2I and (T+1)21 evaluations using Stable Diffusion in Section 6 across five deward-alignmeee scenarios: (i) style, (ii) face (iit) stroke, (iv) compressibility guidance, and (v) aesthetic guidance. For Case Study I, we present tradc-off curves for win rate versus KT-divergence for all basclines. For Case Study II, since calculating KL-divergence in high-dimensional image spaces is intractable, we use Frechet Inception Distance (FID) (Heusel et al., 2017). To ensure capture aligument w.r.t reference image (and avoid using the guidance reward itself) we borrow image alignment metric commonly used in style transfer domain (Gatys et al., 2016; Ych et al., 2020), referred to as I-Gram here. Further, we assess prompt alignment. using CLIPScore (Hessel et al., 2021), referred to as T-CLIP throughout the paper. Additionally, we consider win rate (commonly adopted in the LLM space) as yet another evaluation metric, where it reflects on the number of samples offering larger reward than the base model. To sum up, we consider expected reward, FID, I-Gram, T-CLIP, and win rate.
5 Case Study I: Gaussian Mixture Models (GMMs)
To establish an in-depth understanding of the impact of the proposed methods, we start with a simple model/reward distribution as shown in Fig. 2. For the prior distribution, we consider a 2D Gaussian mixture model p(mo) = E w:NW,, I2),, where o = 2, [,,, pa] = 3), (3, 7). ()], and Ia is an d-dimensional identity matrix. Additionally, we define the reward distribution as p(r(æ) = "Nu,,, H, 314,3] and o, 2. As can be seen in the figure, in this case and by design, reward distribution is far off the peak of the prior. Here, we train diffusion model with 3-layer MLP that takes as input (x,, t) and predicts the noise E,- This model is trained over 200 epochs with T = 1000 denoising steps. Note that all other discussed baselines can straightforwardly be trained in this setting. The results are illustrated in Fig. 2 where we plot win rate vs. KL-divergence for different values of N E 500). The details for computing the KT, divergence have been provided in the appendix F. For the guidance-based methods DPS and UG, the guidance scale is varied between and 50, whereas for the sampling-bane based methods, BoN the number of samples N is varied between and 500, while for SVDD and CoDe, the number of samples N is varied between 2 and 40.
with
As can be seen, BoN offers the upper bound of performance with CoDe achicving on-par performance trade-offs. This aligns with the observations from the realm of LLMs (Beirami et al., 2024; Gui et al., 2024), where BoN has been theoretically proven to offer the best win-rate vs KL divergence trade-offs. However, it is important to notice that CoDe achieves an on-par win-rate vs KL divergence trado-off with BoN for a much smaller N. Specifically, CoDe with N E [2, 10] achieves the same win rate vs KL divergence performance as BoN with N E [30, 500), rendering CoDe roughly 10-15x more efficient than BoN
6

---

Published in Transactions on Machine Learning Research (04/2025)
Boll C
CoDe (B-80)
CoDe In A6) +
C Cobe (0-
Cobe [000
SVD0
Verience of Samppen (nn los scale)
ExD. Raward
(hormalton)
Dstance/Uu).ut)
Ditance//Asaatt)
Figure 3: In contrast to BoN, SVDD-PM, CoDe with and without noise-condii ioningg (1 0.6, 1 1, resp.) are robust against increased distance between reward and prior distributions. SVDD-PM's generated samples offer almost zero variance indicating reward over-optimization.
In contrast, UG and DPS tend to exhibit higher KL divergence, as they often collapse to the mode of the reward distribution when the guidance scale is increased, leading to a reduction in diversity among the sampled data points, phenomenon also noted in prior research (Sadat et al., 2024; Ho et al., 2021). In both scemarios, SVDD achieves a high expected reward (or win rate) but at the expense of significantly higher divergence, even for N = 2. In contrast, CoDe offers flexibility, allowing users to balance the trade-off by adjusting parameters such as N and B, as demonstrated here. For different scenario (and for providing more comprehensive picture), where the reward distribution falls within the distribution of the prior see Appendix D.
Let us dive one step deeper into comparing the performance of CoDe, CoDe(11), BoN and SVDD-PM. To this aim, in Fig. 3, we vary the distance between the mean of the reward and prior distributions, gradually shifting the reward further away. To handle this sconario effectively, we use noise conditioning for CoDe, denoted by (1) = 1, 0.6), by sampling from the known reward distrriuutin and providing it as an input conditioning sample. We also study the impact of block size B for reward-guidance by varying it between B [L. 80, 320], with B 11 corresponding to SVDD-PM. This is shown for N 10, 50 in Fig. 3 where the expected reward sharply drops for BoN regardless of choice of N, whereas it drops less or remains almost intact for CoDe with 17 0.6, B [80, 320). In the case of = 1 for CoDe, we notice that the reward drops sharply for a larger block size (B = 320), while almost remaining constant or dropping lesser for a smaller block size [ = 80). On the other hand, SVDD-PM, imposing token-wisc aggressive guidance with B offers a high, constant reward for both N = [10, 50]. However, WVDD-PM'ss generated samples have a variance that is orders of magnitude lower than BoN or CoDe, as can be seen in the right most part of Fig. 3. This particularly low variance of SVDD-PM's generated samples (almost 10-4) indicates their collapse to single point in the reward distribution. This has been studied extensively in the literature and is referred to reward over-optimization (Prabhudesai et al., 2023), and corroborates the need for keeping a small KL, divergence from the base model, as also empirically and theoretically argued by (Beirami et al., 2024; Gao et al., 2022)
6 Case Study II: Image Generation with Stable Diffusion
We consider five commonly adopted guidance scenarios:
compressibillly, style, stroke, face and aesthelics guidance. We consider T21 tasks for compressibility and aesthetics guidance scenarios and (T+1)21 tasks for compressibility, style, face and stroke guidance scenarios. For each scenario, the reward model is task specific as elaborated in the following. A text prompt as well as a reference image are used as guidance signals. For the first three scenarios, a total of 33 generation settings (i.e., text prompt reference image pairs) are used for evaluations. For compressibility guidance, we have 12 settings. Per setting, we generate 50 samples and estimate the evaluation metrics accordingly. On the qualitative side, to demonstrate the capacity of CoDe(1) compared to other baselines, illustrate few generated examples across two reference images for two different text prompts. In favor of space, the qualitative results for face and tradcoff curves for aesthetics guidance are deferred to the Appendix E, Figs. 15, 16 and Fig. 8, respectively. On the quantitative side, given the -in-differentiable nature of compressibility as guidance signal, we demonstrate the efficacy of CoDe as compared to only sampling-baseo basclines in Table For differentiable reward-guidance scenarios (style, face and stroke), we evaluate the performance across all sconarios/setting combined for further statistical significance in Tables 2, 4.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Ref. img
SDEdIt(n) (2021)
BoN(n) (2021)
SVDD-PM(n) (2024)
CoDe(n) (Ours)
llama
FWLIER
wolf
Figure 4: CnDe(n) demonstrates superior trade-off between compressibility, image and text alignment compared to other basclines on the (T+I)2I scttings.
6.1 Non-Differentiable Reward: Compression
First, consider scenario with non-differentiable reward, where basdie guidance docs not ap- ply. Following Fan et al. (2023), we use image comn pressibility as the reward score which is measured by the size of the JPEG image kilobytes. This way, we guide the diffusion denoising process to generate memory-light, compressible images.
Compressibility Reward
N=60, Br5
reward over-optimization
1.3
N-7,
1.2-
Qualitative Comparisons. A comparative look across baselines and settings is illustrated in Fig. 4. We observe that CoDe(1) generates the best re- sults, offering superior compression as well as image and text alignment. SVDD-PM(y), SDEdit(n) and BoN(n) align well with the image and text prompt, but fall short on providing smooth-textureed content- light compressed images. However, this is to be pected in the case of SDEdit(n) since its generative process is not guided by the compression-reward.
CoDe [n]
BoN [n]
SVDD-PM [n]
10
600
800
1000
1400
KL Divergence
Figure CoDe(1)) offers better reward vs. KL- divergence trade-off as compared to BoN(n) for all N values. SVDD-PM(1) demonstrates a higher reward beyond = but at the cost of a much higher KT.- divergence.
Quantitative Evaluations. Table illustrates the performance comparison of CoDe, CoDe(1) as com- pared other basclines. Sampling-basee basclines (SVDD-PM Li et al. (2024) and BoN Gao al. (2022)) for two scenarios, T2I and (T+I)2I, where in the latter the reference image is omitted. In both scenarios, observe that SVDD-PM and SVDD-PM(n) offer
8

---

Published in Transactions on Machine Learning Research (04/2025)
slightly higher compression reward score compared to other baselines; however, CoDe(1) offers better image (I-Gram) and text. (T-CIIP) alignment and the least divergence from the base distribution (FID, CMMD) as compared to all other baselines. Most notably, Fig. 5 illustrates the reward vs. KL divergence for this scenario, demonstrating that in normal operating regimes (before reward ovcr-optimization occurs, see appendix I, Fig. 19), (n) offers almost. the same reward as its special case of B for SVDD-PM(1) with less than half of its KL divergence. Here, different points on the curves represent sweeping on each method's main set of parameters (NV = [10, 20, 30, 40, 100] for CoDe(n), BoN(r) and N 3, 5, 10, 20,30, 40, 100] for SVDD-PM(1)) Details on the computation of KL divergence are mentioned in the appendix section F.
6.2 Differentiable Rewards
Style guidance. We guide im- age generation based on a reference
Table 1: Quantitative metrics for compression reward.
Compresaibility Reward - T21
style image (Bansal et al., 2024b;
Method
Rew. (f)
FID (4)
CMMD (I)
T-CLIP (1)
I-Geum (1)
He et al., 2024; Yu et al., 2023). Following the reward model proposed in Bansal et al. (2024b), we use the CLIP image encoder to obtain embeddings for the reference style and the generated im- ages. The cosine similarity between these embeddings is then used as the guidance signal. Stroke guidance. A closely re- lated scenario to style guidance is stroke generation, where a high-level reference image containing only coarse colored strokes is used as reference (Cheng et al., 2023; Mong et al., 2021). The objective in this sctting is to produce images that romain faithful to the reference strokes. To achieve this, similar lo style guidance, we employ the CLIP image encoder to obtain embeddings from both the reference and generated images and compute the reward by measuring the cosine similarity between these embeddings. Face guidance. To guide the generation process to capture the face of specific individual (as in (He et al., 2024; Bansal et al., 2024b)), we employ a combination of multi-task cascaded convolutional net work (MTCNN) (Zhang et al., 2016) for face detection and FaceNet. (Schroff et al., 2015) for facial recognition, which together produce embeddings for the facial attributes of the image. The reward is then computed as the negative & loss between fcature embeddings of the reference and generated images.
Base-SD
1.0
1.0
1.0
1.0
BoN
1.23
1.10
1.70
0.99
SVDD-PM
1.83
2.86
61.75
0.88
CoDe
1.65
2.12
32.70
0.95
Compressibility Reward (T+1)22
Base-SD
1.0
1.0
1.0
1.0
1.0
SDEdit (n = 0.8)
0.97
2.19
29.25
0.98
1.34
BoN ( 0.8)
31
0.98
1.32
SVDD-PM 0.8)
1.48
3.54
69.5
0.89
1.15
CoDa 0.8)
L.34
3.08
48.75
0.97
1.20
Qualitative Comparisons. A compar- alive look across baselines, scenarios and settings is illustrated in in Figs. 6 and 7
Table 2: Quant. metrics (+ std.) for (T+T)2T differentiable scenarios.
Method
FID (L)
I-Gram (t)
T-CLIP (f)
Runtime (1)
(and 12 15, 16 and 17 in the appendix). Let us start with style guidance in Fig. 6. As can be seen, CoDe(1) shows arguably a better performance in capturing the style of the reference image, regardless of the text prompt. When it comes to aligument. with the text prompt, UG(r) seems to suffer to some extent with "Eiffel tower" and "woman" " fading away in the corresponding images.
SDEdit(n)
1.0
1.0
1.0
1.0
BoN(n)
1.06
1.08 (# 0.002)
0.98 (#: 0.002)
23.62 (# 0.005)
SVDD-PM (x)
1.29
1.64 (a 0.03)
0.94 (4: 0.002)
103.73 (= 0.05)
DPS (n)
1.01
1.23 (# 0.04)
0.96 (# 0.005)
6.07 (= 0.03)
UG(r)
1.38
1.31 (# 0.05)
0.89 (+ 0.002)
92.07 (# 0.04)
CoDe(m)
1.15
1.60 (= 0.05)
0.98 (#: 0.008)
37.21 (# 0.03)
Note that by excluding noise conditioning from the original baselines (removing 7, see Figs. 12, 22), they all suffer in capturing the style of the reference image, highlighting the importance of using noise-conditioning as is proposed for CoDe for all baselines operating in the (T+I)21 scenarios. Further qualitative results for stroke guidance are summarized in Fig. 7, where the same narrative and observations extend. The results for face guidance are deferred to the Appendix E, Figs. 15 and 16.
Important Remark:
Quantitative Evaluations. Table 2 summarizes the performance across all scenarios (including all settings) over four metrics: I-Gram, FID, T-CLIP and runtime (in second/ recond/image /image, and detailed Section 6.4). The reason why we use I-Gram (instead of expected reward per scenario) in our evaluations is because expected reward has been seen by the model throughout the guidance process.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Ref. img
BoNin) (2021)
DPS(n) (2023)
SVDD-PM(n) (2024)
udin) (2024)
SDEdittn) (2021)
CoDein) (Ouns)
A colorful photo of an Eiffel tower
portrait woman
Figure 6: The style alignment offered by CoDe(1) stands par outperforms other basclines in terms of quality and preserving nuances of The reference image, while adhering to the text-prompt.
Ref. imn
BoNin)
DPSin) (2023)
SVDD-PM(N) (2024)
U@in) (2024)
SDEditin) CoDein)
The skyline of New York City
photo of medieval church
Figure 7: Same narrative in Fig. 6 with CoDe outperforming UG(1) in terms of quality and ref. image- alignment, while standing-on par with all other baselines.
Table 3: Quant. metrics (+ std.) for aesthetics guidance.
The scores here are normalized with respect SDEdit as the baseline, thus indicating the perfor- mance gain over that. We notice that SVDD-PM(y) and CoDe(n) perform par in terms of offering the best image alignment (indicated by I-Gram), while being superior than all other baselines. However, CoDe(n) offers a better trade-off between image, text- alignment and divergence as compared SVDD- PM(n), as indicated by its superior T-CLIP and FID scores. Note that here again by excluding noise-
Aesthetic Guidance
T2I
Method
Rew. (f)
FID (4)
CMMD (4)
T-CLIP (f)
Base-SD (2021)
1.0
1.0
1.0
BoN (2022)
1.10
1.98
6.41
0.99
UG (2024b)
1.30
65.05
MPCD (2024)
1.22
6.55
57.63
0.93
Freedom (2023)
1.29
4.07
22.45
CoDe
1.27
2.59
6.6
0.99
10

---

Published in Transactions on Machine Learning Research (04/2025)
BoN (2021)
UG (2024)
MPGD (2023)
Freedom (2023)
CoDe (Ours)
cheetah
crocodiee
lobster
Figure 8: Qualitative evaluation across methods for aesthetic guidance.
conditioning from the other baselines (as in their original proposition) the gain margin offered by CoDe(n) would be considerably larger as is shown in ablation studies. See Appendix E. for further qualitative and quantitative results.
Aesthetic Guidance. To guide the diffusion denoising process towards generating aesthetically pleasing images, we employ the LAION aesthetic predictor V2 (Schuhmann et al., 2022), which leverages multi-layer perceptron (MLP) architecture trained atop CLIP embeddings. This model's training data consists 176,000 human image ratings, spanning a range from to 10, with images achieving a score of 10 being considered art piece. Table shows the results for sampling based and gradient based inference-time guidance methods on the given T2I scenario. We observe that CoDe offers better rewards compared to MPGD (He et al., 2024) and BoN while being second to best compared to Freedom Yu et al. (2023) and UG Bansal et al. (2024b). However, CoDe offers better text alignment (T-CLIP) and lower divergence from the base distribution (FID, CMMD) as compared to all its gradient based counterparts. This can also be observed in Figs. 18, where CoDe offers almost the same reward as MPGD, Freedom and UG, but at a lower divergence or higher T-CLIP. Additionally, we demonstrate a qualitative comparsion between all baselines Fig. 8. It can be observed that UG generates acsthetic images that do not complctely adhere to the text-prompt lcading reward over-oplimization. This not prominent in Freedom, CoDe and MPGD where the generated images of comparable aesthetic quality while significantly adhering to the text prompt of the animal.
6.3 Ablations
Fig. 9 investigates the impact of varying block size (B) and noise ratio (n) for CoDe on image (I-Gram) vs.
text aligument (T-CLIP). For reference, CoDe(y) = 1) (without image-conditioning) and UG are also depicted. Here, different points per curve represent sweeping on their main parameter (N = [5, 10, 20, 30, 40, 100] for CoDe, and guidance scale of [1,3,6 6, 12,24] for UG). On the left image, incrcasing block size seems limit the image alignment performance; put differently, same performance Regardless of block size, CoDe curves fall top of UG indicating superior overall performance. On
much larger N.
11

---

Published in Transactions on Machine Learning Research (04/2025)
CoDe [8 8,55,
CoDe (Durs) 18.5.-1000
Cuble (Oura) [8 (8-5.000.6)
- CoDe (Ours) (
3.0
Code (Oura) [8 30. 0.6)
3.0
Code (Ours) [8 15 a 0.7]
UG
CoDe (Ounn) (a-
Abistion of block size B
2.5
2.5
Abiation noise
I-Gram
I-Gram
3.0
2.0
2.0
1.8
1.6
1.5-
1.5
1.4
1.0
0.58
0.89
0.90
1.0
0.94
0.96
0.95
0.80
0.85
0.90
0.95
1.00
0.80
0.85
0.90
0.95
1.00
T-CLIP
T-CLIP
Figure 9: Ablation on the block size (B) and the noise ratio (1))
the right, changing the noise ratio L toward higher values, reduces the conditioning strength (as indi- cated also in (Meng et al., 2021; Koohpayegani et al., 2023)) resulting in lower image alignment capac- ity (I-Gram). Yet again, CoDe variants fall on top of the UG curve suggesting better image vs. text alignment performance. More detailed ablation studies and reward alignment trade-off curves are provided in Appendix E. Further note that the operation points with very low T-CLIP scores on UG curves ended up degenerating to the extent that images did not have anything in common with the text prompt (see appendix J, Fig. 20), which was another consideration for choosing the best trade-off point. We also study the impact of dropping the partial-noise conditioning on all baselines, including CoDe in Table For reference, (nDe(1) is also included where the best empirical value for 7) is selected per sce- nario. We report scores across all metrics by normalizing them w.r.t. the base Sta- ble Diffusion model (denoted by Base-SD). As can be seen, CoDe, iee. without noise- conditioning, offers performance gains in terms of image alignment while staying competitive w.r.t. text alignment (I- [[-Gram and T-CLIP scores) and deviating lesser from the base model (FID score), compared to all baselines except UG. Notably, CoDe is also considerably faster than both SVDD-PM and UG in terms of runtime. As stated earlier, here CoDe(1)) i.e. with noise-conditioning offers a much more pronounced gain in terms of I-Gram in terms of the other baselines. We provide general guidelines on setting N, B, 17 for CoDe in Appendix G and an ablation on using adaptive guidance control for N, B in Appendix H.
Table 4:
Ablation on partial-noise conditioning.
Method
FID (1)
I-Gram (t)
T-CLIP (t)
Runtime (L)
Base-SD (2021)
1.0
1.0
1.0
1.0
BoN (2022)
1.19
1.07 (#: 0.004)
0.99 (#: 0.001)
18.90 (#: 0.01)
SVDD-PM (2024)
1.42
1.24 (4 0.02)
0.98
0.004)
99.10 (4 0.08)
DPS (2023)
1.14
1.12 (+ 0.01)
0.98
(#: 0.004)
5.82 (t 0.02)
UG (2024b)
2.91
1.86
0.03)
0.85
0.006)
87.92 0.08)
CoDe
1.17
1.30 (#: 0.009)
0.99 (#: (1001)
34.63 (+ 0.04)
CnDe
3.00
3.19 (1. 0.05)
0.87 (1: 0. 0.006)
23.82 (1. 0.03)
6.4 Computational Complexity.
We provide a comparative look at the complexity of the proposed approach against other baselines. To this aim, we consider two aspects: (i) the number of inference steps, (ii) the number of queries to the reward model. We then measure the overall runtime complexity in terms of time (in sec.) required to generate one image. This is summarized in Table 5. From runtime perspective, within the gradient- based guidance group, DPS is relatively faster across all three generation scenarios. This is due to the m gradient and K refinement steps used in UG, which are not used in DPS. Within the sampling based group, SVDD-PM, imposing token-wise aggressive guidance (B 1), turns out to be an order of magnitude slower than BoN. CoDe, CoDe(n) with its blockwise guidance remains to be faster and more efficient. than BoN as well as UG, offering 4x faster runtime than UG.
Table 5: Computational complexity.
Methods
Inf. Steps
Rew. Queries
Runtime (sce/ing]
Baae-SD (2021)
T
14.12
BoN (2022)
NT
N
266,7 77
SVDD-PM (2024)
NT
NT
1399.36
DDS (2023)
7
82.10
LG (20246)
mKT
wKT
1241.47
CoDe
NT
NT/BB
489.00
CoDe(v)
Nry
NrTT/B
336.39
12

---

Published in Transactions on Machine Learning Research (04/2025)
4.5
16-
4.0
(Normalized) Reward
3.5
1.4
lase ED
CoDe
3.0
CoDe
133
CoDe [n]
CoDe [n]
BoN
2.5
BoN
1.2
BoN [n]
BoN [n]
2.0
DPS
DPS
1.1
SVDD-PM
SVDD-PM
1.5-
SVDD-PM In]
Oa
SVDD-PM [n]
1.0
UG
1.0
100
125
175
50
100
125
150
175
Compute
Compute
Figure 10: Reward vs. Compute and FID vs Compute trade-off curves for Style guidance.
6.5 Performance vs Efficiency Tradeoffs.
In addition to the breakdown of computational complexity, we also illustrate performance-eficieng tradeoff curves in terms of reward vs compute and divergence (FID) vs compute curves for the style guidance scenario in Fig. 10. Compute calculated using the breakdown of the computational complexity terms of the inference steps and reward queries as shown in Table 5. We illustrate both these curves since it is important to analyze both reward and divergence (FID) get holistic picture of performance and reward over-oplimization. Here different points on the curve represent sweeping their main parameters (N [5, 10, 20, 30, 40, 100] for CoDe, N [10, 20, 30, 40] for SVDD-PM, BoN, gradient guidance scale 1.1, 1.3)) for DPS and K = 3,, 3, 6, 12, 24] for UG (with the best gradient scale = 6). As can be observed, CoDe(1)) and SVDD-PM(1) offer the best reward vs compute and FID vs compute tradeofTs as compared to all other baselines. Specifically, while SVDD-PM(1) achieves higher rewards for the same compute as compared to CoDe(n), it also deviates significantly more from the base distribution as compared (n)) It is important note that divergence captures preserving core capabilities not captured by reward, resulting in inferior reward divergence tradeoffss that discussed in the previous sections. Thus, in terms of a tradeoff between performance (captured through reward and divergence) still offers a better tradcoff as compared to SVDD-PM(y) cnabling performance points that are not even achievable by SVDD-PM(T). On the other hand, UG offers high rewards but at the cost of either significantly higher compute or FID.
10.5, 0.7, 0.9,
efficiency, CoDe(1)
7 Related Work
Finetuning-based alignment. Prominent methods in this category typically involve either training a diffusion model to incorporate additional inputs such as category labels, segmentation maps, or reference images (Ho et al., 2021; Li et al., 2023; Zhang et al., 2023; Bansal et al., 2024a; Mou et al., 2024; Ruiz et al., 2023) or applying reinforcement learning (RL) to finetune a pretrained diffusion model to optimize for a downstream reward function (Prabhudesai et al., 2023; Fan al., 2023; Wallace al., 2023; Black al., 2023; Gu et al., 2024; Lee et al., 2024; Uehara al., 2024b). While these approaches have been successfully employed to satisfy diverse constraints, they computationally expensive. Furthermore, finetuning diffusion models is prone to "reward hacking" or "over-optimization" " (Clark et al., 2024; Jena et al., 2024), where the model loses diversity and collapses to generate samples that achicve very high rewards. This is often due to mismatch between the intended behavior and what the reward model actually captures. In practice, a perfect reward model is extremely difficult to design. As such, here we focus on inference-tine guidance-basee alignment approaches where these issucs can be circumvented. Additionally, none of the fine-tuning based methods are built for image-to-image scenarios, which the focus of this work, as we clarified earlier. To compare against them, direct approach could be fine-tuning per reference image, which renders the process
13

---

Published in Transactions on Machine Learning Research (04/2025)
computationally infeasible, taking a meta-learning approach to fine-tuning. However, such fundamental adjustments are beyond the current. scope of our work.
Gradient-based inference-time alignment. There are two main divides within this category: (i) guidance based on a value function, and (ii) guidance based on a downstream reward function. In the first divide, a value function trained offline using the noisy intermediate samples from the diffusion model. Then, during inference, gradients from the value function serve as signals to guide the generation process (Dhariwal & Nichol, 2021; Yuan et al., 2023). A key limitation of such an approach is that the value functions are specific to the reward model and the noise scales used in the pretraining stage. Thus, the value function has to be retrained for different reward and base diffusion models. The sccond divide of methods successfully overcomes this by directly using the gradients of the reward function based on the approximation of fully denoised images using Tweedie's formula (Chung et al., 2022; 2023; Yu et al., 2023). Interesting follow-up research has explored methods to reduce estimation bias (Zhu et al., 2023; Bansal et al., 2024b; He et al., 2024) and to scale gradicnts for maintaining the latent structures Icarned by diffusion models (Guo et al., 2024). Despite such advancements, the need for differentiable guidance functions can limit the broader applicability of the gradient-basee methods.
Gradient-free inference-time alignment. Tree-search alignment has recently gained attention in the context of autoregressive language models (TMs), where it has been demonstrated that Best-of-N (BoN) approximates sampling from a KL-regularized objective, similar to those used in reinforcement learning (RL)-based finetuning methods (Gui et al., 2024; Beirami et al., 2024; Gao et al., 2022). This approach facilitates the generation of high-reward samples while maintaining closeness to the base model. Mudgal et al. (2024) demonstrate that the gap between Best-of-N (BoN) and token-wise value-insed decoding (Yang & Klein, 2021) can be bridged using blockwise decoding strategy. Inspired by this line of research, we propose simple blockwise alignment technique (tree search with a fixed depth) that offers key advantages: (i) it preserves latent structures learned by diffussin models without requiring explicit scaling adjustments, unlike gradient-based methods, and (ii) it avoids "reward hacking" typically associated with Icarning- basod approaches. Concurrently, Li et al. (2024) propose a related method, called SVDD-PM, based on the kell-nnown token-wise decoding strategy in the LM space. In contrast, we devise a blockwise sampling strategy because it allows further control the level of intervention, and offers trade-off between divergence and aligmment, which is of primal interest in the context of guided generation. To enhance the sampling strategy in terms of efficiency, we apply adjustable noise-conditioning which also offers greater control over guidance signals and further improves alignment. Sequential Monte Carlo-based methods (SMC) for diffusion models (Wu et al., 2023; Chung et al., 2023; Phillips et al., 2024; Cardoso et al., 2023) share similarities with tree-search-based alignment methods such as ours, particularly in not requiring differentiable reward models. However, these methods were originally designed to solve conditioning problems rather than reward maximization. Crucially, they involve resampling across an entire batch of images, which can lead to suboptimal performance when batch sizes small since the SMC theoretical guarantees hold primarily with large batch sizes. In contrast, our method performs sampling on per-sample basis. Lastly, using SMC for reward maximization can also result in loss of diversity, even with large batch sizes.
8 Concluding Remarks
We introduce a gradient-free blockwise inference-time guidance approach for diffusson models. By combining blockwise optimal sampling with an adjustable noise conditioning strategy, CoDe, CoDe(n) offer a better reward vs. divergence trade-off compared to mate--of-the-art baselines.
Limitations and future work. Diffusion models are still computationally intensive; as such, extracting quantitative results on the performance of (inference-time) guidance-based alignment methods calls for massive resources, especially when ablating across numerous design parameters. We have used up to 32 NVIDIA A100's solely dedicated to the presented evaluation results. Yet, most commonly adopted settings we have experimented with to arrive at the numerical results in Tables 1 and 4 can be further expanded for the sake of better statistical significance in future work.
Broader Impact. We would like to caution against the blind usage of the proposed techniques as alignment methods are prone to reward over-optimization, which warrants care in socially consequential applications.
14

---

Published in Transactions on Machine Learning Research (04/2025)
References
Arpit Bansal, Fitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise, Advances in Neural Information Processing Systems, 36, 2024a.
Arpit Bansal, Hong- Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal Guidance for Diffusion Models. In The Twelfth International Conference on Learning Representations. IEEE, 2 2024b. doi: 10.48550/ 10.48550/arXiv..o 230/ar.....22....111 URL http:/ /tcc///rree //axxiiivvv.///// org/abs/ 2302.07121.
Omer Bal-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh, Google Research, and Google Deep Theoretical guarantees on the best-of-n aligmment policy. 2024.
URL https: /ttp:///aavvvv/ae http://orgg/bb//////aaee 2ttpb///9000...1111...
Kevin Black, Michael Janner, Yilun Du, Nya Kostrikov, and Sergey Levine. Training Diffusion Models with Reinforcement Learning. 5 2023. URL
wttp://www..rr//////.5..... ab3//255/ 13301v4.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoscini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiu:240..21787, 2024.
Gabriel Cardoso, Yazid Janati, E L Idrissi, Sylvain Le Corff, and Eric Moulines. Monte Carlo guided Diffusion for Bayesian linear inverse problems. 8 2023. URL https://arxdv.orggg org/ ht////bs/22/ http//2238.000088.. 07983v2.
Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hung-Yu Yu Tseng, and Hsin-Ying Lee. Adaptively-realissicc image generation from stroke and sketch with diffusion model. In 2023 IEEE/CVE Winter Conference Applications of Computer Vision (WACV), IEEE, January 2023. doi: http://dx. doi. www.doot......../ooo/oo http://..1119/////65800 2023. 00404.
URL
Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advonces in Neural Information Processing Systems, 35:25683 25696,
2022.
Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. Diffussin Posterior Sampling for General Noisy Inverse Problems. In The Eleventh International Conference Learning Representations, 9 2023. URL / org/ aaaabb//22 abs//209. 14.87v4.
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models differentiable rewards. In The Twelfth International Conference Learning Representations, 2024. URL
https:://penrreview http:://pppnnreeeeeeeeeeerrtttta
Prafulla Dhariwal and Alex Nichol. Diffusson Models Beat GANs on Image Synthesis. Advances in Neural 5 2021. ISSN 10495258. URL https://arxiv.orr org/abs////////oo////oooo/ee
Information Processing Systems, 2105. 05233v4.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Stalistical Associalion 106 (496):1602-1614 2011.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: Reinforcement Learning for Fine-tuning Text-to- Image Diffusion Models. 5 2023. URL https://arxiv.org// wwtps/ii.orr/abbb////aa o2300 16381v3.
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. Proceedings of Machine Learning Research, 202:10835-10866 10 2022. ISSN 26403498. URL https: //e::///// //trx/a..vvv//////////////aaaav/ www.iv.or//at.... org/abs/
2210. 10760v1.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pallern recognilion, pp- 2414-2425, 2016.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning diffusion models through relative preference optimization, 2024.
Lin Gui, Cristina Gârbacea, and Victor Veitch. BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling. 6 2024. URL https://arxiv, org/ abs/2406. 00832v2.
Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient Guidance for Diffusion
Models: An Optimization Perspective. 4 2024. CRL https://axxiii... org/abs/2404. abbb/2404.144431.. 14743v1.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreeiex http://ppa.aiiaaaaeeeeoooaaaaaaa
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image caplioning. In Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing. Association for Computational Linguisties, 2021. doi: 10.18653/v 118653/v1/2021. /2021.
emnlp-main.509. URL http://da dwtpdxx.iooooooooooooooooooooo.t www.dooooo....o..oo.o... org/10. 10653881/20221 http:////2222...mmmmmaaa.. emnlp-main.550
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33:6840- 1001068511112 2020. URL https://github. http:///ggttuub.ooooooooooooo/o/ com/hojonathanbo diffusion.
Jonathan Ho, Google Research, and Tim Salimans. Classifier-Free Diffusion Guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Dounstream Applications, 12 2021.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9307-9315, June 2024.
Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Elucidating optimal reward-diversity tradeofTs in texl-to-image diffusion models, 2024.
Yullwa Kim, Dongwon Jo, Hyesung Jeon, Taesu Kim, Daehyun Ahn, Hyungjun Kim, and jae-joon kim. Leveraging early-stage robustness in diffusion models for efficient and high-quality image synthesis. In seirth-seeenth Conference on Neural Information Processing Systems, 2023. URL httpps://ppenreview net/forum?ide (iw.:rrmmmmmddlllsssiiii...
Soroush Abbasi Koohpayegani, Anuj Singh, K L. Navaneet, Hadi Jamali-Rad, and Hamed Pirsiavash. Genie: Generative hard negative images through diffusiio, 2023.
Tomasz Korbak, Ethan Perez, and Christopher Buckley, RL with KL penalties is better viewed Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, 5 2022. doi: 10.18653/v1 20/55//////2///// findiiiiiiinnnneee -en.gg.-mmlpp77.... URL
11275
Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct consistency optimization for compositional text-to-image personalization, 2024.
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Masatoshi Uchara. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Palue-Based Decoding, 8 2024. URL https: //aaxxiv org/ aww./b//...a..a.e// 08252v3.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the TEEE/CH /CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 22521, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. SDFdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. ICLR 2022 10th International Conference on Learning Regresenlalions, 8 2021. URL httpps:///axiv.orr/aaaaaa/ wttp://bb//208800773332.
Sicheng Mo, Fangzhou Mu, Kuan Hong Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. FrocControl: Training-Fres Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition. 12 2023. URL
https: //tpivv///rivvva/aa org/abs//////eeeooooo.ooose. httpb///112..25555
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. Tiiaadpteer: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.
2024.
Youssef Mroueh. Information theoretic guarantees for policy alignment in large language models. arXiv preprint 2rx: 05883, 2024.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng- The Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled Decoding from Language Models. In Forty-first International Conference on Machine Learning, 5 2024. URL http://arxiv orgg.bs////1..1111111 17022
Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models, 2021. html.
ISSN 2640-3498. URL https://procced///
mlr.
Angus Phillips, Hai Dang Dau, Michael John Hutchinson, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Particle Denoising Diffusion Sampler. Proceedings of Machine Learning Research, 235:
40688-40724, 2 2024. ISSN 26403498. URL https: / /ttpxv/xivv////////////////a.av/ wwrg/bbs/2dee 242.222222....22.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning Text- Vext-to-Image Diffusion Models with Reward Backpropagation. 10 2023. URL https://arxiv. org/ abaaaaa//1110 03739v1.
Gabriel Raya and Luca Ambrogioni. Spontancous symmetry breaking in generative diffusion models. Advances in Neural Information Processing Systems, 36:66377 66389, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- Righ-Rosolution Image Syuthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-Jume:10074--18833 12 2021. TSSN 10636919. doi:
10.1109 1111099CVP5226SS.... 202..0202222222 URL httpp://arraiv.orr/bbb/////
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In 2023 IEEE/CVF Conference Computer Vision and Pallern Recognation (CVPR). TEEE, June 2023. doi: /2tp://vvv2222222222333.
URL httt::/dxx.di..ooo http///..119///PPPR.C /190///2222299 2023.02155.
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. In The Twelfth International Conference on Learning Represenlations, 10 2024. URL nttps://arxiv.org org/abs//30. 17347v2.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFF conference on computer vision and pattern recognition, pp- 815 823, 2015.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models, Advances in neural information processing systems, 35:25278-252999 2022.
Antonio Scloochi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 122(4))224 2025,
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition. arXiv preprint 1409. 1556, 2014.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-lime compute optimally can be more effective than scaling model parameters. arXiv preprint arXim:2408.0333 03311, 2024.
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping,
Abhinav Shrivastava, and Tom Goldstein. Measuring Style Similarity in Diffusion Models. 2024. URL https://arxiis wttp:///.rg/////a////oo//o httpab/a../..02220
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. ICLR 2021 9th International Conference on Learning Representations, 10 2020. URL 02502v4.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. Advances in Neural Information Processing Systems, 32, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott Reed, Dragomir Anguclov, Dumitru Erhan, Vincent. Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceealings of the TEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Masatoshi Uchara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding Reinforcement Learning-Baan Based Fine-Ture Tane-Tuning of Diffusion Models: Tutorial and Review. 2024a. URL https://arxiv. erg/abs/2407 13734v1.
Masatoshi Uchara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review, 2024b.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. 11 2023. URL attps://arxiv. httpa//wwww.rraaa//111...1... 12908vv....
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. Proceedings of the 28th international conference machine learning, 2011. URL
iist. www.pppuuueeeee . edu/
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion-based molecule generation with informative prior bridges. Advances in Neural Information Processing Systems, 35:36533 36545, 2022.
Luhuan Wu, Brian L Trippe, Christian A Naesseth, David M Blei, and John P Cunningham. Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. Advances in Neural Information Processing Systems, 36:31372 31403, 12 2023. URL
com/ wtt.://o//oe/tttttneeeeeeeeeeeee htt.:///11//eeeeeeeeeeeeeeeeeoie htt.d_fffiooonamppppea_
Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical Insights for Diffusion Guidance: Case Study for Gaussian Mixture Models. 3 2024. URL https://arxies wwwpsiii.orraaeeeeeee abs/2403 01639v1.
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. International Symposium Information Theory (ISIT), July 2024.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Kevin Yang and Dan Klein. FUDGE: Controlled Text Generation With Future Discriminators. NAACL-HLT 2021 2021 Conference of the North American Chapler of the Associalion for Computational Linguislies: Human Language Technologies, Proceedings of the Conference, pp. 3511 3535, 4 2021. doi: 10. 10.18653/v1/ 2021. ma2.mnacllmain.221 URL http://aare wttp.pa//g...../////2222 naacl naacl-main. mmain.mmnn... 276.
//xx. doi org/10. 186..111221///
Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stelano Ermon. TFG: Unified Training-Free Free Guidance for Diffusion Models. 9 2024.
Mao-Chuang Yeh, Shuai Tang, Anand Bhattad, Chuhang Zou, and David Forsyth. Improving style transfer with calibrated metries. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 3160 3168, 2020.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Froedom: Training-free energy- guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 23174 23184, 2023.
Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement, 7 2023. URL https://arxii...ot hha//337.00055v/
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and aligument using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499-1500 2016.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to- Diffusion Models. Proceedings of the IEEE International Conference on Computer Vision, pp. 3813-3824, 2 2023. ISSN 15505499. doi: 00.1109/CCCV510000 200.000......0.5
CRL https: http//aaavvvv./aaaaaa// wrg.abs/232....0e///// 05543v3.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Lue Van Gool. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 1219-1229, 2023.
19

---

Published in Transactions on Machine Learning Research (04/2025)
A Proof of Theorem 2.1 Proof of Theorem 2.1.
p(u 20, c)
JA(Ie, c) = Ex, 1m
A (V(r-iipP,) - V (1 c)) + log
(13)
(
I2, c)
pl 1 c) e (21V
-
log
(14)
1 TT (2)))
pl 1 Iee c) ( 1iD10)
= =Eg, 100
log
+ log (m
(15)
π ( 100-115111 c)
p(az-1/00 c) 6 (
= -ETLNNT
log
+ (V(xppp)))
(16)
Now, let.
(ee ( Zx(xe,
px p =
(17)
where the normalizing constant Z (x( is given by
Za( c) = En, 1<p [p(sa- e)e ((e-iii)))))))))
(18)
Putting back Eq. 16, we get
1 c)
(2 c)
Er, i
log π( Tr; c)
Z (I)e c) + +aV(zrppc))
(19)
PA [2, c)
Ex
log
log Zx(zer +V((EED.))
(20)
T( [r. c)
log px |2, c)
EER- wat
log ZA c) AV(xp.c)
(21)
T(x;-1| c)
log * se, c)
+og (x c) (V(G
(22)
( c)
KL c) px 22, (), log (z,, (V(zpp.c)
(23)
Eq. 23 is uniquely maximized by (3 c) = (x
20

---

Published in Transactions on Machine Learning Research (04/2025)
B Sampling from Optimal Model using Langevin Dynamics
Given the optimal policy given in Eq. 10, our goal is to now sample from n instead of p. However, given only P: it is difficult to sample from this optimal policy. To overcome this problem, we look at the score-based sampling approach as in NCSN (Song & Ermon, 2019). Starting from an arbitrary point or, we iteratively move in the direction of 22, log x (xt); which is equivalent to Va, (xt))( We can derive an equivalent form:
(x();) p(z,) (
(24)
log (xt) logp(rt) (V(x.) - log Zu
(25)
logpx (zz) 2x, logg(z:)) x2) + (o. v. log Zs
(26)
(x(x,,t) se(x,, AV 22 V(z)).
(27)
As the above derivation is limited to stochastic diffusion sampling, we loverage the connection between diffusion models and score matching (Song & Ermon, 2019):
Va, logp(rs)
Et.
(28)
VI - aq
Similarity with classifier guidance. Starting from arbitrary point 27, iteratively move in the direction of Va, log p(z(y)) We can derive an equivalent form:
p(x)l)) p(ylz()p(az)
(29)
7
(ogp(()l)) logp(x;) + (ogp(y())) log Z Va, logp(er)y) = V., loggo(()))) + Vn, log xx) V., log Z
(30)
(31)
(x = -so(xe, t) + V., logp(u)(xs)
(32)
21

---

Published in Transactions on Machine Learning Research (04/2025)
C CoDe with Image-Conditionngg CoDe(1)
For (T+I)2I cases, where the reward depends on target image, the reward distribution deviates siguificantly from the base distribution p. Here, sampling-basee bbased approaches would require a relatively larger value of N to achieve alignment. To tackle this, a reference target image Fref, such specifie style even stroke painting, is provided additional conditioning input. Inspired by image editing techniques using diffusion (Meng et al., 2021; Kooh- payegani et al., 2023), we add partial noise corresponding to only + = 17 x T (with E (0, 1)) steps of the forward diffusion process, instead of the full noise corresponding to T steps. This is ratestrated in line and 3 of Algorithm 2. Then, starting from this noisy version of the reference image Σ,,, (nDe(n) progressively denoises the sample for only steps to generale the clean, reference-aligned inage (lines 5 to 10). Specifically, for each block of B steps, we unroll the diffusson model N times independently (Algorithm 2, line 7). Then, based on the value function estimation (line 8), select the best sample (line 9) to continue the reverse process until we obtain a clean image at 0. key ad advantage of CoDe(1)) (1)) is its ability achieve similar alignment-divergence trade-offs while using a significantly lower value of N,, as is demonstrated in Section 5. Note that the inner loop of CoDe(1) (lines 5-10) runs for steps (instead of T) due to adjustable noise conditioning discussed in the following. For the sake of brevity, assume T to be divisible by B; otherwise, block. By conditioning the initial noise sample on the reference image Tref, we can generate images that better incorporate the characteristics and semantics of the reference image while adhering to the text prompt c. As we demonstrate in our experimentation, threshold v provides an extra knob built in CoDe(1) allowing the user to efficiently trade off divergence for reward. Note that the reward-conditioning of the generated image inversely proportional the value of 77. Sctting n = results noise conditioning. A byproduet of this conditioning is compute efficiency, as is discussed in Section 8
Algorithm 2: CoDe(1)
Require p. T, N, B, &refi C, 8
Sample conditional initial noise:
2 ==クx
3 Hy Vā + VI- azz, = N(O, 1)
Initialize counter:
5 for t 1,.., do
if nod(s, B) then
Sample N times over B steps:
(o("]] 1.1.d I p(a; a/z;)
Compute values of all samples:
(o [r(B[zo]=] [[ [D DD
Select the sample with maximum value:
21 + argmax ((p
++1
12 end
Return: me
apply the same steps last but smaller
T and fully deactivates the
Given Theorem 2.1 and its proof in Appendix A, aim to sample from the reward-tilted posterior " ( e) in order to optimize the KL-regularized reward maximization objective Eq. (9). In order to perform CoDe's 's blockwise guidance, we:
1. sample from the prior p(zt-1/æz, c) using the denoising diffusion process, across all N streams,
and then compute the values of each the N samples using V(x,-1iP)
By doing so, the probability of the selected sample #,-1 with the highest value (in Alg. 1 Line 8, 2, Line 9) implicitly incorporates the prior distribution o/ c) as Monte- Carlo estimation technique. Additionally, selecting the highost value sample
argmax ((=[a):p.c)
(33)
is mathematically oquivalent sampling from the categorical distribution
Iz-1 Lid
Catepprica((Botttax///////// [1,
(34)
where the temperature 7 - 0.
22

---

Published in Transactions on Machine Learning Research (04/2025)
D Additional Results for Case Study
For the sake of completeness, we also study a variant of the GMM setting as discussed in Section 5, where the mean of the reward distribution is equal to the mean of one of the components in the prior distribution, as shown in Fig. 11. The prior distribution p(a:) is modelled as a 2-dimensional Gaussian mixture model (GMM) p(2o) = Ei= WyN[p,
with a = 2, [21, Ms [[5, 3), (3, 7), 7)), and La is an d-dimensional identity matrix, as shown in Fig. 11. All mixture components are equally weighted with, i.e., w1 = U2 = W3 = 0.33. In contrast to the previous setup, we define the reward distribution as p(r|æ) = A (,,, I. with μ., [5, and o, Basod on this sctup, we train diffusion model po(x:) estimate the prior distribution p(z). For this we use a 3-layer MLP that takes as input (z), and predicts the noise et- It. is trained over 200 epochs with 7 1000 denoising steps. Then, we implement CoDeto guide the trained diffusion model to generate samples with high likelihood under the reward distribution.
In Fig. 11, we present the trade-off curves for nor- malized expected reward (or win rate) versus KT. divergence by adjusting the hyperparameters the respective methods. For the quidance-based methods DPS and UG, the guidance scale is varied between and 50, whereas for the sampling-based methods BoN, SVDD, and CoDe, the number of samples N is varied between 2 and 500. Similar to the results in Section 5, we observe CoDe achieve the most favor- able trade-off between normalized expected reward and KT, divergence, with BoN performing closely behind. In the case of win rate vs. KL divergence, BoN demonstrates the best trade-off, consistent with findings from the literature Language Model (LM) alignment (Mrouch, 2024; Beirami et al., 2024; Gui et al., 2024). Furthermore, guidance-basee based methods tend to exhibit higher KL divergence, as they of- ten collapse to the mode of the reward distribution when the guidance scale is increased, leading to a re-
Prior: pix)
Reward: pirkx)
Posterior: p(xl) = pixlpfrixt
10
15
BoN
UG (Bansal, 2024)
CoDe (Ours)
-dr-
DPS (Chung, 2023)
SVDD-PM 2024)
0:000
(Normalized) Exp. Rewards
0.8
0.9-
Winn Rote
0.8
1.00
10000
0.4
0:03
duction in diversity among the sampled data points.
For both performance metrics, SVDD-PM PM achieves high expected reward or win rate but at the ex- pense of significantly increased divergence, even for smaller values of N. Whereas CoDe offers the widely sought-after flexibility, allowing users to balance the trade-off by adjusting parameters such N and B.
KL Divergence
Divergence
Figure 11: Setup (top row) and reward vs. divergence trade-off (bottom row). CoDe offers highest reward at lowest divergence with a much lower N than Bo
23

---

Published in Transactions on Machine Learning Research (04/2025)
E Additional Results for Case Study II
Here, we provide further details about the differentiable reward-guidance scenarios' quantitative evaluations summarized in Table 4 and computational complexity analysis in Table 5.
Further details on evaluation metrics. For computing I-Gram, we utilize VGG (Simonyan & Zissorman, 2014) Gram matrices of the reference and generated images to measure image aligument across all scenar- ios/settings, as commonly followed in the literature (Somepalli et al., 2024; Gatys et al., 2016; Yeh et al., 2020). Specifically, these are computed using the last layer feature maps of an ImageNet-1k pretrained VGG backbone (Simonyan & Zisserman, 2014). Image alignment between a reference, generated image pair is then measured by computing the dot product of their gram matrices. Further, we report recently proposed CLIP-based Maximun Mean Discrepancy (CMMD) (Jayasumana et al., 2024) as a divergence measure. It overcomes the drawback of FID stemming from the underlying Gaussian assumption in the representation space of the Inception model (Szegedy et al., 2015).
Qualitative performance. Let us start with style guidance in Fig. 12. As can be seen, CoDe either stands on-par or performs better as compared to all other baselines in terms of capturing both, the style of the reference image and the semantics of the text prompt. This can be seen in comparison with UG for the text prompt of "portrait of a woman", where UG fails to incorporate the text prompt, but latches onto the style of the reference image. The results for face guidance with and without noise-conditioning are illustrated in Figs. 15, 16, respectively. It can be noticed that the poise-conditioned baselines capture the reference face much better than their non noise-conditioned counterparts. Morcover, in the case of noise-conditionin, BoN(n), SVDD-PM(7)) and UG(1) fail to meaningfully capture the semantics of the text-prompt, particularly for "Headshot, of woman made of marble". However, CoDe(1) captures both, the reference face and the text, prompt. In the case of the other text prompt "Headshot of person with blonde hair with space background", SVDD-PM(1) and CoDe(1) offer best results as compared to other baselines. Finally, the results for stroke guidance without noise-conditioning are illustrated in Fig. 17. It can be seen that none of the baselines capture the reference strokes or their color palettes successfully, but only adhere to the text-prompt. This empirically corroborates the need for using noise-conditioning for guidance, when the reward distribution (strokes in this scenario) differs significantly from the base diffusion model's distribution.
Quantitative performance. In this section, we break down the quantitative performance of all methods across the three different differentiable reward scenarios of style, face and stroke guidance. We summarize the results in Tab. 6, 7, 8 with the first row corresponding to the base Stable Diffusion model and R: indicating the reward metric used for guiding the diffusion model. For differentiable guidance scenarios, observe best reward vs prrggonce/tett-alignne
tradcoffs using N, B = 100, 5 for CoDe, 1 = 0.6 for CoDe(1) in the style and stroke guidance scenarios, 1) = 0.7 for face guidance and v = 0.8 for compression guidance. Similarly for SVDD-PM and BoN, using N = 100 renders best results in terms of reward- alignnd generated images without over-optimization (low text alignment or high divergence from base distribution as shown Figs. 19, 20) for differentiable rewards. For compression guidance, we observe best results with N 100 for BoN and N == 40 for SVDD-PM, CoDe for the T2T scneario, and N = 100 for BoN(n), N for SVDD-PM()), N = 40 for CoDe()) for the (T+1)21 scenario. We observe best results for UG with guidance scale of 6, 6 forward gradient steps (m) and 12 refinement steps (K). These have also been reported to work best for style and face guidance by the authors (Bansal et al., 2024b) and we observe these to work best for stroke guidance too. DPS uses a gradient guidance scale of 1.5 for style and stroke guidance and a scale of 200 for face guidance. In the case of MPGD and Freedom used for aesthetic guidance, we observe best results for p = 12.5, 0.15, respectively, and vary o = [8.5, 10.5, 12.5, 15.5, 17.5] for MPGD and p = [0.1, 0.15, 0.2, 0.25, 0.3] for Freedom.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Ref. img
(2021)
DPS (2023)
SVDD-PM (2024)
UG (2024)
CoDe (Ours)
A colorful photo of an Eiffel tower
portrait of woman
Figure 12: Quality evaluation across methods for style guidance without noise-conditioning.
(Normalized) Exp. Reward
(Normalized) Exp. Reward
1.5
1.5
00.00
100
1.4
1.4
1.3
1.3
CoDe In]
CoDe Inl
1.2
BoN [x]
1.2
Bal [n]
DPS
SDEdk:
11
- SVDD
11
SVDD PM
- SVD0-PM (n)
SVDD-PM [x]
10
+ 0G
1.0
UG [n)
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
0
20
40
60
80
100
120
140
CMMD
2.0
-0-000 C
2.0
M000000
1.8
1.8
1.6
Win Rate
1.6
CoBe
-0- Cofe
1.4
-
CoDe
1.4
- CoDe (n]
S0Edi
SCE-I
1.2
SV2D PM
1.2
SVDD-PM
SVDD-PM
1.0
UG
2 UG (n]
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
20
40
60
80
100
120
140
FID
CMMD
Figure 13: Reward vs. divergence Lrade-off curves for style guidance.
25

---

Published in Transactions on Machine Learning Research (04/2025)
Style Guidance. The results summarized in Table 6. Compared to the sampling- based guidance counterparts BoN and BoN(1); CoDe achieves higher reward at the cost of slightly higher divergence (FID and CMMD), with and without the noise-ondditionngg, Yet, with a slightly smaller reward CoDe, CoDe(1) offers a better performance than SVDD-PM, SVPP-PM(1) across FID, CMMD and T-CLIP. Compared to guidance-based counterparls such as DPS, DPS(n) and UG, UG(1), CoDe, CoDe(n) offer a better trade-off terms of reward vs base distribution divergence and reward vs text, image alignment. This also illustrated in Fig. 13 where CoDe(1) consistently outperforms UG, UG(n) in terms of image alignment (normalized expected reward well as win rate), while offering lesser divergence w.r.t. both FID and CMMD.
Table 6: Quantitative metrics style guidance.
R1: Style Guidance
Method
T-CLIP (f)
I-Gram
Base-SD (2021)
1.0
1.0
1.0
1.0
1.0
SDEdit (2021)
1.22
3.25
67.75
0.90
1.51
BoN (2022)
1.14
1.30
2.25
0.99
BoN (n 0.6)
1.34
3.36
84.02
0.87
1.57
SVDID-P (2024)
1.44
1.81
10.93
0.99
1.6
SVDD-PM (7 = (20)
1.60
4.16
96.52
0.82
3.5
DPS (2023)
1.22
1.29
5.46
0.99
DPS (1 (26))))
1.29
3.31
90.06
0.83
2.3
UG (2024b)
1.39
4.27
91.13
0.82
2.9
UG (n 0.7)
1.37
4.43
103.6
0.79
3.5
CoDe
1.34
1.49
7.40
CD 0.6)
1.52
3.64
84.45
0.86
3.2
26

---

Published in Transactions on Machine Learning Research (04/2025)
-6- Code
Code
(Normalized) EXp. Reward
DaGe
(Normaliized) Exp. Reward
BoN
CoDe (n]
SVOD-M
1.5
SVOD PM [d]
1.4
DPS
[n)
SOEdit
SVDD-FF
1.3
1.3
SVDD-PM [g]
UG
1.2
1.2
11
1.1
1.0
1.0
1.0
1.2
1,4
1.6
1.8
2.0
10
20
30
40
50
60
70
FID
CMMD
0:0
1.8
1.8
+ Cobe
+ CoDe
[6]
Win Rate
Bok [n]
Win Rate
1.4
DP5
1.4
CaDe [o]
SVDD-PM
CoDee
1.2
SVDD PM in!
1.2
DPS
SDEdi
1.
1.0
SVDD
[x]
UG
0.8
0.8
UG [n]
1.0
1.2
1.4
1.6
1.8
2.0
20
30
40
50
60
70
FID
CMMD
Figure 14: Reward vs. divergence trade-off curves for face guidance.
Ref. img
BoN(n) (2021)
DPS(n) (2023)
SVDD-PM(n) (2024)
UG(n) (2024)
SDEdit(n) (2021)
CoDe(n) (Ours)
Headshot of a woman made of
marble
Headshot of person with blonde hair with space background
Figure 15: Quality evaluation across methods for face guidance.
27

---

Published in Transactions on Machine Learning Research (04/2025)
Ref. img
BoN (2021)
DPS (2021)
SVDD-PM (2024)
UG (2024)
CoDe (Ours)
Headshot of a woman made of
marble
Headshot of a person with blonde hair with space background
Figure 16: Quality evaluation across methods for face guidance without noise-conditioning
Face Guidance, We summarize the results in Table As the rewards are negative, we first compute the negative log of the reward values and then normalize it with respect to the base.
Table 7: Quantitative metries for face guidance.
R2: Face Guidance
Method
Rew. (1)
FID (
CMMD (4)
T-CLIP (t)
I-Gram (1)
Base-SD (2021)
1.0
1.0
1.0
1.0
1.0
SDEdit (2021)
0.99
1.79
34.91
0.89
1.74
BoN (2022)
1.08
1.22
2.52
0.99
1.0
BoN (1) 0.7)
1.08
1.82
35.3
0.88
1.8
SVDD-PM (2024)
1.42
1.42
9.67
0.97
0.74
SVDD-PM (1) = 0.7) (2024)
1.70
2.07
48.22
0.86
1.77
DPS (2023)
1.04
T.09
1.36
0.99
1.03
DPS (n = 0.7) (2023)
1.21
1.71
33.21
0.86
1.68
(2024b)
1.66
1.69
29.76
0.86
1.06
UG 0.7)
1.77
1.94
61.27
0.85
1.78
CoDe
1.30
1.25
6.76
0.98
0.91
CoDe(1) 0.7)
1.5
1.86
42.40
0.88
1.91
Compared to BoN, BrN(r) and DPS, DPS(1), CoDe, CoDe(1) provides higher rewards but also with higher divergence (FID and CMMD). Although SVDD-PM, SVDD-PM(1) and UG, UG(n) achieve higher rewards, CoDe, Co)e offer a better trade-off in terms of FID, CMMD and T-CLIP. Moreover, CoDe offers the best image-aligument in terms of I-Gram as compared to all other baselines.
Additionally, CoDe(n) provides competitive results as compared to UG, which is the sccond-best method while offering better prompt alignment reflected in higher T-CLIP score. We draw similar conclusions from the reward divergence curves presented in Fig. 14, where CoDe(r) achieves competitive rewards compared to UG. UG()), SVDD-PM, SVDD-PM(y), but on-par win rates as compared to UG, at the cost of slightly higher FID and CMMD scores.
28

---

Published in Transactions on Machine Learning Research (04/2025)
Ref.
BoN (2021)
DPS (2021)
SVDD-PP (2024)
(2024)
CoDe (Ours)
The skyline New York City
photo of a medieval church
Figure 17: Quality evaluation across methods for stroke guidance without noise-conditioning
Stroke. As shown in Table. 8, among the sampling based methods, CoDe provides better results than BoN in terms of expected reward and FID while maintaining the same T-CLIP score. Although UG and SVDD-PM offer higher rewards, CoDe offers lower divergence (FID and CMMD) and better T-CLIP scores. Overall, we observe that CoDe(1)) has the highest rewards while offering competitive FID, CMMD and T-CLIP.
Table 8: Quantitative metrics for stroke generation.
R3: Stroke Generation
Method
(J)
(t)
(f)
Base-BD (2021)
1.0
1.0
1.0
1.0
1.0
SDEdit (2021)
2.79
145.6
0.90
BoN (2022)
1.25
1.05
4.5
0.99
1.12
BoN (g 0.6)
170
SVDD-PM (2024)
1.56
1.04
12.0
0.99
1.38
SVDD-PM (q= 0.6) (2024)
187.1
0.85
DPS (2023)
1.34
1.04
14.0
0.97
1.13
DPS (1= 0.6) (2023)
195.0
UG (2024b)
1.55
2.78
78.0
0.88
1.63
UG 0.6)
236.5
CoDe
1.41
0.78
6.5
0.99
1.38
CoDe(1) 0.6)
178.5
0.87
29

---

Published in Transactions on Machine Learning Research (04/2025)
1.35
1.35
(Normalized) Aesthetic Reward
(Normalized) Aesthetic Reward
1.30
1.30
1.25
1.25
0000
1.20
1.20
1.15
1.15
CoDe
CoDe
1.10
1.10
BoN
BoN
UG
1.05
1.05
1.00
Freedom
1.00
Freedom
0.88
0.90
0.92
0.94
Base FID
T-CLIP
1.0
100-0-99-00
1.0
4.4
0.9
0.9
Win Rate
0.8
Win Rate
0.8
0.7
0.7
CoDe
CoDe
- BoN
BoN
0.6
UG
0.6
UG
MPGD
MPGD
Freedom
Freedom
0.5
0.5
2
5
0.88
0.90
0.92
0.94
0.96
0.98
1.00
Base
T-CLIP
Figure 18: Reward vs. divergence Irade-off curves for aesthetic guidance.
Aesthetic Guidance. Given the tradeoff curves in Figs. 18, we observe that CoDe offers better or on par rewards as compared to MPGD (He et al., 2024) for smaller FID and higher T-CLIP scores, thus offering a better reward vs divergence tradeoff. When compared to Freedom Yu et al. (2023) and UG Bansal et al. (2024b), CoDe achieves competitive or lesser rewards but offers better text alignment (T-CLIP) and lower divergence from the base distribution (FID, CMMD). This also corroborates in Fig. where UG generates aesthetic images that do not completely adhere to the text- leading to reward over-optimization.
30

---

Published in Transactions on Machine Learning Research (04/2025)
Computation Complexity. We present breakdown of the computational complexities of all baselines across each of the guidance scenarios. DPS is considerably faster across all three generation scenarios among the gradient-based based guidance methods. This is due to the m gradient and K refinement steps used in UG, which are not used in DPS. The difference is more pronounced in the case of stylc- and stroke guidance as UG uses a higher number of gradient steps m. Further, among the sampling-based approaches, SVDD-PM is an order of magnitude slower than BoN as it applies token-wise guidance. On the contrary, our blockwise approaches CoDe, CoDe(1) are more efficient than UG and SVDD-PM and closely follow BoN.
Table 9: Computational Complexity
Runtime [scc/img]
Methods
Inf. Steps
Rew. Querics
Style
Face
Stroke
Base-SD 2021
T
14.12
14.12
14.12
BoN 2022
NT
N
266.02
268.43
265.86
SVDD-PM 2024
NT
NT
1168.74
1859.67
1169.68
DPS 2023
T
T
62.52
122.21
61.83
UG 2024b
mKT
mKT
1588.41
543.12
1592.89
CoDe
NT
NT/B
441.81
583.12
442.08
CoDe(1)
NqT
NqT/B
331.42
403.19
274.56
31

---

Published in Transactions on Machine Learning Research (04/2025)
F Details for Estimating KL Divergence
To compute the KL divergence between the guided and the base diffusion model, we draw on some cxisting results that give us an upper bound on the KL divergence between CoDe and the base diffusion model, which is given by the following:
Lemma F.1.
KL(CoDe(N, B) Base) < log..N N - 1 x
T B
(35)
N
Proof. The proof follows the same lines as (Beirami et al., 2024, Theorem B.1), with the exception that we need to resort to (Mrouch, 2024, Theorem 1) to bound the KL divergence cach intervention.
For BoN where the block size B =, the KL. divergence is upper bounded by
KI(BoN(NN) Base) log N N
N
which directly implied by (Mrouch, 2024, Theorem 1) as well. For SVDD-PM where 1, the KL. divergence upper bounded by
N-1 x T.
N
KL(SVDD-PM(N)
|| Base) < (log N
N
Since the coise-conditionns variants of these methods only denoise for 1/ll steps instead of the full T steps, the KL divergences are upper bounded using
KT(CoDe(N, B, n) Base)= S log N
-
nT
(36)
B
N -
KI((BN((N,)) n) Base) & log N
(37)
N
N 1
KL(SVDD-PM(N, 7) Base) < log
nT.
(38)
F.1 Numerical computation of KL divergence for Gaussian models (Case Study I)
In Section 5, to estimate the KL divergence between the base and guided models, first generate 1000 samples from the base diffusion model and the reward guided model each. Then assuming Gaussian densities for both, we compute the mean and variance for each of the distributions and then use the closed-form expression to calculate the KL divergence between two Gaussians. We notice that in this setting when we reach the degeneracy limit, the bounds suggested by Lemma F.1 are loose, particularly for all SVDD-PM experiments Section This known issue with these KI. bounds and has been discussed by Beirami al. (2024).
32

---

Published in Transactions on Machine Learning Research (04/2025)
G
General Guidelines for Setting CoDe's N, B,
CoDe utilizes three parameters N, B, ry in order to guide the diffusion denoising process towards reward-tilted posterior. The interplay between these three parameters has been demonstrated through various reward vs divergence tradeoff curves, reward vs text alignment tradeoff curves (Figs. 13, 14, 18) and performance vs efficiency tradeoff curves (Fig. 10). In this section, we discuss the impact of each parameter on the guidance process and then provide a general set of guidelines on how to choose these values based on different tasks.
~,
Intuitively, N and B impact the exploration of the prior distribution thus controlling the chances of sampling from a higher reward region (modes of the reward-distriutiion while denoising.
Practically, increasing N leads to higher rewards or more reward-aligned generated images. However, increasing N also leads to a higher divergence from the base distribution (FID, CMMD, KL Diver- gence), lower text-alignment (T-CLIP) and a linear increase in computational complexity (inference steps and reward queries).
On the other hand, reducing B increases the number of times the denoising process is diverted towards a high reward region in its distribution thus increasing reward-aligument in generated images. Reducing B also leads to higher divergence from the base distribution (FID, CMMD, KT. Divergence), lower text-alignment (T-CLIP) and linear increase in computational complexity (inference steps and reward queries).
The divergence increases logarithmically in N (Eq. (36) and the compute increases linearly in N (Tab. 9).
On the other hand, divergence and compute both increase exponentially as B reduces (Eq. (36), Tab. 9).
n: Intuitively, 17 controls the degree of conditioning of the input reference image on the generated image. For a smaller 17, the denoising process starts from a slightly noised version of the input reference image and only denoises the image for vT steps instead of the full T steps, thus also reducing the total number of steps that could lead
aegard-aligmment (Section C Alg. 1).
Thus, in practice, reducing r) leads to an increase in reference image aligmment and reduction in reward and text alignment. In cases where the reference image is sampled from the reward distribution (style, face and stroke guidance in (T+I)2I settings with CoDe(n)), reducing 17 leads to an increase in reward alignment.
The computational complexity varies linearly with v7-
Depending on the nature of the task and the divergence the reward distribution from the prior, the guidelines mentioned above can be used mecraas//ecrccse N, B, n for the desired tradcoffs.
33

---

Published in Transactions on Machine Learning Research (04/2025)
H
Adaptive Control on N, B for CoDe
Various related works (Raya & Ambrogioni, 2023; Kim et al., 2023; Sclocchi et al., 2025) have empirically demonstrated that different denoising steps have varying impacts on the final output. The early stages primarily determine the overall structure of the image, whereas the latter stages refine the fine-grained details. Given this observation, CoDe could potentially improve its performance by incorporating an adaptive control mechanism that adjusts its blockwise sampling strategy based on the denoising stage. In order to asses whether this adaptive strategy could help CoDe further improve its performance, we perform ablation experiment by varying N, B across the first half Ti [500, 250) and second half T2 E (250, 0) of the diffusion denoising process, for the T2I compression guidance scenario. We experiment on total of four cases by varying N, B across the two halves of denoising. In the first two cases, N = 40 throughout denoising and B = 1 for Tí, B = for T2 or B = for T1, B = for T2 (rows with = 40, B1 = 1, Bg = 5 and N 40, B = 5, B₂ = in Tab. 10). In the next two cases, B throughout denoising and N = 40 for Ti, = 20 for T2 or N = 20 for Ti, N observe that hyperparameters that enforce stronger reward-guidance offer higher rewards as compared to the ones which enforce it in the first half (Ti). N₁ = 40, No = 20, B offers lower rewards, lower divergence (FID, CMMD) and on-par lext alignment (T-CLIP) as compared to N = [20, 40), B = 5. Similar observations extend for ablating on B where N = 40, B = 1, B2 = offers lower reward, lower divergence (FID, CMMD) and slightly lower text alignment (T-CLIP) as compared to = 40, B = 5, B2 = 1. We would like to clarify that except for the experiments where we swap Ni and N₂ or Bj and B2, the ratio of N/B does not remain the same. Note that using a higher N or lower exert stronger inference- time guidance results in higher rewards and higher divergence compared to their adaptive control counterparts. These observations for using stronger guidance in the second half of denoising intuitively align with the task of generating compressible images, where the fine-gra fine-grained details and the texture of the image is to be controlled. But for other rewards that demand control over the structure and positioning of objects in generated images, stronger reward-guidance signal in the first half denoising might be more effective.
for T2 (rows with Ni = 40, N₂ = 20, B and Ni 20, N₂ = 40,
5 in Tab. 10). We
the second half of denoising (T2)
B throughout denoising
Table 10: Quantitative metrics for compression reward with adaptive control.
Compressibility Reward
T2I
Method
H.P.
Rew. (1)
FID (1)
CMMD (4)
T-CLIP (t)
Base-SD
1.0
1.0
1.0
1.0
BoN
N = 100, B
T
1.23
1.10
1.70
0.99
SVDD-PM
N 40,
1.83
2.86
61.75
0.88
CoDe
N 20, B
1.54
2.01
12.83
0.97
CoDe
= 40, =
1.65
2.12
32.70
0.95
CoDe
40,
Bg
1.69
2.36
42.51
0.94
CoDe
N 40, Bi = 5, B. =
1.76
2.41
48.30
0.95
N₂ 20, B
1.58
2.08
15.35
0.97
CoDe
Ni 20, N2 40, D
1.60
2.11
17.97
0.97
34

---

Published in Transactions on Machine Learning Research (04/2025)
Reward Over-Optimization in Compression Guidance for SVDD-PM(1)
Following section 6.1, Fig. 5, we demonstrate a few images generated in the compressibility guidance sccnario with SVDD-PM(1) for N = [20, 30, 40, 100], where reward over-optimization occurs. As can be seen in Fig. 19, higher values of N for SVDD-PM(1)'s 's guidance lead to degenerate generation images, where the text prompt and reference image alignment is compromised at the cost of high compressibility reward. The generated images roughly follow the color palette of the reference image but fail to meaningfully incorporate the style and aesthetics of the reference image. Moreover, the images also do not resemble natural images, empirically corroborating the high KI.-divergence w.r.t. the base distribution in Fig. 5.
SVDD-PM(n) Compressibility Reward Over-Optimization
Ref. img
N=20
N-30
N=40
N=100
ilama
wolf
Figure 19: Qualitative examples of reward over-optimized images from SVDD-PM(1) for N [20, 30, 40, 100), in the compression guidance scenario.
35

---

Published in Transactions on Machine Learning Research (04/2025)
J UG with a high guidance scale offers low text alignment
Following section 6.3, Fig. 9, we illustrate few generated samples of UG across four settings for style guidance with higher guidance scales of 12 and 24 to qualitatively corroborate their low text-alignment. As can be seen in Fig. 20, the generated images offer high alignment. with respect to the reference image but fail to incorporate any meaningful features of the text prompts. None of the generated images resemble the Eiffel tower or the portrait of woman.
Ref. img
guidance scale=12
UG guidance scale=24
A colorful photo of an Elffel tower
portrait of a woman
Figure 20: Qualitative examples of low text-alignment (T-CLIP) for UG with higher guidance scales, in the style guidance scenario.
36

---

Published in Transactions on Machine Learning Research (04/2025)
K Correlation of I-GRAM with I-CLIP Reward
The reason why we also use I-Gram to indicate reward-alignment (instead of only expected reward per scenario) in our evaluations is because the expected reward has already been seen by the model throughout the guidance process. Thus, we test all guidance methods on the unseen I-GRAM reward-alignment metric to provide a holistic evaluation. That being said, we do notice slight discrepancies in the behavior of the I-GRAM scores and the CLIP-image similarity reward due to the differences in what each of these metries capture. The I-GRAM score has been shown to capture similarities in style and texture of two images (Galys et al., 2016) whereas the CLIP-image similarity score mcasures semantic similarity between any images. To analyze the correlation between these two metric qualitatively, style guidance scenario in Fig. 21. As can be seen, for the first and second generated images, the reward and T-GRAM for the second image is higher than the first, indicating alignment between the two metrics. However, for the first and third image, the reward is higher for the third image but vice-versa for I-GRAM. Similar alignment and misaligument patterus can be observed for pairs of other images. We altribute this discrepancy to the texture and semantic differences of the generated and reference images. Additionally, confirm that I-GRAM can be used as evaluation metric for reward guidance using CLIP similarity, we compuled the Pearson Correlation coefficient between the two metrics across subset of all generated images in the style guidance scenario. We observe correlation coefficient of 0.87 between Reward and I-GRAM, indicating a positive correlation direct proportionality between the two.
present a few generated images for the
Reward: 0.42
Reward: 0.50
Reward: 0.45
Reward: I-GRAM: 92.14
Reward: 0,387
Ref. img
I-GRAM: 91.25
I-ORAM: 93.55
I-ORAM:
A colorful photo of Eiffel tower
Figure 21: Qualitative demonstration of Reward vs I-GRAM for different style guidance generated images.
37

---

Published in Transactions on Machine Learning Research (04/2025)
L Miscellaneous Results
In this section, we illustrate several additional generated images across all basclines and guidance scenarios. We also provide additional results for CoDe, C7De(T)) across various different reference images and lext prompt pairs, that are different from the ones already explored in the main manuscript, in Figs. 22, 23, 24.
Ref. img
DPS (2021)
UG (2024)
BoN (2021)
SVDD-PM (2024)
CaDa (Ours)
CoDe(n) (Ours)
A fantasy photo of lonely road
A fantasy photo of volcanoes
Figure 22: Quality evaluation across methods for style guidance on additional scttings without noise- conditioning.
a cat siting on the grass
Reference
a cat wearing glasses
a cat rides bicycle
a cat rides bicycle
CoDe(n) (Ours)
Figure 23: Quality evaluation of CoDe for style guidance additional settings.
38

---

Published in Transactions on Machine Learning Research (04/2025)
Reference
CoDe(n) (Ours)
Reference
CoDe(n) (Ours)
Headshot of woman in pencil sketch
Cartoon illustration woman
Figure 24: Quality evaluation of CoDe for face guidance on additional settings.
39


Reviews:
[{"review_id": "tfrWz6PfBn", "reviewer": "Reviewer_2j8L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*:  \n1. **Novelty**: The blockwise approach bridges the gap between BoN and stepwise methods, offering a flexible balance between computational cost and reward alignment.  \n2. **Theoretical rigor**: The connection to KL-regularized objectives provides a solid foundation, and the analysis of approximation error (via Tweedie’s formula) is well-motivated.  \n3. **Comprehensive evaluation**: Experiments span synthetic (GMM) and real-world (text-to-image, text-image-to-image) tasks, with both differentiable and non-differentiable rewards, showcasing broad applicability.  \n4. **Practical insights**: The noise conditioning mechanism (CoDe(η)) and ablation studies on block size (B) and η provide actionable guidelines for practitioners.  \n\n*Weaknesses*:  \n1. The method proposed in this paper is relatively straightforward; the truncation at each step does not seem very innovative, and the overall contribution is not particularly substantial.  \n2. As a work primarily focused on providing tradeoffs, it should more comprehensively present the performance–efficiency tradeoff curve space to clearly illustrate the actual contribution. However, the paper only reports efficiency tests under fixed settings, and it does not disclose the search cost used in performance comparisons, which leaves readers unclear about the actual contribution.  \n3. The experimental tasks selected in the paper are relatively simple.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The blockwise approach bridges the gap between BoN and stepwise methods, offering a flexible balance between computational cost and reward alignment. This paper provides a concise algorithmic implementation along with theoretical proofs to demonstrate the effectiveness of blockwise sampling methods. It evaluates performance on tasks such as style transfer, image editing, and GMM model fitting using metrics like I-Gram, FID, T-CLIP, and runtime.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the experiments involving quantitative metrics, the hyperparameters are not clearly specified. However, there are many aspects that can be tuned, and these additional searches could potentially improve performance. The paper should clearly indicate these, and ensure that the runtime in Table 2 remains consistent for a fair comparison.", "broader_impact_concerns": "No more concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rhF1e0pz0J", "reviewer": "Reviewer_5ydM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The proposed method effectively aligns diffusion models with rewards. The authors validate their approach through experiments on both Gaussian Mixture Models and text-to-image diffusion models, demonstrating its effectiveness. The experimental results support their claims.\n2. The proposed method is sound, and the paper is easy to follow.\n\nMajor Weaknesses:\n1. There is a gap between the theory of controlled decoding and the final algorithm. Equation (6) suggests that the probability is proportional to $p\\left(x_{t-1} \\mid x_t\\right) e^{\\lambda V\\left(x_{t-1} ; p\\right)}$, where Algorithm 1 in the pseudo-code only considers  $V\\left(x_{t-1} ; p\\right)}$. From the final algorithm, it seems that the approach simply performs  x_0-prediction at each intermediate step and calculates the final reward based on  x_0, which is not explicitly reflected in the theoretical formulation.\n2. The “best-of-N” strategy, which selects the sample with the highest reward, results in a lower I-Gram score compared to the baseline. This raises concerns about whether the reward is truly aligned with the evaluation metric, that is, if the evaluation metric can effectively show the alignment of the diffusion models.\n\nTypos:\nSeveral typos exist in the paper, such as “algnment” instead of “alignment” and “CoDeis” in the introduction section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the alignment between diffusion models and downstream tasks defined by rewards. The authors propose a blockwise sampling method applied during intermediate denoising steps, enabling alignment with downstream rewards. The key contributions of this work are:\n1. An inference-time sampling technique based on controlled decoding, allowing reward evaluation at intermediate steps without requiring full denoising.\n2. A blockwise approach that reduces the number of intermediate evaluations while maintaining generation quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See  Strengths and Weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BEAOEh8zJh", "reviewer": "Reviewer_qWvG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths: \n1. The paper presents Controlled Denoising (CoDe), an inference-time blockwise sampling method to guide diffusion model outputs without requiring gradient computations. This provides a lightweight alternative to finetuning and gradient-based guidance. \n2. CoDe achieves comparable reward alignment to best-of-N (BoN) sampling while requiring significantly fewer samples, improving efficiency over traditional BoN methods. \n3. Unlike gradient-based methods, CoDe supports both differentiable and non-differentiable rewards, making it more versatile for real-world tasks like image compressibility and stylistic adaptation.\n\n### Weaknesses:\n1. Different denoising steps have varying impacts on the final output. The early stages primarily determine the overall structure of the image, while the later stages refine fine-grained details. CoDe could potentially improve its performance by incorporating adaptive control mechanisms that adjust sampling strategies based on the denoising stage.\n2. CoDe is not sufficiently compared with the more advanced gradient-based models for differentiable rewards.\n3. The effectiveness of CoDe depends on key hyperparameters such as $N$ and $B$, but the paper does not provide a clear guideline on how to choose these values based on different tasks. Offering a structured recipe or heuristic for selecting $N$ and $B$ could greatly enhance the usability of CoDe.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents CoDe, a simple yet effective blockwise control method for diffusion models that optimizes reward alignment during inference while avoiding computationally expensive gradient steps. It achieves comparable or better performance than Best-of-N sampling while being much more efficient.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is better to add a citation or refrence for the claim in Theorem 2.1. \n2. The statement under Eq.(12) is not proper. In Universal Guidance and DPS, they also can use pretrained detection models to extract related features and also use Tweedie's formula to obtain $\\hat{x}_0$. In some sense, it can also been seem as a kind of reward function. Claim that they need to use training a separate model undermines their contributions. \n3. The samples drawn from $\\prod_{i=t}^{t+B} p(x_{i-1}|x_i)$ should be $x_{t+B-1}$ not $x_{t-1}$. \n4. For the comparison, there are also more advance gradient based methods, such as FreeDOM [1], SAG [2], Flowgrad [3], where some are based on SD 1.5 and have better performance than UG and DPS. A comparison with these stronger baselines would provide a clearer picture of CoDe’s advantages and limitations.\n5. Provide a structured recipe for selecting $N$ and $B$. \n\n[1] FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model\n[2] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method.\n[3] Flowgrad: Controlling the output of generative odes with gradients", "broader_impact_concerns": "The author present the broader impact in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper receives mixed reviews. Two leaning accepts and one leaning rejects. The main critiques include lack of novelty, need of more comprehensive evaluations and a potential issue related to the I-Gram score evaluation. The authors provided extensive responses, also fixed a bug in their evaluation which I believe addressed some of the major concerns. I'm also leaning to down weigh the importance of novelty as its flip side is simplicity which is a strong suit on its own. Overall, I think this work is worth including in TMLR with a minor revision by incorporating the reviewer feedbacks.
Decision: Accept (poster)