Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
Authors: Min Cheng
Fatemeh Doudi
Dileep Kalathil
Mohammad Ghavamzadeh
Panganamala Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Reinforcement Learning
Multi-Objective Finetuning
Primary area: generative models
TL;DR: A retraining free, RL based multi-objective alignment algorithm for diffusion model finetuning, with guaranteed empirical performance.

Abstract: Reinforcement learning (RL) algorithms have been used recently to align diffusion models with downstream objectives such as aesthetic quality and text-image consistency by fine-tuning them to maximize a single reward function under a fixed KL regularization. However, this approach is inherently restrictive in practice, where alignment must balance multiple, often conflicting objectives. Moreover, user preferences vary across prompts, individuals, and deployment contexts, with varying tolerances for deviation from a pre-trained base model. We address the problem of inference-time multi-preference alignment: given a set of basis reward functions and a reference KL regularization strength, can we design a fine-tuning procedure so that, at inference time, it can generate images aligned with any user-specified linear combination of rewards and regularization, without requiring additional fine-tuning? We propose Diffusion Blend, a novel approach to solve inference-time multi-preference alignment by blending backward diffusion processes associated with fine-tuned models, and we instantiate this approach with three algorithms: DB-MPA for multi-reward alignment, DB-KLA for KL regularization control, and DB-MPA-LS for approximating DB-MPA without additional inference cost. Extensive experiments show that Diffusion Blend algorithms consistently outperform relevant baselines and closely match or exceed the performance of individually fine-tuned models, enabling efficient, user-driven alignment at inference-time.

Published as a conference paper at ICLR 2026
INFERENCE-TIME MULTI- PREFERENCE ALIGNMENT FOR DIFFUSION MODELS
DIFFUSION BLEND:
Min Cheng
Fatemeh Doudi
Dileep Kalathil Texas A&M University
Texas A&M University
Texas A&M University
Mohammad Ghavamzadeh Qualcomm Al Research
Panganamala R. Kumar
Texas A&M University
ABSTRACT
Reinforcement learning (RI. algorithms have been used recently to align diffu- sion models with downstream objectives such as aesthetic quality and text-image consistency by fine-tuning them to maximize a single reward function under a fixed KL regularization. However, this approach is inherently restrictive in prac- tice, where alignment must balance multiple, often conflicting objectives. More- over, user preferences vary across prompts, individuals, and deployment contexts, with varying tolerances for deviation from pre-trained base model. We address the problem of inference-time reward functions and reference KI. regularization strength, can we design fine- tuning procedure so that, at inference time, it can generate images aligned with any user-specified linear combination of rewards and regularization, without re- quiring additional fine fine-tuning? We propose Diffusion Blend, a novel approach solve inference- siference--tme multi-preference alignment by blending backward diffusion processes associated with fine-tuned models, and we instantiate this approach with three algorithms: DB-MPA for multi-reward alignment, DB-KLA for KL regular- ization control, and DB-MPA-LS for approximating DB-MPA without additional inference cost. Extensive experiments show that Diffusion Blend algorithms con- sistently outperform relevant baselines and closely match the performance of indi- vidually fine-tuned models, enabling efficient, user-driven alignment at inference- time.The code is available on Github
multi-preeerence alignment: given
of basis
INTRODUCTION
Diffusion models, such as Imagen (Saharia et al. 2022a), DALL-E (Ramesh al. 2022), and Sta- ble Diffusion (Rombach et al. 2022). have demonstrated remarkable capabilities in high-fidelity image synthesis from natural language prompts. However, these models are typically trained on large-scale datasets and not explicitly optimized for downstream objectives such semantic alignment, aesthetic quality, or user preference. To address this gap, recent works have proposed re- inforcement learning (RL) for aligning diffusion models with task-specific reward functions (Uchara) et al. 2024al Fan et al. 2023 Black et al., (2024), where the core idea is to fine-tune pre-trained model maximize reward, while constraining the update to remain close the original model via a Kullback- (KL) regularization. The KL regularization term prevents reward overoptimiza- tion (reward hacking), and preserves desirable properties of the pre-trained model (Ouyang et al.)
2022
Rafailov et al. 2023 such as sample diversity and visual fidelity (Fan et al. 2023 Uehara
et al. 2024b).
While RL fine-tuning has improved alignment in diffusion models, it typically assumes a fixed re- ward function and regularization weight. This assumption restrictive in practice, where alignment must balance multiple, often conflicting objectives, such aesthetics and prompt fidelity, and user preferences vary across prompts, individuals, and deployment contexts. Static fine-tuning with fixed reward combinations cannot accommodate this variability without retraining separate models for each configuration (Wang et al.) 20246) Rame et al., [2023] Lee et al., 2024). Moreover, once trained, the trade-offs are fixed, precluding post-hoc adjustment. Similar issues arise with KI. regularization:

---

Published as a conference paper at ICLR 2026
(wi, (P2)
Pronpt
basis
DiflusionBlend
HL Trainng--
Bencber
Doncker F'inzured by
Dasik
Dengiser (Soom)
frrgs elgrad
Wytt
(Prezsined modd)
(a) Diffusion Blend
Overview
(b) RL finetuning
(c) Blanding Diffusion
Figure/ 1: (a). Overview of our Diffusion Blend Multi Preference Alignment (DB-MPA) Algorithm. Given basis reward functions and any user preference weights w = (wi, w2). DB-MPA generates images aligned with combined reward r(w) = wir1 + warz. (b) During the fine-tuning stage, DB-MPA gets an RL fine-tuned model corresponding to each reward function. (c) During the inference time, DB-MPA blends (mixes) the backward diffusion corresponding to each fine-tuned model according the user-specified preference w.
insufficient regularization causes reward hacking, while excessive regularization impedes alignment (Uehara et al. 2024b Liu et al. 2024). In practice, both reward and regularization weights are tuned via grid search, incurring significant computational cost and limiting flexibility.
These limitations motivate the need for a more flexible approach: inference-time multi-preference alignment, where the user specifies their preference vector, i.e., weights over set of basis reward functions such as alignment, aesthetics, or human preference, along with desired regularization strength that controls deviation from the pre-trained model. Crucially, this alignment must occur without any additional fine- tuning or extensive computation at inference time, which is essential for real-time and resource-constraing settings. Unlike trial-and-error prompt tuning, the ideal solu- tion should offer a principled and computationally efficient solution that can achieve Pareto-optimal trade-ofi across multiple preferences. This motivates us address the following questions:
Given a set of basis reward functions (r:),² and a basis KL regularization weight a, can we design fine-tuning procedure such that when the user specifies their reward regularization preferences through parameters w and X at inference time, the model generates images aligned with the linear reward combination r(w) = -Ee wor; and regularization weight (<A) a/λ, without requiring additional fine- ? In this work, we answer this question affirmatively and provide constructive solutions to it. Our main contributions are the following:
We theoretically show that the backward diffusion process corresponding to the diffusion model aligned with reward function r(w) and regularization weight can be expressed as the backward diffusion process corresponding to the pre-trained model with an additional control term that depends on r(w). We propose an approximation result for this control term, which enables us to express it using the control terms corresponding to fine-tuned diffusion models for the basis reward functions (r:)m= We also obtain a similar approximation result corresponding to the regularization weight a.
Leveraging the theoretical results we developed, we propose Diffusion Blend Multi-Preffrence Alignment (DB-MPA) algorithm, novel approach that will blend the backward diffusion pro- cesses corresponding to the basis reward functions appropriately synthesize a new backward diffusion process during inference-time that will generate images aligned with the reward r(w), where w is specified by the user during the inference-time. Using the same approach, we also propose Diffusion Blend KL. Alignment (DB-KLA) algorithm that will generate images aligned with a reward function 7 and regularization weight a a/x, where is specified by the user during the inference time. To reduce the computational overhead associated with DB-MPA, we propose Diffusion Blend Palti--Preeeeeccee Alignment- LoRA Sampling (DB-MPA-LS) algorithm that addresses the increased inference time issue while maintaining similar performance.
We provide extensive experimental evaluations using the Stable Diffusion (Rombach et al. 2022) baseline model, multiple basis reward functions, regularization weights, standard prompt sets, and demonstrate that our diffusion blend algorithms outperform multiple relevant baseline algorithms, and often achieve a performance close to the empirical upper bound obtained by an individually fine-tuned model for specific w and
2

---

Published as a conference paper at ICLR 2026
Tabe
Purm Soc
MCCL
Acsthetic
2
clo ooloned apaile
Inagg alignment score
(8) Visuel Comparison
(b) Pereto front
Figure 2: Comparison of DB-MPP MPA with baselines: Stable Diffusion v1.5 Rombach et al. 2022) CoDe (Sigg
al. 2025). RGG (Chung et al. 2023), rewarded soup (RS) (Rame al. 2023). and MORL RRojjers et al,
2013 Note that MORL included only to illustrate maximum achievable performance by an oracle
algorithm. See section 2 details. For arbitrary preference weight w, algorithms generate images aligned with r(w) wr1 + (1 w)r2, where T1 is text-to-image alignment and r2 is aesthetics.
(a) Images for w E {0.2. 0.5, 0.8). (b) Parcto-front comparison. DB-MPA significantly outperforms baselines and approaches the
MORL upper bound.
P
weight
weight weight 0.50
Prompt
Detratned
Stabie Diflusion
Denoiser (pretrained)
Deno our Finstuned by:
DiffusionBlend
x(1
A)
XÀ
pretrained
aligned with roward
3I and apcilo
seight
bockpook:
22
(a) DB-KL KLA overview
(b) Blending Diflusion
(c) Visual Comparision
Figure 3: (a) Overview of our Diffusion Blend-KL. Alignment (DB-KLA) Algorithm. Given an RL fine-tuned model for reward with KI. weight a, DB-KLA generates images aligned with KI. weight o/ for any specified modification factor (b) During inference, DB-KILA blends the hackward diffusion the fine-tuned and pretrained models according to A. which can be larger than (c) Visual comparisons with >-specific RL retrained models using text-to-ine rext-to-image-al mmnge-allgnment reward and X {0.2, 1.0, 2.0). DB-KLA achieves smooth control by adjusting the effective distance from the pre-trained model via X, generating images similar A- specific RL models without additional fine-tuning.
RELATED WORK
Finetuming-basee algorithms: Prior works align diffusion models via reward-guided finetuning.
Rewards-in-conee (Yang al. (024) conditions multiple reward types, DRaFT (Clark al.)
2024) uses weighted combinations during training, and (Hao et al.) 2023) applies RL with align- ment-aesthetic trade-offs. Parrot (Lee al., 2024) leverages prompt expansion, while Calibrated DPO (Lee et al.) (025) aggregates multiple reward models. Rewarded Soup (RS) (Rame et al. 2023) closest to us. linearly combining parameters from reward-specific models, whereas our DB-MPA
blends backward diffusion trajectories in principled way.
Guidance algorithmss: Gradient-based methods (Chung al. 2023 Yu al. 2023 Song et al.
2023 Bansal et al., 2023 He et al., 2024 Ye et al.) 2024) add reward gradients at each reverse diffusion step, and can handle multiple objectives (Han al. 2023 Kim et al. (2025 Ye et al. 2024). They require differentiable rewards and Tweedie-basee approximations (Efron) 2011). leading to noise and high cost. Gradient-free methods instead generate multiple candidates and select high- reward samples (Mudgal et al.) 22024; Gui et al. 2024: Beirami et al. 2024),
particle/value- guided search (Li et al.) 2024, Singhal et al., 2025 Singh et al. 2025). These avoid gradients but demand heavy sampling and reward access.
Multi- Saltti-Obeectvve RL (MORL): Approaches (Roijers et al. 2013 Yang et al. 2019 Zhou et al. 2022 Rame et al.) 2023) fine-tune separate model for each preference regularization weight. While theoretically optimal, inference-time RL is infeasible; even covering the weight space requires ex- ponentially many models. We thus treat MORL only oracle baseline.

---

Published as a conference paper at ICLR 2026
Multi- Multi- preference alignment in LLMs: Works such (Rame et al. 2023 ang al. [023 Shi al.
2024 Wang et al. 2024b Guo et al. 2024 Zhong et al. 2024b; Wang et al. (024a) extend RL
finetuning to LLMs. For KL-regularized alignment, DeRa (Liu et al. 2024) controls alignment by
combining logits from aligned and reference models. Our diffusion blend methods are inspired by
these but introduce inference-time preference alignment specifically for diffusion models.
LoRA composition for image generation: Recent work on multi-concept fusion in diffusion mod- els (Zhong al. 2024a; Zou et al. 2025) focuses on composing pretrained LoRA modules using fixed, heuristic scheduling to mitigate degradation when multiple concepts are combined. In con- trast, our DB-MPA framework blends searrd-aligned diffusion trajectories, enabling principled and interpretable trade-offs rather than heuristic mixing.
3 PRELIMINARIES AND PROBLEM FORMULATION
Diffusion model and pre-training: A diffusion model (Ho et al. 2020 Song et al.; 2021 ap- proximates an unknown data distribution Pdata by an iterative approach. It consists of forward process and backward process. In the forward process, a clean sample from the data distribution Pdata is progressively corrupted by adding Gaussian noise at each timestep, ultimately transform- ing the data distribution into pure noise. The reverse process involves training a denoising neural network to iteratively remove the added noise and reconstruct samples from the original data distri- bution. The forward process typically represented by the stochastic differential equation (SDE), dr, = = + + ((t)dw,, Vt E T).
where vo Pdata. B(() is predefined noise scheduling function, and represents standard Wiener process, The reverse process of this SDE is given by (Anderson) 1982. Song et al. 2021
dx. = 90)) 3(1)v,, log lo + v ((()dw,, Vt E [T.
(1)
where Pt denotes the marginal probability distribution of Te, FT is sampled according to stan- dard Gaussian distribution, and Va, (x(t)) represents the score function that guides the reverse process. Since the marginal density Pt is unknown, the score function is estimated by a neural net- work sp through minimizing score-matching objective (Song et al. (021) given by the optimization problem, arg ming E Fer. (- A(t)) Va, log (zo) - so(x,, 4) where s8(2,, t) is a neural network parameterized by 0 that approximates the score function and A(t) is a weighting function. In the following, denote the pre-trained diffusion model by the (backward) SDE
dr, = fore t)dt + (()) Vt E [T,0].
(2)
where f I) denotes the term !- B()) V() log px:)) in eq. I and o(l) denotes the noise scheduling function. We will use pere to denote the distribution of xa according to the pre-trained diffusion model given by the backward SDE in eq. (2)
RL fine-tuning of pre-trained diffusion model: Diffusion models are pre-trained to learn the score function, and are not trained to additionally maximize downstream rewards such as aesthetic score. Aligning pre-trained diffusion model with a given reward function r(-) can be formulated as the optimization problem maxpo Ex [r(z)]], with the initialization Pr pre
However, this can lead to reward over-optimization, disregarding the qualities of data-generating distribution po pre learned during the pre-training (Fan et al. 2023). To avoid this issue, similar to the LLM fine-tuning (Ouyang et al., 2022), KL-divergence term between the pre-trained and fine- tuned models is included as a regularizer to the RL objective (Fan et al. 2023), resulting in the diffusion model alignment problem:
max Eapepo [r())) (KT.(pollr5"))
Po
where is the KL regularization weight. Since directly evaluating KL divergence between Po and p is challenging, it to upperbounded it by the sum of the KL divergences between the condi- tional distributions at each step (Fan et al. 2023), resulting in the fine-tuning objective
4

---

Published as a conference paper at ICLR 2026
max E[((x0)) - (PelET
kL (1 Σ 1=7
(4)
where the expectation is taken w.r.t PP ( and 2T Pr. While the optimization is over a sequence of distributions, (pe) (pt)i-T- practice, we learn only the score function parameter 0 which will induce the distributions pt as pl. We assume that this fine-tuning objective will solve eq. (3) approximately, which will result in a fine-tuned model aligned with r and a. Similar to the notation eq. 2 we represent the backward diffusion process corresponding to this fine-tuned model as
dr, = ( t) + (t))dwe Vt E [T.0],
(5)
where f(ra) depends on r, ar The exact form of (ira)) is derived in proposition 1
The fine-tuning problem in eq. (4) typically solved using RL by formulating as an entropy- regularized Markov Decision Process (MDP) (Fan et al.) 2023 Uchara et al., 2024a). The state S and action A spaces are defined as the set of all images X. The transition dynamics at each step t is deterministie, P(( St, ax) = 5(5++1 = a,). The reward function is non-zero only for t = T and 13. The policy ne S + (AA) is modeled as a Gaussian distribution, matching with the discretization of the reverse process given in eq. (2). To align with the diffusion model notation, we define 8, = -r dr --- and IT. (at|))) = PT (TT---1 ( The initial distribution of the state 80 rr is standard Gaussian.
r(s)) = O, Vt E [o....,T T- -7-1]
The main issue with the standard RL fine- tuning (eq. Be. (4) is that the fine-tuned model is optimized for fixed ar, and the model is unchangeable after fine-tuning. So, it is not possible to generate optimally aligned data samples for another reward r or regularization weight a inference time. To address this problem, we follow the standard multi-objective RL formalism (Yang et al. 2019; Zhou et al. 2022), where we consider a
Problem: Inference-Time Multi-Prefence Alignment:
reward (r1, .2..... .. Tm) and assume that the reward represented by a linear scalarization r(w) -i=1 WiT's E A where A.. is the m-dimensional simplex. At inference time, the user communicates their preferences by specifying the reward function weight w. We also assume that the user specifies a regularization modification factor À to propose an effective regularization weight (A) = a/X In this paper, we address the following problem:
How do solve the alignment problem in eq. 33 for arbitrary reward function r(w) and regu- larization weight (A)) without additional fine-tuning at inference time? Note that w and X are user-specified values inference time.
4 DIFFUSION BLEND ALGORITHM
The alignment problem in eq. 3 is the same the one used for LLMs (Rafailov et al. 2023 Liu (ct al. 2022).. Recent results in LLMs, especially those that use the direct preference optimization approach (Rafailov et al. 2023). have leveraged the following closed-form solution eq. to develop alignment algorithms:
p 1 (r)( = ppee (zo) exp(r(2o)/a) / Z
where Z is normalization constant. However, in the case of diffusion models, directly sampling from ptar is infcasible for two reasons. First, computing Z is intractable, as it requires evaluating an integral over a high-dimensional continuous space. Second, unlike in the case of LLM where ppre for any token æ is available at the output layer of the LLM, diffusion models do not offer an explicit way to evaluate p2re(z) for arbitrary x. Instead, we can only sample from pPre by running a backward SDE. In other words, we will not be able to directly sample from pdar by tilting pare as given in eq. (6). even if the value of Z is known. RL fine-tuning achieves sampling from paar by learning model that can synthesize the backward diffusion specified by f(r.a), However, this would suggest an extensive RI. fine-tuning for different values of w and À in r(w) and (()). To address this, we describe an interesting mapping between frre and f(r.o), which we will then exploit to solve the inference-ting time multi-preference aligmment problem without naive extensive fine-tuning.
Let x ~ (P))(.), xha ~ (.). and (e,)T= be an independent, zero mean Gaussian noise sequence with probability distribution Peo i.e., & (.). Consider the forward noise processes
5

---

Published as a conference paper at ICLR 2026
x..T = = C2 and z= =b0 c,, and let p and pt be the marginal distributions of x. and It respectively. As standard in the diffusion model literature (Song et al. 2021: Ho et al. 2020). assume that under the forward noise process, the distributions of 27 and 2T are Gaussian. Let Pole denote the conditional distribution of xb
pre given xt pre. Then, we have the following result.
Proposition 1. Let f((s0) and frre be as specified in eq. ((re) (11, 1) p (xt, 1) - 3(1)u(ra)) (xr,1) 1), where
5 and eq.
respectively. Then,
r(x0)) a ]
((ria) (It, = Vu, log Eg pee Boo (H1x))
exp
Remark 1. We prove proposition [ following the SDE interpretation of diffusion models (Song al., 2021). and by showing that two SDEs initialized at time at pere and y and sharing the same forward noise injection process, can be reversed similarly. In particular, we show that the key parameters of the corresponding two reverse SDEs remain unchanged, except that the latter includes additional control term (()a) in the score function. We note that (Uehara al. 2024b) derives a similar result by analyzing the RL. objective in eq. ( and leveraging results from stochastic optimal control. In contrast, our approach analyzes the original alignment objective in eq. (3) and derives a simpler, first-principles proof without relying on stochastic optimal control theory, under the standard mild assumption that for large T, the terminal distribution of both forward SDEs is Gaussian.
We now consider an approximation @(rea)) to ((raa), motivated by the Jensen gap approximation idea that has been successfully utilized in algorithms for noisy image inverse problems (Chung et al. [223]] Rout al. 2023 2024). Let be a random variable with distribution p. For non- linear function f. the Jensen gap is defined as [[f(x)] - f[E[=]). In our case, we interchange the expectation [E|]] and the nonlinear function exp(.) in (maa) to obtain the following approximation:
r r(zo)
u ((ra) t) = a((a)(2 t) + A (ia)) t), where (xna)(x, t) = V 20 popoe (-Ix)
(8)
We now prove an upper-bound on the approximation error A (m:a (x, t) in eq. 8 under certain reasonable assumptions. Refer to appendix A.2 for more details.
Lemma 1. Assume r(uh" = F(x:"e t) p t),, where n(w, x: .) specifies the randommess
induced by the backward on diffusion process. Denote prtt as the conditional distribution of the random
variable == r(x," )/c given xr. Then,
IA (1.a) (1, < 1z,1(()) x (x,((x) + Lr,((x),
where
(o,(2)= = / B|ll Van(w, (w, x. t)]|3 x.ere = =x], LT2((z) (xr(exx(r(zz (xxx))))))/
pie
[lexp(((>="*) x x]
L,5(x)= = (1 ) supl Va ( 1. = 2) + log af = x)
Remark 2. In lemma [ the term I,,11 quantifies the local Lipschitz sensitivity of the stochastic component of R with respect to changes in Zy; L, denotes the conditional coefficient of variation (i.e.. the ratio of standard deviation to mean) of R given xı; and Lr,3 measures the deviation of the conditional distribution p(zp 1z) from pure shift family. A shift family (or location family) refers to class of conditional distributions where changing the conditioning variable results in a simple translation of the distribution without altering its shape, e.g., P(X = x y) = PP(X = x y) (Casella and Berger 2024). For a perfect shift family, we may write Lr,3 O. In diffusion models, pere
as / gets closer to 0, Poje (zo)2z)) becomes more deterministic and concentrates around w,, with variation of x, reducing to mean shift, resulting L,3 to get closer to 0. When the reward function is more predictable from the noisy image x, or becomes closer to 0, both Lr,1 and L.,2 will be small. Note that Lt.2 and Lt.3 will increase when the regularization coefficient a becomes very small, suggesting that our algorithm might fail when we decrease a dramatically. We note that (Uehara et al.] (2024b) exchanges the order of [[]]] and exp(.) under the assumption r(20) k(xz)) +c. where e i8 an independent noise term. This is consistent with our result, as would be easy to derive T 1 = L,3 under their assumption.

---

Published as a conference paper at ICLR 2026
The key motivation behind the approximation in eq. (8 is that now leverage the linearity of expectation available in alro)) to approximate f(rlw))aaA) in terms of f ((rva) i 1,..., m.
Lemma 2. Let f(rra) be as specified in eq. 5p Then, we have
p(r(w))))
(Ie, Σ
wf
w/
(Iz, t) + 3(t)
Σ w (1, (I, (2,
(9)
==1
((aa()))
= (1 - ( + X (xr,1) + 3(1) (1,()s)) (xr,
(10)
Using the result in lemma
2 we now introduce our diffusion blend algorithms, with pseudo code
provided in appendix B.2
Diffusion Blend-Multi-Preferene Alignment (DB-MPA) Algorithm: Our goal is to solve the alignment problem in eq. 3 for an arbitrary reward function r(w) with user-specified parameter w, without additional fine-tuning at inference time. This is equivalent to obtaining the diffusion term f(r(w)).) and running the backward SDE in eq. (5). At the fine-tuning stage (before deployment), independently fine-tune the pre-trained model for each reward (rx) with fixed o, obtaining m use lemma 2 to approximate f(r(w))))) (Iz, t) = >i= w; (f.f( (Iz, t), where each ((r.,)) is computed using the RL fine-tuned model 0r. We then generate samples by running the backward SDE in eq. (5)
RL fine-tuned models (0 by solving the fine-tuning objective in eq. (3) At inference,
Diffusion Blend-KL Aligmment (DB-KLA) Algorithm: Our goal is to solve the alignment prob- lem in eq. 3 for an arbitrary regularization weight a(A) with user-specified parameter X, without additional fine-tuning at inference time. This is equivalent to running the backward diffusion in eq. 55) with f(rsa(A)), At fine-tuning,
fine-tune the pre-trained model for reward r and regular- ization weight a, obtaining RL fine-tuned model d from (pre, At inference, we use lemma 2 to approximate ((raa())) t) s (1 X) (x, t) + (f(raa) (zze, f). where frre and f(r,a) are com- puted using (pre and (F), respectively. We then generate samples by running the backward SDE in eq. (5)
4.1
INFERRENCE TIME EFFICIENT VARIANT
While DB-MPA and DB-KL successfully achieve reward alignment through score merging, this approach requires evaluating all m diffusion models at each denoising step, resulting in mx com- putational overhead during inference. To address this limitation, we propose 1B-MMPA-with-LoR Sampling (DB-MPA-LS), a novel algorithm that approximates the score merging process by ran- domly sampling reward fine-tuned LoRA adapters at each denoising step with probabilities propor- tional to their assigned weights. This approach reduces the inference cost to that of the original pre-trained Stable Diffusion model, eliminating the multiplicative overhead inherent in inference- time realignment methods, including our DB-MPA and existing LLM variants Liu et al. (2024); Shi et al. 2024).. Note that this sampling approximation cannot be applied to DB-KL since the KL reweighting terms may be negative. The key insight is that unlike LLM realignment which mixes probabilities over discrete and finite tokens, diffusion models operate through continuous stochastic processes where the noise-adding nature enables different mathematical treatment, which we show by the following proposition.
For the Lipschitz continuous functions fi and fa. the following two SDE have the same marginal probability px! = Px? for V 1 E [0, SDE I is dx1 = (xf1 + (1 (x)x dtt adt)d with Xo Po- t T). and {wi] being the Winner process. SDE is dx? = (X)) + (21-)))2(Xx7))tt + (t)ddi, with Xo ~ po. t E [0, T). and {wi] being the Winner process, where Y, a Bernoulli random variable with probability to be 1 and probability
Proposition 2.
1 a to be 0, and Y, is independent of (x?); and Y, for any s + t.
Remark 3. Without loss of generality, we present the theoretical result for the two-reward case (m = 2). as the extension to arbitrary finite m rewards follows straightforwardly by replacing the Bernoulli variable with categorical random variable. Details of the proof are in appendix A.4

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
In this section, we present comprehensive experimental evaluations that demonstrate the superior performance of our DB-MPA and OB-KLA algorithms compared to the baseline models.
Reward models. We use four reward models in our experiments: (i) ImageReward (Xu et al.
2024
which measures the text-image alignment and is used in its original form; (ti) VILA (Ke et al. which measures the aesthetic quality of generated images and outputs in [0, 1], is rescaled to 2,2
2023
via r + 4r 2 to normalize its influence relative other rewards; and (zii) PickScore (Kirstain et al. 2023). which measures how well an image generated from a text prompt aligns with human preferences, is shifted by 19 match the scale of other rewards. (iv) We further test our algorithm on JPEG compressibiliiyy reward, which opposes aesthetics by favoring smooth images, enabling analysis of adversarial alignment.
Baselines: We compare the performance of our algorithms with the following baseline algorithms:
(i) Rewarded Soup (RS) (Rame al. ) (it) CoDe (Singh et al. 2025). a gradient-free guidance algorithm with -head search, where we use N 20 particles for the search and B = 5 look-
ahead steps. (iii) Reward gradient-baser basse guidance (RGG) (Chung et al. 2023 Kim al. 2025). and (iv) Multi-Objective RL (MORL) (Rame al. 2023 Wu et al. 2023). Details of these baselines are given in appendix [.. Note that we report MORL performance only as an oracle baseline, illustrating the maximum alignment achievable.
We use two benchmark datasets in our experiments. (i) We first seleet the color subset from DrawBench (Saharia al. 2022b), comprising 25 prompts of the full 183 prompts across 11 categories. RL training can reliably converge at this small-scale setup, which aligns with our theoretical assumption that RL converges the closed-form solution of eq. (3), while our DB algorithm is interpolating between those optimal solutions under individual reward functions. For evaluation, we generate a test set 1,000 prompts using pipeline similar to GenEval with ran- dom color-object combinations. The candidate lists of colors and objects are generated by GPT-4 (Achiam et al.] 2023) be semantically similar the training (implementationn details and full list in our code). (ii) We also validate performance of DB on the GenEval dataset (Kirstain al. 2023), which contains 550 prompts across six compositional tasks: single objects, two ob- jects, counting, colors, spatial positions, and color attribution. An additional 700 test prompts generated using the official GenEval prompt generation script.
Prompt datasets:
We use Stable Diffusion v1.5 (SDv1.5) (Rombach et al., 2022) as the base model for our experiments, which a text-to-ing pelttto-image image model capable of generating high- resolution images. We use the DPOK algorithm (Fan et al. (2023) for RL fine-tuning. For the experi- mental results given in the main paper, we use the 1000 test prompts. Details of the implementation, including training configurations and hyperparameters, are given in appendix B.3
Training and evaluation details:
5.1 DB-MPA ALGORITM RESULTS
Table Quantitative comparison of DB-MPA and baseline methods
SD
MORL
DR-MPA
DB-MPA-LS
RS
CoDe
RGG
w=0.0
-0,22
-0.14
-0.19
0.47
-0.19
0.47
-0.19
0.47
-0.19
0.47
-0.002
0.01
-0.16
0.42
0=0.2
.0.22
0.14
0.24
0.38
0.05
0.38
0.07
0.36
00.20
0.23
0.16
0.00
-0.25
0.32
Reward 0.5
-0.22
-0.14
0.34
0.14
0.27
0.21
0.31
0.14
-0.12
-0.04
0.30
-0.04
-0.08
0.14
0=0.3
-0.22
-0.14
0.43
40.11
0.36
-0.01
0.37
-0.04
0.21
-0.14
0.41
-0.10
0.21
-0.04
-0.17
-0.17
Inference Time (1 scefing)
5.46
5.46
11.11
5.64
5.46
185.26
121.58
We first consider two reward functions (m = 2), with as ImageReward (Xu et al. 2024). which measures text-image alignment, and 72 as VILA (Kirstain et al. 2023). which measures aesthetics. During the inference time. the user specifies preference weight E [0. 1] to obtain data samples aligned to the reward wr, + (1 w)r2. We fix a = 0.1 for these experiments, where a is the KL weight.
8

---

Published as a conference paper at ICLR 2026
Ansterk
Teclls Angreee
0.
Srable MfPfeddo
Agron Due
Figure 4: Illustration the smooth control of DR-MPA to generate images aligned with r(w) for any w e [0, 1]. DB-MPA generates images that are better aligned with both rewards, especially for w E [0.4, 0.8). RS generates images with wrong interpretation objects (orange) or missing objects (cellphone).
In fig. b)), we present the Pareto-front of DB-MPA against baseline algorithms under the Short- DrawBench setting, evaluated on the 1k test prompts. For DB-MPA and RS, we evaluate the performance for E {0.1..... 0.9). For other baselines, we evaluate their performance for E {0.2, 0.5, 0.8) due to their high inference cost. Our DB-MPA algorithm consistently outper- forms the baseline in all these experiments and achieves a Pareto-front very close to that of MORL, which represents the theoretical optimum obtainable by RL fine-tuning. We further evaluate the lightweight variant OB-MPA-LS, which achieves nearly identical Pareto performance while match- ing the inference speed of standard Stable Diffusion. As shown in fig. TT. the outputs of DB-MPA and DB-MPA-LS are also visually close. In table 1 we provide a quantitative comparison of this Pareto-front result. If take the weighted reward r(w) as a metric of comparison, for DB-MPA (0.42) has close performance to DB-MPA-LS 0.39 and outperforms RS, CoDe, and RGG by 3,92x, 1.95x, and 1.33x, respectively. table also shows the inference time comparison. DB- MPA uses two tinettune models, making its inference time about twice that of Stable Diffusion. CoDe and RGG, though dangle-mode methods, incur far higher costs due multi-partiche sampling and gradient steps. The lightweight DB-MPA-LS matches DB-MPA's performance while running at nearly the same speed as Stable Diffusion. As analyzed in appendix C.4 DB-MPA-LS main- tains performance close to DB-MPA as the number of rewards increases, with DB/LS ratios of the average gains under each fixed reward setting being 1.02, 1.15, 1.08 for 2, 3, rewards, while RS performance drops significantly (DB/RS 3.3, 6.4, 8.2).
0.5,
We further scale DB-MPA to the GenEval benchmark by fine-tuning models all 550 prompts. Evaluation on the 700 held-out GenEval test prompts (table 4 and its corre- sponding Pareto boundary in fig. 5) shows that both DB-MPA and DB-MPA-LS consistently dominate the baselines across all preference weights. We also verify scalability on a larger base model (SDXL) in appendix C.o where single-promph experiment demonstrates consis- tent performance gains.
DO HE- FOUn"
CoOe
RGG
Aesthetic score
0.0
fig. 4 illustrates the smooth control of DB-MPA in generating images aligned with r(w) for any w E [0, 1]. We use RS as the primary base- line due to the high inference cost of other meth- ods. DB-MPA consistently produces images bet- ter aligned with both r1 and r2, particularly for W E [0.4, 0.8). Additional results in appendix C.8 and appendix C.9 further confirm that DB-MPA generates images of higher quality than the baseline and comparable those from MORI..
Text-to-lmage alignment score
Figure 5: Pareto-frout comparison of DB-MPA and DB-MPA-LS algorithm with other baselines, evalu- ated on GenEval test prompt set.
9

---

Published as a conference paper at ICLR 2026
We further examine challenging case of conflicting rewards by introducing the JPEG compressibil- ity reward, which favors smooth, low-detail images and directly opposes the VILA aesthetic reward that emphasizes fine- tne-grained visual quality. As shown in appendix C.3 DB-MPA maintains supe- rior performance even under this adversarial setting. We also extend our study to the three-reward case (m 3) by incorporating PickScore (Kirstain et al. 2023), which measures human preference alignment. The corresponding results, provided in appendix C.5 validate the generalizability of our method to multi-reward settings while consistently achieving superior performance.
5.2 DB-KLA ALGORITHM EXPERIMENTS
(A-m (k- 6.7)
DB (
(
02
apple purple backpack.
An colored spndwich (a) degree of control of DB-KLA
[5] Numerical Comparison
Figure 6: (a) Images generated by DB-KLA for different values of A. We consider SDv1.5 as X = 0 (infinite regularization weight), and as we increase the aligned model moves further away from SDv1.5. These examples demonstrate that DB-KLA can smoothly control the level of text-to-image alignment by selecting different X, without any additional fine-tuning. (b) The quantitative comparison of DB-KLAA and A-specific RI. fine-tuned models among the test prompt Result for the train set is given in appendix D
For the DB-KLA algorithm, we first fine-tune the baseline SVv1.5 model with the ImageReward and KL regularization weight evaluation. In fig. ((c), we show that even without any additional fine-tuning, the images generated by DB-KLA are similar to those of X-specific RI. fine-tuned models. In fig. [a), we illustrate that DB-KLA enables smooth and continuous control over alignment strength via the rescaling factor X. Additional experiment results given in appendix D In fig. (/bb, we can observe that the average reward obtained by OB-KLA closely follows that the MORL retrained model. As observed in fig. a(),, in scenarios where the retrained model fails to fully align, DB-KLA with stronger alignment setting (A > 1) can generate more semantically accurate outputs, such as correcting object colors preserving scene elements. This highlights its potential as diagnostic tool for understanding and mitigating under- or over-optimization reward-guided diffusion finetuning.
0.1. using the Short- DrawBench prompts for fine-tuning and
CONCLUSIONS
We introduced Diffusion Blend, framework for inference-time multi-preferenee alignment in dif- fusion models that supports user-specified reward combinations and regularization strengths without requiring additional fine-tuning. Our proposed algorithms. DB-MPA, DB-MPA-LS, and DB-KLA, consistently outperform existing baselines and closely match the performance of individually fine- tuned models. Notably, DB-MPA-LS eliminates the linear scaling of inference time that plagues tra- ditional inference time realignment methods, enabling efficient multi-preference alignment at scale.
ACKNOWLEDGEMENT
This material based upon work partially supported by the National Science Foundation (NSF) under grants CNS-2328395, Army Contracting Command under Contract Numbers W911NF2520046, 191 and 191 and the US Office of Naval Research under contract 140014-24-122661
and NSF CAREER EPCN-2045788 2045788, the US
This research was conducted using the advanced computing resources provided by the Texas A&M High Performance Research Computing.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774 2023.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 843-852, 2023.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' Amour, Jacob Fisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint ar arXiv:2401.0188 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. The Twelfth International Conference on Learning Rep- resentations, 2024.
George Casella and Roger Berger. Statistical inference. CRC press, 2024.
Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations, ICLR, 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations, ICLR, 2024.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- lion, 106(496):1002211114 2011.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885, 2023.
Lin Gui, Cristina Garbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness best-of-n sampling. Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems, 2024.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi- objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1437-1454, 2024.
Xu Han, Caihua Shan, Yifei Shen, Can Xu, Han Yang, Xiang Li, and Dongsheng Li. Training- free multi-objectivv diffusion model for 3d molecule generation. In The Twelfth International Conference on Learning Representations, 2023.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36:66923-66939 2023.
Yutong He, Naoki Murata, Chich-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In International Conference on Learning Representations, ICLR, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840-6851, 2020.

---

Published as a conference paper at ICLR 2026
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoe parameter merging. arXiv preprint arXiv:2310. 1564, 2023.
Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. Vila: Learning image aesthetics from user comments with vision-language pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10041-10051, 2023.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Kyungmin Lee, Xiaohang Li, Qifei Wang, Junfeng He, Junjie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi-preferee optimization for aligning diffusion models. arXiv preprint arXiv:2502. 025888 2025.
Seung Hyun Lee, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang, Fei Deng, Glenn Entis, Junfeng He, al. Parrot: Pareto-optimal multi-reward reinforcement learning frame- work for text-to-ine -ottte generation. In European Conference Computer Vision, pages 462-478. Springer, 2024.
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Bian- calani, Shuiwang Ji, Aviv Regev, Sergey Levine, et al. Derivative-fre guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08233 08252, 2024.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares-López, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- time realignment of language models. International Conference on Machine Learning, pages 31015-31031. PMLR, 2024.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, Yaguang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from lan- guage models. In International Conference on Machine Learning, pages 36486-36503. PMLR, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01553728-57744 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights finetuned on diverse rewards. Advances in Neural Information Processing Systems, 36:7 10671095--7134 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-makinn making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
12

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkot- tai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems, 2023.
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36499 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-360 -64994, 2022b.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A Smith, and Simon Du. Decoding-time time language model alignment with multiple objectives. Advances in Neural Information Processing Systems, 37:48875-4988
2024.
Anuj Singh, Sayak Mukherjee, Ahmad Beirami, and Hadi Jamali-Rad Rad. Code: Blockwise control for denoising diffusion models. arXiv preprint wwXiv:2502.00966 2025.
Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06m 06848, 2025.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, pages 32483-32499 PMLR, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding rein- forcement learning-based fine-tuning of diffusion models: A tutorial and review. arXiv preprint M.XV:22407.13734 2024a.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous- time diffusion models as entropy-regullrized control. arXiv preprint arXiv:2402. 15194, 2024b.
Haoxiang Wang, Yong Lin. Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Iims for diverse user preferences: Directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 8642-8655, 2024a.
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Kumar Dubey, et al. Conditional language policy: A general framework for steerable multi-objective finetuning. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 2153-2186. 2024b.
13

---

Published as a conference paper at ICLR 2026
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine- line-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008-59033 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In International Conference on Machine Learning, 2024.
Runzhe Yang. Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training- freeingg-reeggeeee guidance for diffusion models. Advances in Neural Information Processing Systems, 37:22370-22417, 2024.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-frr energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 23174-23184, 2023.
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. Transactions on Machine Learning Research, 2024, 2024a.
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for 1lms. Advances in Neural Information Processing Systems, 37:75522-75558. 2024b.
Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized nat- ural policy gradient for multi-objective reinforcement learning. Advances in neural information processing systems, 35:13584-13596 2022.
Xiandong Zou, Mingzhu Shen, Christos-Savvas position for multi-concept image generation. arXiv preprint arXiv:2502.04923, 2025.
Savvas Bouganis, and Yiren Zhao. Cached multi-lora com-
14

---

Published as a conference paper at ICLR 2026
A TECHNICAL PROOFS
In this section, we provide the theoretical details referenced in Scction 4. We begin by analyzing the solution to eq. (3). followed by discussion of the Jensen gap approximation error introduced by interchanging the expectation operator Bi-] and the exponential function exp(-). We then describe how to approximate general function (((());a(A)) under the Jensen gap approximation. Finally, we provide analysis for the sampling-based approximation algorithm based on SDE theory showing its equivalent marginal distribution.
A. PROOF OF PROPOSITION
Proof of Proposition 7 By definition, (x)
Sy po (y) p
(2 po (wo.ae)) Pt (m))
y) dy. pl (x)
Sy mr (v)pe,(r v) dy. We also have, Poje (zoo with an appropriate normalization constant C,
(co)pen (w0-#o) Now, P3 (me)
(x)) = /
((())
po tar (s)pe, (x -
dy = C p0 (y)exp
Pe, (xr y) dy
(x) /, exp
r(y)
Pa
pre
(y) (y)pe. (x - v)
dy Cp. (x)
exp
((())
Por (ulz) dy
Pt (x)
a
r((z)) ]
= Cpp pr (x:) Bzompeo (112)11)) (11))
exp
a
From this, we get,
r(xp)
V log tar (x) Vx log (x) + Va log B (oo
exp
(11)
The result now follows from the definition of f(rsa) (2))) = B)t()2 - B(t) x. log pl (x,) and frre (x;) - - V n, log (x,))
We also present a more general formulation of proposition To offer a clearer understanding.
Proposition 3 (General statement of proposition [] Let X be random variable distributed ac- cording to Pu( Y be random variable distributed as go(y) = Cpo(y) exp(r(y)/or and Z be an independent noise. If the probability density of X Z pe- then the probability den- sity of Y + Z is (x(x) = Cpe(x))(exp((( "X + = x). The score of Y + is given by V. log Elexp( x(X) | +Z (z)
Proof. Note X + Z is distributed w.r.t. the probability density:
pr(x)) rolu)pz(x y)dy.
Density of Y + 7 is:
(x)
= / co(v)pz(x- - vidy
-c / po(s) exp( ((y) 1pz(x y)dy
exp(
=Cpi(t)) / expl r(u), Polu)pz(x -y) dy
{ploss)pZE ssds
-Opo(t)) / (exp( r(U)
IxxXx+ - (X y)dy
=CP:(T)LE eeppl (exp((((x) [(x)
X + 2 = 2].
15

---

Published as a conference paper at ICLR 2026
Remark 4. Consider the variance-exploding forward process proposed by Song al. (2021) i..
X, = =Xo+ Xo+var vā, with Z ~ >N(0, I). proposition] 3 iimpies that for two different initial distributions X Pn and Y qo Po exp(r/a), both following the same noise-exploding forward process, the marginal probability density (v( of Y, Yo + va, Z is equal to the product of the density of X2 and posterior mean term Rlexp(((Xx)/oo)
X, y]. Recall that the reverse process involves adding an extra score term Vz log (z) into the drift. proposition 3 suggests that approximating Ph exp(r/a) in the reverse process can be achieved by replacing the original score V (ogg((2) in Xt, with a shifted score Vx (x) + V. log E E(cxp(r(Xo)/o X = z].
The same analysis applies the variance-preserve 2020), defined as X, vā,Xo + vI- Z with Z ~ (0, I). In this case, the same conclusion follows by replacing the random variable in proposition with Vo,
forward process used in DDPM (Ho et al.
A.2 APPROXIMATION ERROR UPPER BOUND FOR EQ. 8
Lemma 3. Decompose the reward function into two parts
-(xxp" .) + n7(w,, x pro ;t)
.a "(ab"" [op"r]] only depends zhe and n(w, x ,t) contains randomness w induced from the noise injection prucess. Let PRte denote the conditional distribution of random variable R == r(=p" )/a given vt 2.. Then,
where F(x,"e,
r(
f(xtpe i))
||V log ( |/))/=? = x]] V L(( pro) )///x = xi]l S (,1(2) x (z,((x) + Le,(
where
(xx(x) = V B(I V V, x:"00 =) = z/,
[1,(2)) v (arleexpr(aa (/a /a) x)
E/exp(r(<b) pro) )/a)/= = æ]
L,,(x) = (1 + sup V= logPR) = x) + V log (r|= =]]]l-
Proof. To shorten the notation, denote random variable R == (=/"")/aa and constant Cj maxx/r(x)/ar -. Let PRaa be the conditional probability of It given
We first make one assumption about the boundary condition that the conditional probability density decreases exponentially fast lim PRI (F(ar O, and lim e" (RI((((z) 0. It is known that Sab-Gaussian distributions satisfy those assumptions with an exponentially decreasing tail p(x) S Ce-l#l" for large enough x. Remember that in our experimental setting, all reward models output in bounded range [-2, 2] and is fixed real number. Therefore, for the bounded random variable R, belongs to the sub-Gaussian distribution and satisfies our boundary assump- tion.
Let F(x)= = 2R)x = z] and g(x) = E[R|2," = æ], then
V, log [(x) Vag(z)= = v.l(x) Vag(())
F(x)
Ve exp(r) PRI (r/z)dr + exp(r) (PPma(r)))d
F(r)
V ( + rv VV
EN V pm expl (R) pap = x]
E(V pire Rla:pre = x]
F(x)
/
F(x)
exx((I)
/eep(r)V=pp((rll)) ( F(x)
E
-1
V 2pr
R
x = +
F(x)
16

---

Published as a conference paper at ICLR 2026
We first bound the first term 11. Decompose into two parts R F(x .t) (t,, t) where the
first part F(rt t) only depends on 2!!".. Note that E
(()p(()
F(x) -1/x, xx] El exp(
1 | 2] O.
|4
R
pre
= x
F(x)
R
V (a 1)) + V Var n(w, st 11)
F(x)
E
V Vapennlss xt 1) 20 =
F(x)
eR
xip = 2) x 1B I1 a nlw " .01B ap
E
-1
F(x)
(x) x (xr)
For the second term 1z. we note that 17 under the assumption proposed by [eeaaa et al. (2024b) that R = f(xz) + e can be decomposed to the summation of a function related to T2 and an independent noise e, which is induced by the translation invariance Pnje(r(z)) = Pmoise (r - r). Inspired by this observation, we define A v log PRepPRe (rlx)) + V, log Pata measure the shift from such translation invariant family (p Profse: s.t. (ni((((x)
Pootse (r - x))
S exp(r)pp)( 20) log (ogg
/ (rpmlrlx) log
F()) (r) [1) - V, ( (PRie(r)=)(A Pri
(exp(r)PR(((
F(z)
-Ep
pre =] |
Elr |zt pre = s]
J <xpp())pR)(((=>) v log PRI (rlwddr
(Ppn( Prae (ra
F(x)
where (r(r(r)) = (xp(r)pnn,((1)) is the reweighted probability. Under the boundary condition, we F(2)
can show that IF 1g = -
1g = / v VVDR((rle)dd / V.. (r(mn(((rl))) - Praje Prie(r (r)a))r
=|pR) -=000
f c (r))))) / / (xp(())x(((((c))) dr (exp()
IF
F(x)
F(x)
( (exp(r)pr)((11) (r/zz)
-=
V (exp(r))
F(z)
F(x)
Therefore, | = (A æ| -ETrA = =|]l| s (1+CR) sup,) ||Aell = Lxx(x))
A.3 PROOF OF L.MMA
Proof of Lemma
2 By leveraging the linearity expectation available in ((((()))) we get
m - warr(oo) )
a(r(w)) (x,
Vil (11))
Σ w. (m. t).
17

---

Published as a conference paper at ICLR 2026
Using this, we get
(1(7)))))))) (s 1) - -B())(rtussa (rt, = frre 1) - 3()a (x. - ((1)) (r(w)))) (x,
m
m Σ
=fpro(xet t) B(t) Σ u 6) + 3(t)
w. A (n..)) (2, - (z,
m
Σ wi f (fore(22,)) t) ((t)ulria)) (z, (),
+ ((t)
wi (1i.a) (x A (x,
wa (xr,t) + B(1)
Σ ws (1s,a (x,, - ()
Similarly, using the fact that ((na(s)) Xitina) we wet
p(ra(A)))x
(x2 t) - ((t))(na(A)) (x), = (ore - - (() B()) ((tt))(n())(()a)))
(sr 1) - ((1))al)5)) (x, 1) - ((1) (ma))))))))
- (()) (ro()))) t) =xp(naa) (2., t) + (1 f f) + 3(t) (A (z, t) - A (xxA(A))
=)(fPre(x11)) t) - B(t)a(na) (2, t)) + (1 -
A.4 PROOF OF PROPOSITION
Proof of Proposition
2
Denote - (())==(((()++-10)))))) For any function vi E (IR))) It0's
formula gives:
o²(t)
dt [(x(X))] = / o(t)V du, + EVV ((x,)) + dep (x!)
du
2 Ir (r(V2
tr
2
de, is a martingale with mean 0, therefore the first Note that tr (v( = Drb and I V de, term in the RHS is O. We got for SDE 1:
_ - = -VXXX))) 2²(1)
Ax(X))
2
Move to the SDE 2. Similarly, we can derive:
d
x)A(?)) + x²(1)
E [(
E E V
(XF)
Ax(x3)
dt
2
E|Y, (X3) + + (X) E (2()) Ax((3))
+ -X (x(?) + +E| (t)
A
2
22()
=[E[Ve(X))) ( (1- (XXX)
EI B1 2
Ax
VVV 8² 02(t) (x
where the independence of (Y:) is applied.
18

---

Published as a conference paper at ICLR 2026
Both {X+) and {X?) satisfy: =F
+ - Aab( with Xo ~ Po. for
any tb E C Denote Pt as the probability density of X; we got:
dt / (+))(())dd== / V+(+))((x))
0 (t)
Ax(()) pt pr(x)dx
/ (()) -v( (b())(()))))))) 0² o²(t) ((z)
where the change of integral is used. Then we have:
i ((s) (a:p( + v(b())p(())) (1)
Ap(())) de O, vy V E
2
Therefore, both pxt and Pxę are solutions of this PDE: (x)x( (MM)))d=++(())))))))))))))>>)))) (x) O with p/r-o = po- Since the drift term b is Lipschitz continuous, this PDE has only one solution. Therefore, Px; px?-
B EXPERIMENTAL DETAILS
B. BASELINE ALGORITIMS
MORL: For MORL, we fine-tune the SDv1.5 base model using RL, following the same procedure as in Fan et al. (2023). In particular, we obtain separately fine-tuned models for (r(w), a())), for different values w and X. This used as an oracle baseline for both DB-MPA and DB-KLA.
Rewarded Soup (RS): We use RS, introduced by Rame et al. (023)) as baseline for the DB- MPA. We first RL. fine-tune SDv1.5 separately for each reward each reward (r:) with a fixed a. So, starting from the pre-trained model parameter (pre, we obtained m RL fine-tuned models with parameters (0:))m At inference time, given the preference w given by the user, we construct a new model with parameter (w) =
Ex= w,O+!, and generate images using this model. We only average the U-Net parameters from the fine-tuned models.
Reward Gradient Guidance (RGG): We follow the gradient guidance approach (Chung et al, 2023 Kim et al.) 2025) where the diffusion process at each backward step is updated using the gradient of the reward function. The update rule is given by:
( t) A v., f(x)),
(12)
where Ho (z+, t) denotes the base model's predicted mean, of is the noise variance timestep t, a is the KI. regularization weight, and λ is a time-dependent scaling factor defined by the expo- nential schedule X (1 + 7 with 7 0.024, as introduced in Kim et al. (2025). The reward function is defined as f(xx) r(xo(=x)), where xx²x)) denotes the Tweedie approximation-based reconstructed image obtained from x, using the denoiser network.
We adapted this approach to the multi-objective setting by combining gradients from two reward functions T1 and r2 as,
V2, f(x)) = (zo (x:)) + (1 - wi) Va, (2(o((z)))
(13)
However, to ensure that the influence of each reward is independent of its scale gradient magni- tude, normalized the individual gradients before combining them,
V.. (z,)
V (2,oo((((z.))
V f(x)) = W1 I ri (an (2:))
+ (1 - wi) | a, ra (2
(14)
This normalization ensures that the reward guidance strength is controllable and not biased by the nature or scale of individual rewards.
CoDe: Introduced by Singh et al. (2025). CoDe is gradient-free guidance method for aligning diffusion models with downstream reward functions. CoDe operates by partitioning the denoising process into blocks and, each block, generating multiple candidate samples. It then selects the sample with the highest estimated lookahead reward to proceed with the next denoising steps. For experiments, configured CoDe with 20 particles and lookahead of steps.
19

---

Published as a conference paper at ICLR 2026
B.2 PSEUDO CODE
In this section, we present the pseudo-code for the three inference-timg algorithms introduced in section 4 DB-MPA, DB-KL, DB-MPA-LS. All algorithms leverage the approximation result from lemma 2 proposition to enable controllable sampling without requiring additional fine-tuning inference time.
Algorithm oottlines the DB-MPA procedure, in which user-specified preference vector is used to linearly combine the drift functions of m RL fine-tuned models, each optimized independently for distinct reward basis. Algorithm 2 presents the DB-KLA procedure, which instead blends the drift of a fine-tuned model with the pre-trained model. Algorithm H approximmaes Algorithm | by sampling a drift with the assigned preference weights. All fine-tuned models used in algorithms are obtained by applying the single-reward RL fine-tuning algorithm individually to each reward function.
Note that all algorithms adopt basic Euler- Buler-Maruyama discretization to simulate the reverse SDE eq. (5). In practice, this integration step can be replaced by any diffusion model's 's reverse process (e.g., DDIM (Song 2000)) other solvers), as long the drift term (or the predicted denoising output) appropriately mixed.
Algorithm DB-MPA
Input: RL-fine-tuned drifts { (fr weights w E R", E W/ 1; time grid 0 to <
t <... < fx = -
1: Sample Tiy ~ N(,0 I)
2: for k < N down to 1 do
3:
A/4 t- 1 4k-1
b positive
4: noise N0,
5: fmix Ei' Wi f(rt, (Ete) tx)
6: Tta Itt
fmix Ale allx)/Al.
7: end for
Output: Ita
Algorithm 2 DB-KLA
Input: RL-fine-tuned drift (r.a); original pretrained drift fere; KL-reweight parameter X 2 0; time grid 0 to 4 <... IN =
1: Sample Zty ~ NO, I)
2: for k + N down to 1 do
3:
At +- 4 f=-1
b positive
4: noise I)
5: fKL. - >s (mm) (It,, (x) + (1
pre () (b) 6: Ite-1 Ita fKI At, + atta)
7:
end for
Output: Ito
B.3 PROMPT SETS
We evaluate our methods on two datasets: Short-DrawBench and GenEval.
Short-DrawBench. The original DrawBench dataset (Saharia et al. 20226) contains 183 prompts across 1l categories. For our experiments, we isolate the 25 prompts in the color category, forming the Short-DrawBench subset. This reduced scale enables direct comparison with MORL, since training separate models for each preference or KL weight is computationally feasible. To test generalization. further construct set of 1,000 evaluation prompts generated by GPT-4 (Achiam jet al., 2023). These prompts introduce novel object-color and multi-object compositions not present in the training subset. The instruction used for generating this test set was:
20

---

Published as a conference paper at ICLR 2026
Algorithm 3 DB-MPA-LS
Input: RL-fine-tuned drifts { {f(ro)))m weights w E R", Ex= wi 1; time grid 0 to <
f1 <.... ix =
1: Sample Ciy ~ N ~/ 1)
2: for k <- N down to 1 do
3: Aty f- t -1--1
b positive
4: noise (0,
5: sample in from {1, 2,,,..... . m) with probability (Wi, W2,..., Wm)
6: Ita-1 Fte
f(r,, (Iez tu) Ats + ((tx)/At
7: end for
Output: Itt
"Please generate 1000 testing prompts that are similar to the following training prompis, which are cober+ combinations. You should use colors that appeared in the train set have similar semantic meaning. Objects can be a little more common or random."
GenEval. The GenEval benchmark (Kirstain al. 2023 consists of 550 prompts designed to test compositional generalization, spanning attributes such as color, counting, spatial relations, and multi-object scenes. In addition the official prompts, we generate an 700 held-out evaluation prompts using the official GenEval prompt generation toolkit. These follow the same construction rules as the original dataset, obtained by varying random seeds and object/color assignments.
B.4 COMPUTING HARDWARE AND HYPERPARAMETERS
Fine- Fine-tuning of Stable Diffusion for each KL weight and reward composition was performed on NVIDIA A100 GPUs using mixed precision. We used the AdamW optimizer with learning rate of x 10-6 for policy updates and applied LoRA with rank Gradient accumulation was set to 12, with per-GPU batch size of for policy updates and for prompt sampling.
Policy updates followed a clipped PPO-style objective with a clipping ratio x 10 following Fan al. (2023). Each outer iteration performed policy gradient steps and value function up- dates (batch size 256, learning rate x 10-1), using a replay buffer size 1000. Training required approximately 96,000 online samples converge for the Short-DrawBende subset. A similar size of online samples is used for GenEval.
C DB-MPA ALGORITHM: ADDITIONAL RESULTS
In this section, we present additional experimental results for the DB-MPA algorithm. We begin with reward evaluations the training prompts. Next, demonstrate that DB-MPA naturally extends to alignment with three rewards (m = 3). We also show that DB-MPA-LS produces outputs visually close DB-MPA. We then provide additional qualitative comparisons with baselines to further highlight DB-MPA' effectiveness. Finally, we evaluate DB-MPA's ability to achieve fine- grained multi-preference alignment.
C.1 RESULTS ON TRAINING PROMPTS For the rithms relative to Stable Diffusion, evaluated on the training prompts (30 random seed per prompt, 750 images in total). Across all preference weights, DB-MPA consistently outperforms RS, CoDe, and RGG.
setting, Table 2 reports the performance of DB-MPA and baseline algo-
next consider the GenEval dataset, which evaluates compositional generalization.. Table reppors the reward improvements of all methods relative to Stable Diffusion. DB-MPA and DB-MPA-LS achieve the best or near-best gains across most preference weights. Although the table lists separate Ar1 and Arz, also computed the weighted reward A AWR wr1 (1 w)r2. On this aggregate metric, DB-MPA and DB-MPA-LS consistently outperform all other baselines, confirming their superior trade-off performance.
21

---

Published as a conference paper at ICLR 2026
Table 2: Quantitative comparison DB-MPA baseline methods on train prompts. Ar, = r -
5D
DB-MPA
RS
CoDe
RGG
Ar1
Ar2
Ar1
Ar2
Ar1
Ar2
Ari
Arz
w=0.2
0.19
0.74
-0.01
0.61
0.14
0.23
0.12
0.59
Improvement (t)
w=0.5
0.49
0.50
0.12
0.20
0.29
0.19
0.13
0.39
w=0.8
0.65
0.18
0.54
0.02
0.34
0.12
0.03
0.16
Table 3: GenEval train results: reward improvements (Ar) relative to Stable Diffusion.
DB-MPA
DB-MPA-LS
RS
CoDe
RGG
Ar1
Ar2
Ar2
Ar1
Ar2
Ar2
0.2
+0.122
+0.497
+0.143
+0,477
-0.021
+0.378
+0.254
+0.107
+0.064
+0.497
1.5
+0.267
+0.357
+0.376
+0.310
+0.161
+0.192
40.344
+0.067
+0.104
+0.327
Reward (1)
10=00.8
+0.382
+0.185
+0.411
+0.180
+0.340
+0.089
40.374
+0.007
-0.246
+0.111
C.2 QUANTITATIVE RESULTS ON GENEVAL TEST DATA
Table 4 reports the raw numerical values corresponding to fig. shown the main paper.
Table 4: GenEval (test): numerical results (r1, r2) corresponding to the Pareto-front plot fig. in the main text. DB-MPA and DB-MI DB-MPA-LS consistently dominate baselines across preference weights.
DB-MPA
DE-MPA-LS
F1
Y2
12
F1
12
f12
122
71
12
f1
12
w==0
-0.2
-0.04
-0.19
0.83
-0.19
0.83
-0.19
0.83
-0.03
0.07
0.03
0.52
w=0.2
-0.04
-0.05
0.73
-0.07
0.54
-0.07
0.02
0.44
Reward (1)
w=0.5
-0.2
-0.04
0.13
0.47
0.19
0.37
-0.09
0.21
0.05
0.01
-0.35
0.27
w=1
-0.04
0.26
0.01
0.26
0.01
0.26
0.01
0.12
-0.09
-0.20
-0.06
C.3 RESULTS FOR CONFLICTING REWARDS
To evaluate DB-MPA under adversarial objectives, we consider conflict between JPEG com- pressibility and VILA aesthetics. The JPEG reward incentivizes smooth, low-detail images, whereas VILA prioritizes fine-grained, high-quality visuals. These objectives are naturally at odds: opti- mizing for JPEG typically harms aesthetics. For example, while the Stable Diffusion (SI) base- line scores F1 = -0.09 on JPEG and r2 = 0.48 on VILA, an RL-fine-tuned JPEG model attains ri 1.52 but drops 72 -0400.
We train a JPEG-aligned model (ri) and combine it with our VILA-aligned model (r2) using DB- MPA. Because JPEG compressibility is non-differentiable, gradient-based methods such as RGG cannot be applied. We therefore compare DB-MPA against Rewarded Soup (RS) and CoDe, Re- wards are reported as function of the blending weight w [0, 1]. with w preferring JPEG and = 0 preferrig VILA.
Across all weights, DB-MPA achieves substantially higher weighted rewards than both RS and CoDe. This demonstrates that DB-MPA can effectively balance two strongly conflicting objectives far better than competing baselines.
C.4 EFFECT OF INCREASING THE NUMBER OF REWARDS
We study how the performance DB changes as the number of reward models increases. In Fig- fig. 7 evaluate DB under 2-, 3-, and 4-reward settings and compare DB-MPA. DB-MPA-LS, and RS, all of which interpolate the same set of finetuned reward- basis models. can be observed that the improvements of DB-MPA and DB-MPA-LS over the pretrained model remain stable as
22

---

Published as a conference paper at ICLR 2026
Table 5: Performance on Short-drawbench 1k prompts under conflicting rewards: JPEG com- pressibility (r1) and VILA aesthetics (r2). We also report the weighted reward WR w71 + (1 w)r2. The best weighted reward is bold.
DB-MPA
RS
CoDe
WR
WR
WR
==0.2
0.52
0.40
0.44
0.11
0.28
0.21
0.04
0.02
0.03
Reward (t)
w=0.5
1.00
0.18
0,59
0.30
0.02
0.16
0.22
0.03
0.12
w=0.8
1.35
-0.18
0.88
0.93
-0.13
0.72
0.37
-0.01
0.29
more rewards are introduced. All experiments use uniform average weights. From table 6 we ob- serve that DB-MPA achieves the largest improvement, while DB-MPA-LS performs slightly worse but reduces inference cost to essentially the same unit-time speed as SD v1.5. In contrast, the base- line RS, despite interpolating the same 2-4 reward- -asisd-basis models, performs substantially worse than DB, and its performance degrades noticeably the number of rewards increases.
D2-MIA-LS
02-MIALS
CB-M0A-5
2.5
FS
500
Renemmance con IV2 5D
Rentermence Doem 500
0.5
Iemurmante
0.3
0.
31
R2
83
64
Figure 7: Performance comparison Short-drawbench 1K test prompts of DB-MPA and baseline algorithms under different numbers reward models. = ImageReward, PickScore. Performance improvement is computed as (algorithm reward) (SD-v1.5 reward). DB-MPA and DV-MPA-LS consistently outperform RS as the number of rewards increases.
VILA, Compressibility, R4
Table 6: Performance improvements under 3, and 4 reward settings. Each column shows Ar; method r SDv1. the group-wise mean.
2-Reward
3-Reward
4- Reward
Method
Ar;
Ar2
Avg
Ar's
Arz
Arg
Avg
Ar:
Ar2
Ars
Ara
Avg
DB
0.54
0.38
0.46
0.30
0.29
0.75
0.45
0.40
0.22
0.62
0.39
0.41
0.58
0.31
0.45
0.37
0.26
0.56
0.39
0.37
0.28
0.50
0.38
0.38
0.15
0.13
0.14
-0.00
0.08
0.14
0.07
0.03
0.05
0.10
0.03
0.05
23

---

Published as a conference paper at ICLR 2026
C.5 RESULTS FOR REWARD SETTING
To evaluate DB-MPA in a more complex multi-objective setting, we conducted experiments using three distinct reward functions: ImageReward for text-image alignment, VILA for aesthetic quality, and PickScore Kirstain et al. (2023) as proxy for human preference w. We evaluate the result on smaller test set with 25 prompts generated by GPT-4 (Achiam al. 2023 with identical 30 fixed random seeds for each prompt. The full list of this smaller test set is in our code. Figure iillustrates
DB-MPA performance across various weight combinations in the three-reward setting. DB-MPA consistently adapts its outputs to reflect user-specified preferences, demonstrating scalable control without requiring additional retraining.
16.2, 322.355
05.825
L
13-
Sertere Suee
Whil USS, E.5, 8231
Waye 11
HLL
Fulfal RNWWRoo200r
Figure 8: Performance comparison of DB-MPA and baseline algorithms in the m rewards setting. using LewgeReward (text- (text-image aligmment). VILA (aesthetic quality), and PickScore (human preference). Each bar shows the improvement SDv1. 1.5 for the corresponding reward. DB-MPA consistently outperforms all baselines across different weight combinations and all reward dimensions.
C.6 EXTENSION TO SDXL
To examine whether the preference trade-off behavior observed in SD 1.5 carries tially larger backbone, we extend our experiments to Stable Diffusion XL. (SDXL). The SDXL base UNet contains 1.6B parameters, more than 5x larger than the 300M UNet in SD 1.5, and the full SDXL pipeline totals approximately 2.6B parameters. Following seminal RL works on fine-tuning diffusion model Fan al. (002);; ); Black al. (2024) that train and validate on single prompt, we also fine-tune SDXL on the prompt ("an orange colored sandwich"). As in the SD-1.5 setup, we use two reward models Imaeels-amaaeRewarrd (alignment) and VILA (aesthetics). The corresponding training reward curves are shown in Figure
substan-
After training, we evaluated the model across different preference weights. For DB-MPA, DB- MPA-LS, and two baselines RS and CoDe, we swept w in increments of 0.1. For RGG, due to huge inference time cost, we only evaluated five points: w 0.2, 0.5, 0.7, 1.0). For each point, 64 random seeds used. The resulting Pareto front is shown in Figure TO Table repoots the numerical results for the representative weights. The qualitative trends are consistent with our observations in SD 1.5: DB-MPA and DB-MPA-LS continue to provide controllable reward trade- offs in the SDXL setting, achieving larger performance improvements compared to other baselines. training-based methods (DB-MPA, DB-MPA-LS, and RS) demonstrate superior performance training-fret approaches (RGG and CoDe). Notably, CoDe exhibits similar performance across different reward weightings, which may be attributed SDXL's larger VAE being more sensitive to noise in the intermediate-step predicted images during CoDe's lookahead best-of-N sampling scheme.
24

---

Published as a conference paper at ICLR 2026
(a) ImageReward training curve.
(b) VILA training curve.
Figure 9: SDXL training curves for the two reward models. Training was run for roughly 2000 epochs over
72 GPU(A100) 100) hours.
0.7
DB-MPA (ours)
DB-MPA-LS (ours)
Rewarded Soup
0.6
CoDe
RGG
0.5
VILA score
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
ImageReward score
Figure 10: Pareto front of the SDXL LoRA model across preference weights.
Table 7: SDXL single prompt results:
ImageReward, r2 VILA. The pretrained SDXL has: = -105,
0.01.
DB-MPA
DB-MPA-LS
RS
CoDe
RGG
V1
72
F1
12
T1
V22
F1
F2
V22
==0.0
-0.34
0.70
0..4
0.70
-0.34
0.70
-0.87
0.18
-0.87
0.32
w=0.2
-0.10
0.65
-0.09
0.64
-0.21
0.50
-0.89
0.16
-0.84
0.29
Reward (t)
w=0.5
0.25
0.55
0.20
0.56
-0.10
0.27
-0.99
0.16
-0.8
0.21
w-0.7
0.14
0.47
0.39
0.48
0.23
0.30
0.89
0.15
-1.07
0.16
w=1.0
0.58
0.42
0.58
0.42
0.58
0.42
0.88
0.16
0.82
0.15
C.7
VISUAL SIMILARITY OF DB-MPA AND DB-MPA-LS
We present visual comparisons among DB-MPA, DB-MPA-LS, and RS under the two-reward setting
using ImageReward (text-image alignment) and VILA (aesthetic quality). The results indicate that, for interpolating the same pair of diffusion reverse processes, DB-MPA and DB-MPA-LS yield visually similar outputs, both surpassing the baseline RS in image quality.
25

---

Published as a conference paper at ICLR 2026
w-0.5
DB-MPA
DB-8/A--L
RS
st0ee oftusion
dag purple colihono!
AA yallow
[6]
Figure 11: w denotes the weight assigned to IewgeReward, with 1 - w corresponding to the weight for VILA.
Both DB-MPA and DB- 0B-MPA-LS produce visually similar results, and each generates images better aligned to the user's preference than baseline
C.8 VISUAL COMPARISON WITH BASELINES
We provide additional qualitative comparisons between DB-MPA and the baselines fig. 12 us- ing prompts from both the train and test sets, for w {0.2, 0.5, 0.8). Despite requiring extra fine-tuning, DB-MPA generates images that are visually close to the MORL oracle baseline, and outperforms all other baseline methods.
26

---

Published as a conference paper at ICLR 2026
( A red colored dog
[b) A biue bird and brown beer!
Eabe
Smarded Soap
MC4L
Cabe
Enaaried Soup
orange colored sandwich.
book
Rewarded
05-
Rewarded
yallow
timale
(3) A blue car and pink shoap.'
(h) 'A rod book and yallow vase.
Figure 12: Qualitative comparison DB-MPA with baselines. Subfigures (a) (d) correspond to training prompts, and (e)(h) to test prompts. In several cases, such as (a) and (e), Stable Diffusion produces cartoonish or unrealistic outputs. In contrast, DB-MPA generates more realistic and se- mantically aligned images by effectively leveraging multi-reward alignment, without requiring any additional fine-tuning.
27

---

Published as a conference paper at ICLR 2026
C.9
ALIGNMENT WITH FINER GRANULARITY
In Fig. 13 we present additional results of DB-MPA and RS for multi-preference alignment with finer granularity of w, with (0.1, ..... 0.9}. As observed, both algorithms exhibit smooth transition from aesthetically pleasing results to outputs that are more aligned with the input prompt. However, DB-MPA typically achieves better alignment with the input prompt, especially for w [0.3, 0.7].
W 0.1
0.2
W
0.4
W
w 0.6
W 0.7
w
0.8
R 0.9
red book and yellow vase.
DB- NPAN
RS
blue cup and green cell phone.
D8 -P-NPA
blue car and pink shoop.
DB-AIPA
RS
green yellow
D8 BB-NPA
RS
Figure 13: Qualitative comparison between DB-MPA (first row) and RS (second row) with Im- ageReward and aesthetic score rewards. Each block shows generations the ImageReward weight w increases from 0.1 to 0.9 (left to right). The first two examples are from the training prompt set, and the last two from the test prompt DB-MPA demonstrates smoother transitions and more precise alignment with the target reward preferences compared to RS, supporting the trends observed in the quantitative results.
28

---

Published as a conference paper at ICLR 2026
D DB-KLA: ADDITIONAL RESULTS
This section presents additional results for DB-KLA, including qualitative comparisons with the MORL oracle across diverse prompts. It also evaluates DB-KLA's controllability under fine-grained variations of the KL weight.
D.1 QUALITATIVE COMPARISON WITH BASELINES
We provide additional qualitative comparisons between DB-KLA and the MORL oracle baseline in fig. [14] using prompts from both the train and test sets. These examples show how DB-KLA adapts generation quality as the KL regularization strength changes, producing outputs that closely resemble those of the oracle baseline.
A
PROC.
US- KLA
(a) white and red sheep.'
A 0.2
05
A 0.7
A
A-2
A U
DS- KLA
(b) 'A book and yellow vase.'
0.2
MORL
Us- KLA
(c) 'A blue colored apple. !
A-2
==0
PROYL
DG-KLA
'A purple hat and black clock.'
Figure 14: Qualitative comparison between DB-KLA and MORL for KL weights À (0.2, 0.5, 0.7, 1.0, 1.5, 2.0}. The first two rows show the results with train prompts, and the last two show the results with test prompts. DB-KLA generates images of similar quality to those of the MORL oracle base- line without any additional fine-tuning.
29

---

Published as a conference paper at ICLR 2026
D.2 KL ALIGNMENT WITH FINER GRANULARITY
In Fig. 15 we present additional results of DB-KLA with finer granularity of X, for À [0, 0.2, 0.5, 0.7, 1.0, 1.5, 2.0]. As regularization increases, the model shifts more toward optimiz- ing the text-to-image reward, producing images that better match the prompt but drift further from the original Stable Diffusion output.
red car and a white sheep.
blue cup and green cell phone.
brown bird and blue bear.
A blue car and' red giraffe.
yellow colored dog.
A blue book and an orange colored sandwich.
Figure 15: KI. weight alignment in DB for À = [0, 0.0.2,00 0.5, 0.7, 1.0, 1.5, 2.0), with À increasing from left to right.
E IMPACT STATEMENT
Diffusion Blend enables flexible, inference-time alignment of diffusion models with user-specified preferences over multiple reward objectives and regularization strengths, without additional fine- tuning. This significantly reduces computational costs and increases adaptability for personaliza- tion. By leveraging a small set of fine-tuned models, it opens the door to scalable, user-controllable generative AI and sets the stage for more preference-awe implications for alignment and deployment as it enhances the existing diffusion models' perfor- mance, do not foresee any immediate societal concerns that require specific highlighting.
deployment. While work has broad
30

---

Published as a conference paper at ICLR 2026
F
THE USE OF LARGE LANGUAGE MODELS
Portions of this work were prepared with the assistance a large language model (ChatGPT, GPT-5, by OpenAl). The model was used writing assistant improve clarity, grammar, and organization of the manuscript, and to suggest alternative phrasings of technical content written by the authors. All ideas. experiments, analyses, and final decisions regarding the content remain the responsibility of the authors. The test prompt set for the Drawbench prompt dataset is generated by LLM, fol- lowing the standard benchmark like GenEval. The model was not used generate research ideas, perform experiments, or create unverifiable scientific claims.
31


Reviews:
[{"review_id": "nfQPrblLxG", "reviewer": "Reviewer_RqHN", "summary": "This paper addresses the problem of inference-time multi-preference alignment for diffusion models, where the users can adjust reward trade-offs and regularization strengths on-the-fly without additional finetuning. The DiffusionBlend pipeline blends the backward diffusion trajectories of multiple RL-finetuned models to distinct reward functions. The method is theoretically motivated and empirically validated on a series of benchmarks and alignment tasks.", "questions": "see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem is novel and well-motivated. The paper formalizes the inference-time multi-preference alignment problem from the perspective of MORL, which is practically important and underexplored in diffusion literature. \n\n2. The approach is theoretically justified with clear derivations and bounds on approximation errors. \n\n3. Strong empirical results on multiple datasets and reward functions.", "weaknesses": "1. The Jensen-gap approximations in Eq. 8 are only empirically validated via downstream metrics; direct error analysis on $ \\Delta(r,\\alpha) $ (beyond Appendix bounds) would strengthen claims. \n\n2. The paper builds on KL-regularized RL fine-tuning for diffusion models, but does not sufficiently discuss or compare to DiffusionDPO (Wallace et al., 2024), DDPO (Black et al., 2024), etc, which are dominant lines of work in this space. For example, the author cites DPO (Rafailovetal., 2023) for diffusion model alignment in section 4, but does not mention DiffusionDPO (Wallace et al., 2024), which is the direct diffusion analog and is relevant. \n\n2. The training cost still scales linearly with respect to the number of rewards. This can be costly in applications. \n\n3. Based on the provided pages, evaluations focus on two rewards (alignment + aesthetics) and Stable Diffusion v1.5. Scaling to m > 2 or diverse rewards (e.g., human preferences via PickScore, diversity) and larger base models (e.g., SDXL) is not shown. \n\n4. Strong compared to RS/CoDe/RGG, but no comparison to DPO-inspired diffusion variants (e.g., Calibrated DPO) or recent multi-LoRA fusion methods (Zhong et al., 2024a). The best individual RL-finetuned model per w (the oracle) is also missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ia7dKGxuA3", "reviewer": "Reviewer_b58F", "summary": "This paper tackles the challenge of aligning diffusion models with multiple, potentially conflicting objectives at inference time. Existing RL-based alignment methods typically optimize a single reward under fixed KL regularization, which limits flexibility. The authors introduce Diffusion Blend, a novel framework that enables inference-time multi-preference alignment, which allow the model to generate outputs based on any user-specified combination of reward functions and regularization strengths without additional fine-tuning. They instantiate this idea with three algorithms: DB-MPA (for multi-reward alignment), DB-KLA (for controllable KL regularization), and DB-MPA-LS (a low-cost approximation). Empirical results demonstrate that these methods outperform existing baselines and perform comparably to models fine-tuned individually, offering a practical way to achieve user-driven, flexible alignment during inference.", "questions": "Please refer to the answer in Weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written and easy to follow.\n\n2. The studies of conducting inference time alignment on diffusion models are novel and interesting.\n\n3. The authors conduct extensive experiments to verify the effectiveness of thier method.", "weaknesses": "1. It's better for authors to have some results on larger models like SDXL to further prove the effectiveness of thier method.\n\n2. It's better to demonstrate that the model can be used in wide applications like image editing.\n\n3. The authors use DPOK for fine-tuning models, is this method sensitive to different RL algorithms?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JGwDBCFFHC", "reviewer": "Reviewer_bBuv", "summary": "This paper proposes Diffusion Blend that aligns diffusion models' generation process with user-specified multi-preferences. Crucially, this is done at inference time such that users can dynamically control preferences as they wish. Diffusion Blend trains different versions of diffusion models that maximize the given reward functions and then combine their scores during the generation process. This paper introduces theoretical justification for such a process and demonstrates its effectiveness in quantitative and qualitative experiments.", "questions": "It would be great if the rebuttal could comment on the points in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Importance of the problem: Dynamically aligning diffusion models' generation process with user preferences is a challenging problem that can have a great impact on content creation applications by providing users with knobs to adjust their content. \n- This paper proposes a simple algorithm that achieves this purpose by combining scores of diffusion models trained for specific rewards. This is theoretically justified well, and the simplicity of the algorithm would enable easy adoption of the method.\n- Maintaining multiple copies of diffusion models could be heavy in storage and computation, but the experiment with LoRA shows it is possible to achieve efficient alignment by maintaining relatively small copies.\n- The evaluation in the experiments section shows a consistent boost in performance over the baselines considered in the paper.", "weaknesses": "- The experiments are done with a relatively small number of reward models (most with two) and a single class of diffusion model, Stable Diffusion. The results could have been stronger by pushing the limit with more rewards (e.g. 6~10) and with more recent diffusion models such as Flux. \n- In practice, LoRA-based diffusion model weight combination techniques (e.g. Zou, Shen, Bouganis, and Zhao, ICLR 2025) could be a strong candidate to achieve the same purpose. How does it perform, and is there a practical advantage to using Diffusion Blend over it?\n\nMinor: in equation 11) $\\log p^{tar}_t$ in the right hand side needs to be  $\\log p^{pre}_t$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqZlXaymdA", "reviewer": "Reviewer_PHNS", "summary": "This is a classifier‑free guidance–like formulation designed for reward alignment in a score-based generated model.", "questions": "As noted in strength, I prefer to see the reformulation written in Karra's SDE instead.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, the paper mainly makes a theoretical contribution. \n\n## Presentation: ~75th percentile\n\nThis paper is comprehensible to diffusion model researchers, but it possesses some weaknesses in presentation.\n\n## Soundness: 80~90th percentile\n\nA great portion of the soundness stems from the reliability and the success of CFG. The formulation appears correct based on my understanding of CFG and score-based SDE.\n\n## Contribution: ~90th percentile\n\nTo the best of my knowledge, you are the first, aside from any concurrent work, to propose this \nformulation. Your contribution is particularly strong because it is presented within a score‑based SDE \nframework. It could have been even more impactful had you expressed it in Karras’s SDE, although that is \nnot strictly necessary.\n\n## Note\nI hope the AC is aware that the rating is calibrated using percentiles to reduce evaluation noise effectively.", "weaknesses": "## Presentation\n\nPresenting advanced mathematics elegantly is always a challenge, and a successful example in writing I’ve read is the score-based SDE by Yang et al. Perhaps it would be better to present the main idea without too many interruptions from details, making the mathematical notation less daunting in the main paper for a wider range of readers.  But that is impossible without a significant modification of the manuscript.  If accepted, I recommend rewriting your final manual script or, if any, your preprints.\n\nThere are some minor formatting issues, such as inconsistent use of italics and bold across sections, and the abuse of italics applied to the entire paragraph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)