Sample Reward Soups: Query-efficient Multi-Reward Guidance for Text-to-Image Diffusion Models
Authors: Yinghua Yao
Yuangang Pan
Guoji Fu
Ivor Tsang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion model
Text to Image
Sample Reward Soups
Training-free
Black-box alignment
Primary area: generative models
TL;DR: We propose the first inference-time soup strategy, named Sample Reward Soups (SRSoup), for Pareto-optimal sampling across the entire space of preferences.

Abstract: Recent advances in inference-time alignment of diffusion models have shown reduced susceptibility to reward over-optimization. However, when aligning with multiple black-box reward functions, the number of required queries grows exponentially with the number of reward functions, making the alignment process highly inefficient. To address the challenge, we propose the first inference-time soup strategy, named Sample Reward Soups (SRSoup), for Pareto-optimal sampling across the entire space of preferences. Specifically, at each denoising step, we independently steer multiple denoising distributions using reward-guided search gradients (one for each reward function) and then linearly interpolate their search gradients. This design is effective because sample rewards can be shared when two denoising distributions are close, particularly during the early stages of the denoising process. As a result, SRSoup significantly reduces the number of queries required in the early stages without sacrificing performance. Extensive experiments demonstrate the effectiveness of SRSoup in aligning T2I models with diverse reward functions, establishing a practical and scalable solution. The code is available at https://github.com/EvaFlower/Sample-Reward-Soups-ICLR26.

Published as a conference paper at ICLR 2026
SAMPLE REWARD SOUPS: QUERY-EFFICIENT MULTI- REWARD GUIDANCE FOR TEXI-TO-IMAGE DIFFUSION MODELS
Yinghua Yao!², Yuangang Pan'"* Guoji Fu" Ivor W. Tsang
Centre for Frontier Al Research, Agency for Science, Technology and Research, Singapore Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore DDepartment of Computer Science, National University of Singapore, Singapore 'College of Computing and Data Science, Nanyang Technological University, Singapore Slao-yinghua, pan--uanae -uangann, ivor ivor.tsang sa-star. edu. sg guoji fu@u. nus edu
ABSTRACT
Recent advances inference-time alignment of diffusion models have shown reduced susceptibility to reward over-optimization. However, when aligning with multiple black-box reward functions, the number of required queries grows expo- nentially with the number of reward functions, making the alignment process highly inefficient. To address the challenge, we propose the first inference-time soup strat- egy, named Sample Reward Soups (SRSoup), for Pareto-optimal sampling across the entire space of preferences. Specifically, at each denoising step, we indepen- dently steer multiple denoising distributions using reward-guided search gradients (one for each reward function) and then linearly interpolate their search gradients. This design effective because sample rewards can be shared when two denoising distributions are close, particularly during the early stages of the denoising process. As a result, SRSoup significantly reduces the number of queries required the early stages without sacrificing performance. Extensive experiments demonstrate the effectiveness of SRSoup in aligning T21 models with diverse reward functions, establishing practical and scalable solution. The code is available at https: //github. com con/evaFlowee//eeeeeoooaaoaoooee www.oo/wwwe//eeeeeeeeeeee Reward- Soups ICLR26
INTRODUCTION
Text-to-Image (T2I) diffusion models have emerged at the forefront of high-quality image synthesis conditioned on text prompts. pushing forward various applications in fields such as art, design, and
beyond (Rombach et al.) 2022; Ramesh al. 2022 Saharia et al.) 2022] Podell et al. 2024, Chang
et al. 2025). Despite their success, controlling T21 models to meet complex user objectives, such as compressibility and subjective aesthetics, remains challenging through simple text prompting alone. This has driven interest in techniques for fine-tuning T21 diffusion models with reinforcement
learning [Black et al. 2024). differentiable reward functions (Prabhudesai et al. 20233 Clark et al. 2024), and direct preference optimization (Wallace et al.) 2024 Yang et al. 2024a).
Multiple reward functions are often used to provide supervision for fine-tuning. A common approach assign different weights these functions to account for diverse human preferences regarding different reward criteria. This, however, incurs substantial computational costs due to the large space of possible weight combinations. One line of research (Rame et al.) 2023) adopted the idea of model soups (Wortsman et al.) 2022) to reduce the fine-tuning cost to scale linearly with the number of reward functions. Another effort (Yang al. 2024b) is to fine-tune model that conditions multiple reward values in the prompts using supervised learning. Nevertheless, these fine-tuning strategies may lead to reward over-optimization. where the fine-tuned models lose generation quality
or diversity (Gao et al. 2023 Jena et al. 2024). Additionally, these models may not generalize well unseen prompt distributions that drift far from their training set.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
SRSoup Weightrdium
SRSoup 0o8- Wiightediom
Skseip WeglleeSum
Hiperesuum (ARV
Mypervolume (H1))
3-277 i efficiency
602
efficieccy
efficierry
350
Number of reward queries Ix 1x
Number reward queries (x107)
Number reward queries (x101)
Aesthetic Compressibility
Aesthetic & PickScore
Aesthetic & HPSv2
SRSoup
5R5oup
SRtoup
KeightedS.um
WeighteeSum
WelghooddSmm
(12)
Hyperas (Yee)
Syperoalume (AA
6.22
638
225
Inferenie
3ia 1/2
321
382
Inference time (GPU hours)
04 18
Inference time (GPU hours)
12
24 16 32
Aesthetic Compressibility
Aesthetic & PickScore
Aesthetic & HPSv2
Figure 1: Quantitative evaluation two-objective optimization, showing that our SRSoup is more query-efficient and achieves comparable results with less reward queries and inference time.
In this work, instead of fine-tuning models for alignment, we explore guiding the denoising process of diffusion models using multi-reward signals. This type of inference- terrenece-tmeeeeee alignment has been shown to be less prone to reward weerooptimization (Kim et al. 2025 Eyring et al. 2024 Yeh al., (025). However, main challenge in the multi-reward setting is that traversing weighted combinations of multiple reward objectives often requires a large number of reward queries. To address this, we propose Sample Reward Soups (SRSoup) as a query-efficient solution. Specifically, we begin by optimizing the stepwise denoising distributions of diffusion models using reward-gun -uiddedd search gradients for each individual reward function. Then, we interpolate these search gradients weighted sum of the reward functions. The
to approximate the scarch gradient corresponding key insight is that the denoising distributions eesiiuutoons--peciallll the early stages of the process-overlap when optimized from the same noise point under different weightings of multiple rewards (Fig. We summarize our contributions as follows:
We advocate multi-reward alignment approach inference time of diffusion models enable Pareto-optimal sampling across the entire preference space w.r.t. different reward functions.
We propose Sample Reward Soups (SRSoup), which significantly reduces the number of reward queries compared to the weighted sum strategy.
We empirically demonstrate that our approach obtains better solutions than fine-tuning based rewarded soup variants and achieves comparable performance the weighted-sung sum strategy while requiring fewer reward queries in Fig.
2 RELATED WORK
Fine-tuning-based Alignment. Fine-tuning T2I diffusion models using reinforcement learning (RL) has been proposed align them with arbitrary black-box reward functions (Black al. 2024 Fan et al. 2024 Deng et al., 2024). These approaches interpret the reverse diffusion process as multi-step decision-makin process and apply policy gradient algorithms for fine-tuning based on pre-specified rewards. In parallel, direct back propparopngttion through differentiable reward functions has been proposed for T2I model alignment (Clark et al. 2024 Prabhudesai et al. 2023 Xu et al. 2024). In addition, reward fine-tuning with supervised learning has been proposed (Lee al. 2023 Wu
et al. 2023 Dong et al. 2023). In the approaches, images are generated with the tre-trained diftusion model, and the model is then fine-tuned using weighted likelihood, where the weights determined by the rewards or by a selected set images with high rewards. Another popular approach involves fine-tuning T2I models directly on preference data using Direct Preference Optimization (DPO)
2

---

Published as a conference paper at ICLR 2026
objective (Yang et al. 2024a Wallace et al. 2024 Liang al. 2024 Venkatraman al. (2024) casts
reward fine-tuning as a stochastic optimal control (SOC) problem and proposing
SOC for better sample quality and diversity. Upon it, Dominoo--mnnnin
et al. (2025) further proposes more simplified and elegant solution, Adjoint Matching, to solve SOC problems. Recent works (Wang
et al.) 2025 Liu al. 2025) apply group relative policy optimization (GRPO) on flow-based T21 models for reward alignment, establishing a promising foundation for this direction.
Alignment at Inference Time. Diffusion model exhibits flexibility in its inference, beyond large language models, allowing for the incorporation of more controls during the inference stage. Classifier guidance (Dhariwal Nichol, 2021) is popular strategy steer the denoising process in alignment with the target objective by adding an auxiliary objective gradient to the score function of diffusion models. Many works (Wallace al. 2023] [Bansal et al. 2023) Eyring et al. 2024 Kim al., (025) have proposed incorporating guidance from differentiable reward model at the inference stage of T21 models. A recent work (Yeh et al.) (2225) proposed optimizing noise distribution for the diffuusion sampling based rewards from black-box function. DyMO (Xie & Gong, 2025) focuses on improving semantic alignment in the single-reward setting by introducing a dedicated semantic alignment objective.
Multi-reward Alignment. Weighted sum is a popular method for combining multiple reward functions into a single one, which can then be used to fine-tune models (Li et al. 2020 Zhou et al.) 2024). However, this approach typically requires using many different sets of weights to obtain Pareto-optimal solutions for the multiple objectives, resulting in significant computational overhead. Rewarded soups (Rame et al. 2023) has been proposed to reduce the cost (Clark et al. 2024 Prabhudesai al. 2023). first obtains models fine-tuned independently on each reward function and then interpolates the model weights to find diverse solutions that account for various trade-off's among the objectives. Supervised tne-tuning methods have also been proposed for multi-reward alignment (Lee et al. [024]] Yang et al., 2024b). Specifically, (Lee et al.) (024)) generates images with pre-trained T21 model and applies multiple reward functions to select Pareto set of images. The T21 model is then fine-tuned using RI. based on this Pareto set of samples. Similarly, (Yang) ctt al. 2024b) generates images with the pre-trained model, applics multiple reward functions to score the images, and extends the text inputs with their corresponding reward vectors to fine-tune the T2I model. The Pareto set of images is then obtained by specifying the reward vectors during the inference process. In summary, existing approaches either require fine-tuning multiple models for multi-objective alignment or rely on large supervised datasets. These fine-tuning fine-tuning methods often suffer from issues of reward over-optimization, leading to a loss in generation quality and diversity (Gao et al. 2023 Jena et al.) 2024). It is also possible to apply the weighted sum to provide multi-reward guidance at inference time in diffusion models (Kim et al. 2025). However, the need for numerous reward queries across various combinations of weightings for different reward objectives leads to a high computational cost. This work will target a query-efficient multi-reward alignment method without any training of diffusion models.
3 PRELIMINARIES
Diffusion Models (Ho et al. 2020 Yang et al. (022) progressively diffuse data to noise, and then learn to reverse this process for sample generation. Considering sequence prescribed noise scales 0 < 31 B2 <... < Br < 1, clean data æu ~ Adaia (x) are diffused to noise σy (0, I) via constructing a Markov chain ( I1, ... ,ry,...,,,,.. where q
N( vI B 1, 8 I). In particular, qa (())))))= = Nar;; Va,zo, (1 ar) I). where a = I[ (1 - B.). The training of the reverse diffusion process is to minimize simplified variational bound of negative log likelihood. Namely,
(m [[le- co ()),t)|1?
(1)
where a,, / 30 + vI are, and c0( is neural network-based approximator to predict the noise from z, After obtaining the optimal noise predictor cø( t), samples can be generated by starting from ØT NN(0, I) and reversing the Markov chain following the step sampling distribution as below:
Øt-1 (( (x t), B,1)
3r
(2)
(e( t) =
2t
c0(
VI Dz
vI
aq
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Reward-guided Search Gradient for Denoising Distribution Optimization
1: Input:
reward function f(x), denoising distribution Na 1c- 3,1), random
noises { J==1 from (0, I), step size Tz.
2: Construct N denoised samples { = p4-1 + vBiz" (Biz"N
Tn=1
I==1-
3: Obtain the black-box rewards (f()-1)))
4: Estimate the gradient VM, ((( VA, Σn (x
5: Obtain the high-reward mean 11 = P1-1 + Tr V - F F(p
6: Output: search gradient V VH.- F(m-1)). optimal mean p and rewards (f(27-1))m)
where T,T 1, .,.... 1. The variance 3, can be to either 3e or I-o
1-ae-1
Be
Ho et al. 2020 In
the following, we set 3e as 3t-
Multi-objective Optimization Let x E RD a sample interest, e.g., an animal of specific
appearances, material with desired functions. Assuming that F(z) [ (z), f.( fas (x))
be a set of M rewards, each of which represents a property and is preferred to have a smaller value. The multi-objective optimization problem (Chinchuluun & Pardalos 2007 Yao et al. 2024) can be conventionally expressed as:
min F(x) min mERD
mERD
In the context of multi-objective optimization, for any #1, 22 E RD, 21 is said to dominate Z2. i.e. 21 < 22, iff (a)) s (1(22), Vi = 1, 1.2...., M, and F(m)) + F(x2). A point x" is called Pareto ...
optimal iff there is no any other x'' E TRD such that x' < x=
4 SAMPLE REWARD SOUPS
In this section, first propose training-free free guidance strategy for multi-objective alignment of diffusion models. Building on it, we further propose sample reward soups to improve the query- efficiency of multi-objective guidance.
4.1 REWARD-GUIDED SAMPLING FOR BLACK-BOX REWARD ALIGNMENT
While fine-tuning T2I diffusion models using RL for non-differentiable reward optimization has shown success, we in parallel propose training-fi free strategy that simply steers the denoising sampling process of diffusion models based on sample rewards.
Specifically, propose optimizing the denoising distribution N( (o BrT) in Equation at each step to encourage high-reward generations. Namely,
max Af (( 1898 (3.11)
EE (x) = I.....
This idea is inherited from black-box optimization Wierstra al. (2014b); Nesterov & Spokoiny (017)), which enables reward-guide optimization without requiring gradients from the reward function. For simplicity, we only optimize the mean of the Gaussian distribution. Defining p4-1 Ha (xe, 1), we have the following objective:
max F(p-11) = tw (f Vt = T,T 1.....1
(5)
Theorem (Reward Neeward-guided search gradient). The gradient of F(M:-1) w.r.t a learnable u can be obtained as follows:
1 EN( [f(M-1 + VBiz)z). (6)
V Hr- F(m--1)=
J (.
vaf(a))
VB.
See Appendix A.1 for detailed derivation.
4

---

Published as a conference paper at ICLR 2026
Equation 6 provides solution to determine the optimal mean parameter M411 in Equation 5 Juiin sample rewards. To avoid computational overhead, we adopt a single-step gradient descent. Namely,
3. VI-a Eo (Z.,)) + + V V
P
ct
where is the step size and is set to Bg for compatible with the original sampling schedule. With a slightly abused notation, we let Ph-1 = p t), then 14-1 VI-3. it vI-a. ea E0( and p. = 1 TO V F(pe-)). Finally, a sample with a high reward can be sampled via: Tt-1 1,
In practice, we use Monte Carlo approximation for Equation 6 Specifically, at each denoising step we first draw N standard Gaussian noise x" ~ (0, I) and obtain the denoise sample x;- via the transformation -²-1 P + VB,"". Then, we obtain the rewards {f{ }) h= which will be used to update 1 towards f via Equation 7 To avoid extra noise distraction (Prabhudesai al. 2023 Clark al. 2024), we adopt the deterministic formulation of the reverse generative
process, i.e., Denoising Diffusion Implicit Models (DDIM) Song et al. (2020) to obtain 1t-1, namely, 1t-1 = pt-1- The whole process is summarized Algorithm
4.2 WEIGHTED SUM FOR
ALIGNMENT
Weighted sum is commonly used strategy to solve the multi-objective optimization. Let WI:M denote {w; Ms i=1 for short. Each point WI.M E A lies on the standard (M 1)-simplex:
w; 2 0, Ee = 1. Given a weight vector WI.M. the multi-objective problem (Equation is formulated into a weighted objective as follows:
M
F(x, W1:M)
Σ w;fi(x)=
= (1)( + w2)2(2)) ... WMMMM(()
(8)
1=1
where larger wi places more emphasis the corresponding reward function fi-
Following the reward-guide guided sampling introduced in Section 4.1 we can apply weighted rewards to obtain samples for
alignment by replacing f with F. Namely,
max F(p-11 WI.M) (e ..11) (x.=1, VI
1.
14-1
To accommodate different trade-offs among objectives, we need to construct a set of L(> M) weight vectors { -- These weights yield L samples { which collectively approximate the Pareto front eross multiple objectives. However, directly evaluating all L weighted combinations requires MT M 1) reward evaluations per prompt. This is because each w typically has support over M objectives. Among these, the M one-hot weight vectors each require only NT evaluations, while the remaining L M combinations require N evaluations per prompt. Although this approach mitigates the high training cost of diffusion models, the large number of required reward evaluations can significantly increase inference-time sampling cost, especially when the black-box reward is expensive to compute.
4.3 SAMPLE REWARD SOUPS FOR MULTI-REWARD ALIGNMENT
Inspired by Rewarded Soups (Rame et al. 2023), which interpolates the weights of diffusion models fine-tuned on distinct reward functions to approximate model optimized for their weighted sum, propose Sample Reward Soups (SRSoup) in the training-free scenario. Our SRSoup is a sample-level analogue that interpolates reward- -giided search gradients (Equation ()) derived from distinct reward functions at each denoising step, to approximate the search gradient corresponding to their weighted combination.
Definition 2. Let (cT=) J denote set of exemplars, each individually guided by a distinct reward function. These exemplars serve as the mean parameters in the denoising distribution, similar the Pi-1 in Eq. 7 As only one reward function is active, the weight vector of each exemplar becomes one-hot. Therefore, for each exemplar ciT Eq. 9 can be simplified as:
max F(c 1, pm) = RRyee-ee (.d) [m((x.11))
YI= =;= -- 1,...,l,
(10)
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Sample Reward Soups (SRSoup) at the Denoising Step L
1: Input: M reward functions f2,, ; L weights M #query N, denoising timestep step size Tt.
2: if't == then
3: Initialize samples (# M= using same point from No I).
4: else
5: {./)=1 obtained by using SRSoup at the denoising step - 1 (Algorithmn 2
6: end if
7: Sample N random noises [=")== from from.N(O, I).
### Obtain search gradients w.r.. each reward function based on the top M samples {= 11111 with one-hot weights em (1 at the m-th entry).
8: for m 1,2,....4 M do
2): ci vT- p. c Vi-ai co (2:" .t). Obtain the search gradient Verz F(cT Cm). the optimal mean c and the black-box rewards (x."")) for (a=""1 1; 1- B,I) with fm and (=") using Algorithm Obtain the high-reward denoised sample: o; = 1-
9:
Obtain the denoised mean (exemplar in Definition
10:
)
11:
end for ###
Interpolate search gradients. for - .M 1, M + 2,...., , do
14:
Obtain the denoised mean: p1-1 -I-8, (
3e vi-aa
co
15:
Calculate the interpolated gradient:
M
N
Σ w/m F(e VB,N
fm (m
m.n (c ui-
m=
Obtain the high-reward denoised sample: E/_1 = -/
m F (Hi w(in))
17: end for
18:
Output: (a/-1)t)) that optimizes M reward functions with respective weights
Har
where m = 1,2,.... ............ M and em E M-1 is a one-hot vector with the m-th entry equal to 1.
Proposition 3. Given a of exemplars {c]-]]] 1M introduced in definition 2
Let 8m Mh- c
Vm = 1.2..... M, the gradient of ut-1 in Equation 9 be approximated by the gradients of these exemplars as follows:
M
V
= Σ
Wm
F(e
Vog F(c em )6n 0(16mm 13)
m=
See Appendix A.2 for a detailed derivation.
According proposition 3 the gradient approximation by Ven F(c" is accurate only when the difference 6m. very small. Otherwise, we need calculate the second deriva- tive V F(c cm), which is however computationally expensive. In the following, we derive another correction term to reduce the approximation gap without calculating the second derivative.
Proposition 4. Given two Gaussian distributions P = ((TT 1,11) and Q = N (1 Then, the
Suppose the Euclidean distance between the means is bounded ||c| _e__1- 1--1-14-11 S total variation distance (TV) between the product distributions of N independent samples satisfies TV(P Q8N)= Ns See Appendix A.3 for detailed derivation.
VAB.
Proposition 4 relaxes the assumption of closeness between two points to the overlap between two distributions. It shows that samples drawn from N(c]" T) can be statistically indistinguishable from those drawn from ((4-1, 3,I) under certain conditions. Motivated by this, our Sample Reward Soup method reuses the search gradients computed solely from exemplars achieve multi- reward alignment, leveraging the distributional overlap to ensure valid gradient estimation.
Sample Reward Soup (SRSoup): Given set of exemplars {c J introduced in defini- tion 2 let (ar""aa 1==1 denote of samples drawn from the exemplar distribution [[c[",,,

---

Published as a conference paper at ICLR 2026
10, and {fm In=1 be the corresponding rewards. Following proposition each component
|m-1 (1 cm) in Equation can be further approximated as
M F( = Ver (c:=1:mm)
1
Σ (( (1-1),
(11)
V
where the second term is regarded as the correction term for the gradient approximation.
Furthermore, we propose two strategies to enforce overlap between the distributions. The first is to initialize from a same noise sample, VT (N0,, I), so that PT-1 = cT_1
= ci (i.c.,
the initialization stage in Fig. 2 The second is sample the same for querying the M reward functions. This minimizes divergence among exemplars optimized for different reward functions, thereby reducing the gap between 14-1 and 1 1
noises construct inputs
Nevertheless, as 3, gradually decreases over time in the denoising process, the distributions may no longer overlap. So we adopt a hybrid strategy. At the early stage the denoising pro- cess (i.e., the red Gaussians in Fig. 2. where the distribution A ( I) exhibits overlap with the exemplar distributions (CT" 3. for all m = .......... M, we perform Sample Reward Soups (Algorithm 2 At the later stage of the denoising process (i.e., the green Gaussians in Fig. 2 where the distribution (V 3,I) be- comes distinct from the exemplar distributions, we adopt the true weighted reward update w.r.t. Equation 9 Let K denote the number of soup steps (Equation [T). We perform SRSoup sam- pling for the first K steps (i.e., when / > T K) and use weighted-sumt sampling thereafter (i.e.. when T K).
cheake
a.4.
20.8 201
440
Remark 1. In the context of diffussoo models, the denoising distributions are isotropic. There- fore, each dimension of pt-1 can be treated as updated independently. Under this setting, our assumption about the overlap of the denoising distributions can be considered dimension-wist making it easier to hold, as il avoids the curse of dimensionality. Empirically, udopt the Bhuttacharyya coefficient (Bhattucharyya evaluate the average dimension-wis difference between the distributions at each time step. which aligns with our theoretical analysis. This observation further supports the feasibility and validity of our proposed method. Please refer to the Appendix for details.
Three blue Gaussians (fully overlapped) denote the initialization stage = T); three red Gaussians (partially overlapped) denote the early stage of the denoising process; three green Gaus- sians (non-overlapped) denote the later stage.
Figure 2:
1943) to
EXPERIMENTS
In this section, we apply our SRSoup to align T2I diffusion models with a diverse variety of reward objectives. Due space limits, we put more results and additional experiment details in Appendix.
5.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments animal prompts from ImageNet (Deng al. 2009) and prompts from Human Preference Dataset (HPD) (Wu et al., (023). For fine-tuning methods, we use 45 animal prompts and 750 HPD prompts the training set following previous settings (Black et al.) 2024) Prabhudesai et al. 2023 Zhang et al. 2024). Evaluation is performed on unseen prompts from the datasets.
7

---

Published as a conference paper at ICLR 2026
625
6.4
Compre sobiiity
038
0.2
-393
6:31
-225
(Ours)
ArgnProp (soup)
Alcerropp (soug)
AlgnProp (soup)
-359
TDPO Isoup)
TDPO tsoup!
TDPO (soup)
DDPO (soup)
DDPO usoup)
DOPO (soup)
50
Aesthetic
Ansthatic
Aesthetic
Figure 3: Pareto front comparison SRSoup and model soup baselines, aligned with two rewards wifi + w2f2. The depth of the marker color denotes the value of W1; deeper indicates larger.
#1
0.0
0.8
SRSoup (ours)
TDPO (soup)
DDPO (soup)
AlignProp (soup)
More Compressible
More Aesthetic
Figure 4: Qualitative comparison of T2I model aligned with the aesthetic reward and the compressibility reward fz. Here, wi + ---w)) w1) ) fz. Prompt: "cheetah".
consider commonly used reward models, namely, compressibility func- tion Black et al.) (222),, LAION aesthetic predictor (Schuhmann al. 2022). HPSv2 (Wu et al.) 2023 and PickScore (Kirstain et al. 2023).
Reward Models.
Baselines. To demonstrate the superiority of our method, compare with DDPO (soup) B Black et al. (024). TDPO (soup) (Zhang et al. 2024), and AlignProp (soup) (Prabhudesai et al. 2023). In particular, DDPO and TDPO fine-tune T21 models with RL, while AlignProp tue-tunes models using differentiable reward functions. The soup version means that we first fine-tune models on each reward function independently, obtaining ).M.M. and then interpolate models by {EM w L See Appendix for comparison with other multi-reward fine-tuning methods.
Hyperparameters. Unless otherwise specified, we set the number of denoising timesteps T to 50, the number of query samples per reward function at each timestep N to 30, and the number of soup steps K to 20 (introduced in Section
5.2 COMPARED WITH BASELINES
We adopt Stable Diffusion (SD1.5) (Rombach al. 2022 as the pre-trained T2I model for all methods. Fig. 3 presents a comparison for two-objective alignment between our SRSoup and various model soup baselines in T2I generation across three scenarios. SRSoup consistently achieves superior Pareto fronts, demonstrating its effectiveness leveraging wwward-giided sampling for multi-objective optimization.
8

---

Published as a conference paper at ICLR 2026
Fig. visualizes the generated images for the prompt "cheetah" under different trade-offs between compressibility and aesthetic quality. Results of other animal prompts are put in Appendix. The fine-tuning methods suffer from over-optimization- common issue in such approaches (Kim et al. 2025 wwhich causes the generated images to lose diversity. The images of different prompts optimized for aesthetic reward by DDPO, TDPO and AlignProp show similar backgrounds as shown in Appendix.
Sitsoup (Guns)
Scaling to three objectives. To access the scalability of our SRSoup, we optimize three reward functions, i.e., aesthetic score, HPSv2 and PickScore, on HPD prompts. Fig. shows the superiority of our SRSoup this more chal- lenging scenario. In Fig. 14 (Appendix), our SRSoup achieves the best prompt alignment. TDPO suffers from severe over-optimization, generating images that do not correspond to prompt, when optimizing aesthetic rewards.
AlignProp (soup)
TDPO (soup)
UDPO (soup)
0.25
0.2 g
0.21
0.19
0.29
28
6.0
5.3 COMPARED WITH WEIGHTED SUM GUIDANCE UNDER LIMITED QUERY BUDGET
Aembetic
7.0
0.26
Pareto front comparison for three- objective alignment.
Figure
To demonstrate the efficiency of method, compare with the commonly used weighted- sum guidance for diffusion models inference time, under comparable query budgets. The budget setup is detailed in Appendix. As shown in Fig. []] our SRSoup consistently achieves better multi- reward optimization while requiring reward queries comparable weighted-sum guidance. Moreover, under limited query budget, our SRSoup can achieve comparable performance with 1.8x fewer reward queries for the first two cases and about 2.7x for the last case.
5.4 APPLIED OTHER T21 DIFFUSION MODELS
To demonstrate the generality of our method, we apply our method to SDXL (Podell et al. 2024), which is regarded as the pretrained T2I model. Fig. shows the improvement im- age quality with more advanced Stable Dif- fusion model. The quantitative results can be seen in Appendix.
Figure 6: SRSoup with SDXL base for the aesthetic reward function fi and HPSv2 reward function f2. The weights are 0.0, 0.5, 1.0, respectively.
5.5 ABLATION STUDY
In this section, we study the effects of two
hyperparameters the SRSoup framework: (1) K, the number of our soup steps before the true search gradient from the weighted sum is applied; and (2) N, the number of queries per reward function at each timestep. We also conduct an ablation on the sample reward soup component.
The left panel of Fig. shows the impact of varying K with N fixed at 30. Even without weighted sum guidance (i.e., K 50), SRSoup achieves reasonable trade-off between objectives benefiting from the effective interpolated gradients at the early stage. Reducing K K--incoroorating search gradients from the weighted sum earlier the process-leads to improved performance by leveraging more true rewards. Notably, K < 20 reduces reward queries substantially (by up to 40%) without significantly sacrificing performance. The middle panel varies N with fixed at 30, demonstrating performance gains with more queries. The right panel highlights the effectiveness of the sample reward soup strategy, which provides informative guidance and yields a significantly better Pareto front than the unconditional sampling counterpart.

---

Published as a conference paper at ICLR 2026
Compress obaiity
Compressibbilty lililssliiiityy
100
-40
Welghtedsum
-00
-00
St
SRSo (K-20)
5R Soup (K-30)
SRSoup (K=33)
2-
SeSu (K-40)
SR (K=20)
K=
-18
(K-50)
(N-10)
Aesthetic
Aesthetic
Aesthetic
Figure 7: Ablation study of SRSoup. K refers to the number of the soup steps before switching weighted-sum sampling. N refers the number of queries per reward per time step. The solid line corresponds to SRSoup with different K, while the dashed line represents the method with unconditional sampling for the first K steps and weighted sum sampling thereafter.
CONCLUSION
We introduce SRSoup, the first training-free soup strategy that interpolates reward-guided search gradients from individual rewards for multiple-reward alignment. We provide theoretical analysis supporting the effectiveness of SRSoup, which uses sample reward sharing mechanism fundamen- tally different from the parameter sharing design of existing model soup approaches. In particular, SRSoup consistently outperforms various model soup baselines in both two-reward and three-reward alignment settings, without suffering from vereoopimization. Moreover, it achieves performance comparable to inference-timg weighted-sum alignment while requiring fewer reward queries.
ETHICS STATEMENT
The techniques studied in this paper (multi-rewart alignment), while capable of balancing complex and conflicting human values to enable more nuanced, personalized, and safer AI behavior, may also introduce risks such as biased reward weighting, value conflicts, and potential misuse. Ensuring careful design and governance is crucial to maximize societal benefits while minimizing unintended harms.
REPRODUCIBILITY STATEMENT
Section 5 and Appendix D provide detailed implementation information for our method and baselines, covering hyperparameter settings as well as training and sampling procedures. All experiments use publicly available datasets and evaluation metrics.
ACKNOWLEDGMENT
This work was supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: LISG-NMM.P22224-00 findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Any opinions,
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Anil Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distribution. Bulletin of the Calcutta Mathematical Society, 35:99-1100 10, 1943.
10

---

Published as a conference paper at ICLR 2026
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations. 2024.
Yuanyuan Chang, Yinghua Yao, Tao Qin, Mengmeng Wang, Ivor W. Tsang, and Guang Dai. Instruct- ing text-to-image diffusion models via classifier-guided semantic optimization. In International Joint Conference on Artificial Intelligence, IJCAI, pp. 747-755. ijcai.org, 2025.
Altannar Chinchuluun and Panos M Pardalos. A survey of recent developments in multiobjective optimization. Annals of Operations Research, (:2(1))29-500 2007.
Hyungjin Chung, Jeongsol Kim, Michael T McCann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations, 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations, 2024.
Fei Deng, Qifei Wang, Wei Wei, Tingbo Hou, and Matthias Grundmann. Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7423-7433, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Carles Domingo-Enrin Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In International Conference on Learning Representations, 2025.
Hanze Dong, Wei Xiong, Decpanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang. Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.0676; 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems, 37: 15:477777-22551, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances Neural Information Processing Systems, 36, 2024.
Alexei A Fedotov, Peter Harremoës, and Flemming Topsoe. Refinements of pinsker's inequality. IEEE Transactions on Information Theory, 49(6): 1911-1488, 2003.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. pp. 18835-10866. PMLR, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6851. Curran Associates, Inc., 2020.
Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Elu- cidating optimal reward-diversity tradeoffs in text-to-image diffusion models. arXiv preprint arXiv:2409.0644 00699, 2024.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. In The Thirteenth International Conference on Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 006533662-366655 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv: /2192, 2023.
Seung Hyun Lee, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang, Fei Deng, Glenn Entis, Junfeng Hc, et al. Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-image generation. In European Conference on Computer Vision, pp. 462-478. Springer, 2024.
Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics, (166)3303--114 2020.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint wxXh:22406004314 2024.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl arXiv preprint arXiv:2505.05470, 2025.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-fres free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean- Stan-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Transactions Information Theory, 62(11):5933-6066 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2529. 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
12

---

Published as a conference paper at ICLR 2026
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, et al. Amortizing intractable inference in diffusion models for vision, language, and control. Advances in neural information processing systems, 37:76080-761 14, 2024.
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7280-7290, 2023.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-8233 8238, 2024.
Yibin Wang. Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning. arXiv preprint arXiv:2508.2077 2025.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(27):949-99) 2014a.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-99) 9800 2014b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Love Lontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferences of ext-to--mmae synthesis. arXiv preprinl arXiv:2306. 09341, 2023.
Xin Xie and Dong Gong- Dymo: Training-free diffusion model alignment with dynamic multi- objective scheduling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13220-13230, 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024a.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2022.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In International Conference on Machine Learning, 2024b.
Yinghua Yao, Yuangang Pan, Jing Li, Ivor W. Tsang, and Xin Yao. PROUD: pareto-guided diffusion model for mbbti-obeective generation. Mach. Learn., (13(9):6511-653) 2024.
Po-Hung Ych, Kuang- Huei Lee, and Jun-Cheng Chen. Training- diffusioo model alignment with sampling demons. In International Conference Learning Representations, 2025.
Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: perspective of inductive and primacy biases. In Proceedingg of the 4/st International Conference on Machine Learning, pp. 60396-60443, 2024.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one- to---peffeeeeefitteeeeee alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 10586-10613, 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILED PROOFS
A.1 PROOF OF THEOREM I
Proof. The core idea of the proof is apply the log-derivative trick decouple the gradient computation from the reward function. To be specific,
E (x u4 1a) $ (( 1 1og (: BcLa)) pe- F(H
1
E N( f(xr-1) -M-11) m1-1)]
Br
vB. RN ((m-)))
where I applies the log-derivative trick (Wierstra et al. with non-differentiable reward function.
2014a), which enables gradient estimation
A.2 PROOF OF PROPOSITION 3
Proposition 3 Given set of exemplars {e] J.M introduced in Definition 2
(Ct-1/==
Let 6m 14-1
c Vm = 1,2, ....., M, the gradient of f4-11 in Equation can be approximated by the gradients of these exemplars as follows:
M
Me_ F( W.M)
=
Σ
Wm V u 1 F(
m=1
Verz F(ch em) Ve FlcT-1 +
Wm
Proof. Our goal is to approximate the search gradient, guided by weighted rewards, using search gradients computed each exemplar c.- First, have
V V F(u-11, W1.M))
Wm V pI_, F(He-11, em)
Σ Wm V V Em D (f
Then, we approximate each component V E IN 1. fm (mmmfm/x,a. (x)-1)) using exemplar-wist gradient Ven W 1,2, 1, .M.
under the assumption that [|mm is small, m =
Therefore, we expand F(ut-1,, L [fm (x(-1) around oτ using Taylor expansion:
6m) = + 8m, Cm) =[(c" em) + F(p-
-- 5m
T m m F(u-
Om + 0116mm11))
Me 1--10
Taking the gradient of both sides with respect to 1 gives:
pmm F(pt-11 em) Ver F(c" + cm F(c 0(16mm))))
Summing over all m, the weighted gradient becomes:
M
M
Σ
Wm V p._, V F(ui-1,
Σ
Wm (Ver, elc V2 el 0(16mm l3))
m=1
m=1
(12)
where 8n 14-1 -c Vm 1,2,.... M,
M, and complete the proof.
14

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF PROPOSITION 4
Proposition Given two Gaussian distributions P 11.11 and Q 1, 1.
Suppose the Euclidean distance between the means is bounded as || ph-1l z E. Then, the total variation distance (TV) between the product distributions of N independent samples satisfies TV(PeN Q8NN Nc
Proof. In the following, we first compute the KL divergence between P and Q, which, for Gaussian distributions with the same covariance 3,I, depends only on the distance between their means. We then apply Pinsker's inequality Fedotov al. (2003) to bound their total variation distance.
First of all, since both P and Q are Gaussian distributions with identical covariance matrices, their Kullback-Leibler (KI.) divergence has closed-fort form expression:
8²
KL(PIIQ)=
|
14-111 28,
23,
By Pinsker's inequality, which relates KL. divergence to total variation, we have:
TV(P,Q) < (I((P)Q) S
V43.
For the product distributions N independent samples, the total variation distance satisfies the following inequality:
Ne
TV(PSN QaN) s N TV(P.Q) <
V4B
This implies that the statistical distance between N samples drawn from P and N samples drawn from Q is at most
Sample Reward Soup (SRSoup): Given set of exemplars {c Jm introduced Definition let {ox m.n J==1 denote set of samples drawn from the exemplar distribution (cc 1,, and (fm M be the corresponding rewards.
(m=11)
For each exemplar ci- we have
F cL where Wi.h NNeT"1::
where vn = N.
M Following Proposition 4 these samples (- In=1 can be also considered as sampling (11-1,,811) is sufficient small. So, we have
when v
zn x 41-1 = 2" c pt-1, where x! N(H1-1,8;1),
where Vn- 1,2. N.
Then, these samples can be leveraged for reward-guided search gradient approximation of each component vn-,J F(u--1, in Equation 3
p F(m-
Σ Im (m""")))"
V
Σ Jm (x""))) + (c - 11-1))
N
N Σ (( ((-1);
V oz F (c em)
n=l
15

---

Published as a conference paper at ICLR 2026
B EMPIRICAL VERIFICATION USING BHATTACHARYYA COEFFICIENT (BC)
Following Proposition 4 to empirically demonstrate that samples from the exemplar distributions can be reused for other denoising distributions, we adopt Bhattacharyya Coefficient (BC) Bhattacharyya (1943) to measure the overlap between two distributions. BC is normalized value in the range [0, 1|. making it convenient for reference across different image generation cases. Given two continuous probability density functions p(z) and ((x)), BC is defined as
/ p(() ((x)
BC(p, ④)
(13)
A higher BC value indicates greater similarity (i.e., more overlap) between the two distributions. Note that the total variation distance (TV) between two distributions upper bounded by the BC (Sason & Verdú 2016), i.e., (p. 9) S 2(1 BC(p. q)
In the context of diffusion models, the sampling distribution is isotropic, and each dimension of æ follows an independent Gaussian distribution with identical covariance 8r at each time step t. The Bhattacharyya coefficient can be calculated dimension menession-wiseaee as:
BC(c Md,t-1) exp (-
1 (-83,
83, Md,t-1 Cd.c-
(14)
where Mudc-11 and cd denote the d-th dimensions of μ and ci respectivlly.
We plot BC values for 14-1 and c. in Fig. 8 under the setting of optimizing the aesthetic and compressiblity rewards using the true black-box gradient of weighted sum objectives. Note that the weights for ci-1, d and are [1,0), [0, 1], [0.5, 0.5). respectively. In Fig. 8 at the beginning, have BC(CTTT-1 (d,T-1) 1 because 4T-1 ct-
cx resulting from initialization with the same noise æy /N. I). At the early stage of denoising process (i.e., t 30), more than 50% of the dimensions have BC(c), Ma,x-i) > 0.4. which suggests that our SRSoup provides a
reliable approximation. At the later stage, this term approaches zero, i.e., BC(c Md.1-1 (- 0. making the approximation ineffective.
1.0
between cj, and c/
BC between c and Mer
BC between di, and
Bhattacharys Coefficent (BC)
0.8
0.6
0.4
0.2
30 Inference step
Figure 8: The Bhattacharyya coefficient (BC) is caleulated between any two of the following de- noised distributions, i.e., N' 1, 3,I), the legend, they are simplified as "BC between distribution mean * and ** respectively. Since BC is calculated dimension-wise the solid line represents the median BC across all dimensions, while the shaded area covers the range between the 25th and 75th percentiles. Note N ( S,I) optimizes the aesthetic reward function f1. (x,-1: 3,1) optimizes the compressibility func- tion fa and N (#+-1; Mt-1, 3.I) optimizes 0.5ft + 0.5f2.
In // [x 1,T1) and N (2
16

---

Published as a conference paper at ICLR 2026
Table 1: Cross-reward evaluation for the of aligning with the aesthetic compressibility reward Wi fi + w2f2 2 on animal prompts.
Cross-reward
Method
W1 0 W1
0.2 w1
0.4 w1 = 0.6 w1
0.8 w1
SRSoup
0.2692
0.2672
0.2735
0.2690
0.2549
0.2532
CLIP (SD1.5: 0.2589)
AlignProp (soup)
0.1844
0.2298
0.2650
0.2446
0.2527
0.2464
TDPO (soup)
0.2379
0.2628
0.2553
0.2403
0.2590
0.2361
DDPO (soup)
0.2583
0.2665
0.2582
0.2561
0.2573
0.2552
SRSoup
0.2401
0.2423
0.2581
0.2676
0.2680
0.2673
HPSv2 (SDI.5: 0.2731)
AlignProp (soup)
0.2189
0.2338
0,2571
0.2708
0.2669
0.2637
TDPO (soup)
0.2372
0.2538
0.2629
0.2638
0.2556
0.2514
DDPO (soup)
0.2440
0.2562
0.2644
0.2681
0.2687
0.2621
This justifies our hybrid strategy for multi-reward guidance. At the early stage, when the dis- tribution N (1 I) exhibits significant overlap with the exemplar distributions. [N 8) Vm = 1, 2,2,... ...,. M,, we perform Sample Reward Soups (Algorithm 2 the later stage, when the distribution ( I) becomes distinct from the exemplar distributions, we adopt the true update guided by weighted rewards (Equation 9)
C ADDITIONAL EXPERIMENT RESULTS
C.1 CROSS-REWT REWARD EVALUATION
We include the cross-evaluation metrics for the three scenarios in Fig. Table 1 Table and Table show the evaluation using the CLIP score, which measures prompt alignment and the evaluation using other unseen reward models during alignment.
The evaluation results for the pretrained Stable Diffusion 1.5 (SD1.5) can be taken as the reference. From the above tables, we can find that: (1) AlignProp (soup) has significant drop (denoted in red color) in the cross-evaluation metrics compared SD1.5 when aligning with the aesthetic and compressibility rewards, which indicates reward over-optimization, consistent with the visual results shown in Fig. 4 (2) TDPO (soup) also suffers from the similar issue when optimizing the aesthetic reward HPD prompts (Fig. 14.. (3) When aligning with the aesthetic reward (w² 1), observe some drops in the CLIP metric compared to SD1.5 across all methods, possibly indicating loss of prompt alignment. But our SRSoup still achieves better results than other baselines.
Table 2: Cross-reward evaluation for the case of aligning with the aesthetic & HPSv2 reward
Wi fi + W2f22 on animal prompts.
Cross-reward
Method
W1 = 0 W1 = 0.2 W1
0.4 wh = 0.6 w1
0.8 w1
SRSoup
0.2571
0.2566
0.2591
0.2545
0.2445
0.2324
CLIP (SD1.5: 0.2574)
AlignProp (soup)
0.2627
0.2646
0.2635
0.2631
0.2477
0.2171
TDPO (soup)
0.2542
0.2613
0.2662
0.2583
0.2110
0.1128
DDPO (soup)
0.2579
0.2568
0.2516
0.2509
0.2490
0.2237
SRSoup
0.2300
0.2291
0.2277
0.2236
0.2189
0.2142
PickScore (SD1.5: 0.2190)
AlignProp (soup)
0.2198
0.2216
0.2213
0.2214
0.2162
0.2069
TDPO (soup)
0.2198
0.2221
0,2218
0.2201
0.2056
0.1827
DDPO (soup)
0.2219
0.2235
0.2242
0.2229
0.2163
0.2065
C.2 SRSOUP APPLIED IN SD3 AND SDXL
Our SRSoup method also generalizes to other T2I diffusion models, such as SDXL, and flow models based diffusion transformers (DiT), such as Stable Diffusion 3.5 Medium (SD3) (Esser al. 2024). Fig. d) shows that SRSoup improves both the aesthetic reward and the PickScore reward compared to the pretrained SD3, and achieves trade-off between the two rewards.
17

---

Published as a conference paper at ICLR 2026
Table 3: Cross-reward evaluation for the of aligning with the aesthetic & PickScore reward
Wi + w2f2 on animal prompts.
Cross-reward
Method
W1 0 W1
0.2 w1
0.4 W1
0.6 w1
0.8 w1
SRSoup
0,2625
0.2594
0.2562
0.2543
0.2451
0.2238
CLIP (SDL55: 0.2574)
AlignProp (soup)
0.2404
0.2423
0.2500
0.2560
0.2460
0.2171
TDPO (soup)
0.2468
0.2522
0.2548
0.2526
0.2063
0.1128
DDPO (soup)
0.2439
0.2460
0.2475
0.2524
0.2449
0.2237
SRSoup
0.2878
0.2872
0.2860
0.2833
0.2762
0.2714
HPSv2 (SD1.5: 0.2787)
AlignProp (soup)
0.2785
0.2793
0.2797
0.2790
0.2747
0.2655
TDPO (soup)
0.2742
0.2803
0.2794
0.2783
0.2662
0.2493
DDPO (soup)
0.2826
0.2829
0.2820
0.2796
0.2735
0.2647
SRSoup (Ours
SR5oup (Oum)
SRSoup (Oum) D3PO (20ug)
FTWS
RIC
Compressilbilty
Compressionn
-12
-36
6.0
25
Aesthatic
(a)
(b)
(c)
SiSuup (Durs)
SRSoup (Ouss)
GAPO
E205-
(209
PiclScore
<240-
6285
4299
6230
(300
1280-
6385
6225
6330
520
5.15
6.5
635
64
64
CA
75
Ansthetic
Ansthatic
(d)
(e)
Figure (a) SRSoup with the SD1.5 base vs. tine-tuning with weighted-sum objectives. (b) SRSoup with the SD1.5 base vs. RiC. (c)SRSoup with the SD1.5 base vs. D3PO. (d) SRSoup with the SD3 base vs. Flow-GRPO. (e) Pareto front of SRSoup with the SDXI. base, aligned on two rewards.
Fig. b shows that SRSoup improves both the aesthetic reward and the HPSv2 reward compared to the pretrained SDXL, and achieves trade-off between the two rewards. Visual results provided in Fig [5] Compared to the pretrained SDXL, SRSoup improves both the visual aesthetics and the prompt consistency.
C.3 COMPARED WITH OTHER FINE-TUNING METHODS
DPO methods (e.g., DiffusionDPO Wallace al. 2024). D3PO (Yang et al., 2024a)) are designed to guide T2I models using human preferences, which is not directly applicable to the setting of guiding T2I models using multi-reward models studied in this paper. To make a comparison, we follow D3PO, which adopts reward models for providing preference pairs. Specifically, the aesthetic reward model and compressibility reward model are used to provide preference pairs for fine-tuning SD1.5 with the DPO loss, respectively. Then similarly to DDPO (soup), interpolate the two finetuned models for obtaining multi-reward guided samples. Fig. c) and Fig. 12 demonstratess that our SRSoup achieves better performance than D3PO (soup).
18

---

Published as a conference paper at ICLR 2026
200
122
(WV)
14
19 Number soup stops
Number of soup staps K
Number soup staps
Aesthetic & Compressibility
Aesthetic & PickScore
Aesthetic & HPSv2
Figure 10: Hypervolumn (HV) versus different soup steps K.
Flow-GRPO (Liu al. 2025) recently proposed for reward alignment using GRPO. To compare with it, we apply Flow-Git GRPO to align the pretrained SD3 model with the aesthetic and PickScore rewards the HPD prompts, respectively, obtain fine-tuned models. We then apply the soup strategy to generate multi-rewartlaligned samples. Fig. () shows that SRSoup outperforms Flow- GRPO in terms of multi-reward alignment. Meanwhile, the visual results in Fig. demonstrate that SRSoup achieves better prompt alignment.
C.4 COMPARED WITH MULTI-REWARD FINE-TUNING METHOOSS
As diseussed in Section multi-reward training methods have some drawbacks. Fine-tuning weighted sum requires training a large number of models to trade off different reward functions. Parrot (Lee al., 2024) and Rewards-in-contett (RiC) (Yang al. 2024b) rely large supervised dataset. We include comparisons with fine-tuning using weighted sum objectives (FT-WS) and RiC, as shown Fig. 9 Since Parrot's code has been released, we do not compare against it.
SRSoup vs. FT-WS. Fig. a(a shows the results of SRSoup and FT-WS when aligning with the aesthetic and compressibility rewards (wr) + (2)22) on the animal prompts. The depth of the marker color denotes the value of wi: a deeper color indicates a larger value. The values of w1 are set to 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0, respectively. The result demonstrates that SRSoup outperforms FT-WS.
SRSoup vs. RiC. For an case of comparison, we follow the text-to-image setting in RiC to conduct the comparison experiments, which aligns with the aesthetic and compressibility rewards (w,f1 + w2f2) on the CoCo dataset. Fig. (()) shows that SRSoup outperforms RiC.
C.5 HARD PROMPTS vs. EASY PROMPTS
We find that different prompt samples exhibit varying levels of optimization difficulty. As shown in Fig. 16 with a query number of N 20, the prompt "snail" is optimized effectively, while "hippopotamus"
when optimizing the compressibility reward. As N increases, the optimization of "hippopotamus" improves. This is reasonable, as the concept of a "snail" is simpler than that of "hippopotamus". As discussed in the limitations of the main text, future work could focus on developing an adaptive query allocation strategy based on the optimization difficulty of individual samples under fixed query budget.
C.6 COMPARISON OF REWARD QUERIES AND INFERENCE TIME
As elaborated in the main text, the update guided by weighted rewards (Equation 9 a.k.a., weighted sum) requires NL M 1) per prompt. Our SRSoup would require NTM M NKM(L- M). The reduction in reward queries is N (L M).
In Table compare SRSoup under different settings of K to the weighted sum method in terms of the required reward queries and inference time for generating a single image. For SRSoup with K 20, which achieves performance relatively close that of the weighted sum method (Fig.) 7 in the main text), the reduction in reward queries reaches 4800. In fact, the reduction can be around 40%
when large number of weight vectors are considered (limp-+to NTM M+1]
NKM M)) K 40%).
19

---

Published as a conference paper at ICLR 2026
We plot the HV in terms of K in Fig. and present GPU memory and inference time breakdown in Table 3 and Table respectively.
Table 4: The total number of reward queries and inference time required for one prompt under different setting of K while optimizing the aesthetic reward fi and the compressibility reward f2- N,T, , L are 30, 50, 6, respectively.
SRSoup K 30
Method
WeightedSum
K 50
K 40
K 20
K 10
#Query
3,000 9.7 0.1
5,400
7,800
10,200 21.1 + 0.2
12,600
15,000 28.8 = 0.3
Time (min)
13.7 0.3
17.3 0.2
2 0.2
Table 5: The peak GPU memory required for SRSoup and WeigthedSum (WS) when optimizing
aesthetic & compressiblity rewards the animal prompts and when optimizing aesthetic & PickScore rewards and aesthetic PickScore rewards on the HPD prompts in Fig. [ See Section Dor detailed setup. The difference of required GPU memory due to the different setup of N.
SRSoup
WS(N = 26)
WS(N = 21)
WS(N = 16)
WS(N = 11)
WS(N = 6)
animal 20.9GB
20.9GB
20.9GB
20.9GB
20.9GB
18.4GB
HPD
24.6GB
24.6GB
24.6GB
24.6GB
24.6GB
22.1GB
Table Inference-time (minute) breakdown required for one prompt under different numbers of reward queries. Total refers to the total time for full pipeline inference. Decode refers to time spent
decoding latent embeddings back into images for reward evaluation; Reward refers time spent evaluating decoded images with reward models.
Aethetic & Compressibility
Aesthetic & PickScore
Aesthetic & HPSv2
Method
#Queries
Total
Decode
Reward
Total
Decode
Reward
Total
Decode
Reward
SRSOUP
3000
9.2
8.8
0.2
9.7
9.0
0.3
9.7
9.0
0.3
5400
13.0
12.2
0.4
14.0
13.0
0.6
13.7
12.7
0.6
7800
17.1
15.9
17.4
16.2
0.8
17.6
16.2
10200
20.4
19.2
0.8
21.6
20.0
1.1
21.6
20.0
1.2
12600
24.3
23.1
0.9
25.1
23.3
1.4
25.4
23.4
1.6
WS
3000
8.2
7.5
0.3
9.3
8.4
0.5
9.4
8.5
0.5
5500
12.8
12.1
0.4
14.4
13.3
0.7
12.2
11.1
8000
17.7
16.8
0.6
18.4
17.0
1.0
15.7
14.3
1.0
10500
21.7
20.4
0.9
23.0
21.4
1.2
21.1
19.2
1.4
13000
24.8
23.4
1.0
27.1
25.3
1.4
25.8
23.7
1.7
C.7 ADDITIONAL RESULTS FOR TWO-OBJECTIVE ALIGNMENT
We present more results for the optimization of the aesthetic and compressibility rewards animal prompts in Fig. 18 Fig. Fig. 20 and Fig. 21 The images generated from different prompts, optimized for the aesthetic reward by DDPO, TDPO, and AlignProp, exhibit similar backgrounds. indicating over-optimization in these fine-tuning methods. In contrast, our SRSoup produces images with greater diversity.
A qualitative comparison of the optimization of the aesthetic and HPSv2 rewards is shown in Fig. 17 demonstrates the superiority of our method in prompt alignment. Additionally, the baselines still tend to over-optimize the aesthetic reward on HPD prompts. The results for the the optimization of
aesthetic and PickScore rewards on HPD prompts can be seen in Fig. 22 Fig. 23 Fig. 24 and
Fig. 25
D ADDITIONAL IMPLEMENTATION DETAILS
SRSoup. We run our experiments on NVIDIA RTX 6000 Ada GPU (48GB). Typically, we set the
denoising step T as 50 following previous works (Black al. 2024 Prabhudesai al. 2023 Zhang
20

---

Published as a conference paper at ICLR 2026
et al. 2024). While more queries often lead to better results, N 30 is sufficient to achieve good performance. The number of soup steps is set to 20, based on the analysis of BC explained above. In order to coordinate scales of different reward functions, their mean and standard deviation. The reward model trained on clean images may have inaccurate evaluations noisy images. Thus, we follow commonly used technique reduce errors, which performs evaluation on the image denoised from the noisy images (Eq. (11) in (Chung et al. 2023 ). For the three-objective experiment in Fig. 5 of the main text, we use weights [1.00, 0.0, 0.0), [0.0, 1.0, 0.0), [0.0, 0.0, 1.0). : 3 3 3 0.1, 0.8), [0.1, 0.8, 0. [0.8, 0.1, 0.1),
normalize the reward values using
[0.2, 0.2, 0.6], [0.2, 0.6, 0.2]. 0.6, 0.2, 0. (0.4. 0.4, 0.2) [0.4, 0.2, 0.4], [0.2, 0.4, 0.4]] Pareto front.
to depict the
Baseline. We use the official code from DDPO, TDPO, and AlignProp, with an effective batch size of 256 achieved through gradient accumulation. Following their original settings, DDPO and TDPO are fine-tuned for 200 and 100 epochs, respectively, by default. Because they collapse on the animal prompt (generating meaningless images), evaluated their performance using checkpoints before the collapse. For DDPO, we used the checkpoint at the 130th epoch for the aesthetic reward and the 80th epoch for the compressibility reward. For TDPO, we used the checkpoint at the 80th epoch for the aesthetic reward and the 30th epoch for the compressibility reward. Since AlignProp tends to collapse more easily, we also used pre-collapse checkpoints for comparison. For animal prompts, we early stop at 6 epochs for the aesthetic reward; and 2 epochs for the compressibility reward; for HPD prompts, we early stop at 12 epochs for the aesthetic reward, epochs for the HPSv2 reward and 23 for the PickScore reward. We plot the rewards evaluating on different checkpoints in Fig. 11
Query setup in Section 5.3 In Fig. T the number of denoising timesteps T set to 50 for both methods, and the number of weight vectors is 6 in the two-objective cases. For SRSoup, the num- ber of query samples per reward function at each timestep N fixed at 30, while K is varied to control the overall query budget. Specifically, K is set to 50, 40, 30, 20, and 10, corresponding to 3000, 5400, 7800, 10200, and 12600 required queries per prompt, respectively. For weighted- sum guidance,
match similar query budget, N is set to 6, 11, 16, 21, and 26, resulting in 3000, 5500, 8000, 10500, and 13000 required queries per prompt, respectively. The reference points for calculating HV values in the three cases (aligning with the aesthetic and compressibility re- ward; with the aesthetic and HPSv2 reward; and with the aesthetic and PickScore reward) are chosen as values slightly lower than the worst value among all obtained evaluation results, namely (4, -100, (5, 0.23), (5, 0.18), respectively.
E USAGE OF LARGE LANGUAGE MODEL (LLM)
The LLM (ChatGPT from OpenAl) was applied to refine sentence structures, improve readability, and ensure grammatical correctness, without altering the scientific content, analysis, or conclusions in this paper.
21

---

Published as a conference paper at ICLR 2026
W
Animal
Animal
HPD
HPD
HPD
1,70
520
6224
E272
Checseet
Checkzene
CHECEEE FM
Creckpaar
Animal
Animal
HPD
HPD
HPD
By:
6.5
360-
2284
85.
Chocepolr:
Checiport
Checkpoor
Chepppin
Animal
Animal
HPD
HPD
HPD
Figure 1I: Reward values during the finetuning AlignProp (first row), TDPO (second row) and DDPO (third row) on the animal prompts and HPD prompts.
0.0
0.2
0.4
0.6
0.8
1.0
SRSoup (ours)
D3PO (soup)
More Compressible
More Aesthetic
Figure 12: Qualitative comparison of T2I model aligned with the aesthetic reward and the PickScore reward w1/1 + W2/2. Prompt: "snail".
W:
211
0.2
81
N11
Wtj
SRSoup (ours)
Flow-GRPO (soup)
More PickScore
More Aesthetic
Figure 13: Qualitative comparison of T2I model aligned with the aesthetic reward and the PickScore reward wi/1 /1 + W2/2. Prompt: "Three people are preparing meal in a small kitchen."
22

---

Published as a conference paper at ICLR 2026
1,
w
00, 1.0
0, 1
3
SRSoup (ours)
TDPO (soup)
DDPO (soup)
AlignProp (soup)
Figure 14: Qualitative comparison of T2I model aligned with the aesthetic reward f1. the HPSv2 reward f2 and the PickScore reward fa- w is used to weight fi, f2 and f3- Prompt: "An eye level counter-view shows blue tile, faucet, dish scrubbers, bowls, a squirt bottle and similar kitchen items.".
23

---

Published as a conference paper at ICLR 2026
Pretrained SDXL
W1 0.0
w1 0.2
W1
0.4
Wy
0.8
W
Figure 15: Our SRSoup with SDXL base aligned with the aesthetic reward fi and HPSv2 reward fz. The weights for fi and fz are wi and we backpacks parked in a public place", "Yellow and red motorcycle with man riding on it next to grass", "Classic cars on city street with people and a dog". "A large commercial airliner silhoetted in the sun", "A motorized bicycle covered with greens and beans", "There is a bathroom that has lot of things on the floor", "Ornate archway inset with matching fireplace in room". "A wooden skate with toy elephant inside of it".
W1, respectively. Prompts are "Bicycles with
24

---

Published as a conference paper at ICLR 2026
0.0
W1 0.2
MP1 0.4
0.6
W1
0.8
Wi
1.0
N = 20
N = 30
Prompt: "snail"
N = 20
N 30
Prompt: "hippopotamus"
More Compressible
More Aesthetic
Figure 16: Our SRSoup for different prompts. The target is to optimize the aesthetic reward fi and compressibility reward fz. The weights for fi and are wy and W2 W1, respectively.
0.0
0.2
0.4
0.6
0.8
1.0
SRSoup (ours)
TDPO (soup)
DDPO (soup)
AlignProp (soup)
More HPSv2
More Aesthetic
Figure 17: Qualitative comparison of T2I model aligned with the aesthetic reward fi and HPSv2 reward f2. The weights for fi and wi and w2 wi, respectively. Prompt: "yellow and red motorcycle with a man riding on it next to grass".
25

---

Published as a conference paper at ICLR 2026
Figure 18: Our SRSoup aligned with the aesthetic reward fi and compressibility reward f2- The weights for fi and f2 are wi and W2 = W1, respectively. From left to right, w1 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "snail"; "hippopotamus". "crocodile"; "lobster"; "octopus".
Figure 19: TDPO aligned with the aesthetic reward f1 and compressibility reward fz. The weights for fi and f2 are wy and W2 = 1 - wi, respectively. From left to right, w1 = 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "snail"; "hippopotamus"; "crocodile": "lobster"; "octopus".
26

---

Published as a conference paper at ICLR 2026
Figure 20: DDPO aligned with the aesthetic reward fi and compressibility reward f2. The weights for S1 and f2 are wi and W2 = 1 Wi, respectively. From left to right, W1 = 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "snail"; "hippopotamus"" "crocodile" "lobster"; "octopus".
Figure 21: AlignProp aligned with the aesthetic reward fi and compressibility reward fz. The weights for fi and f2 are and W2 = 1-w,, ww, respectively. From left to right, w 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "snail"; "hippopotamus"; "crocodile". "lobster"; "octopus".
27

---

Published as a conference paper at ICLR 2026
Figure 22: Our SRSoup aligned with the aesthetic reward fi and PickScore reward fz. The weights for fi and f2 are wh and w2 1 1- wT, respectively. From left to right, w1 0.0, 0.2, 0.4, 0.6, 0.8. 1.0. Prompts: "a bunch of people on skiing on a hill"; "A dog standing in front of a doorway."; "A man sitting at table in front of bowls of spices.", "; "A person sitting on motorcycle in the grass."; "A TV sitting on top of wooden stand."; "Several bicycles sit parked nest to each other""; "A bunch of people posing with some bikes."; "Ornate archway inset with matching fireplace in room.".
28

---

Published as a conference paper at ICLR 2026
Figure 23: TDPO aligned with the aesthetic reward f1 and PickScore reward fz. The weights for fi and f2 wi and W2 1 w1, respectively. From left right, w1 0.0. 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "a bunch of people on skiing on a hill"; "A dog standing in front of a doorway."; "A man sitting at table in front of bowls of spices.", "; "A person sitting on motorcycle in the grass."; "A TV sitting on top of wooden stand.": "Several bicycles sit parked nest to each other."; "A bunch of people posing with some bikes."; "Ornate archway inset with matching fireplace in room.".
29

---

Published as a conference paper at ICLR 2026
ALLL FIEL
Figure 24: DDPO aligned with the aesthetic reward fi and PickScore reward fz. The weights for fi and f2 wi and W2 1 wi, respectively. From left right, w 0.0. 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "a bunch of people on skiing on a hill"; "A dog standing in front of a doorway."; "A man sitting at table in front of bowls of spices.", "A person sitting on motorcycle in the grass."; "A TV sitting on top of a wooden stand."; "Several bicycles sit parked nest to each other."; "A bunch of people posing with some bikes."; "Ornate archway inset with matching fireplace in room.".
30

---

Published as a conference paper at ICLR 2026
Figure 25: AlignProp aligned with the aesthetic reward fi and PickScore reward fz. The weights for fi and f2 wi and w2 1 w1, respectively. From left right, wN 0.0. 0.2, 0.4, 0.6, 0.8, 1.0. Prompts: "a bunch of people on skiing on a hill"; "A dog standing in front of a doorway."; "A man sitting at table in front of bowls of spices.", "A person sitting on motorcycle in the grass."; "A TV sitting on top of wooden stand.": "Several bicycles sit parked nest to each other.", "A bunch of people posing with some bikes."; "Ornate archway inset with matching fireplace in room.".
31


Reviews:
[{"review_id": "aVpQcmx5ch", "reviewer": "Reviewer_GEV9", "summary": "The paper Sample Reward Soups (SRSoup) addresses the challenge of multi-objective optimization in Text-to-Image diffusion models by proposing the first query-efficient, inference-time \"soup\" strategy. The core problem is that standard methods for balancing multiple black-box rewards (like aesthetic quality and compressibility) require an exponentially large number of expensive reward queries across the preference space. SRSoup solves this by independently calculating a reward-guided search gradient for each individual objective at every denoising step, and then linearly interpolating these gradients (not the model weights or rewards themselves). This allows the model to efficiently share and reuse sample rewards across different weighted objective combinations, drastically reducing the number of required queries and enabling superior or comparable performance to the Weighted Sum baseline.", "questions": "1. Please provide a detailed breakdown of the wall-clock time and GPU memory consumption across the full inference pipeline, explicitly weighing the reduced reward query time against the overhead of running M parallel guidance steps.\n2. Given the risk of sub-optimal configurations in multi-objective optimization, how does SRSoup compare to simple weighted-sum methods in terms of output balance (i.e., avoiding cases where one reward is disproportionately maximized)?\n3. Please provide a quantitative ablation study showing the performance (Hypervolume) and efficiency (Query Reduction) trade-offs across a range of K values (e.g., K=20, 40, 60).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novelty of Inference-Time Gradient Interpolation: The paper introduces the first inference-time “soup” strategy (SRSoup) for diffusion process, effectively adapting the “Model Soups” concept not to model weights but to search gradients. This approach achieves Pareto-optimal sampling across a preference space without the risks of reward over-optimization often seen in fine-tuning methods.\n2. Addresses a Scalability Bottleneck with High Efficiency: The paper addresses a limitation in aligning with multiple reward functions, which requires exponentially growing number of reward queries, which makes the process highly inefficient. The proposed SRSoup resolves this by significantly reducing the number of queries in the early denoising stages, establishing a more scalable solution for multi-objective T2I alignment.\n3. Strong Empirical Results and Practical Impact: The method achieves significant query efficiency (e.g., up to 2.7x speedup) over the weighted-sum baseline, as demonstrated by the Hypervolume (HV) versus reward query plots.", "weaknesses": "1. Lack of Cost-Benefit Justification: While the paper shows a reduction in the number of reward queries, it fails to provide a comprehensive cost-benefit analysis for the entire inference pipeline. The reward query computation time, which is model-dependent, must be weighed against the overhead of running multiple parallel single-reward guidance steps required by SRSoup's gradient estimation. \n2. Insufficient Analysis of Multi-Reward Trade-offs and Stability: The analysis, primarily relying on Hypervolume, does not rigorously demonstrate that SRSoup maintains a better balance or stability across the Pareto front compared to weighted-sum methods, especially with non-convex reward functions. This omission limits validation of the method's robustness against potential reward over-optimization.\n3. Sensitivity and Justification of the Hybrid Strategy: The reliance on a hybrid strategy using SRSoup only for the first K steps lacks rigorous investigation into the determination and sensitivity of the boundary K.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ajzXCewWBT", "reviewer": "Reviewer_itLM", "summary": "This paper proposes a training-free multi-reward optimization framework that improves generated images of text-to-image diffusion models. In particular, the authors proposed a multi-reward alignment approach that optimize the denoising distribution at each sampling step using black-box optimization strategy. Additionally, they also proposed Sample Reward Soups, a mechanism that combines multiple reward objective that interpolates reward-guided search gradients. Experiment results show that the proposed inference-time framework achieves competitive performance compared to RL and other finetuning methods. They also demonstrated that the proposed Sample Reward Soups is more preferable than naive weighted sum of multiple rewards.", "questions": "I currently recommend this paper for weak acceptance. \nI'm willing to increase my score based on author's responses. In particular, I hope the author can provide missing baselines and related literature.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is sufficiently novel in that it extends the idea of Reward Soap that interpolates multiple finetuned diffusion models to a training-free setup, making it more accessible while provided meaningful insight about the nature of this type of interpolation methods.\n2. The author provided a detailed theoretical analysis of the characteristics of the proposed method, supported by toy experiments on mixture of Gaussian, which are informative.\n3. The experiments are thorough. I especially appreciate the additional experiments and analysis in the supplementary material.", "weaknesses": "1. The author only showed experiments on SD-series U-Net, which is quite outdated at this time. I understand that many of the RL literature still focuses on SDv1.5 and SDXL due to computation constraints. However, as the proposed method is training free, it would be good to show results on state-of-the art rectified flow models based on diffusion transformers (DiT), such as Sana, SD3, Flux, etc. Such inclusion would make the paper more relevant.   It would also be interesting to see how the proposed method compared with the latest RL literature, such as Flow-GRPO. These latest works should also be discussed in the related works.\n\n2. offline policy learning methods such as DPO are discussed but not included in the comparison. these model learns directly from human preference dataset which encompasses a diverse set of preference. It would be interesting to see however this method compare. While it is hard to apply soup to these models as they cannot be directly trained for different rewards, the author can still use other inference-time methods such as best-of-N sampling by matching the inference time compute.\n\n3. for SDXL comparison in appendix, an RL baseline is missing, such as D3PO[2].\n\n\n[1] Liu, Jie, et al. \"Flow-grpo: Training flow matching models via online rl.\" arXiv preprint arXiv:2505.05470 (2025).\n[2] Yang, Kai, et al. \"Using human feedback to fine-tune diffusion models without any reward model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "moG2hgAGES", "reviewer": "Reviewer_k88a", "summary": "This paper proposes Sample Reward Soups (SRSoup), a training-free, query-efficient multi-reward alignment method for text-to-image diffusion models.\nThe core idea is to pre-compute per-reward gradients (“search gradients”) and to linearly interpolate them to approximate the gradients for arbitrary reward weightings.\nBy sharing reward evaluations across weights during the early diffusion steps—when the sample distributions are still overlapping—the method aims to achieve efficient Pareto-optimal sampling under multiple rewards.\nExperiments on Stable Diffusion 1.5 and SDXL demonstrate similar or slightly better image quality compared to fine-tuning methods (e.g., DDPO, TDPO) while reducing reward queries by ~1.8–2.7×.", "questions": "1. How does the performance change when K is varied?\n\n2. Would it be possible to compare the inference time with existing methods such as Rewarded Soups, AlignProp, TDPO, and DDPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles the important problem of reducing query computation cost in multi-objective preference optimization.\n\n2. The number of queries can be explicitly calculated (as shown in Section C.3), and each parameter can be set according to the desired reduction target.\n\n3. The proposed method is clearly formulated and easy to understand; unifying the Gaussian initialization to achieve a first-order gradient approximation is an interesting design.", "weaknesses": "1. The paper does not discuss the impact of varying K on performance.\n\n2. There is no comparison of processing time with existing methods other than weighted-sum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0CwERLNSm", "reviewer": "Reviewer_5nQC", "summary": "- This paper introduces SRSoup, a training-free and query-efficient diffusion-time alignment method for multiple rewards.\n- The core idea is to avoid combinatorial blow-up by decomposing multi-objective optimization into $M$ simpler single-objective problems.\n- The theory clarifies when the approximation is accurate: it holds when the objective’s curvature (second derivative) is small and the initialization is suitably chosen.\n- In text-to-image experiments, SRSoup attains higher rewards under comparable query budgets.\n- The empirical study also examines the roles of key hyperparameters.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "questions": "- Could you extend your method to a per-subproblem optimization strategy with $M_2$ rewards per subproblem, where $M_2 < M$? I believe this could offer a favorable trade-off between quality and cost.\n- For example, with 10 reward functions in total, you could approximately divide them into 5 subproblems, each containing 2 reward functions.\n- Please note that I am NOT requiring any additional experiments for this.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed multi-reward alignment method is training-free and query-efficient.\n- Its query efficiency and accuracy are supported by both theory and experiments.\n- The authors prove they can approximate the search gradient efficiently and accurately under stated assumptions, and they evaluate the finite-sample approximation error.\n- The Pareto front is clearly visualized in experiments, and empirical Pareto near-optimality is demonstrated in Figures 3, 5, and 7.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "weaknesses": "- As noted in lines 327–335, the main concern is that the assumption for Proposition 3 may not hold, leading to inaccurate approximations. The authors suggest switching from SRSoup to the standard method at the K-th step; however, the computational cost will then approach that of the standard “WeightedSum” method.\n- For these reasons, I hope the computational cost as a function of K is clarified concretely in the main paper (I only found Table 4 in the appendix).\n- The authors empirically validated density overlap using the Bhattacharyya coefficient (BC), whereas their theory uses Total Variation (TV). In the appendix and Proposition 4, they provide only an upper bound on TV. I think a lower bound on TV is also needed to justify the choice of K. The true TV might be small if BC is small.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)