BalancedDPO: Adaptive Multi-Metric Alignment
Authors: Dipesh Tamboli
Souradip Chakraborty
Aditya Malusare
Biplab Banerjee
Amrit Singh Bedi
Vaneet Aggarwal
Venue: Accepted by TMLR ()

Abstract: Diffusion models have achieved remarkable progress in text-to-image generation, yet aligning them with human preference remains challenging due to the presence of multiple, sometimes conflicting, evaluation metrics (e.g., semantic consistency, aesthetics, and human preference scores). Existing alignment methods typically optimize for a single metric or rely on scalarized reward aggregation, which can bias the model toward specific evaluation criteria. To address this challenge, we propose BalancedDPO, a framework that achieves multi-metric preference alignment within the Direct Preference Optimization (DPO) paradigm. Unlike prior DPO variants that rely on a single metric, BalancedDPO introduces a majority-vote consensus over multiple preference scorers and integrates it directly into the DPO training loop with dynamic reference model updates. This consensus-based formulation avoids reward- scale conflicts and ensures more stable gradient directions across heterogeneous metrics. Experiments on Pick-a-Pic, PartiPrompt, and HPD datasets demonstrate that BalancedDPO consistently improves preference win rates over the baselines across Stable Diffusion 1.5, Stable Diffusion 2.1 and SDXL backbones. Comprehensive ablations further validate the benefits of majority-vote aggregation and dynamic reference updating, highlighting the method's robustness and generalizability across diverse alignment settings.

Published in Transactions on Machine Learning Research (04/2026)
BaPancedDPO: Adaptive Multi-Metric Alignment
Dipesh Tamboli
Souradip Chakraborty
Aditya Malusare
Biplab Banerjee
Amrit Singh Bedi
Vaneet Aggarwal'
Purdue University Uniiersity of Maryland
Indian Institute of Technology Bombay
University of Central Florida
Reviewed on OpenRevicw: httpps//pppanorii...eee//o/motoa
Abstract
Diffusion models have achieved remarkable progress in text-to-image generation, yet aligning them with human preference remains challenging due to the presence of multiple, sometimes confficting, evaluation metrics (c.g., semantic consistency, acsthetics, and human preference scores). Existing aligmment methods lypically optimize for single metric or rely on scalar- ized reward aggregation, which can bias the model toward specific evaluation criteria. To address this challenge, we propose BALANCEDDPO, a framework that achieves multi-metric preference alignment within the Direct Preference Optimization (DPO) paradigm. Unlike prior DPO variants that rely on single metric, BALANCEDDPO introduces a majority-vote consensus over multiple preference scorers and integrates it directly into the DPO training loop with dynamic reference model updates. This consensus-based formulation avoids reward- scale conflicts and ensures more stable gradient directions across heterogeneous metrics. Experiments on Pick-a-Picc Partilrompt, and HPD datasets demonstrate that BALANCED- DPO consistently improves preference win rates over the basclines across Stable Diffusion 1.5, Stable Diffusion 2.1 and SDXL, backbones. Comprehensive ablations further validate the benefits of majority-vote aggregation and dynamic reference updating, highlighting the method's robustness and generalizability across diverse alignment scttings.
1 Introduction
Text-to-image (T21) diffusion models achieved remarkable progress in generating images that exhibit strong semantic alignment, aesthetic appeal, and overall visual quality (Ramesh et al. 2022 Saharia et al. 2022 Rombach et al.) 2022 Nichol et al. 2021). However, aligning T2T models with diverse and muance human preferences remains an open challenge, particularly in real-world scenarios where multiple, often conflicting, objectives must be optimized simultancously. For example, consider the prompt: "A warning sign for a flooded street." " A successful model must. accurately depict. the key safety element a clear, legible warning sign while also ensuring the context, such as visibly flooded street, is realistic and unabigguus. Balancing clarity, scmantic alignment, and aesthetic quality is crucial in such safety-critical sccnarios, where misaligned or ambiguous oulputs could lead to misinterpretation and potential harm.
Existing alignment methods, such as DiffusionDPO (Wallace et al. 2024), leverage direct preference opti- mization (DPO) (Rafailov et al. 2024) to fine-tune T2I diffusion models using labeled preferences.
Although the methods discussed above are effective for individual metrics, these approaches struggle to balance competing objectives such aesthetics, text-image alignment, and human preference scores. A naive solution is to perform multi-objective optimization (Zhou et al. 2024), which involves directly modifying the loss function to combine multiple metrics or rewards. However, this approach faces critical challenges: (C1) reward rescaling issues, as rewards often operate on different scales (aesthetics and HPS scores have different ranges), loading to dominance by specific rewards rewards/metries metrics and biased alignment: (C2) conflicting
*Code:

---

Published in Transactions on Machine Learning Research (04/2026)
Prompt: A cyberpunk golden retriever coding
Scorers
Image:
Image
S
S2
S3
S4
Sn
Figure 1: Overview of the BALANCEDDPO framework for the prompt "A cyberpunk golden retriever is coding." Multiple reward models (Scorers S₁ to S.) evaluate a pair of generated images. Each scorer casts a binary vote for their preferred image. Image 2, receiving the majority of votes, is designated the Winner. This majority-votie voting mechanism resolves the scaling disparities between metrics (e.g., Si [0, 1] vs. S2 [1, and avoids gradient conflicts, outperforming traditional normalization strategies demonstrated in results.
gradients, where improving one reward may compromise others, making convergence unstable; and (C3) pipeline complerity, as integrating multiple metrics would requires substantial modifications the standard DPO framework, complicating implementation and reducing model generalization.
To address these limitations, propose BALANCEDDPO, a novel multi-metrie alignment framework that extends the existing DPO (Rafailov et al. 2024 methodology with minimal changes. The core innovation lies in defining how preferences are aggregated; instead of mixing multiple rewards directly in the loss function, which results in scaling and dominance issues, as shown in our ablation studies, BALANCEDDPO operates in the preference distribution space. By leveraging majority-votig based aggregation strategy inspired by social choice theory (Praaad 2018), our method curates balanced preference dataset that.
2

---

Published in Transactions on Machine Learning Research (04/2026)
DiffusionDPO
BalancedDPO (Ours)
Aesthetics
CLIP
HPS
PickScore
Pick-a-Pic
Figure 2: Comparison images generated by models trained on image-text pairs from the Pick-a- Pic dataset and preference labels based on different score metrics (Aesthetics (Schuhmann) 2022), CLIP (Ramesh al. 2022,, HPS (Wu et al. 2023), PickScore (Kirstain et al. 2023), Pick-a-Pic labels (Kirstain et al.] 2023 and PA (combining all metrics) across four prompts: "a boy playing chess", "A Pivor style blue rabbit", "person riding shark", and "pirale guinea pig". Results show that single-metric models often fail in either aesthetics or prompt alignment. For instance, the Aesthetics model generates cartoonish "shark rider,' while Pick-a-Pic labels produce a realistic but incomplete image. contrast, BALANCEDDPO achioves superior performance across all cases, highlighting the benefit of multi-metric optimization.
dynamically combines diverse metrics such as CLIP score, aesthetic quality, and Human Preference Score (HPS). As shown in Fig. [ multiple scorers are used to cast votes in favor which image they find superior. This relative choice mitigates the need for normalization over widely varying scoring scales and helps us continuously incorporate feodback into the alignment process. This approach captures the muancdd trado-offs between multiple objectives, allowing the model align with diverse preferences without requiring complex modifications the DPO pipeline. Fig 2 demonstrates how single-metric models can fail in alignment aesthetics, while BALANCEDDPO achieves multi-metric optimization through the aggregation strategy.
We summarize our contributions as follows:
novel paradigm for multi-metric alignment: We introduce BALANCEDDPO, a framework that performs preference- based alignment in text-to-image diffusion models using a multi-metrin majority- vote consensus. Unlike prior approaches that optimizo a single reward or scalarized combination, our method aggregates diverse scoring models (e.g., CLIP, HPS, PickScore, Aesthetic) into unified binary preference label used directly within the DPO loss (Eq. 7-9). This design avoids scale-dependent gradient conflicts and enables robust alignment across heterogencous objectives.
Dynamic reference model updating: In contrast to traditional T21 alignment methods that fix a base reference model, BALANCEDDPO periodically updates the reference model with the current model parameters (Alg. 1), maintaining stability while allowing exploration in new alignment.
3

---

Published in Transactions on Machine Learning Research (04/2026)
Prompt
realistic furry anthro fox
Incineroar from Pokémon
Corgi with helmet bicycle
An elephant furry anthro
To beat game
SDXL
1H:25 0:0.1 A:4.57
|8 C.0.35 A44.43
H.0.28 C:0.41 JA4452
(9:029 [C:41 JA4.11
P.0.18 H:0.7
DiffusionDPO
10 |0035||Ac6.4 A48.44
|180 1A44.00
|1:0 C:0.39 /A4.322
(A4.55
BalancedDPO
(Ours)
|1:002000C..0000 H:0.20
Figure 3: Pick-a-Pic (Kirstain et al. 2023 comparison for SDXL. Comparison of images generated by various SDXL fine-tunes (specify versions if applicable) and BALANCEDDPO (Ours) on the Pick-a-Pic dataset across diverse prompts. BALANCEDDPO generally creates images that are more realistic and possess finer details. They are also superior in terms of prompt alignment and visual attractiveness, The columns emphasize BALANCEDDPO's strength in areas like background details, object details, prompt adherence, and overall aesthetic. The values displayed underneath each image indicate PickScore (P), Human Preference Score (H), CLIP score (C), and Aesthetic score (A).
directions. As demonstrated in our experiments, this mechanism substantially improves convergence and consistency across multiple datasets and evaluation metrics.
Empirical Evaluations: We conduct extensive exporiments demonstrating that BALANCEDDPO outperforms prior alignment methods across both Stable Diffusion 1.5, SD 2.1, and SDXL backbones. For instance, the Pick-a-Pic dataset (Kirstain et al. 2023 BALANCEDDPO achieves 85.2% win rate in HPS, 78.4% win rate in CLIP score, and 69.4% win rate in aesthetic quality, outperforming DiffusionDDPO by 14.5%, 20.2%, and 16.6%, respectively. On the newly added SD 2.1 experiments, BALANCEDDPO further demonstrates consistent. gains: it achieves 62.33 in HPS, 65.33 in CLIP, 69.33 PickScore, and 65.33 in Aesthetic score when compared against SD 2.1, and surpasses DiffusionDPO by 33.67, 65.67, 69.67, and 70.00 in the respective metries. Consistent improvements also observed the PartiPrompt dataset Yuu et al. 2022), discussed in 4.2
4

---

Published in Transactions on Machine Learning Research (04/2026)
Mechanical bee flying
A 45 year old African American woman casual clothes standing pork, ongled light, professional marketing photography
Prompt
Beautiful goddess, detailed face, focus eyes, masterpiece, realistic
nature, electronics, motors, wires, buttons, Itd, instead eyes, antennes instead of feet
Ruby oval ring arabic jali design with net style pattern diamonds feminine delicate
Pirate guinea pig
SD VZL3
H027 (00.33 JA4.30
P:0.10
P.0.8 |,0.22 [C:042 (A4.38
C:0.33 A4.36
H | AA4.00
DiffusionDPO
P.0.8] H0.27 1A4.42
C:0.37 JA4.34
BalancedDPO
(Ours)
C:0.39|
|60.220
[C:032
Figure Pick-a-Pic (Kirstain et al., 2023) comparison. Comparison of images generated by SD1.5, DiffusionDPO, and BALANCEDDPO (Ours) across various prompts. BALANCEDDPO consistently produces more realistic and detailed outputs, outperforming the other models in aligning with prompts and visual appeal. Each column highlights BALANCEDDPO's superior performance in aspects like facial detail, dynamic motion, adherence to prompt details, and image reflection. The scores below each image represent PickScore (P), Human Preference Score (H), CLIP score (C), and Aesthetic score (A).
2 Related Works
RLHF Based Alignment of Diffusion Models: Reinforcement Learning from Human Feedback (RLHF) has been pivotal in aligning diffusion models to human preferences, optimizing output quality beyond standard
training. Foundational work in this field Fan et al. 2024 Wang et al. 2022 Lee et al. 2023 extended RLHF methods to domain-spocific diffusion models, ensuring outputs met specialized constraints, such as those high-resolution photorealistic generation. This improvement was realized in these models through training with Proximal Policy Optimization (PPO) (Schulman et al. 2017). allowing iterative tuning that addressed visual fidelity and semantic alignment (Zheng et al. 2023a Black et al. (Black et al. 2023) highlighted enhancements in complex generation scenarios by integrating RLHF with training strategies for diffusion models in specific reinforcement learning contexts. Chen et al. (Chen et al. (024) explored the application of RLHF to diffusion- based text-to-speech synthesis, using mean opinion scores as a proxy loss and introducing luss-d reinforcement learning policy optimization of the diffusion model improve speech quality and naturalness. Furthermore, Dong al. (Dong al. 2023) proposed framework that uses RLHF to quantify
5

---

Published in Transactions on Machine Learning Research (04/2026)
human preferences and guides diffusion planning for the customization of zero-shot behavior, effectively aligning agent behaviors with diverse human preferences. But all of the RIHF-based approaches require access to reward function during fine-tuning as well as restricted to the use of PPO algorithm, which exhibits instabilities in practice (Zheng et al. 2023b
Direct Preference Based Alignment: DPO (Rafailov et al. 2023 designed to align generative models with human aesthetic and content preferences by directly optimizing the model's parameters based on human feedback, thus eliminating the need for intermediate reward model. DiffusionDPO (Wallace et al 2024 extended DPO from language-modee model-based formulation (Rafailov et al. 2023 to account for the unique likelihood structures of diffusion models, utilizing the evidence lower bound to derive a differentiable objective. This approach is then applicd to fine-tune the Stable Diffusion XL (Podell et al. 2023 model using the Pick-a-Pic (Kirstain el al., 2023) dataset, resulting in significant improvements in visual appeal and prompt alignment as evaluated by human judges. The direct preference for Denoising Diffusion Policy Optimization (D3PO) method (Yang et al. 2023), which fine-tunes diffusion models dircctly based on human feedback without the need for reward model, demonstrates effectiveness in reducing image distortions and generaling safer images. Other work extends this idea to relative (Gu et al. 2024) and diffusion step-aware (Liang et al, 2024) preference optimization, as well as to other modalities like structure-based drug design (Cheng et al. 2024). But existing DPO-based approaches for diffussio models are limited to single preference leedback, and aligned models suffer from suboptimal performance with respect to multiple metrics, which are important in real-world applications.
3
Problem Formulation
3.1 Background and Preliminaries
DiffusionDPO: DPO [Rafailov et al. 2024 is method originally developed for aligning language models with human preferences (Rafailov et al. 2024). DPO focuses on learning from pairs of ranked outputs, where one sample is preferred over another. Let zi and d' represent the "winning" and "losing" samples, respectively, for a given conditioning c. The Bradley-Terry (BT) model (Bradley & Terry 1952) then used. to express human preferences as
PRT(2" > =gge)) = (r(e, a8) r(c, (c)))));
where a is the sigmoid function, and r(c, represents the latent reward model that scores the preference of sample To given conditioning c. Reward model is learned via maximum likelihood estimation for binary classification:
LBT(r) --Bo, =n (rg,(((()))))) r(c, m/) xb))]]
(2)
where the expectation is taken over text prompts and data pairs (=)) z/) from a dataset annotated with human preferences. The direct. preference oplimization in (Rafailov el al. 2023) bypasses the reward learning step and proposes to directly optimize the model from preference feedback. DiffusionDPO (Wallace et al.) 2024) adapted DPO specifically for diffusion models, introducing a preference optimization objective that aligns the reverse process pe (No:L) with the human-preferred distribution. The optimization objective Eq. (12) of (Wallace et al. 2024 is expressed as:
-2 -- B logo Eg log
po po log
po (o(ob) pror(z()) poltb)))
(D.
(ret(x?)))
where the outer expectation Ey is with respect to (x)"==)) ~ D, inner expectation E2 is with respect to TT.1. ~ "P(aat 2i.r. ~ Po(e laf) where æ1.L denotes the latents of the diffusion path leading to Zo, a(.) denotes the sigmoid function, B 0 is regularization constant, and D represents the dataset of paired human preferences.
Remark: The objective in equation aligns the model based on given preference dataset, which typically reflects single utility metric. For example, the dataset might prioritize selecting images with better aesthetic

---

Published in Transactions on Machine Learning Research (04/2026)
punk rock squirrel in studded leather jacket shouting into microphone while standing an a lily pad
Prompt
helicopter hovering over Times Square
map of United States with a pin an San Francisco
Anubis wearing sunglasses and sitting astride hog matorcyle
Graffiti of furny dog street wall
SD Seva
P:0.10 H..28 10025AA418
P:0.17 H:024|| 1 C:0.22 (A4.34
C:0.98
P:0.19| H:0.24 C. 1..00/AAA A44.0
DiffusionDPO
H50.25| CC0.55 1/4.25
C:0.5 (A4.43
BalancedDPO (Ours
Figure PartiPrompt (Yu et al., 2022) comparison. Comparison of images generated by SD1.5, DiffusionDPO, and BALANCEDDPO (Ours) out-of-distribution prompts from the PartiPrompt dataset. BALANCEDDPO consistently generates accurate and realistic outputs, including specific elements like helicopter, microphone, and lifelike dog, while the other models produce incomplete or irrelevant results.
quality over others. However, this approach fails to capture the multifaceted requirements of real-world image generation, where preferences often span multiple attributes such as semantic alignment with text prompts, aesthetic appeal, diversity of generated content, and adherence human preference scores. Balancing these diverse and sometimes conflicting objectives is essential for creating robust and versatile text-to-image models suitable for practical applications.
3.2 Multi-metric Alignment for Diffusion Models
We start by highlighting the limitations of single-utility-baser preference optinization methods in the context of the text-image diffusion models (see Figure 2). For given text-to-image diffusion model pe pe(zolc) where Zo is the generated image and conditioning text prompt, the goal to align the model outputs (which is a generated image) to human preferences and diverse attributes. As discussed earlier, traditional approaches typically optimize a single reward function r(c, 20), which reflects specific aspect of user preference, such semantic alignment, aesthetic quality (Wallace al. 2024 The standard single-reward based alignment

---

Published in Transactions on Machine Learning Research (04/2026)
be expressed as:
max (en 3 (Tole) (ree(tlee))
(4)
Pe
where Dr represents the distribution over prompls and ro)) is the reward function capturing alignment. with a particular preference criterion. For diffusion models, since reward evaluated for the final generated image out of trajectory ToT; we can define the reward function as r(c, the reward for the full chain To. mentioned in (Wallace et al. 2024). For the sake of discussion and to motivate the contributions in this section, it is sufficient to consider the reward model r(c, only.
Exr [R(c, Morr),, where is
To this end, we remark that the alignment formulation in equation is limited in capturing multi-faccted human preferences because optimizing solely for r(c, zo) may result in overfitting to specific altributes, reducing the general performance. As a naive (vanilla) solution to it, we define the multi-objective preference optimization problem next.
Multi-objective Preference Optimization: We assume access to diverse reward functions {ri(ee zo) J each representing a distinct alignment metric. These metrics may include human preference score (HPS), CLIP score, acsthotic quality, etc. In an idcal scenario, there cxists a latent, unknown aggregation function f that combines these reward functions to reflect the true composite preference which can be captured as
s(c, xo) f(ri (c, r2(c, zo), "r,0)".,"xx( ,rx(c,
where s(c, zo) is the aggregated score representing the true overall alignment quality the generated image zo given the prompt c. Unfortunately, it almost impossible have access to this function f(11,22 rk).
A fundamental challenge: Due to the unavailability of the truc aggregation function f(11, r2. rkk directly applying DPO methods becomes challenging since we cannot compute single, invertible reward (refer to (Rafailov et al. 2023 Wallace et al., 2024) for the invertibility requirement) that accurately captures the latent preferences, leading to issues in defining a consistent optimization objective, similar to one we have equation 3 On the other hand, direct aggregation approaches, such as a weighted sum or averaging of reward functions would face several issues: 1. Unknown weights: The optimal weights for combining rewards are typically unknown, and estimating these weights requires separate optimization process, which can introduce additional loss terms and complexity. 2. Scale and Dominance: Each reward function may operate on different scales, which can cause direct summation to disproportionately favor certain metrics leading to sub-optimal and biased alignment. 3. Conflicting Objectives: Directly optimizing the aggregated function can introduce conflicts, where improvement in one reward, complicates convergence.
Therefore, an effective and robust preference aggregation mechanism required mitigate the issues which is agnostic to the scale and dominance of individual reward attributes.
3.3 Our Method: Majority Voting Aggregation
Key idea: To mitigate the issues discussed above, we take motivation from Social choice theory majority rule Prasad 2018) to perform aggregation of the preference labels after collecting them from different rewards for the same pairs of data points. This approach is agnostic to the reward scale and the need for optimal weight parameters required in direct aggregation methods.
Our proposed method: We have K attributes given as 72. and a fixed dataset prompts and image-pairs as D {(c, x]))]. First, we define score function sy (c,c, .n. mg) corresponding to each reward model as
{
if re(c, 2 ra(c, by otherwise.
sk = +3)
(6)
Based on the above scores, define our aggregation strategy motivated by the majority rule
s(c, -/h 22) sign
Σ
8k
(7)
8

---

Published in Transactions on Machine Learning Research (04/2026)
which indicates that = 1, when majority of the attributes prefers z over znj and s -1, otherwise. This aggregates these individual preferences via majority voting. We utilize the score a(c, =b) to decide whether the pair (xb, 2g) is (2t =x) (when =), =ū) =1) ) or =f") (for s(c,a), 25) = -1). We do it for all
the data points in data set D. Using score function definition in equation 7 and the DiffusionDPO loss in equation 3 holds that.
Lagg (0)
B (a)x)=0)~D [log (B.s [oo 2 =f)1)]
(8)
where, l0. xz} = E2 log Next, with this definition, the gradient of the loss function is given by
po(ele)) log po (ugle) po(alee)
po (ugle) for notation simplicity and Eg same as in equation 3
VaLugz (0) = EE [3. 8- a (-3 lo(s, ab. = Volo(e, =d.(()
The above formulation results in a simplified and efficient objective almost similar to direct preference optimization and agnostic to the scale of the reward functions.
Advantage over direct aggregation strategies: We show that using naive direct aggregation strategy (see Table 3) might lead to an issue in aggregating gradient due to potentially conflicting objectives leading to slower convergence. The naive direct aggregation strategy given as optimizing the direct preference loss
(o) -R (1( Σ we log a (B 8h lale, e, =b
(10)
where Wk: Sk represents the weights (unknown) and preference for the 8th attribute. Similarly, the gradient of the objective boils down to
(o) -ED
Σ wh 3yso (-Blo(c, ab. =5)) Volo a/. 2n))
(11)
To mathematically illustrate the gradient contradiction, consider scenario with two reward models, and Tj that disagree on a preference (s; = 1, = -1). In the naive aggregation strategy (Fq. [T. the resulting gradient term for a given pair is proportional to (w; s; + +wy 8,) Valg. If the weights W;, Wy are similar, the terms (1) and w-1- effectively cancel each other out, leading to vanishingly small update. This "gradient cancellation" stalls the optimization process, as the model receives clear direction despite the presence of preference data.
In contrast, our majority-vote signal s = sign sn) in Eq. 7 resolves this disagreement in the preference space. It distills the conflicting rewards into a single, high-confidence directional indicator, ensuring that VoLags provides a non-vanishing update toward the majority- prreitt--peeeredd manifold. This prevents the destructive interference of gradients and leads the more efficient convergence observed in our experiments.
Updating the based reference model: Another important aspect of our approach is that since we are already catering to multiple diverse rewards through the aggregation strategy, it makes sense to allow our model to move a little away from the reference model base, and hence we also replace Pref with the recent pa, during training and it provides remarkable performance, as shown in the experimental section next.
Theoretical Rationale:
Before performing the gradient analysis (Eqs. 10 11), we highlight the underlying motivation for our approach. This analysis shows that naive weightiig can lead to conflicting gradients, whereas majority voting ensures consistent update direction across metries.
The Rationale for Binary Discretization: While converting real-valued rewards into binary votes (sk E ()) results in a loss of magnitude information, it provides a critical advantage in multi-metric alignment: directional consensus. In practice, reward models like HPS, Aesthetic Score, and CLIP operate on fundamentally different distributions and scales. Traditional scalar aggregation (Eq. 5 is highly sensitive to these heterogencous ranges, often requiring exhaustive hyperparameter tuning of weights wy to prevent a single high-variance metric from dominating the gradient (the Reward Rescaling issue, C1). By discretizing the
9

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm 1 Majority Voting Aggregation for BALANCEDDPO
Require:
Initial model parameters 0g
Datasel D = ((e,)). +8))
Reward functions {ra)
Learning rate temperature B
Reference model update interval T
Ensure: Optimized parameters 8*
1:
Initialize reference model Pref +- pdo for training step { = 1 to Nateps do
3: Sample batch {(c, xg. x0)} ~ D
4: for each (c, x0. =g) in batch do
5: Compute attribute preforences:
for = 1. K do
1 if (r(= cx(g
8 - otherwise
8:
end for Aggregate preferences: s st sign === S/b
10:
end for Compute loss gradient: VoLags t- -E [Bso (-31g Blo) Volol Update parameters: 0411 +- O2 - 1
11:
12:
13:
if t = mod 7 then
14:
Update reference model: Pref < Po. end if
15:
16: end for
preference into vole, we achieve scale-invariance. This ensures that the model optimizes for the preference direction agreed upon by the majority, making the training process robust to outliers and preventing "reward hacking" where a model might exploit the numorical peaks of a single noisy metric at the expense of others.
Flexibility of the Voting Schema: While we employ a simple majority vote our experiments, the BALANCEDDPO framework inherently flexible. In scenarios where specific reward models are deemed more reliable critical than others, our method naturally supports weighted voting. By redefining the consensus signal as = sign(E W&ss), where represents the confidence or priority of the k-th metric, practitioners can incorporate domain-specific priors into the alignment process without reverting to the instabilities of scalar-sum optimization.
3.4 BalancedDPO Aggregation
The proposed method implements the majority voting strategy through three key phasos, as detailed in Algorithm D
I. Preference Collection (Lines 3-8): For each image pair (a x3) and prompt c, we query all K attribute- specific reward models obtain binary preferences 8І E (-1,1). 1}. This converts continuous reward values into directional indicators, eliminating scale sensitivity.
2. Majority Aggregation (Line 9): The function sign sx) implements Condorcet's Majority Principle by selecting the option favored in pairwise comparisons and satisfies Independence of Irrelevant Alternatives (ITA) by ensuring decisions remain unaffected by the presence of other options, making it robust. to outliers.
3. Gradient Update (Lines 11-13): The unified preference signal drives a single gradient update through the BALANCEDDPO loss, avoiding conflicting updates from multiple rewards. The reference model refresh (Line 14) follows curriculum learning principles, gradually tightening the KL constraint the model improves.
10

---

Published in Transactions on Machine Learning Research (04/2026)
Notably, the algorithm maintains (()) complexity for preference collection but only O(1) complexity for gradient updates, making it as efficient single-attribute DPO training. The periodic reference model update (Line 14) prevents premature convergence while maintaining training stability.
This consensus- based formulation provides a principled theoretical loundation for resolving the primary challenges of multi-metric alignment. By distilling multiple objectives into single, unified preference signal 8, BALANCEDDPO ensures that the diffusion model converges toward image distributions most consistently favored across diverse panel of "voters" (reward models). Furthermore, the relative preference between image pairs remains stable and decoupled from the presence of other samples in the batch. This theoretical framework directly addresses the Conflicting Gradients (C2) issue; rather than navigating a landscape of divergent updates from hetcrogencous metrics, the model follows a singular, high-confidence gradient direction rooted in social choice consensus.
4 Experiments and Analysis
4.1 Experiment Setup
Datasets We use the Pick-a-Pic dataset (Kirstain et al. 2023) for training, which provides pairwise preference images for each caption. Preference labels derived from ensemble of score models (PickScore Kirstain et al. 2023 HPS (Wu et al. 2023 CLIP (Ramesh et al. 2022). and Aesthetic (Schuluaann)
2022
creating a fultiefaceted image quality evaluation. For evaluation, we use Pick-a-Pic, PartiPrompt (Yu et al.
2022,, and HPD (Wu et al. 2023 Safe prompts are extracted using the Huggingface
LL 2023 model.
Detailed datasel information provided in Appendix A
Model We validate BALANCEDDPO using Stable Diffusion v1.5 (SD1.5) (Rombach et al. 2022 and compare it with the publicly available SD1.5 weights from the Hugging Face model hub (Rombach et al.)
Our model is trained for 2000 steps with updates to Pret every 100 steps, using the AdamW optimizer (Loshehilov] 2017) on a single NVIDIA A100 GPU. The batch size is effectively 128, and the learning rate is set to 1 x 10-8 with warmup, following the DiffusionDPO (Wallace et al. 2024 setup with adjustments for efficiency.
Hyperparameters
Evaluation We evaluate BALANCEDDPO against. SD1.5 and DiffusionDPO by generating five images per caption using random seeds. Images are assessed using four metrics: Human Preference Score (HPS), CLIP score, PickScore, and Aesthetic score. The highest score for each metric and prompt is sclected to mitigate variations. Win rates are computed for model comparisons (BALANCEDDPO vs. SD1.5, BALANCEDDPO vs. DiffusionDPO, DiffusionDPO vs. SD1.5), representing the proportion of times one model's best score surpasses the other's. Detailed evaluation procedures are available in the supplemental material (see Appendix D
4.2
Results
Comparison of win rates (%) for DiffusionDPO vs SD1.5, BalancedDPO vs SD1.5, and BalancedDPO DiffusionDPO across three datasets (Pick-a-Pic, PartiPrompt, HPD) and four evaluation metrics (HPS, CLIP score, PickScore, Aesthetic score). BALANCEDDPO outperforms SD1.5 and DiffusionDPO in most cases, with significant improvements in CLIP and Aesthetic scores on Pick-a-Pic and PartiPrompt. In HPD, while DiffusionDPO leads in HPS, BALANCEDDPO excels in CLIP, PickScore, and hesthetic scores.
Table 1:
Model
Picksa-Pie
PartiPronpt
HPJ
DFF
CLIP
PickScore
Acsthotte
IIPS
CLIP
PickScore
Aesthotic
HPS
CLIP
PickSoore
Aestactie
DI S01.5
70.71
67.24
56.23
82.99
67.76
86:24
30175
30.40
74.00
50.67
47.33
56.00
PALANCHSDPO SDL5
85.19
78.45
70.06
65.19
72.67
72.67
BALANCRODPO va Di
73.06
73.40
65.99
69.70
61.20
62,80
50.95
02.62
57.53
69.33
64.00
60.00
11

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Comparison of win rates (%) for DiffusionDPO vs SDXL, BalancedDPO vs SDXL, and BalancedDPO vs DiffusionDPO across three datasets (Pick-a-Pic, PartiPrompt, HPD) and four evaluation metrics (HPS, CLIP score, PickScore, Aesthetic score). BALANCEDDPO outperforms SDXL and DPO in most cases, with significant improvements in CLIP and Aesthetic scores on Pick-a-Phe and PartiPrompt. In HPD, while DPO leads in HPS, BALANCEDDPO excels in CLIP, PickScore, and Aesthetic
scores.
Model
Piik
PartiPrompt
HPD
HIS
CLI
PictScure
Letac
HP'
CLI
PickScore
Anslaeiic
HTS
CL.I
PickScore
Aestmelic
DO SDXL
78.00
59.00
45.00
21.00
76.00
58.00
52.00
26,00
78.00
55.00
38.00
11.00
BALANCEDDPO SDXL
98.00
88.00
62.00
44.00
96.00
91.00
76.00
66.00
97.00
78.00
50.00
42.00
BALANCRDDOO va DP
82.00
77.00
64.00
81.00
88.00
86.00
80.00
93.00
80.00
74.00
75.00
83.00
Table 3: Ablation study on the Pick-a-Pic dataset comparing BALANCEDDPO with models trained on individual metrics (HPS, CLIP, PickScore, Aesthetic) and Vanilla Aggregation. The table shows win rates against SD1.5 and DiffusionDPO across four metrics. Models trained on single metric (e.g., HPS) excel in that metric but perform poorly in others. BALANCEDDPO, using a multimetric consensus approach, consistently outperforms all ablated versions, highlighting the advantage of optimizing across multiple metrics
for superior overall performance.
Label Preference
Win againet SDL5
Win against DiftusiouDIO
HFS
CLIP
PickScore
Aesthetic
HPS
CLIP
PickScore
Aesthetic
89.90
49.49
87.54
12:46
CLIP
11:45
57.91
58.92
63.30
5.05
51.52
55.56
67.00
PirkScore
80.47
58.25
43.77
47.81
73.74
52.19
44.11
43.77
Acshhtic
52.19
26.94
57.58
55.22
56.23
Ours (Random Score Function)
83.83
56.22
49.15
38.04
78.45
45.11
45.79
35.01
Ours (Vanilla, /Nave Aggrogation)
60.70
31.65
53.54
53.54
54.21
26.60
53.20
52.
Ours (Normalized Aggregation)
84.28
42.54
60.20
62.23
50.87
60.49
61.38
63.47
Ours (BALANCEDDPO))
85
78.45
64.31
73.06
65.99
69.70
Comparison with baseline Table I compares the win rates of BALANCEDDPO (Ours) and DiffusionDPO across three datasets: Pick-a-Pic, PartiPrompt, and HPD. In the Pick-a-Pic dataset, BALANCEDDPO significantly outperforms DiffusionDPO with win rate of 85.19% in HPS (compared to 70.71%) and 78.45% in CLIP (compared to 57. 27...%) Similarly, in the PartiPrompt dataset, BAL achieves higher scores across all metries, particularly in HPS (70.6%% vs. 67.76%) and Aesthetic (62. 18% vs. 50.49%). In the HPD dataset, while DiffusionDPO has a slight edge HPS (74.00% vs. 72.67%), BA outperforms in CLIP (72.67% vs. 50.67%), PickScore (60.67% vs. 47.33%), and Acsthetic (63.33% vs. 56.00%). Overall, BALANCEDDPO consistently surpasses DiffusionDPO across most metrics and datasets, demonstrating superior alignment with human preferences and improved visual quality.
Table 2 further presents pairwise relative preference scores comparing DiffusionDPO vs. SDXL, BalancedDPO vs. SDXT., and BalancedDPO vs. DiffusionDPO across three datasets (Pick-a-Pic, PartiPrompt, HPD) and four evaluation metrics (HPS, CLIP, PickScore, and Aesthetic). BalancedDPO consistently achieves higher preference relative to both SDXL and DiffusionDPO across nearly all metrics and datascts. Specifically,
Table 4: Ablation study on the Pick-a-pic dataset comparing BALANCEDDPO with its ablated versions and DifiDPO across four metrics: Human Preference Score (HPS), CLIP score, PickScore, and Aesthetic score. Results show that removing either the multimetric scoring system reference distribution update significantly lowers performance, especially in Aesthetic and CLIP scores. Both ablated versions still outperform DiffDPO, with the full BALANCEDDPO model achieving the highest scores across all metrics, highlighting the importance
of both components.
Method
Pref update
Multi-Metric
HPS
CLIP
PickScore
Aesthetic
DPO
X
x
70.71
57.24
56.23
52.86
x
62.96
60.94
47.14
53.87
BAL
x
82.49
62.29
46.13
43.43
85.19
78.45
64.31
69.36
12

---

Published in Transactions on Machine Learning Research (04/2026)
in Pick-a-Pic, BalancedDPO is strongly preferred over SDXL (98% HPS, 88% CLIP, 62% PickScore, 44%
Aesthetic) and also clearly preferred relative to DiffusionDPO (82% HPS, 77% CLIP, 64% PickScore, 81% Aesthetic). On PartiPrompt, the relative preference remains high, favoring BalancedDPO significantly over SDXL (96% HPS, 94% CLIP, 76% PickScore, 66% Aesthotic) and DiffusionDPO (88% HPS, 86% CLIP, 80% PickScore, 93% Aesthetic). Similarly, for HPD, BalancedDPO demonstrates a strong relative preference over SDXL (97% HPS, 78% CLIP, 50% PickScore, 42% Aesthetic) and over DiffusionDPO (80% HPS, 74% CLIP, 75% PickScore, 83% Aesthetic). DiffusionDPO itself is consistently proferred relative to SDXL, though generally by smaller margins compared to BalancedDPO.
To further assess backbone generalization, we extend our evaluation to the Stable Diffusion 2.1 model (see supplementary Scction C.I.. As summarized in Table BalancedDPO again cxhibits substantial gains over both SD 2.1 and DiflusionDPO across all metrics on the Pick-a-Pic dataset, achieving 62.33 HPS, 65.33 in CLIP, 69.33 in PickScore, and 65.33 in Aesthetic, Compared to DiffusionDPO, which records 57.33, 47.67, 50.67, and 45.67 respectively, these results confirm that the proposed multimetric consensus and dynamic reference updaling remain elfective even on more recent diffusion backbones.
Pick-a-Pic dataset analysis Figure 4 compares the outputs of SD1.5, DifiusionDPO, and BALANCEDDPO
(Ours) across various prompts, highlighting the qualitative differences between the models. In the first row "Beautiful goddess, detailed face...") "), BALANCEDDPO generates a more realistic and lifelike face, while SD1.5 and DiffusionDPO produce stonc-like, statuc-like faces. Similarly, in the second row ( ("Mechanical bee flying...") "), BALANCEDDPO accurately captures the action of a bee in flight, unlike SD v1.5 and DiffusionDPO, where the bees remain static. In the third row ("Pirate guinea pig"), only BALANCEDDPO successfully includes the pirate cap as specified in the prompt, while the other models miss this key detail. For the fourth row ( ("Ruby oval ring...) '), BBALACCEDPP produces clearer and more complete rellection of the ring compared to SD1.5 and DiffusionDPO. Lastly, in the fifth row ( ("A 45-gear-old Africon Americon woman...") ), BALANCEDDPO preserves facial structure and details better than the other models, which produce less realistic faces. Overall, BALANCEDDPO demonstrates superior alignment with prompts and visual quality across all categories, often outperforming SD v1.5 and DiffusionDPO in both aesthetic appeal and prompt adherence.
PartiPrompt dataset analysis: Figure 5 compares the outputs of SD1.5, DiffusionDPO,, and BALANCED- DPO (Ours) across five prompts from the PartiPrompt dataset, which is out-of-distribution (OOD) for these models. In the first row, with the prompt "A helicopter hovering over Times Square", ", only BALANCEDDPO successfully generates a helicopter, while both SD1.5 and DiffusionDPO fail to include it. In the second row, for the prompt "A map of the United States with a pin on San Prancisco', ; all models generate a recognizable map, but none correctly place the pin. However, BALANCEDDPO produces a more detailed and visually appealing map compared to the others. In the third row, with the prompt "A punk rock squirrel in a studded leather jucket shouting inlo microphone while standing a bily pad", BALANCEDDPO accurately includes the microphone as specified in the prompt, while SD1.5 and DiffusionDPO generate irrelevant objects like sushi or random items. In the fourth row, for the prompt "Anubis wearing sunglasses and sitting astride a hog motorcycle', " BALANCEDDPO produces a more realistic Anubis with aviator sunglasses, while SD1.5 and DiffusionDPO generate less detailed, cartoonish versions. Finally, in the fifth row, for the prompt "Graffiti of funny dog on street wall", B BALANCEDDPO generates a more lifelike dog in graffiti style compared to the abstract representations from SD1.5 and DiffuusioNDP.. Overall, BALANCEDDPO consistently outperforms SD1.5 and DiffusionDPO in both prompt alignment and visual quality, showcasing superior acsthetic appeal and adherence to textual descriptions across all prompts.
HPD dataset analysis: Figure G illustrates the qualitative performance of SD1.5, DiffusionDPO, and BALANCEdDPO (Ours) across five prompts from the HPD (Wu et al., 2023) dataset. The results highlight the limitations of SD1.5 and DiffusionDPO in generating images that are both semantically aligned with the prompts and aesthetically appealing. For example, in the first row ("A blue airplane in a blue, cloudless sky'), DAL generates a more realistic airplane with accurate proportions and lighting compared to the less convincing outputs from SD v1.5 and DiffusionDPO.
In the second row ("A gummy chameleon hanging on a tree branch'), BALANCEDDPO captures both the gummy texture and natural lighting effectively, while the outputs from other models fail to balance realism
13

---

Published in Transactions on Machine Learning Research (04/2026)
hybrid creature concept pointing of zebra stripedd unicorn with bunny ears and colorful mane.
A portrait of two women with purple hair flying in different directions against dark background.
steampunk pocketwatch owl is trapped inside glass jar buried in sand. surrounded by an hourglass and swirling mist.
Prompt
airplane cloudless sky
gummy chameleon hanging on tree branch.
SD V1.
P:0.88 27 (c0.33 JA4228
F:0.18 H:0227 C.038 A44.09
F:0.18 H:0 28 C:0.39 JA4422
P:0.20 1H:0 29 C:0.29 (A4.48
PG.18100.27 C.0.20 IA4.47
DiffusionDPO
P.0.88 H..23] H C.0.3 IA4.31
P:0.9 H:0.8 0:041| A34
P.O18 H:0.9 C.0351AAA.T
F:0 H:0.30 CC00.39 AA4.42
P.C.99 P.C.9|HO.20 H0 C:0.9 IA4.49
BalancedDPO
(Ours)
P:0.99 C:0.34 AA4455
P:0.19 H-0.29| C903/ A:446
P- H:0.9 C:0.8 A-631
P-0.00 (60.30 C-9.42 A:644
P.019) H:0.29 C:0.34 A:4.53
Figure Comparison of images generated by SD v1.5, DiffusionDPO, and BALANCEDDPO (Ours) on prompts from the HPD dataset. Each row corresponds the generated outputs for each method. BALANCEDDPO consistently produces more realistic, detailed, and aesthetically pleasing images while aligning better with the given prompts compared to SD v1.5 and DiflusionDPO. The scores below each image represent PickScore (P), Human Preference Score (H), CLIP score (C), and Aesthetic score (A).
specific prompt, and the columns display
and prompt adherence. Similarly, in the third (*A hybrid creature concept painting of zebra-striped unicorn with bunny ears and colorful mane'), BALANCEDDPO produces vibrant and visually appealing image, whereas other models generate less pleasing outputs.
The fourth row (*A portrait of two women with purple hair flying in different directions against dark background') demonstrates BALANCEDDPO's ability to preserve facial details while adhering to prompt specifics, outperforming SD1.5 and DiffussinDPO., which produces less realistic faces with poor text alignment (background is not dark). Finally, in the fifth ("A steampunk pocket watch owl is trapped inside glass jar buried in sand, surrounded by an hourglass and swirling mist"), BALANCEDDPO generates intricate and visually compelling image that captures both steampunk aesthetics and prompt details, whereas other models fail to achieve this level of detail.
Overall, BALANCEDDPO consistently outperforms SDL.5 and DiffusionDPO across all prompts by producing images that are better aligned with textual descriptions while maintaining superior visual quality across all evaluated metrics.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Effectiveness of different score methods:
To rigorously evaluate the necessity of majority voting, we compare BALANCEDDPO against three alternative
aggregation strategies in Table 3
Random Score Function: At each training iteration, one of the K reward functions is randomly
sampled to determine the preference 8, representing a stochastic approach to multi-metric alignment.
Vanilla Aggregation:
The preference is determined by the sign of the unweighted sum of raw scalar rewards: s = sign( kk=1 rk). This baseline is highly susceptible to the "Reward Rescaling" (C1) issue.
Normalized Aggregation: To mitigate scale dominance, rewards are Z-score normalized based on batch statistics (z+ = -x/// before being summed to determine the preference signal.
Table 3 highlights the effectiveness of training models using individual score functions (HPS, CLIP, PickScore, Aesthetic) versus combining all score functions as done in BALANCEDDPO (Ours). Models trained on HPS score show strong performance in their respective metric but underperform in others. HPS-trained model achieves high HPS scores (89. 999.00% against SD1.5 and 87.54% against DiffusionDPO) but performs poorly in Aesthetic (14.4%% against SD..5). Similarly, the CLIP-trained model excels in CLIP score (57.91% against SD1.5) but struggles with HPS. In contrast, BALANCEDDPO, which optimizes across all metrics simultaneously, consistently outperforms across most metrics, achieving the highest scores in CLIP (78.45%), PickScore (64.31%), and Aesthetic (69.36%) against SD1.5, and similarly outperforming DiffusionDPO across all metrics.
Even when utilizing all four scoring methods, Vanilla Aggregation and Normalized Aggregation do not yield optimal results. The Random Score Function approach, which involves selecting a random score function at each iteration, shows improved performance over individual metrics but still falls short of BALANCEDD performance. Notably, Vanilla Aggregation struggles to balance the different metrics effectively, with lower win rates across the board compared to BALANCEDDPO. Normalized Aggregation, while showing some improvement over Vanilla Aggregation, still fails to match the consistent high performance of BALANCEDDPO across all metrics.
This demonstrates that combining multiple score functions in a meaningful way (BALANCEDDPO) leads to better overall performance and balanced results across different evaluation criteria.
Effectiveness of Balanced Multimetric Alignment and Reference Model update: Table ! highlights the importance of Multimetric Alignment and frequent. updates to the reference model (Pret) in achieving superior performance. When removing the multimetric scoring system, the model still performs well in HPS (82.49%) and CLIP (62.29%) as it is able to chase and overfit the human preference score but underperforms in PickScore (46.13%) 13%) and Aesthetic (43.43%). Similarly, without frequent updates to Pref, the model achieves a higher Aesthetic score (53.87%) but suffers in HPS (62.96%) and PickScore (47.14%). In contrast, BALANCEDDPO (with both multimetric alignment and frequent Pref updates) consistently outperforms other variants across all metrics, achieving the highest scores in HPS (85.19%), CLIP (78.45%), PickScore (64.31%), and Aesthetic (69.36%). This demonstrates that optimizing multiple metrics simultancously and frequently updating the reference model are crucial for achieving balanced and superior performance across different evaluation criteria. These results underscore the effectiveness of our BALANCEDDPO approach in simultancously optimizing multiple image quality metrics. Unlike Diff which shows uneven improvements across different metrics, BALANCEDDPO consistently outperforms across all evaluated dimensions. This balanced improvement suggests that our method successfully addresses the limitations of previous approaches, achieving a more holistic enhancement of image generation quality. The consistent. performance across both the in-distribution Pick-a-Pic dataset and the out-of-distribution PartiPrompt dataset further demonstrates the robustness and generalizability of our BALANCEDDPO method. This strong OOD performance is particularly significant, as it indicates that BALANCEDDPO can effectively handle a wide range of prompt styles and complexities beyond its training distribution.
The synergistic relationship between multimetric aligmment and the dynamic reference model (pret) update is critical. In single-metric DPO, frequent Prof updates can lead to instability "reward backing" as the
15

---

Published in Transactions on Machine Learning Research (04/2026)
BalancedDPO (Ours)
SD v1.5
DiffusionDPO
Added by authors for publication
Figure 7: Comparison image generations by SD1.5, DiffusionDPO, and BALANCEDDPO (Ours). In the first row (Prompt: " "Person riding shark"), BALANCEDDPO accurately includes the person, while SD v1.5 and DiffusionDPO do not. In the second row (Prompt: "SFW Trustworthy ...'), BALANCEDDPO generates high-quality, safe-for-wont image, while SD1.5 and DiffusionDPO produce NSFW content. All images are generated with the same seed.
model overfits to narrow signal. However, in our multi-metric setting, the dynamic update acts as moving anchor. This prevents the model from being pulled excessively toward one specific metric (e.g., maximizing Aesthetic scores while sacrificing Prompt Adherence). Instead, allows the model to incrementally discover a "consensus manifold" where the requirements of all reward models are satisfied. This mechanism ensures that the KL-divergence constraint remains informative and anchored to the most recent balanced state of the model, leading to the superior win rates observed in Table 4
As shown in Figure individual reward models exhibit distinct "corner-case" biases. Aesthetics-only training prioritizes photographic flair (e.g., saturation, bokeh) but often suffers from semantic drift, sacrificing prompt details for visual appeal. Conversely, CLIP TLIP/PickScore-on0 models ensure semantic fidelity but lack artistic polish, often appearing compositionally flat. While HPS v2.1 provides stronger baseline, it remains susceptible to stylistic biases in its training data. By requiring a majority consensus, BALANCEEDDO acts consensus filter. It retains the visual sophistication of aesthetic metrics while using CLIP and HPS signals to ground the imagery in the prompt. This evidence suggests BA successfully navigates the Pareto front of these conflicting objectives, distilling a holistic generation from heterogeneous reward signals.
Qualitative Analysis of Metric Bias:
Adherence to prompts during image generation: In Figure 7 the comparison of image generations by SD1.5, DiffusionDPO, and BALANCEDDPO (Ours) demonstrates a clear advantage of the proposed multi-objective optimization strategy in adhering to user prompts.
16

---

Published in Transactions on Machine Learning Research (04/2026)
In the first row, where the prompt is "Person riding shark" BALANCEDDPO successfully generates image that includes both the person and the shark, directly aligning with the prompt's requirements. In contrast, both SD1.5 and DiffusionDPO fail to include the person, suggesting their limitations in prompt adherence. These models either omit critical clements or interpret the prompt too loosely, leading to incomplete or inaccurate representations of the requested scene. Similarly, "SFW Trustworthy ...;, further highlights the strengths of BALANCEDDPPO. While SD1.5 and DiffusionDPO generate NSFW content, BALANCEDDPO reliably produces safo-for-work image that meets both the prompt's requirements and the expectation for content safely. Overall, these results emphasize the advantage of BALANCEDDPO 's multi-objective optimization strategy, which not only ensures higher fidelity to the specified prompt but also improves prompt adherence.
5 Conclusions
In this work, we introduced BALANCEDDPO, method for aligning text-to-image models with human preferences by optimizing multiple metrics simultaneously. Our experiments show that BALANCEDDPO outperforms existing approaches like DiffusionDPO and SD1.5 across key metries, including Human Preference Score (HPS), CLIP, PickScore, and Aesthetic quality. While single-metric training leads to imbalances, BALANCEDDPO's multimetric approach ensures balanced, high-quality image generation. The frequent updates to the reference model (Pret) further enhance its ability to align with human preferences.
Furthermore, while our evaluation focuses text-to-image synthesis, the underlying consensus-based based mechanism of BALANCEDDPO is inherently modality-agnostic. The stralegy of distilling multiple conflicting reward signals into a unified preference signal could be extended to other generative tasks, such as Large Language Model (LLM) alignment for helpfulness and safety, or video generation where temporal consistency and aesthetic quality must be balanced. This suggests that majority-vote preference aggregation is a robust, general-purpose framework for multi-objective alignment.
Overall, BALANCEDDPO offers a scalable and effective solution for multi-metric optimization, demonstrating strong performance across both in-distribution and out-of-distribution datasets, with broad potential for real-world applications.
References
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.130 13301, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-33 345, 1952.
Jingyi Chen, Ju-Seung Byun, Micha Elsner, and Andrew Perrault. Reinforcement learning for fine-tuning text-to-specch diffusion models. arXiv preprint arXiv:2405.14633 14632, 2024.
Xiwei Cheng, Xiangxin Zhou, Yuwei Yang, Yu Bao, and Quanquan Gu. Decomposod direct preference optimization for structure-based drug design. CoRR, abs/2407.13981 2024. URL httpp:://diioorg/:
48550/arXiv. 2407. 13981
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjee Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. arXiv preprint arXio:2310. 02054, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-fors international conference on machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to poat-to-ime to-image diffuuson models. Advances in Neural Information Processing Systems, 36, 2024.
17

---

Published in Transactions on Machine Learning Research (04/2026)
Yi Gu, Zhendong Wang. Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning diffusion models through relative preference optimization. arXiv preprinl or Xix:24406. 06382, 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advonces in Neural Information Processing Systems, 36:36652 36663, 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint ar arXit:2302.12192 2023.
Michelle Jie Li. Inappropriate text classifier, 2023. URL https///huggnngfeeeeo///atllllll
Shaproprrite_texxe terpriase_eett_clsssiiieer
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint arXiv:2406.0433 2024.
Loshchilov. Decoupled weight decay regularization. arXic preprint arXiv:1711.05101 2017.
Alex Nichol, Prafulla. Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXix:2112.1074 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint ar arXiv:230701152. 2023.
Mahendra Prasad. Social choice and the value alignment problem. Artificial intelligence safety and security, pp. 291-314, 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Dircct preference optimization: Your language model is secretly a reward model. In Thirty-seuenth Conference on Neural Information Processing Systems, 2023. URL http:://ppperrreeeeeeeeetoorrree
HPuSIXJaa9
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint XxXi:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Huggingface stable diffusion v1.5 model.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Essor, and Björn Ommer. High-resolttion image synthesis with latent diffuusio models. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recogmition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-
36494, 2022.
Christoph Schuhmann. Laion-acsthetics, 2022. URL http:e//aaaooaa////////aaooeeeer cessed: 2023-11-10.
Ac-
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar 1 2017.
18

---

Published in Transactions on Machine Learning Research (04/2026)
Bram Wallace, Meihua Dang, Rafacl Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model aligument using direct preference optimization, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-828888888 8238, 2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiw:2208.0611 061998 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arxiv preprint Xiv:22306. 09341, 2023.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arXiv:2311.198 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint ar Xrv:22006. 10789, 2(3):5, 2022.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part Ppo. arXiv preprint ar aan:2307.4966 04964, 2023a.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, et al. Delve into ppo: Implementation matters for stable rlhf. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023b.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one- cweference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 10586 10613, 2024.
19

---

Published in Transactions on Machine Learning Research (04/2026)
This appendix provides extended insights and results to support the claims made in the main paper.
In Section A we describe the datasets used in our experiments, including PartiPrompts, Pick-a-Pic, and HPD, and showcase images generated from randomly sampled prompts to highlight the diversity and quality of the dataset.
Section provides more information on Diffusion Model Formulation.
Section expands on key findings:
Demonstrates the limitations of Vanilla Aggregation for combining metrics, emphasizing the necessity of our proposed BALANCEDDPO method.
Presents additional qualitative results on the HPD dataset, showing BALANCEDDPO's abil- ity to generate semantically aligned and visually appealing images compared to SD1.5 and DiflusionDPO.
Analyzes the effect of seed variation, highlighting BALANCEDDPO's robustness across random seeds while maintaining semantic alignment and aesthetic quality.
In Section D we detail the evaluation procedures used to assess model performance across metrics, ensuring fair and comprchensive comparisons between methods.
A
Datasets
Figure 8: Pick-a-pic Kirstain et al. 2023 dataset.
For validation, employ three distinct datasets to provide comprehensive evaluation:
20

---

Published in Transactions on Machine Learning Research (04/2026)
Figure 9: PartiPrompt dataset
Yu et al. 2022)
The Pick-a-Pic dataset is extensive, publicly accessible collection of over 500,000 examples, encompassing more than 35,000 unique prompts, each paired with two Al-generated images and corresponding user preference labels. This dataset was compiled through the Pick-a-Pic web application, which enables users to generate images from text prompts and indicate their preferences between image pairs. The primary objective of Pick-a-Pic is to facilitate research aligning text-to-image generation models with human preferences, thereby enhancing the quality and relevance of generated images. In addition to its in training, utilize the validation set for the evaluation, which contains 500 unique prompts.
Pick-a-Pic (Kirstain et al. 2023
PartiPrompts Yu et al. 2022): PartiPrompts is comprehensive dataset comprising over 1,600 English prompts, designed to evaluate and enhance the capabilities of text-to-inage generation models. Fach prompt is categorized across 12 distinct themes such as abstract concepts, animals, vehicles, and world knowledge and is further classified into one challenge aspects, including basic, quantity, words and symbols, linguistics, and complex scenarios.
Human Preference Dataset v2 (HPD v2) Wu al. 2023): This dataset serves as the foundation for creating the Human Preference Score v2 (HPS v2), and contains 798,090 human preference choices 760 pairs of images. It is specifically curated to minimize potential biases present in previous datasets and covers wide range of image sources and styles. It has 400 unique prompts.
We show examples of generated images from prompts sourced from Pick-a-Pic (Figure 8 PartiPrompts (Figure Ø and HPD v2 (Figure 10 below.
21

---

Published in Transactions on Machine Learning Research (04/2026)
A RPE BOES
Figure 10: HPD Dataset
Wu et al. 2023
B Diffusion Model Formulation
A generative model class that learns denoise e ~ [/, I), into structured data. The core component is a dcnoiser D, predicting E[xg|æ2, given noisy image xy time t. The noisy image 24 is x, agz9 dcC, where and d, control noice addition.
Given g(zo) and noise scheduling functions a and 0,, the generative model (=o) follows a discrete-time reverse process po (=(oT) -II'_1 Pa(#111117)) (x)), with transition dynamics as,
pe pe (z.) = (ozz)
(12)
with (z())) being the predicted mean and - representing the variance at cach timestep. This reverse process ileratively refines noisy samples into clean data. The training of diffusion models is performed by minimizing the Evidence Lower Bound (ELBO), which results in objective
Lpm Exo.e.t. [w(A:)]le - c0 1)
(13)
where, (O, represents the noise added to the data, / sampled uniformly from [), ep denotes the noise predicted by the model being trained, and x, is sampled from the distribution q(z2/20) N(xx; agto, ofT).
The term A = of represents the signal-to-noise ratio at time step and w(Xc) is pre-specified weight function that adjusts the importance of different timesteps during training.
C Extended Results
This section provides additional experiments to support our claims in the main paper. First, we conduct ablation study to demonstrate the limitations of the Vanilla Aggregation approach, highlighting why
22

---

Published in Transactions on Machine Learning Research (04/2026)
our proposed BALANCEDDPO method is essential for effectively optimizing multiple metrics simultancously. Second, we present qualitative and quantitative (see Table [ results on the HPD dataset (Wu et al. 2023 showcasing BALANCEDDPO's ability to generate images that are both semantically aligned with prompts and aesthetically superior compared to SD1.5 and DifrusionDPO. Finally, we analyze the effect of seed variation on image generation, demonstrating that PALANCEDDPO consistently produces high-quality and semantiially aligned outputs across diverse prompts, proving its robustness to random initialization.
C.1 Results on SD 2.1
To further validate the generalization ability of our approach, we evaluate BALANCEDDPO the Stable Diffusion 2.1 backbone using the Pick-a-Pie Pic dataset [Kirstain et al. 2023 As shown in Table 5 BALANCED- DPO consistently improves over both the base SD 2.1 model and DiffusioNDPO across all evaluation metrics, confirming that our multimetric consensus strategy and dynamic reference model updates remain effective on newer diffusion architectures.
Table 5: Comparison of BALANCEDDPO against SD 2.1 and DiffusionDPO on the Pick-a-Pic dataset across four evaluation metrics: HPS, CLIP, PickScore, and Aesthetic.
Comparison
HPS
CLIP
PickScore
Aesthetic
DiffusionDPO vs SD 2.1
57.33
47.67
50.67
45.67
BALANCEDDPO vs SD 2.1 BALANCEDDPO vs DiffusionDPO
62.33
65.33
69.33
65.33
53.67
65.67
69.67
70.00
C.2 Generalization to Transformer-based Architectures
To address concerns regarding the modernity of diffusion backbones, we evaluate BALANCEDDPO on SD3-Medium (Esser et al. 2024) (using a Multimodal Diffusion Transformer, MMDiT). Since BALANCED- DPO operates at the objective function level by aggregating rewards into unified consensus signal, it is fundamentally architecture-agnostiic.
As shown in Tables D. BALANCEDDPO provides consistent improvements over the base transformer models and the mingle-metric DiffusionDPO baseline across all evaluation criteria. Notably, while standard DPO on transformer architectures can sometimes lead to decline in Aesthetic scores while improving prompt adherence (CLIP), BALANCEDDPO maintains high win rate across both dimensions. These results demonstrate that the consensus- based alignment strategy generalizes effectively to the latest generation of transformer-based diffusion models.
Table 6: Win rates (%) on SD3-Medium (Transformer backbone) using 500 captions from PartiPrompts.
Comparison
HPS
CLIP
PickScore
Aesthetic
DiffusionDPO vs. Base SD3
55.42
52.11
49.87
44.34
BALANCEDDPO vs. Base SD3
62.21
63.80
61.58
58.12
BALANCEDDPO vs. DiflusionDPO
58.45
62.27
60.11
58.35
C.3 Limitations of Vanilla Aggregation
In addition to our proposed multimetric consensus strategy, we experimented with straightforward approach to combine scores, referred Aesthetic) are directly aggregated using simple weighted sum. As shown in Table 3 this naive approach fails to achieve competitive performance compared to BALANCEDDPO, While Vanilla Aggregation performs slightly better than models trained on individual metrics alone, it struggles to balance competing objectives effectively, resulting in suboptimal performance across multiple metrics.
Vanilla. Aggregation, where individual metric scores (HPS, CLIP, PickScore,
For instance, Vanilla Aggregation achieves win rate of 69.70% in HPS and 53.54% in both PickScore and hesthetic against SD1.5, which is significantly lower than the corresponding scores achieved by BALANCED-
23

---

Published in Transactions on Machine Learning Research (04/2026)
DPO (85.19%, 64.31%; and 69.36%, respectively). Similarly, against DiffusionDPO, Vanilla, Aggregation underperforms across all metrics, with CLIP score of only 26.60% compared to BALANCEDDPO's 73.40%. This performance gap highlights the inability of Vanilla Aggregation to handle the inherent scale differences and conflicts between metrics, leading to suboptimal alignment.
The limitations of Vanilla Aggregation can be further understood through the following optimization equation shown in Eq. 10
(0)
(14)
Σus k
ur log a (B 8k lo c, To: =g))
TE (e,,
where we represents the weights for each metric and 8& indicates the preference for the t attribute. Similarly, the gradient of the objective boils down to
V VoLda
(15)
=-Ep ED
Σ WrBSUo (-Blolc, =g)) polale,20.10
This formulation suffers from issues such as unknown optimal weights (wx), scale dominance by certain metrics, and conflicting gradients that hinder convergence. These challenges make Vanilla Aggregation unsuitable for effectively optimizing multiple metrics simultaneously.
In contrast, BALANCEDDPO's multimetric consensus approach dynamically aggregates preferences through majority voting, ensuring balanced optimization across all metrics without being dominated by any single score. This enables BALANCEDDPO to consistently outperform both Vanilla Aggregation and single-metric models, demonstrating the importance of our proposed method for achieving superior overall performance.
C.4 Effect of Seed Variation
Figure demonstrates the robustness of BALANCEDDPO across different prompts and sceds, showcasing its ability to generate high-quality and semantically aligned images consistently.
For prompts such as "the Taj Mahal at sunrise' (Parti dataset), BALANCEDDPO produces photorealistic images with varying lighting and perspectives, maintaining semantic alignment across all scods. Similarly, for "A hybrid creature concept painting of zebra-striped unicorn with bunny and colorful mane' (HPD dataset), BALANCEDDPO consistently captures the intricate details of the prompt, such the vibrant colors and unique features of the creature. Prompts like "a calm and peaceful office" (Pick-a-Pic PPc dataset) highlight BALANCEDDPO's ability to render realistic indoor scenes with diverse layouts while adhering to the described atmosphere.
This figure underscores BALANCEDDPO's capability handle diverse prompts from multiple datasets while maintaining consistency across seeds. The results demonstrate that BALANCEdDPO not only excels in semantic aligmment but also generates aesthetically pleasing images with significant variation across seeds, making it robust solution for text-to-image generation tasks.
C.5 Scoring Functions
Table 7 presents the average scores for three - DiffusionDPO, and our proposed BALANCEDDPO- across four key metrics: Human Preference Score (HPS), CLIP score, Pickscore, and Aesthetic score. The results demonstrate that BALANCEDDPO consistently outperforms both SD1.5 and DiffusionDPO across all evaluated metrics. Specifically, BALANCEDDPO achieves the highest average scores HPS (0.2745), CLIP (0.3771). Pickscore (0.1899), and Aesthetic (4.5034). While the improvements may appear incremental, they represent significant advancements in model performance, particularly given the challenging nature of
24

---

Published in Transactions on Machine Learning Research (04/2026)
Figure 11: Comparison of images generated using BALANCEDDPO across five different seeds for each prompt. Rows correspond to prompts from the PartiPrompts, Pick-a-Pic, and HPD datasets, while colunns represent outputs for different seeds. The figure highlights BALANCEDDPO's ability consistently generate visually appealing and semantically aligned images across diverse prompts and seeds.
25

---

Published in Transactions on Machine Learning Research (04/2026)
simultancously optimizing multiple metrics. DiffusionDPO shows slight improvements over SD1.5 in HPS and CLIP scores but performs marginally worse in Aesthetic score. These results highlight the effectiveness of PALANCEDDPO in achieving a balanced optimization across diverse image quality metrics, demonstrating its superior capability in generating images that align well with human preferences while maintaining high visual quality.
Table 7: Average scores for SD1.5, DiffusionDPO, and BALANCEDDPO (Ours) across four key metrics: Human Preference Score (HPS); CLIP score, Pickscore, and Aesthetic score. BALANCEDDPO consistently
outperforms the other models across all metrics. Model
HPS
CLIP
Pickscore
Aesthetic
SD1.5
0.2708
0.3686
0.1897
4.4860
DiffusionDPO
0.2730
0.3715
0.1898
4.4840
BALANCEDDPO (Ours)
0.2745
0.3771
0.1899
4.5034
Table presents statistical measures (mean, minimum, maximum, and standard deviation) for four scoring metrics applied to images generated by the SD1.5 model. These metrics include the Human Preference Score (HPS), CLIP score, Pickscore, and Aesthetic score. The data shows variation in scores across the different metrics, with Aesthctic scores having a notably different scale than the other three metrics.
Table 8: Statistical summary of image quality metrics for SD1 1D1.--generated images across four evaluation scores: Human Preference Score (HPS), CLIP, Pickscore, and Aesthetic.
Statistic
HPS
CLIP
Pickscore
Aesthetic
Mean
0.2708
0.3686
0.1897
4.4860
Min
0.2233
0.2383
0.1656
4.0969
Max
0.3114
0.5161
0.2195
4.7878
Std
0.0130
0.0461
0.0086
0.1049
Evaluation Procedure
We evaluate BALANCEDDPO in comparison to Stable Diffusion 1.5 (SD1.5) (Rombach et al. 2022 and DiffusionDPO (Wallace et al. 2024), addressing the limitation of these approaches in simultaneously improving scores across all metrics.
Image Generation: For each caption from our validation datasets, we generate five images using different random seeds for all models (BALANCEDDPO, SD1.5, and DiffusionDPO).
Metrics Used: We evaluate generated images using four key metrics: Human Preference Score (HPS), CLIP score, PickScore, and Aesthetics score following DiffusionDPO.
Best Score Selection: For each metric and prompt. we select the highest score from the five generated images for all the models. This method mitigates random variations in image generation, ensures each model demonstrates its best performance, and maintains consistency across all models by automating the selection process.
4. Win Rate Caleulation:
We computed the win rale for each model pair comparison (BALANCEDDPO us. SD1.5, PALANCEDDPO us. DiffusionDPO, and DiffusionDPO us. SD1.5). The win rate represents the proportion of times model's best score is preferred (i.e., higher) compared to the other model's 's best score for the same prompt and metric.
Ethics
While our model, BALANCEDDPO, demonstrates significant advancements in text-to-image generation, is crucial to acknowledge the ethical considerations associated with its use. Text-to-image models, including
26

---

Published in Transactions on Machine Learning Research (04/2026)
ours, inherently carry risks such potentially generating harmful, biased, inappropriate content. These risks may arise from biases present in the training data or unintended misuse of the model. We emphasize that it is the responsibility of users to employ this technology ethically and avoid generating content that is harmful, hateful, or otherwise inappropriate. Our model is designed for research and creative purposes, and we strongly discourage use for generating misleading, offensive, or exploitative content.
27


Reviews:
[{"review_id": "UQRjiVUYlq", "reviewer": "Reviewer_WLjH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes BalancedDPO, a new method to align diffusion models to multiple evaluation metrics. The authors leverage ideas from social choice theory where different metrics are balanced via majority vote consensus. This idea is very interesting and different from approaches that rely on scalar value optimization. Extensive quantitative and qualitative experiments demonstrate the effectiveness of the proposed approach", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claim is that the proposed approach outperforms simple aggregation methods and methods that only focus on one evaluation criteria. This claim is supported by quantitative experiments over many datasets. The authors also provide example output images for various prompts, which qualitatively show that the proposed method has better image generation qualities in general.", "audience_yn": "Yes", "audience_explanation": "This work should appeal to people working on generative models (including diffusion). The method proposed to balance multiple evaluation metrics might be useful beyond image generation.", "requested_changes": "I have a few concerns. They are not critical issues, but if addressed will strengthen the paper.\n1. It seems that by using majority vote, we must inevitably convert real-valued metrics to binary ones, which might suffer loss of information. It would be helpful if the authors could argue why this would not be a critical issue in the training process.\n2. Could the alignment method apply beyond image generation?", "broader_impact_concerns": "The ethical implications are adequately addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "69XOcNpl91", "reviewer": "Reviewer_Hhxp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A new text to image diffusion model called BalacedDPO is proposed. It improves adherence to multiple image generation criteria via simple mechanism that uses majority voting to derive new preference pairs from multiple metrics.\n\n(Motivation) Text to image generation is a problem that requires satisfying multiple metrics that focus on diverse criteria like alignment with the prompt or aesthetic appeal. Given 1 of M metrics, existing approaches tend to perform better at just one of these to the exclusion of others. One can consider combining M metrics into a single metric via some fusion function (eq. 5), but it is difficult in practice to find the right fusion approach because the different metrics can have imbalanced and conflicting effects on the loss surface.\n\n(Approach) The proposed approach BalancedDPO is a simple extension of the DiffusionDPO model that applied DPO to T2I diffusion models. Given an image pair, each of the M metrics votes for the image it prefers. These votes are then aggregated via majority vote into a single preference and this new preference is used with the standard DiffusionDPO objective to train the model. An additional change to the approach is made where the reference model in the DPO objective is dynamically updated to the most recent model every 100 training steps.\n\n(Experiments) Stable Diffusion 1.5, and SDXL are used as base models and in almost every case BalancedDPO outperforms both these base models and a DiffusionDPO version of them across 4 metrics (HPS, CLIP, PickScore, Aesthetic) and 3 datasets (Pick-a-Pic, PartiPrompt, HPD). Ablations show the approach outperforms both single metric optimized models and more naïve metric fusion approaches. Ablations also show that both the dynamic reference model updates and the majority vote approach are necessary for peak performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claim 1: A good solution to the T2I problem requires satisfying multiple diverse measures of quality and baseline approaches have trouble satisfying all of these diverse metrics at once.\n* This can be seen in experiments that compare e.g., DiffusionDPO and SDXL (Table 2). DiffusionDPO wins 78% of the time on the HPS metric while it only wins 21% of the time on the Aesthetic metric. While this is a more extreme example, this happens often enough in tables 1 and 2 to justify the claim. Additional justification comes from table 3 where models trained on just one metric clearly suffer in performance on the other metrics.\n\nClaim 2: The BalancedDPO approach, including both dynamic reference model updates and majority voting based preferences, improves image generation quality.\n* This is shown clearly by quantitative 4 metrics across 3 datasets in table 1 through 4. There are also lots of qualitative examples and they seem to usually align with the prompts more accurately than baselines.\n\nClaim 3: A multi-objective optimization approach would suffer from (C1) reward rescaling issues, (C2) conflicting gradients, and (C3) pipeline complexity.\n* These don't appear to be justified with individual evidence. Perhaps C1 and C2 are justified by the other Ours baselines in Table 3, but that does not appear to distinguish between the two challenges. However, the reasons make intuitive sense and are presented as motivation rather than scientific claims about how alternatives perform, so they aren't misleading and I'm willing to accept them.", "audience_yn": "Yes", "audience_explanation": "Engineers and researchers building T2I models and generally using DPO will be interested in this since it is both easy to implement and shows strong improvements.", "requested_changes": "None of these are critical to my recommendation. Addressing them would just strengthen the work.\n\n* It does not look like the Random Score Function, Vanilla/Naive Aggregation, and Normalized Aggregation ablations from Table 3 are described precisely anywhere. I can guess what they are doing, but it would be good to include a more precise description.\n* I appreciate the p_ref update ablations in Table 4, but I would still like to understand the impact of dynamic reference model updates better. In those ablations adding just p_ref update to DiffusionDPO has a wildly variable effect (sometimes hurts, sometimes helps). It would be useful if the paper included more intuition about how it interacts with the Multi-Metric aspect of the approach and some tips about how and when to apply it more generally.\n* A primary motivation of the paper is that different metrics capture different image generation criteria. I can see the difference between the 4 metrics chosen in the quantitative results, but the paper should also provide intuition about what the qualitative differences are between those 4 metrics if possible.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x7QOb9ngkc", "reviewer": "Reviewer_NdHQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes BalancedDPO to align samples generated by diffusion models with human preferences. Its core design focuses on making generated samples conform to a broader range of reward models. To achieve this goal, the paper introduces a voting mechanism based on social choice theory, which favors samples that demonstrate superiority across multiple reward models. Results demonstrate that this approach effectively enhances the generative quality of the diffusion model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The author conducte experiments under the SD1.5 model using various reward models (PickScore, HPS, etc.), performing both quantitative and qualitative analyses of the results, which align with the conclusions presented in this paper.", "audience_yn": "Yes", "audience_explanation": "This paper explores aligning models with multiple reward models, a research question of significant value that enables various generative models to produce samples aligned with human needs.", "requested_changes": "### Major \n+ Insufficient experiments: All experiments are built upon and evaluated with the Unet-SD, a model that is outdated and no longer representative of the current state of the field. Given the rapid shift toward transformer-based architectures like DiT, MMDiT, FLUX, and SD3, it is essential to validate the proposed method on these more advanced models. Without this, the experimental claims are difficult to generalize or take seriously.\n\n+ The proposed method is overly simplistic. I believe the author should further explore additional ways to validate winning samples. The current voting strategy appears too simplistic and unconvincing. I do not believe every reward carries equal weight.\n\n+ Equation 10 and Equation 11 do not appear to demonstrate a contradiction in the gradient. Could the author provide further evidence?\n### Minor\n+  Figure 1 is missing critical information. In Figure 1, it should be labeled which one is the winner.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)