ViPO: Visual Preference Optimization at Scale
Authors: Ming Li
Jie Wu
Justin Cui
Xiaojie Li
Rui Wang
Chen Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Image Generation
Video Generation
Visual Generation
DPO
Primary area: generative models
TL;DR: Scaling preference optimization in visual generation through our proposed large-scale datasets and algorithmic improvements

Abstract: While preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm for visual generation remains largely unexplored.
Current open-source preference datasets typically contain substantial conflicting preference patterns, where winners excel in some dimensions but underperform in others. Naively optimizing on such noisy datasets fails to learn meaningful preferences, fundamentally hindering effective scaling. To enhance the robustness of preference algorithms against noise, we propose Poly-DPO, which extends the DPO objective with an additional polynomial term that dynamically adjusts model confidence during training based on dataset characteristics, enabling effective learning across diverse data distributions from noisy to trivially simple patterns.
Beyond biased patterns, existing datasets suffer from low resolution, limited prompt diversity, and imbalanced distributions. To facilitate large-scale visual preference optimization by tackling key data bottlenecks, we construct ViPO, a massive-scale preference dataset with 1M image pairs (1024px) across five categories and 300K video pairs (720p+) across three categories. Leveraging state-of-the-art generative models and diverse prompts ensures consistent, reliable preference signals with balanced distributions.
Remarkably, when applying Poly-DPO to our high-quality dataset, the optimal configuration converges to standard DPO. This convergence validates both our dataset quality and Poly-DPO's adaptive nature: sophisticated optimization becomes unnecessary with sufficient data quality, yet remains valuable for imperfect datasets.
We comprehensively validate our approach across various visual generation models. On noisy datasets like Pick-a-Pic V2, Poly-DPO achieves 6.87 and 2.32 gains over Diffusion-DPO on GenEval for SD1.5 and SDXL, respectively. For our high-quality ViPO dataset, models achieve performance far exceeding those trained on existing open-source preference datasets. These results confirm that addressing both algorithmic adaptability and data quality is essential for scaling visual preference optimization. Code, models and open-source datasets will be released at: https://github.com/liming-ai/ViPO

Published as a conference paper at ICLR 2026
VIPO: VISUAL PREFERENCE OPTIMIZATION AT SCALE
Ming 1i Jie Wu², Justin Cui2.3, Xiaojie Li², Rui Wang", Chen Chen' University Central Florida, "ByteDance Seed, SULA
ABSTRACT
While preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm for visual generation remains largely unexplored. Cur- rent open-source preference datasets typically contain substantial conflicting preference patters,, where winners excel in some dimensions but underperforn in others. Naively optimizing on such noisy datasets fails to learn meaningful preferences, fundamentally hindering effective scaling. To enhance the robustness of preference algorithms against noise, we propose Poly-DPO, which extends the DPO objective with an additional poly- nomial term that dynamically adjusts model confidence during training based on dataset characteristics, enabling effective learning across diverse data distributions from noisy to trivially simple pattems. Beyond biased patterns, existing datasets suffer from low res- olution, limited prompt diversity, and imbalanced distributions. To facilitate large-scale visual preference optimization by tackling key data bottlenecks, massive-scale preference dataset with IM image pairs (1024px) across five categories and 300K video pairs (720p+) across three categories. Leveraging state-of-the-an gener- ative models and diverse prompts ensures consistent, reliable preference signals with bal- anced distributions. Remarkably, when applying Poly-DPO to our high-quality dataset, the optimal configuration converges to standard DPO. This convergence validates both our dataset quality and Poly-DPO's adaptive nature: sophisticated optimization becomes unnecessary with sufficient data quality, yet remains valuable for imperfect datasets. We comprehensively validate our approach across various visual generation models. On noisy datasets like Pick-a-Pic V2, Poly-DPO achieves 6.87 and 2.32 gains over Diffusion-DPe on GenEval for SD1.5 and SDXL., respectively. For our high-quality ViPO dataset, models achieve performance far exceeding those trained existing open-source preference datasets. These results confirm that addressing both algorithmic adaptability and data quality is essential for scaling visual preference optimization. Code, models and open-source datasets will be released at: https://gihubb.om/iiiinguuuu
construct ViPO,
INTRODUCTION
Preference optimization techniques, such as Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. (2022) and Direct Preference Optimization (DPO) Rafailov et al. (2023). have proven essential for aligning large-scale models with human values. Building on this success in language models, researchers have extended these paradigms to visual generation. Among various approaches, off-policy methods like Diffusion-DPO Wallace al. (2024)
particularly promising for large-scale
applications. Unlike on-policy RL approaches Xu et al. (2023); ;; Liang et al. (2025): ) Liu et al. (2025a); Xue al. (2025): Black al. (2024) that require costly iterative sampling, off-policy methods leverage pre-collected preference datasets without expensive policy deployment, making them inherently more suitable for scaling Wu et al. (2025a). However, while preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm remains largely unexplored.
We argue that the primary obstacle to scaling lies in the conflicting preference patterss prevalent in current datasets. Specifically, existing open-source preference datasets Wu et al. (2023b ay; Ma et al. (202): Kirstain et al.] (2023) are usually constructed by early diffusion models, contain substantial conflicts where winner images excel in certain dimensions (c.g., aesthetics) but underperform in others (e.g.. text-image alignment). Naively optimizing on such noisy datasets fails to learn meaningful preference patterns, fundamentally hindering effective scaling of preference optimization. Without proper handling of these conflicting signals, models struggle to extract genuine preference patterm, leading to suboptimal
This work was done during the first author's 's internship USA

---

Published as a conference paper at ICLR 2026
Crepatie
Performance Scaling with Training Preference Samplex
0238
Anahatic
Aandating
Qualit
Tharning Samplo (Mition)
Preference Sealing under Differeut Algorithmms and Data
Training Pick-a-pie Datasct
(e) Training ViPO-Image-1M Dataset
Figure 1: (a) Preference scaling with our Foly-DPO and ViPO-Image' IM dataset. (b) When training biased preference dataset such as Pick-a-pic V2, our Poly-DPO outperforms Diffusion-DPO in all evaluation dimensions. (c) Our proposed ViPO-Image- IM dataset comprehensively improve the SD3.55Medium.
performance that fails to further improve with data scale, as demonstrated in Figure | (a). Beyond biased preference patterns (conflict or over-simple samples), existing datasets suffer from multiple limitations: low visual resolution (typically 512-768), limited prompt diversity, imbalanced data distributions from random collection strategy, and constraints from outdated generation models, as shown in Table These factors collectively hinder the effective scaling of preference learning.
To better leamn from biased preference datasets, we propose Poly-DPO, which extends Diffusion-DDD with polynomial term that dynamically adjusts sample weighting based on prediction confidence. This mechanism enables effective learning across diverse data characteristics: for existing datasets that contain conflicting preferences (e.g., Pick-a-pic V2), it helps models focus on informative samples despite contradictory signals and improves the final generation quality as shown in Figure (b). To comprehensively address data quality barriers, we construct ViPO, a massive-scale and high-quality visual preference dataset comprising IM image pairs (1024px) across five categories and 300K video pairs (720p+) across three categories. By leveraging state-of-the-ar generative models (FLUX Labs| (2024). Qwen-Image Wu et al. (222a)), Wan Video Wan et al. (2025)) and systematic categorization, we ensure reliable, balanced preference signals that enable robust preference learning at scale.
Extensive experiments validate the synergy between our contributions. On noisy datasets like Pick-a-Pic V2, Poly-DPO significantly outperforms standard Diffusion-DPe by handling conflicting preference patterns. Training on our ViPO dataset, the 5 model achieves state-of
results far exceeding those trained on existing datasets Figure (a) and comprehensively improves the SD3.5-Medium as shown in Figure (c). Remarkably, when applied to IIPO-mmaee- IM, Poly-DPO converges to standard DPO (a + 0) and remains robust across a neighborhood around zero, indicating it works equally well on high-quality data without tuning. This convergence mutually validates both contributions: ViPO's quality enables stable optimization across different a values, while Poly-DPO adaptively simplifies through a single hyperparameter when data quality permits. These findings show that scaling visual preference optimization requires both algorithmic robustness for imperfect data and systematic data curation.
Our contributions are summarized as follows:
New Insight for Visual Preference Scaling: We demonstrate that the biased preference distributions characterized by conflicting patterns constitute a fundamental bottleneck for preference scaling, We reveal that standard Diffusion-DDD fails to extract effective signals from such data, leading to performance saturation despite data scaling.
Poly- DPO Optimization Algorithm: We introduce Poly-DPO, which dynamically adjusts sample weighting based on confidence levels, enabling effective learning from conflicting patterns noisy datasets while preventing over-confidence trivially distinguishable preferences.
Large-Scale High-Quality Dataset: We construct ViPO dataset with 1M high-resolution image pairs and 300K video pairs using state-of-the-art models and systematic categorization, providing reliable and balanced preference signals that establish a new benchmark for preference learning at scale.
Mutual Validation of Approach: Our experiments demonstrate that Poly-DPO excels on biased datasets while converging to standard DPO (a 0) with robustness across neighboring values high-quality ViPO- Image- IM data, confirming that sophisticated optimization becomes unnecessary with sufficient data quality yet remains essential for imperfect datasets.
2

---

Published as a conference paper at ICLR 2026
ViPO-Image-In IM
ViPO-Video-300G
Iu
SANA SANALS
Queen-tmage
HiDreom--11
PLUXX2-ter
PLUX.I-der
PIAARE
Wae2.2
WanZi1
HinyannVide
Yeed
Text Rendering
Composition
Aesthetics
Human Quality
Alignment
Motion Quality
Alignment
Visual Quality
Figure 2:
Overview of our ViPO-Image- IM and ViPO-Video-300M 300K dataset.
Dataset
Prompt
Image/Videe
Pair
Resolution
Construction
Generative Models
Image Dataset
HPDv1
25,205
98,807
25,205
[512-960]
Random
SDL.4
HPDv2
103,700
430,060
645,090
[480-640]
Random
SD2.0, Cog View2, DALL-E2
Pick-a-pic
37,523
623,694
583,747
[512-768]
Random
SD2.1 SDXI, Dreamlike,
Pick-a-pic v2
58,960
928,068
959,040
[512-768]
Random
SD2.1, SDXL, Dreamlike,
HPDv3
202,274
1,088,274
1.17M
[256-1024]
Random
SD1.4, SDXL, FLUX.I dev, etc
Ours
1,000,000
2,000,000
1,00M
1024
Categoried
Qwen-Image HiDream-I1, etc
Video Dutuset
VideoDPO
10, 000
20, 000
10, 000
480p
Random Categoried
CogVideo, VideoCrafter2, etc WanVideo, Vea3, etc
Ours
30, 000
60, 000
30, 000
720p, 1024p
Table I: Comparison with existing open-source preference datasets.
2 RELATED WORKS
Diffusion- based Visual Generation. Building upon pioneering diffusion models, SShl- Dickstiin et al.
(; Ho et al. (2020); Song et al. (202 b); Lipman et al. (2023) and their successful scaling Rombach Ho & Salimans 2022): Dhariwal & Nichol (2021), lj. visual generation has achieved remarkable
al. 2
progress. Ad Advanced models like FLUX Labs (2024). Owen-Image Wu et al. (2025a) for images, and
Hunyuan Video Kong al. (202)), Wan Video Wan et al. (2025) for videos, have enabled stunning visual
content creation across diverse applications Zhang et al. (2023); Mou et al. (2024); |uiz et al. (2023): Ye al. (2023); BBrooks et al.) (2023); Li et al. (2025). Despite these advances, two key challenges remain: aligning outputs with complex user prompts and optimizing multiple quality dimensions simultancously.
Reinforcement Learning from Human Feedback (RLHF). RI has demonstrated remarkable success in aligning large language models with human values Ouyang et al. (2022): Touvron et al. (2023a);
Bai et al. (2023); Wang et al. (2024); Team et al. (2025). Current approaches fall into two categories: on-policy methods (PPO Schuulmuul et al. (2017), GRPOOShao et al. (2224) that require iterative sampling and reward model evaluation during training, and off-policy methods (DPO Raaailov et al. (2023) that
leam directly from pre-collected preference datasets, Off-policy methods avoid the computational overhead of online sampling, making them more efficient Touvron et al. (2023b); Rafailov et al. (2023). though
their effectiveness depends on preference dataset quality Morimura et al. 2024); Wu et al. (2025a).
Reinforcement Learning for Visual Generation. Recent research extends RL. success from L.Mss to visual generation. On-policy methods include ReFL-based approaches Xu et al. (2023); y; Clark et al. (2024); Li et al. (2024a): Wu et al. (2025b) that integrate reward maximization into diffusion training. and PPO-based methods [Black al. (2024); Xue al.] (2025); Liu al. (2025a) that model diffusion as an MDP. However, these face scalability constraints from computational intensity and reward hacking vulnerability. Off-policy methods, particularly DPO-based approaches Wallace et al. (2024); Yang et al.) (2024] Dong et al. (2024): [Liu et al. (2025b); Karthik et al. (2025); Zhu et al. (2025); [hang et al. (2025a); Liu et al. (2026), offer superior computational scalability by training on preference pairs without online sampling, but they require high-quality preference datasets and effective optimization algorithms.
3 DIFFUSION PREFERENCE OPTIMIZATION WITH POLY-DPO
3.1 PRELIMINARIES FOR DIFFUSION-DPO
Diffusion Models. Denoising diffusion models operate through two complementary processes: forward process that progressively corrupts data by introducing noise, and reverse process that reconstructs clean data from the corrupted versions. Specifically, during the forward process, a clean data point x undergoes noise comupptin at timestep E [0,1]. resulting in conditional distribution g(x.|x) characterized by X, = O₁,X + oré, where E ~ (=, Cke, oy represent predefined noise scheduling parameters, and X log(a, /af) denotes the logarithmic signal comal-tt--noie noise ratio (SNR). With the input condition the training process optimizes a weighted noise prediction objective formulated as:
( = Ex -tt.///o...lwwww///////xx:.:):::
(I)
where W/ represents a weighting function and X: = dD/dt. Notably, most diffusion and flow matching training objectives can be expressed in the form of Eq. (I) through appropriate choices of w, and Aç-
3

---

Published as a conference paper at ICLR 2026
Reward Models. For given image and input conditioning c, reward model R(x,c) represents a function that quantifies the quality of the generated output. A widely adopted framework for modeling human preferences is the Bradley-Terry (BT) model, which establishes the preference probability distribution over a triplet (c, x", x)): P(x > x'|c)= == a [R(x") ,c) (()x'.))), where a denotes the sigmoid function, and x"", x' represent the winner and loser images, respectively. The objective of reward fine-tuning is to optimize the diffusion model py such that maximizes the expected reward of generated outputs while incorporating KL regularization DRL. to prevent reward over-optimization: maxe Ee, (x,x--o(((e) [R(x,c)] BDI.. (po(: |c) llpes (. (c)) where Pref reference model and 3 is a hyperparameter that controls the strength of KL regularization.
Diflusion-DPO. Following the DPO framework Rafailov et al. (2023), the training objective can be reformulated to enable direct optimization through the conditional distribution px(x))):
[ log Po(x")) (rer(ack)
3 log pa(a'") (at))
pres(ae)) )
po
(0) (o s²)) log
However, directly applying Eq. to diffusion models presents a fundamental challenge, as the logllikelihoods of diffusion models are intractable. To address this limitation, Diffusion-DPO Wallace ct al. (2024) introduces an approximation that connects the diffusion denoising process with the forward trainin objective in Eq. ① Specifically, timestep t, the log- ligeliillioood ratio can be approximated as:
log po(a)) == we (11E0(x - <t (le (x - eall))
(3)
(x))
By substituting Eq. 3 into Eq. (2). we obtain the final Diffusion-DDOO loss function:
LDillusive-DNN (f) = E(ee" we ee). logo (-Bu
(4)
- -(1eo(xx):c,)) 6x l2 Ever (x/3,,) -, (l]
3.2 POLY-DPO: POLYNOMIAL EXPANSION FOR PREFERENCE OPTIMIZATION
Diffusion-DPOO as the Binary Classification Task. Building upon the Diffusion- DPO framework. we propose Poly-DPO, which leverages insights from poly loss Leng al. (2022) design to enhance preference learning. We begin by reinterpreting the standard Diffusion-DDe objective into the standard binary classification task. Specifically, we can define the preference probability:
Po(x"") Prer(x
Po(a')
[ log
B log
(5)
(ref(xe))
which quantifies the model's relative preference for the winner image x over the loser image x' compared to the reference model. This allows us to rewrite the Diffusion-DPO loss as:
DDiffusion-DOO (0)
- -
(log(">>))
(6)
This reformulation reveals that Diffusion DPO can be regarded a cross-entropy loss for binary classification, where the model learns to maximize the probability of correctly ranking preferred generations.
Polynomial Expansion of Preference Learning, Inspired by poly loss Leng et al. (2022), we can get the Taylor expansion of the standard cross- moos-entropp loss in the context of Diffusion-Dar framework:
Lpimusion- DPO llog(p">))
-1 -1------- 1 + 2 (1 221
) =1
(7)
The core idea of the Poly Loss is to add a perturb term ar for the Top-N polynomials that contribute the most to the gradient and keep others, and we can obtain the Poly-N loss:
L = (1+a:) +... (LLaa////NNN N
+1(N+1)(1---)) N+1
perturbed by ag
as LCR
(
=-oog(p">1 + Σ az (1-p
4

---

Published as a conference paper at ICLR 2026
1 hyper-parameter
2 lines of code
3 preference distributions
Ammmiie nFO (1))))
doo_1oss ignoid logits)
a 0: Noisy Dataset (Hard to learn)
poly_ loss sigroid (lggitt) loss = dos__
a Over-simple Dataset (Casy to leurn)
Lydy DeO log (1 plo)
atpha toss
High-Quality & Ralanced Dataset
Figure 3: Summary of our Poly-DPO. By adjusting only one hyperparameter and introducing only two new lines of code, our Poly-DPO can handle preference datasets with three different data distributions.
However, it is unrealistic perturb and adjust parameters for all polynomials. A simple form is modify only the first term that contributes the most to the gradient Lendg et al. (2022). thus obtaining Poly-DPO loss:
Toyy DPO = -ogo w>!)
(p">)) +1(11-1>>)
Hence, Poly-DPO rescales the DPO gradient (1-p">1) ( (I+ap=>>), ) where - ologitt):
i Paiy-DPO Ölogit
(p"
1
op op">>(1 -p=>t)= - (1-p"t -p 1
(10)
Poly factor
a >>0 0 (Confidence Enhancing). When datasets contain conflicting preference patterns, models struggle to extract consistent signals. Setting a > 0 upweights uncertain samples (probability near 0.5) and downweights extreme cases (near 0 or 1). This prevents the model from being confused by conflicting patters,, and instead focuses learning on borderline cases where consistent improvement is possible.
a < 0 (Confidence Reducing). When datasets contain trivially distinguishable preferences (e.g., synthetic dataset with shuffled losers in Section (5.2). models quickly achieve high confidence but only learn surface-level distinctions. Setting
reduces gradient contributions from high-confidence samples, preventing over-fitting and forcing continued exploration of winner-loser differences.
a (Standard DPO). When datasets contain balanced, high-quality preference signals without significant conflicts or trivial patterss, the optimal configuration of Poly-DPO converges to standard DPO and is highly robust to the choice a.
Remark. As shown in Figure 3, our Poly-DPO augments Diffusion DPO with single additive term that makes training explicitly confidence-aware. By tuning a, it dynamically reweights samples across models and preference datasets, pushing the learning process toward informative samples while tempering over- and under-confidence, making the diffusion model better capture diverse preference patterns and achieve higher generation quality. In Section 5.2 and Figure 4 we verify the effectiveness of for these three different preference distribution datasets.
LARGE-SCALE VISUAL PREFERENCE DATASET CONSTRUCTION
Motivation and Design Principles. Current open-source preference datasets suffer from three critical limitations that fundamentally impede scaling: (i) low resolution (512 (512-768px) and limited prompt diversity restrict learning of fine-grained details; (ii) reliance on early- aurly-generation models produces unreliable preference signals; and (iii) random collection creates imbalanced distributions where simple patterns dominate while critical aspects remain underrepresented. To address these challenges, we construct a large-scale dataset using mate-of-the-arr models (FLUX, Qwen-Image for images; Wan Video, Seedance for videos) with systematic categorical organization to ensure balanced, reliable preference signals. Specifically, construct IM high-resolution (1024px) image preference pairs across five categories and 300K vidco pairs across three categories, as illustrated in Figure Details on specific construction pipelines, filtering procedures, and labeling strategies are provided in the Appendix.
ViPO Image-ma. We organize image preferences into five dimensions, each with 200K pairs: (1) Aesthetics: visual appeal and artistic merit: (2) Text-Image Alignment: semantic correspondence with prompts; (3) Text Rendering: accuracy of rendered text elements; (4) Portrait Quality: anatomical correcnesss and realism; (5) Compasition: spatial arrangement and visual organization. For data construction, we leverage publicly available prompts from HuggingFace, employ state-of-the-are generators to create high-quality pairs, and use multiple VLMs for filtering and labeling.
Video preferences span three dimensions, each with 100K pairs: (1) Motion Quality: temporal dynamics and smoothness; (2) Video-Text Alignment: semantic correspondence throughout temporal sequences; (3) Visual Quality: frame clarity and temporal consistency. We employ diverse generation strategies, including 12V based on our image dataset and T2V/T21VV with different models to create varied preference pattems..
5

---

Published as a conference paper at ICLR 2026
Open-Source Datasets Due to licensing constraints of proprietary models, the original dataset version containing Seedream-3.0 and Seedance- 1.0 outputs may not be publicly released. To better serve the open-source community, we provide altemnttie version of ViPO that substitutes these proprictary outputs with publicly available models (FL.UX22-de) for images, Wan2.2-A 14B-12V for videos), enabling full reproducibility and broader downstream applications such diffusion distillation for open-sourced models. Our validation confirms that the released dataset maintains comparable quality, with both SFT and Poly-DPO consistently yielding significant improvements acToss all evaluated models and benchmarks. Detailed dataset construction updates and comprehensive experimental results are provided in Appendix
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Generation Models and Training Datasets. We conduct experiments on image generation using SDL.5, SDXL, SD3, and FLUX models, and video generation using Wan2.1-T2V-1.33 For SD1.5, we train on PickaPic-v2 for fair comparison with previous methods and test on multiple datasets to evaluate resilience to preference noise. We train all image models on our ViPO-Image- IM dataset (excluding text rendering subset for SD1.5 due to its limited text capabilities) and train Wan2.1-T2V-1. on ViPO-Video OPPO-Vide-300K.
Evaluation Protocol. For SD1.5, we follow established protocols using CLIP- reward models (ImageReward [Xu et al. (2023), 1PSv2.1 |Wu et al. (2023a), Aesthetic Predictor [Schuhmann et al. (2022)
and test datasets (HPSv2 Wu et al. (2023a). Pick-a-Pic Pick-a-PieKKirstainn et al. (2023), Parti Yu et al. (2022) ). For
high-resolution models (SDXL, SD3, FLUX), adopt mulii-dimensional evaluation: (1) Aesthetics:
DeQA You et al. 22025; (2) Aligmment: DPG-Bench Hu et al. (2024): (3) Text Rendering: CVTG-2K Du
al.) (2025); (4) Human Quality: GPT-4o evaluation; (5) Composition: GenEval Ghosh et al. (2023
Video generation is evaluated on VBench2.0 Huang et al. (2024).
5.2 ABLATION STUDIES FOR POLY-DPO
To comprehensively demonstrate that our proposed Poly-DPO can adapt to different preference datasets by adjusting the single hyperparameter a, we conduct a series of ablation experiments based on the SD1.5 model. For these experiments, we randomly sampling 300 prompts from each of four sources: the test set of the Parti dataset, the test set of Pick-a-pic V2, the test set of HPD) v2, and the "Validation Unique" set of Pick-a-pic V1. This resulted in a total of 1,200 prompts, for which single image was generated for each. These prompts are then used to simulate three scenarios with distinct characteristics as discussed in Section 3.2 (I) noisy dataset with conflicting preference patterns, (2) over-simple dataset dominated
by simple preference patterss, and (3) high-quality datasets with balanced preference distributions.
30.5
27.00-
HPSV2 1 Scora
20.75
20.0
28.00
HP5/2
26.25-
26.00-
-1.0
-is
1.0
(a) Naisy Preference Datasel
(b) Ovensimple Preference Datasel
(c) Our Full Preference Datasel
Figure 4: Ablation studies with different a on datasets with varying noise properties. While only the
HPSv2.1 score is visualized for elarity, similar trend is observed across all other evaluation metrics
Noisy Preference Dataset. As the largest publicly available preference dataset, Pick-a-Pic Pic V2 exhibits significant minlli-dimensionall conflicts in preference signals. Specifically, when we evaluate image pairs using five different reward models (PickScore, ImageReward, HPSv2, Aesthetic Score, and CL.IP Score), only 20.79% of pairs show consistent preference rankings across all five dimensions, where one image con- sistently scores higher than the other. This dimensional conflict prevent models from learning meaningful preference patterns, as illustrated in Figure (a). Consequently, this dataset benefits from Poly-DPO with a>0 O, which enables the model to better navigate these conflicting signals by adaptively weighting samples based on prediction confidence. In our experiments, we found a =8 8 has the best experimental results.
Over-simple Preference Dataset. To validate that Poly-DPO with a< O mitigates overconfidence, we construct a synthetic dataset where simple patterns dominate. We first perform SFT on SD1.5 using winner images from HIPO-ImageIIN IM, then create preference pairs by randomly shuffling losers within batches while maintaining original winners. This setup causes critical failure under standard DPO: the model quickly becomes overconfident and overfits to reproducing winner images rather than learning winner- distinctions. The high confidence from trivial preference patterns prevents the model from learning subtle preferences essential for alignment. We show that Poly-DPO with a < 0 can penalize overconfident predic- tions, and forcing the model to learn more meaningful preference patterns in this scenario in Figure (b).
6

---

Published as a conference paper at ICLR 2026
High-quality and Balanced Preference Dataset. While Poly-DPO with 0 and a < 0 performs well on noisy and imbalanced datasets respectively, we observe an interesting phenomenon when training SFT-imitialized SD1.5 on our complete ViPO-Image IM dataset: the optimal a value converges to approximately zero, where Poly-DPO converges to standard DPO and exhibits robust performance across different hyperparameter settings, as demonstrated in (e. This convergence validates dataset quality-when preferences are reliable and balanced, adaptive optimization becomes unnecessary, confirming that data quality remains the primary factor for successful and scalable preference optimization.
5.3 RESULTS ON PICK-A-PIC V2 TRAINING DATASET
To validate Poly-DPO's effectiveness, we conduct experiments using SD1.5 and SDXL trained on Pick-a- Pic V2, which contains substantial conflicting preference patterss analyzed in Section 552 making it an ideal testbed for demonstrating robustness to noisy real-world data. Table[]presents evaluation results across four test datasets. Poly-DPO consistently outperforms both Diffusion-DPO and Diffusion-KTO across all metrics. On Pick-a-Pic V2 test set, Poly-DPO achieves 4.4% improvement in PickScore and 13.1 %% in HPSv2.1, significantly surpassing Diffusion-DPOI gains of 1.8% and 4.4% respectively. The most sub- stantial improvements appear in ImageReward scores (+0,594 vs. +0.212), This pattern holds across other test sets: on HPD V2, Poly-DPO achieves 15.9% HPSv2.1 improvement versus Diffusion-DDN DPO's 5.3%; on Parti, the RewgeReward gain reaches +0.542 versus +0.158. These consistent improvements confirm Mly--DO''s ability to extract meaningful preference signals despite conflicting patterns. T compositional understanding using GenEval benchmark. Poly-DPO achieves the highest overall scores among off-policy methods for both SD1.5 (49.87) and SDXL (60.34), even surpassing on-policy SPO while avoiding iterative sampling costs. Notably, Poly-DPO excels at challenging tasks: for SDL.5, it achieves 51.25 on counting (vs. Diffusion-DPOY tribute binding reaches 31. .000 compared to Diffusion-DPO 18.50. These substantial gains demonstrate that confidence-bart based reweighting enables learning nuanced preference patterns beyond simple visual attributes.
38.75) and 14.00 on attribute binding (vs. 3.75); for SDXL, at-
Table 2: SD1.5 comparison results when trained on the Pick-a-Pic V2 dataset and evaluated on multiple datasets. For each prompt, we generate 4 images and report the average reward scores. Baseline results evaluated with official released checkpoints, and all evaluations conducted under the same setting.
Eval Dataset
Method
Paradigm
PickScore
HPSv2.1 1
Aesthetic 1
ImageReward
SD1.5
20.57
25,02
5,42
0,085
Pick-a-Pic V2 (Test)
Diffusion-DPO
Off-Policy
2..95
26. 12.4.4%
5.55_244
0.297_
Diffusion-KTO
Off-Policy
21.06
18.06
5.66....%
0.628
Poly-DPO (Ours)
Off-Policy
21.48 4..4%
28.30,
5.67.....%
0.679.0.504
SDI.S Diffusion-DPO
20.86
0.246
5.58
0.139
HPD V2 (Test)
Off-Policy
21.31
0.239 45.3%
5.71.
0.338
Diffusion-Ke KTO Poly-DPO (Ours)
Off-Policy
21.45.2..%%
0,2844.15.4%
5.80,.3.9%
0.690
Off-Policy
21.87 +-4.9%
0.285 10.99
4.83.-4.5%
0.716
SD1.5
21.28
0.253
5,36
0,194
Parti (lest)
Diffusion-DPO
Off-Policy
21 21.52.112%
0.261 43.22%
5.44
0.352
Diffusion-KIOO
Off-Policy
21.50 +1.0%
01.279.
4.55.
0.615
Poly-DPO (Ours)
Oft-Policy
21.89 42.99%
0,280 0,.80,,10.000 10.7%
5.56 4.3.7%
0.736, 77.36,0.542
SD1.5
20.56
24.05
5.47
0.008
DDPO
On-Policy
21 21.066+2.47%
24. 24.91 68%
5. 5.59
0,082_.0074
D3PO
On-Policy
20.76..1.08
23.9 23.97_
5.53..1.1%
0. 0.1224...132
Pick-a-Pic V1 (Validation Unique)
SPO
On-Policy
21.22, 41.22.....2
25.83.47.%
5.93 8..9%
0.168-..100
Diffusion-DPO
Off-Policy
20.99. 1..19
25.54 16.256
5.60
0.302 .029
Diffusion-KIT -TTO
Off-Policy
21.12
28. 19 17.22%
5.68 +3.8%
0.6422...o0 0.034
Poly-DPO (Ours)
Off-Policy
21.48 +-4.5%
28.32 17.8%
5.68 +3.8%%
0.671
5.4 RESULTS ON VIPO-IMAGE- IM TRAINING DATASET
Composition. Table demonstrates the effectiveness of our ViPO-Image- 1M dataset across multiple model architectures the GenEval benchmark. All models show substantial improvements when trained with our dataset. SD1.5 improves from 0.42 to 0.52 overall (+23.8%), with particularly strong gains in two-object generation (0.38-+0666) and attribute binding score, surpassing many baseline models, with attribute binding improving dramatically from 0.16 to 0.42. SDD..- already strong 0.69, reaches 0.83 after training, approaching the performance of HiDream-I1 FI-Ful (0.83), a model specifically designed for compositional generation. #1.UX.1-der shows consistent improvements across all metrics, reaching 0.79 overall score.
SDXL achieves 0.63 overall
7

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation results on GenEval (Ghosh et al. 2023 with Pick-a-pic V2 training dataset. The ww..S/SXX//TOO/ffffisiiii
results are evaluated with their officially released models under the same setting as LPO Zhang al. (025b). The SPO/L baseline results are from the LPO paper.
RL Paradigm
Single Object
Two Object
Attribute Binding
Model
Counting
Colors
Position
Overall
SD1.5 SPO
37.63
37.81
74.73
3,50
4.57
42.34
On-Policy
36.62
34.83
72.34
6,50
41.53
LPO
On-Policy
97.81
55.30
42.19
80.59
6.75
10.00
48.77
Diffusion-DPO
Off-Policy
96.88
39.90
38.75
75.53
3.25
3.75
43,00
Diffusion-K0 KTT
Off-Policy
97,50
35.35
36.25
79.79
7.00
6.00
43.65
Poly-DPO (Ours)
Off Pot-Policy
96.25
46.46
51,25
87.23
4.00
14,00
49.87
SDXL
98.12
75.25
43.75
89.63
11.25
15.75
55.63
SPO
On- On-Policy
96.88
69.70
37.19
83,51
9.50
19.75
52,75
LPO
On-Policy
99.69
84.34
43.13
90.43
13.75
27.75
59.85
Diffusion-DPO
Off Off-Policy
99.38
82.58
49.06
85.11
13.05
18.50
58.02
Off-Policy
66.41
84.31
18.75
Poly-DPO (Ours)
Off-Policy
98.75
82.83
46,25
87.23
16.00
31,00
60.34
Table 4: Evaluation results on GenEval Ghosh et al. (2023)
with our ViPO-Image-IM training dataset.
Single Object
Two Object
Attribute Binding
Model
Counting
Colors
Position
Overall
PixArt-o
0.98
0.50
0.44
0.80
0.08
0.07
0.48
SD3.5 Large
0.98
0.89
0.73
0.83
0.34
0.47
0.71
HiDream-I1-Ful
1.00
0.98
0.79
0.91
0.60
0.72
0.83
SD1.5
0.96
0.38
0.38
0.75
0.04
0.05
0.42
+ SFT
0.99
0.49
0.38
0.78
0.46
+ SFT Poly-DPO
0.98
0.66
0.50
0.84
0.07
0.17
0.54
SDXL
0.98
0.75
0.44
0.90
0.11
0.16
0.56
SFT
0.98
0.77
0.43
0.88
0.13
0.21
0.57
+ SFT Poly-DPO
1.00
0.88
0.45
0.93
0.09
0.42
0.63
SD3.5-MMdium
1.00
0.87
0.68
0.80
0.20
0.57
0.69
+ SFT
1.00
0.97
0.74
0.91
0.43
0.77
0.80
+ SFT & Poly-DPO
1.00
0.97
0.75
0.91
0.47
0.86
0.83
FLUX.1 [Dev]
1.00
0.86
0.80
0.78
+ SFT
1.00
0.90
0.74
0.87
0.38
0.62
0.75
+ SFT & Poly-DPO
0.99
0.97
0.83
0.85
0.40
0.70
0.79
Table 5: Evaluation results on DPG-Bench Hu et al. (024)) with our ViPO-Image-IN training dataset.
Model
Global
Entity
Attribute
Relation
Other
Overall
Hunyuan-DiT
84.59
80.59
88.01
74.36
86.41
78.87
PixAnt-
86.89
82,89
88.94
86.59
87.68
80.54
DALL-E3
90.97
89.61
88.39
90.58
89.83
83.50
SD3 Medium
87.90
91.01
88.83
80.70
88.68
84.08
HiDream-I1-Fa
76.44
90.22
89.48
93.74
91.83
85.89
GPT-Image
88.89
88.94
89.84
92.63
90.96
85.15
SD3.5-Mcdium
91.70
90.59
89.49
92.21
85.12
84.24
+SFT
84.80
89.97
88. 14
93.69
82.00
84.24
+SFT & Poły-DPO
84.80
92.64
90.10
94.81
89.20
87.71
FLUX.1 [Dev]
74.35
90.00
88.96
90.87
88.33
83.84
SFT
85.41
89,21
85.17
92.72
80,40
83.59
+FT & Poly-DPO
90.99
91.05
90.91
93.73
91.12
87.31
Image- Text Alignment. both 3D3.5-Medium and FL.UX. dev achieve state-of-the-ar performance after training, with overall
Tables 5 present evaluation results on text-image alignment. On DPG-Bench,
8

---

Published as a conference paper at ICLR 2026
Table 6: SD3.5-Medium FLUX-dev comparison results when trained on our ViPO-Image- IM dataset and evaluated across multiple benchmarks. For each prompt, we generate 4 images and report the average score. We provide more details about these experiments in the Supplementary Material.
Aesthetics DeQA f
Alignment DPC--ench
Text Rendering Human Quality
Composition GenEval f
Method
CVTG-2K +
GPT-4o Acc f
SD3.5-Medium
4.27
84.24
0.4378
73.25
0.69
+ SFT
4.31
84.24
0.5887
77.50
0.80
+ SFT Poly-DPO
4.31
87.71
0.6995
85.25
0.83
FLUXXI-dew
4.37
83.84
0.4878
80.00
0.69
+ SFT
4.32
83.59
0.2126
81.75
0.75
+ SFT & Poly-DPO
4.40
87.31
0.6859
88.75
0.79
Table 7: W- Experiments on VBench-2.0 when trained with our ViPO- Videviddo-30K dataset.
Models
Human Identity
Material
Thermotics
Dynamic Spatial Rel.
Dynamic Attribute
Motion Order Und.
Human Interaction
Camera Motion
Motion Rationality
Wan2.1
62.18
69.75
72.36
24.64
53.48
35.35
74.00
31.79
43.68
Poly-DPO
67.99
71.57
68.53
33,82
$7,00
38.62
78.00
32.49
47.70
scores of 87.71 and 87.31 respectively, surpassing commercial models like GPT-Image 1 (85.15) and approaching HiDream-II-Fus (85.89). The models excel particularly in relational understanding, with SD3.5-Medium achieving 94.81 on relation tasks.
Aesthetics and Human Quality Evaluation. We evaluate aesthetic quality and human generation capabilities as shown in Table 6 For aesthetic assessment using DeQA You et al. (2025). we observe modest but consistent improvements -SDd.u-Meduum: 4.27-+4.31, FLUX: 1.37-+4.44 on DrawBench, demonstrating that our training maintains aesthetic quality while improving technical capabilities. For human quality evaluation, we use GPT-4o to assess anatomical correctness on 400 human-related prompts. The results show substantial improvements: SD3.5-Medium's accuracy increases from 73.25% to 85.25%, while FLUX improves from 80.00% to 88.75%. These gains address persistent challenges human image generation, including correct proportions, realistic poses, and proper body structure. Our proposed ViPO-Image- IM achieves simultaneous improvements across multiple visual dimensions.
Text Rendering. Training on our dataset significantly improves performance on the challenging CVTG-2K text rendering benchmark Du et al. (2025). As shown in Table 6 our full pipeline boosts -De.imMedium's word accuracy by 59.8% (from 0.4378 to 0.6995). Notably for PLUX.1-dew. overcomes an SFT-induced performance degradation achieve a strong final score of 0.6859. A more detailed analysis, including results on multi-region text, is available in Tabe 15 in the Appendix.
5.5 RESULTS ON VIDEOVIDDO--00E 300K TRAINING DATASET
We evaluate the effectiveness of our ViPO- Video-300K dataset using Wan2. 1-T2--1.3333 model on VBench-2.0 benchmark Zheng et al. (2025). as shown in Table Training with ViPO-Video-300 Video-300K yields consistent improvements across nearly all evaluated dimensions, Most notably, the model shows significant gains in motion-related metrics: Dynamic Spatial Relationship improves from 24.64 to 33.82 (+37.44 3.)). Motion Order Understanding increases from 35.35 38.62, and Motion Rationality rises from 43.68 to 47.70. These improvements demonstrate that our video preference dataset effectively captures temporal dynamics and motion quality distinctions. Human-centric metries show substantial improvements, with Human Identity increasing from 62.18 to 67.99 7.99 and Human Interaction from 74.00 to 78.00, validating the quality of human motion preferences in our dataset. While Thermotics shows a slight decrease, the overall pattern of improvements across diverse evaluation criteria confirms that ViPO-Video- Video-3008 enables balanced enhancement of video generation capabilities, particularly in challenging aspects like motion understanding and temporal consistency.
5.6 HUMAN EVALUATION ON VIPO DATASETS
Human Evaluation Setup. We randomly sampled 40 images and 20 videos per category, recruiting 18 annotators to identify the superior sample in each pair based on aategory-specifii instructions, yielding 4,378 annotations. We define rater accuracy alignment with the majority vote across raters. All raters

---

Published as conference paper at ICLR 2026
exceed 70% accuracy with mean of 87.2%, confirming high annotation reliability (see Figure Figure in the Supplementary).
g
and
Reliability of ViPO Dataset Annotations. We benchmark the VLM-basedd rater against human annotators using majority-vote consensus as ground truth. As shown Figure 5 the VLM achieves an overall agreement rate of 81. 81.2%, surpassing the average human annotator (74.7%). The VLM excels particularly images (84.0% vs. 74.9%), driven by strong performance on Aesthetic (95.0%) and Alignment (92.5%). For videos, VLM and human performance comparable (7 (71.7% vs. 72.2%), though the VLM lags Motion Quality (55.0% 0% vs, 67.2 67.2%), indicating that current VLMs still struggle with fine-grained temporal dynamics.
VLM
VLM Human
Video
Wrong 18.8%
Wrong 28.3%
72.2%
Correct 81.2%
Correct 74.7%
84 0%
Image
74.9%
66
0.2
64
36
06
Accuracy/ Agreement Rate (b) Comparison of Performance by Modality
(a) Comparison of Overall Performance
11.4%
Human
73.8%
72.9%
73.0%
17.2%
41.7%
01.45
66
Accuracy
64
60
(eAgsmwe))
jocmpsellise
(humon))
(ecrenddring)
(dlgeemett)
penual quaRy)
Category
(c) Comparison and Human Performance Category
Figure 5: Performance Comparison between VLM and Human Raters. Accuracy (or Agreement Rate) is defined as the frequency with which choice aligns with the consensus label (majority vote among human raters, excluding VLM predictions). (a) Overall: The VI.M (81 .2)% demonstrates higher consistency with the consensus than the average individual human annotator (74.7%). (b) By Modality: The VLM significantly outperforms humans images (84.0% vs. 74.9%) but performs comparably on video tasks (71.7% vs. 72.2%). (c) By Category: The VLM excels most metrics like Aesthetic (95.0%) but only struggles with temporal Motion Quality (55.0% 67. 07.2%).
CONCLUSION
In this paper, we demonstrated that conflicting preference patterns in existing datasets limit visual preference optimization scaling. We introduced Poly-DPO, which dynamically adjusts sample weighting based on confidence levels, enabling effective learning across diverse data characteristics. We also constructed ViPO, a large-scale dataset with IM image and 300K video pairs, ensuring reliable preference signals across multiple quality dimensions. Our experiments show Poly-DPO significantly improves performance on noisy datasets like Pick-a-Pic V2 while achieving state-of-the-an results on ViPO. Remarkably, Poly-DPO converges standard DPO on ViPO, confirming that sophisticated optimization becomes unnecessary with sufficient data quality. This reveals that scaling visual preference optimization requires addressing data quality and algorithmic robustness in tandem.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Gc, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Kegin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprini arXiv:2511.21631 2025.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Leamingg follow image editing instructions. In CVPR, 2023.
Yixiong Chen, Li Liu, and Chris Ding, X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models. arXiv preprint arXiv:2305.70843, 2023.
Christopher Clark, Jieyu Zhang, Zixian Ma, Jac Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint 2026.
preprime arXiv:2601. 106/
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on diffcrentiable rewards. In ICLR, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. TMLR, 2024.
Nikai Du, Zhennan Chen, Shan Gao, Zhizhou Chen, Xi Chen, Zhengkai Jiang, Jian Yang, and Ying Tai. Textcrafter: Accurately rendering multiple texts complex visual scenes. arXiv preprint 1oXiv:2503.23400 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to-image alignment. NewrlPS, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprinl arXiv:2207. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeuriPS, 2020.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome, and Jongheon Jeong. Margin- pref-
erence optimization for aligning diffusion models without reference. arXiv preprint arXiv:2406.06422 2024.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arX1v:2403.05135, 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024.
Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for -ittte generation. ICCV, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03603, 2024.
Black Forest Labs. http::///gthhhb..cm/////dad----
2024.

---

Published as a conference paper at ICLR 2026
Black Forest Labs. FLUX.2: Frontier Visual Intelligence. http://///bb.aaabloo////eee 2025.
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. arXiv preprint arXiv:2204. 12511, 2022.
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang. Xuefeng Xiao, and Chen Chen. Controlnct++: Improving conditional controls with efficient consistency feedback. In ECCV, 2024a.
Ming Li, Xin Gu, Fan Chen, Xiaoying Xing. Longyin Wen, Chen Chen, and Sijie Zhu. Superedit: Rectifying and facilitating supervision for instruction-basel image editing. In ICCV, 2025.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. diffusion models by optimizing human utility. NeurIPS, 2024b.
Aligning
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. In CVPR, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl arXiv preprint arXiv:2505.05470. 2025a.
Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Videodpo: Omni-preference alignment for video diffusion generation. CVPR, 2025b.
Xinxin Liu, Ming Li, Zonglin Lyu, Yuzhang Shang, and Chen Chen.
Learning from noisy preferences: A semi-supervised learning approach to direct preference optimization. In ICLR, 2026.
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng L.i. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:2508.03789, 2025.
Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization. In EMNLP, 2024.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint (tXix:2307.01955 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurlPS, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aanush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurlPS, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (tt:::777.34444 2017.
12

---

Published as a conference paper at ICLR 2026
Bytedance Seed. Seed1.8 model card: Towards generalized real-world agency. Technical report, 2025.
Team Seedance, Heyi Chen, Siyan Chen, Xin Chen, Yanfei Chen, Ying Chen, Zhuo Chen, Feng Cheng, Tianheng Cheng, Xinqi Cheng, et al. Seedance 1.5 pro: A native audio-visual joint generation 2025.
foundation model. arXiv preprint arXiv:2512.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint wrXiv:2402.03300 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICMI,,
Jiaming Song, Chenlin Meng, and Stefano Ernon. Denoising diffusion implicit models. In ICLR. 202
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 12599, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288, 2023b.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In CVPR, 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-War Xic. Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint wrXi:2250.220314 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint (xXiv2409.1219) 2024.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02322 2025a.
Jie Wu, Yu Gao, Zilyu Ye, Ming Li, Liang Li, Hanzhong Guo, Jie Liu, Zeyue Xue, Xiaoxia Hou, Wei Liu, et al. Rewarddance: Reward scaling in visual generation. arXiv preprint arXiv:2509.08822) 2025b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint 23X.v:2306.09341, 2023a.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv: 2023b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS, 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07816 2025.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In CVPR, 2024.
13

---

Published as a conference paper at ICLR 2026
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 3081:22308.0671,,222 2023.
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. In CVPR, 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Daoan Zhang. Guangehen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, and Jiebo Luo. Bridging sft and dpo for diffusion model alignment with self-sampling preference optimization, 2025a. URL
htttss//arxiv.one alos 2410. 05255
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023.
Tao Zhang, Cheng Da, Kun Ding, Huan Yang, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model as noise-aware latent reward model for step-level preference optimization. arXiv preprint wrXh::2502.00051, 2025b.
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, et al. Vhench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.2/755, 2025.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model aligmment. In ICLR, 2025.
14

---

Published as a conference paper at ICLR 2026
A
OVERVIEW OF APPENDIX
The appendix is organized into the following sections:
Section
B
Dataset Construction Details.
Section
More Experiments and Analysis.
Section
Implementation Details.
Section
Results and details of the open-source datasets.
Section
Discussion, Limitation and Future Work.
Section
G
The Use of Large Language Models (LLMs).
B DATASET CONSTRUCTION DETAILS
B.I VIPO-IMAGE-IN 1M DATASET
Image-Text Aligmment. To construct DPO preference pairs (win/loss) for image-text alignment while minimizing impact on other attributes, utilize a single image generation model conditioned on distinct prompts to generate the corresponding image pairs.
Our data construction pipeline begins with sampling images and prompts from the open-source LAION- Aesthetics dataset. We then use Swve2.55-VL-3222 to generate a detailed caption for each image and sub- sequently filter out any samples containing inappropriate content. Following this, we employ Sced-VL-11 to perform image-grounded perturbations on these clean captions. This approach requires the model to first comprehend the image content, ensuring that all modifications are semantically consistent with the visual in- formation. For instance, person-related attributes are only altered if human subjects are present in the image.
Specifically, we modify one, two, or three of these dimensions in the original prompt with probabilities of 70%, 20%, and 10%, respectively. The primary dimensions include: (1) style, (2) rendering, (3) lighting, (4) atmosphere, (5) time, (6) color-scheme, (7) saturation, (8) perspective, (9) depth-of-field, (10) composition, (11) weather, (12) season, (13) location, (14) background, (15) detail-level, (16) texture, (17) mood, (18)
quantity, (19) size, (20) pose, (21) action, (22) interaction, (23) emotion, (24) clothing, and (25) age.
In this setup, the image generated from the original, unperturbed caption serves as the "winner", while the image generated from the perturbed caption designated as the "loser", Based on preliminary experiments where Seedream-3.0 achieved the highest alignment scores on small internal test set, we selected it to generate all 200K image pairs for this task.
Text Rendering, The text prompts used for our text rendering dataset are constructed from three primary sources. The first component consists of 208K prompts from the CoverBook subset of the TextAtlas5M dataset. The second is a collection of 100K prompts from the 'stzhao/movie posters 100k controlnet" dataset on HuggingFace. The third source comprises prompts selected from the LAION-Aesthetics dataset that correspond to images containing visible text; we ensure these samples do not overlap with those used for the aforementioned image-text alignment task when sampling from -sste-Ass
After aggregating these text-centric prompts. we filter them by character count to exclude excessively long or short text strings and perform additional step to remove inappropriate content. This process yields a final set of 200K prompts dedicated to text rendering. To construct the corresponding image pairs for these prompts, we exclusively employ Qwen-Image, HiDream-II. Seedream-3.. 3.00, and FLUX.1-dey as other generative models exhibit inferior text rendering capabilities.
To annotate the preference pairs for the text rendering task, we implement a two-stage evaluation process involving PaddleOCR-3. and Seed-V Mi-d-V--1... First, we use PaddleOCR-3.0 for an initial assessment. If one image in a pair accurately renders the text specified in the prompt while the other contains character-level errors, the former is automatically labeled as the "winner". However, if both images succeed or both fail in rendering the correct text, we proceed to the second stage. In this stage, we employ Seed-VL-11. to perform the comparison. The model determines the winner based on holistic evaluation of several criteria, including the clarity of the rendered text, the precision of character formation, and the degree to which the text's position and shape align with the prompt's description.
15

---

Published as a conference paper at ICLR 2026
wolf-hhnned figure donning heavy, armored ottrre wields forch and sword oninously misry library
Aestheties: Amnored skeleral and dogs under a threatening, cloudy sky.
Aligoment:
stripes holding piece mozzareila with its two hooves, placing
Funnt
Aestheties
Alignment
umi umious
mimus
Text Rendering
Human Quality
Image Composition
Text Rendering: vibrani display the word luminouss spelled out in colorfal Lego bricks.
Human Quality: girl in purpie animal-prinn print
pajamas sifs crozz bog-eegged, touching her haur.
ViPO-Image-IM
Composition: Peoceful beach scene with vibrunt umbreflax, a person umbrefia.
Composition: restroom with green chais, murror, sink zoilet.
Human Quality: close portrait of blonde
woman, focusing on expression
(a) Image- IM dataset visualization
Motlen Quality:
Motion Quality
Visual Quality
Spatial Angunrent: moiks band
Visual Quality: beanly unded wamenr betling
shough menior the moe is worring dark bive
funrticle gue contounly exiers the cohepprk meto fiom
and verue & marrg white vaddin dess
snoly industriit aver with sball the boskgound
Video-Text Alignment
ViPO-Video-300k
Mation Aligament: Pegile despvotely 7ot From
ginst spider closinng Mevi.
(b) Video-300K dataset visualization
Figure 6: ViPO-Image- IM and ViPO-Video-300K dataset visualization.
16

---

Published as a conference paper at ICLR 2026
Human Quality. To construct human-centric DPO dataset, first gathered 100K images from existing open-source datasets. We began by filtering the ProGumeeGoo/syytttiiiee flataset-mmdallle wawaalle--igh-uulllll high-quality- captions dataset on HuggingFuce with Seed-VL-1.5, selecting 44,501 images with exhibited human anatomical flaws. We augmented this dataset with 2,009 images from the dataset on HuggingFace and 56,444 images from the HumanRefiner dataset HuggingFace. This aggregated pool was then filtered for inappropriate content (c.g.. violence or nudity) using Seed-VL-1.5, and finally we randomly sampled 80K images from the filtered pool.
To further diversify our dataset, we generated another 120K images. We used Seed- 1.-e-l to select 120K SaaaannGooosymmhiiiddataaee
new nentanccetrii prompts from the dataset, ensuring they were distinct from those used in the first step, We then prompted a suite of ten differ- ent open-source models to generate around 10K 1024x1024 images for each model (including CogVin View4, ILUX.1-dey HtDrream-1--Ful Hunyuan-DiT, Kolors, Pix SANA1.5 1..11..-.88--111 8B-1024px, SD3.5-Medium, SDXL). In addition, we also deploy Qwen-Image to generate 20K 1024x 1024 images. This resulted in collection of 200K human- images sourced from a wide variety of generative models.
1m-dalle3-high-eee high--uullttycaation
To create the paired preference data, we generated counterpart for each of the 200K images using the Seedream- 3.0 model with the identical prompt. Finally, Seed- VL-155 was employed as an automated judge to assign the final preference labels (i.c., identifying the "winner" and "loser" image in each pair) based on which image rendered the human subject more accurately. This comprehensive pipeline yielded final dataset of 200K unique, high-resolution DPO image pairs.
Image Composition. We constructed our dataset by sourcing 200K unique prompts from two primary HuggingFace datasets: fuckyhate/tex-to-mmeet -orattttom-m--MMMM- and hetrrommlllehhghh-uallyyye middomiidddosreeeeeeee prompts from -oophatteeee-----mm----e--- paired image using the same prompt with a randomly selected model from either Qwen-Image or HiDreemm-evv. For prompts from peteromallethhigh-uuall sugualttymidjouuey-ssree MidJoumey-V7 image and generated its counterpart with Seedream-3.0. 3.0. Acknowledging the subjective and complex nature of evaluating image composition, we employed a multi-VLM voting system for robust preference labeling. Specifically, a panel of three diverse VLMs Qwen2.5-VL-322 1e-n..--VL-32B-nsstrru Beed-VL-1.33 and Q- Insighht basighttwwas was used to judge which image in each pair exhibited superior composition. The final preference was then determined by a majority vote from these three judges.
midiiiigaalyyyyy-doooeeyeeeeee srefs.. For
we generated one image using Seedream-3.0 and a second,
we utilized the original
Aesthetics. To construct our aesthetics DPO dataset, we first sampled 200K prompts and corresponding images from the ensuring there was no overlap with the samples previously used for the other DPO datasets. For each prompt, we generated another image using Seedream-3.0. To establish preference pairs based on aesthetics, we utilized three VL.Ms, i.e., Qwen2.5-VV VIL-33BBInnsteen Seed-VL-115, and I-Inightt-to to judge two images was more aesthetically pleasing. The final preference was then determined by a majority vote.
Sar.GGoynheeticddttaeee 1m-dalle3-higg high-hhugliig quality-captions dataset on HuggingFace,
judge which of the
B.2 VIPO-VIDEO-300
Motion Quality. For our Motion Quality task, we construct all video pairs using an Image-to-Vidde (12V) pipeline ensure that the spatial information between the two videos in each pair remains as consistent as possible. Our data generation process integrates samples from four distinct datasets, all sourced from HuggingFace: (I) We collect 6,763 videos and prompts from the WenhaoWang/SShan Veo3 dataset, originally generated by Veo3, extract the first frame of each, and use Seedance- .0-Pro to synthesize generate initial videos with Hunyuan Video-T2V-13B. extract their first frames, and then use Scedance- 1. 1.0-Lite to create the corresponding pairs. (3) We gather 32K videos from the FastVideo/Wan2.2 2-t..-2--nn11222 10xxxxxx 1280 32k dataset, generated by the Wan Video2.2 TI2V-5B model, extract the first frame and prompt for each, and use Seedance-
720p video pairs (2) We take 11K prompts from SiaggguggBiidd//po-Sooo---e-
to generate the paired videos. (4) We select 50K image-text pairs from the LAION-Aesthetics dataset, augment the prompts with motion details using Veed-V and then generate video pairs using both Seedance- 1.0-Pro and Seedance- 1.0-Lite. After generating all pairs, we use Seed-VL- to score the motion quality of each video, designating the higher-scoring one the 'winner'. We then filter the dataset by removing pairs with the largest and smallest score differences to discard trivial or ambiguous examples, resulting in final dataset of 100K preference pairs for this task.
17

---

Published as a conference paper at ICLR 2026
Visual Quality. To construct the Visual Quality subset of our ViPO-Video- ------idd30O- dataset, we first sample 100K image pairs from ViPO- IPPO-Image-Aa IM. We specifically select samples for which all participating VLMs (Qwen2.5-VL-328) Seed- VLL and Q-Insight) unanimously assigned the same preference label. Subse- quently, for each selected pair, the images are fed into Seed- -1..1- to generate a single motion prompt that semantically suitable for both. This motion prompt then integrated with the shared image description to form the final video generation prompt. Using this prompt and the two source images, we employ Seedance- 1,0-Lite to perform an image-to-video synthesis task, generating the corresponding video preference pair. The preference label for each resulting video pair is directly inherited from its source image pair.
Video- Text Alignment. For the Video-Text Aligmment task, we construct preference data by addressing two key aspects: spatial alignment and temporal alignment. To generate data for spatial aligmment, we first select 50K image-text alignment pairs from HiPO-Image-11 IM, which feature subtle visual differences. We then employ Seed- VL-115 to generate single, common motion prompt suitable for the main subject in both images. Subsequently, Seedance-1 Ciainneee-----icc L0-Lite executes an 12V task for each image using this shared prompt, creating video pairs where preference is determined by the inherited spatial characteristics. For temporal alignment, we select 50K images from the V generate two distinct motion prompts (e.g. "a person running" vs. "a person walking") Seedance- 1.0-Lito then generates two videos from the same source image, each conditioned on one of the different motion prompts. In both scenarios, the winner-loser designation based on the correspondence between a video and its prompt; the video that accurately reflects its conditioning prompt is the winner.
dataset. For each image, we use
C MORE EXPERIMENTS AND ANALYSIS
Detailed Text Rendering Results. A distinctive advantage of our dataset is the significant improvement in text rendering on CVTG-2K benchmark Du et al. (2025). which is a historically challenging task for diffusion models. As shown in Table [5] MD3.5-Medumm' 's average word accuracy improves from 0.4378 to 0.6995 (+59.8%) with the NED score reaching 0.8853. FLUX. 1-dvv demonstrates even more dramatic gains, improving from 0.4878 to 0.6859 in word accuracy despite SFT alone causing degradation (0.2126). These improvements are particularly notable for multi-region text rendering, where SD3.5-Medium achieves 0.6252 accuracy on 5-region text compared the baseline's 0.3933.
Table Quantitative evaluation results of English text rendering on CVTG-2K Du al. 2025
Word Accuracy?
Model
NED
CLIPScore
2 regions
3 regions
4 regions
5 regions
average
SD3.5 Large
0.7293
0.6825
0.6574
0.5940
0.6548
0,8470
0.7797
AnyText
0.0513
0.1739
0.1948
0.2249
0.1804
0,4675
0,7432
TextDiffuser-2
0.5322
0.3255
0.1787
0,0809
0.2326
0.4353
0.6765
RAG-Diffusion
0.4388
0.3316
0.2116
0.1910
0.2648
0.4498
0.7797
3DIS
0.4495
0.3959
0,3880
0.3303
0.3813
0.6505
0.7767
TexiCrafter
0.7628
0.7628
0.7406
0.6977
0.7370
0.8679
0.7868
SD3.5-Mcdium
0.5104
0.4788
0.4197
0.3933
0.4378
0.7325
0.7548
+SFT
0.7474
0.6485
0,5625
0,5027
0,5887
0,8228
0.8107
+SFT & Poly-DPO
0,8188
0.7422
0.6900
0,6252
0.6995
0.8853
0,8287
FLUX.1 [dev]
0.6532
0,5273
0.4491
0.4312
0.4878
0.6727
0.7265
+SFT
0.3530
0.2462
0.1962
0.1459
0.2126
0.4623
0.7303
+SFT & Poly-DPO
0.7733
0.7203
0.6893
0.6169
0.6859
0.8489
0.7939
Supervised Fine-Tuning on ViPO-Image-M. The results presented in Table hhighight the optimal strategy for integrating Supervised Fine- Tuning (SFT) with our Poly-DPO method. All models in this ablation are evaluated on the same 1.200-prompt test set detailed in Section 5.2 We first observe that an initial SFT stage is crucial for achieving the best performance. Applying Poly-DPO directly to the SD1.5 baseline yields only modest improvements, whereas models that first undergo SFT before DPO training demonstrate substantially higher scores across all evaluation metrics.
Furthermore, our experiments reveal that the composition of the SFT dataset is critical. By comparing models trained with SET on the full winner-loser pairs versus only the winner images, we consistently find that the latter approach is superior. This is evidenced by our top-performing model, "+ SFT (Winner Only)
18

---

Published as a conference paper at ICLR 2026
& Poly-DPO," which surpasses all other configurations. This demonstrates that fine-tuning exclusively on high-pre prefeerrfeeecee (winner) data provides a more effective foundation for the subsequent preference alignment with Poly-DPO.
Table 9: Ablation study the integration of Supervised Fine- (SFT) and Poly-DPO for the SD1.5 model. The results demonstrate that an initial SFT stage using only winner images is the optimal strategy to achieve the best performance. We utilize this optimal setting for all experiments in the main paper.
Method
PickScore t
HPSv2.1
Aesthetic
ImageReward
SD1.5
20.89
25.04
5.46
0.1757
+ Poly-DPO
21.51
26.40
5.60
0.6391
+ SFT (Winner-Loser)
21.74
28.75
5.71
0.7671
+ SFT (Winner Only)
21.92
29.00
5.72
0.8355
SFT (Winner-Loser) & Poly-DPO
22.06
29.57
5.76
0.9955
SFT (Winner Only) & Poly-DPO
22.19
29.69
5.78
1.0161
0L Gradient Analysis on of Our Poly-DPO. Figure visualizes how the gradient magnitude (-1-))(1+ap)) of Poly-DPO adapts to different data characteristics through the a parameter, where p=o(zz) represents the model's confidence in preferring the chosen response. The visualization reveals three distinct optimization regimes that directly correspond to our experimental findings. When a >> (blue and purple curves), the gradient is amplified in the region p (0.5,0.8, maintaining substantial parameter updates even for moderately confident predictions. This enhancement proves crucial for noisy datasets like Pick-a-Pic V2, where only 20.79% of samples show consistent preferences across evaluation dimensions- the sustained gradient (approximately 2-3x stronger than standard DPO at 0=6 when a = 8) prevents premature convergence spurious patterns and encourages continued exploration to identify genuine preference signals amidst dimensional conflicts. Conversely, when a << 0 (red and orange curves), the gradient decays more rapidly as confidence increases, actively penalizing overconfident predictions. This mechanism addresses the overconfidence problem in our synthetic dataset experiment, where negative a values enforce faster gradient decay beyond p p00.6, maintaining the model in a "humble" learning state that prevents memorization of superficial patterns. Remarkably, when training on our high-quality ViPO-Image- 1M dataset, the optimal a converges to approximately zero (green curve), where Poly-DPO reduces to standard DPO with linear gradient decay | (1-p)). This convergence serves as an empirical validation of dataset quality- whenn preference labels are reliable and balanced, additional gradient modulation becomes unnecessary, confirming that data quality remains fundamental for successful preference optimization. The visualization also provides practical insights: the optimal a value serves as a diagnostic tool for dataset quality (large positive values suggest noisy labels, negative values indicate oversimplified patters, while a == 0 validates well-balanced data), and explains why different datasets achieve different convergence points. This adaptive gradient mechanism enables Poly-DPO to achieve robust performance across diverse dataset characteristics without requiring dataset-specific algorithmic modifications.
1.0
1.0
Gradient Magnitude logits
0.0
0.8
-1.0
0.6
0.2
6 0: Enhencing the confidence
Reducing the contidence
Standard DPO gradient
0.0
0.0
0
0.4
0.6
0.8
1.0
Probability Confidence (p w>1)
Figure 7: Gradient magnitude of Poly-DPO loss with respect to logits under different values. The gradient (1 adaptively controls learning dynamics based on confidence p. a > 0 enhances gradients for medium-confidence predictions combat noisy labels, a 0 suppresses overconfident predictions to prevent overfitting, while a 0 (standard DPO) proves optimal for high-quality balanced datasets.
19

---

Published as a conference paper at ICLR 2026
Human Evaluation Details To ensure the rigorous quality standards the ViPO dataset, we conducted large-scale evaluation by recruiting 18 annotators. This scale significantly exceeds that of related visual generation works, such as ControlNet (12), thereby offering higher statistical confidence and mitigating individual bias. Figure g details the rater reliability, defined the consistency between an individual's choices and the majority vote consensus. The empirical results highlight exceptional agreement: every rater surpassed 70% accuracy, with 14 out of 18 exceeding 80% (Mean: 87.2%, Median: 97.6%). This distribution confirms that our collected preference labels are stable and trustworthy, Such high inter-rater agreement further evidences that the ViPO tasks well-posed and the instructions effectively minimizing the noise often inherent in subjective visual assessments. Consequently, the derived consensus labels provide robust ground truth for benchmarking. Finally, Figure illustrates the annotation interface; rater IDSs utilized strictly for tracking and resuming management to guarantee fully anonymous evaluation process.
Distribution Per-Rater Accuracy
Mear: 87.2%
Median: 57.8%
Number of Raters
unambiguous,
70% Rater Accuracy (Agreement with Majority Vote)
90%
90%
100%
Distribution of human rater accuracy on our ViPO datasets.
Figure 8:
(Required-
starting. allowed)
Plevoverte for you Gove Bort Ir will cound uith connrratioon mork the saurto and you proggees Prng cumomnationn ny iootedehhe seeviing
eig Jone
Modality
Category
Jumpp Index
Jump
Resumee Last Progress
Image
Video
Category Guidelines (image/ aesthetic)
lihting
Then consider texture, details, and noise centrol.
Finally consider subjeett promiineee and image talonne.
Drompe
In imags, an enigmetic woman with long, wovy brown hair and plercing eyes stores noennly et the camera through her warmly illuiinated sunounding
Option
Option
Image
Image
Bettter Choice
Previous
Submit and Next
Skip
Figure 9: The UI inferface used for our human evaluation.
20

---

Published as a conference paper at ICLR 2026
SD155 & SDXL Experiments on DPG-Bench. Table 10 presents the comparative results on the DPG-Bench benchmark. As shown, our proposed Poly-DPO consistently outperforms existing baselines across both SD1.5 and SDXL backbones, achieving the highest Overall scores of 67.02 and 75.67, respectively. This demonstrates the superior capability of our Poly-DPO in aligning diffusion models with human preferences. regarding the baseline selection, it is worth noting that we report results for Diffusion-KTTT exclusively on SDI.5 and MAPO on SDXL, as their respective official repositories have only released model weights for these specific architectures.
Table 10: Evaluation results on DPG-Bench Hu et al 2024) with the Pick-a-pic V2 training dataset
Model
Paradigm
Global
Entity
Attribute
Relation
Other
Overallt
SDI. Rombach al. Diffisson DPO Karaloyy
2022
Off-Policy
74.63
74.23
75.39
73,49
67.81
63.18
2023
Off Pflicy
71.50
72.53
75.25
73.55
72.84
63.29
Diffusion-KTO Li et al Poly DPO
Oft-Policy
72.45
76.51
78.09
78,08
73.20
66.69
On Policy
73.36
78.15
76.50
75.81
73.42
67.02
SDXI Pulell et all
2023
Ofllcy
83.27
82.43
80.91
86.76
80.41
74.65
Diflusson DPO Karanoy al. 2023
Off-Policy
83.67
83.50
81.89
81,56
81.58
75.12
MAPO]Hong el al.
On Policy
78.22
81.31
80.65
85.35
79.85
73.80
Poly-DPO
Off-Policy
84.03
83.86
83,07
75.67
Inference on Different ViPO Sub-datasets. Table 11 comprehensively evaluates the performance of the SD3.5-Medium model under various fine-tuning strategics, leveraging distinct sub-datasets from ViPO. Initially, the base SD3.5-Medium model serves as our benchmark, demonstrating solid performance across all metrics. The subsequent rows clearly illustrate the significant benefits of Supervised Fine-Tunine Tuming (SFT) using individual ViPO sub-datasets. For instance, SFT the "Aesthetics" dataset noticeably improves DeQA and DPG-Bench scores, while SFT on "Text Rendering" leads to a substantial jump in CVTG-2K. This initial phase highlights the high quality and specificity of ViPO sub-datasets, targeted training on specific aspects like aesthetics or text rendering yields immediate and measurable improvements in their corresponding evaluation metrics.
A crucial observation is the inherent overlap among these diverse datasets. For example, datasets primarily focused on "Aesthetics" or "Alignment" inevitably contain elements pertaining to "Human Quality" and "Text Rendering." Consequently, fine-tuning on a seemingly specific dataset can still positively influence other, indirectly related metrics. This is evident in several SFT rows, where improvements are not strictly confined to the explicitly targeted metric. When SFT is applied to "All Datasets," we observe more generalized enhancement, albeit with some trade-offs, indicating the complexity of balancing multiple objectives through SFT alone.
The most compelling results emerge from the combination of SFT (on "All Datasets") followed by DPO using individual ViPO sub-datasets. This two-stage approach consistently achieves superior performance across all evaluation metrics, significantly surpassing both the base model and models trained with SFT alone. Notably, the "All Datasets" DPO fine-tuning achieves the highest scores across most metrics, including a remarkable 85,25 for GPT-4o Accuracy and 0.6995 for CVTG-2K, representing a substantial leap from the SFT-only and base models. This profound improvement underscores two key points: first, the high quality and preference-rich nature of our ViPO datasets are exceptionally well-suited for preference learning: and second, DPO effectively hamesses this high-quality preference data to further refine the model's capabilities, leading to more robust and human-alignee outputs across various dimensions like aesthetics, alignment, text rendering, and overall human quality. The consistent gains across different DPO fine-tuning setups further validate the effectiveness of our comprehensive training methodology and the superior leaming signals provided by the ViPO dataset.
Inference on Different SFT Training Steps. Table 12 presents an ablation study on the number of training steps during the SFT phase, ranging from 1,000 to 4,000 steps. As observed, extending the training duration yields a continuous and significant improvement in complex capabiliitess such as Text Rendering (CVTG-2K) and Human Quality (GPT-40 Acc), with the latter increasing from 73.25 to a peak of 77.50. While some metrics like Alignment (DPG-Bench) saturate or slightly fluctuate after early stages, the steady gains in text rendering (reaching 0.5887) and overall human preference indicate that the model requires more training steps fully absorb the fine- fine-grained details present in our high-quality dataset. Consequently, we select the 4,000-step checkpoint for subsequent stages, as it offers the most robust foundation for generating high-fidelity, human- images.
21

---

Published as a conference paper at ICLR 2026
Table 11: SD3.5-Medium performance on various metrics after fine-tuning with different ViPO sub-datasets using SFT and DPO, demonstrating the impact of specific and comprehensive data training.
Aesthetics DeQA 1
Alignment DPG-Bench 1
Text Rendering
Human Quality GPT-4o Acc
Composition GenEval
Method
Dataset
CVTG-2K t
SDA.S-Mediiu
4.27
84,24
0.4378
73.25
0.69
SFT
Aesthetics
4.32
87.02
0.5051
76.91
0.78
SFT
Alignment
4.30
86.63
0,4904
76.89
0.77
SIT
Composition
4.30
86.57
0,4815
76.52
0.78
SFT
Human Quality
4.29
87.05
0,5174
77.42
0.79
SFT
Text Rendering
4.25
85.85
0,5319
74.45
0.76
SFT
All Datascts
4.31
84.24
0,5887
77,50
0,80
SFT (AIl) + DPO
Aesthetics
4.31
86.91
0,5668
82.32
0.79
SFT (All) + DPO
Alignment
4.31
88.55
0.6680
82.14
0.79
SFT (All) + DPO
Composition
4.31
86,41
0.6190
81.78
0,80
SFT (All) + DPO
Human Quality
4.30
86.70
0.5729
83.02
0.81
SFT (AI)) DPO
Text Rendering
4.28
86.13
0.6344
80.18
0.79
SFT (AIl) DPO
All Datascts
4.31
87.71
0.6995
85.25
0.83
Table 12: Ablation study on the effect of training steps during the Supervised Fine- Fine-Tuning (SFT) stage.
Aesthetics DeQA f
Alignment DPG-Bench
Text Rendering
Human Quality
Composition GenEval f
Method
CVTG-2K t
GPT-4o Ace f
SD3.5-Medium
4.27
84.24
0.4378
73.25
0.69
+ SFT 1000 Steps
4.28
86.84
0.5134
73.98
0.79
+ SFT 2000 Steps
4.31
86.72
0.5334
75.16
0.81
+ SFT 3000 Steps
4.30
86.27
0.5614
76.34
0.81
+ SFT 4000 Steps
4.31
84.24
0.5887
77.50
0.80
Training Stability for Diffusion-DPO and Poly-DPO. To address concerns regarding potential model collapse, we visualize the training dynamics of both the baseline Diffusion-DPO and our proposed Poly-DPO. As illustrated in Figure I0 track four key evaluation metrics- Piciics-siPcSrrrr,,, ImageReward, Aesthetic Score, and HPSv2 -troouhoou the training process on the Pick-a-Pic V2 dataset. Contrary to the instability often associated with on-policy RL methods, both off-policy approaches demonstrate remarkable stability. The reward scores exhibit consistent, steady increase followed by a smooth plateau, indicating stable convergence process with no signs of sudden performance degradation or collapse. Notably, Poly-DPO maintains the robust stability inherent to the DPO framework while achieving a higher performance ceiling than the baseline.
217
0.00
212
0.75
65-
214
0.77
0403
211
010
3.0
42
0200
180 aho A00 NO0 Wot 1200 201 2018 WOR 30t8 Vil 100) 200 A/0 2300 UIT 100: 1850 Tarr
10h:
Figure 10: Training dynamics of Poly-DPO and Diffusion DPO on the Pick-a-Pic V2 dataset. Both methods exhibit high training stability, with evaluation metrics steadily increasing to convergence without any signs of model collapse.
D IMPLEMENTATION DETAILS
Training on Pick-a-pic V2 Dataset. We validate our proposed Poly-DPO method by training the SD1.5 model on the Pick-a-pic V2 dataset. Our training implementation and hyperparameters are based on the official open-source code of Diffusion-DPO. Specifically, we use batch size of 512 and base learning rate of 4e-9 (the final learning rate is 512 x 4e 9= 2.048e -6), 6), the training resolution is 512x512. We perform a grid search for the hyperparameter of Poly-DPO over the set -0.5, 0, 0.5, 1, 2, 4, 6. 8, 10} and find that a = yields the best results. addition, we observed that the original Diffusion-DPO algorithm converges in approximately 8,000 steps, whereas Poly-DPO method achieves convergence in
22

---

Published as a conference paper at ICLR 2026
4,500 steps. Throughout the training process, we do not update the reference model or use the Exponential Moving Average (EMA).
1M Dataset. For our experiments on the ViPO-Image- IM training set, first conduct validation on the SD1.5 model. Based on our conclusions in Section 9 we adopt a two-stage training process for all models. First, we perform SFT using only the winner images. Following this, apply Poly-DPO training. For this second stage, it is important to note that both the policy model being trained and the reference model are initialized from the checkpoint of the SFT-tuned model. We found that there was no significant difference in the evaluation indicators when a was in the range of [-1, 1], for both the SD1.5 and the SDXL model, so we simply set = for all experiments. The training resolution is 512 x 512 for SDI.5 and 1024 x 1024 for other models. No reference model update or EMA is used for all experiments. The specific implementation details for each model architecture are as follows:
Training
SD1.5. We use a batch size of 512 for both stages. The base learning rate is 4c-9 for SFT and Ic-9 for Poly-DPO, with both stages trained for 8,000 steps. We observed that after the initial SFT, a smaller value for / in Equation 5 was better, so we set 3= 500.
SDXL. The batch size is 512. The base learning rates are 2c-9 for SFT and 5c-10 for Poly-DPO, with both stages trained for 4,000 steps. We use B == 1000 for this model.
SD3.5-Medium. For the SFT stage, we use a batch size of 2048 and a base learning rate of 1e-8. For the Poly-DPO stage, the batch size is 512 with a base learning rate of 5c-9. The SFT stage is trained for 4,000 steps and the Poly-DPO stage for 2,000 steps, with 500
FLUUX.1-dev. For the SFT stage, the batch size is 2048 with a base learning rate of 1e-9. For the Poly-DPO stage, the batch size is 512 with a base learning rate of 5c-9, Similar to SD3.5-Medium, SFT is trained for 4,000 steps and Poly-DPO for 2,000 steps, using 500.
Training HIPO-Video-000K Dataset. We conduct experiments by applying Poly-DPO directly to the W base model, using the for 2,000 steps with batch size of 256 and a base learning rate of 1e-8. For this experiment, DPO hyperparameter 3= 550 and the Poly-DPO hyperparamettr a = 0. During training, we utilize a dynamic resolution approach and do not perform any resizing operations the videos in the dataset. This means we consistently train on video data with its original 16:9 and 1:1 aspect ratios, For final evaluation, the VBench2.0 score is calculated by averaging the results from both the 16:9 and 1:1 generated videos.
dataset for training. The model is trained
set the
E OPEN-SOURCE DATASET CONSTRUCTION
E. MOTIVATION
As discussed in Section 4 the original ViPO dataset leverages several proprietary generative models, including Seedream-3.0 for image generation and Seedance- 1.0 for video generation. Due licensing and intellectual property constraints associated with these proprietary models, we are unable to release the data generated by them. However, we believe that open access high-quality visual preference data is essential for advancing research in this area. To this end, we construct and release an alternative version of the ViPO dataset that replaces all proprietary model outputs while preserving the dataset's 's scale, categorical structure, and preference quality.
Importantly, this replacement limited in scope. The majority of the ViPO dataset is constructed using outputs from open-source generative models (e.g., FLUX. 1-dev, HiDream-11, Qwen-Image Image, Wan Video) and prompts sourced from publicly available datasets on HuggingFace. These portions of the dataset remain unchanged in the released version. Only the subsets originally generated by Seedream-3.0 (for images) and Seedance- 1.0 (for videos) are substituted, with FLUX.2-dev [abs (2025) and Wan2.2-A1 A--1144-- 14V-I12 Wan et al. (2025) serving as their respective replacements. Both replacement models are publicly available and demonstrate strong generation capabilities comparable to their proprictary counterparts. We name the released datasets HiPO-Image-IM-Oppp and ViPO-Vide--00KK-00 respectively.
E.2 ANNOTATION PIPELINE
To ensure consistent and reliable preference labels across the released dataset, we adopt a unified Mul voting pipeline for all annotations, covering both image and video preference pairs. Specifically,
23

---

Published as a conference paper at ICLR 2026
each candidate pair is independently evaluated by three Vision-Language Models: Qwen3-VL VL-30B-AA3 Thinking Bai et al. (2025). Molmo2-8B Clark et al. (2026), and Seed- 1.8 SSed (2025). Each VLM is prompted with category-specie
instructions (e.g., assessing aesthetic quality, text-image alignment, or motion quality) and independently selects preferred sample from each pair. The final preference label is determined by majority vote among the three judges. This multi-VI.n voting strategy mitigates individual model biases and produces robust, reproducible annotations without relying on proprietary labeling services.
E.3 EXPERIMENTAL VALIDATION
To verify that the released dataset maintains the quality and effectiveness of the original version, re-run the main experiments reported in the paper. Specifically, we train SD1.5, SDXL., SDD,,-MMddium, and FLUX.I-der on the released ViPO-Image- IM-Open, by Poly-DPO) described in Section D
IPPP--ma----- following the same two-stage pipeline (SFT followed
Table 13: Evaluation results on GenEval Ghosh et al. (023) with different training datasets.
Training
Single Object
Two Object
Attribute Binding
Model
Dataset
Counting
Colors
Position
Overall
SDI.5
0.96
0.38
0.38
0.75
0.04
0.05
0,42
ViPO-IIagg-IIM
0.99
0.49
0.78
0.06
0.09
0.46
+ SFT & Poly-DPO
ViPO-Inage-IM
0.98
0.66
0.50
0.84
0.07
0.17
0.54
ViPO-Image- 1M-Open
0.77
0.08
STT & Poly-DPO
VIPO Image IM 1M-n
0.98
0.57
0.63
0.78
0.08
0.17
0.53
SDXL,
0.98
0.75
0,44
0.90
0.11
0,16
0.56
SFT
ViPO-Inag-II ViPO-Image-11
0.98
0.77
0.43
0.88
0.13
0.21
0.37
Poly-DFO
SST
VIPO Image 1M-0per Open
0.99
0.73
0.58
0.83
0.13
0.28
0.59
Poly-DPO
ViPO-Inage- I IM-0pen
0.37
SD3.5 SDD.S-Medium Medium
1.00
0.87
0.68
0.80
0.20
0.57
0.69
ViPO-Image-IM ViPO-Imags-Ie IM
0.7
091
0.43
0.77
SFT Poly-DPO
1.00
0.97
0.75
0.91
0.47
0.86
0.83
-FT
ViPO-Inage- MM-Open
1.00
0.97
0.69
0.90
0.42
0.72
0.78
SFT Puly-DFO
ViPO-Image IM Open
1.00
0.99
0.76
0.90
0:45
0.71
0.81
FLUX.I [Dev]
0.86
0.78
0.45
SFT
ViPO-Image-Ih IM ViPO-Innge-I0
1.00
0.90
0.74
0.87
0.38
0.62
0:75
SFT Poly-DPO
0.99
0.97
0.85
0.40
0.70
0.79
SFT
ViPO-Image- 1M--------ee
0.99
0.89
0.80
0.83
0.29
0.67
0.74
SIT PoO
VIPO Image IM Open
1.00
0.92
0.79
0.85
0.29
0.68
0.76
Table 14: Evaluation results on DPG-Bench Hu 2024) with different training datasets.
Model
Training Dataset
Global
Entity
Attribute
Relation
Other
Overall f
SD3.5-Meding
89.49
92.21
85.12
84.24
+SFT
ViPO-Image- 1M
84,80
89.97
14
93.69
82.00
84.24
+SFT & Poly-DPO
ViPO-Imuee-1 1M
84,80
92.64
90,
94.81
89.20
87.71
+FT
MPO-Image-IM--O IM-Open
92.06
93.60
91.56
91.13
88.68
87.57
SFT Poly-DPO
IPOOImaae-INee IM-Open
94.47
94.08
93.19
91.86
93.29
89.85
FLUX.I [Dev]
74.35
90.00
88.96
90.87
88.33
83.84
+SFT
VIPO-Inage- IM
85.41
89,21
85.17
92.72
80.40
83.59
+FT & PoO
VIPO IPO-eeage-/ 1M
90.99
91.05
90.91
93.73
91.12
87.31
SFT
VPPO-Image1MM Maaagge----ppp 1M-Open
92.21
91.44
89, .47
91.66
89.51
85.41
SPT Poly-DPO
1o0-Image-IM-Oppe
91,94
92.12
91.91
90.44
86.99
The results confirm that all core conclusions of the paper hold with the released dataset: (1) SFT on winner images consistently improves base model performance across all architectures and evaluation dimensions; (2) Poly-DPO further yields significant gains on top of SFT, demonstrating the effectiveness of preference optimization; and (3) the overall performance trends across benchmarks remain consistent with those reported using the original dataset. These findings validate that the released ViPO dataset is faithful and effective substitute, suitable for reproducing our results and supporting future research in visual preference optimization.
24

---

Published as a conference paper at ICLR 2026
Table 15: Quantitative evaluation results of English text rendering on CVTG-2K Du et al. (2025) with our ViPO-Image- IM and ViPO-Image- IM-Open training datasets.
Training
Word Accuracy
Model
NED
CLIPScore!
Dutaset
2 regions
3 regions
4 regions
5 regions
iveriiee
SD3.5-Medium
0.5104
0.4788
0.4197
0,3933
0.4378
0.7325
0.7548
+SFT
SIO-Image IM
0.7474
0.6485
0.5625
0.5027
0,5887
0.8228
0.8107
+SFT & Poly-DPO
ViPO-Inage- IM
0.8188
0.7422
0.6900
0.6252
0,6995
0.8853
0,8287
+SFT
ViPO-Inage- IM- Open
0.6863
0.6070
0.5978
0.5599
0.5995
0,8372
0,805
+SFT & DPO
ViPO Image 1M Open
0.7324
0,7013
0.6798
0.6370
0.6791
0.8944
0,8189
F I [dev]
0.6532
0.5273
0.4491
0.4312
0,4878
0.6727
0.7265
+SFT
ViPO Image- 1M
0.3530
0,2462
0.1962
0.1459
0.2126
0.4623
0.7303
+SFT & Poly-DPO
ViPO-Image- IM
0.7733
0.7203
0.6893
0.6169
0.6859
0.8489
0.7939
+SFT
VIPO-InageIIM-C
0.5551
0,5164
0.4271
0.3715
0,4491
0,6787
0,7499
+SPT & Poly-DPO
ViPO-Image-IM-- IM-Open
0.6327
0,5603
0,5139
0.4231
0.5134
0.7427
0.7656
Table 16: Wan2.1-T2V-1.33 Experiments on VBench-2.00 when trained with different datascts.
Models
Human Identity
Material
Thermoties
Dynamic Spotial Rel.
Dynamic Attribute
Motion Order Und.
Human Interaction
Camseaa Motion
Motion Rationality
Wan2.1
62.18
69.75
72.26
24.61
53.48
35.35
74.00
31.79
43.68
-iPO-Video-000
67.99
71.57
68,53
33,82
$7.00
38.62
78.00
32.49
47,70
-PP0-Nidoo-300--Oeem
68.42
31.26
54.34
39.19
31,87
F DISCUSSION, LIMITATION AND FUTURE WORK.
Discussion. Our work presents dual contribution to scaling visual preference optimization: the Poly-DPO algorithm and the high-quality ViPO dataset. The most significant finding is the symbiotic relationship between algorithmic design and data quality. Our experiments demonstrate that while robust algorithm like Poly-DPO is critical for extracting meaningful signals from noisy, rerl-world datasets such as Pick-a-Pic V2, the need for such sophisticated algorithmic adjustments diminishes as data quality improves. The convergence of the optimal Poly-DPO hyperparameter a to zero when training on our ViPO dataset serves powerful empirical validation of ViPO's quality and balance.
This suggests that the hyperparameter a can itself serve as a valuable diagnostic tool for assessing preference dataset characteristics. A large positive optimal a may indicate significant noise or conflicting preference signals, whereas a negative optimal a could suggest the dataset is dominated by trivially simple patterns leading to model overconfidence. An optimal o near zero, as observed with ViPO, indicates well- bell-balanced and reliable dataset where standard optimization is sufficient.
Furthermore, our construction of the ViPO dataset highlights a scalable paradigm for future data curation efforts. By leveraging a suite of state-of-the-ar generative models and a panel of powerful Vision Language Models (VLMs) for automated filtering. generation, and labeling, we demonstrate a pipeline that largely bypasses the immense cost and scalability issues of collecting human preferences directly. This Al-driven approach is fundamental to achieving preference optimization "at scale."
Limitation. Despite the promising results, our work has several limitations. First, the preference labels in the ViPO dataset are generated exclusively by AI models (VLMs). While we used multiple state-of-the-ar VLMs to ensure robustness and consistency, these Al-generated labels are a proxy for, not direct measurement of, true human preferences. We did not conduct large-scale study to measure the correlation between our VLM--ssigned labels and those from human annotators, and the inherent biases of the judge VLMs may be encoded in our dataset.
Second, while Poly-DPO's effectiveness is demonstrated across datasets with different characteristics, the optimal value for the hyperparameter a was determined via grid search. This process can be computationally intensive, and the ideal may depend on factors beyond data noise, such as the base model architecture or the specific domain of the content. A more automated or dynamic method for setting would improve the method's practicality.
Finally, the creation of the ViPO dataset itself required significant computational resources, involving gener- ation from over dozen ofate-of-the-ar models. While our work helps democratize the use of nigh-quality preference data through its public release, the initial construction of such datasets remains a costly endeavor, potentially limiting the ability of smaller research groups create similar resources for new domains.
25

---

Published as a conference paper at ICLR 2026
Future Work. Based on findings and limitations, we propose several avenues for future research. critical next step is to conduct a large-scale human validation study of the ViPO dataset and explore more robust pseudo-labeling, with better reward models Chen et al. (223)) and better generative models like Seedance- 1.5-Pro Seedance et al. 2025). Comparing the VL-M-generatd labels against human judgments would not only quantify the quality of ViPO but also provide valuable insights into developing next-generation judge VLMs that are even better aligned with human values.
Another promising direction is the automation of the hyperparameter in Poly-DPO, Future work could explore methods to make a a learnable parameter that is dynamically adjusted during training based on batch statistics or the model's evolving confidence distribution. This would create truly self-adaptive preference optimization algorithm.
The categorized structure of the ViPO dataset opens up possibilities for more fine-grained and controllable preference optimization. Future research could investigate methods for explicitly modeling the trade-offs between different quality dimensions (e.g., prioritizing "Text Rendering" over "Aesthetics"), potentially leading more personalized and instruction-guided visual generation. Lastly, we believe the Al-driven curation pipeline itself can be extended, both to new modalities like 3D and audio, and into an iterative, self-improving loop where models trained on ViPO are used to generate new data that, after being filtered by judge VLMs, used to further refine the dataset.
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
All technical content, dataset design, experimental results, and analyses presented this paper were produced by the authors. Large Language Models (LLMs), such as GPT and Gemini, served only tool for language polishing and enhancing readability; they were not used to generate any of the core ideas, data, experimental results.
ETHICS STATEMENT
This work develops preference optimization methods and datasets for visual generation models. All experiments were conducted using publicly available models and datasets, with newly generated synthetic data created from text prompts publicly available image datasets. Our ViPO dataset construction involved Al-generated content from state-of-the-art models (FLUX, Qwen-Image for text-to-image; Video for image-to-video using LAION-Aesthete images). While we use publicly available datasets that may contain human images, we follow established practices for responsible use of such data. We recognize that visual generation models can potentially be misused for creating misleading or harmful content. To mitigate these risks, we emphasize responsible use guidelines, transparent documentation of our methods, and acknowledge that generated content should be clearly labeled as Al-created. While our work aims to improve generation quality and alignment with human preferences, we encourage ongoing research into detection methods and ethical deployment practices for generative Al systems.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. A comprehensive description of our dataset construction, including the entire collection and processing pipeline for our proposed ViPO datasets, provided in Section B All implementation details, including models, training hyperparameters for each experiment, and the evaluation setup, thoroughly documented in Section [ We believe these resources provide all the necessary components for the community to reproduce our results and build upon our work.
26


Reviews:
[{"review_id": "zeTprahu6j", "reviewer": "Reviewer_Dmfu", "summary": "The paper presents several contributions for diffusion DPO, including large-scale datasets of 1M image pairs and 300K video pairs, and Poly-DPO as an extended algorithm to standard DPO. The new datasets are to address limitations of existing datasets, e.g. low resolution, and conflicting winning patterns. Methods trained on the new datasets got substantial improvements. The Poly-DPO is obtained by taking DPO as binary classification, and Taylor expansion of the standard cross-entropy loss. In the experiments, Poly-DPO outperforms Diffusion-DPO. It can also reveal problems of a dataset for being noisy or over-simple.", "questions": "- How to demonstrate that ViPO doesn't have the problem of conflicting winning patterns?\n- How to verify if ViPO is aligned with human preference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The large-scale preference datasets are valuable to the community. \n- The Poly-DOP improves the standard Diffusion DPO.\n- Experiments are comprehensive and convincing.", "weaknesses": "- Poly-DPO is considered as an incremental change to DPO\n- Preference pairs are judged by one or more VLMs. It’s not clear about the quality of annotation by aligning with human preference. To improve, authors can conduct cross checks with human raters on a small portion to verify the alignment.\n- Minor writing issues\n  - Line 075: learning -> learn\n  - Line 086: SD15 -> SD1.5", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4kVcBqYFIW", "reviewer": "Reviewer_Cn5U", "summary": "This work proposes ViPO, visual preference optimization at scale. It includes two contributions: (a) Poly-DPO which proposes including a polynomial term alongside the standard diffusion loss to dynamically weight preferences in the face of uncertainty in the preference datasets, (b) ViPO dataset, a massive-scale preference dataset with 1M image (1024resolution) pairs across five categories  and 300K video pairs (720P+ resolution) across three categories. Numerical experiments have been done to evaluate the proposed method on preference optimization compared to the baselines.", "questions": "- New insights regarding existing datasets is not that new. i.e., These datasets also suffer from low resolution (512-768resolution), limited prompt diversity, outdated generation models. Many existing works have demonstrated the same thing. For instance, see RankDPO.\n- Why not evaluate SDXL and SD1.5 on DPG-Bench benchmark like in Tab. 5? Why only show the GenEval scores?\n- What’s the recommended value of alpha in poly-loss? How does one tune this hyper-parameter for a new preference dataset?\n- For the proposed new dataset, since the recommended value of alpha=0 ==> its standard diffusion DPO. Does it mean that Poly-Loss does not help at all? Because the illustrated examples in which alpha!=0, are very old datasets and would no longer be used due to their outdated generation models and low-res images.\n- In Tab.5, why does SFT not improve SD3.5-medium and Flux.1 Dev performance? Technically, the ViPO-1M image dataset is better in quality. \n- How robust is the proposed Poly-DPO is to model collapse which is observed very frequently with DPO formulation with longer training?\n\nMissing References:\n- DSPO — Direct Score Preference Optimization for Diffusion Model Alignment https://openreview.net/forum?id=xyfb9HHvMe \n- RankDPO — Scalable Ranked Preference Optimization for Text-to-Image Generation:  https://arxiv.org/abs/2410.18013  \n- Flow-GRPO— Training Flow Matching Models via Online RL : https://arxiv.org/pdf/2505.05470 \n- Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization https://arxiv.org/abs/2410.05255", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Large scale preference optimization dataset for image and video generation \n- Evaluation on a range of image and video generation models to show the efficacy of the proposed method", "weaknesses": "- It is unclear what’s the impact of Polynomial component in Poly-DPO on a decent DPO formulation. Most of the datasets referred in the paper are quite old and would not be used by any modern SOTA T2I models for preference optimization. Besides the alpha becomes yet another hyper-parameter in the DPO formulation which is already quite prone to collapse\n- One concerning issue with the VIPO dataset is that SFT with this dataset does not seem to improve models like SD3.5-medium and FLUX.1 dev on DPG bench which is harder benchmark compared to GenEval. This could point to biases in the dataset construction. \n- Claims the new insights regarding existing datasets (low resolution images, outdated models, conflicting win/lose pairs) as a contribution, but this has been observed earlier in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kPv98SujU1", "reviewer": "Reviewer_kHDq", "summary": "This paper introduces a two-pronged approach to advance the scaling of visual preference optimization. The first contribution is Poly-DPO, a novel algorithm that modifies the standard DPO loss with an adaptive polynomial term, enabling more robust learning from datasets with noisy or conflicting preference labels. The second contribution is ViPO, a new large-scale, high-quality dataset of 1 million image pairs and 300,000 video pairs, designed to provide a more reliable and balanced preference signal than existing resources. A key finding is the synergy between these contributions: Poly-DPO significantly outperforms baselines on noisy datasets, but gracefully converges to standard DPO on the high-quality ViPO dataset.", "questions": "1. The paper would be even more impactful if it included a small-scale human validation study to measure the correlation between the VLM-generated labels and actual human judgments. By the way, one of related reference papers can be cited (X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models).\n2. A discussion on how dataset statistics (like the conflict rate mentioned for Pick-a-Pic) could inform the initial search range for `α` would significantly enhance the method's insight for other researchers.\n3. The categorical construction of the ViPO dataset is a major strength. Have you performed any analysis on the relative importance of these different categories (Aesthetics, Composition, etc.)? For instance, do you find that training on certain categories, like Text-Image Alignment, contributes disproportionately to the overall improvements seen on general-purpose benchmarks like GenEval?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper presents a significant and compelling contribution to the field of visual preference optimization, marked by its dual focus on both algorithmic innovation and high-quality data curation. My evaluation of this work is highly positive, and its primary strengths are as follows:\n\n*   **Elegant and Effective Algorithm Design:** The proposed **Poly-DPO** is a standout contribution. It is exceptionally well-motivated, directly addressing the critical and practical challenge of learning from datasets with noisy and conflicting preference signals. The design is both simple and powerful—by augmenting the standard DPO loss with a single, intuitive polynomial term, it adaptively re-weights samples based on model confidence. This elegant modification requires minimal code changes yet yields substantial performance gains on noisy datasets, demonstrating a high degree of effectiveness and practical utility.\n\n*   **A Landmark Dataset Contribution:** The introduction of the **ViPO** dataset is a massive contribution to the community. Beyond its impressive scale (1M image and 300K video pairs), its true strength lies in its thoughtful, categorical construction. By organizing preferences into distinct dimensions such as Aesthetics, Text-Image Alignment, and Composition, the authors ensure the dataset is balanced, reliable, and capable of driving comprehensive improvements across a wide range of model capabilities. The data generation strategies are both clever and pragmatic—for instance, creating image-text alignment pairs through prompt perturbation and using OCR to generate robust preference labels for text rendering. This meticulous approach sets a new standard for preference dataset quality.\n\n*   **Comprehensive and Rigorous Experimental Validation:** The paper's claims are substantiated by an extensive and exceptionally well-designed set of experiments.\n    *   The **ablation studies** (particularly Figure 4) are cleverly constructed. They not only provide convincing evidence that the \"noisy preference\" problem is a real and significant obstacle in existing datasets but also clearly demonstrate Poly-DPO's ability to mitigate this issue effectively.\n    *   The experiments systematically validate each component of the work. The authors wisely use public benchmarks like the Pick-a-Pic dataset and standard models like SD1.5 to transparently demonstrate the standalone effectiveness of the Poly-DPO algorithm. Subsequently, they showcase the powerful synergy of their algorithm and new dataset, demonstrating state-of-the-art results and significant improvements across a diverse array of modern generative models, including SDXL, FLUX.1, and Wan2.1. This thorough validation leaves little doubt about the efficacy of the proposed methods.", "weaknesses": "Despite the paper's numerous strengths and high-quality contributions, there are a few areas that could be further clarified or represent inherent limitations of the chosen approach. These points are offered to help refine what is already an excellent piece of work:\n\n*   **Reliance on AI-Generated Labels for the ViPO Dataset:** The primary limitation is that the ViPO dataset's preference labels are generated exclusively by AI models (e.g., VLMs). While this AI-driven pipeline is a key strength for achieving unprecedented scale, it introduces a fundamental trade-off. \n\n*   **Practicality of Hyperparameter Tuning for Poly-DPO:** While the Poly-DPO algorithm is simple in its formulation, its effectiveness on a new dataset relies on finding an optimal value for the hyperparameter `α` through a grid search. This process can be computationally intensive and presents a practical barrier to easy adoption. The authors correctly identify the automation of `α` as a direction for future work.\n\n*   **Minor Typos and Notational Inconsistencies:** The paper contains several minor but noticeable typographical errors that should be corrected for the final camera-ready version. These do not detract from the core scientific contributions but do affect the overall polish and clarity. For instance:\n    *   There is an inconsistency in the naming of the dataset, appearing as both \"**ViPO**\" and \"**VIPO**\" in the abstract and elsewhere.\n    *   In the introduction, \"SD1.5\" is incorrectly written as \"**SD15**\" (line 86).\n    A thorough proofread to catch these small errors would improve the professionalism of the final manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)