Towards Better Optimization For Listwise Preference in Diffusion Models
Authors: Jiamu Bai
Xin Yu
Meilong Xu
Weitao Lu
Xin Pan
Kiwan Maeng
Daniel Kifer
Jian Wang
Yu Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-image generation
Diffusion Model Alignment
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We recover listwise human preferences from image annotations and introduce Diffusion-LPO to directly optimize such listwise rankings for diffusion models.

Abstract: Reinforcement learning from human feedback (RLHF) has proven effectiveness for aligning text-to-image (T2I) diffusion models with human preferences. Although Direct Preference Optimization (DPO) is widely adopted for its computational efficiency and avoidance of explicit reward modeling, its applications to diffusion models have primarily relied on pairwise preferences. The precise optimization of listwise preferences remains largely unaddressed. In practice, human feedback on image preferences often contains implicit ranked information, which conveys more precise human preferences than pairwise comparisons. In this work, we propose Diffusion-LPO, a simple and effective framework for Listwise Preference Optimization in diffusion models with listwise data. Given a caption, we aggregate user feedback into a ranked list of images and derive a listwise extension of the DPO objective under the Plackett–Luce model. Diffusion-LPO enforces consistency across the entire ranking by encouraging each sample to be preferred over all of its lower-ranked alternatives.  We empirically demonstrate the effectiveness of Diffusion-LPO across various tasks, including text-to-image generation, image editing, and personalized preference alignment. Diffusion-LPO consistently outperforms pairwise DPO baselines on visual quality and preference alignment.

Published as a conference paper at ICLR 2026
TOWARDS BETTER OPTIMIZATION FOR LISTWISE PREFERENCE IN DIFFUSION MODELS
Jiamu Bai" Xin Yu", Meilong Xur, Weitao Lu", Xin Pan',
Kiwan Maeng', Daniel Kifer', Jian Wang', Yu Wang?
Penn State University
TikTok Inc.
Stooy Brook University
Figure I: Sample images generated from SDXL trained with Diffusion-LPO, Diffusion-LPO gener- alizes Diffusion-DPO by optimizing the preference under list of ranked images. After finetuning with Diffusion-LPO, SDXL produces images with higher visual aesthetics and prompt alignments.
ABSTRACT
Reinforcement learning from human feedback (RLHF) has proven effectiveness for aligning text-to-image (T2I) diffusion models with human preferences. Al- though Direct Preference Optimization (DPO) is widely adopted for its compu- tational efficiency and avoidance of explicit reward modeling, its applications diffusion models have primarily relied on pairwise preferences. The precise opti- mization listwise preferences remains largely unaddressed. In practice, human feedback on image preferences often contains implicit ranked information, which
Co-first author. Work done during Jiamu Bai's internship at TikTok. |: jvb6867, xm25152, kvm6242, 1uk17)@ppp edu 2: Iu, xin. pan, jian.wangl, yuwang. w) com 3:me i xuêcs. atonybrook.....

---

Published as a conference paper at ICLR 2026
conveys more precise human preferences than pairwise comparisons. In this work, we propose Diffusion-LPOO, a simple and effective framework for Listwise Pref- erence Optimization in diffusion models with listwise data. Given caption, we aggregate user feedback into a ranked list of images and derive a listwise exten- sion of the DPO objective under the Plackett-Luce model. Diffusion-LPO en- forces consistency across the entire ranking by encouraging each sample to be preferred over all of its lower-ranked alternatives. We empirically demonstrate the effectiveness of Diffusion-LPP LPO across various tasks, including sext-to-image generation, image editing, and personalized preference alignment. Diffusion-LIF consistently outperforms pairwise DPO baselines visual quality and preference alignment.
INTRODUCTION
Diffusion models, including prominent architectures like Stable Diffusion (Podell et al. 2023 Rom- bach et al. 2022 Imagen (Saharia et al., 2022), and DALL-E 2 (Ramesh et al., 2022),, have become a dominant paradigm in text-to-image synthesis due to their ability to generate high fidelity and se- mantically aligned images (Ho at. (020). While these models pretrained massive web-scale datasets to establish a powerful foundation, real-world deployment necessitates further finetuning to align their outputs more closely with human preferences. Drawing inspiration from alignment tech- niques in large language models (LLMs), recent works have adapted methods like Direct Preference Optimization (DPÓ) for diffusion models, enabling preference learning without an explicit reward model Yang et al. 2024, Wallace et al. 2024, Zhu et al. 2025 Li et al.) 2024 Gu et al. 2024). These approaches have demonstrated superior performance over standard supervised fine-tuning by directly incorporating preference signals into the optimization process.
Despite these advances, prior works mostly rely on pairwise human preference data. Pair- wise comparisons are relatively easy to collect and annotate, making them the dominant form in existing human preference datasets [Bai al. 2022 Kirstain et al., [023). However, human preferences are inherently expressed as ranked lists rather than isolated pairs, since
Figure 2:
An example of a ranked list of images under human preference. The caption is "Rocket Raccoon, furry art, fanart, digital painting." "
preferences
simply absolute approvals or rejections but relative orderings among multiple
options, as shown in the example of Figure In this sense, pairwise annotations implicitly contain richer ranking information: we consider a user indicating their preferences as response x(a) >
(6) and (b) > x(c), this naturally implies the transitive ordering x(a) > (b) > x(o), which more faithfully represents the user's overall preference. Notably, observe that in the Pick-a-Pic dataset (Kirstain et al., 2023). where each user provides pairwise preferences for images, 56% of such annotations can be benefited from grouping into consistent rankings with size greater than 2 to form more informative human preferences. This motivates the need for listwise optimization, which directly models the ranking nature of human preferences. A straightforward approach formulating an objective for listwise preference is to decompose rankings of size m into m (m 1)/2 pairs to apply the pairwise DPO objective. Several works (Chen et al., 2025 Karthik et al. 2024 have
explored this direction, but their methods rely on auxiliary evaluators to assign reward scores to each image, introducing extra computational overhead and resource burden.
In this work, we introduce Diffusion-LPPP (Listwise Preference Optimization), new framework for aligning diffusion models with user-level preferences. Instead of reducing user feedback pairwise comparisons, Diffusion-LPP directly models listwise preferences with the Plackett- Luce ranking model (Plackett 1975). Unlike pairwise DPO, which only compares a winning sample against a single loser, Diffusion-LPO enforces consistency across the entire ranking by encouraging each sample to be preferred over all of its lower-ranked alternatives, thereby preserving the full relative order within the list. When the list size equals 2, Diffusion-LPO degenerates Diffusion-DPO As a general approach for listwise preference optimization. the formulation of Diffusion- can potentially enhance many existing methods built upon the Diffusion-DPO family.
2

---

Published as a conference paper at ICLR 2026
To evaluate the effectiveness of our method, train SD1.5 (Rombach al. 2022) and SDXL (Podell et al.] 2023) on the Pick-a-Pic dataset (Kirstain et al.) 2023), a dataset of real-world user preferences that contains human preference pairs of images. We uncover the inherent listwise preference within the dataset, with detailed information of dataset reformulation illustrated in Sec- tion 4.1 Empirical results show that Diffusion-LPOO improves PickScore win rates by 12% and by 4% over Diffusion-DDP on SD1.5 and SDXL. Specifically, our contribution can be listed as follows:
We first provide a novel perspective that ranked human preferences are implicitly embedded within pairwise annotations. motivates us intentionally enforce the diffusion model to learn such structural rank information.
We propose Diffusion-LPO, a direct preference optimization framework from listwise hu- man feedback. As a generalization of pairwise DPO, Diffusion-LPO can integrated into any DPO-based method to further boost the performance.
We empirically demonstrate the effectiveness of Diffusion-LPPO across various T2I tasks. Diffusion-LPPC exhibits higher alignment with human preferences over baselines.
2
RELATED WORK
RLHF Alignment Reinforcement Learning from Human Feedback (RLHF) has emerged as the dominant paradigm for aligning large language models (LLMs) with human preferences (Chris- tiano et al. 2017 Stiennon et al. 2020: Mnih et al. 2016; Ziegler et al. 2019: Bai et al. 2022 Bakker et al. 2022 Jiang et al., 2025p. In its standard formulation, reward model is trained to approximate user preferences, and the policy is subsequently optimized via reinforcement learning with this reward signal. While effective, this two-stage pipeline is computationally expensive and can be susceptible to reward hacking (Singhal et al., 2023 Chen et al. 2024). To address these limitations, recent work has explored direct preference optimization (DPO) (Rafailov et al., 2023). which bypasses explicit reward modeling and instead optimizes the policy to prefer human- preferred outputs directly. Most existing approaches rely on pairwise preference data and the Bradley-Terry (BT) model (Bradley & Terry, 1952) to parameterize the likelihood of a winning sample over a los- ing one (Ethayarajh et al. 2024; Meng et al. 2024). However, pairwise comparisons may provide limited information about human preferences, as they reduce complex judgments into binary wins and losses. Several recent works have extended DPO to the listwise ranking setting (Liu et al.) 2025b, Song et al., 2024 Hong et al. 2024a), enabling richer supervision signals from ranked lists of responses than pairs.
Recent work has shown that reinforcement learn- ing is an effective tool for aligning diffusion models with human aesthetic and semantic prefer-
Preference Alignment with Diffusion Models
ences (Fan et al. 2023 Black et al.
2023
Xue al. 2025 Clark et al.
2023 ZZhoo et al.
2025
Ue-
hara et al. 2024 Prabhudesai et al.
2023 Lee et al. 2023 Eyring et al. 2024 Sun et al. 2025a Lee
al. 2025 Zhang et al. (025).. A Among existing reinforcement learning optimization approaches,
Direct Preference Optimization (DPO) stands out as an attractive option by defining an implicit
reward through preference comparisons, eliminating the need for explicit reward models (Wallace
ct a 2024: Yang et al. [024: Li et al., 2024 Gu et al. 2024; [Zhu et al. 2025 Hong et al.] 2024b
Lee et al. [2255: Wang et al. 2025 Liang al.] 2024| Sun et al. 2025b Lu et al.) 2025] Wu al. 2025 Hu et al. 2025; Zhang et al. (024). Diffusion- DPO (Wallace et al. 2024
2025 Cai et al.
adapts this framework to the diffusion setting. Follow-up work includes DSPO (Zhu et al. 2025
which derives the DPO objective through score matching, Diffusion-KTO (Li et al. 2024 which formulates the objective by maximizing human utility, MaPO (Hong et al. 2024b) which maxi- mizes likelihood margin between preferred and dispreferred image sets. Despite these advances, the above DPO-based methods for diffusion rely exclusively on pairwise preference data, while human feedback can come a list of rankings.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
3.1 DIFFUSION MODELS
Diffusion models are generative models that learn data distributions by reversing gradual noising process (Ho et al. 2020; Song et al., (021). The forward (diffusion) process starts from clean data To ~ Pústas where Pdatu is the data distribution, and progressively adds Gaussian noise across time steps with noise scheduling B1,..., ST:
(x+ xo) = (VVorxo (1 o.))) Or II O2s
Be.
The reverse process (denoising) is parameterized by a neural network eg. which predicts noise er given a noisy image x, at timestep t. The standard training objective is a weighted denoising score matching loss (Ho et al. 2020):
(G)
E x;
(1 ppo (1mm) po (i) ||c co 0)
[le
where A [log(a]/o]] is the signal-to-noise ratio, pe is the reverse process, and w(x) pre- defined weighting function.
3.2 DIRECT PREFERENCE OPTIMIZATION (DPO)
Large generative models can be aligned with human feedback via reinforcement learning from hu- man feedback (RLHF) (Christiano et al. 2017 Stiennon et al. (020). The goal of RLHF is to optimize the policy my by maximizing the reward r(c, Xo) given prompt e sampled from prompt set D. and sample Xo, with regularization KL-divergence from reference policy Ter (xolc):
max Be [r(c, xo) 31Dxr. [/][//((X//]/))]
Classical RLHF requires training separate reward model and then optimizing the generative policy with reinforcement learning (e.g., PPO (Schulman et al. 2017)). Direct Preference Optimization (DPO) (Rafailov et al., 2023) eliminates the need for explicit reward modeling by modeling human preferences with the Bradley-Terry (BT) model (Bradley & Terry 1952):
PBT (x) > x c) = (r(c,
r(c, x))
where e is the conditioning prompts and (x xxx is the pair of human-labeled winning and losing samples. Here, Xo) is the latent reward, and is the sigmoid function. Use the log- likeliiold ratios to get the implicit rewards, the DPO loss becomes
(ex x IND [og (3(log (xxx (c) [x c)
log To (x lc) Ter (c) )]
(0)
Here, (c,xx xx ~ D indicates ~ De. xt (x(xx)l)) xo co(x
3.3 DIFFUSION-DPO
The key observation of adapting DPO to diffusion models is that the diffusion denoising objective fDM serves implicit reward. Specifically, Diffusion-DPO (Wallace al., 2024) defines reward for an image x at timestep as the improvement of the fine-tuned model eg over the reference model Gets specifically re(c,
dg(c, Xx, t), where
8g(c, x., 1) = - (lee eg (x,, c, 0)11 ||e Ecot
-re- (xe, (x,, c)))
The resulting preference objective is
LDur-DPO (0) -B B x (BTgoBTTWi (6o x;" xx,"t) (x(c,x,,))))
where T' is the number of diffusion steps and w reweights timestep contributions.

---

Published as a conference paper at ICLR 2026
4 METHOD
4.1 LISTWISE PREFERENCE OPTIMIZATION FOR DIFFUSION MODELS
Existing preference alignment methods for diffusion models, such as Diffusion-DPO. are built on the Bradley-Tere (BT) model, which assumes pairwise preferences. In this work, we adopt listwise preference optimization, x(1) x(2)
> x(m), which leverages the ranking as more fine- grained information for preferences. To capture this higher-order structure, we adopt the Plackett- Luce (PI.) model (Plackett 1975), a probabilistic model for listwise rankings.
x xm))] denote a group of m candidate images generated under prompt c, ranked by a user as x(1) x(2) x(m), The PL. model defines the probability of this ranking as
Reward Under Plackett-Luce Model. Let G = (x(1), x(2)
exp(r(c, x(1))
(x > x
x m)) cc) = II
Ei= (x,xprcc,
where r(c, x) denotes the latent reward the image x conditioned on the prompt c. This formula- tion can be viewed as applying softmax over each suffix sublist x(0) x(++1) x(m), ensuring that every step the selected item is assigned higher preference than all remaining lower-ranked candidates.
Objective for Diffusion-LPO We would like finetune the diffusion model pe (x,(cc with the RLHF objective as in Equation [ For diffusion models, KL-divergence regularizes the whole dif- fusion chain, and the reward is defined as r(c, Xo) = Epo(xir (x),)) Ri(c, Xo.T)], where (XI,... Xr) are variables defined on the diffusion path. Maximizing the reward from the objective of Plackett- Luce model, we can get the listwise Diffusion DPO objective:
Diifusiion (0) E
(0
( DD
(1am) "iT"
(1:m) ippo (x 1x6 (1:m))
m
3 log
Po
0.T
(5)
log
m exp 3 log
Po
(A) D-T
(3)
1=1 Pref X
k=j
Prof O.T
Let co(,, and (ror(, c, () be the learned and reference denoisers. Using the standard Diffusion-
DPO surrogate (x 0:r c) ~ Tw(Ae) So(c, x() (), where (c, Xx, 1) is defined in Equation 2
the listwise objective becomes
DDiffusionLLool (0) (0)== = E (1mm)) (1om) ppp (x (1 (11m))
m
BT BTw(A)) da(e, xf log Σ exp (0TT(A))ooc,x .00)
m
(4)
Σ
k=j
The proof of deriving Equation from the RLHF objective (Equation] D can be found in Appendix B.T As listwise optimization is a more general form of pairwise, it is possible to extend Diffusion- LPO with other pairwise DPO methods. We use DSPO (Zhu al., [2025) as an example and provide the derivation and relevant discussion in Appendix
E We observe improvement when using list- wise preferences DSPO compared with original pairwise DSPO, with detailed results shown in Appendix E.2
Besides Diffusion-LPO we also found similar existing works that also aim optimize at the list level. We include discussion of the comparison between our method and theirs in Appendix B.2
Constructing Listwise Groups. The Pick-a-Pic dataset provides user preferences in pairwise form, e.g., X, > X₁ and X₁ > x. under the same prompt c. To construct a more faithful repre- sentation of human feedback. we form list of preferences from the above pairwise preferences as X. > X₁ > X.. We conduct aggregation as directed acyclic graphs (DAG) preferences and extract listwise paths, which we treat ranking sublists. We provide statistics of transforming the pick-a- a- pic dataset into listwise in Appendix A An example of the preference group can be found in Figure
5

---

Published as a conference paper at ICLR 2026
4.2 ADVANTAGE OF LISTWISE PREFERENCE MODELING
Given a list of preferences, prior work on pairwise optimization (Wallace et al. 2024) can be ex- tended naively. One can find the optimal policy model by minimizing the objective below, where we refer to it as Group Pairwise DPO(GP-DPO):
LGP-DrO -E (c.xx JoD, (1:m)
(1:m)) (1:m)
Σ
logo (
(5)
180
15)ckmm
Such an objective is also derived in Chen et al. (2025. Conducting pairwise optimization in group
of size m will introduce mmm - 1)/2 pairs for comparison with equal importance. For our proposed
method, denote the score $ ( pa p (r)e)
(3) Par Purlee which is the part of reward R(c, x related to policy
model up to scalar 3. In objective (4. when treating x(6) as the positive sample and optimizing over
its set of negative samples (x(++1), ... x(m)), yields the following reward difference
PDiffusion-I ( (x(1) (k) Vk E
.m)) x B log 8g () log(
(8)
)o)
(6)
Besides, find for Group Pairwise DPO with objective (5 the reward difference when treating x() as the positive sample and computing with all the preference pairs
x( (0), wk U......... .m]) log
log((s())) ()
k=
The detailed analysis can be found in Appendix B.3 GP-DPO and Diffusion-LPO derive different
reward functions in terms of human preference of x(5) over the negative samples. Compare their rewards for negative samples, we can find
Σ kmf-1
1og((s(0))++ (1000)) m --j1
Σ
max log((s(0)p) (00)) Rehes
-- j+1
kmff11
(8)
S log Σ
(8
From the above analysis, we observe that GP-DPO leads to an underestimate of the aggregate reward of negative samples, since it reduces the ranking into equally weighted pairwise comparisons as shown in (7) In particular, the reward assigned to negatives under GP-DPO is upper-hounded by the corresponding term in Diffusion-LPO,, shown in (8). This underestimation inflates the margin between the current positive sample x(i) and its negatives. In contrast, Diffusion-LPo directly normalizes over the entire negative group as shown in (6) which ensures higher-ranked samples are favored over the lower-ranked ones.
EXPERIMENT
5.1 EXPERIMENT SETUP
Models, Datasets, and Evaluations. We finetune two text-to-image diffusion backbones: Stable Diffusion 1.5 (SD1.5) Rombach al. (022) and SDXL (Podell 2023). Training data drawn from the Pick-a-Be Fick-a-Pic vl preference dataset (Kirstain et al. 2023). ) and our listwise data construction follows the procedure described in Appendix[A] In our Diffusion-LPO implementation, the extracted preference lists have variable lengths. and we cap the maximum list length at 8, so Diffusion-LLPO trained on lists of size from to 8: when the list size is 2, the objective reduces to the standard pairwise Diffusion-DPO loss. We evaluate models on: (1) General Text-to-Image Alignment: We Bick-a-Pic test dataset (Kirstain et al., 2023), Parti- Parti-Prompts Yu et al. 2022), HPSV2 test dataset (Wu et al.) 2023). (2) Image Editing: We use two image editing datasets, InstructPix2Pix (Brooks et al. 2023 and ImgEdit benchmark (Ye et al.) 2025) for evaluation. (3) Personalized Preference Generation: Testing whether models adapt to
use the prompts from three datasets for evaluation:

---

Published as a conference paper at ICLR 2026
individual user preference embeddings. We train the SD1.5 model with the pipeline in PPD DDng jt al., 2025) and evaluate the personalization performance on the Pick-a-Pic test dataset. Compre- hensive details of the evaluation pipeline and prompts given in Appendix C.I Implementation Computation overhead discussion is in Appendix
details are in Appendix C.2
Baselines. We consider baselines of: SFT (supervised fine-tuming on preferred images). pairwise Diffusion-DPO (Wallace et al.) 2024), and DSPO (Zhu et al. 2025), along with the original pre- trained SD1.5 and SDXL. Besides finetuning on these baselines, we also include Diffusion-DPO SD1.5, and SDXL tuned on the Pick-a-Pic v2 preference dataset, which has almost twice the data size as Pick-a-Pic v1. We directly use the released checkpoint for evaluation. Notice that we only consider baselines that do not require additional reward models or evaluators for fair comparison.
Evaluation. For text-to-ine -mage alignment, we adopt five widely used automatic evaluators: PickScore (Kirstain al. (023). HPSV2 (Wu et al. (2023). CLIP score (Radford al.) (021), Image Reward (Xu et al. 2023). and Aesthetic Score (AES) (Schuhmann) 2022). For image-editing task, we adopt three metrics for the InstructPix2Pix dataset: LI, CLIP, and DINO. L1 calculates the pixel-level absolute difference between the generated image and ground truth target image. CLIP and DINO reflect the edited image quality by measuring the cosine similarity of the CLIP (Radford ct al. 202T) and DINO (Caron et al. (021) embeddings between the generated images and ground truth target images. We use the GPT4o (Hurst et al. (024) to evaluate the images generated under ImgEdit benchmark, following its suggested evaluation pipeline. For personalized preference gen- eration, where no standard automatic metric exists, we follow recent evaluation practice in (Dang let al. 2025) and rely on GPT-4o (Hurst al., 2024) judge, prompting it to rate which generated image better aligns with a user's 's historical preferences.
5.2 GENERAL EVALUATION: TEXT-TO-IMAGE ALIGNMENT
Table I: Winrate results over original SD1.5 and SDXL for Diffusion-LPO compared with other baselines. For abbreviation, we use "PS", "IM", and "AES" to represent PickScore, Image Reward, and Aesthetics score. "Diff." is short for "Diffusion". Note: best results trained under Pick-a- Pic v1 (Kirstain et al., 2023) are in boldface. Diffusion-DPO* is trained under Pick-a-Pic v2 dataset, with almost twice the data size than v1. We directly use the checkpoint trained by (Wal- lace et al. 2024). All results averaged over generations from 5 random seeds. t: higher values indicate better performance.
Stable Diffusion 1.5
Stable Diffusion XI.
Dataset
Method
PS
HPS f CLIP
IMf
AEST
PSt
HPS f CLIP
IM 1
AES f
SFT
73.4%
80.7%
56.3%
74.9%
71.7%
22.5%
39.3%
46.6%
37.0%
24.3%
Diff.-DPO* *
73.3%
69.8%
57.1%
61.7%
63.4%
77.1%
78.4%
66.0%
70.0%
50.8%
Pick-a-Pic
Diff--DPO
68.2%
70.7%
54.8%
65.2%
60.1%
73.0%
81.1%
64.5%
66.4%
47.1%
DSPO
73.1%
82.6%
57.8%
75.2%
70.9%
59.2%
77.4%
56.6%
62.9%
45.9%
Di
80.4%
74.6%
58.5%
68.1%
64.0%
77.1% %
87.1% 1
64.0%
73.7%
51.5%
SFT
65.0%
79.0%
54.2%
69.7%
72.3%
24.2%
39.6%
47.9%
40.9%
32.0%
67.3%
64.9%
53.7%
60.2%
62.3%
70.0%
80.5%
64.6%
74.0%
58.6%
Parti-Prompts
Diff.-DPO
59.7%
64.0%
52.9%
61.6%
58.9%
66.8%
77.7%
56.0%
68.7%
56.0%
DSPO
63.8%
78.2%
54.3%
70.3%
71.1%
55.5%
78.1%
54.2%
65.5%
55.4%
Diff.-LPO
71.9%
69.7%
57.4%
65.9%
62.5%
72.8%
82.9%
60.1 %
73.6%
59.7%
SFT
75.1%
85.0%
56.4%
77.7%
72.9%
20.4%
40.1%
47.8%
42.3%
25.9%
Diff--DPO*
76.5%
71.0%
57.4%
64.4%
67.2%
72.5%
79.2%
59.3%
70.4%
55.5%
HPSV2
Diff.-DPO
69.5%
70.9%
51.5%
65.8%
62.0%
72.1%
80.0%
59.2%
70.8%
47.7%
DSPO
74.4%
85.3%
56.8%
77.4%
71.6%
58.4%
77.3%
47.8%
63.7%
48.2%
DiO
82.9%
76.6%
57.7%
69.0%
65.1%
74.6%
85.0%
56.3%
72.3%
51.9%
We report text-to-image generation results in Table Diffusion-LPO achieves substantial gains over its pairwise baselines and even surpasses Diffuusio -PO trained on Pick-a-Pic v2, on most metrics. On SD1.5, Diffusion-LPO shows clear improvements over pairwise Diffusion-DPO. In particular, the PickScore increases by more than 12% across all evaluation sets, indicating more reliable alignments with human preferences. On SDXI., Diffusion-L.PO consistently outperforms all
7

---

Published as a conference paper at ICLR 2026
baselines. Notably, Diffusion-LPO improves the PickScore win rate harti-prompts with 6% and HPS score win rate on HPSV2 with 5% compared with its pairwise baseline, Diffusion-DPO While SFT achieves competitive results SD1.5, its performance drops significantly SDXL, with win rates falling below 50%. This degradation is likely due to a mismatch between the relatively lower quality of training data and the higher intrinsic quality of SDXL. generations. In contrast, Diffusion- LPO exploits relative ranking information effectively, enabling the model to better capture human preferences and outperform its pairwise baseline, Diffusion-DPO.
Figure I qualitatively illustrates these improvements. Diffusion-LPO demonstrates stronger capa- bility handle fine-grained structures. For the last column, although all methods produce visually appealing images of the dolphin mid-jump, Diffusion LPO generates dolphin tails with smoother curvature and more accurate shape, highlighting its superiority in capturing detailed features. More qualitative illustrations of Diffusion-L.PO over other baselines are shown in Appendix D.1 We also report additional results of training DiT family diffusion models Appendix F.T. We fine-tune the SD3.5 SD3.5-Medhe 5-3eium (Esser et al. 2024) model on Diffusion-DPO and Diffsson- -LlfsionnLLO... Diffusion-LPO consistently outperforms Diffusion- DDtuuo demonstrating the effectiveness our method on both traditional U-Net backbones and modern DiT-based diffusion architectures.
penguin wearing mask beside canal Venice
A cat space suit walking on the moon
dolphin doing
Sunflower
spectacular jump in a river
SDXL
Diff.- DPO
DSPO
Diff.-LPO
Figure 3: Images generated from original SDXL, Diffusion-DDD DPOO, DSPO, and Diffusion-LPo Diffusion-LPO demonstrates improved image generation quality over other baselines regarding gen- eral aesthetics and detail handling. The last column indicates the zoom-in parts.
5.3 IMAGE EDITING
further evaluate Diffusion-LPOO in the context of instruction-based image editing. Following prior work, adopt the InstructPix2Pix dataset (Brooks et al.] 2023). which provides paired source images, natural language edit instructions, and target outputs. We perform editing with Stable Diffu- sion 1.5, comparing the original model and its tuned variants. For editing, apply SDEdit (Meng et al. 202T) with a noise strength of 0.6. Diffusion-LPO achieves notable improvements over its pairwise counterpart, Diffusion-DPO,, with win-rate gains of 4.3% DINO score, 3.6% CLIP
8

---

Published as a conference paper at ICLR 2026
score, and 3.8% L1 distance respectively. We also adopt recent image editing bench- mark Ye et al. (002). which includes both single-turn and multi-turn image editing instructions, and GPT4o evaluate the image editing result. Diffusion-LPO achieves winrate of 56.3% compared to images generated from Diffusion-DPO. These results indicate that listwise supervision provides stronger alignment signals for following editing instructions. Qualitative results shown in Figure 8 in Appendix D.2
User Profile
SD 1.5
SD 1.5+Profile
PPD
PPD-LPO
This user prefers images that ... capture the beauty nature and technology coexisting, such as trains traveling through scenic landscapes, evoke feelings of adventure and progress...
cinematic still of highly reflective stainless steel train the desert, sunset
The preferred images share common traits such as high visual quality, detailed textures... and immersive scenes. They often feature majestic dragons in natural landscapes, evoking sense of wonder and fantasy....
majestic vicious dragon habitat landscape, boss fight scene, refractions realistic, cinematic lighting, antview, natural, horrific atmosphere, sharp focus
The user prefers images that have high visual quality, with detailed and dynamic elements, vibrant colors, ... The user values depth, lighting, and color saturation in their preferred images....
swirling water tornados epic fantasy
The preferred images consistently exhibit more detailed character design, richer colors, and engaging backgrounds... dispreferred images lack these qualities, appearing flatter, less detailed...
paladin in full plate armor, render, realistic, photorealism
Figure 4: Images generated under Diffusion-L-PO with personal preference alignment with other baselines. User profiles are summarized by VL... "SD 1.5+Profile" represents images generated using SD 1.5 with user profile appended to the caption. We highlight the user preferences in green and dispreferences red.
5.4 PERSONALIZED PREFERENCE ALIGNMENT
Beyond general text-to-image alignment, we further evaluate Diffusion-I-PO under the personalized preference setting. Following the pipeline of Dang al. (2025), condition the diffusion model on user embeddings extracted by vision-language model that summarizes each user's historical preferences (see Appendix C.3 for details).
We use SD1.5 as the backbone and evaluate it on the Pick-a-Pic test set, which is partitioned into held-in users (seen during training) and held-out users (unseen during training). As baseline, adopt Personalized Preference Diffusion (PPD) (Dang et al., (025), which aligns user embeddings with diffusion models using the Diffusion-DPO objective. We then replace the pairwise loss with

---

Published as a conference paper at ICLR 2026
listwise formulation, denoted as PPD-LPO. Incorporating Diffusion-LPO into the personaliza- tion pipeline yields consistent gains: the win rate improves from 71.1% to 72.3% on held-in users and from 70.3% to 80.2% on held-out users, demonstrating that listwise optimization both enhances personalization for known users and substantially improves generalization to unseen users. Qual- itative analysis is shown in Figure 4 While the original PPD improves alignment relative to the untuned SD1.5 model, PPD-LPO further enhances both visual appeal and fine-grained detail.
6 ABLATION
6.1 RANK ENFORCEMENT FROM DIFFUSION-LPO
To further assess the effectiveness of Diffusion-LPPe we compare it against Group-Pairwise DPO (GP-DPO), with the objective in Equation to finetune Stable Diffusion 1.5. Both objectives are trained under the same objective. For each prompt, we calculate the scores of images generated by each evaluator. Table 2 reports, for each evaluator, the win rate of Diffusion-LPO over GP-DPO, where the win rate is defined as the percentage of prompts on which images from Diffusion-LPOO receive higher scores than those from GP-DPO. Since Diffusion-LPO outperforms GP-DPO most metrics and evaluators, this indicates that explicitly optimizing the model on listwise human prefer- ence data leads better alignment quality, consistent with theoretical analysis in Section 4.2 We provide the corresponding significance tests in Appendix [22 For all metrics except CLIP, the improvement of Diffusion-LPO over GP-DPO is statistically significant.
Table 2: Winrate for Diffusion-LPO on SD1.5 in comparison to SD1.5 trained under GP-DPO.
Dataset
PSt
HPS t
CLIP 1
IM
AES
Pick-a-Pic
52.3%++0002
50.6%++.0114 51.3%=0.010
50.8%+0.022
53.7%=0.011
HPSv2 Parti-Prompts
52.4%%11.001
50.6%.10.018 54.0%=20.015
51.3%10.00s
49.1%.10.0m
52.4%::0..012
53.0%=2.017
50.8%11.012
51.4%:10.012
52.1%:10.010
6.2 DIFFERENT MAXIMUM LIST SIZE
To assess the influence of the maximum list length, we compare performance under maximum list sizes of 4, 8, and 12 Table 3 The result shows that, increasing maximum list size from 4 to 8 brings 1% of overall improvement, while increasing list size from 8 to 12 yields only marginal differences. This suggests that list size 8 for Diffusion-LPPO in our setting is sufficiently large, which also consistent with the empirical distribution of data, where approximately 95% of preference lists have length at most 8.
Table 3: Winrate for Diffusion-LPo on SD1.5 vs SD1.5, under different maximum list size. Results reported Pick-a-Pic dataset.
Max List Size
Avgt
PS
HPS
CLIP 1
IM t
AES
m 68.2%
79.4%
73.0%
58.5%
68.5%
61.6%
m 8
69.1%
80.4%
74.6%
58.5%
68.1%
64.0%
m 69.3%
79.9%
74.7%
60.5%
69.2%
62.1%
CONCLUSION
In this work, we introduce Diffusion-LPO., novel framework for aligning text-to-image models us- ing listwise preference optimization. By modeling full preference rankings with the Plackett-Luce model, our approach surpasses pairwise methods like Direct Preference Optimization (DPO). Exper- iments Stable Diffusion 1.5 and SDXL demonstrate that Diffusion-LPPP produces outputs with superior coherence and human preference alignment, highlighting the efficacy of listwise supervi- sion for fine-tuning generative models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint MxXiv:2204005800 2022.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler. Luey Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models find agreement among humans with diverse preferences. Advances in neural information processing systems, 35:38176 76-188666889, 2022.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, arXiv preprint arXiv:2305.1330// 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVV conference on computer vision and pattern recognition, 2023.
Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, and Yunhe Wang. Dspo: Direct semantic preference optimization for real-world image super-resolution. arXiv preprint arXiv:2504. 15/76, 2025.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp.
2021.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In International Conference on Machine Learning, pp. 7935-7952. PMLR, 2024.
Renjie Chen, Wenfeng Lin, Yichen Zhang, Jiangehuan Wei, Boyuan Liu, Chao Feng, Jiao Ran, and Mingyu Guo. Towards seff-improvement of diffusion models via group preference optimization. arXiv preprint 2505: 1070, 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized prefer- ence fine-tuning of diffusion models. In Proceedings of the Computer Vision and Puttern Recog- nition Conference, pp. 8020-8030, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference muchine learning, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems, 37: 17:112588-1255199 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tunnn ainettunig text-to-image diffusion models. Advances in Neural Information Processing Systems, (6679858-798855 2023.
11

---

Published as a conference paper at ICLR 2026
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning dif- fusion models through relative preference optimization. arXiv preprint arXiv:2406.063822 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 681006688111118 2020.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without refer- ence model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-1111 11100-11199, 2024a.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome,, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024b.
Zijing Hu, Fengda Zhang, and Kun Kuang. D-fusion: Direct preference optimization for aligning diffusion models with visually consistent samples. arXiv preprint arXiv:2505.22002 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2 1276, 2024.
Daniel R Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, and Zheqing Zhu. Improving generative ad text on facebook using reinforcement learning. arXiv preprint FrXiv:2507.218833 2025.
Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for text-to-image generation. arXiv preprint arXiv:2410.18011 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-inn sot-to--mmae generation. Advances in neural information processing systems, 36:36652-366666 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023.
Kyungmin Lee, Xiahong Li, Qifei Wang, Junfeng He, Junjie Ke, Ming- Hssan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi- nulti-preference optimization for aligning diffusion models. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 18465-18475, 2025.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. Advances in Neural Information Processing Systems, 37:24897-24925, 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint 2rxXiv2406.043144 2(5):7, 2024.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.054 05540, 2025a.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Moham- mad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through -aannnn In Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers). pp. (2000,220,22 2025b.
Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang. Xiaoyin Xu, and Min Zhang. Inpo: Inver- sion preference optimization with reparametrized ddim for efficient diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2025.
12

---

Published as a conference paper at ICLR 2026
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-1242355 2024.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1929 1937. PmLR, 2016.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):133222 1975.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 015533288-37444 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125, 1(2)33 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36499 2022.
Christoph Schuhmann.
Laion-acsthetics. [liio-aeettheicss 2022. Accessed: 2023 11- 10.
https://aion.ai/bloo
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 170.007777 2017.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.037166, 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang- Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990-18998, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
13

---

Published as a conference paper at ICLR 2026
Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text-to-image generation with preferences through f-divergence minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 220444-2762,,222 2025a.
Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Positive enhanced prefer- ence alignment for text-to-image models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025b.
Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding rein- forcement -easeng-bbssed fine-tuning of diffusion models: A tutorial and review. arXiv preprint MrXiv:2407.13734 2024.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8228-822833 8238, 2024.
Fu- Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, and Hongsheng Li. Diffusion-npor Nega- tive preference optimization for better preference aligned generation of diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Mare Najork. The lambdaloss framework for ranking metric optimization. In Proceedings of the 27th ACM international con- ference information and knowledge munagement, pp. 1-31222 2018
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096-2105, 2023.
Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, and Aliaksandr Siarohin. Densedpo: Fine-grained temporal preference optimization for video diffusion models. arXiv preprint (tx.iiv:55660351177 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 555903-19333 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint ar Xix::2505.078/8 2025.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-88189 1-951,911, 2024.
Yang Yc, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint 2505. 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Xin Yu, Zelin He, Ying Sun, Lingzhou Xue, and Runze Li. Understanding the statistical accuracy- communication trade-off in personalized federated learning with minimax guarantees. In Forty- second International Conference on Machine Learning, 2025.
Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, et al. Seppo: Semi-policy prefer- ence optimization for diffusion alignment. arXiv preprint arXivv 2024.
Lingfan Zhang, Chen Liu, Chengming Xu, Kai Hu, Donghao Luo, Chengjie Wang, Yanwei Fu, and Yuan Yao. When preferences diverge: Aligning diffusion models with minority-awe aware adaptive dpo. arXiv preprint arXiv:2503. 16921, 2025.
14

---

Published as a conference paper at ICLR 2026
Yulai Zhao, Masatoshi Uehara, Gabriele Scalia, Sunyuan Kung, Tommaso Biancalani, Sergey Levine, and Ehsan Hajiramezanali. Adding conditional control to diffusion models with rein- forcement learning. In The Thirteenth International Conference Learning Representations, 2025.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint wr1iv:1999.00000 085990 2019.
15

---

Published as a conference paper at ICLR 2026
A PICK-A-PIC DATASET LISTWISE CONSTRUCTION
We restructure the Pick-a-Pic dataset into user-prompt specific directed acyclic graphs (DAGs) of image rankings. Specifically, we collect all images generated under the same prompt and annotated by the same user, and construct graph where each image corresponds to node. Whenever the user provides pairwise annotation indicating that image X is preferred over image xB, we add directed edge X, - Xg. By aggregating all such edges, transitive structure of the user's annotations. From these DAGs, we can extract consistent listwise rankings of images (e.g. XA Xg xc) whenever sufficient pairwise information is available, thus transforming independent pairwise judgments into richer listwise preference orders.
obtain DAG that encodes the
Table 4: Statistics of Pick-a-Pic after listwise group construction.
List Size
# of list
2
225,656
81,704
4
42,113
22,216
6
14.191
8,953
28
23,087
A
A:
B:
B
D
C:
C
D:
E
Figure 5: An example of a group with preferences that forms a DAG. The arrow pointing from image X to image Xn represents human preference: X > Xp. The prompt is "Rocket Raccoon, furry art, fanart, digital painting"" Here, valid rank list will be (xA > Xc > xx) and (xA > XD xg).
Data usability. Table summarizes the statistics of the reformulated lists. total, we analyze 1,848 preference pairs. Among them, 225,656 pairs (44.09%) form lists of size 2, which naturally reduce to standard pairwise DPO training. Importantly, 274,895 pairs (53.71%) belong to lists of size > 2, making them directly applicable to listwise preference optimization. Only 11,289 pairs (2.20%) fall into inconsistent groups and are discarded. Such inconsistencies correspond to cyclic preferences (e.g., X > XB. XB XC, > XA), which cannot be represented as DAGs and thus reflect low-quality or noisy annotations. The very small fraction of such cases demonstrates that the overwhelming majority of human annotations aggregated into rankings. In our training, lists of size still contribute through Diffusion-DPP while longer lists provide richer preference information for Diffusion-L.PO.
internally consistent and can be reliably
16

---

Published as a conference paper at ICLR 2026
Constructing lists from DAGs. Each group is represented directed acyclic graph (DAG), where nodes correspond to images and edges encode user-indicated pairwise preferences. To trans- enumerate valid topologial paths within each DAG. Each path yields ordered list of images {z(1) > x(2) >
form this structure into training examples for listwise optimization,
2!m) } that consistent with all local pairwise edges. These lists are then used directly in the Plackett-Luce likelihood of our objective. In practice, multiple valid paths may exist within single DAG; use all valid paths for Diffusion-LPO.
B DIFFUSION-LPO OBJECTIVE
B.I DIFFUSION-LPO OBJECTIVE
Recall the RLHF objective:
max Ec [r(c, xo)] - 5L [xa (PPer((o))))) [c]]
(9)
where / is parameter that controls how much po po (xale) deviates from Prer(Xalc).
We define R(c, Xy.r) as the reward on the whole diffusion chain to define the reward r(c, Xo) as rcc, Xo) = Bpxxrr/xx [R(c, Xo.r)]. Given we have
min (xo e)
[ r,, xo) DeL
[c]
P6
E Exo R((c Xo:T) + DKL [xo c)) (Pee(XXo:T c)]
min
log (x
min Enp (x)
Ríc, Xoer)
Prer(Xo:r c)
po (x(xoTl(c)
(10)
min Epp
log Prer Prer(Xo:T c) exp [R(c, Xp:T )/3)
pe (xx.Tlc)
min Epo
log -
Zic (xo. exp [R(c, Xo:T)/3) log Z(c)
min DKL (x0:T (xo:Tl)) exp(II(c, x0r)
where Z(c) = Ex (x. exp (r(c, 3)) is the partition function, and we get rid of log ((c)
it is independent from pa. The optimal p (xorrlc) of Equation (10) has unique closed-form solution:
p (xxT (c) (rer(X) exp(II(c, xx.T) (B.T)/)().
Therefore, we have the reparameterization of the reward function
p (xu:rc)
R(c, Xo.T) Blog
+ 3 log Z(c).
(ror(X).T/c)
Plugging this into the definition of r, we have the following
(x(x) r(c, xo) pep log Pror(Xo.Tio) c)
log Z(c).
Substituting this reward reparameterization into the maximum likelihood objective of the Plackett- Luce model, the partition function cancels out, and we get a maximum likelihood objective defined on diffusion models, for the group (c, x ...., X ("")):
pā (xx T
exp
BE
(x
G)) 1:2
(x) .))
log
Pir (xx
Diilussion LPO (P)())
=log
po (x (c) (6) x c)
1=1
k= exp
BB
(o(x (2
log
17

---

Published as a conference paper at ICLR 2026
By applying Jensen's inequality, can push the expectation out, and with some simplification, can have,
Diffussion (P) = EB
(e.xh )n (umm))opo
(xp0
(Rom) (
Po (x1-7 0.7
(x (R) Po Po
)
3 log
log
exp 3 log
Pref O:
(1)
|c)
k=
Pref O.T
(k)
Since sampling from pe ( Xo: c) is intractable, we utilize the forward process ((x1))|x)) for approximation. As x ((11:T x .c) for I,..., m,
Lappenx (0) = Σ
8
log Po (x(xx-71c
Prer 0:T P
(3)
log
k=j
exp
8 E
x1.7
log
Prer X.
Pe (x(xx:11
0:0 c)
B BE (1)
log
po (x
1-1
c)
Pior (i)
(i)
1==
,c)
(k)
Pe xx 1-1
log
b=j
exp
BE (N)
log
Pret i-1
(k)
xx c)
po X
.)
6 E (3)
T E, log
Pret x
X .c)
log
exp
B
x11
T TE log
Piot (x)
po (2x
1-1
(k)
(1)
X
Xi
(k)
(b)
.c)
m
(x(
(5)
(3) .c
BT E, 0) 1-1
log
Pref X
.c)
j=1
m
Pa (x
x
(b)
1)
log
exp
E (
log
Prot (c)
(k)
X
Po (x(
BT E, X
(1m)
log
(i) (1)
Preff xx
po [x
(k)
X c)
)
log
exp
(TE,
(1)))
x
log
(k)
x Piot
1-1
x
(11)
(i) where xIT q(xIT |
c), xx
q(x, | x and x ()) ~ 9(x:-1 x x
()) for
j I,..., Since the stagewise negative Plackett-Luce term E, {sj - log E Σو>s exp (sx)
is convex in the score vector by Jensen's inequality we can push E, x (l:)) outside and obtain an upper bound:
po (x)
(1) (1)
x c)
Lupprox (0) - E,
3T E
log
1,
[1im)
()
Prer (x( x x c)
po (x
(k)
log k=
exp
3T E
log
x
(x
x
Using the standard reverse- reverse-Gaussinn Gaussian parameterization of the diffusion model, each inner expectation reduces to KL-ddifeerncce that, in turn, yields the denoising-error form with the usual timestep
18

---

Published as a conference paper at ICLR 2026
weight (Xxe):
Lapprox (0) - -E 0,x0
(1:m)
3T whe) -co(e()) ,c, ||e0) - (x)( ,,, e.t)l(?)
log
exp
BT (Te) ( -co ,c, + =||((() - cx((xx .c, c.tt)la))
,c,t)l12
k=j
where e (O,,I), x
Vo
(1) X
-y and X af/o?.
B.2 COMPARISON WITH EXISTING RANK PREFERENCE OPTIMIZATION METHODS.
We found similar existing works that also aim to optimize at the list level. Defining the score as 2 ||e() cx .,, ||e(] (x) c, 1) Chen et al. (025) derives a group pairwise loss (GPO) the form of LGPO 25 [m 2j 1)s("" from the pairwise DPO loss, for (r6) 1æ=1 representing the reward for
and introduces standard rewards Aj each image. The reward scored by some external evaluators. The image reward replaces the term (m 2j 1), and the final objective becomes.
(e)
sui(r)
7
LaPo Σ (A,(Ilo)) co (x c,,) || - (x c, (.
Also, Karthik et al. (2044) proposes RankDPO, which uses lambda loss (Wang et al. 2018) for the list images:
TRankDPO
j.A logo((B((s)) s(())).
1=1 1)]
[g]] represents the weight between where AJ,. ((20()) 1) -1) [og(1++())
image pairs x())), with Φ(i). <(k) and r(), being the true scores and true ranks of (xx),x(* (*)). As the true scores are not available, Karthik et al. (2024) considers evaluation models such HPS. Pick Score. Image Reward, etc., compute the simulated scores for the images.
Different from existing methods, Diffusion-LPO does not require any additional scoring model to learn the ranked preference, as (a) the evaluator can cause extra computation cost, and (b) the auto- mated responses from evaluators cannot replace the human annotation. As the goal is to learn user preferences, evaluator feedback may cause misleading information and contradict the actual human feedback.
B.3 DETAIL COMPARIOSN BETWEEN GP-DPO AND DUFFUSION-LPO
Let's consider the data point ( the preference ranking and its negative samples (x(i+)),... .... xm)) When using the Group Pairwise DPO, the preference of data point x(*) over its negative samples is reflected in the pairwise comparison:
log (x x (k) Vk E (j 1. .m))) = -log (T x(0) r(c, )))
(1mm) For simplicity, we omit the expectation over e and x our analysis. With the notation we po(x(i)-e))
stated before, 80 (a)
we can find
m Σ
log (8 (Blog gl1) B log s0))
Bog s0i) log ((s60)8 (6(3)00)
Am
kmj+1
m log (150)38 + (1(0))0).
x 3 g(3)
m j + k=j+1
19

---

Published as a conference paper at ICLR 2026
Here, by scaling it with the scalar (/ 1), easily check the effect of pairwise preference
optimization over sub-list (x(²) x x(")) in setting.
For Diffusion-LPO in 3). can find that the advantage of x over its negative samples is clearly characterized by a softmax function
m
log Dillussion (P x (6) wk (j 11+1,... 1, .m]) ,.,,,,,))))
=3 BBlog (0) log
Σ
(1log s(6))
exp
k=j
3 log g0) - logg
(8 (k))
Up now, we have explicitly characterized how GP-DPO and Diffusion-LPPO optimize the sub-list preference x,,,,.,,,,)))))))
C EXPERIMENT SETTINGS
C.I EVALUATION DATA DETAILS
We provide the dataset details we used as follows:
Pick-a-Pic dataset Pick-a-Pic is a large-scale, publicly available dataset of human preferences for text-to-image generation (Kirstain et al. (202)). It is collected via web application where users submit prompts, receive images from multiple diffusion backbones, and indicate their preferences between pairs of images. Version (v1) contains over 500,000 examples, while the updated Version 2 (v2) extends this to more than one million. Each entry includes a prompt, two generated images, and a label for the preferred image (or a tie). our experiments, as the Pick-a-Pic v2 is currently not available online, we use Pick-a-Pic vl for training and its held-out test set for evaluation.
The Parti-Prompts dataset Yu al. 2022) introduced to evaluate compo- sitional and semantic capabilities of large-scale text-to-image models. It contains carefully curated prompts spanning diverse categories, such as objects, attributes, styles, and complex multi-object lations. The prompts are designed to systematically probe generation fidelity, compositionality, and visual grounding, making a widely used benchmark for assessing the generalization of diffusion- based models. We adopt Parti-Prompy for standardized evaluation of text-to- -ontteo-mmae generation qual- ity.
Parti-Prompts
HPSV2 The Human Preference Score v2 (HPS v2) dataset (Wu et al. 2023 is built from over 25,000 prompts and 98,000 images generated by Stable Diffusion, accompanied by 25,205 human preference annotations collected from the Stable Foundation Discord community. Each annotation compares multiple candidate images for the same prompt, with the user selecting preferred image. This dataset was used to train a preference classifier fine-tuned from CLIP, which defines the HPS metric- strong predictor of human judgments. We the HPS v2 test set benchmark how well our aligned diffusion models capture human preferences.
InstructPix2Pix InstructPix2Pix (Brooks et al. 2023) is a large-scale dataset for instruction- based image editing. It was synthetically generated by combining GPT-3 for producing editing instructions and Stable Diffusion with Prompt-to-Prompt for creating paired before/after images. The resulting dataset comprises over 450,000 examples of input images, natural language editing instructions, and corresponding edited outputs. Models trained on this dataset can generalize to real-world user-written instructions at inference time. In our evaluation, we use a standardized subset of 1,000 test samples assess performance instruction-uuides image editing.
ImgEdit The ImgEdit-Benne Beeddt dataset (Ye et al. 2025 is comprehensive benchmark for instruction-based image editing. contains 8I1 test cases across sub-tasks, organized into three suites: basic edit suite with nine common operations (add, remove, alter, replace,
20

---

Published as a conference paper at ICLR 2026
style transfer, background change, motion change, hybrid edit, and cut-out), Understand- ing Grounding- (UGE) suite that stresses spatial reasoning and tinegrainedd localization on challenging scenes, and a multi-turn suite that evaluates content understanding, content memory. and version backtracking. Fach test case pairs a real image with natural-language editing instructions, and the benchmark defines multi-dimeesional scores for instruction adherence, editing quality, detail preservation, and fake detection, computed using GPT-4o-based judgments and forensic detector. In our experiments, use only the ImgEdit-Bende evaluation data and follow its official test split and scoring protocol to assess image-editing performance.
C.2 IMPLEMENTATION DETAILS
Hyperparameters. We adopt AdamW for training SD1.5 and Adafactor for training SDXL. The learning rate is set 1 x 10-8 with linear warmup, scaled by effective batch size. The global batch size is 2048: for pairwise methods, the effective batch contains 2048 pairs, and for listwise B 2000 for Diffusion-DPO, Diffusion- LPO, and 3 = 0.001 for DSPO and DSPO-LPO. For SDXL, we set 5 = 5000 for Diffusion-DPo Diffusion-LPo and 3 3000 for DSPO and DSPO-LPO, following previous work.
methods, contains 2048 groups. For SD1.5,
C.3 PERSONALIZED PREFERENCE IMAGE GENERATION
We introduce the pipeline of our personalized preference alignment tasks here. The setting is adapted from Dang et al. 2025).
Personalized Preference Alignment PPD is framework designed to align diffusion models with individual-level human preferences. Personalization would lead to better generalization per- formance across different individuals (Yu et al., 2025). The pipeline operates in two stages. In the first stage, vision- model (VLM) summarizes each user's historical pairwise preference data into dense user embedding. This embedding captures stylistic and semantic signals specific to the user, derived from only few examples. In the second stage, the diffusion model is fine-tuned with these user embeddings as additional conditioning, injected via cross-attention layers. Train- ing uses variant of the Diffusion-DPO objective, which optimizes the model to generate images aligned with each user's preferences while maintaining regularization against a reference model.
User Embedding Generation Each user is represented using small set of 4-shot preference pairs (caption, preferred image, dispreferred image). These examples are processed by the multimodal (with a Qwen2 language backbone and multi-image capability). The final user embedding is obtained by extracting the hidden state of the last token of this profile from the Qwen2 encoder. A Ofain-of-TThoulhe (CoT) prompting is used: the VLM describes and compares each preference pair, then generates textual user profile. The template of prompting can be found in Tabld[]
VLM
C.4 COMPUTATIONAL OVERIEAD DISCUSSION
Diffusion-LPO vs Diffusion-DPO Diffusion-LPPO can yield extra training time and GPU memory usage compared to Diffusion-DPO under one global step, and the additional overhead can increase as the list size increases. proovies the GPU memory usage and training time under different list size. Note that set maximum list size to be 8 doesn't 't mean for all optimization steps have to suffer from the longer training time, As 54% of the data in Diffusion- size 2, more than half of the time Diffusion-LPO are having same training time and memory as Diffusion-DPO.
also only have
Table 5: Training time and memory cost under different list size. All measurements obtained on a single NVIDIA RTX A5000 GPU.
List size
6
Time Memory
0.229s 12815MB
0.255s
0.294s
0.326s
0.348s
0.406s 19007MB
0.449s
14065MB
15285MB
16542MB
17753MB
20234MB
21

---

Published as a conference paper at ICLR 2026
Diffusion-LPO vs DP-GPO Given same global step for Diffusion-LPO and GP-DPO, the com- putation time for calculating the score of each image is the same. The main difference mainly comes from calculating the objective as single sum(as in Equation 4 double sum(as in Equation We measure the training time for one global step and the time difference less than Is, and the memory usage for both method is the same.
D ADDITIONAL QUALITATIVE ANALYSIS
D.1 TEXT To IMAGE QUALITATIVE RESULTS
We show more examples of images generated by Diffusion-LPO. Figure 6 provides examples for Diffusion-LPO over other baselines for SD1.5. From the top to the bottom, the prompts are:
Beautiful girl;
Woman Argentina;
Realistic owl;
mechanical bee flying in nature, electronics, motors, wires, buttons, led;
beautiful, tranquil garden of love and peace;
JEM doll 80s in a fur coat the snow, thick outlines, bright colors, digital art, hard edges, detailed, anime style, dynamic pose, character design, art by sora kim, rinotuna, ilya ku- vshinov;
Duck.
Figure 7 shows more examples for Diffusion-LPO over other baselines for SDXL. From the top to the bottom, the prompts are:
A blue airplane in blue, cloudless sky;
Portrait of anime maid by Krenz Cushart, Alphonse Mucha, and Ilya Kuvshinov;
Two Somali friends sitting and watching Studio Ghibli movie;
A jellyfish sleeping in a space station pod;
A painting by Raffaello Sanzi portraying Kajol and symbiots Riot during the Renaissance era, showcased on Artstation;
A person holding very small slice on pizza between their fingers.
D.2 IMAGE EDIT QUANTITATIVE AND QUALITATIVE RESULTS
We show more qualitative results for image editing in Figure Edit prompts and original images are from the InstructPix2Pix dataset.
E GENERALIZZAION FROM OTHER DPO FRAMEWORKS
The idea switching from pairwise to listwise can be applied to most of the families of Diffu- sion DPO works. Here, show example of adapting Listwise preference into DPO work, DSPO (Zhu et al. 2025). DSPO leverages score matching from human preference, and its original objective is,
min (t)ll (x. (Vx, x, logp(xx/e) (c) + VV log log [x., c))
(12)
where wi)) is the time-dependent function for score matching Song et al. 2021). DSPO model hu- man preference with p(y|xe, c) = p(x; > x/|x! c) = a (r((c xx) r(c. x)) and reward calculated
by r(c, x,) & ([]cg(X(+1: t 1, <+1)ll Ger (x, 1, (2). Plug into the
objective(Equation 12p, their loss function becomes.
LDSPO A(1illea441 ee+1 Ax(1 a(r(c,x,) r(c, x)) (.fet))) (13)
22

---

Published as a conference paper at ICLR 2026
SD 1.5
Diff.-DPO
DSPO
Diff-LLPO
Figure 6: Qualitative illustrations for Diffusion-LPPO other baselines for SD1.5.
23

---

Published as a conference paper at ICLR 2026
SDXL
SFT
Diff.-DPO
DSPO
Diff-LLPO
Figure 7: Qualitative illustrations for Diffusion-LPPO over other baselines for SDXL.
24

---

Published as a conference paper at ICLR 2026
Original
SD 1.5
Diff.-DPO
Diff.-LPO
Makke it Lon is Mertioe
Edit instruction: make photo
Edit instruction: make the cottage a castle
Figure 8: For qualitative results, Diffusion-LPO faithfully follows the edit instructions while pro- ducing high-quality pictures. Diffusion-LPP produces more faithful and higher-quality edits. In the first example, only Diffusion-LPPO successfully transforms paint-like image into realistic photo- graph, accurately reflecting the edit instruction. In second example, while all methods modify the cottage into castle instructed, Diffusion-LPO generates visually sharper and more coher- ent castle with realistic architectural details, underscoring its superior ability to handle fine- areggrained edits. More results are available in Figure
To extend DSPO into listwise preference modeling, given list of images with preferences (x(1) (2) > x)) we can score-match the human preference through
exp(r(c,xx11))
p(y|x,, c) = p(x; X1
(++1)
(i+1) (mn) [
(m)
.c)
En exp(r((c,
(14)
Therefore, listwise preference optimization for DSPO(we denoted as DSPO-LPO) is.
exp(r(e,x(1))
CDSPO-LPO = A(t)lleo.t+1 ce+1 AT(1 En exp(r(c, x(())
) [60.)41 Get.41)). (15)
In practice, choose the highest rank image, x(1,, the target of score matching (i 1). A detailed derivation can be found at Appendix E.
E. DSPO-LPO OBJECTIVE
exp(r(cx(11)
We start with the original objective of Equation 12 By modeling po c) mm (x((((x(())))
we convert Equation 12 into:
Po (x(x((c)
exp(r(c,x(0)
min (t) log
V log
(16)
p(X: [c]
Σmh exp(r(c,x1k))
Po
(x(x(c)
exp cxp(r(c,x x())
= M =t)n log
-1(1
Σn exp(r(cc |x, ((,
(17)
p(X)
From the derivation in Zhu et al.
2025).
write the first term
po po cx
Vx. log
60,111 G++1
(18)
p(x,
25

---

Published as a conference paper at ICLR 2026
Original
SD 1.5
Diff.-DPO
Diff.-LPO
Edit instruction: have her wear a cape
Edit instruction: make it painting
Edit instruction: make a space ship
Edit instruction: make it a stormy night
Figure 9: Qualitative results for image edit using SD1.5, finetuned over Diffusion-LPO and other
baselines.
26

---

Published as a conference paper at ICLR 2026
and that V(, can be expressed in terms of the reference model:
[c, 2 68, Eref.t+11
(19)
Thus the objective simplifies to:
DSPO A(t) c0,++1 - G+1 -41 Ax o(r(c, x,) r(c,,(,x))) ((1)x (co.0:1 Gete+1)
(26)
where A(t) absorbs the timestep-dependent weight and À is constant factor arising from the Gaus- sian parameterization.
E.2 TEXT-TO-IMAGE ALIGNMENT OF DSPO-LPO
We show the quantitative result of DSPO and DSPO-LPO in Table 6 incorporating listwise preference optimization with DSPO, we further boost the performance across all metrics.
We observe that, by
Table 6: Winrate results over original SDXL for Diffusion-LPO compared with DSPO, both fine- tuned on SDXL.
Dataset
Method
PS
HPS f
CLIP f
IM 1
AES
DSPO
59.2%
77.4%
56.6%
62.9%
45.9%
Pick-a-Pic
DSPO+LPO
77.1%
87.1
64.0%
73.7%
51.5%
DSPO
55.5%
78.1%
54.2%
65.5%
55.4%
Parti-Prompts
DSPO+LPO
72.8%
82.9%
60.1%
73.1%
59.7%
DSPO
58.4%
77.3%
47.8%
63.7%
48.2%
HPSV2
DSPO+LPO
74.6%
85.0%
56.3%
72.3%
51.9%
F MORE RESULTS
F.1 RESULTS ON DIT FAMILY MODELS
We provide additional experiments SD3..5 (Esser al. 2024), DiT-based rectified-flow model.
As shown in Table 7 Diffusion--LPO outperforms Diflusion-DPO across most cases.
Table 7: Winrate results over original SD3.5 for Diffusion-LPPP compared with Diffusion-DDP
Dataset
Method
PSt
HPS t
CLIP
IM t
AES f
Diffusion-DPO
54.3%
54.6%
53.5%
52.1%
43.6%
Pick-a-Pic
Diffusion-LPe
58.6%
58.8%
51.4%
53.6%
47.2%
Diffusion-DDD
53.7%
53.5%
52.4%
50.9%
47.9%
Parti Potti-Prompss
Diffusion-LPPP
57.3%
57.7 %
51.3%
54.0%
49.2%
Diffusion DPO
52.7%
54.6%
51.1%
52.2%
44.3%
HPSV2
Diffusion-LPO 61.2%
58.8%
51.3%
55.5%
51.1 %
Discussion over online RL methods. We also test FlowGRPO (Liu et al. 2025a) on the text to image alignment metrics. The result is in Table 8 RL-based methods such as FlowGRPO operate in fundamentally different setting from ours: they perform online optimization using the reward model during training, whereas our method is fully offline and does not require any annotator or evaluator to score generated images in the training loop. For fairness, we additionally report com- parisons using the publicly available FlowGRPO checkpoint. We observe that, FlowGRPO adopting PickScore as the reward model significantly outperforms the one using GenEval s the reward model, which heavily strengthens the choice of external evaluator in terms of helping the alignment. Our method, however, doesn't t rely on any external evaluator.
F.2 SIGNIFICANT TEST OF DIFFUSION-LPO vs GP-DPO
We provide further significant tests the result in Table
27

---

Published as a conference paper at ICLR 2026
Table 8: Winrate results over original SD3.5 for FlowGRPO, with PickScore and GenEval as the reward model.
Method
PS
HPS 1
CLIP
IM 1
AES
SD3.5 FlooSSR
96.6%
77.7%
43.5%
77.7%
86.3%
SD3.5- 10335-looGRPOGGeeEEee
27.0%
13.8%
36.5%
41.0%
36.5%
Table 9: P-values for Diffusion-LPPO on SD1.5 in comparison to SDI.5 trained under GP-DPO.
PS
HPS
CLIP
IM
AES
Pick-a-Pic
0.0065 0.0204 0.1923 0.2722 0.0008
HPSv2
0.0056
0.0019 0.2467 0.0110 0,8112
PartiPrompts
0.0054 0.0082 0.1067
0.0097 0.0129
G THE USE OF LARGE LANGUAGE MODELS
The paperis is primarily written by the authors, with only minor assistance from large language models (LLMs). In particular, LLMs were used for light editing tasks such as polishing grammar, refining phrasing, and suggesting alternative wordings. All technical content, derivations, and experimental design were conceived and carried out by the authors. No LLMs were used generating research ideas, implementing methods, conducting experiments.
H ETHICS STATEMENTS
We build on publicly available datasets and models, following established practices in prior work (Wallace et al. 2024 Podell et (0033 Rombach et al. 2022 Kirstain et al. (023). All models are obtained from official repositories such HuggingFace, and respect associated licenses. While the Pick-a-Pic dataset may contain potentially unsafe content (e.g., depictions violence or nudity), use is strictly limited research purposes within the dataset's intended scope. No additional human annotation was conducted in this study.
We mindful of the broader societal impacts of generative models. Our work is intended to ad- vance alignment techniques so that diffusion models better capture user preferences in a transparent and responsible manner. We do release user-identiffabee data and take care avoid privacy risks. We also emphasize that preference alignment should be deployed in ways that respect di- versity, minimize potential harm, and promote accessibility. All experiments were conducted in accordance with ethical and legal standards, with commitment to openness, reproducibility, and integrity reporting results.
I REPRODUCIBILITY STATEMENT
We use H100-80G for our experiments, with GPU hours around 50 hours for finetuning SDXL for each experiment. We will release the model checkpoint and code source in the future.
J LIMITATION
In this work, the maximum list size is restricted to 8. We believe further exploration with larger lists and a systematic study of how list size influences optimization can provide additional insights. Moreover, evaluating Diffusion-LPO on broader range of preference datasets would help further verify its effectiveness and generality.
28

---

Published as a conference paper at ICLR 2026
Table 10: Confidence intervals for Diffusion-LPO on SD1.5 in comparison to SD1.5 trained under GP-DPO.
PS
HPS
CLIP
IM
AES
Pick-a-Pic
[51.1%. 53.5%]
[50.3%, 52.3%]
(49.2%, 52.0%
[48.1%; 53.5%
[52.6%, 54.8%]
HPSv2
[50.9%, 53.9%]
[52.2%, 55.8%]
[48.4%, 52.8%]
[50.3%, 52.3%]
[46.6%; 51.6%]
PartiPrompts
[50.9%, 53.9%]
[50.9%, 55.1%]
[49.3% 52.3%]
[50.3%, 52.5%]
[50.6%, 53.6%]
Table 11: Instructions for Embedding Generation and User Preference Prediction
System Prompt
You are an expert image aesthetics and have been asked predict which image user would prefer based on the examples provided.
COT Assistant Prompt
You will be shown few examples of preferred and dispreferred images that has labeled.
Here is Pair 1:
Here is the caption: [Caption for Pair 1]
Here is Image 1: [Image 1]
Here is Image 2: [Image 2]
Prediction of user preference: [1 or 2]
[...]
Here is Pair 4:
Here is the caption: [Caption for Pair 1]
Here is Image 1: [Image 1]
Here is Image 2: [lmage 2]
Prediction of user preference: [1 2]
1. Describe each image in terms of style, visual quality, and image aesthetics.
2. Explain the differences between the two images in terms of style, visual quality, and image aesthetics.
3. After you have described all of the images, summarize the differences between the preferred and dispreferred images into user profile.
Format your response follows for the four pairs of images:
Pair 1:
Image 1: [Description]
Image 2: [Description]
Differences: [Description]
[...]
Pair 4:
Image 1: [Description]
Image 2: [Description]
Differences: [Description]
User Profile: [Description]
Additional Assistant Prompt for User Preference Prediction
Finally, you provided with new pair of images, unlabeled by the user. Your task is to predict which image the user would prefer based on the previous examples you have seen.
Format your response follows:
Prediction of user preference: [I or 2]
29


Reviews:
[{"review_id": "h9v0tPa6Av", "reviewer": "Reviewer_FSKG", "summary": "This paper proposes to align diffusion models with listwise preference. Instead of learning human preferences from ranking pairs, this paper learns human preferences from a ranked list. It extends the DPO objective under the Plackett-Luce model and proposes LPO, which is a framework that can be combined with any DPO-style training losses and achieve further gains. The authors construct a listwise ranking dataset by postprocessing the existing Pick-a-Pic dataset and extracting ranked lists. Experiments show that the proposed method is effective compared to baselines.", "questions": "see Weaknesses. \n\nIs Diffusion-LPO implemented as an extension of Diffusion-DPO, DSPO, or both? The method is presented as a general framework compatible with arbitrary DPO-style objectives, but most results focus on Diffusion-DPO. Can the authors provide additional results showing that Diffusion-LPO also improves DSPO or other DPO variants, under matched conditions? Demonstrating consistent gains across multiple base algorithms would reinforce the method’s generality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of extending DPO to listwise preference data is novel. The use of the Plackett-Luce model to extend the Bradley-Terry model is well-motivated. The derivation is theoretically grounded, and the authors provide a clear connection to the diffusion processes. \n2. The observation in the Pick-a-Pic dataset that a large portion of it can be aggregated into lists is important, allowing richer training signals without additional labeling costs. \n3. Strong empirical results show consistent improvements over prior methods.", "weaknesses": "While the authors claim that LPO-style methods outperform prior DPO-style approaches, it is unclear whether the comparison is controlled for the same amount of data and training signal. For instance, are the models trained on the same set of image pairs or groups, and do they receive an equal number of optimization steps or preference annotations? If the listwise method effectively utilizes more comparisons per group (e.g., m(m−1)/2 relations in a list of size m), then it may benefit from a richer signal.\n\nAdditionally, the paper lacks a detailed analysis of why the improvement occurs. Is it primarily due to better preference supervision (e.g., higher-order transitivity), reduced noise, or more structured gradients? Such an analysis would significantly strengthen the empirical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0OzmPtjq7U", "reviewer": "Reviewer_ATd9", "summary": "This paper proposes Diffusion-LPO, a listwise preference optimization framework for aligning text-to-image diffusion models with human preferences. Unlike prior Direct Preference Optimization (DPO) methods that rely solely on pairwise comparisons, the proposed method introduces Plackett–Luce–based listwise optimization, allowing consistent learning over the entire ranking of generated samples. The authors reconstruct the Pick-a-Pic dataset into ranked lists using DAG aggregation and demonstrate consistent improvements over Diffusion-DPO and DSPO on both SD1.5 and SDXL models, with +12% PickScore gain and improved HPSv2/CLIPScore.", "questions": "1. Could the authors provide results when training with a fixed list size? I am curious about which list length contributes most effectively to performance improvement.\n\n2. How do training time and GPU memory usage compare to GP-DPO? Since LPO introduces additional computation, please clarify how the training cost (in both time and memory) scales relative to the group-pairwise baseline.\n\n3. The paper mentions that 54% of pair-wise data could be converted into list-wise form.\nWhat kind of pairwise annotation design would be effective for increasing this percentage ? For example, are there particular annotation patterns or graph connectivity structures that facilitate more transitive ranking reconstruction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The problem setting is clearly defined. The authors’ claim that human preferences are inherently based on ranked lists is well supported by the analysis of  Pick-a-Pic.\n\n2. The paper provides a theoretical justification for why the Plackett–Luce model is more suitable than GP-DPO.\n\n3.  A practical method is proposed for constructing listwise datasets from existing pairwise annotations.", "weaknesses": "1. There is no analysis of performance with respect to different list sizes.\n\n2. The paper does not discuss the computational overhead introduced during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CId7necJK4", "reviewer": "Reviewer_Kumb", "summary": "This paper proposes a novel algorithm to generalize the pair-wise preference learning to list-wise preferences, where a ranking of generations are available. The proposed method use a Plackett–Luce Model to directly model listwise preferences, as opposed converting one list preference to multiple pair-wise preferences like previous works. Through extensive experiments, the author show that the proposed method outperform baselines on T2I generation and image editing.", "questions": "See weakness\nAdditional questions that did not affect my judgment:\nWhat is the complexity of the proposed LPO and other methods listed in appendix B? are they both O(m^2) where m is the size of the list?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is well-motivated and is built on the solid theoretical foundation of Plackett–Luce Model, as opposed to navie converting list preferences to pair-wise ones like GP-DPO.\n2. The experiments are comprehensive, covering a wide range of tasks, datasets, metrics and base models. I'm fully convinced about the effectiveness of the proposed LPO based on the presented results.", "weaknesses": "1. I understand that the authors followed the base model choices of most literature in T2I preference learning and used SD 1.5 and SDXL as the base model. However, it would be better if the authors can demonstrate the effectiveness of proposed method on modern rectified-flow models in DiT architecture, such as Sana, Flux, SD3. This would make the contribution more relevant to the community.\n2.  While I'm convinced of the effectiveness of Diffusion-LPO compared to baselines presented in Table 1, Table 2 is less impressive, with the win rate against GP-DPO around 50% (i.e random). The author should perform statistical significance analysis (i.e. confidence interval or p-value) to determine if these results are significant or indistinguishable from the null hypothesis (i.e. LPO and GP-DPO has same performance).\n3. Pickscore and HPS in general are not a good metrics for image editing tasks. I suggest the author report more common metrics used in image editing. These includes conventional metrics like DINO, L1, CLIP [1].  Alternative, the author may choose to adopt a more \"modern\" image editing benchmarks that uses GPT4 as the judge model, such as ImageEdit, \n\n[1] Zhang, Kai, et al. \"Magicbrush: A manually annotated dataset for instruction-guided image editing.\" Advances in Neural Information Processing Systems 36 (2023): 31428-31449.\n[2] Ye, Yang, et al. \"Imgedit: A unified image editing dataset and benchmark.\" arXiv preprint arXiv:2505.20275 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIaVyzwyNf", "reviewer": "Reviewer_n3m2", "summary": "The paper proposes an extension of DiffusionDPO from pairwise preference to listwise preference where the elements within the list is ranked. The main idea applies Placket-Luce model instead of the usual Bradley Terry model for reward construction. The authors show the advantage of DiffusionLPO over naive extension of DPO such as GP-DPO and show noticeable gains in performance over other baselines on classical datasets and SD model series.", "questions": "I would like the author to address above questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The presentation is clear and the idea is straightforward. \n- The theoretical justification that naive extension of DPO overestimates the positive/negative gap is illuminating.\n- The results show good improvements over baselines over multiple standard datasets.", "weaknesses": "- The baseline comparisons are not sufficient. There many RL-based methods such as FlowGRPO that also optimize for reward signals that the authors miss. How does LPO compare with these online RL techniques?\n- The optimized models are quite outdated (e.g. SD1.5 and SDXL). How does the method perform on newer models such as FLUX?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)