Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
Authors: Xinxin Liu
Ming Li
Zonglin Lyu
Yuzhang Shang
Chen Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Primary area: generative models
TL;DR: diffusion dpo， RLHF

Abstract: Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise—images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training.

Published as a conference paper at ICLR 2026
LEARNING FROM NoISY PREFERENCES: A SEMI- SUPERVISED LEARNING APPROACH TO DIRECT PREF- ERENCE OPTIMIZATION
Xinxin Liu, Ming Li, Zonglin Lyu, Yuzhang Shang, Chen Chen University of Central Florida
ABSTRACT
Human visual preferences are inherently multi-dimensional, encompassing aes- theties, detail fidelity, and semantic alignment. However, existing datasets pro- vide only single, holistic annotations, resulting in severe label noise-images that excel in some dimensions but are deficient in others simply marked winner or loser. We theoretically demonstrate that compressing multi culli-dimennional pref- erences into binary labels generates conflicting gradient signals that misguide Dif- fusion Direct Preference Optimization (DPO). To address this, we propose Semi- DPO, semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on consensus-fitteedd clean subset, then uses this model as implicit classifier generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of of-the-ar performance and sig- nificantly improves alignment with complex human preferences, without requiring additional human annotation explicit reward models during training. We will release our code and models at: https://githhbb.c www.re//oo--ddngaaaaeeeeeeeeeeee
INTRODUCTION
Diffusion models (Ho al. 2020) have achieved remarkable success in text-to-image (T2I) gen- eration (Ramesh et al. 2022 Pernias et al.) 2023 Ramesh et al. 2021). However, aligning T21 diffusion models with human preferences typically requires training with separate reward models, creating significant computational bottlenecks (Wang et al. 2024 Wu et al. [2023a Xu et al.) 2023)
To address this limitation, Diffusion-DPO (Wallace et al. 2024) adapts the direct preference op- timization (DPO) approach from large language models (LLMs) (Rafailov et al. 2023b), elimi- nating the need for explicit reward models by optimizing directly on preference pairs. However, inherently multi-dimensional (Zhang et al. 2024), annotated datasets collapse this into binary choices. This destabilizes training by creating contradictory signals that penalize learning desirable attributes from "loser" images while simultancously rewarding undesirable ones in "winner" images.
Diffusion- DPO overlooks fundamental distinction: while human visual preferences
As shown in Figure I for the prompt "A vast green grassland with blue sky and two white clouds, a mother cow and her calf both eating grass, natural landscape, extremely detailed, 4k resolution, perfect lighting, fine textures". Image may excel in semantic alignment and composition but appear aesthetically flat, while Image B may excel in texture but lack semantic alignment. When a human annotator is forced to choose, their judgment may hinge on a single dimension, yet the label is recorded as overall preference. This produces noisy, contradictory signal, as the model is implicitly taught to prefer all attributes of the winning image, including its flaws.
Collapsing multi-dimensional preferences into single binary label introduces critical challenge for Diffusion-DPO training: dimensional conflicts across the dataset cause conflicting gradient sig- nals leading to suboptimal convergence. We formalize this phenomenon in our theoretical analy- sis 3.2 To address this challenge, we reformulate the problem through the lens learning with noisy labels (LNL) (Zhang et al.. 2016; Song al., 2022). dominant paradigm for tackling LNL to reframe as a semi-superviser learning (SSL) problem, where noisy samples are treated as unlabeled set that requires relabeling. Within this paradigm, iterative self-training stands out

---

Published as a conference paper at ICLR 2026
CONFUSED
Prompt: proon grosslndd sky two mother eatting now landeccpp, graas, detailes, resoution. porfect lighing. fine textures.
Why different lesrned before?
Diffusion DPO
Conflict Signal
Composition: A
Samartic Aligmment X Texture:
Diffusion Model
What earth do humans like ?
Image B
Refined Timestep-Awor Signal
Training
Human Visual Preferences
Semi-DPO (Ours)
Win
Composition:
Label:A XB Binory Lobel
Semarfic Alignment A
Decoupled
Lese
Diffusion Model
Textune: B
Antotator
Human
M
Pacudo-labdlig by Stself (.3).
Figure I: Resolving label noise from multi-dimensional preferences. Standard Diffusion- learns from a noisy, conflict signal created when a binary label (A > B) collapses multi multi-dimensional preferences (c.g., g., A's composition vs. B's 's texture). Our method, Semi-DPO, resolves this by decoupling the conflict into refined timestep aware signals via self-training, leading to a more robust alignment.
powerful and widely adopted methodology. It involves training classifier on small clean set and using it to generate pscudo-labes for the larger unlabeled set, progressively refining the model (Xic) ct al. 2020 Li et al. 22200; Arazo et al. 2019). Inspired by this, we treat dimensionally conflicted samples as our unlabeled set. The central question for this approach then becomes which model should serve as the classifier for generating these sseudo-labels.
Our answer lies within the diffusson model itself. We leverage the fact that the DPO loss compels the model to distinguish between preferred and dispreferred samples, effectively turning it into an implicit preference classifier. This inherent capability allows the diffusion model to generate fine- grained preudo-label, for dimensionally conflicted data without requiring any architectural changes. Furthermore, echoing research (Hertz et al. (2022) on the hierarchical nature of the diffusion process, where early stages govern global composition and later stages refine local details. This hierarchy allows us to reframe single, conflicting preference label (e.g., A's good composition vs. B's good preferences. Therefore, we apply this
texture) as a series of non- confolfiic,,, implicit classifier across the diffusion timeline. This transforms a single noisy preference signal into fine-grained, timestep-conditioned pseudo-labels, effectively decoupling noisy confliciing signals.
To this end, we propose Semi-DPO, two-stage framework. In the first stage, Multi-Reward Con- sensus, we partition the dataset by using consensus of diverse, pre-trained reward models to filter the data. A preference pair is added to clean labeled set only if all models unanimously agree with the human label; otherwise, it is assigned to a noisy, unlabeled set requiring relabeling. This process designates about 21% of the Pick-a-Pic V2 dataset as clean. In the second stage, Itera- tive self-training, we initially train the model on the clean labeled dataset. The trained model then generates timestep-conditional pseudo-labels for the noisy unlabeled dataset. To mitigate confirma- tion bias and model drift (Zhang et al.) 2021 Cascante- Bonilla et al. 2021 Xie et al. (020), only high-confidence pseudo-labels are selected for retraining with a composite objective anchored to the clean set. This creates virtuous cycle, synergistically improving the model's 's alignment with multi-dimensional human preferences.
In summary, our main contributions are: (1) We provide theoretical analysis proving that con- flicting dimensional signals in holistic labels cause conflicting gradient signal during Diffusion- DPO training, leading to suboptimal convergence. (2) We propose Semi-DPO, a novel self-training framework that reframes the preference alignments as semi-supervised learning under label noise, leveraging cimestep-conditional pseudo-labeling to generate fine-grained signals that decouple con- flicting preference dimensions. (3) We demonstrate through extensive experiments that Semi-DPO achieves state-of-the oof-the-art art performance, significantly improving the model's ability generate im- ages that align with complex, multi-dimensional human preferences without extra annotation costs or fine-tuning with an explicit reward model.
RELATED WORKS
Diffusion Models and Diffuusio Alignment. In recent years, diffuusion models (Ho et al. 2020) Song & Ermon 2019 Song al. 2020) have achieved remarkable success in text-to-image gen-
2

---

Published as a conference paper at ICLR 2026
eration. These models are traditionally trained large-scale text-image datasets scraped from the web. However, they are not well-aligned with human preferences. To achieve better alignment, T2I adapted Reinforcement Learning from Human Feedback (RLHF) (Ouyang al. 2022) from the LLM domain (Dai et al., 2023; Miao et al. 2024; Clark et al., 2024), which requires train- ing an explicit reward model on human preference data to guide diffusion model optimization [i] iet al. (2024a): Xu et al. (2023). However, developing reliable reward models remains computa- tionally expensive and requires large-scale annotated datasets (Xu et al., 2023 Wu et all 2023a Wang et al. 2024), creating a significant bottleneck. Inspired by Direct Preference Optimization (DPO) (Rafailov et al. 2023b) in LLMs, recent work has adapted this approach to T2I alignment, eliminating the need for an explicit reward model. Methods like Diffusion-DPO (Wallace et al.) 2024) directly optimize diffusion models on human-annotated preference pairs by maximizing the relative probability of preferred images. Follow-up work on Diffusion-DPCC falls into two cate-
gories: offline (Li et al., 2024b, Zhu et al. 2025 Lee et al., 2025 Hong et al. 2024 Li ct al. 2026) and online methods (Liang et al. 2025 Black et al. 2023] Zhang al.) 2025] Yang et al. (022)). We
provide detailed comparisons between online and offline diffusion DPO methods in Appendix 6.5
and discuss how Semi-DPO relates to existing work in Appendix 6.6
Noise Data & Semi-Supervised Learning, The success of deep learning models largely depends on large-scale, high-quality annotated datasets. However, common data collection methods-such as web scraping and crowdsourcing (e.g.. Amazon Mechanical Turk)
introduce label errors (Song et al., 2022). Due their high capacity, deep neural networks can memorize these incorrect labels, which degrades generalization (Zhang et al., 2016 Song et al. (002)) To address this challenge, dominant paradigm reframes learning with noisy labels (LNL) as a semi-supervised learning (SSL) problem (Arazo et al. 2019)_ This approach partitions training data into a clean labeled dataset and noisy unlabeled dataset
(Li al. 2020 Han et al.) 2018, Yu et al., 2019) Wei et al. 2020). This partitioning strategy has been applied in several influential frameworks. Co- teaching (Han et al.] (2018) trains two networks that select small-loss samples for each other. Noisy Student Training (Xie et al., 2220) employs self-training where teacher generates pseudo-labels
for a larger, noised student model, enabling it to learn more robust representations.
METHOD
3.1 PRELIMINARIES
Diffusion Models. Diffusion Models are latent variable models designed to learn the reverse of fixed, T-step Markovian noising process. The forward process, is defined as g(x; x;-1) := A(x;; /1 -, B,I), which admits a closed-form sampling distribution any timestep ((x: xo) = N(x:; v Vo (1 T), where o₁ = 1 Br and ã = I Os.
The model learns the reverse process, (x x,, c). by training network co(x,, f, e) to predict the noise component c from a noised sample x,. This is achieved by optimizing a simplified objective on the negative logeliiel:oood:
LDM Ex,,,. (ot()) |c- co (Vopxo + vI- t, e)
Generation is performed via ancestral sampling, starting from XT ~ No 1) and iteratively applying the learned reverse transition.
Reinforcement Learning from Human Feedback (RLHF). A prevalent approach for model align- ment is Reinforcement Learning from Human Feedback (RLHF). For text-to-image (T2I) models, human preference data is collected as paired comparisons (xx" x c), where xi" and x" represent the preferred ("winning") and dispreferred ("losing") final images for given text prompt
First, a reward function r(Xp, e) is trained to model these preferences using the Bradley- -erry model, where the likelihood of preferring x over xh is given by:
PDT Pe [x x c) = =o (r(x) r(xf c))
where (()) is the sigmoid function. Subsequently, the diffusion model policy pe is optimized to maximize the expected reward, regularized by a KL-divergence term to prevent large deviations from reference policy Prot, with B 0 controls the KL penalty strength:
LRLur Epp((ole) (xo, c)] ,3 DRr.. [po (Xo.r c)) l|prot (Xo:T c)]
3

---

Published as a conference paper at ICLR 2026
Direct Preference Optimization for Diffusion Models. Direct Preference Optimization (DPO) simplifies RLHF by re- p--arammterizing the reward function directly in terms of the policy and ref- erence models, thus bypassing the need for an explicit reward model. When adapting this concept to diffusion models by framing the denoising process as a Markov Decision Process, the implicit, time-step-wise reward function be expressed as:
Xe, c) Pre1 X1, c)
Pa (x
r(x-11, -Blog
(4)
Substituting this reward definition into the Bradley-Terry likelihood yields the conceptual Diffusion- DPO loss, which is optimized directly with respect to the policy parameters A:
Po
xy" .)
Blog pa (x x c) Piet (x X,, c)
LpPo-Diffusion 13
log
(Blog
(5)
(rer
x
The expectation is taken over the preference dataset 'D and timesteps & E [1,7]. where the noisy states x2 sampled from the forward noising process.
3.2 CONFLICTING GRADIENTS SIGNAL FROM MULTI-DIMENSIONAL PREFERENCES
Riffusion-DPO Gradient Formulation. To understand the training dynamics, we begin by decom- posing the per-sample. per-timestep DPO gradient, Ve (t)
VoCipo -11 a (1) )) 3 V log (x xx] Vo Po pe (x
(6)
(0)
where 2/" :0:=B log
pa 1x Pier (x", xx) log
po pa(xi_,ixi.c) This decomposition shows that the Piet x r))
Diffusion-DPO update adjusts the model's parameters along the direction of the feature difference, Aog aiming to increase the log-pree pog-probability of the preferred sample relative to the dispreferred one. While this update mechanism effective for clean preference pairs, its behavior becomes problem- atic in the presence of multi-diimessoonl conflicts. Detailed derivation is provided in Appendix 6.1 We now analyze how these conflicts impact the Diffusion-DPO training process.
The Source of Conflicting Signals in Preference Optimization. In this section, we provide a theo- retical analysis for the training instability outlined previously. We demonstrate how collapsing multi- dimensional preferences into binary labels mathematically guarantees the presence of conflicting gradient signals, leading to suboptimal convergence. Our analysis begins by partitioning the dataset based on whether sample's 's preference along a specific dimension aligns with its holistic label. For a given dimension (e.g.. composition), let the reward difference be Ark := r& rx(xx); .) - r& (xh- c). This partitions the dataset into an alignment set A, (where the dimensional preference matches the holistic label, Ar, > 0) and a conflict set C2 (where the dimensional preference opposes the holistic label, Ars < 0), occurring with probabilities Pa,& and Po,k, respectively.
To analyze the training dynamics, we define the per-sample gradient gø = -1 Ae( (t) and
the oracle direction, t0, sign Arx ) The oracle direction represents the ideal update direction for improving dimension k. For an aligned pair in Ar, Uk is the standard Diffusion- DPO update direction. However, for conflicting pair in Cx. Uj points in the opposite direction, representing the corrective update that should have been applied for this specific dimension.
(+) en (O,(0)))) This quantity We then examine the inner product. The inner product defined as (-90²
measures how well the actual gradient update aligns with the ideal oracle direction. A positive value indicates that the update making progress on dimension k while a negative value indicates the update is actively degrading performance that dimension. Therefore, the variance of this inner product serves as a direct mathematical measure for the severity of conflicting signals that cause training instability.
As we prove in Appendix 6.2 this variance is bounded below by:
Vark (-9gg Var(<-go0)
(t) Tn
(t) + m 1)>) 2 Pa,kPo,k (m
(1)
4

---

Published as a conference paper at ICLR 2026
Multi- Nulti-RRewrrd Consensus (1) Train
Iterative Self-Training
(2) Timestep-Conditions Pseudo-Labeling
(4) Repeat
Py DM
Labeled Semples
Latent
Deto)
Imaga
RM
Timestep-Condiiionl Pseudl-Lobeling
Pl DM
forward
Ref DM
Agree
Latent
Image
prompt
(3) Re-train
RM
Any Disagree
lad ( and 45
latila (" and af
RM
Original Dataset
Unlabeled Semples (Noisy Dato)
Pseudo-Label Sumples with Few labeled samples
low confidence, not select
Figure 2: Semi-DPO framework for resolving label noise. Stage MMulti-Reward Consensus): A committee of reward models partitions the original dataset into small, clean labeled dataset (based unanimous agrec- ment) and large, noisy unlaheled dataset. Stage (Iterative Self-Training): (1) An initial model is trained on the clean labeled set. (2) This model then generates pseudo-labels for the noisy unlabeled set. A pseudo-label accepted if its confidence score (the logit magnitude |zš] exceeds a dynamic threshold T; ( The sign of the logit, sign(=)), determines how the new label is applied: positive sign keeps the original "winner" and "loser" assignment for the image pair, while negative sign swaps them. (3-4) The model is then retrained on the high-confidence pseudo-labele and the original clean set, and this cycle is repeated until convergence.
(1) where ma,k and m ma,k are the expected magnitudes of the gradient updates conditioned on the align- ment and conflict sets, respectively. This inequality reveals the mechanism behind the instability. The product term Pa.kPe, is greater than zero if and only if a conflict set exists Pa.k > 0 ), which mathematically guarantees that the training signal must contain updates that are both aligned with and opposed to the oracle direction. This co-existence gives rise to gradients with inconsistent di- rections updates from C2 that directly oppose the "oracle" updates desired for dimension k.
Conflicting gradient signals force the model's parameters to oscillate, with the update direction fre- quently reversing. This behavior introduces two critical issues: (1) it renders the learning process highly inefficient, progress from one step is largely negated by the next, impeding the minimiza- tion of the loss function. (2) this constant directional conflict makes the training instability, and the model struggles to find consistent optimization path, leading to suboptimal convergence.
3.3 THE SEMI-DPO FRAMEWORK
Our theoretical analysis shows that collapsing multi-dimensee dimensional conflicts into single binary labels generates conflicting gradient signals, causing suboptimal convergence. To address this, we reframe the alignment task as a semi-supervised learning (SSL) problem designed to handle these noisy labels. We detail our proposed framework in the following section.
Data Partitioning via Multi-Reward Consensus. To obtain clean labeled set (Diatelea) for stable cold-start training, we filter the original dataset D using a multi-reward consensus approach. This approach is motivated by existing work that reveals strong correlations between widely-used reward models and different dimensions of human preference Zhang et al. (2024) (see Table Figure 4 For instance, CLIP Score (Radford et al. 202I) demonstrates strong correlation with the semantic alignment dimension, while Aesthetic Score (Schuhmann et al. 2022a) is highly correlated with the aesthetics dimension (detailed explanation provided in Appendix 6.8). We employ set of K pre-trained models [r*)* A preference pair (c, ar)) is included in Diabeled only if all reward models unanimously agree with the holistic label, i.e., Vk, Ar: rx( c) TA (#/ e) > O. The remaining data, which contains dimensional confticts, forms the noisy unlabeled set D Duntaheled- This filtering step yields a high-quality dataset that provides an unambiguous initial gradient direction.
Timestep-Conditional Pseudo-Labeling Our pseudo-labeling strategy stems from key insight: the DPO loss function is equivalent to a binary cross-entropy loss. This means the training process implicitly trains binary classifier at each timestep to distinguish between the denoising predictions associated with the preferred and dispreferred samples. Consequently, the per-timestep margin, =/") functions as the logit for this implicit classifier. This provides a principled signal for self-training, where the sign of the logit, sign(2)"), determines the predicted preference, and its magnitude, |z serves the confidence score (see Figure 2 right).
5

---

Published as a conference paper at ICLR 2026
Dynamic Connestta-Condiiionall Thresholding. Our experiments find that the model's confidence and prediction accuracy are not uniform over the diffusion timesteps (Table Therefore, instead of single fixed threshold for pseudo-labeling, timeline into N distinct intervals, (7 } and assign a unique threshold to each. This dynamic
employ dynamic strategy. We partition the
threshold, 7,-1 is updated at each training iteration and is specific to the interval e<l) containing timestep Consequently, a pseudo-label is used for retraining only if confidence score |=8--11 exceeds the corresponding threshold for its specific time interval, as shown in Figure right.
Iterative Self-Training with Composite Objective. The iterative self-training process begins with a "cold-start" phase. We first train an initial model, pg. using only the clean labeled set Diabeled This step is crucial as it provides stable and reliable foundation for the self self-training loop. The training objective at this stage consists solely of the anchor loss (defined in Eq. 9 For each subse- quent iteration i, we leverage the model from the previous step, P3 to generate pscudo-labels for the noisy unlabeled set Dunlaheled- The new model, P/. is then trained using the composite objective (defined in Eq. ( that combines stable, anchoring signal from the clean set with learning sig- nal from high-con conficoone pseudo-labels (see Figure 2 left). These principles are formalized in our composite objective for the iterative refinement stage O):
Dem (0) Liabeted (0) - unlabeled (0)
The two loss components are defined as:
Anchor Loss (Ctabeled). The standard Diffusion- DPO loss on the clean labeled set, which acts as ground-truth regularizer prevent model drift.
Llabeled (0) = Bee,og an)) Draces [- log (+)
Pseudo-Label Loss (())) (Cunlabeted) The DPO loss filtered subset of the noisy data, using pseudo-
labels generated by the model from the previous iteration (p)-1).
unlabeled (0) E (naa ( miateeee [1I=62 ( (-logo(E))))
(10)
In Eq. 10 the indicator function II()) filters for high-confidence predictions using the dynamic thresh- old 7-1- The new preference pair (zi pscudoo psccudo) used to compute the loss term af² is determined
by the sign of the previous model's logit, (t) =0,-1' A positive sign retains the original label, while negative sign swaps the "winner" and "loser" images.
This pseudo-labeling mechanism the core of our solution to the inflated gradient variance prob- lem identified in Section 3.2 The variance originates from dimensional conflicts within Dunlaheled where a sample's holistic label provides a supervisory signal that opposes the ideal gradient for a specific attribute. Our method resolves these conflicts granular, timestep-conditione level. By
re-labeling pairs based on the model's own learned preference-the sign of the logit =0; -it actively corrects the noisy original annotations. This process effectively reduces the proportion of conflicting samples (the term Pe,k in our analysis) that generate gradients with inconsistent directions. Conse- quently, this self-correction mitigates the source of gradient conflict, leading to a more consistent and effective training signal from the noisy dataset.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets and Models. We select SD1.5 (Rombach et al., 2022) and SDXL (Podell et al., (023) base model. We train Semi-DPO on the Pick-a-Pic V2 (Kirstain et al. 2023) dataset. After excluding approximately 12% of ties pairs, there are 851,293 preference pairs across 58,960 unique prompts in the training dataset.
Baselines. We compare Semi-DPo against strong baselines, including the original base models (SD1.5 Rombach et al. (0022) SDXL. Podell et al. (202)) and state-of-the-art alignment methods

---

Published as a conference paper at ICLR 2026
Chi quiciig
skic musiig srrse kuna pai p0x, uodio
as luk salln skap hoas
Figure 3: Sample images generated by different models for various prompts.
such as Diffusion- DPO Rafailvv et al. (2023a), Diffusion- PiffusioonKTTLLLI et al. (2024b), MaPO Hong et al. (2024), and InPO Lu et al. (2025). All comparisons are conducted using their officially released checkpoints.
Training Details. During the multi- consensus stage, we employ five proxy reward models to filter the dataset: PickScore (Kirstain et al. 2023), HPS v2 (Wu et al. 2023a). CLIP Score (Radford ict al. 2021), the LAION Aesthetics Classifier (Schuhmann et al. 2022a). and ImageReward (Xu let al. 2023). This filtering process, which is further detailed in Appendix 6.8 and Section 3.3
yields a clean, consensus-labelee dataset of 176,999 pairs, with the remainder classified as noisy. We then split this clean dataset into training portion of 173,007 pairs and a test portion of 3,992
pairs, which is used to evaluate model accuracy after each training iteration (See Appendix (6.9.. An ablation study investigating the impact of using different numbers of reward models is presented in Section 4.3 Additional training details can be found in Appendix 6.9
Evaluation. Following the protocol from (Li et al., 2024b), we evaluate our Semi-DPO method on the SD1.5 model. The model's performance is assessed using a suite of established met- rics: ImageReward. PickScore, HPS v2, the LAION Aesthetics Classifier, CLIP Score, and Gen- Eval (Ghosh et al.) (2023). Furthermore, to demonstrate Semi-DPO' ability to generate outputs that align with multi-dimensional human preferences, we employ the Multi-dimennional Preference Score (MPS) (Zhang et al. 2024
4.2 ALIGNMENT RESULT
Qualitative Result. Figure 3 presents qualitative comparison between Semi-DPO and base- line models using the same prompt, demonstrating our method's significant improvements in text- alignment, detail fidelity, and aesthetics. For instance, given the prompt, "a photo of Pikachu cook- ing at restaurant, wearing a chef's 's hat," Semi-DPO is the only method that successfully generates Pikachu with the specified chef's hat, highlighting its superior text-alignment. We provide more visualization comparison results in Figure 6
Quantitative Result. Tables I demonstrate that Semi-DPO consistently outperforms baselines on both SD1.5 and SDXL, excelling in multi-reward metrics and win-loss rates (refer to Ta- bles Consistent with prior studies (Zhang al. 2025 Lee et al. 2025). we further evaluate on specialized benchmarks: GenEval (Ghosh et al. (203) for Abject-focused generation and T2I- CompBench (Huang et al.) (205)) for compositional generation. Results in Table 2 and 3 confirm Semi-DPO's significant advantage, validating its capability in generating complex, high-fidelity im- ages.
4.3 ABLATION STUDY
Iterations. To validate the contribution of our iterative self-training process, we conducted an abla- tion study on the key stages of Semi-DPO, as shown in Table Our hypothesis is that each iteration enables the model to generate better pseudo-labels, leading to further performance gains. We evalu- ated the performance of different iterations: IterO, the initial model trained only on the high-quality clean dataset; Iterl, the model after the first round of pseudo-labeling and retraining; and Iter2, the model after the second round of pseudo-labeling and retraining.
Our ablation study results substantiate our hypothesis. We find that each iteration yields a model ca- pable of generating more reliable pseudo-labels, which in turn enhances the performance of the sub- sequent model. Specifically, we observe significant performance improvements from IterO to Iterl and from Iterl to Iter2. However, our results indicate these performance gains diminish and stabi- lize after the second iteration. We therefore conclude two rounds of self- self-training are sufficient for convergence, striking an effective balance between performance and computational efficiency.ssee Appendix for a detailed cost analysis).
7

---

Published as a conference paper at ICLR 2026
Table I: Reward Score comparisons on HPS v2, Parti-Prompt and Pick-a-Picc V2 datasets. We compare hase- lines across both SD1.5 and SDXL architectures. Best results are in boldface.
Model
Dataset
Method
ImageReward
HPSv2.1
PickScore
Aesthetic
CLIP
MPS
SDI.5
0.139
0.246
20.862
5.578
0.293
12.211
Dift-DPO
0.339 (12200))
0,259 (533)
21.308
0.297 (.714
12.739 (14.8)
HPS v2
Dit'-KTO
0.690
0.284 (+1)
21.454 (-28%)
5,803 (4400)
13.016 (-
Semi-DPO
0.816 (10677)
0.287 (11678)
21.945
5,899
(120))
13.514 (11976)
0.194
21.284
5.358
9.754
Diff-DPO
0.352 (40L159)
0.262 (+3.9))
21.520 (11.1%)
5,443 (x1.65)
(:0.7%)
10.135 (13.9%)
SD1.5
Parti Prompt
Ditt-KTO
0.615 (404211)
0.279 (+9
21.594 (-1
5,552 (+3,68)
0.277 (=2
10.202 (+1.68)
Semi-DPO
0.798 (-0,04))
0.284 (+:1 8R3)
21.964 (-3.2%
5,706 (66.5%)
(.226
10.771 (+10.99)
SD1.5
0.085
0,250
20.566
5,421
0.273
9.635
Dit--DOO
0.297 (-02222)
0.261
20.948
5.549 (+2.4%)
(.279))224)
10.144 (45.3%)
Pick-a-Pic
Dift-KTO
0.629 (:0.44))
0.281
21.064 (12.4%)
5.659
0.281 (1299)
10.226
Semi
24)
(-1.%)
(+7.00)
(-2.9%
(-4.59
SDXI.
0.839
0.280
22.611
6.127
0.301
14.472
Diff-DPO (.000)
0,298 (16449)
22.953 (11.5%)
6,163 (10.5)
0.307 (12.09)
14.831 (12.5%)
HPS v2
MaPO
(4
0.292 (+4.M83
22. 22.68220m
6.244 (+1.9))
0-303
14.635 (41.1))))))
Semi-DPO
1,095 (40.256)
0,301
22.816 (.0.9%)
6,269 (+2,3%)
0.302 (40.3))
15,001 (43.7%)
SDXL
0.736
0.272
22.407
5,766
0.278
11.255
SDXL
Parti Prompt
Difi-DPO
1,066
(+5.9%)
22.681 (-1.2%)
$.813
(-2.9%)
(14.8)
0.855 (+01119)
0.280 (+2.9%)
22.412 (-000%)
(+3.0%)
(-0.79)
11.431 (+1.6))
Semi DPO
0.983 (40.247)
0.291 (+7.00%)
22.715 (-1.4%)
5.845 (+1.4%)
0.281 (-1.1%)
11.633 (-3,4%)
SDXL
0.716
0,282
22.031
6,055
0.283
11,809
DHI--PO
0.953 (441201)
0.299 (+6/083)
22.467 (-2.0%)
6.031 (4049)
0..99 (-3.22))
12.210
Pick-a-Pic
MaPO
0.852 (10.136)
0,290 (12.59))
22.091 (10.3%)
6,189 (x2.25)
(8118))
11.887 (10.7%)
Semi-DPO
1,056
0.304
22.352
6.210
0.287
12.548
Table 2: Quantitative results on GenEval (Ghosh et al. SD1.5 and SDXL baselines. Best results are in boldface.
2023 with 50 inference steps. Comparison between
Base
Method
Single
Two
Counting
Colors
Position
Color_attr
Overall
SD1.5
95.62
37.63
37.81
74.73
3.50
4.75
42.34
Diff-DPO
96.88
39.90
38.75
75.53
3.30
3.75
43.00
SD1.5
Diff-KTO
97.50
35.35
36.25
79.79
7.00
6.00
43.65
InPO
95.00
45.45
45.00
82.98
4.00
8.00
46.74
Semi-DPO
98.75
49.75
42.19
77.93
6.00
9.25
47.31
SDXL
98.12 18
75.25
43.75
89.63
11.25
15.75
55.63
Diff-DPO
99.38
82.58
49.06
85.11
13.05
18.55
$8.02
SDXL
MaPO
96.56
66.41
40.00
84.31
10.75
18.75
$2.80
InPO
97.50
74.75
46.25
84.04
10.00
18.00
55.09
Semi-DPO
97.50
80.81
50.00
86.17
14.00
22.00
58.41
Table 3: Quantitative Results on Ten-Chm Huang et al. SDXL baselines. Best results boldface.
2025
Comparison between SD1.5 and
Base
Method
Color
Shape
Texture
2D-
3D-
Numer.
Non- Spa.
Complex
Spa.
Spa.
Original
0.378
0,362
0.417
0.123
0.297
0.449
0,310
0,300
Diffusion-DPPP
0.409
0.366
0.425
0.134
0.312
0.454
0.312
0.304
SD1.5
Diffusion-KTT
0.465
0.416
0.466
0.157
0.341
0.461
0.314
0.309
InPO
0.482
0.424
0.493
0.159
0.341
0.468
0.314
0.319
Semi-DPO
0.471
0.433
0.493
0.183
0.340
0.481
0.310
0.320
Original
0.5833
0.4782
0.521
0.1936
0.3319
0.4874
0.3137
0.3327
Diff-DPO
0.6941
0.5311
0.6127
0.2153
0.3686
0.5304
0.3178
0.3525
SDXL
MaPO
0.6090
0.5043
0.5485
0.1964
0.3473
0,5015
0.3154
0.3229
InPO
0.6409
0.5067
0,5720
0.2107
0.3671
0,5221
0.3129
0.3674
Semi-DPO
0.6624
0.5079
0.5727
0.2133
0.3723
0.5410
0.3154
0.3728
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of the iterative self self--taining process SD1.5.
Dataset
Method
ImageReward
HPS v2.1
PickScore
Aesthetic
CLIP
MPS
Semi-DPO (Iter))
0.569
0.269
21.493
5.806
0.300
13.039
HPS
Semi-DPO (Iterl)
0.798
0.284
21.892
5,902
0.300
13,495
Semi-DPO (Ite22)
0.816
0.287
21.945
5.899
0.299
13.514
Semi- RPO (Iter0)
0.557
0.272
21.679
5,548
0.275
10,386
Parti Prompt
Semi-DPO (Iterl)
0.779
0,283
21.929
5,691
0.277
10.743
Semi-DPO (Iter2)
0.798
0.284
21.964
5.706
0.276
10.771
Semi-DPO (Itrr))
0.563
21.153
5,660
0.282
10.554
Pick-a-Pic
Semi-DPO (Itrrl)
0.789
21.490
5.794
0.282
11.018
Semi-DPO (Iter2)
0.801
21.524
5,801
0.281
11.030
Number of Reward Models for Consensus Filtering. To determine the optimal number of proxy reward models for multi-reward consensus, ablated consensus sizes of SD1.5. As shown in Table 5 performance consistently improves across all metrics as the com- mittee size increases. Notably. incorporating a greater number of reward models reduces the bias from individual evaluators and not only boosts performance on the metrics used for filtering but also enhances the model's generalization ability other preference evaluators not included in the consensus (MPS). We therefore adopt five-model consensus to ensure the initial model is trained on the most reliable clean data.
five models on
Table 5: Ablation study on the number reward models for consensus filtering. This study evaluates the performance of model trained on data filtered by consensus of an increasing number of reward models (from two to five) to determine the optimal configuration. Metrics included in the consensus committee are marked in green, while metrics that evaluated but not used for filtering marked in red. The baseline SD1.5 and its results are marked in gray. Best results for each dataset are bold. The results demonstrate that performance consistently improves across all metrics and datasets as the consensus committee grows.
Dataset
Method
CLIP
Aesthetic
HPSv2
Image Reward
PickScore
MPS
SD1.5
0.293
5.578
0.246
0.139
20,860
12.211
CLIP+Aes
0.299
5.759
0.263
0.442
21.438
12.891
HPS v2
CLIP+Acs+HPS
0.299
5.775
0.265
0.459
21.430
12.927
CLIP+Acs+HPS+IR
0.299
5.789
0.268
0.524
21.464
13.018
CLIP-Acs--IP++IRPPP
0,300
5,806
0.269
0,569
21.493
13.039
SD1.5
0.270
5.358
0.254
0.194
21,284
9.754
CLIP+Aes
0.273
5.501
0.265
0.444
21.620
10.280
Parti Prompt
CLIP+Acs+HPS
0,273
5,519
0.268
0.470
21.699
10,331
CLIP-Acs+HPS+1R
0.274
5,530
0,269
0,513
21.651
10.363
0.275
5.548
0.272
0.557
21.679
10.386
SD1.5
0.273
5.421
0.250
0.085
20.566
9.635
CLIP+Aes
0.279
5.609
0.267
0.417
21.099
10.399
Pick-a-Pic
CLIP+Aes+IIPS
5.627
0.469
21.143
CLIP-Acs+HPS+IR
0.281
5.632
0.271
0.499
21.127
10.477
CI Pick
0.282
5.660
0.273
0.563
21.153
10.554
CONCLUSION
In this work, we theoretically demonstrate that collapsing multi-dimensional preferences into binary labels generates conflicting gradients, hindering iiffusion-DPO optinization. To resolve this, propose Semi-DPO, a semi-supervised framework that identifies clean data via Multi-Reward Con- sensus and corrects noisy labels through iterative self-training. By leveraging timestep-conditional pseudo-labels, our method effectively decouples conflicting dimensions. Experiments confirm that Semi-DPO achieves state-of-the ooftthe-aee art performance, significantly improving aligmment with complex human preferences without requiring additional annotations or explicit reward models. Moreover, findings highlight the intrinsic capability of diffusion models to serve their own latent reward models, paving the way for more scalable, self-contained alignment frameworks that are robust to real-world data imperfections.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our research enhances lext-to-image model alignment while relying solely on publicly available resources, specifically the Pick-a-Pic V2 (Kirstain et 2023) dataset, SD1.5 Rombach al. (2022) and SDXL (Podell et al. 2023). In line with responsible research practices, we acknowledge the broader societal implications and potential for misuse associated with generative technologies.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we will release our code. Our method is detailed in Section 3.2 and our training hyperparameters are documented in Appendix 6.9 allow for the replication of results.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference machine learning, pp- 312-321, PMLR, 2019.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.1330/ 2023.
Paola Cascante-Bonin Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Re- visiting pseudo-labeling for semi-supervised learning. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6912-6920, 2021.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth ICLR, 2024. URL htt.ttsss//peeeeeeiieee net/
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.1580) 2023.
Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh. Qiang Zhang, Yan Kang, Zhe Lin, and Yuchen Liu. Dollar: Few-step video generation via distillation and latent reward optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 17961-17971, 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. volume 33, pp. 6840-6851, 2020.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. arXiv preprint arXiv:2406.004424 2024.
Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. Tiicoompeench+:: An enhanced and comprehensive benchmark for compositional text-to-image generation. JEEE Transactions Pattern Analysis and Machine Intelligence, 47(5):3563-3579 2025.
10

---

Published as a conference paper at ICLR 2026
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to- -oxt-to-image generation. Advances in neural information processing systems, 36:36652-36663, 2023.
Kyungmin Lee, Xiahong Li, Qifei Wang, Junfeng He, Junjie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi-preference optimization for aligning diffusion models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18465-18475, 2025.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. arXiv preprint arXiv:2002.07394, 2020.
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. io/controlnet plus plus. European Conference on Computer Vision, pp. 129-147. Springer, 2024a.
Ming Li, Jie Wu, Justin Cui, Xiaojie Li, Rui Wang, and Chen Chen. Vipo: Visual preference optimization at scale. In The Fourteenth International Conference on Learning Representations, 2026.
Shufan Li. Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. Advances in Neural Information Processing Systems, 37:24897-24922 2024b.
Zhanhao Liang. Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post- eesttraining diffusion models from generic preferences with step-by-step preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13199-13208, 2025.
Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang, Xiaoyin Xu, and Min Zhang. Inpo: Inver- sion preference optimization with reparametrized ddim for efficient diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28629-28639, 2025.
Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang. Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10844-10853, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35; 27730-27744, 2022.
Pablo Pernias, Dominic Rampas, Mats L Richter, Christopher J Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. arXiv preprint 2tt.:2330600063.7 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01152. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 23, Red Hook, NY, USA, 2023a. Curran Associates Inc.
11

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-5333 53741, 2023b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-25294 2022a.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurlPS, 35:25278- 25294, 2022b.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 34(11) (1)811)):33--1 8153, 2022.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Gohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arX1v:2011.13456, 2020.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-8238, 2024.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training pop-performing reward models. Advances in Neural Information Processing Systems, 37:1474- 1501, 2024.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13726-13735, 2020.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306.099 09341, 2023a.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303. 14420, 1(3), 2023b.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698. 2020.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 15903-15535, 2023.
12

---

Published as a conference paper at ICLR 2026
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In CVPR, pp. 3941-8951, 2024.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International conference on machine learning, pp. 7164-7173. PMLR, 2019.
Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling. NeurIPS, 34:18408-18419, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 16xiv:661..3355 2016.
Sixian Zhang, Bohan Wang. Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimensional human preference for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018-8027, 2024.
Tao Zhang, Cheng Da, Kun Ding, Huan Yang, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model noise-aware latent reward model for step-level preference optimization. arXiv preprint arXiv:2502.0100 20.01100....,,..... 2025.
Huaisheng Zhu, Teng Xiao, and Vasant Honavar. DSPO: Direct score preference optimization for diffusion model alignment. In The Thirteenth ICLR, 2025. htRps::::: htthttps:: //tp///eeee..eeeeeeeeeee net / Eorum? id d- xyfb9HHvMe
13

---

Published as a conference paper at ICLR 2026
APPENDIX
6.1 DERIVATION OF DIFFUSION-DPO GRADIENT
We derive the gradient of the per-timestep Diffusion-DPF loss, (DPO (0), with respect the model parameters The loss defined as:
(e) (0) = - log (8 log po (x(x xx, ,c) log po(x |x/ c)
c)) Prof / |x(, c) 1)
(11)
Peet Pret 1/xx
To simplify the notation, let 110 = 3 loog no px(xx 1 10.
Do (x (x,,0) log
Applying the chain
(xee(xp 1- 1800 180,,0)
Prot [2
rule and using the identity dz
- (x))), have:
o Fpp
Vo
(- log logg(())))
d
(-logo(hh)) Vohg
dhg
(1 -o((h))) Vohk
(12)
po (x] xx ,))
pul (x|-111 |x/, c)
- (1 o(h)))) log
Pref (x |x( ,,,,, log
Pref (x Xx/
= (1 o(h (Vo Po | 12, - V log pogpo(x log -
6.2 PROOF OF VARIANCE INFLATION
For clarity, we first restate the relevant definitions from Section 3.2 Let be the dataset prefer- ence tuples d (c,2%' =n)). For any sample d DD and given preference dimension k, we define the reward difference as Ar (d) == Fк (xi c) - Tk (a). c). This induces partition of the dataset into alignment set A4 == {d D Ark > 0} and conflict set C. == {d E D Arx
0), with respective probabilities Pa.k = P (dd E As) and Pe, = P E Ck). For a given sample
d and timestep f, we define the per-sample gradient Do
(e) = - f(") Adg the oracle direction
en (0, t) := sign (Ark) Aøg and the inner product & = (-90 90 (0, Finally, we define the conditional expected magnitudes as
(t)
(4) m k = (t)
Ao
(1)
Ax
and makk = E (
1100
(1) ||
Cx
2
By the law of total variance, we can decompose the variance of & based on whether sample is in the alignment set Ag or the conflict set Cą:
Var &
E Var Z]] + Var ZJ)
(13)
Intra-group variance
Inter-group variance
where is a random variable indicating membership in (As. Cx)-
Since variance is non-negative, the total variance is bounded below by the inter-group variance term:
Var[gu]
Var(E[E. Z])
(14)
We now compute the Var (E (S. Z)). The conditional expectations of & are:
ElG An] E(((") (+1)
(t) 10 As]
(0) EE Cx] ElS!" ((11) 1440 (1) 1 Cx) -m
14

---

Published as a conference paper at ICLR 2026
The Var [E [e Z]) can now be calculated using its definition, Var(Y) E[Y (B[Y))?. Let
= E Z/. Then:
Var(E[<
2] Pa.a(m (1) vcx(--mak) (pa,am -aaam (0)²
Pa, - Po, Po,N)(m Pa, (1- - Po.a)(m (1))2 2 Pma (1) (t)
(15)
=PakPac (m + (m).)) + 2mm
Pa,kPe.k (m a,k + m c.k)
where we used the fact that Pa,k + Pe,k =.
Thus, the total variance has lower bound determined by the conflict
Var[ 2 Po.k Pe,k (m + m m²,k)
(t)
(16)
This proves that any non-zero conflict mass (Pe.k > 0) introduces variance term that grows quadrat- ically with the sum of the conflicting and aligned update magnitudes.
6.3 THE USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) such as GPT were used solely for language polishing and clarity improvements in the writing of this paper. All technical content, dataset design, experimental results, and analyses were created by the authors. The models were not used to generate ideas, data, or experimental outcomes.
6.4 LIMITATIONS
Like many classical SSL methods (Xie et al., 2020: Zhang al. 2021 Cascante-Bonilla et al., 202T), our approach requires multiple cycles of greudo-labeling and model retraining. However, detailed in Appendix 6.9 this iterative process does not result in higher computational costs; in fact, our method is more efficient than the single-stage bascline (132 vs. 192 GPU hours) due to the effective filtering of noisy data. Therefore, the primary limitation lies not in computational resources, but in the operational complexity of managing a multi-stage pipeline. Future research could explore unified, single-pass frameworks mitigate this engineering overhead.
6.5 COMPARISON ONLINE AND OFFLINE DPO PARADIGMS IN IMAGE GENERATION
The methodologies for Direct Preference Optimization (DPO) in diffusion models can be broadly categorized into offline and online approaches. Offline campling-based DPO T2I, the paradigm under which Semi-DPO operates, utilizes a static, pre-collected dataset of human preferences for the entirety of the training process. This approach computationally efficient and offers greater stability and reproducibility, as it fine-tunes the model in single stage fixed dataset. However, its primary limitation is that the model's 's ultimate performance is fundamentally constrained by the diversity and quality of the initial preference data; it cannot learn beyond the scope of the examples is given (Wallace et al.) 2024 Zhu et al.) 2025 Lee et al. 2025 Hong et al. 2024).
In contrast, online sampling-basee DPO in T2I involves dynamically generating new preference data during the training loop. In this setup, the diffusion model iteratively produces new images that are then evaluated, typically by auxiliary reward model, create new preference pairs for continued training. While this allows the model to learn continuously and potentially surpass the quality of the initial dataset, introduces significant computational overhead and complexity. Furthermore, online methods risk overfitting to the biases of the reward model and depend on having reliable reward signal available throughout the resource-intensive training process (Liang et al.,
2025
Black et al. 2023 Zhang et al. 2025 Yang et al. 2024).
6.6
COMPARISON WITH EXISTING METHODS
Offline DPO. Unlike existing offline approaches that focus on modifying algorithmss (Zhu et al.
2025
[i al. 2024b Hong al. 2024) generating new datasets through multiple reward models
15

---

Published as a conference paper at ICLR 2026
for re-annotation and training (Lee et. 2025), Semi-DPO addresses the noise label problem caused by multi culli-dimeesional preference conflicts in DPO datasets. We reclassify the dataset into clean labeled data and noisy unlabeled data. Following the principle of semi semi-supervisee learning, Semi- DPO leverages both laheled and unlabeled data to achieve superior performance compared to using labeled data alone. This approach maximizes the utilization of existing datasets.
Our theoretical analysis and empirical results serve dual purpose: validating our central claim that collapsing multi-dimensional preferences into single binary labels introduces significant noisy label problem, and demonstrating that diffusion model trained with the DPO loss can correct these noisy labels by acting its own implicit classifier.
Latent Reward Model. Semi-DPo leverages key property of the Iiffusion-DPo framework: its loss function transforms the diffusion model into an implicit latent reward model. The principle is straightforward: the DPO objective compels the model to increase the relative probability of pre- ferred samples over dispreferred ones. To achieve this, the model must learn distinguish between "better" and "worse" latent representations at every timestep, capability that serves as inherent reward signal.
This approach contrasts with existing latent reward model methods (Zhang et al. 2025 Ding et al., 2025). Those methods typically require architectural modifications to construct an explicit reward model that is then trained separately. In contrast, the latent reward model in Semi-DPO is the original diffusion model itself. This implicit method is more efficient and requires no architectural changes.
6.7 FUTURE WORK: AN ONLINE EXTENSION THE SEMI-DPO PARADIGM
Pixel-space reward models (e.g., ImageReward) constrained by gradient propagation issues, which restrict their effective training signal to only the final stages of the diffusion process. La- tent reward models overcome this limitation by providing deep training signal across all timesteps (from / = 999 down to = ), allowing them to guide the entire generation process.
However, existing latent reward models (Zhang al., 2025; Ding al., 2025) have significant limitation: they are architecturally specific. They must share latent space with the diffusion model they are training, which requires shared VAE encoder. As different generative models (e.g., SD1.5 and SDXL) use different VAEs, a latent reward model trained for one is incompatible with another. This gives them great power but prevents the plug-and and-play versatility of pixel-space models.
Our work offers a path to resolve this trade-off. At its core, diffusion model trained with the DPO loss functions as an implicit latent reward model, learning to assign preference labels each timestep. This insight allows for powerful online extension of the Semi-DPO paradigm:
Cold-Start: An initial model is trained on small, multi mulii-dimensionally consistent dataset to learn the basics of human preference.
Online Training: During online training, the model inherits our iterative self-training phi- losophy. To begin the + 1) iteration, new data is generated by the model from iteration and then labeled by an ensemble of implicit reward models composed of the models from iterations and i 1.
By developing this iterative, self-trainingg online method, we would no longer need to train a new, bespoke latent reward model for each T2I architecture. Instead, the model would correct itself through an internal process. This presents a path toward a universal, model-agnostic alignment strategy that captures the deep-signal benefits of latent-space rewards without being constrained by their architectural limitations.
6.8 MOTIVATION BY
REWARD SELECTION
In our methods (see Section 3.3 we employ committee of five proxy reward models PickScore (Kirstain et al. 2023 HPS V2 /Wu et al., 2023a), CLIP (Radford et al. 2021). LAION Aesthetics Classifier (Schuhmann et al. 2022b), and ImageReward (Xu et al. 2023) for data fil- tering. This approach is motivated by the study that introduced the Multi Score (MPS) (Zhang et al. 2024), which constructed dataset reflecting real human preferences by
Preference
16

---

Published as a conference paper at ICLR 2026
Table 6:
The evaluation of MPS and scoring functions for the prediction of multi- dimensional human preferences(*) Copied from MPS (Zhang al. 2024).
ID
Preference Model
Overall
Aesthetics
Alignment
Detail
CLIP score Radford et al.
2021
63.67
68.14
82.69
61.71
2
Aesthetic Score Schuhmann et al. 2022a
62.85
82.85
69.36
60.34
3
ImageReward Xu et al.
2023
67.45
74.79
75.27
58,31
HPS Wu et al. 2023b
65.51
73.86
73.86
62.05
PickScore Kirstain et al. 2023
69.52
70.95
70.92
56.74
HPS
2023a
65.51
73.86
73.87
62.06
MPS
Zhang et al.
2024
74.24
83.86
83.87
85.18
Overall score
Aesthetic
Semantic alignment
Detail quantity
Re0.185
R=-0.002
R=0.540
R= 0.155
CLIP score
Ru0,573
R=0.439
Rs-0345
R=0.105
PickScare
Re0/702
Rs0.710
0.8.86
MPS (Ours)
Figure 4: Correlation between real user preferences and model predictions. The x-axis of each subplot represents the annotated real human preferences. and the y-axis denotes the model's predictions. We examine three models: CLIP score, PickScore, and MPS. Each subplot is annotated with the calculated correlation coefficient R-value, where a higher R-value indicates a closer alignment of the model's predictions with actual human preferences. Figure reproduced from (Zhang et al. 2024)..
ensuring image source diversity and having human annotators perform pairwise comparison scoring across four dimensions: aesthetics, detail quality, semantic alignment, and overall assessment.
They compared the performance of multiple existing reward models against their proposed MPS on multi-dimensional preference dataset, revealing a core phenomenon: existing reward models exhibit sigmificant specialization when predicting human preferences (see Table and Figure 4 For instance, CLIP Score (Radford et al., 2021) shows strong correlation with the semantic alignment dimension, while Aesthetic Score (Schuhmann et al.] (022a) is highly correlated with the aesthetics dimension. Meanwhile, models such as HPSv2 (Wu et al.) 2023a), ImageReward (Xu et al., (023), and PickScore (Kirstain et al. (2023) show higher consistency with the overall score. This finding indicates that no single model can comprehensively evaluate image quality.
As shown in Table 6 and Figure 4 (Both Table and Figure 4 are reproduced from MPS paper. They are Tab. and Fig. relation with molti-dimensionall human preferences. However, a publicly available version capable of providing scores for individual dimensions was unavailable the time of our work. Therefore, to ensure our initial training data was reliable and free from label noise caused by dimensional conflicts, we used five distinct preference models in concert, selecting only the data points that all models agreed upon to form high-quality, dimensionally consistent subset. Figure provides visual comparison of these two subsets, highlighting the prevalence of label noise and dimensional conflicts within the data rejected by our consensus mechanism. To validate the effectiveness of this five-model filtering strategy, we also conducted an ablation study to investigate the impact of using different numbers of reward models 4.3
their original paper.), the MPS model shows the strongest cor-
17

---

Published as a conference paper at ICLR 2026
Winner
Loser
Winner
Loser
Lah Hannig Can Hurgrgiegl
mins
TEAK!
WET
dd 2OX
0934195
g CEN Arprges B; X 90m
SE TI'S TIVE ER LIITS I MITTIER
CAT LIVES MATTER
Figure 5: Qualitative comparison of samples from the labeled dataset (Left, Blue) and the unlabeled dataset (Right, Green). The annotations Winner (green) and Loser (red) indicate the human preference labels. The corresponding full prompts are: Top-Lefft: "Cosmic hamburger in space"; nottom-Leftt "Cat with a sign that says 'Cat lives matter' "; Top-Right: "An empowering view of a orca warrior wearing royal robe, sitting in cafe drinking coffee next to kangaroo warrior with eye menacing, by artist Philippe Druillet and Tsutomu Nihei, volumetric lighting, detailed shadows, extremely detailed";; Bottom- Right: "A wide angle photo of large gold head Caesar on display in a smokey roman villa burning, 18mm smoke filled room debris, gladiator, floor mosaics fire smoke, a photo, roman, a digital rendering. inside the roman colosseum, brick, indoor, plants overgrown outstanding detail, room flonded with water, in front of a huilding, by claude-joseph vernet, luxury hotel".
6.9 TRAINING DETAILS
For SD1.5, the training was utilizing total of 32 NVIDIA A100 40GB GPUs for distributed training. We configured local batch size of 4 for each GPU and performed gradient accumu- lation over 4 steps, which resulted in a global batch size of 512. Iteration 0 uses a learning rate of 4 10 and is trained for 1,600 steps, while iterations 1 and use learning rate of 4 x 10 10 and are trained for 4,000 steps each. The DPO parameter was set to B 2500 for all iterations, in line with the hyperparameters specified in the official Diffusion-DPO (Wallace) et al. 2024) code repository. All iterations incorporate a linear warmup over the first 400 steps.
Table 7: The model's prediction accuracy, evaluated on a clean test set, varies across the diffuusio timeline.
To implement our dynamic thresh- olds for pscudo-labeling (as men- tioned in Section 3.3 we first par- tition the diffusion timeline (t E [0, 999]) into ten discrete intervals (e.g.. 0-100, 100-200). We initially set the threshold for each interval at the 80th percentile of its confidence scores to ensure consistent number of samples are initially selected. However, when we evaluated this strategy on our accuracy test portion (3,992 pairs), we found that the model's pre- diction accuracy was not uniform across different timestep intervals (see Table 7 specifically, for timesteps greater than 650, its accuracy dropped below 70%. To mitigate confirmation bias from these less reliable labels, for any interval failing to meet the 70% accuracy level, we raised its con- fidence threshold. Although this adjustment reduces the number of pseudo-labeled samples from later stages of the diffusion process, ensures the model primarily trained on labels that are both high-confidence and high-accuracy.
Timesteps
50
150 250 350 450
550 650 750 850 950
Accuracy (%) 72 73 73
72
72 71 69 67 65 59
Computational Efficiency, Under the setup described above, completing the first two stages (Iter () and Iter 1) requires approximately 132 GPU hours. This is significantly more efficient than the
18

---

Published as a conference paper at ICLR 2026
standard Diffusion-DPO baseline, which necessitates 192 GPU hours. This efficiency gain stems directly from the dynamic thresholding mechanism described previously. which effectively filters out low-confidence samples from the noisy unlabeled dataset. While extending training to Iter increases the total cost to 228 GPU hours, the performance gains are marginal compared to Iter (see Table 4 indicating that the Iter 0 configuration offers the optimal balance between training efficiency and performance.
Inference & Memory Costs. Since our final model retains the exact same architecture as base model, the inference time and memory footprint are identical to the baseline, incurring additional computational cost during deployment.
6.10
ADDITIONAL QUANTITATIVE RESULTS
Table 8: Win rate (%) comparisons on Pick-a-Pic V2, HPS V2 and Parti-Prompt datasets. Comparisons are performed for both SD1.5 and SDXI. baselines.
Base
Dataset
Method1
Method2
ImageReward
HPS v2
PickScore
Aesthetic
CLIP
MPS
Diff-DPO
SD1.5
60.80%
70.10%
75.70%
68.10%
54.70%
67.20%
Diff-KTO
SD1.5
77.80%
90.40% %
74. 10%
72.20%
55.30%
68.60%
HPS v2
Semi-DPO
SD1.5
79.80%
88. 88.10%%
87.40 %
78.80 %
55.80%
73.70%
Semi-DPO
Diff-DPO
74.60%
83.00%
77.40 %
68.40 %
52.20%
66.40%
Semi-DPO
Diff-K KTO
56.10%
52.80%
72.90 %
60.60 %
51.20%
61.70%
Diff-DPO
SD1.5
58.90%
64.90%
67.90%
63.20%
50.80%
62.80%
Diff-KTO
SD1.5
69.80%
83.40%
65.20%
19.10%
57.30%
59.60%
Parti
SD1.5
Prompt
Semi-DPO
SD1.5
75.80%
83.40%
78.80 %
80.00 %
56.00%
71.70%
Semi-DPO
Dift-DPO
71.90%
78.80%
70.80 %
74.00 %
$5.00%
65.70%
Semi-DPO
Diff-KTO
60.20%
54.80% %
70.40 %
65.70% %
48. 48.60%
66.20%
Diff-DPO
SD1.5
63.20%
69.90%
74.60%
66.00%
58.00%
66.00%
Diff-KTO
SD1.5
75.50%
85.80%
73.40%
71.80%
60.90%
62.70%
Pick-a-Pic
V2
Semi-DPO
SD1.5
79.10%
87.30%
85.20
80.70 %
60.60%
77.00%
Semi-DPO
Diff-DPO
72.90%
82.10%
76,30 %
72.60% %
53,50 %
69.00%
Semi-DPO
Diff-KTO
60.20%
59.50 %
73.00 %
64.50 %
50.40 %
70.20%
Diff-DPO
SDXL
69.10%
80,90%
69.90 %
54.30%
59.70%
61.90%
MaPO
SDXL
64.60%
78.90%
55.80%
67.10% %
55.40%
56.10%
HPS v2
Semi-DPO
SDXI
68.90%
82.90%
60)40%%
62.50%
52.40%
64.70%
Semi-DPO
Diff-DPO
52.90%
52.80 %
41.90%
61.20% %
42.50%
54.40%
Semi-DPO
MaPO
60.10%
65.50% %
57.20 %
49.40%
46.60%
60.90%
Diff-DPO
69.20%
76.80%
63.80 %
55.70%
61.10%
63.60%
MaPO
SDXL
61.30%
70.20%
48.30%
70.80%
55.10%
57.10%
Parti Prompt
SDXL
Semi-DPO
SDXL
68.40%
80.90%
58.90%
71.30% %
56.80%
70.70%
Semi-DPO
Diff-DPO
51.40%
58.00%
45.10%
70.20 %
46.30%
59.40% %
Semi-DPO
MaPO
62.20% %
71.00% %
61.20 %
55.20 %
53.40 %
67.50%
Diff-DPO
SDXL
68.00%
77.80%
71.20%
48 48.80%
63.30
61.40%
MaPO
SDXL
66.50%
72.40%
53.70%
67.50 %
55.80%
51.60%
Pick-u-Pic V2
Semi-DPo
SDXL
74.00% %
82.00%
65.00%
65.30%
57.20%
70.00%
Semi-DPO
Diff-DPO
56.20%
55.20%
42.90%
68.70% %
44.00%
60.50%
Semi-DPO
MaPO
63.90%
71.20%
62.90% %
50.90%
53.80
69.30%
19

---

Published as a conference paper at ICLR 2026
6.11
ADDITIONAL QUALITATIVE RESULTS
SD1.5
Diffusion DPO
Diffusion KTO
Semi-DPO (Ours)
pathering,
furry knight adventurer,
proms Painted impresisuist style
Anthropomorphicc
morring ceherpusl
eyberponk futuristic city hyper realistic
drumaati libriing
crowded street, bocy street, cluitrred, chosic, hyper detuiled, intricalee,
di, unreal engine, maya render
space warrier
stapls, hohl dynamic composition,
bright sstrattd huns, strong condracts, sibrant, energetie clahorate, hyper detiiled risually stunning
and captivating art style
altra
photorcalitii.
detailed image
sogine,
pase
eyberpusk, erganic
metemparary
hiremy
cathedral
stasmommy night
Figure 6: Qualitative comparison of Semi-DPO against baseline models
20


Reviews:
[{"review_id": "iDI8yTyrVb", "reviewer": "Reviewer_2XbV", "summary": "The authors argue that compressing multi-dimensional human preferences into binary labels creates conflicting gradient signals during Diffusion-DPO training. They propose a two-stage method: 1) “Multi-Reward Consensus” filters a dataset into clean and noisy subsets based on the unanimous agreement of five pre-trained reward models, and 2) “Iterative Self-Training” uses a model trained on the clean set to generate timestep-conditional pseudo-labels for the noisy set, which are then used for further training. Experiments on Pick-a-Pic V2 show improvements over Diffusion-DPO and Diffusion-KTO baselines.", "questions": "1. The SSL literature has many techniques for handling noisy labels (e.g., co-teaching, robust losses). Why was the self-training/pseudo-labeling paradigm chosen over others?\n2. Can further comparative experiments be conducted on T2ICompbench?\n3. Is the training time, memory footprint, or inference time comparable?\n4. Are there any failure modes observed?\n5. How does your model perform in controlled generation tasks, and could it lead to the loss of certain properties?\nThe final score is contingent upon the following revisions: For score 4 :supplemental baseline comparison mentioned，For score 6: Provide experimental efficiency metrics and comparative results on the T2ICompBench.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The Semi-DPO framework is a clever and practical solution: Using a committee of diverse reward models for robust data partitioning. Leveraging the diffusion model itself as an implicit, timestep-wise preference classifier. The dynamic, timestep-conditional thresholding for pseudo-label selection, which accounts for the varying reliability of the model across the denoising process.\n2. The paper excels at identifying and formalizing a fundamental issue in preference learning for generative models. The argument that multi-dimensional preferences are a source of label noise is well-illustrated.", "weaknesses": "1. The innovation is quite limited, and I believe the “Multi-Reward Consensus” in the first stage is very similar to CaPO. It involves using multiple rewards for evaluation and selecting samples where the winner outperforms the loser across all dimensions for learning. Additionally, the idea of phased training is somewhat similar to [5], relabel is somewhat similar to [8].\n2. There is a lack of comparison with many baselines.  like [1]SPO(SD1.5 SDXL), [2]DDIM-InPO(SD1.5 SDXL), [3]Flow-GRPO(SD3.5-Medium) [6]DSPO (SD1.5) [7] MaPO(SD1.5 SDXL). These baselines (checkpoints) are open-sourced.\n \n   [1]Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization. CVPR2025\n\n   [2]InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment. CVPR2025\n\n   [3]Flow-GRPO: Training Flow Matching Models via Online RL\n\n   [4]Calibrated Multi-Preference Optimization for Aligning Diffusion Models. CVPR2025\n\n   [5]Curriculum Direct Preference Optimization for Diffusion and Consistency Models. CVPR2025\n\n   [6] DSPO: Direct Score Preference Optimization for Diffusion Model Alignment. ICLR2025\n\n   [7] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference\n\n   [8] Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences. ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIlANfoW0n", "reviewer": "Reviewer_f4ae", "summary": "This paper points out a key issue faced by the current image preference alignment methods based on Diffusion-DPO: human visual preferences are inherently multi-dimensional (such as composition, details, semantic alignment, aesthetics, etc.), but existing preference datasets only provide a single \"overall winner\" label, leading to significant label noise. The authors theoretically analyze that compressing such multi-dimensional preferences into binary labels will cause conflicting gradient signals, thereby hindering model optimization. To address this problem, the authors propose Semi-DPO, a new method that reconfigures preference alignment as a semi-supervised learning problem. This method first uses the consensus of multiple pre-trained reward models to filter out \"clean\" preference pairs as labeled data, and the rest are regarded as noisy unlabeled data. Then, through iterative self-training, it uses the model itself as an implicit classifier to generate fine-grained pseudo-labels at different time steps of the diffusion process, thereby decoupling conflicting signals. Experiments show that Semi-DPO significantly outperforms existing methods on multiple metrics, reaching state-of-the-art (SOTA) levels, without the need for additional manual annotations or explicit reward models.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical analysis is solid: through the derivation of the lower bound of gradient variance, it rigorously proves how dimension conflicts lead to training instability, providing theoretical support for method design.\n\n2. The method design is ingenious: combining the DPO framework with semi-supervised learning, it utilizes the inherent discriminative ability of the diffusion model in DPO training to generate time-step conditional pseudo-labels without modifying the model architecture.", "weaknesses": "1. The method relies on multiple rounds of self-training (2–3 rounds are used in the paper) and multi-reward model consensus filtering, making the training process more complex and time-consuming than standard DPO (see Appendix 6.4), which may limit its application in resource-constrained scenarios.\n\n2. The initial clean set only accounts for approximately 21% of the original data. If the consensus reward model itself has systematic biases (e.g., poor performance on certain prompt types), it may affect the quality of cold start.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzLSNUuwix", "reviewer": "Reviewer_KSxh", "summary": "This paper is about generating T2I results that can incorporate human preferences by going beyond of collapsing the multi-dimensional preference to a single binary indicator. The paper proposed a method that by recognizing human preferences are of multi-dimensional and noisy (or in other words, emphasizing various aesthetics perspectives). The method is to divide a training set into clean and noisy subsets and start the training of a model with the clean subsets. The model then moves on to the noisy preference subset to train iteratively to align the model's generation with human preference.", "questions": "Please see weaknesses.\nIt would be helpful if the authors could give examples of \"noisy labels\", as well as some examples of multi-dimensional human preferences.\nDoes iter 0 mean no refinement and only the clean subset was used?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strengths of the work is the recognition of the noisy nature of human preference in assess AI-generated images. Building upon this strength is the strength of highlighting the drawback of simply classifying a generated image as winner or loser as do so would collapse the original multi-dimensional human preference to an overly too simple binary criterion. Another strength is the division of a training set to clean and noisy subsets and the design of method that iteratively trains on the noisy subset to align the model's output iteratively with human preference.", "weaknesses": "It is not clear, given a training set, when it is divided into clean and noisy preference subsets along dimension k, does it mean for a different dimension, the clean and noisy subsets will be different.\nIt is unclear how the loss function of Eq. (8) was derived. It seems from the beginning of the paper, the iterative training only applies to the noisy preference subset, which is understandable, but then why for iterative refinement step, Eq. (8) includes L_labeled? Shouldn't Eq. (8) only need to be concerned with L_unlabeled^i for each iteration i?\nContrary to what the paper claimed in line 418, it seems there was not big improvement from iter 1 to iter 2 in ablation study. It seems that the improvement came from iter 0 to iter 1. From this perspective, it would likely suggest that iterative refinement is not necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)