Diffusion-RainbowPA: Improvements Integrated Preference Alignment for Diffusion-based Text-to-Image Generation
Authors: Haoyuan Sun
Bin Liang
Bo Xia
Jiaqi Wu
Yifei Zhao
Kai Qin
Yongzhe Chang
Xueqian Wang
Venue: Accepted by TMLR ()

Abstract: Although rapidly increasing capabilities of text-to-image (T2I) models have profound implications across various industries, they concurrently suffer from numerous shortcomings, necessitating the implementation of effective alignment strategies with human preference. Diffusion-DPO and SPO have emerged as robust approaches for aligning diffusion-based T2I models with human preference feedback. However, they tend to suffer from text-image misalignment, aesthetic overfitting and low-quality generation. To tackle such matters, we improve the alignment paradigm through a tripartite perspective, which are the calibration enhancement (Calibration Enhanced Preference Alignment), the overfitting mitigation (Identical Preference Alignment, Jensen-Shannon Divergence Constraint) and the performance optimization (Margin Strengthened Preference Alignment, SFT-like Regularization). Furthermore, combining them with the step-aware preference alignment paradigm, we propose the Diffusion-RainbowPA, a suite of total six improvements that collectively improve the alignment performance of Diffusion-DPO. With comprehensive alignment performance evaluation and comparison, it is demonstrated that Diffusion-RainbowPA outperforms current state-of-the-art methods. We also conduct ablation studies on the introduced components that reveal incorporation of each has positively enhanced alignment performance.

Published in Transactions on Machine Learning Research (07 (07/2025)
Improvements Integrated Preference Alignment for Diffusion-based Text-to-lmage Generation
Haoyuan Sun
Tsinghua University
Bin Liang Universily of Technology Sydney
bin. din.lionn@uuts edu.ou
Bo Xia* Tsinghua University
siab2 www.nnill.tinnguu..........
Jiaqi Wu Tsinghua University
mu.jjg//mmail..sinnggee edu.cn
Yifei Zhao Tsinghua University
zhaogife @mmaal.. edu.co
Kai Qin Tsinghua University
tank24 dhe
Yongzhe Chang
chenggonghhe@sss.tiinnuu.....
Tsinghua University
Xueqian Wang"
wong wwng.zzg@sz.tsinghuu..ll
Tsinghua University
Reviewed on OpenReview: http:://oorrrvwww.ee///////taaaa
Abstract
Although rapidly incrcasing capabilities of text-to-image (T2I) models have profound im- plications across various industries, they concurrently suffer from numerous shortcomings, necessitating the implementation of effective alignment strategies with human preference. Diffusion-DPO and SPO have emerged robust approaches for aligning diffusion-based T2I models with human preference feedback. However, they tend to suffer from text-image misalignment, aesthetic overfitting and low-quality generation. To tackle such matters, we improve the alignment paradigm through a tripartite perspective, which are the calibration enhancement (Calibration Enhanced Preference Alignment), the overfitting mitigation (Iden- lical Preference A Alignment, Jensen-Shannon Divergence Constraint) and the performance optimization (Margin Strengthened Preference Alignment, SFT-like Regularization). Fur- thermore, combining them with the step-aware preference aligument paradigm, propose the Diffusion-RainbowPP A,, a suite of total six improvements that collectively improve the alignment performance of Diffusion-DPP. With comprehensive alignment performance evalu- ation and comparison, is demonstrated that state-of- the-art methods. We also conduct ablation studies on the introduced components that reveal incorporation of each has positively enhanced alignment performance.
Raaniio--R A perferos current
"Corresponding authors: Bo Xia, Xuegian Wang

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Figure In this study, we propose the Diffusion- Cettsioo-RaanbowPhh method that integrates six improvement components, collectively designed to enhance the alignment performance of Diflusion-DPO Wallace al. 2024). As shown in samples above, after the fine-tuning process Stable Diffusion v1.5 model, our method has yielded images that possess an extraordinarily high level of visual allure and text alignment.
1 Introduction
"The whole is greater than the sum of its parts."
Aristolle
As generative capabilities of diffusion-based text-to-image (T21) models advance, the challenge of aligning them with human preference has garnered extensive attention within the community. In relatively carly works, alignment was commonly achieved through supervised fine-tuning (SFT) Reinforcement Learning from Human Feedback (RLHF) (Black et al. 2024 Fan et al. 2024). While these methods are effective, they are hindered by convoluted implementation process. With notable success of Direct Preference Optimization (DPO) (Rafailov et al.) 2024] in aligning Large Language Models (LLMs), preference- based alignment for T2T diffusion models offers promising methodology that eliminates the need of reward modeling, thus strcamlining training process. Diffusion-DPO (Wallace al.] (0244) and D3PO (Yang al. 2024a) take the pioneering step for scaling of diffusion model aligument methods, introducing innovative alignment paradigm that learns from human preferences.
In the past year, numerous studies have been developed based on the paradigm (detailed in Section and Appendix Bj. In particular, the recent. SPO (Liang et al. 2022) introduces step-aware preference paradign lo address the misalignment between preference label and denoising performance at each stage. Nonetheless, there are still limitations persist. Firstly, as demonstrated in Table | exccution of human preference alignment could worsen the misalignment between text and image. Furthermore, as depicted in Figure they often tend to suffer from aesthetic overfitting and low-quality generation.
2

---

Epoch 10 Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Epoch
Epoch
Epoch 10
Epoch 1
Epoch 5 Diffusion-RainbowPA
SPO
In the aligmment training process 10 epochs, generative examples of SPO (Liang et al.) 2024 (left) and Diffusion-Rainle Rainboww.A (right) with the prompt of "A beautiful lake". It can be observed that SPO leads to aesthetic overfitting, resulting in scenes with an unauthentic presence of snowy mountains adjacent to the lake, accompanied by chaotic coloration and excessive glare, thereby degrading quality of generated images. Conversely, Leffasion-RainbowPP has effectively mitigated such issues, providing an authentic depiction of scene and a natural rendering of color and light.
Figure 2:
To address such issues, improve alignment. paradigm of diffusion-based T2I models from three aspects: the calibration enhancement (Section 4.2p, the overfitting mitigation (Section 4.3) and the performance optimization (Section 4.4) In order to achieve such goals, we conduct an exhaustive investigation into the current alignment methods for LLMs (Wang et al. 2024 Winata et al. 2024) and carefully study their feasibility for adaptation to diffusion models (DMs). For the purpose of calibration enhancement, we introduce the Calibration Enhancement Preference Aligument (CEPA), introducing the calibration terms between the scaled ground-truth reward and the implicit reward function. In order to mitigate the issue of overfitting, we introduce the Identical Preference Alignment (IPA) and the Jensen-Shannon Divergence Constraint. Identical Preference Alignment (Section 4.3.1 skillfully circumvents the Bradlcy-Terry modcling assumption and employs identical mapping to the preference function, thereby mitigating alignment preference overfitting, Jensen-Shannon Divergence Constraint (Section 4.3.2) substitutes Kullback-Leibler divergence with Jensen- Shannon divergence to stabilize alignment training and alleviate overfitting. Moreover, analysis of DPO-based methods for diffusion-basee based T2I models from the contrastive loss perspective shows that they only consider the "dissimilarity" part. Based on such observation, we introduce two performance optimization lechniques: the Margin Strengthened Preference Alignment (MSPA) and the SFT--like Regularization. To accelerate increase of the diffusion win ratio and the margin between ratios, introduce the Margin Strengthened Preference Alignment (Section 4.4.1). Furthermore, the SFT-likee Regularization (Section 4.4.2 is proposed, which introduces term to improve the impact of positive items. Based on the aforementioned five improving components that grounded in three perspectives, further integrating the step-aware preference alignment paradigm (Section 4.1b we propose the Diffusion-Raine RainbowPA (Section 4.5 a novel method that integrates total six improvements on the Diffusion-DPO.
In the experiment evaluation, we utilize the Stable Diffusion v1.5 (Rombach al., 2022 our benchmark model, which is widely recognized and frequently utilized within the domain of T2I model alignment (encompassing both preference-based and RL-based methodologies). We evaluate the alignment performance
with four published automatic metrics: (VQAScore [Lin ct al.) 2025), CLIPScore (Radford et al. 2021
HPS-V2 (Wu et al. 2023) and IeageRewwar (Xu et al. ]024]} on four zero-shot datasets: [GenEval Ghosh
et al. 2024). 121-CompBench++ (Huang et al. 2025). GenAl-Bench (Li et al., 2024a). and
HHu
et al. 2024 ). Results (Section 5.2 indicate that alignment performance of Diffusion- RainbowPA outperforms
current state-of-the-art (SOTA) methods. In Figure 1 we show generation examples of the aligned model that utilizing
Further ablation study (Section 5.3) the introduced components
3

---

Published in Transactions on Machine Learning Research (07/2025)
of Iiffusion-Rainbow22 demonstrates that each component has positive impact on the improvement of alignment performance.
Our contributions are summarized as follows:
We point out that current existing SOTA methods still tend to suffer from text-image misalignment, aesthetic overfitting and low-quality generation.
2. To address such issues, improve the alignment paradigm through tripartite perspective: the calibration enhancement, the overfitting mitigation and the performance optimization.
We propose the Diffusion- Diffusion-RRinbowP.. RainbowPA, a novel method combining total six improvements on Diffusion- DPO, which outperforms current SOTA methods.
2
Related Work
Fueled by resounding success of Direct Preference Optimization (DPO) (Rafailov et al. 2024). which has revolutionized alignment processes for LLMs by obviating the requirement for explicit reward model, Diffusion- DPO (Wallace et al.] 2024) and D3PO (Yang et al.)
2024a have taken the forefront in aligning diffusion-based T21 models with human preferences. Amid their remarkable success, several subsequent works (Yang et al.,
2024) have expanded upon their foundation. For instance, -iffuuifusio Yuan et al., 2024), empowers the model to outperform its predecessors, thereby fostering emphasized that there is a discrepancy between the preference for the final image and the performance each denoising step. To tackle such challenge, step-aware preference strategy is effectively deployed. Recently, in the study divergence for Kullback-Leibler
2024b Li et al. 2024b Gu el. al., 2024 Hong et al.
an itcrative cycle of continuous self-improvement. In the work (Liang et al. 2024),
(unn et al. 2025b), the authors suggest that substituting Pensen-Shanmin divergence can significantly improve alignment performance. ScPPO (Zhang et al. 2024 employs the concept of online sampling, leveraging previously saved checkpoints as reference models, and has become one of the state-of the-----tt--tt methods. We provide more details about diffusion- based T21 models in Appendix [A] and offer more in-depth and detailed discussion on iiffusion-bbsed T2I alignment in Appendix [B] Although they have achieved promising results, challenges remain, including text-image misalignment, aesthetic overfitting and low-quality generation. This study adopts integrated approach to address them concurrently.
3 Preliminaries
In this part, provide brief description on the Diffusion-DPO and the SPO for readers who are unfamiliar with diffusion-bass text-to-image alignment.
3.1 Diffusion-DPP DPO
Within framework of diffusion models, Diffusion-DPO (Wallace el al. 2024] extends the optimization objective from pa(zalc) to po(zorrle). By incorporating latent variables 31.7, an implicit reward model is established
across the entire sequence. Based on this, the optimization objective pertaining to the conditional distribution can be formulated as follows:
log a Bog (o( 33 log po(aburle) Pret(x))
((() -E
(1)
(e.ar WD. pp (x (1 .)). 21 (uf (4(.))
pro((zbrrlc))
where a fixed dataset D {(c, +/)} is employed with c denoting the input prompt, x indicating the human preferred generation and of signifying the dispreferred one; po(xo:r(c) represents the reverse process parametrization, Pret(Xoorle /c) pertains to that of the reference model; is the measure of regularization intensity and a(.) is the Sigmoid function.
4

---

Published in Transactions on Machine Learning Research (07/2025)
Through application of reverse decompositions to pe and Prefs leveraging Jensen's inequality and convexity of function log the authors derive the following upper bound optimization:
Po po(=;-1le, x44
BTllog Po(s 1 (c,
((0) S - 18 (e,e ( log #1-1, po (1 (o,00 o 11 (_. (ca5))
BT 1T Pret(#ū-1 [()))
(2)
Pret (2 xx4)
3.2 Step-aware Preference Optimization
In the work [Liang et al. 2024), it is noted that previous methods, such as Diffusion-DPO (Wallace et al., 2024) and D3PO (Yang et al. 2024a), solcly evaluate the whole generation trajectory based on the final image wo, assigning all intermediate states the same preference 20. Hence, based on such observation, the authors introduce Step-aware Preference Optimization (SPO) methodology. It primarily hinges two key aspects. Firstly, during cach denoising step, image pair is determined from the gencrated image subset, and the winner and loser are thereby ascertained. To achieve the target, a step-aware preference model is trained to synchronize with denoising performance. Furthermore, a step-wise resampler is designed to eliminate trajectory-level dependencies. Merging objectives across all T timesteps, the ultimate objective function for SPO training can be derived follows:
log Blog Po 1 ll.,t, Prot( (,,t 24)
B log po(24-1le,t,1 t,, iic, Pret(x) Ic, xx)
[(0)
(3)
(wf- (0--::--1)Np00e000 (=
where the preference pair 1=--1.4/11 1), comprising the most preferred item x;-1 and the most dispreferred item 2|_1 from the sampling set {at 22 1..., 1
4 Method
Diffusion-DPO (Wallace et al. 2024) has emerged as a significant milestone in bridging the gap of align- ment paradigm between large language models (LLMs) and diffusion models (DMs). Recently, numerous improvements have been proposed to enhance the preference-based alignment, paradigm of DMs. However, they still tend to suffer from the issues of text-image misalignment, aesthetic overfitting and low-quality generation. Subsequently, to mitigate such issues, improve the alignment paradigm with five additional components through tripartite perspeclive: calibration enhancement (Section 4.2 overfitting mitigation
(Section 4.3) and performance optimization (Section 4.4]. Further integrating the step-aware preference
alignment paradigm (Section 4.1 we propose the Seffusion-Rainbowe A (Section 4.5p, a suite of total six improvements integrated that collectively enhance alignment performance of DPf
4.1
Step-aware Preference Alignment Paradigm
In Section
3.2 we conduct a brief review of SPO, which advocates the employment of a step-aware preference model and a step-wise resampler facilitate precise step-aware supervision, while maintaining the same initialization throughout the subsequent denoising phase. Throughout our study, we consistently integrate the paradigı of step-aware preference and utilize the open-source step-aware preference model in our alignment. framework.
Morcover, unlike original SPO, we abandon multi-sample setup utilized in step-wise resampler (which samples images in SPO) and adopt a setup that samples 2 images. Such a choice is informed by the following observations, as depicted in the left side of Figure 2 it found that SPO leads to overfitting in aesthetic rating. This is further corroborated by Table of SPO (Liang et al. 2024), where, despite optimal acsthetic rating the sampling number of 4, human preference ratings (HPS- V2 and ImageReward) are the lowest.
4.2 Calibration Enhancement
As previously mentioned, the issue of text-image misalignment still persists. Hence, we introduce the Calibration Enhancement Preference Alignment (CEPA) to address it. Firstly, we draw inspiration from
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Cal-DPO Xiao al. 2024 and define the calibrated objective for diffusion-based preference optimization in Definition 1
Definition 1. If have B log po(xx-1lett,21)
pral#tt-1cct..z r(c, we call that the estimated implicit reward B (e.ttt)) for the sampling probability of DM pe is calibrated with the ground truth reward. log (11(211116322211)
Hence, we introduce the CEPA term with the purpose of constraining distance between the learned implicit reward and the ground-truth reward. Furthermore, it is observed that formula form of the term is similar to SPPO (Wu et al., 2024b, which treats alignment problem as constant-sum two-player game and approximate Nash equilibrium through iterative policy updates. Such observation further illustrates its effectiveness and inspires us to set the reward for preference feedback as: r(c,x5) x5") = 1/2 and r(c, xb) = -1/2. In practice, the setting works well. Then, we show terms of the CEPA as follows:
B log Po(21_1 C, x7)
1
Po po(r4-1let,,t) BBoog
1
1]2
CCEPA (0)
+ (4)
Prot(a | tc,t, 2
Pret(e!-1 xx)
From Equation we can see that CEPA foues on increasing the discrepancy between the diffussion win ratio and the diffusion lose ratio to a scale of 1. Furthermore, it endeavors to adjust the diffusion win ratio close to 1/2 and the diffusion lose ratio close to -1/2 for simultaneously increasing log- bog-likelhhood of the preferred item and decreasing that of the dispreferred one.
4.3 Overfitting Mitigation
Large diffusion models that have undergone extensive data pre-training are prone to overfitting during and alignment process (Gao et al. 2023 Clark et al. 2024 Kim et al.) 2025). In this study, we effectively mitigate propensity for overfitting during the alignment process by employing the Identical Preference Alignment (IPA) and the -hhen-Shannon Divergence Constraint.
4.3.1 Identical Preference Alignment
In the work (Azar et al. 2024). it is proposed that overfitting issue is partly due to the replacement of pairwise preferences with pointwise reward model, as implemented in Bradley-Terie Terry (Bradley & Terry 1952 model, encountering challenges when preferences are predictable or nearly predictable. Hence, in instances where the preference leans towards determinism, as is often the case in image generation, discrepancy between reward functions would asymptotically tend towards infinity, thereby substantially diminishing the offectiveness of divergence constraint enforced by B. Based on such observation (Azar et al. 2024) propose the IPO to tackle such challenge, which adeptly sidesteps the Bradley-Terry modeling assumption concerning preferences and applies an identical mapping to the preference function. Furthermore, the very recent work (Sun et al., 2025ch suggests that such paradigm can mitigate overfitting in the alignment process of text-to-image models, leading to improved alignment. performance and generation diversity. Hence, inspired by the idea of identical mapping, we introduce the Identical Preference Alignment (IPA) that further extends it to diffusion- based T2I alignment and concurrently merges the step-aware preference aligument paradigm:
2
log [po(="_1ee r/c, t, xt Preer (2
CIPA (0) (e.. (x1 (/ ile.a.:))
(5)
(po(s/_ Ilc, t, at) Pe)r (x t, xxr)
2,3
where B is the regularization intensity.
4.3.2 Lansen-Shannor Divergence Constraint
Kullback-Leible (KL) divergence has been typically chosen as the divergence constraint for policy training in traditional methods such as Diffusion-DPO and SPO. In the work (Sun al. (002b)). extends the KL divergence constraint to f-divergence, including the Forward KL divergence, the Reverse KL divergence, the a- divergence and the Jensen-Shannon (JS) divergence. Their detailed analysis and comprehensive experiments

---

Published in Transactions on Machine Learning Research (07/2025)
show that JS divergence exhibits superior generative diversity well as better alignment performance compared to KT. divergence. Generalized formula under the constraint. of f-divergence is presented as:
logo (gr (0) = 1
35 ( (oolor_llctt,aa)) -38
(o(.2_11( Pret(at-1lc,tt x4
po [Po 11,
(6)
Presla (1,,,,,,,,
6 where ('(. represents the derivalives of generalor function s()). Upon further comparing Equation
and Equation (3, it can be observed that generalization from KL divergence to f-divergence only requires the substitution from log z to f(()), where x represents the win ratio and the loss ratio. Therefore, if
this component is activated, for JS divergence, ((x) ælog x+1 + log - and ('(x) log 2+1 following substitution is needed:
Po (n(0--1le,07)
po(o po(0i-1le, /11
(e) Pref(et 1 c,,
log
log 2
Prot (
Piot 1 c,, x7)
4.4 Performance Optimization
In practice, aligned diffusion models still suffer from low-quality gencration. In this section, we aim achieve further performance optimization.
Let's begin by re-examining DPO-based methods from the perspective of contrastive loss functions Sun
ct al. 2025d Lv et al.) 2025). With the purpose of minimizing distances of similar items and maximizing
distances of dissimilar oness contrastive loss (Hadsell el al. 2006) has been widely applied in large model
training, such as the CLIP model (Radford et al. 2021
Lcon -[
ED(y: Uj)
Σ min (c(w))c,,)) -,0)
-
(8)
dissimilar items' distance
similar items' term
where D(yi, Uj) is the distance between a positive item Yi and negative item Vy: À represents regularization intensity of positive itemss and e is the introduced threshold. Define the dissimilar instances' distance between the positive (preferred) item xx-1 and the negative (dispreferred) item x/_1 as:
Po 10(x:-1le,,,,, (c,,, Pref (1 Il, 2,))
log Po(x x:))
D log
(9)
Pret 22
x1)
Hence, it can be observed that conventional DPO framework incorporates only the dissimilarity component (distance between positive and negative items) within the framework of contrastive loss, while neglecting to consider the constituent element that are relevant to the similar items' term. In this part, we break the balance between positive and negative items in two ways to improve the aligument. process's learning of positive items, i.e. the Margin Strengthened Preference Alignment (MSPA) and the SF'-llike Regularization.
4.4.1 Margin Strengthened Preference Alignment
As shown in the Equation 3 we aim to maximize the diffusion win ratio while concurrently minimizing the diffusion lose ratio during the training phase. However, is usually the case that both ratios decline concurrently to achieve margin expansion, with the diffusion lose ratio exhibiting a more pronounced rate of decrease. Continuous decline of the diffusion win ralio is highly detrimental to the slabilily of training process and the safety of fine-tuned model. Similar to LLM contexts (Yan et al.) 2025), it will gradually fail to control alignment direction of the preferred items, ultimately devolving into simply unlearning the dispreferred items. Furthermore, such training process can lead to dispersion effect on unseen responses, e.g. model's capacity of generating unseen content. in images has risen dramatically, posing a substantial risk to generative safely.
Based on such obscrvations and reflections, we would like to optimize the alignment process by targeting the following two objectives: increasing margin between the diffusion win ratio and the diffusion lose ratio; decelerating decrease and subsequently accelerating increase in the diffusion win ratio. In order to achieve
7

---

Published in Transactions on Machine Learning Research (07/2025)
them simultancously, diffusion win ratio strengthened term is introduced within the o()) function of Equation 3 thus transforming it into the following form:
Do (17-)1(,,,,4) log
po po (x Pret(x 10,,,,,, xz)
(MPPA( -E-u((1 (.op:))) logo n3 log Pret( 1,,, 10111- ( 1l(t,))
(10)
pa pa(a tT))
BB Prot(1) 11
where 7/ is strengthening intensity of the margin and the diffusion win ratio.
Moreover, it is evident that the formulation presented in Equation [10 substantively disrupts equilibrium between the win ratio and the loss ratio, thus enabling the formula exhibits similar items' terms within the modified expression.
4.4.2 SFT-like Regularization
Based on the aforementioned observations and the gains achieved by modifications within MSPA, further consider another common performance optimization method in LLM aligument: the SFT regularization Yan et al. 2025 However, it's also worth noting that diffusion models actually exhibit different paradigm compared to LLMs. Totally, there are three enhanced strategics: cnhancing probability preferred (pp); enhancing log probability preferred (log (e); enhancing log probability preferred ratio (log ptr) According to the very recent work (Sun et al.] (225dd), it is suggested that enhancing log probability of preferred ratio (log Pe. not only yields excellent effectiveness but also ensures robust training stability. Furthermore, we introduce a threshold and the minimum operation to align its form with that of similar items' term in Equation (8). While, note that this is not the SFT loss, we designate as the SFT- Regularization:
CSFT-L(0) (epp) min (log po 1e,,t (0e====))--00(11111
log Pref(x"" 2,,,)
(11)
where À is the enhancement intensity and c is the threshold.
4.5
Diffusion-RainbowPA
Inspired by Aristotle's assertion that "the whole is greater than the sum of its parts", and further motivated by Rainbow (Hessel et al.) 2018) and RainbowPO (Zhao et al. 2025). we propose combining these improvements. Building on the aforementioned improvements to Diffusion-DPO. encompassing five components through a tripartite perspective and further integrating the step-aware preference alignment paradigm, we introduce the Raaeiin-RRinbooP... a comprehensive combination of six improvements to the Diffusion-DDP. Combining the formula in Equation (4) Equation
we can obtain following form: (5) Equation (10) and Equation [I
(f) (0) + LSPPA (f) + (PEPA + (MSPA(
(12)
specifically by exccuting transformation defined in Equation 77), we convert (() into EDiffusinn-RainbowrPr (0), which serves as the aligument objective of Diffusion-Rann RainbowPA:
Furthermore, with activation of the component in Scction 4.3.2
(io Trausformation
DDiflusion CDiitussionRRaabbooo... (0)
(13)
Equation

---

Published in Transactions on Machine Learning Research (07 (025)
Table Comparison on the GenEval.
Table 2: Comparison on the "21-CompBench
GenEval
VQAS 1
CLIPS 1
HPS t
IR t
T2I-Comp++
VQAS f CLIPS t
HPS
IR f
SD-1.5
61.85
33.77
26.93
-0.200
SD-1.5
60.71
32.06
26.55
-0.245
DiffusionDDPO
63.68
34.52
27.17
-0.030
Diffusion-DPO
62.61
32.66
26.78
-0.089
SPO
61.22
33.41
27.30
0.083
SPO
62.73
31.19
26.87
-0.023
SPO (LoRA)
61.38
33.40
27.30
0.081
SPO (LoRA)
62.77
31.19
26.87
0.021
SPIN-Diffusion
60.81
32.79
27.31
-0.073
SPIN-Diffusion
60.84
31.15
27.00
-0.086
ScPPO
63.10
33.94
27.35
0.003
SePPO
63.10
32.30
27.03
-0.006
Ours
68.41
35.01
27.45
0.202
Ours
67.60
33.48
27.15
0.230
Table 3: Comparison on the GenAT-Bench.
Table 4: Comparison the DPG-Bench.
GenAI-Ben VQAS +
CLIPS f
HPS f
IR
DPG-Bench
VQAS f
CLIPS f
HPS
IR f
SD-1.5
59.50
33.62
26.94
0.237
SD-1.5
76.28
33.42
26.31
-0.187
Diffusion-DPO
60.15
34.11
27.15
0.376
Diffusion-DPO
77.13
34.01
26.51
-0.036
SPO
59.52
32.40
27.27
0.360
SPO
75.61
32.18
26.73
-0.087
SPO (LoRA)
59.48
32.41
27.27
0.359
SPO (LoRA)
75.60
32.12
26.73
-0.091
SPIN-Diffusion
58.06
32.92
27.35
0.437
SPIN-Diflusion
76.59
32.86
26.84
0.109
SePPO
61.11
34.09
27.48
0.536
SePPO
78.30
34.09
26.85
0.175
Ours
62.58
34.47
27.53
0.593
Ours
79.18
34.72
26.95
0.166
5
Experiments
5.1
Experimental Settings
Model and Dataset. In the study, we utilize the Stable Diffusion v1-5 (SD-1.5) as our benchmark model, as it is the most widely applied model in both the RI-based aligmment. research group and the preference-based alignment research group. In selecting the training dataset, to ensure a fair comparison, we adopt the same datasot utilized by SPO (Liang et al. 2024). which consists of 4K randomly chosen prompts from the Pick-a-Pie V1 dataset.
Hyperparameters. We simultaneously set all terms in Equation 12 to share f 10, corresponding to the SPO condition. Based on the tuning results reported in (Sun et al. 2025d), the positive enhancement intensity in Equation [1 is set as 100 and the threshold as log 0.9; for the MSPA term, the margin strengthening intensity 7) is empirically set to 0.5.
Evaluations. To comprehensively verify the improvement alignment performance, we adopt a total of four zero-shot alignment performance metrics, which are: the VQAScore (Lin et al. (025) (abbreviated as VQAS), the CLIPScore (Radford et al. 2021 (abbreviated as CLIPS), the HPS- V2 (Wu al.) 2023 (abbreviated HPS), and ImageReward (Xu et al. 2024 (abbreviated as IR.. We provide more details the evaluation metrics in Appendix [C.1] In our selection of evaluation benchmark, we opt for total of four widely utilized benchmark datasets in the area of text-to-image generalization: the GenEval (Ghosh et al., 2024), the CompBench++++++ (Huang et al.) 2025), the GenAl-Bench (Li et al. 2024a),, and the DPG-Bench (Hu et al., 2024). Unlike some previous work, our evaluation uses an unified approach: comparing the quality of generated images with the four zero-shot metrics described above, as opposed to contragting them with their individual benchmarking results. We provide more details on the evaluation datasets in Appendix C.2
5.2 Comparison with State-of-the-. Methods
In order demonstrate the superiority of Diffusion Diffusion-RainboPAA. PA, we compare quantitative results with those of state-of-the-art methods, including RI.-based methods (the SPIN-Diffusion (Yuan et. al. 2024) and the
RainbowPA,
9

---

Published in Transactions on Machine Learning Research (07 (225)
5m0
1338
SUPU
18.24
IPPO
55335
kno
62.90
05.42
6041
62.55
11
10.15
faes D40 Railonn Viydders
Vosliy Difiaii is Ocaava
Viciay Ras CCriioootaottto Oisa lecesoonnd
Figure 3: Winning rate from model feedback on the GenEval.
16.1
58.50
KOLLON
11233
71.66
Figure 4: Winning rate from model feedback on the T21-CompBench+H
1425
5PPO
563
Smo
12.13
36.75
20
3496
61.00
3026
Figure 5: Winning rate from model feedback on the GenAl- -Bnnch.
61.11
35
61.6
12.11
11.5
01.75
11.6
Imo
Whatig hon di Voldow
Officie CLESoo
Whalgg ww
Wiillng Ran >'De Ralloni Oden hoapd
Figure 6: Winning rate from model feedback on the DPG-Bench
2024g) and preference-basee methods (the Diffusion-DPO Wallace et al. 2024b and the SPO (Liang et al. 2024]). ). We provide more details the comparison methods in Appendix C.3 In Table D Table 2 Table 3 Table 4 we report quantitative comparison results across four zero-shot alignment metrics on the GenEval, the T2I-ComBeench++: the GenAI-Bench, and the DPG-Bench, respectively. It in both text-image aligmment and
SePPO (Zhang al.
is particularly noteworthy that improvement of Selluuiion-Rainbowe human preference alignment is unparalleled. Specifically, in the context wherein short prompts are employed (GenEval and (22p-CopBeench++) on GenEval, our method achieves a VQAScore improvement of 6.56, while the best other SOTAs only achieves maximum of 1.83; on Comnehh our method improves the VQAScore by 6.89, while the highest among other SOTAs is only 2.39. Simultaneously, the results
10

---

Published in Transactions on Machine Learning Research (07/202) /025)
Diffusion- RainbowPA
SD-1.5
Diffusion-DPO
SPO
Figure 7: Qualitative comparison of Diflusion-RaintowP2 with Diffusion-DPO and SPO. Top: "A pilot with aviator sunglassos."; Bottom: "A photo of an astronaut riding a horse on Mars." It can be observed that Diffusion-Raiinoww/ have achieved superior text-image alignment performance (aspects such as scene, attribute, and style), alleviative overfitting and generation with higher quality.
pertaining to long prompts are exciting: on GenAl-Bench, our method yields an improvement in VQAScore of 3.08, whereas the most superior SOTA attains peak improvement of merely 1.61; on DPG-Bench, our method demonstrates an improvement in VQAScore of 2.9, but top-performing SOTAs merely achieve a maximum of 2.02. We also provide detailed quantitative results on the eight separate categories of T2I- + Appendix D Furthermore, in Figure Figure 4 Figure 5 Figure we present the winming rate of Diflusion-Rainboww A against other algorithms based model leedbacks (VQAScore, CLIPScore, HPS-V2 V2 and ImageReward) the GenEval, the T21- Men-CommBennch++ the GenAl-Bench and the DPG-Bench respectively. The results indicate that Raiiiionn A consistently achieves superior performance.
Furthermore, we display qualitative comparisons between Diflusion-Rainbow/ and the SOTAs, as detailed in Figure 7 It can be observed that, Diffusion-Rainboww RainbowPA exhibits superior text-image alignment performance, more realistic scene depiction, more precise attributes, and more expected generation style. For example, the first line, our method generates more accurate, aesthetically pleasing, and logically coherent output for the subject "pilot"; in the second line, method is more accurate for scene ("Mars"), attribute (like leg number of "horse") and style ("photo").
5.3 Ablation Study
To further validate the improvement of alignment performance by each component, we conduct ablation study the five introduced components with the aforementioned four benchmark datasets. In Table 5 Table 6 Table 7 Table S show the alignment performance GenAI-Bench, and the DPG-Bench, respectively. For the five scenarios, each involving the removal of one of the five components from Diffusion-RainbowwPP (the label "w/o X" indicates the alignment performance without the component "X"). Firstly, observing all test results across the four datasets, it can be found that each component has positively contributed to the alignment performance of biffusion-Rainbowe the observed bias in GenEval, primarily stems from GenEval's nature of short prompts and a relatively small prompt set (comprising only 553 prompts). Of the three remaining datasets, the model excluding JS
the GenEval, the T21-CommBBnnch++ the
For
11

---

Published in Transactions on Machine Learning Research (07 (075))
Table 5: Ablation Study on the GenEval.
Table 6: Ablation Study on the D21-ComBBench-H
GenEval
VQAS CLIPS
HPS t
IR
T21-Comp++
VQAS 1
CLIPS f
HPS
IR f
SD-1.5
61.85
33.77
26.93
-0.200
SD-1.5
60.71
32.06
26.55
-0.245
Ours
68.41
35.01
27.45
0.202
Ours
67.60
33.48
27.15
0.230
w/o CEPA
68.05
31.77
27.49
0.239
w/o CEPA
66.84
33.14
27.18
0.203
w/o IPA
68.42
34.81
27.50
0.218
w/o IPA
66.55
33.23
27.17
0.172
w/o JS
68.14
34.95
27.54
0.210
w/o JS
67.29
33.36
27.23
0.219
w/o MSPA
68.94
35.04
27.47
0.224
w/o MSPA
67.16
33.20
27.12
0.184
w/o SFT-L
67.25
34.43
27.45
0.197
w/o SFT-I
67.48
33.06
27.13
0.180
Table Ablation Study on the GenAl-Bench.
Table & Ablation Study on the DPG-Bench.
GenAI-Bench
VQAS
CLIPS
HPS
IR
DPG-Bench
VQAS t CLIPS
HPS f
IR.
SD-1.5
59.50
33.62
26.94
0.237
SD-1.5
76.28
33.42
26.31
-0.187
Ours
62.58
34.47
27.53
0.593
Ours
79.18
34.72
26.95
0.166
w/o CEPA
62.23
34.12
27.57
0.531
w/o CEPA
78.73
34.38
26.95
0.088
w/o IPA
61.55
34.25
27.54
0.517
w/o IPA
78.30
34.57
26.95
0.075
w/o JS
62.24
34.16
27.59
0.542
w/o JS
78.79
34.36
26.97
0.111
w/o MSPA
62.78
34.04
27.51
0.505
w/o MSPA
78.78
34.49
26.93
0.054
w/o SFT-L
62.24
34.01
27.53
0.485
w/o SFT-L
78.56
34.46
26.94
0.023
demonstrated superior performance on HPS-V2 compared to the full model. Such observation is related a potential conflict between the optimization objective of JS divergence and the evaluation dimensions of HPS-V2: JS divergence aims stabilize the training process and alleviate overfitting by symmetrizing KL divergence. Its core objective is to improve the distribution consistency of generated samples, rather than directly optimizing the fine-grained attributes of human preferences (such color accuracy, reasonable composition); then the distribution over-smoothing and optimization direction shift introduced by the JS divergence paradigm might slightly decrease the HPS-V2 results. Despite this, performance with the JS divergence always holds significant. advantages over that with the KL divergence in all other metrics: for example, as shown in Table 5 ablating the JS results a 1.05 decrease in VQAScore and a 0.06 decrease in CLIPScore; Table G removing the JS leads to reductions of 0.31 in VQAScore, 0.12 in CLIPScore, and 0.011 (5%) in ImageRewarde in Table 7 JS removal decreases VQAScore by 0.34, CLIPScore by 0.31, and Iewaedeward by 0.051 (9%); in Table removing JS lowers VQAScore by 0.39, CLIPScore by 0.36, and ImageReward by 0.143 (86%). Hence, considering all test results comprehensively, omission of any single component would result in a degradation of the alignment performance: that is to say, regarding the introduced five components from three aspects as "the whole" can get "greater" performance than that of "its parts".
conclude that
6 Conclusion
In this study, we point out limitations of current state-of-the-art diffusion-based text-to-image alignment meth- ods, which tend to suffer from text-image misalignment, aesthetic overfitting and low-quality generation. To address such issues, improve the alignment paradigm through tripartite perspective: calibration enhance- ment, overfitting mitigation and performance optimization. For calibration enhancement, we introduce the Calibration Enhancement Preference Alignment (CEPA); for overfitting mitigation, we introduce the Identical Preference Alignment (IPA) and the Jensen-Shannoo Shannon Divergence Constraint; for performance optimization, we introduce the Margin Strengthened Preference Aligument (MSPA) and the SFT-like Regularization. Further combining the introduced five components with step-aware preference alignment paradigm, we propose the Diflusion-Rainboww A, suite of six improvements that collectively enhance the alignment, performance of
12

---

Published in Transactions on Machine Learning Research (07/2025)
Diffusion-DPO. Comprehensive evaluations demonstrate that state-of-the-art methods. Furthermore, ablation study conducted on the five introduced components indicates that each component of MaasiionRainbowPA. has positively contributed to the alignment performance.
outperforms current
Broader Impact Statement
Performance of D is promising, while actually any effort in text-to-image generation presents ethical risks. The improved models might be misused to generate harmful, hateful, fake or sexually explicit content. We further utilize supplementary safety filtering mechanisms during inference to ascertain the elimination of toxic content. Specifically, we train only the UNet model and do not train the safety checker model to prevent potential vulnerabilities, such as the reward hacking, that could attack the safety checker model during the training process. Moreover, with all wreference-basey alignment. methods, the biases of preference that has been encoded in the dataset might be introduced. For this problem, we adopt the solution consistent with previous methods like Diffusion-DPP and SPO, i.c. performing data filtering; and we utilize the same datasets with them in our training process.
Acknowledgments
This work was supported by the Natural Science Foundation of Shenzhen (No.JCY110220007711100000) the Natural Science Foundation of Guangdong Province (No.2024A15150100033) National Key Research and Development Program of China (No. 24.2002274400040000 and Cross-disciplinaty Fund for Research and Innovation (No.JC2024002) of Tsinghua. SIGS.
References
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, et al. Improving image generation with better captions. Computer Science. hllps://odnn. openai. com/yupers/ue /alle--- pdf, 2(3):8, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: 1. the method of paired comparisons. Biometrika, 33(3/4):324-345 1952.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata, ReNO: Enhancing one-step text-to-image models through reward-based noise optimization. In The Thirty-eighth Annual Conference on Newral Information Processing Systems, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to too-to--mmna to-image diffuusio models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, and Ilya Makarov. Aligning diffusion models with noise-conditioned perception. orXiv preprint ar Xix::2400.. 17636, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learnöng, pp. 10835-10866 PMLR, 2023.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for
evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning diffusion models through relative preference optimization. arXiv preprint arxiv:2406.. 06382, 2024.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 TEFE computer sociely conference on computer vision and pattern recogrition (CVPR 006), volume 2, pp. 1735-1742. IEEE, 2006.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advonces in neural information processing systems, 33:6840-6888 2020.
Jiwoo Hong, Sayak Paul, Noah Loe, Kashif Rasul, James Thorne, and Jonghcon Jcong. Margin-aware preference optimization for aligning diffusion models without reference. arXio preprinl arXiv:2406 06424, 2024.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2433.0511 05135,, 2024.
Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Qihan Huang, Long Chan, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, and Jie Song. Patchdpo: Patch-level dpo for finetuning-free personalized image generation. arXiv preprint 2rXhv:241.003177, 2024.
Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for text-to-image generation. arXiv preprint arXiv:2410.1801 2024.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Alignment without over-optimization: Training-free solution for diffusion models. arXiv preprint arXim:2501.0558 2025.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Emily Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Genai-bench: A holistic benchmark for compositional text-to-visual generation. In Synthetic Data for Computer Vision @orkshop@ CVPR 2024, 2024a.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusson models by optimizing human utility. In The Processing Systems, 2024b.
Annual Conference on Neural Information
Shufan Li, Harkanwar Singh, and Aditya Grover. Popalign: Population-level alignment for fair text-to-image generation. arXiv preprint arXiv:2406. 19668, 2024c.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each stop. arXiv preprint arXiv:2406.04314, 2024.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366 384. Springer, 2025.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xic. Aligument of diffusion models: Fundamentals, challenges, and future. arXiv preprint arXiv:2409.00 072500 2024a.
Runtao Liu, Chen I Chich, Jindong Gu, Jipeng Zhang, Renjie Pi, Qifeng Chen, Philip Torr, Ashkan Khakzar, and Fabio Pizzati. Safetydpo: Scalable safety alignment for text-to-image generation. ar preprint 0rXiv:2412104490 2024b.
Xufei Lv, Haoyuan Sun, Xuefeng Bai, Min Zhang, Houde Liu, and Kehai Chen. The hidden link between rIhf and contrastive learning. Xiv preprint ar (xxx:2506.22576 2025.
Yong-Hyun Park, Sangdoo Yun, Jin-Hwa Kim, Junho Kim, Geonhui Jang, Yonghyun Jcong, Junghyo Jo, and Gayoung Lee. Direct unlearning optimization for robust and safe text-lo-image models. In The Tiirth- Annual Conference on Neural Information Processing Systems, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-42000 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. ar preprint arXiv:2810.03730 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Newral Information Processing Systems, 36, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp- 8821-8831. Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06120 061122 1(2):3, 2022.
Scott Reed, Zeynep Akala, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1068 PMLR, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image hests with latent diffusion models. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 10684- 10695, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1:Xiu:1707.063.4 2017.
Shivanshu Shekhar, Shreyas Singh, and Tong Zhang. See-dpo: Self entropy enhanced direct preference optimization. arXiv preprint ar arXio:2411.04712. 2024.
Haoyuan Sun, Jiaqi Wu, Bo Xia, Yifu Luo, Yifei Zhao, Kai Qin, Xufei Lv, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Reinforcement fine-tuning powers reasoning capability of multimodal large language models. ar preprint arXiv:2505. 18536, 2025a.
Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text-to-image generation with preferences through f-divergence minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025b.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Identical human preference alignment paradigm for text-to-image models. In ICASSP 2025 2025 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025c.
Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Positive enhanced preference alignment for text-to-image models. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025d.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural informalion processing systems, 12, 1999.
Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung-Hut Chang. Tuning-free aligmment of diffusson models with direct noise optimization. arXiv preprint arXiv:2405. 18881, 2024.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of 1lm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407 16216, 2024.
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey. arXiv preprint ar Xiv:24009 1164,, 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint XrXi::2306. 09341, 2023.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXie preprint orXiv:2405.000 2024.
Bo Xia, Haoyuan Sun, Bo Yuan, Zhiheng Li, Bin Liang, and Xueqian Wang. A delay-robusi method for enhanced real-time reinforcement learning. Newral Networks, 181:106769, 2025.
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-DPO: Calibrated direct preference optimization for language model alignment. In The Thirty-eighth Avnuul Conference on Neural Information Processing Systems, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image gencration. Advances in Neural Information Processing Systems, 36, 2024.
Yuzi Yan, Yibo Miao, Jialian Li, YipinZhang, Jian Xic, Zhijic Deng, and Dong Yan. 3d 3d-properties: Identifying challenges in DPO and charting path forward. In The Thirleenth International Conference on Learning Representations, 2025.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to finc-tune diffusion models without any reward model. In Proceedings of the /EE//VV Conference on Computer Vision and Pattern Recognition, pp. 8941 8951, 2024a.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-3 2023.
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. In Forty-first International Conference on Machine Learningg 2024b.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-lo-image generation. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, et al. Seppo: Semi-policy preference optimization for diffusion aligument. Xiv preprial or Xrxiv:2 10. 00.05255, 2024.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Hanyang Zhao, Genta Indra Winata, Anirban Das, Shi-Xiong Zhang, David Yao, Wenpin Tang, and Sambit Sahu. RainbowPO: unified framework for combining improvements preference optimization. In The Thirleenth International Conference Learning Represenlations, 2025.
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A Diffusion-based Text-to-lmage Models
Zhang al. Early rescarches in text-to-image (T2I) generation mainly focus GANs (Reed ct al.
2016
2017
More recently, however, diffiusion-based models have demonstrated remarkable proficiency in producing images of exceptional quality, adhering to the input descriptions better (Ho et al.)
2020:
Rombach et al.
2022: Podell et al. 2024 Essor et al.) 2024 Ramesh et al. 2021 2022 Betker et al. 2023 Yang ct al. 2023
To achieve more precise control over the generative process, recent studies have explored various techniques
aimed at enhancing the guidance of diffusion models. DiT (Peebles & Xie 2023 exemplifies better generation
with utilization of superior transformer backbones; Pixart-alpha (Chen al. 2024 exemplifies efficacy of advanced text encoders in the accomplishment of robust language conditioning: DALL-E (Betker et al. 2023) exemplifies enhanced generative performance through improved captioning. Despite their ellectiveness, aligning diffusion models with human value still presents significant challenge and remains critical issuc. In this study, we utilize the Stable Diffusion v1.5 (Rombach et al. 2022) as our backbone, primarily based on the goal of aligning with the RL-based T2I aligument research group. Herein, we would like to briefly introduce the core of diffusion-based T2T for readers who are not very familiar with it, which the mechanism of Denoising Diffusion Probabilistic Models (DDPMs).
Let's consider the data distribution Fo (o(2)), 20 E R". DDPM algorithm approximates the data distribution 40 with parameterized model with the form of po( = J Pa le)d21r where po po (=))_))))==p(=r)) IT= (1 c), and e is the conditioning information, (i.e., the image category and the image caption). Then, we can describe the reverse process to be an Markov chain with dynamics follows:
p(aT) = NO, 1), po( c) (1 Ext.
Furthermore, DDPM further exploits an approximate posterior ( (oo, (),,),, namely the forward process, adding Gaussian noise to the data acccording to the variance coefficients $1,..,
T ((11T/20 c) = II (11))
(1 =N(VI
1- 00-1
= 1 1 Br, 0
11,
ai B.
1
Based on these, in the work (Ho et al. 2020 the parameterization is further applied as follows:
1 Vog
no( c)
(#t
6 (x())))) at
VI
B Brief Survey of Text-to-lmage Alignment
Reinforcement learning (RL) has recently exhibited significant potential for application across diverse array of domains (Sun et al.] 2025a Xia et al.] 2025). Furthermore, in the domain of aligning diffusion- based text-to-image models with human preferences, prevalent method family the Reinforcement Learning from Human Feedback (RLHF). To summarize, they encompass the maximization of target reward function in conjunction with the minimization of Kullback-Leibler divergence between current policy and reference policy. Methods such as DDPO (Black et al. 2024h, DPOK (Fan et al. 2024), ReFL (Xu et al. 2024). DRaFT (Clark et al., 2024), and AlignProp (Prabhudesai et al.) 2023) initially entail training a reward model for modeling human value, followed by employing RL pipelines like PPO (Schulman et al. 2017) and REINFORCE (Sutton et al. 1999) to fine-tune the policy, thereby optimizing for the rewards offered by the reward model. Despite their effectiveness, traditional RLHF methods involve the modeling of reward from relatively limited preference datasets, which results in the training pipeline suffer from disadvantages such high computational costs and inadequate training stability.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Direct Preference Optimization (DPO) (Rafailov et al. 2024 achieves remarkable success in Large Language Models (LLMSs) alignment by implicitly estimating the reward model. Based on this, some prior work has been conducted grounding in the idea of implicit reward models (Liu et al. 2024a). Diffusion-DPO (Wallace ct al. 2024 re-formulates DPO by utilizing the evidence lower bound (ELBO) to derive a differentiable objective function, and further approximation the reverse process with the forward. D3PO (Yang et al. 2024a) regards the denoising process as a multi-step Markov decision process (MDP), drawing the conclusion that directly updating the policy based on human preferences within MDP is equivalent to first learning the optimal reward function. DenseReward (Yang et al. 2024b) further advances the DPO paradigm by introducing a temporal discounting, which accentuates the initial stages of denoising while comparatively weaken posterior ones. Recently, SPO (Liang et al. 2024 is proposed to evaluate and adjust the denoising performance at step-level for ensuring accurate step-specific preference signal. These paradigms have laid a robust groundwork for researchers to pursue further advancements.
Building upon them, recent researches on aligning diffusion- text-to-image with human preference primarily focus on three directions. The initial aspect involves deriving inspiration from existing numerous alignment paradigms of generative models (such LLMs), exploring and modifying them for application in the context of diffusion-based text-to-image. The sccond aspect is to further integrate DPO paradigm with diffusion model itself for further leveraging the advantages of denoising process. The third aspect involves exploration of novel training paradigms to achieve superior performance and better efficiency during the training process.
In the first aspect aforementioned, some typical works are as follows. Diffusion-KTO (Li et al. 2024b) formulates the alignment objective as the maximization of expected human utility and further offers robust framework based existing KTO methodology. Diffusion-RPO (Gu et al. 2024) applies contrastive weighting to similar prompt-image pairs based on the success of RPO. MaPO (Hong et al. 2024) eschews conventional reference paradigm, concurrently optimizing the likelihood margin between the preferred and the dispreferred image sets and augmenting the likelihood of preferred set, which is also partly inspired by SimPO. Overall, there still exists substantial potential ial for further investigation within this aspect.
In regard to the second aforementioned aspect, some representative works are delineated as follows. In the studies of DNO (Tang et al.) 2024), NCPPO (Gambashidze et al.] 2024). and ReNO (Eyring et al. 2024 researchers delve into examination of the impact of noise resullant outcomes, employing methodologies primarily centered around the noise optimization. DAS (Kim et al. 2025) offers a training-free approach to aligning diffusion models with arbitrary reward functions, thereby maintaining their generalizable performance.
Concerning the third aspect, several exemplary works are discussed as follows. SPIN-DifTusion (Yuan el al,, 2024) enables the diffusion model to compete with its carlier versions, thereby facilitating an iterative process of self-improvement. DUO (Park et al., 2024) endeavors eliminate Not Safe For Work (NSFW) content from T21 models, simultaneously safeguarding their proficiency across unrelated subjects. SafetyDPO {Liu et al. 2024b) facilitates safety alignment of T2T models, precluding the generation of inappropriate outputs without impacting generative capabilities on safe prompts. PopAAlggn (Li ct al., 2024c) proposes a concept of population-level preference optimization as a means of mitigating the population bias. RankDPO (Karthik
let al.] 2024) weighs the preference loss with discounted cumulative gains, which further enhances DPO-based methods with the help of ranking fcedback. ScPPO (Zhang et al.) (202)) further employs previously saved checkpoints as reference models, concurrently utilizing them to produce ou-policy samples. SEE-DPO (Shekhar
et al. 2024h incorporates self-entropy regularization term into conventional KT-regularized formulation of RLHF objective function. In the study (Sun et al. 2025b), it is proposed that employment of Jensen Shannon divergence yields superior performance in achieving human value alignment comparing to Kullback-Leibler divergence, while concurrently facilitating an optimal trade-off between alignment performance and generative diversity. PatchDPO (Huang et al.) 2024 estimates the quality of image patches within each generated image and accordingly trains the model.
In summary, exploration on aligning diffusion-based text-to-image models with human preferences is yet. to traverse substantial expanse; concurrently, this field is undergoing burgconing phase. The aforementioned three aspects, along with their interwoven integration, constitute the predominant pathways for future researches.
19

---

Published in Transactions on Machine Learning Research (07/)025)
C More Details on the Evaluation and Comparison
In this part, we provide more detailed descriptions on the evaluation metrics, evaluation datasets and comparison methods.
C.1 Details on the Metrics
In this study, we utilize total four published zero-shot metrics: the VQAScore (Lin et al., 2025), the CLIPScore (Radford al.) 2021), the HPS-V2 (Wu al.) 2023), and the ImageReward [uu et al. 2024
VQAScore. VQAScore utilizes the Visual Question Answering (VQA) model for evaluation. Its core idea is to transform the given text into simple yes/no question: "Does this image depict text?" The image and the question are then input into a pre-trained VQA model, and the probability of the model predicting "Yes" is used as the matching score between the image and the text. To further enhance the performance of VQAScore, the authors also trained an internal VQA model named CI.IP-Flan's T5. This model employs a bidircctional image-question encoder, allowing image embeddings to be adjusted according to the question and vice versa, which is more in line with how humans understand images. CLIP-FlanTS has achieved a new state-of-the-an the-art level in all benchmark tests. The simplicity and effectiveness of VQAScore make it powerful tool for evaluating text-to-visual generation models, particularly demonstrating significant advantages in handling complex text prompts.
CLIPScore. CLIPScore is a widely used metric for evaluating the quality of generated images, which leverages OpenAT's CLIP (Contrastive Languago-Image Pre-training) model mcasure the semantic consistency between generated image and given text description. The core idea is: first, use the CLIP model to separately compute the embedding vectors for the generated image and the text description; then, calculate the cosine similarity between these two vectors. The higher the similarity score, the higher the semantic alignment between the image and the text, indicating that the image better expresses the content of the text description. Advantage of CLIPScore lies in its strong generalization ion ability and atlention to image details, as it is trained on large-scale image-text datasets, capable of capturing richer semantic information, and not just focusing on the visual quality of the image, but also on its aligmment with the text description. Although CLIPScore can some extent reffect human perception of image quality and text consistency, it is not perfect and may be influenced by certain specific text prompts, and there may be limitations in understanding complex scenes and abstract concepts. Nevertheless, CLIPScore remains important tool for evaluating the performance of generation models, often used in combination with other metrics to more comprehensively assess the quality of generated images.
HPS-V2. HPS-V2 is trained on a large human preference dataset, the Human Preference Dataset v2 (HPD-V2), which contains 798,090 human preference choices on 433,760 pairs of images, making it the largest dataset of its kind. To avoid biases present in previous datasets (such as primarily containing images generated by specific models or using biased prompts), HPD-V2 deliberately collects images from 9 diffcrent text-to-image generation models (including Stable Diffusion, DALL-E 2, etc.) and the COCO dataset. It also uses ChatGPT to clean up the prompts, making them more concise and clear, reducing stylistic language and contradictory information. HPS-V2 is obtained by fine-tuning the CLIP model on HPD-V2. Experiments show that it has better generalization capabilities than previous metrics (such as HPS-V1, PickScore, etc.), can better predict human preferences for generated images.
ImageReward. ImageReward is general human preference scoring model. It is trained through a systematic amnotation process that includes two stages: scoring and ranking, where over 1.37 million expert comparison data has been collected. The process first classifies and identifies issues in the text prompts, then scores generated images on three dimensions: text-image alignment, fidelity, and harmlessness, and finally ranks the images to capture human preferences. Experiments show that ImageReward outperforms existing scoring models in understanding human preferences, such as CLIP, Aesthetic, and BLIP. In evaluations of real user prompts and the MS-COCO 2014 datasct, ImageReward highly consistent with human preference rankings and has a higher degree of discrimination between samples, making it promising automatic evaluation metric for text-to-image generation models.
20

---

Published in Transactions on Machine Learning Research (07/2025)
C.2 Details the Datasets
In evaluations of this study, we utilize four published benchmarking datasets. Based nature of prompts in the dataset, we can categorize them into the short prompts and the long prompts. Herein, the short
prompts include the GenEval [Ghosh et al., 2024) and the T21-CompBench+++ (Huang et al. 2025 ; the long
prompts include the GenAl-Bench (Li et al. 2024a and the DPG-Bench (Hu et al. 2024).
GenEval. GenEval is a relatively small dataset (containing only 553 prompts) designed for evaluating the generation capabilities of lexl-to-image (T21) models. GenEval decomposes texl. prompls into their constituent. parts, such as number, attribution, color, and relative position. It then uses an object detection model to identify objects in the generated images, extracting bounding box and segmentation mask information. This information is used to verify the presence of specified objects, the accuracy of their counts, and the adherence to the described spatial relationships between objects. In this research, we do not use the benchmarking results from the GenEval dataset itself but instead use the quantitative results from metrics in Appendix C.1 with the purpose of unifying results' comparison across different datasets.
ToI-CompBench++ a benchmark comprised of 8000 synthetic text prompts designed to evaluate the compositional capabilities of text-to-image generation models. It categorizes prompts into four main areas: Attribute Binding, Object Relationships, Generation Arithmetic, and Complex Combi- nations, further subdividing them into eight subcategories, including newly introduced 3D spatial relationships and numeracy. Attribute Binding comprises subcategories for Color, Shape, and Texture, assessing the model's ability to associale altributes with the correct objects. Object Relationships includes subcategories for 2D33D Spatial Relationships and Non-Spatial Relationships, evaluating the model's understanding and generation of various object interactions. Generation Numeracy assesses the model's capability to handle text prompts specifying different object quantities. Complex Combinations involve combining multiple objects or categories, testing the model's ability to handle more complex scenarios. To evaluate these diverse compositional challenges, T'2l-CompBench++ proposes improved evaluation metries, including detection-based metrics for 3D spatial relationships and arithmetic, as well as analytical metries leveraging multimodal large language models (MLLMs) such as GPT-4V. By benchmarking 11 text-to-image models, including 1LUX.1, SD3, DALL-F 3, Pixart-a, and SDXL, on T2I-CompBench+4 and conducting thorough evaluations to validate the metrics' effectiveness, as well as exploring the potential and limitations of MLLMs, T2l-CompBench++ provides a robust framework for assessing and advancing compositional capabilities in text-to-image tomeee generation. Similarly, we do not use the benchmarking results itself but instead use the quantitative results from metrics in Appendix C.1 with the purpose of unifying results' comparison across all different datasets.
GenAI-Bench. GenAl-Bench is a novel benchmark designed for comprehensive evaluation of compositional text-to-visual generation models. Unlike previous benchmarks that primarily focus on basic visual elements like objects, attributes, and simple relationships, GenAI-Bench incorporates a wider range of compositional skills, calegorized into "basic" (objects, altribules, spatial/actiog aattiiall/ccio/pparr relations, scenes) and "advanced" (counting, comparison, differentiation, negation, universality) skills. This expanded skill taxonomy reflects the complexities of real-world user prompts, sourced directly from professional dosigners familiar with tools like Midjourney. The benchmark includes 1600 challenging prompts, each meticulously tagged with all relevant skills, enabling fine-grained analysis of model performance across different aspects of compositional reasoning. The study utilizes over 80,000 human ratings (38,400 for initial model evaluation, and an additional 43,200 for a GenAI-Rank sub- nab-benchmark focused on image ranking) on images and videos generated by ten leading models (including both open-source and closed-source options), revealing significant shortcomings in handling advanced rcasoning tasks even for stato-of-tho-art models. Furthermore, GenAT-Bench facilitates the evaluation of automated metries, demonstrating the superior performance of VQAScore metric leveraging VQA models compared to existing methods like CLIPScore, especially in correlating with human judgments on compositional prompts. Similarly, we do not use the VQAScore merely but use the four quantitative results from metrics in Appendix C.I with the purpose of comprehensively carrying out the comparison.
DPG-Bench. DPG-Bench is a dataset introduced to evaluate ability of text- text-to-image models to handle complex and dense prompts. Previous benchmarks like T2I-CompBench and PartiPrompts primarily focus on shorter prompts with limited descriptive detail. DPG-Bench addresses this limitation by providing a more challenging evaluation set. It consists of 1,065 lengthy and detailed prompts, significantly exceeding the length
21

---

Published in Transactions on Machine Learning Research (07/2025)
and complexity of those in previous benchmarks. These prompts describe multiple objects, each with diverse attributes and intricate relationships between them. The average prompt in DPG-Bench contains around 84 tokens, compared to 10-20 tokens in previous benchmarks. The increase reflects a much richer semantic content within cach prompt. Source data is gathered from COCO, PartiPrompps, DSG-1k, and Object365. And the existing short prompts are exlended using GPT-4 to incorporate detailed descriptions of objects, attributes, and relationships. They are then verified by human annotators to ensure quality and accuracy. Similarly, we use the quantitative results from metrics in Appendix C.1 with the purpose of comprehensively carrying out the comparison.
C.3 Details on the Comparison Methods
In the comparison of this paper, we mainly focus on the Diffusion-DPO, the SPO (with and without LoRA), the Diffuiifusi,n, and the SePPO.
Diffusion-DPO. (CVPR 2024) Diffusion-DPO is the first attempt designed to align text-to-image diffusion models with human preferences. It. departs from traditional approaches using reinforcement. learning or specific datasets by directly optimizing the diffusion model. Instead of crafting an explicit reward function, DPO subtly learns human preferences by directly tweaking the model's parameters. It utilizes paired image comparisons where one image is preferred over another for given prompt. The aim is to improve the win ratio of preferred images relative to dispreferred ones under the model's distribution.
SPO. (CVPR 2025) Step-by-Step Preference Optimization (SPO) (with LoRA and without LoRA) is a novel post- post-training method for enhancing the aesthetic quality of diffusion models. Unlike existing DPO-based mothods that propagate preferences across entire generation trajectories, SPO focuses on fino-grained detail improvements at each denoising step. SPO operates by first sampling a pool of candidate images from a shared noisy latent at each step. A Step-aware Preference Model (SPM), trained on open-source preference data, then evaluates these candidates and identifies win-lose pair exhibiting the largest quality difference: crucially, these pairs are visually similar, highlighting subtle aesthetic details rather than large layout discrepancies. The diffusion model is then fine-tuned using a modified DPO loss function based on this win-lose pair. Finally, a candidate image is randomly selected from the pool to initialize the next denoising step. Such iterative process, repeated across multiple steps, accumulates minor aesthetic improvements, leading to significantly enhanced overall visual appeal. Key advantage of SPO is its ability to leverage generic preference data effectively by focusing on small, detail-level differences at each step, circumventing the limitations of existing DPO methods that struggle with noisy and conflicting preferences in holistic image evaluations.
SPIN-Diffusion. (NeurIPS 2024) SPIN-Diffusion is a novel self-play fine-tuning method for diffusion models, addressing the limitations of standard supervised fine-tuning (SFT) and reinforcement learning from human feodback (RLHF). Unlike SFT, which platcaus with limited data and doesn't directly optimize for human preference; and unlike RLHF, which requires paired "winner-loser" images per prompt,, SPIN-Diffusion iteratively improves a diffusion model by pitting it against previous versions. Core idea of is a minimax game: "main player" (the current model) trics to distinguish between real images from the target distribution and those generated by an "opponent player" (a previous model version). The opponent aims to fool the main player. Crucially, the opponent is simply a copy of the main player from a previous iteration, enabling self-play. SPIN-Diffusion overcomes challenges inherent in applying self-play to diffusson models by (a) designing an objective function considering the entire diffusion trajectory (not just the final image), and (b) decompasing and approximating the probability function using score functions (gradients of probabilitics), leveraging the Gaussian reparamctorization technique from DDIM for efficient computation. Hence, it allows for an unbiased objective function calculated from intermediate samples, further approximated for computational efficiency to avoid storing all intermediate images. The iterative self-play process continues until convergence, theoretically achieving point where further SFT improvement is impossible.
SePPO. (arXiv preprint) Semi-Policy Preference Optimization (SePPO) is a novel method for aligning diffusion models (DMs) with human preferences without relying on reward models or paired human-annotated data. It addresses limitations of both on-policy and off-policy reinforccment learning from human feedback (RLHF) approaches. ScPPO leverages previous DM checkpoints as reference models. They generate on-policy reference samples, acting as replacements for "losing images" in preference pairs. Tt further optimizes using
22

---

Published in Transactions on Machine Learning Research (07 (025)
only off-policy "winning images" from supervised fine-tuning (SFT) dataset. A key innovation is the reference model selection strategy. Instead of using initial or latest checkpoint, SePPO randomly samples from all previous checkpoints, broadening exploration in policy space and thus preventing overfitting. Furthermore, ScPPO employs an Anchor-bass Adaptive Flipper (AAF). AAF assesses whether reference samples are truly inferior current model's output using a criterion based on the probability of generaling the winning image. This adaptive mechanism prevents performance degradation due uncertain reference sample quality, selectively learning from generated samples based their likely classification winning or losing.
D More Detailed Results of Comparison with SOTAs the T21-CompBench+4 with Eight Categories Separately
Table 9: Comparison of Diflusion-Rainboww with SOTAs on the T21-CompBench++4 (eighl categories
separately).
VQAScore
T
3d spatial
color
complex
non-spatial
numeracy
shape
spatial
texture
SD-1.5
50.95
55.82
70.50
76.37
49.52
59.72
61.07
61.70
Diffusion-DPo
54.26
57.94
72.11
77.67
50.46
61.06
64.47
62.95
SPO
57.57
59.01
72.83
72.96
50.32
63.67
63.07
62.44
SPO (LoRA)
57.63
59.20
72.91
73.00
50.40
63.74
63.00
62.31
SPIN-Diffusion
49.96
57.01
71.85
74.65
47.17
62.47
62.48
61.15
SePPO
54.42
60.68
72.56
78.09
50.26
60.70
65.52
62.54
Ours
61.69
66.23
74.48
79.16
53.64
65.63
69.06
70.89
CLIPScore
SD-1.5
31.46
32.76
30.98
33.19
31.54
30.94
33.91
31.67
Diffusion-DPO
32.26
33.60
31.31
33.61
32.22
31.48
34.78
32.03
SPO
31.33
32.21
29.87
31.15
30.80
30.68
33.07
30.38
SPO (LoRA)
31.32
32.21
29.87
31.14
30.79
30.68
33.10
30.36
SPIN-Diffusion
30.58
31.85
29.67
32.65
30.53
30.79
33.56
29.61
SePPO
31.63
33.40
31.07
33.36
32.36
31.11
34.18
31.31
Ours
33.45
34.58
31.53
33.68
33.37
32.12
35.66
33.46
HPS-
SD-1.5
26.87
27.06
25.61
26.62
26.17
26.27
27.91
25.91
Diffusion-DPO
27.12
27.29
25.74
26.80
26.39
26.50
28.29
26.10
SPO
27.21
27.44
25.99
26.81
26.37
26.61
28.42
26.15
SPO (LoRA)
27.21
27.44
25.99
26.81
26.37
26.61
28.42
26.15
SPIN-Diffusion
27.38
27.52
25.99
27.05
26.47
26.69
28.62
26.25
SePPO
27.39
27.64
26.03
27.10
26.65
26.63
28.55
26.27
Ours
27.54
27.71
26.06
27.13
26.77
26.76
28.68
26.51
ImageReward
SD-1.5
-0.552
-0.496
-0.922
0.549
-0.500
-0.431
-0.043
-0.393
Diflusion-DPO
-0.341
-0.322
0.030
0.670
-0.399
-0.320
0.244
-0.275
SPO
-0.113
-0.331
0.116
0.601
-0.228
-0.163
0.285
-0.352
SPO (LoRA)
-0.108
-0.327
0.116
0.606
-0.228
-0.164
0.293
-0.359
SPIN-Diffusion
-0.329
-0.359
0.121
0.723
-0.452
-0.176
0.233
-0.446
SePPO
-0.246
-0.179
0.157
0.805
-0.295
-0.281
0.313
-0.321
Ours
0.059
0.106
0.200
0.819
-0.051
-0.017
0.553
0.168
23

---

Published in Transactions on Machine Learning Research (07 (025))
E More Detailed Results of Ablation Study on the T21-CompBench with Eight Categories Separately
Table 10: Ablation study of Dillbsion-RRinbowNAA on the T21-CompBench+ (eight calegories separately).
VQAScore
+
3d ad_spatial
color
complex
non-spatial
numeracy
shape
spatial
texture
SD-1.5
50.95
55.82
70.50
76.37
49.52
59.72
61.07
61.70
Ours
61.69
66.23
74.48
79.16
53.64
65.63
69.06
70.89
w/o CEPA
60.96
64.74
74.95
78.73
52.85
64.54
68.57
69.37
w/o IPA
60.24
64.76
74.34
79.09
52.96
64.39
67
68.80
w/o JS
61.64
64.57
74.74
79.08
54.07
64.66
69.89
69.62
w/o MSPA
60.57
65.30
74.60
78.62
53.89
64.43
69.19
70.69
w/o SFT-L
61.07
66.76
74.55
79.46
53.78
64.91
69.13
70.16
CLIPScore
SD-1.5
31.46
32.76
30.98
33.19
31.54
30.94
33.91
31.67
Ours
33.45
34.58
31.53
33.68
33.37
32.12
35.66
33.46
w/o CEPA
33.10
34.32
31.63
33.39
32.91
31.65
35.18
32.95
w/o IPA
33.13
34.48
31.72
33.48
32.92
31.98
35.31
32.83
w/o JS
33.22
34.36
31.51
33.51
33.45
32.00
35.61
33.25
w/o MSPA
32.94
34.22
31.47
33.44
32.85
31.80
35.33
33.58
w/o SFT-L
32.76
34.21
31.50
33.05
32.84
31.65
35.01
33.46
HPS- V2
SD-1.5 Ours
26.87
27.06
25.61
26.62
26.17
26.27
27.91
25.91
27.54
27.71
26.06
27.13
26.77
26.76
28.68
26.51
w/o CEPA
27.53
27.75
26.16
27.24
26.77
26.77
28.72
26.54
w/o IPA
27.56
27.76
26.15
27.16
26.76
26.81
28.70
26.48
w/o JS
27.65
27.80
26.14
27.21
26.89
26.84
28.82
26.54
w/o MSPA
27.46
27.70
26.09
27.12
26.75
26.77
28.62
26.51
w/o SFT-L
27.50
27.78
26.11
27.14
26.73
26.76
28.61
26.48
ImageReward
SD-1.5 Ours
-0.552
-0.496
-0.092
0.549
-0.500
-0.431
-0.043
-0.393
0.059
0.106
0.200
0.819
-0.051
-0.017
0.553
0.168
w/o CEPA
0.018
0.073
0.243
0.796
-0.057
-0.097
0.543
0.104
w/o IPA
-0.010
0.028
0.206
0.759
-0.109
-0.066
0.518
0.047
w/o JS
0.079
0.069
0.194
0.791
-0.017
-0.067
0.583
0.121
w/o MSPA
-0.025
0,035
0.186
0.765
-0.108
-0.066
0.513
0.175
w/o SFT-L
0.025
0.041
0.137
0.759
-0.134
-0.095
0.550
0.158
F Further Discussion on Computational Overhead and Complexity Introduced
In this study, the experiments are conducted on a machine equipped with 4 x NVIDIA A100-PCIE-40GB GPUs. The training process for Aillusion-Rainbor
requires approximately 15 hours and the maximum GPU memory consumption required by Diffusion- RainbowPA is 21.74 GB per GPU, indicating that is a relatively lightweight method. Furthermore, we perform experiments on consumer-grade graphics cards, specifically utilizing machine equipped with NVIDIA GeForce RTX 3090 GPUs (each with 24GB of memory); and the entire training process this setup requires approximately 27.5 hours.
24

---

Published in Transactions on Machine Learning Research (07/2025)
In addition, we provide an intuitive explanation of the complexity introduced by integrating multiple alignment components. The six alignment improvements in Dnffusioon-Rinbow... are implemented as additional regularization termss within a similar training loop to that of the baseline SPO (Liang et al. 2024). Herein, in contrast to SPO, which employs a multi-sample setup in the step-wise resampler by sampling four images, we instead adopt a configuration that samples only two images. This modification partially reduces computational complexity. For the CEPA, two additional squared error terms are introduced, which increase only scalar multiplication, addition, and squaring operations for the computation; hence, CEPA can be considered a quadratic regularization extension of log pra((at- algorithmic complexity. Similarly, IPA only adds scalar subtraction and squaring operations; JS introduces only scalar multiplicatton,, addition, and division; MSPA adds only scalar multiplication and addition; and SFT-L requires only comparison and truncation operations. Hence, none of them significantly increase the algorithmic complexity. Overall, the added complexity resulting from the integration of multiple alignment components is deemed acceptable.
polst_1le.f.22) without significantly increasing
G Further Discussion on Hyperparameter Tuning
In this study, hyperparameter tuning is primarily guided by two main principles. Firstly, we employ a unified initialization for the regularization intensity 3, assigning the same value to all of them. For a fair comparison, we set this value to 10, consistent with the configuration used in D3PO (Yang et al. 2024a) and SPO (Liang et al. 2024). Secondly, we empirically anchor each component by fixing their values based on results from preliminary experiments. Specifically, we perform an individual grid search for cach hyperparamcter, including the SFT intensily X, the threshold 0, and the margin strengthening intensity y, While this might not be the globally optimal combination of hyperparameters, we want to convey that this paper transcends the pursuit of immediate performance enhancements and its contribution more lies in the articulation of novel rescarch avenues within the field, facilitated by methodological innovation.
In future engineering applications, especially when handling sigh-dimensional data or large-scale models, hyperaaraeeter optimization can be achieved through the following method. To begin, we need to define the acceptable range of values for each hyperparameter; and if there are sufficient resources, we can set it as a continuous range for finer-grained tuning, while, if resources are limited, we can empirically set it as discrete values to reduce the search space. For example, for the regularization intensity B, it could be [10, 100, 1000]; for the positive enhancement intensity X, it could be [50, 100, 500]; for the margin strengthening intensity n, it could be [0.1, 0.5, 1]. Further, we can expand this by automating hyperparameter tuning (like the methodology of Bayesian optimization that achiove dynamic balance between exploration and exploitation to approximate the global optimum with the fewest. number of evaluations). integrating with an AutoML framework (such as the Optuna). We believe that this methodology could enable superior performance when applied to larger models and more complex datascts in engineering applications.
H Theoretical Perspectives on CEPA, IPA and MSPA
In this section, we present further theoretical insights into CEPA, IPA, and MSPA from the perspective of functional analysis. We begin by defining the model function space in Definition 2 Subsequently, we explicitly formulate the functionals of CEPA, IPA, and MSPA within the model function space in Definition
Definition (Model Function Space). Setting that X is the sample space and that F is the real-valued measurable function space on X. For any function (model) fo E F, Vx E X, we have that fo(x) X - ohh where w and h represent the width and height of the image, respectively.
Definition 3 (Functional Expression).. Let the preference training dataset be D = ((x)) x/) C X x X. In the Hilbert space FD (D), consider the three functionals: C[So] (for the CEPA), I[fo] (for the IPA) and M[fo] (for the MSPA). Hence, for each pair sample (x.2 ,,)), the win ratão X and the lose ratio X! are uniquely determåned; thuus, we have that:
1
1
Clfal = B log (x("
+ B log fo(X!) +
(14)
2
2
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
1
][o]] log so(()) loggo((X)) 2,3
(15)
M[Jo] = - log ( (1+1) B (x(x)) 3 log Sa (x()))
(16)
Furthermore, let consider the Fréchct derivative of the functionals. The Fréchet derivative measures the linear variation of functional's value in response to small perturbations of its input function. In Proposition]] we present the calculation results of the Fréchet derivatives for CEPA, TPA and MSPA.
Proposition 1 (Fréchet Derivative of the Functionals). Let So be function in Hilbert space. We introduce rendom perturbation h and e denotes the magnitude of the perturbation. Then, under mild assumption, we can get the voriation of C[fel, I[fo] and M[fol as:
Bh(X)) fo(
8h] = )
2/3 log fo
+
233 10g fo(x)) -1)
(17)
(h) = log fo(X!)) - 2oo fo(X)) 1 h((P)))
MXx)
(18)
(X(')
So(X!)
8M(fol(h) ((11 (1) f(x!)) - 3 log f(x))) (17 1)53
h(x:")
((x))
(19)
fx(")))
So(X!)
Proof. Part 1. Variation Derivation of the C[Jo].
Firstly, introduce the random perturbation ch to the C[fol:
17
+ B (fo(X)) + ch((')) + 1
Cife chk log (Xo(X + ch(?")) 2
For each of the two terms, we perform a Taylor expansion as follows:
B log (Ja(X.") + ch(x)") Bog Bog [x(" + )
(X!) log (Jo(X)) + ch((x)) se Blog fo(X!) fox)) <3h(X!)
So(
Sctting that A Blog fo and B log fo 2 we have that:
1
<Bh(X:")
<3h(XF))
log +h(xx"))))+++)))
A
A 2A
+O(e):
2
[o(
To(X!")
1
(X)))
<Bh(x:)
I log (fo(x)) ch(:)))
B
B² +2B
+ Oc)).
2
So(X!)
So(X!)
Performing the differentiation to the e at
d clf + ch]
Bh(XP))
3h(())
de
2A (x(
2B
fo(X!)
e=U
Hence, the variation for C[fo] can be expressed as:
2 log fx(x))) fo(x)) + Bh(x))
23 log fo(x)) -1).
(C]
fx("
So(X!)
Part 2. Variation Derivation of the I[fo].
Firstly, introduce the random perturbation ch to the I[fo]:
17
Iise + ch) = log (Jo(X!)) + cl(x))) - log (Jo(X)) + +x((x))
23
26

---

Published in Transactions on Machine Learning Research (07/2025)
Similarly, perform the Taylor expansion to log (fo(X))) ch(X;")) and log (fo(X)) + ch(())))
log (fo(X))) + ch())") = log (fo(X:)) + c fo())))
log (fo(X))+ ch(x:)) = log (fo(x;)) + eh(X!)
fo(X!)
Thus, we have that:
log + - log (fo(X)) + ch(x))) = log fa(x;") - log fo(!) + (((x;") M(X))
[x([F) fo(X!)
Setting that C log So (X!") So(!))
23 and have that:
If + ch] ~ C+ (xx:)) MXX)
fo(! (x( ]
M(x:))
M(X))
C² + 2Ce
+ +0())
Fo(X!)
So(X!)
Performing the diffcrentiation to the c at c = 0:
h(x:")
h(x!)
de
Ilf + ch
2C
[x("))
fo(x!)
c=0
Hence, the variation for I[fo] can be expressed as:
8I 2og fo(X)) - 2 21og 2log [o(x)) - 1 (
((xF))
M(X))
To(XP)
fo(X!)
Part 3. Variation Derivation of the M[fel. Firstly, introduce the random perturbation ch to the M[fo]:
- log (1 (1+1)ap Blog fX( + cc((x()) foogfo(X; + ch(x3))
Performing the Taylor expansion (1+1) Blog Sx(" + +eh(X(") 3 log log lo(: + (x()))))
~ (n+ 1) log fxX)) + (X.")
ch(X."))
B log log (X.)
ch(X!)
[x()
= (1+ 3 log so(!") - 3 log fo (X!) + (1++1) So (x fo(X!)
h(x²)) h(x!))
Setting that D = (1) + (x(F) - 3 log fo (xt) and that E= (1) + 1).8 MXx))
TUXP 8 N(X))
then:
To(
Mifo + chl log (D+
As we know that, du logo(u) = 1 o( = 1 - o(u); then, performing the differentiation to the at O:
_ MIS + chi
E = (-(1o(D)) E.
de
Hence, the variation for M[fo] can be expressed as:
SM[So[(h) (0:1000--1---- (1-10
10g ((x") Blog s(x))) ) (1) (1)1))3
1)33 h(XP")
3 A(x!) [o(X!0)
fo()
which completes the proof.
27

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Based on the above calculation results, the following conclusion can be readily drawn.
Fact 1. Intuitively, MMfo) includes the nonlinear operator o()); therefore, therefore, it neither overlops with nor is compatible with C[fa] or I[
Furthermore, based on the type of perturbation, we reformulate the Fréchet derivatives of C[fo] and I[fo]
presented Proposition 1 into vector form as follows:
c(So] = (M(X))))xx)
(
B (To(xT))
28 log fo(X!") - -1).
B f0(x;"
28log So(X:) -1) -1))
(x!
Gc
I[fol= = (Mxx)))(())
2 log 2log fo( 2og - 1/3 fo()!")
2 log fo(")") - 22 log fo(X.) 1/3 T
fo(X!)
Gz
Finally, based on the results above, further elucidate the relationship between the Fréchet derivatives of C[fa] and I[fe], as presented in Theorem [
Theorem 1. The Fréchet derivative of the functionals Clfol and I[fe] are non-redundant (linearly independent) in the gradient space for almost every So E F(D) that salisfies fo(X > 0 and Jo(X!) > for all
Proof. As we know, if C(Jo] and I[fe] are redundant, there must be two constants o, b that makes:
Ge =a +b,,
where I is the alltono-vector, representing the constant offsct. Hence, there should bc:
B
2,3 [x(?") -1)= =a a (2log (X!:) -2 2og fo(X!) 1/B
(20)
"
)x(
3
23 log fo (X.) +1) = -a. (2log
(2log (o
2 log fo (x.) 1/3
+
(21)
(x!)
For all fo(X)) 0, fo(X!) 0 and so (x(")) + [o())), we further derived that:
a 3 fx(X)) (2 log fe (X."") 1) fe(X))) (og fo(X)) + 1)
(2 log fo (X.") - 2 log fx( 1/8) (fo(X;) fo(X!"))
b 2,3 log (x(()) log (x())))
+
It evident that both a and are functions of fo(X!") and fo(X),, rather than fixed constants. Therefore, in this case, the functionals C[fo] and [[fa] are non-redundant.
Furthermore, for all fe(X 0, fx() and fo fo(X))) 2, we have that:
82
= 282 log Z Z
We can observe that a is constant, whereas b is not. Consequently, in this context, C[fo] and I[fo] are also non-redundant.
Therefore, based on the precoding discussion, we can conclude that the Fréchet derivatives of the functionals Cl.fo] and [[fo] are non-redundan (i.e., linearly independent)) in the gradient space for almost every fo E F(D) that satisfies So Jo(X)) > 0 and fo(X!) > 0 for all i. This completes the proof.
28

---

Published in Transactions on Machine Learning Research (07 2025)
Results across Different Initialization Seeds
To further validate the stability and robustness of the results across different initialization sceds, we evaluate the using five different random seeds (seed=40, 41, 42, 43, 44). Additionally, it is important to note in practice that the seed value used during generation and testing should be consistent with the one set during model training. For example, if the seed is set to 40 during training, the same seed value should also be set 40 during generation and testing.
performance of
Table 11: Performance Comparison over Five Seeds the GenEval.
Diffusion -DPO
GenEval
SD-1.5
SPO
Seed=10
Seed=11
Seed=12
S3ed=13
Seed=11
Average
VQAScore
61.85
63.68
61.22
69.89
70.35
68.41
69.89
68.66
69.44
CLIPScore
33.77
34.52
33.41
34.04
35.73
35.01
35.70
35.42
35.18
HPS-V2
26.93
27.17
27.30
27.52
27.44
27.45
27.67
27.64
27.54
ImageReward
-0.200
-0.030
0.083
0.234
0.322
0.202
0.359
0.236
0.271
Table 12: Performance Comparison over Five Seeds the T21-CompBench++H
Diffusion -DPO
Alllusion-Rainbowee
T
SD-1.5
SPO
So
Se
42
Seed=43
Sced=44
Average
VQAScore
60.71
62.61
62.73
68.98
67.73
67.60
69.20
67.59
68.21
CLIP'corre
32.06
32.66
31.19
33.13
33.87
33.48
34.16
33.55
33.64
HPS-V2
26.55
26.78
26.87
27.19
27.17
27.15
27.31
27.30
27.22
Iewae
-0.245
-0.089
-0.023
0.299
0.232
0.230
0.339
0.247
0.269
Table 13: Performance Comparison Five Seeds the GenAT-Bench.
Diffusion -DPO
GenAl-Bench
SD-1.5
SPO
Se 40
Soed=41
Scod=42
Seed=43
Seed=44
Average
VQAScore
59.50
60.15
59.52
63.03
62.24
62.58
63.99
62.49
62.87
CLIPScore
33.62
34.11
32.40
33.75
34.34
34.47
34.60
34.33
34.30
HPS-V2
26.94
27.15
27.27
27.56
27.47
27.53
27.65
27.62
27.57
ImageReward
0.237
0.376
0.360
0.567
0.490
0.593
0.584
0.514
0.550
Table 14: Performance Comparison over Five Seeds on the DPG-Bench.
Diffusion -DPO
Mnifusion-RRinnowPP..
DPG-Bench
SD-1.5
SPO
Seed=10
Seed=11
Seed=12
Seed=13
Seed=11
Average
VQAScore
76.28
77.13
75.61
79.19
78,88
79.18
78.66
77.80
78.74
CLIPScore
33.42
34.01
32.13
34.76
34.56
34.72
34.86
34.80
34.74
HPS-V2
26.31
26.51
26.73
27.11
26.86
26.95
27.01
26.98
26.98
ImageReward
-0.187
-0.036
-0.087
0.157
0.100
0.166
0.143
0.075
0.128
29

---

Published in Transactions on Machine Learning Research (07 2025)
J Detailed Prompts for Figure 1
We summarize the detailed text prompts utilized in Figure as follows (cach line, from left to right):
Line 1 (on generation of animal subjects):
1. A fluffy white Samoyed dog standing in colorful flower garden in front of a rustic house, highly detailed, digital art
white puppy sitting playfully in autumn leaves, surrounded by fallen red apples, soft natural lighting
cute duck wearing a chef uniform covered cookie batter, unreal engine render 8k
4. A realistic Venusaur animal among the trees, forest lake, moss, cold weather, dark teal and amber, Sony A7 IV
5. A fluffy white cat standing on a lush green hillside under clear sky with scattered clouds, serene natural landscape, cartoon style
6. Create image of cat as gardener, wearing straw hat, gardening gloves, and surrounded by colorful flowers
Line 2 (on generation of humanity):
1. A whimsical pink cloud-shaped building with minimalist windows and doors, floating above vibrant blue sky with couton-like clouds, Studio Ghibli-style animation movie lexture
2. Vibrant city skyline during sunset, modern skyscrapers, colorful abstract style, warm gradient sky, digital art, urban landscape, vivid colors
3. Vibrant Christmas tree, glowing lights, abstract background, festive atmosphere, painterly style, bright, and vivid color
Heavily decorated pile of donuts in dark red, black and gold with icing and lace trimmingg dramatic lighting, dark background
Mystical forest with glowing mushrooms and babbling brook
6. Album art of hand holding balloon emerging from the water against red sky
Line 3 (on generation of portrait, with 3 for male and for female):
1. 4d photographic image of full body image of cute little chibi boy realistic, vivid colors octane render trending on artstation, artistic photography, photorealistic concept art, soft natural volumet cinematic perfect light, UHD no background
2. Heroic elf warrior, golden glowing background, detailed fantasy armor, cinematic lighting, epic fantasy art, high detail
3. An intricately detailed close-up portrait of elderly man with long gray beard, insane face details, extremely intricate, high res, 8k, award winning
Extreme close-up shot portrait of short Monde-haired beautiful woman, capturing the essence with blue eyes, lit by overhead lighting
bcautiful brunctte pilot girl, bcautiful, moody lighting, best quality, full body portrait, real picture, intricate details, depth of field, in a cold snowstorm, Fujifilm XT3, outdoors, Beautiful lighting, RAW photo, 8k, film grain, unreal engine 5, ray trace
30

---

Published in Transactions on Machine Learning Research (07/2025)
6. Anime illustration of Princess Mononoke from Studio Ghibli, by artgerm, stunning artwork
Line 4 (on generation of futurism):
dragon front face, backgrounds like hell, chain effects, angry, 3d, 8k, realistic and details, storm effective
2. A surreal alien landscape with a massive planet in the sky, rocky terrain, dramatic lighting, cinematic atmosphere, sci-fi theme, warm toncs, highly detailed, 4K resolution
3. A lone astronaut standing on volcanic landscape, detailed lava rocks, dramatic clouds in the sky, cinematic lighting, vibrant colors, realistic textures, 8k
4. Futuristic metallic humanoid robot, highly detailed face, sci-fi background, cinematic lighting, dystopian cityscape, 4K resolution
Futuristic cyberpunk city at night, neon lights, high-tech car, vibrant colors, cinematic lighting, highly detailed, sci-fi atmosphere, resolution, unreal engine
6. A woman in black bodysuit, standing in dark room, futurism, dramatic lighting, full-body view, cyberpunk, high contrast, detailed, 4k
31


Reviews:
[{"review_id": "AG4da7uxpb", "reviewer": "Reviewer_CZKt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Its very intutive paper, especially reasoning of each approach is well documented.\n2. They build on existing approach but provide justification of each modification step.\n3. Good amount of experiments to support their hypothesis.\n4. The use of Jensen-Shannon divergence and the explicit focus on promoting generative diversity over the typical KL divergence penalty is a significant contribution. It adds more stability and flexibility in optimization without compromising output quality. I am really impressed by this finding, as we all are aware of the similar improvements for stability in GAN.\n5. The adoption of the SPO paradigm, with preference supervision at each diffusion step, offers a more efficient training process. This step-wise training helps mitigate errors in earlier steps, leading to better final outputs.\n\nWeakness\n1. The modifications introduced in the paper, such as enforcing a margin in MSPA and step-aware preference optimization, could face scalability issues in more complex or large-scale generative tasks. The scalability of these methods, especially when applied to higher-dimensional data or larger models, isn't fully discussed.\n\n\nNote: I also appreciated the clarity of the paper and the clear evidence provided as supplementary material to verify the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is interesting as it derives most of the alignment and improvements from first principles. At first glance, this paper is not introducing any new model or method, but rather it is reintroducing existing solutions to problems by carefully building up from first principles. It carefully reviews most of the components, such as alignment issues, preference optimization, and loss functions supporting preference optimization.\n\n1. Step-Aware Preference Optimization\n\nRainbowPA adopts the SPO paradigm (Section 4.1 of the paper) as part of its framework. The authors use a step-aware preference model (sometimes a separate model that can evaluate noisy partial images) and ensure that each denoising step’s output is aligned. They also simplified the original SPO setup by using two sample trajectories instead of a larger pool (to reduce complexity and overfitting). By consistently integrating the paradigm of \"step-aware preference\" in RainbowPA, the model receives preference supervision at each diffusion step. By the time the diffusion reaches the final steps, the model has consistently preferred better partial solutions, so the final image is of higher quality. In contrast, a DPO-only model might accumulate errors in earlier steps (since it didn’t specifically train on them) and only correct some of them at the end.\nSPO provides a much denser training signal (on every step), which can make learning more data-efficient and stable. Instead of one comparison = one gradient update, one comparison provides supervision over T steps.\n\n2. Calibration Enhancement (CEPA)\n\nCEPA calibration has two benefits: First, it prevents the model from becoming overconfident in its preferences. Without CEPA, the optimal solution to DPO’s logistic objective is to make the reward gap as large as possible. With CEPA, the model instead targets a moderate reward gap of 1.0 between preferred and non-preferred (0.5 – (–0.5)), which is sufficient to reflect human judgment without overshooting. Second, it improves alignment – because the model’s notion of a “win” or “loss” is kept consistent with the ±0.5 reward, the model is less likely to sacrifice prompt fidelity for the sake of marginal preference gains.\n\n3. Overfitting Mitigation\n\nThe authors point out that models which are extensively fine-tuned tend to overfit on the trained data. To avoid this issue, the authors explore a solution from first principles of mathematical understanding.\n\n“We circumvent the Bradley–Terry modeling assumption and employ an identical mapping to the preference function to alleviate the over-confidence.”\nThis means that, instead of converting reward differences into probabilities (via a sigmoid/logistic model), they train the reward difference directly to match a known value (e.g., a calibrated target).\n\n4. Jensen-Shannon (JS) Divergence\n\nThis is an amazing idea derived from KL divergence. In simple words, KL is forward, while JS is symmetric and bounded. Using JS divergence leads to more stable and diverse optimization than KL. Since JS treats both the model and the reference more equally, the new model is not as harshly penalized for generating novel but plausible images that the original might have overlooked. This helps generative diversity.\n\n5. Performance Optimization\n\nDPO’s preference objective (and even with IPA) only requires that the preferred image’s log-likelihood be higher than the dispreferred’s. In practice, especially under logistic loss, a small increase is enough to classify correctly – there’s no built-in requirement for a large gap once the ordering is right. This can lead to cases where the model’s outputs for preferred vs. non-preferred samples are very close, making it brittle (a slight change could flip the preference). MSPA (Margin Strengthened Preference Alignment) introduces an explicit margin to preference alignment: the model is trained to separate the log-likelihoods (or scores) of preferred vs. dispreferred outputs by a larger amount.\nApproach: MSPA modifies the preference loss to encourage a minimum gap (Δ) between the model’s treatment of winners and losers. This can be implemented in a margin-based loss style, such as hinge loss.\nEnforcing a margin leads to more decisive preferences and smoother training dynamics. Gradient flow is stabilized because once the margin is satisfied, the loss gradient goes to zero (in a hinge formulation), preventing the model from endlessly increasing the gap.\nIn terms of outcomes, MSPA improves alignment by making the model reliably distinguish high-quality vs. low-quality generations.\n\n6. SFT-like Regularization\n\nRainbowPA introduces an SFT-like element for diffusion models. Pure preference training (as in DPO) only cares about relative rankings of two samples, not their absolute quality. This means if both images in a pair are poor (or both are decent), as long as one is slightly better, the model’s update is small – it doesn’t directly push the winner to be as good as it could be. SFT-like regularization addresses this by explicitly encouraging high likelihood for preferred samples in an absolute sense, beyond just beating the other sample by a tiny margin.\n\nBy injecting this supervised-like signal, the model is directly taught to increase the likelihood of human-preferred outputs instead of only worrying about their relative ranking in pairs. This yields two main benefits:\n\n1. Higher absolute image quality – because the model is encouraged to reproduce images (or image features) that humans liked, it leans more into modes of the distribution that correspond to aesthetically pleasing or on-point images. It’s as if we fine-tuned the model on a set of high-quality generations (even though it’s implicit in comparisons).\n2. Better alignment beyond the comparison baseline – the model doesn’t stop at just barely preferring the better image; it learns to make the preferred image really likely, pushing the distribution in the direction of human preference in a way similar to classical fine-tuning on a curated set. This term likely helps combat any residual tendency of the model to output mediocre images that just slightly edge out others.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None. I really enjoyed reading the paper", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aWl3hKSPlG", "reviewer": "Reviewer_uURe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* Clearly articulated rationale for each methodological improvement.\n* Comprehensive and rigorous experimental evaluations on multiple standard datasets.\n* Consistent improvements demonstrated across diverse metrics and datasets.\n* Thorough ablation studies effectively validate the contribution of each proposed component.\n\n**Weaknesses:**\n\n* Insufficient theoretical discussion of potential overlaps or redundancies among the six methods, particularly between CEPA, MSPA, and IPA, which seem to partially address similar alignment and optimization issues.\n* The manuscript lacks a theoretical analysis of the interaction and compatibility between the proposed components, potentially raising questions about their collective efficacy.\n* Relies primarily on automated metrics without human evaluation to assess subjective visual quality improvements, which is essential for a method addressing aesthetic preferences.\n* Experimental results do not sufficiently discuss randomness and stability across repeated trials.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces \"Diffusion-RainbowPA,\" an integrated framework aimed at improving the alignment of diffusion-based text-to-image (T2I) models with human preferences. It builds upon the Diffusion-DPO framework, incorporating six distinct enhancements: (1) Step-aware Preference Alignment, (2) Calibration Enhancement Preference Alignment (CEPA), (3) Identical Preference Alignment (IPA), (4) Jensen-Shannon Divergence Constraint, (5) Margin Strengthened Preference Alignment (MSPA), and (6) SFT-like Regularization. The authors highlight the core issues of existing T2I models, specifically text-image misalignment, aesthetic overfitting, and poor image quality. Empirical evaluations confirm superior performance of Diffusion-RainbowPA across multiple standard benchmarks, such as GenEval, T2I-CompBench++, GenAI-Bench, and DPG-Bench.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Provide detailed theoretical analysis and clear discussion regarding the potential redundancy and compatibility between the CEPA, MSPA, and IPA components.\n\n* Discuss explicitly whether the coefficients used in the combination process of Equations (12)-(13) were carefully considered, and justify why or why not these coefficients significantly impact the effectiveness of the integrated approach.\n\n* Include additional experimental results involving human evaluators to validate the subjective quality improvements suggested by automated metrics.\n\n* Expand the discussion on hyperparameter selection, clearly explaining the strategies and criteria for tuning each component and their interactions within the final integrated framework.\n\n* Provide additional analysis or experiments demonstrating the stability and robustness of the results across different initialization seeds and data subsets, helping to clarify the consistency of improvements.", "broader_impact_concerns": "The authors acknowledge potential ethical implications and propose safety measures.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6FKI0wABFZ", "reviewer": "Reviewer_QTKW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. Clearly identifies critical shortcomings (text-image misalignment, aesthetic overfitting, low-quality generation) in existing T2I alignment methods.\n\n2. Introduces a comprehensive alignment improvement framework (Diffusion-RainbowPA) integrating calibration enhancement, overfitting mitigation, and performance optimization.\n\n3. Demonstrates strong experimental results, significantly outperforming state-of-the-art methods across multiple benchmarks.\n\n4. Provides thorough ablation studies, effectively validating the contribution of each individual improvement component.\n\n5. The paper is clearly written with structured methodology, facilitating good reproducibility.\n\n---\n\n### Weaknesses:\n\n1. The complexity of integrating six improvements could raise concerns about practical ease-of-use and hyperparameter tuning efforts.\n\n2. Limited exploration of computational overhead introduced by the additional components.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies shortcomings in current SOTA diffusion-based text-to-image alignment methods, text-image misalignment, aesthetic overfitting, and low-quality generation.\n\nTo tackle these issues, the authors introduce the Diffusion-RainbowPA, an approach integrating six enhancements addressing three key aspects: (1) calibration enhancement, (2) overfitting mitigation, and (3) performance optimization.\n\nExperimental evaluations demonstrate that Diffusion-RainbowPA outperforms existing methods across multiple benchmarks, validating each introduced improvement through detailed ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide explicit analysis or discussion about the computational overhead and complexity introduced by integrating multiple alignment components.\n\nOthers (But not necessary in my opinion):\n\n2. Include results from human evaluations alongside automatic metrics to more convincingly demonstrate practical alignment effectiveness.\n\n3. Offer guidelines or empirical insights on hyperparameter sensitivity and tuning strategies for the Diffusion-RainbowPA components.", "broader_impact_concerns": "The Broader Impact is well stated in page 12. It would be better to give more details of safety filtering mechanisms.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)