SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedback
Authors: Xiaoxuan He
Siming Fu
Wanli Li
Zhiyuan Li
Dacheng Yin
Kang Rong
Fengyun Rao
Bo Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion models;RL
Primary area: generative models

Abstract: Aligning diffusion models with human preferences remains challenging, particularly when reward models are unavailable or impractical to obtain, and collecting large-scale preference datasets is prohibitively expensive. This raises a fundamental question: can we achieve effective alignment using only minimal human feedback, without auxiliary reward models, by unlocking the latent capabilities within diffusion models themselves? In this paper, we propose SAIL (Self-Amplified Iterative Learning), a novel framework that enables diffusion models to act as their own teachers through iterative self-improvement. Starting from a minimal seed set of human-annotated preference pairs, SAIL operates in a closed-loop manner where the model progressively generates diverse samples, self-annotates preferences based on its evolving understanding, and refines itself using this self-augmented dataset. To ensure robust learning and prevent catastrophic forgetting, we introduce a ranked preference mixup strategy that carefully balances exploration with adherence to initial human priors. Extensive experiments demonstrate that SAIL consistently outperforms state-of-the-art methods across multiple benchmarks while using merely 6\% of the preference data required by existing approaches, revealing that diffusion models possess remarkable self-improvement capabilities that, when properly harnessed, can effectively replace both large-scale human annotation and external reward models.

Published as a conference paper at ICLR 2026
SAIL: SELF-AMPLIFIED ITERATIVE LEARNING FOR DIFFUSION MODEL ALIGNMENT WITH MINIMAL HU- MAN FEEDBACK
Xiaoxuan He2.2 Siming Fu Wanli Li', Zhiyuan Li,, Dacheng Yin", Kang Rong?. Fengyun Rao?, Bo Zhang
ZheJiang University,
2 WeChat Vision, Tencent Inc
3 Independent Researcher
ABSTRACT
Aligning diffusion models with human preferences remains challenging, particu- larly when reward models are unavailable or impractical to obtain, and collecting large-scale preference datasets is prohibitively expensive. This raises a fundamental question: can we achieve effective aligument using only minimal human feedback, without auxiliary reward models, by unlocking the latent capabilities within diffu- sion models themselves? In this paper. we propose SAIL (Self-Amplifidd Iterative Learning), a novel framework that enables diffusion models to act as their own teachers through iterative self imprrvemment.. Starting from a minimal seed of preference pairs, SAIL operates in a closed-loop manner where the model progressively generates diverse samples, self-annotates preferences based on its evolving understanding, and refines itself using this self-augmented dataset. To ensure robust learning and prevent catastrophic forgetting, we intro- duce ranked preference mixup strategy that carefully balances exploration with adherence to initial human priors. Extensive experiments demonstrate that SAIL consistently outperforms state-of-the-an art methods across multiple benchmarks while using merely 6% of the preference data required by existing approaches, revealing that diffusion models possess remarkable self-improvement capabilities that, when properly harnessed, can effectively replace both large-scale human annotation and external reward models.
INTRODUCTION
Diffusion models have revolutionized generative AI, enabling the synthesis of high-fidelity images with remarkable diversity (Ramesh et al.] 2022 Saharia et al. 2022: Rombach et al. 2022, Podell
al. 2023 Esser et al. 2024). However, aligning these models with human preferences remains a fundamental challenge, particularly in practical scenarios where reward models are unavailable or impractical obtain (Black et al. 2023b Fan al.) 2023 Clark et al. 2023 Wallace al. 2024). This alignment problem becomes even more critical as diffusion models are increasingly deployed in real-world applications requiring nuanced understanding of human aesthetic and semantic preferences (Li et al. 2024: Hong et al. 2024)..
Current approaches to preference alignment face critical dilemma. Methods like DiffusionDPO (Wal- lace et al.] 2024) achieve strong alignment but require massive human-annotated preference datasets often millions of ranked maiisg them prohibitively expensive and inflexible to evolving preferences. Alternatively, approaches utilizing external reward models (c.g., Aesthetic- based scorers (Black et al. 2023b Fan et al.) 2023)) introduce secondary biases and are vulnerable to reward hacking (Fu al. 2022 Liu et al. 2024), while struggling with distributional shifts beyond their training data. Both paradigms create problematic dependencies either on exhaustive human
Equal Contribution.
1 Project Leader.
t Corresponding authors.

---

Published as a conference paper at ICLR 2026
Traditional Offline (with Fixed Preferences)
Traditional Online DPO
Our Method: SAIL
(with External Reword/ )nnort/oon)n))
imxte pairs
Diffusion
biffusion
External Model
Iteration Training
Generation
Iteration Training
Generation & Label
Label
Text-inage pairs
Humen
Annotator
External Model
Large Prefenence Detaset
Label
Human
DPO Training
Annotctor
fiffision Model
Large Preference Dataset
Small Preference Dataset
Figure 1: Comparsion of three direct preference optimization methods. Different from Offline DPO and Online DPO, SAIL iteratively update without large preference dataset and external reward model.
annotation efforts or on auxiliary models that may not generalize well-fundamentally limiting their practical applicability.
This raises a crucial question: Can we achieve effective preference alignment using only minimal human feedback, without auxiliary reward models, by unlocking the latent alignment capabilities within diffussion models themselves? We argue that diffusion models, once exposed to even a small of human preferences, possess the inherent ability to act as their own expanding their understanding through iterative self seff-imprrvement. This insight fundamentally reimagines the alignment process: rather than treating models as passive learners requiring constant external supervision, we can leverage their generative and discriminative capabilities in tandem. In this paper, we propose SAIL (Self-Amplified Iterative Learning), the first implicit self-rewarding framework that enables diffusion models to achieve strong preference alignment through autonomous bootstrapping. As illustrated in Figure T SAIL operates through a closed-loop learning process: starting from minimal seed set of annatttte preference pairs, the model iteratively generates diverse samples, self-annotates preferences based on its evolving understanding, and refines itself using this self-augmented dataset. The key innovation lies in mathematical quantification of relative reward values between image pairs, enabling the diffusion model to serve dual roles as both generator and evaluator when conditioned on fixed reference parameters.
To ensure robust learning and prevent distribution collapse, we introduce a ranked preference mixup strategy that carefully balances exploration of the preference space with adherence initial human guidance. This mechanism addresses the critical risk of catastrophic forgetting in self-training scenarios, maintaining alignment with human priors while enabling the model to discover nuanced preference patterns beyond the original annotations. Figure demonstrates not only the effectiveness of our approach but also remarkable stability over extended iterations. Our contributions to the community include:
We propose SAIL, the first self-amplifiet iterative learning framework that enables diffusion models achieve effective preference alignment through autonomous bootstrapping, elimi- nating dependencies on large-scale annotations and external reward models by developing mathematical framework for self-reward quantification.
We design ranked preference mixup strategy that prevents catastrophic forgetting and ensures stable self-imprrvementt, enabling the model to balance exploration of the preference space with adherence to human priors across extended iterations.
Extensive experiments demonstrate that SAIL consistently outperforms state-of-the-art methods HPSv2, Pick-a-Pic. and PartiPrompts benchmarks while using merely 6% of typical preference data, achieving superior qualitative results in texture and textual detail generation.
2

---

Published as a conference paper at ICLR 2026
Acsthetics
ImageReward
HPSv2
DiffusionDPO
DijjusionDPO
0.20
3.6
26.6-
DiffusionDPP
Scores
Scores
0.1
Scores
0.0
542
005
263
538
SD1.5
000
SD1.5
SD1
2
Iterations
Iterations
Iterations
Figure 2: Iterative performance improvement with generated data of SAIL on Pick-a-Pic validation dataset Aesthetics, ImageReward, and HPSv2. During the iterative process, SAIL demonstrates steady improvement and ultimately surpassed DiffusionDPO (as indicated by the dashed line).
2
RELATED WORK
2.1 HUMAN PREFERENCE OPTIMIZATION
Beyond mere visual fidelity, a critical frontier in text-to-image generation is aligning outputs with nuanced human preferences. A predominant strategy has been inspired by RLHF. For example, IeageReward ReFL (Xu al.) (023a) first train reward model on 137K pairwise human comparisons and then fine-tune a diffusion model by backpropagating reward gradients, yielding substantial gains in aesthetic and caption alignment. Denoising Diffusion Policy Optimization (DDPO) (Black et al. (023a)) treat the denoising trajectory as an MDP and apply policy gradients directly optimize black-box rewards such as CLIP similarity aesthetic scores. DiffusionDPO (Wallace et al., 2024) align diffusion models to human preferences by directly optimizing on human comparison data. Building upon this framework, several advanced variants have emerged, MaPO (Hong et al.) 2024) jointly maximizes the likelihood margin between preferred and dispreferred image sets, the absolute likelihood of preferred samples. SPO (Liang et al. 2024) introduces employs step-by-step optimization strategy, enabling finer control over localized quality improvements. The above methods rely on either large-scale preference datasets or pre-trained reward models. A critical yet understudied direction lies in self-alignment: leveraging the model's I's inherent generative capabilities to bootstrap preference learning without external supervision. This capability highly valuable in scenarios that require either strong sense of realism distinct artistic style (no reward model).
optimization,
2.2 ONLINE DIRECT PREFERENCE OPTIMIZATION
The Direct Preference Optimization (DPO) framework (Rafailov et al., 2023). initially for large language model alignment (Wu et al., 2025) directly refines policies with preference pairs without training an explicit reward network. A critical challenge in aligning diffusion models with human preferences lies in the inherent off-policy nature of conventional approaches: while the model continuously updates during training, the preference dataset is typically collected priori, leading to a growing divergence between the model's current behavior and the static training data. Online AI feedback (Guo et al.) 2024), uses an LLM as annotator: sample two responses from the current model and prompt the LLM annotator to choose which one preferred, thus providing online feedback. Some methods eliminate the need for external annotators altogether by repurposing the DPO model itself as an implicit reward model (Kim et al. 2024| Chen et al.) 2024 Cui et al., 22025), enabling iterative self-impru velf-improvemmnt. This approach is particularly appealing for diffusion models, where collecting high-quality preference data is inherently more challenging than in language tasks, and where single reward model often fails to capture the nuanced, multi-diimensonal aspects of image quality (e.g., composition, realism, and aesthetic appeal).
3 PRELIMINARY
Direct preference optimization. Direct Preference Optimization (DPO) [Rafailov et al,, 2023) is recently developed approach for aligning LLM up with human preferences. The key idea behind DPO reparameterize the reward function in terms of the policy itself, eliminating the need for explicit
3

---

Published as a conference paper at ICLR 2026
Sanpled Poins
Sna loft Anwor Ranking
Serpled Seed Pars
Mlup with def anuturice
Preses Seed
Sampled Daha
Sel boesting DPD Medule
Pronpts Pool
Medel
Self Rewording
Horation
siable haratian
boasting loop
Figure 3: Illustration of the proposed SAIL framework. The SAIL framework incrementally refines the alignment diffusion models through iterative cycles consisting of generating new preference data and conducting preference learning using mixup ranked preference data complemented by self-refinement mechanisms. This closed-loop self-boosting process operates with minimal initial data input, aiming optimize performance by capitalizing on the intrinsic capabilities of the model, independent of external reward systems.
reward modeling. Specifically, the optimal reward function is derived from the RLHF objective, with the target LLM my and the reference model Tral-
To(x)y) Tief (x
r x) 3 log
B log (zy)
Then, the preference between two responses could be measured using this reward derivation, and optimized maximize this preference of x" over x' using the preference dataset D.
Po( ( > x (1y = o(3 <x - Blog
To (x (1)
(2)
Tee( (
(TDo(TT) Figy Flyy,=m=')ED x') log (a > a a11y)]
METHOD
The key insight is the development of self-rewarding direct preference optimizaation framework, which is designed to iteratively maximize alignment with human preferences via closed-loop without any external reward model within few seed data. Specifically, we start with seed preference dataset Dim {x"" x², ) and a pre-trained diffusion model eq. i.c. SD1.5 or SDXL. The initial step involves fine-tuning cD on Din using the DiffusionDPO (Wallace et al. 2024) to update cg. Subsequent updates to the diffusion model are performed iteratively, leveraging self self-generated data and self-rewarding mechanisms to continually improve the model's performance. The overview framework is demonstrate in Figure
4.1 HELF-REWARDING PERFERENCE RANKING WITH SELF- DATA
Given the candidate images 4 and BB, the reward difference between two images is shown as Equation 2
The Equation can be derive
A x (y) = (r(y,, x² - r(y, x0
(4)
In diffusion models, the optimal reward function can be derived as:
(o y t. qe (xo)) + 3 log Z(y, t, (xo)) (x y. t, gr (2o))
r(y. x) = 3 log
(5)
4

---

Published as a conference paper at ICLR 2026
Algorithm 1
Input: Diffusion Model eli, seed preference dataset Dinst. number of improving iterations T, new prompt sets (Yi)==
Obatin Human preference model eß using Diffusion- DPO with cj and Dixiit
for to 7' do
Candidate Generation. Sample N candidate images (ax(1), (.. ax()) from the model eg- Self- belf-Rewarding Ranking. Rank the candidate images and choose the best image and the
worst image in N images construct paired preference data D; with ea and eg.
Mixup Ranked Preference Data. Mix the generated data with seed preference dataset.
o'D; + inni
Closed-loop Boosting Preference Optimization. Update the current model with Eq. 3 and
obtain cg
end for
return <)
T+1
where is the timestep and (zo) vojan + v1 04 is the combination of ap and ~ NO, I). Of noise scheduler. With the noise prediction co of diffusion models, we can derive and simplify the term po y, L, (x=))::
(mr)
0460
po t, Na
6,I)
(6)
Va
|e-
22g
Since log (ly, (e()))) is the same for fixed prompt y. The reward function for image ' can be formulated:
3 (IlE* (x y, t)l1 || - Exe((a y, 1)1))
r (y
(8)
Finally, can apply the Equation reward value of images 4: and
8 in Equation 4 lnd obtain the following term to judge the relative
y, 2) 3 [[1e -x
A > x (1)) or
||E (xr(( ( y. 1)
(||E B - ca (m 0)11 ||. " (xr 1 1)1)))
In enhancing the precision of estimates, it proves beneficial
average across multiple samples,
denoted as (1, ge (2t). We compute estimates based on 10 random draws of (t, q:(xn)) Consequently. the assignment of preference labels to the tuple (x= 2x²) is governed by the following formulation:
(a" x²) = (x A B) if (x
x y) > 0.5 else (a A
(10)
We choose the best image and the worst image in N images construct paired preference data. Following the construction of the dataset D, the i-th iteration of preference learning is executed by fine-tuning the diffusion model eg. This training on the self-generated dataset DD is aimed at
enhancing alignment by propagating the human preference priors encapsulated in Do through the capabilities of the diffusion model. More details in Appendix
4.2 CLoSED-LLOPP BOOSTING DIFFUSION MODEL. WITH MIXUP RANKED PREFERENCE DATA
Despite these advantages, there exists the risk of distributional collapse and overfitting synthetic data during iterative self-improvement. To address these challenges, we propose an enhancement to the preference learning methodology by integrating inspired by experience replay (Zhang & Sutton (017) in reinforcement learning, designed to stabilize the learning process against such perturbations and ensure robust preference alignment.
mixup ranked preference data strategy
Especially, for the i-th iteration (i = 1, 1....), we assume that the new prompt set Y; = {v} is available, ie., Y, O Y, 0 for all O,...,i As summarized in Algorithm During each iteration i,
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with other methods on SD1.5 and SDXL. For HPSv2, we report the score in Anime, Concept-Art, Painting and Photo. For Pick-a-Pic v2, we apply PickScore, ImageReward, Aesthetics and HPSv2 metrics evaluate all methods. The best results are highlighted in red and the performance gain are highlighted in boldface.
HPSv2
Pick-a-Pic V2
Model
Method
Data
Ani.
Con.
Paint.
Photo
P.S.
L.R.
Aes.
HPSv2
27.23
26.65
26.52
27.41
20.62
-0.0130
5.38
26.21
Dif
0.8M
27.64
26.97
26.90
27.56
21.07
0.2056
5.48
26.57
DiffusionSPO
0.8M
28.05
27.49
27.61
27.55
21.20
0.1577
5.68
26.75
SAIL (IterO)
0.05M
27.41
26.75
26.72
27.50
20.80
0.0715
5.42
26.38
SD1.5
SAIL (Iterl)
0.05M
27.56
26.88
26.83
27.59
20.89
0.1137
5.46
26.49
SAIL (Iter2)
0.05M
27.75
27.06
27.03
27.74
20.95
0.1729
5.47
26.65
SAIL (Iter3)
0.05M
27.88
27.16
27.15
27.79
21.00
0.2329
5.49
26.75
+0.65
+0.51
+0.63
+0.38
+0.38
+0.2459
+0.11
+0.54
28.03
27.17
27.22
27.50
22.13
0.6891
6.04
26.80
DiffusionDPO
0.8M
28.71
27.75
27.82
27.89
22.59
0.9336
6.02
27.27
MaPO
0.8M
28.39
27.60
27.58
27.74
22.24
0.8227
6.16
27.05
SDXL
SAIL (Iter0)
0.05M
28.41
27.50
27.49
27.76
22.40
0.8705
6.04
27.08
SAIL (Iterl)
0.05M
28.59
27.62
27.59
27.90
22.53
0.9355
6.08
27.21
SAIL (Iter2)
0.05M
28.74
27.80
27.78
28.09
22.51
0.9844
6.16
27.32
+0.71
+0.63
+0.56
+0.59
+0.38
+0.2953
+0.12
+0.52
For each prompt y E Yú, we sample N candidate images (..")).....? by utilizing the intrinsic generation and reward modeling capabilities of the diffusion models cj. where ep is the resulting model from the previous iteration. Then, using the reward captured with e) and ei (Eq. 5 we measure the relative preference between (!) and x (N) and construct generated preference dataset D,. After that, we construct the mixed dataset D; by sampling a proportion of data from D; and (1 a)) proportion of data from Dinit- Finally, DPO training is conducted on D; using e as both the initial policy and the reference policy, resulting in the updated model e
EXPERIMENTAL RESULTS
5.1 EXPERIMENT SETTINGS
Implementation Details. We demonstrate the effectiveness of SAIL across a range of experi- ments. We apply Stable Diffusion 1.5 (SD1.5) (Rombach al. (022) and Stable Diffusion XL-1.0 (SDXL) (Podell et al. 2023) as our base model. For preference learning dataset, we utilize Pick-a-Pic dataset (Kirstain al. 2023), following the previous work. We use the larger Pick-a-Pic v2 dataset. After excluding the 12% of pairs with ties, we end up with 851,293 pairs, with 58,960 unique prompts. We first randomly select 50K preference data from the larger Pick-a-Pic v2 dataset, and then choose the remaining prompts from the remaining prompts for self improvement. We apply SAIL three iterations on SD1.5 and iterations on SDXL, each iteration with 10K, 20K, 20K prompts. Morever, we set the mix ratio of human preference data and generated preference data as 0.25 in each iteration.
Hyperparameters. Following DiffusionDPO. We use AdamW for SD1.5 15 experiments, and Adafac- tor for SDXI, to save memory. An effective batch size of 128 (pairs) used. For image generation, set N 8 for quickly sampling. Morever, we apply DDPM with 50 steps for SD1.5 and DDIM with 20 steps in SDXL for quickly sampling in training and testing. All test images are generated classifier free guidance scale of 5 (SDXL) or 7.5 (SD1.5) during inference. For DPO training, present the main SD1.5 and SDXL results with 3 5000.
Evaluation. We evaluate the proposed SAIL three popular benchmarks: Pick-a-Pic, PartiPrompts and HPSv2. For Pick-a-Pic, We evaluate quantitative results based on the 500 validation prompts, i... validation unique. PartiPrompts contains 1,632 prompts encompassing various categories. Meanwhile,

---

Published as a conference paper at ICLR 2026
HPSv2 comprises 3,200 prompts. covering four styles of image deseriptions: animation, concept art, paintings and photo. For metrics, we use multiple evaluation metrics, indluding PickScore (general huamn preference) (Kirstain al. (023), Aesthetics (no-text- bastd visual appeal) (Meyer & Verrips 2008). HPSv2 (prompt alignment) (Wu et al. 2023) and ImageReward (general human preference) [uu et al. 2023b). For all metrics, higher values indicate better performance.
5.2 PRIMARY RESULTS: ALIGNING DIFFUSION MODELS
Qualitative Comparison Given the effectiveness and implementation efficiency of Direct Preference Optimization (DPO) (Ratailov et al. 2202),, we adopt DPO as the foundational framework for iterative aligmment. We compare SAIL with the
diffusion model (e.g.. SD1.5, SDXL), vanilla DPO, and its variants (DiffusionSPO (Li et al. 2024), MaPO (Hong et al., 2024)) for fair comparison. DiffusionDPO and MaPO are training on Pick-a-Pic v2 dataset. Specically, SPO is different from the above method, which considering step-wise perference optimization not image-wise wise. Morever, SPO apply Pick-a-Pic to train a step-wise reward model. We demonstrate the quantitative result in Table [ In HPSv2, experimental results demonstrate consistent performance improvement with increasing iterations, ultimately achieving a 0.71% (Anime), 0.63% (Concept Art), 0.56% (Painting), 0.59% (Photo) gain over the base model SDXL. Compared to DiffusionDPO with equivalent preference data, our method yields 0.33% (Anime), 0.30% (Concept Art), 0.29% (Painting), 0.33% (Photo) improvement SDXL.
Remarkably, using only 6% human preference data (0.05M vs 0.8M sam- ples), our approach surpasses the full- data DiffusionDPO baseline. Com- prehensive evaluation on Pick-a-Pic dataset (measuring human prefer- ence, aesthetic quality, and text-image alignment) shows significant improve- ments across all four key metrics (0.38% in PickScore, 0.2953% in Im- ageReward, 0.12% Aesthetics, 0.52% HPSV2) in SDXL. Meanwhile, as il- lustrated in Table I our method ro- bustly adapts to varying model scales (SD1.5 and SDXL), also achieving 0.38% in PickScore, 0.2459% in Im- ageReward, 0.11% Aesthetics, 0.54% HPSV2 in SD1.5, confirming its generalizability. In SD1.5, SAIL achieves similar performance with the DiffusionSPO. which uses a specific reward model and step-wise align human preference. Table | further verifies that fully exploiting the model's intrinsic potential can achieve impressive human alignment without external data expansion.
Table 2: The performance of each iterations of SAIL in Par- tiprompts, Stable Diffusion 1.5, We report PickScore, Aes- thetics, ImageReward and HPSv2 to evaluate the effective- ness of our method. The performance gain are highlighted in boldface.
Model
P.S.
Aes.
LR.
HPSv2
SD1.5
21.37
5.26
0.1177
26.82
SAIL (IterO)
21.48
5.28
0.1951
26.94
SAIL (Iter2)
21.54
5.31
0.2661
27.04
SAIL (Iter3)
21.62
5.34
0.3198
27.19
SAIL (Iter4)
21.61
5.36
0.3072
27.26
+0.24
+0.10
+0.1895
+0.44
Meanwhile, we also evaluate SAIL in Partiprompts. Partiprompts can be used to measure model capabilities across various categories and challenge aspects. Partiprompts can be simple and can also be complex, which brings challenge in model evaluation. We conduct SAIL in SD1.5 and present introduues the consistent performance gain in each score, 0.24% in PickScore, 0.1895% % in ImageReward, 0. 0.10% Aes, 0.44% HPSV2 improvement.
the result in Table 2] Table 2
Quantitative Comparison As shown in Figure 4 SAIL demonstrates significant qualitative improve- ments the base SDXL model. Quantitative results demonstrate generated data and self-rewarding can also achieve effective human preference aligmment. SAIL achieves consistent visual improve- ment with the iteration increase. Quantitative experiments demonstrate our method's significant improvements across structural coherence and aesthetic quality.
5.3 INITIAL ON LARGE SEED DATA
We conducted experiments to explore initializing the model with more data. Specifically, we used the entire Pick-a-Pic v2 training dataset for initialization and selected prompts from JournyDB (Sun cet al. 2023) as our subsequent prompt pool. Since our iter0 model is essentially DiffusionDPO. we directly continue training from this baseline. The experimental results presented in the Table 3
7

---

Published as a conference paper at ICLR 2026
SDXL
SAIL-IterO
SAIL-Iterl
SAIL-Iter2
SDXL
SAIL-Iter2
ice dragon, castle, medieval fantasy, winter, digital illustration, 8k
An aircraft corrier perfectly encased within glass bottl
griffon with seagull head and lion body
old sailor with beard sitting on..
Art illustration, strong bautiful woman sit down relaxing on green fancy couch, ww2 we can do it poster, black short hair,
Two cats watering roses greenhouse
Figure 4: The qualitative results demonstrate the effectiveness of our method.
Table 3: Comparison with other methods on SD1.5. We build our SAIL on DiffusionDPO and mark SAIL* For HPSv2, report the score in Anime, Concept-Art, Painting and Photo. For Pick-a-Pic v2, we apply PickScore, ImageReward, Aesthetics and HPSv2 metrics to evaluate all methods. The best results are highlighted in red.
HPSv2
Pick-a-Pic V2
Model
Method
Data
Ani.
Con.
Paint.
Photo
P.S.
L.R.
Aes.
HPSv2
27.23
26.65
26.52
27.41
20.62
-0.0130
5.38
26.21
DiffusionDPO
0.8M
27.64
26.97
26.90
27.56
21.07
0.2056
5.48
26.57
SD1.5
DiffusionSPO
0.8M
28.05
27.49
27.61
27.55
21.20
0.1577
5.68
26.75
SAIL" (Iterl)
0.8M
27.86
27.15
27.08
27.65
21.16
0.2761
5.56
26.68
SAIL" x (Iter2) 0.8M
28.08
27.37
27.31
27.72
21.27
0.4303
5.60
26.81
We apply SAIL" for two iterations and each iterations with 10K, 20K prompts. Table 3 demonstrates the result SAIL. compared with SD1.5, DiffusionDPO DiffusionSPO (Liang et al. 2024).
the Pick-a-Pic validation set, SAII.* outperforms DiffusionSPO across three metrics, most notably on ImageReward, where we achieve performance of 0.4303%. Furthermore, on the HPSv2 dataset, our method also surpasses DiffusionSPOO in two subclasses, Anime and Photo.
5.4 COMPARSION WITH ONLINE DPO
As illustrated in Figure in LLM, existing approaches employ external models for online DPO optimization, but their core limitation lies in heavy reliance on extensive annotated data to train robust reward models. Taking DDPO (Black et al. 20236) as an example, this method requires concurrently training four separate model to comprehensively evaluate human preferences. Our experiments follow the setting in DDPO's framework (using only Aesthetic as the single reward model). Table presents the comparsion be- tween Online DPO and SAIL. While OnlineDPO-AA -ees achieves remarkable improvement in aesthetic metrics, which surpass SAIL by 0.07%. But its gains in human preference and text-image alignment remain limited. This confirms that single reward model struggle with comprehensive enhancement, including human preference, aesthetic quality, and text-image alignment though the ideal solution may be introducing multi reward models, how to balance their weight presents new research challenges. In contrast, our method demonstrates dual advantages: Eliminates dependency on ex- ternal annotations through self-rewarding based closed-loop optimization; Balanced Performance:
wonpressbiiityllnnmpesssitii hwwpsiiiittyy/topt-mmmaag Alignment)
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparsion with Online DPO in Iterl. Base on the IterO model, we apply self- rewarding and Aesthetics rewarding to rank preference pairs given the generated data.
Table 5: Comparsion of different selection strat- egy in constructing preference data of Iterl. Base on the Iter model, we apply Best-worst and random to choose win-lose pairs.
Model
LRR
Aes.
HPSv2
Model
LRR
Aes.
HPSv2
SD1.5
-0,0130
5.38
26.21
SD1.5 IterO
-0.0130
5.38
26.21
IterO
0.0715
5.42
26.38
0.0715
5.42
26.38
Self Rewarding & Aesthetics Rewarding
Select Strategy
SAIL.
0.1137
5.46
26.49
best-worst random 0.1055
0.1137
5.46
26.49
OnlineDPO 0.0936
5.53
26.35
5.44
26.40
Achieves consistent improvements across all metrics. Morever, Pure aesthetic optimization may cause aesthetic overfitting (e.g., oversaturated colors), SAIL outputs better align with composite human preferences.
5.5 ABLATION STUDY
Pair Selection Strategy. We compare two pair-selection methods for constructing preference data from N candidates: (1) Best-wors, Selection: select the best and the worst sample to construct preference data; (2) Randomized Selection: randomly select two samples and construct preference data. We conduct experiments and present the result in Table Table shows that SAIL achieves the best performance when using Best-worst Selection, 0.1 1177% in ImageReward, 5.46% in Aesthetic and 26.49% in HPSv2.
The role of Mixup Ranked Preference Data. reveal the critical role of mixing gener- ated and human preference data in maintaining model stability. As shown in Table 6. SAIL without mixed data suffers from significant performance drop in the second iteration, at- tributed to two key factors: (I) Overfitting to Pairs: The model overfits to high-reward sample pairs during training phrase, drastically reducing generation diversity. For instance, images generated from different seeds become similar, while reward scores artificially inflate (e.g., exceeding 90%). (2) Catastrophic Forgetting: The absence of human preference data leads to progressive degradation of the model's discriminative ability. Compared to the first iteration, the reward model's accuracy declines measur- ably, which incurs the generated preference pairs is not accuracy. More discussion about the role of Mixup Ranked Preference Data is in Appendix.
Table 6: The influence of Mixup Ranked Prefer- ence Data.
Model
P.S.
IRR.
HPSv2
Base
20.62
-0.0130
26.21
SAIL (IterO)
20.80
0.0715
26.38
SAIL (Iterl)
20.89
0.1137
26.49
Iter2
20.95
0.1729
26.65
Iter2 w/o mix
20.86
0.1564
26.55
6 CONCLUSION
In this work, we propose SAIL, a novel self-rewarding framework for aligning diffusion models with human preferences without relying on large-scale annotated datasets external reward models. By leveraging iterative self- ieprovmprvee through closed-loop loop generation and preference learning, SAIL effectively expands limited human seed annotations into robust alignment signals. Our approach addresses key limitations of existing taathods-costie data dependency and bias propagation- introducing mixup-ranked preference data to mitigate catastrophic forgetting and stabilize training. Experiments demonstrate that SAIL outperforms state-of-the-art methods even with only 6% of human preference data, highlighting its efficiency and scalability.
Limitations. We primarily focus on leveraging a small amount of human preference data and model- generated data for human preference alignment. However, compared to the image domain, preference data in video generation is significantly harder to collect. We therefore foresee promising future for exploiting SAIL in video human preference alignment and investigating intermediate reward mechanisms that can provide step-wise guidance throughout the denoising process.

---

Published as a conference paper at ICLR 2026
Use of LLMs. We utilize LLMs to assist with experimental design and writing refinement.
ACKNOWLEDGEMENTS
This work was partially supported by the National Natural Science Foundation of China under Grant No. 62402434.
REFERENCES
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023a.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023b.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arXiv preprint arXiv:2406.097 2024.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 2023.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominík Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, (6:79858-798855 2023.
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502. /8770, 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprini 24Xi:22402.0479. 2024.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome,, and Jongheon Jeong. Margin-awart preference optimization for aligning diffusion models without reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
Dongyoung Kim, Kimin Lee. Jinwoo Shin, and Jachyung Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. arXiv preprint arXiv:2406.04411, 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.044m 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint a arXiv:2406.044. 04314 2(3), 2024.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409. 13156, 2024.
10

---

Published as a conference paper at ICLR 2026
Birgit Meyer and Jojada Verrips. Aestheties. In Key words in religion, media and culture, pp- 36 46. Routledge, 2008.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.0195: 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-5374) 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2:Xi:22204.061255, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Seyed mehdi S Kulkarni, S Sara Ghasemipour, Burcu Karagol Ayan, Soroush Shaker Mahdavi, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:20/0.02500. 2020.
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 36:49659-4999 2023.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-8238, 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306. 00000.99111... 20233
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 006.99 2025.
Jaazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Li, Yujia Du, C.-L. Li, Ziqi Shen, Jianfeng Wang, Lijuan Lin, Meng Wang, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.0m 0.59777 2023a.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903-15999 2023b.
Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. arXiv preprint arXiv:1712.01227 2017.
11

---

Published as a conference paper at ICLR 2026
A MORE DETAILS ABOUT SELF-REWARDING STRATEGY
According to the Equation 5 in the main paper, which is derived in the DiffusionDPO Wallace et al. (2024). We have the reward function for images æ as follows:
rly x) = 3 log (a to y. t, 9r (Xo))
+ 3 log (yy t, 9r (2o))
(11)
Pref (oo y L, (:())))
where is the timestep and q: (zo) = vn + II OvE is the combination of æ and e ~ A0 I). Or noise scheduler. Following DDIM Song et al. 2020), The pe pa t, (2(o) be dirve
(x (x(oo)) (N0o; pred 6,d)
-a ge (((2))
v1 OrEA ,or1)
Vos
OGE VI
arE8 8,1)
= =N 20
Voa.
T-me (c-en))) vize
(mo
1wot
287
(2nd))
(12)
T-me ||
(2nd7)id/2)
60-
20r
(2763)(d/2))
o
||e-c011
26
(2md7) (d/2)
In Equation 82
To evaluate the effective-
70
ness of SAIL,, we demonstrate the accuracy of several reward methods (PickScore Kirstain et al. (2023), Aesthet- ics Meyer & Verrips (2008), Radford et al. (2021) and Seff-rewarding). We choose 1,000 preference data from Pick-a-Pic v2 validation set and take the human preference as the ground truth. Figure 5 shows the results. SR', SR² SR3 represent different versions of self-rewarding method IterO. The main difference between the above methods the number of (t, q:), SR SR2, SR3 base on 10, 20, 30 draws, respectively. It demonstrates the only with 0.05M data, the self-rewarding of iterO surpasses the Clip and Aesthetics.
60.5
61.45
61
55.7
55
50
4...2
45
40
35
30
Acs. Clip. P.S. SR
SR²
SR3
Methods
Figure 5: Comparsion of five reward mod- cls.
B VISUALIZATION OF SAIL*
we present visualizations comparing outputs from SD1.5, DiffusionDPO, and our model after SAIL* iterations. These visualizations demonstrate that after SAIL* iter- ations in Figure 6. the generated images exhibit higher quality, improved text-image alignment, enhanced aesthetics, and better overall texture.
C THE NUMBER OF CANDIDATES N
12

---

Published as a conference paper at ICLR 2026
SD1.5
DiffusionDPO
SAIL*-Iters
Cute and adorable cartoon rabbit baby rhea facing the camera, fantasy, dreamlike, surrealism, super cute, trending on artstationm volumetric light, cinematic, post processing, 8K
A small purple duck
A Minecraft character named Herobrine standing on a grass block
A landscape featuring unique digital cointing-stylee building
Figure 6: Qualitative visual results for several methods.
Table 7: The influence of Candidates N.
We conducted experiments to deter- mine the optimal number of candi- date images per iteration. The results, shown in Table 7 illustrate the impact of varying candidate image counts on the outcome of the first iteration. It evident that increasing the num- ber of candidate images significantly improves the results. However, con- sidering both image generation effi- ciency and performance, we opted to N in study.
Model
P.S.
L.R.
HPSv2
Base
20.62
-0.0130
26.21
Candidates N
2
20.70
0,0536
26.35
20.78
0.0879
26.40
20.89
0.1137
26.49
16
20.96
0.1356
26.54
13

---

Published as a conference paper at ICLR 2026
Table 8: The performance of SAIL with different seed preference dataset on SD1.5,
Pick-a-Pic V2
Method
P.S.
I.R.
Aes.
HPSv2
20.62
-0.0130
5.38
26.21
SAIL (seed
21.00
0.2329
5.49
26.75
SAIL (seed2)
21.05
0.2381
5.50
26.79
SAIL (seed3)
21.02
0.2253
5.49
26.76
D THEORETICAL ANALYSIS
Theoretically. SAIL shares the same global objective as standard DPO: finding the policy x' that maximizes the reward (the underlying human preference) subject KI--divergence constraint. SAIL (leraaive/Semi-Supers -Sperviss))) can be viewed as a Self-Training process. The model explores the latent space to generate new samples and uses current implicit reward function to estimate rankings (pseudo-labels). This propagates the reward signal into previously unexplored regions.
Therefore, SAIL converges to distribution that consistent with human preferences over much broader support set than standard DPO. It approximates the optimal solution of DPO over the full data distribution, rather than just the limited set.
E SAIL ON DIFFERENT INITIAL DATASET
Table g indicates that SAIL is reasonably robust to seed selection. The Ranked Preference Mixup strategy plays crucial role here. By mixing generated preferences with the high-quality seed data, SAIL prevents the model from drifting too far if the self-generated signals are noisy the early stages.
F SAIL WITH MORE ITERATIONS
We perform SAIL with more iterations SD1.5, shown in Table 9 In iter 1-3, there rapid improvements. The model effectively learns to align with the seed preferences and generalizes to the unlabelled set. While in iter 4-5, we observed that the probability in Equation 4 exceeds 0.8, indicating that the model has approached a saturation state.
Table 9: The performance of SAIL with more iterations on SD1.5.
Pick-a- Pic V2 L.R.
Method
P.S.
Acs.
HPSv2
20.62
-0.0130
5.38
26.21
SAIL (Iter0)
20.80
0,0715
5.42
26.38
SAIL ([terl)
20.89
0.1137
5.46
26.49
SAIL (Iter2) 20.95
0.1729
5.47
26.65
SAIL (Iter3) 21.00
0.2329
5.49
26.75
SAIL (Iter4)
21.04
0.2803
5.51
26.77
SAIL (Iter5) 21.06
0.2971
5.51
26.78
G VISUALIZATION
We present visual comparisons showcasing the effectiveness of SAIL across different base models.
14

---

Published as a conference paper at ICLR 2026
SDXL
SAIL-Iter2
SDXL
SAIL-Iter2
Figure 7: Qualitative visual analysis results for several methods. The images are presented row by row (top to bottom), and within each row, from left to right. The respective prompts are: (1) "A rich, ruthless and courageous warrior king from Africa", (2) "A robotic bull, full body shot, standing, athletic, steel and hard rubber, glowing eyes, humanoid, looking camera, arms crossed". (3) "A girl with silver hair in post apocalyptic setting portrayed in cinematie illustration by Yoji Shinkawa and Krenz Cushart", (4) "beautiful goddess, detailed face, focus on eyes, masterpiece, realistic", (5) painting depicting wuxia character standing on
under moonlit night", (6) "Clone Trooper". (7) "A teddy bear inspired by Vincent van Gogh", (8) "The image is a stunning illustration of knight warrior wearing Nordic armor and Skyrim mask, with intricate details and dynamic lighting that make perfect for RPG portraits and cosplay".
15

---

Published as a conference paper at ICLR 2026
DifFusionDPO SDXL
SAIL Iter0
SAIL Iterl
SAIL Iter2
SDXL
Figure 8: Qualitative visual analysis results for different methods. The prompts used for each row (from bottom) follows: (1) "An old with bird on his head". (2) "A lemon character with sunglasses on the beach". (3) "Cat wearing cowboy hat rides on corgi during sunset in the Wild West", (4) "There an anthropomorphic male wizard in the image wearing 3D cinema glasses". (5) "An elderly man is sitting on couch", (6) "A man standing in front of bunch of doughnuts". (7) "A man and two dogs are riding scooter".
16

---

Published as a conference paper at ICLR 2026
DifOusionDPO SDXL
SAIL IterO
SAIL Iterl
SAIL Iter2
SDXL
Figure 9: Qualitative visual analysis results for different methods. The prompts used for each row (from top to bottom) are as follows: (1) "A portrait of a character in a scenic environment", (2) "'A A stylized portrait featuring sliced coconut, electronics, and Al in cartoonish cute setting with dramatic atmosphere".. (3) "A side profile portrait of Maya Ali mage with intricate details, neon and sweat drops in highly detailed digital painting". (4) "The image depicts beautiful goddess of spring wearing wreath and flowy green skirt, created by artist wlop", (5) kinetic sculpture of colorful bird with a long tail surrounded by swirling lines and shapes", (6) "A concept art digital CG painting of place in Bali, trending on ArSStation and created using Unreal Engine", (7) "The image portrays surreal scene of hybrid creature consisting of a great leviathan, cybernetic turtle, and cephalopod terrapin in magical universe surrounded by cozy hot springs, forest, and lush plants amidst luminous stellar sky". " (8) "A realistic digital art depicting dwarven automobile".
17


Reviews:
[{"review_id": "wr8YgWTbiq", "reviewer": "Reviewer_sMVs", "summary": "This paper proposes SAIL, a novel framework for aligning diffusion models with human preferences without reward models or the use of large-scale human-annotated data. The idea is to bootstrap alignment through self-amplified iterative learning using a minimal set of preference data. Experiments demonstrate strong performance while using only a small part of the preference data.", "questions": "see Weaknesses 1. 2 \n\n1. How sensitive is the proposed method to the choice of the initial seed dataset? What happens if the seed preferences are noisy or biased?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a self-improving framework to align diffusion models with human preferences without large-scale annotated datasets, which is novel. \n2. Thorough empirical evaluation results show that the proposed method is effective, outperforming existing alignment methods using only 6% of the annotations.", "weaknesses": "1. Lack of Theoretical Guarantees. While the reward formulation (Eq. 8–9) is mathematically correct in the DiffusionDPO framework, the paper does not provide theoretical analysis of what distribution SAIL converges to. What is the target distribution of this method? Is it the same as DPO? If so, what explains the performance with fewer annotations? It's unclear where the observed gains come from. Whether and why the self-reward metric aligns with true human preference distributions?  A thorough analysis would benefit the paper.\n\n2. Even though the method needs fewer human annotations, it introduces additional cost to generate samples in the training loop, which is not efficient. Could the author provide some discussion on the trade-off between annotation efficiency and training efficiency? \n\n3. The central reward estimation strategy, which computes preference scores using differences in squared denoising errors between current and reference models (Eq. 6–9), closely follows the DiffusionDPO formulation and thus is not novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2dle2pQE8Z", "reviewer": "Reviewer_SeQh", "summary": "This paper presents a method for aligning diffusion models with human preference using limited preference data. It utilizes the implicit reward function from DPO to create online preference data. To address overfitting, it mixes online preference data with the initial preference data. Experiments show that the proposed method can achieve comparable or better metrics than some previous work, using less data.", "questions": "- What are the drawbacks when performing more iterations of SAIL?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Using self-rewarding to rank online data is relatively new in text-to-image generation. \n- The proposed method works well with limited data.", "weaknesses": "- Implicit reward is adopted from previous work in LLM.\n- The mixup of online and initial preference data is straightforward.\n- Some generated images seem to have a color saturation problem.\n- There are a lot of problems in writing, e.g.\n  - Line 144-145: grammar error\n  - Line 145: some -> Some\n  - Line 344: Pic-a-Pic -> Pick-a-Pic\n  - Line 357: use -> uses\n  - Line 362: bringing challenge -> brings challenges\n  - Line 373: fix reference\n  - Line 374-375: Thus, …, so … \n  - Line 454: reveals -> reveal\n  - Line 457: suffers -> suffers from", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLgoex1vVM", "reviewer": "Reviewer_neyU", "summary": "This paper deals with the limitations of existing approaches to preference alignment for diffusion models: (1) DiffusionDPO requires large-scale human-annotated preference data; (2) Auxiliary reward modeling approaches introduce more biases, are vulnerable to reward hacking, and struggle with distributional shifts from training data. To address these issues, the authors propose their methodology with a novel argument for diffusion model.\n\nThe authors argue that diffusion models' potential hasn't been fully exploited through supervised fine-tuning on human-labeled datasets alone. Instead, they propose iterative self-improvement by cold-starting with only a small seed set of human preference data. Based on the under-exploitation assumption, they propose SAIL, an iterative self-improvement closed-loop learning process and also introduce a ranked preference mixup strategy to prevent distribution collapse.\n\nExperimental results show that SAIL achieves comparable performance to state-of-the-art methods while using only 6% of the human preference data, highlighting the sample efficiency of the proposed algorithm.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of fully exploiting the base model's potential is innovative. By eliciting this potential through the proposed iterative self-improvement method SAIL, the authors achieve comparable or better preference performance using only 6% of the human preference data.\n\n2. The consistent improvement across multiple iterations further demonstrates the effectiveness of the proposed iterative self-improvement paradigm.", "weaknesses": "1. The tables lack multiple trials and confidence intervals, which are necessary to demonstrate the statistical significance of performance improvements and validate the effectiveness of the algorithm design in ablation studies.\n\n2. It would be valuable for the authors to include results over a larger range of iterations to illustrate the performance trajectory and reveal how the improvement trend evolves as the number of iterations increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)