Self-Supervised Direct Preference Optimization for Text-to-Image Diffusion Models
Authors: Liang Peng
Boxi Wu
Haoran Cheng
Yibo Zhao
Xiaofei He
Venue: NeurIPS 2025 poster ()
Keywords: diffusion
generative model
Primary area: applications

Abstract: Direct preference optimization (DPO) is an effective method for aligning generative models with human preferences and has been successfully applied to fine‑tune text‑to‑image diffusion models.  Its practical adoption, however, is hindered by a labor‑intensive pipeline that first produces a large set of candidate images and then requires humans to rank them pairwise. We address this bottleneck with self‑supervised direct preference optimization, a new paradigm that removes the need for any pre‑generated images or manual ranking.  During training, we create preference pairs on the fly through self‑supervised image transformations, allowing the model to learn from fresh and diverse comparisons at every iteration.  This online strategy eliminates costly data collection and annotation while remaining plug‑and‑play for any text‑to‑image diffusion method. Surprisingly, the on‑the‑fly pairs produced by the proposed method not only match but exceed the effectiveness of conventional DPO, which we attribute to the greater diversity of preferences sampled during training.  Extensive experiments with Stable Diffusion 1.5 and Stable Diffusion XL confirm that our method delivers substantial gains.

Self-Supervised Direct Preference Optimization for Text-to-Image Diffusion Models
Liang Peng + Boxi Wu
Haoran Cheng** Yibo Zhao
Xiaofei Hel.2
FABU Inc. *Zhejiang University
(pengliang, wuboxi, chenghaoran)0zjju.oo edwijjueeducoom
Abstract
Direct preference optimization (DPO) is an effective method for aligning generative models with human preferences and has been successfully applied to fine-tune text-to-image diffusion models. Its practical adoption, however, is hindered by labor-intensive pipeline that first produces large set of candidate images and then requires humans to rank them pairwise. We address this bottleneck with self-supervised direct preference optimization, new paradigm that removes the need for any pre-generated images or manual ranking. During training, we create preference pairs the fly through superviee image transformations, allowing the model to learn from fresh and diverse comparisons at every iteration. This online strategy eliminates costly data collection and annotation while remaining plug-and-play for any text-to-image diffusion method. Surprisingly, the on-the-fly pairs produced by the proposed method not only match but exceed the effectiveness of conventional DPO, which we attribute to the greater diversity of preferences sampled during training. Extensive experiments with Stable Diffusion 1.5 and Stable Diffusion XL confirm that our method delivers substantial gains.
Introduction
Text-to-image diffusion models 11 5 have emerged dominant paradigm for high-quality image generation conditioned on natural language prompts. They have demonstrated the ability synthesize diverse and visually appealing images across a wide range of prompts and styles. However, while these models are typically pretrained on large-scale datasets, they can fail to align with human preferences, especially in applications requiring text-image alignment subjective aesthetic quality. To bridge this gap, recent works have explored preference-base post-training strategies that adjust the model's outputs based on human feedback.
One prominent method this direction is Direct Preference Optimization (DPO) 10) which trains generative models using pairwise human preference data. DPO has shown strong results in aligning text-to-image models with user intent. Nevertheless, its practical adoption is hindered by a costly and rigid training pipeline: it first requires the offline generation of large of image candidates, followed by extensive human annotation in the form of pairwise ranking. This process is not only cons-minn and expensive but also inflexible- adapt to new prompts or domains. Furthermore, the fixed nature of the dataset may limit the diversity of learning signals, potentially affecting generalization.
the ranking data collected, cannot easily
In this work, we present Self-Supervised Direct Preference Optimization (Self-DPO), a novel frame- work that eliminates the reliance pre-generated images and manual rankings. Instead of requiring
Equul contribution
i Work was done at FABU Inc.
I Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

R
IR
DPO
Self-DPO
We propose Self-DPO, incorporating direct preference optimization in a self-supervised manner. We provide the comparisons SDXL base model with the same seed. Our method requires less data, yet generate more visually appealing results. Best viewed in color with zoom in.
Figure 1:
external preference data, our method constructs training pairs dynamically using self-supervised image transformations. During each training iteration, the model first identifies "winning image" that satisfies human-aligned quality criteria. We then generate a corresponding "losing image" by in- tentionally degrading the winner, either through visual-quality reductions or text-image misalignment. The resulting lon-lose pair supplies an immediate preference signal, allowing the model to perform online direct preference optimization. By learning from this continuous stream of synthetically generated preference pairs, Self-DPO achieves effective human alignment.
This self-supervised, on-the-fly approach brings several key advantages. removes the need for costly pre-generated images and human ranking efforts, enables scalable and dynamic training, and introduces greater di- versity into the preference supervision signal. We summarize the data requirements in Ta- ble I Remarkably, we find that Self-DPO not only matches but surpasses conventional DPO in both qualitative and quantitative met- rics (e.g., boosting ImageReward win rate from 61 to 85). We provide some visualiza- tion results in Figure Experiments on Stable Diffusion 1.5 1 and XL (2 demonstrate consistent improvements in visual quality and text-image alignment. Our method is plug-and-play, generalizable across architectures, and easily integrable into
Data requirements
SFT Self-DPO
DPO
Text Caption Image
Extra Preference Image
<
Table I:
Post-training data requirements. Self-DPO shares the same data requirements as SFT (TT]. DPO [1O] requires extra preference images that are col- lected and annotated by humans, which highly time- consuming and expensive.
existing diffusion model training pipelines.
2 Related Work
2.1 Text-to-Image Diffusion Models
Text-to-image diffusion models have recently attracted significant attention in generative modeling due to their ability to produce high-quality and diverse images from textual descriptions. The introduction

---

of Denoising Diffusion Probabilistic Models (DDPMs) [2 I3 marked a major breakthrough in this field, establishing diffusion models as a powerful generative approach. Diffusion models [22 14 [5]
operate by reversing a gradual diffusion process, where noise is incrementally added to the clean latent and then learned to removed, ultimately synthesizing a coherent and realistic image/video. Building upon this foundation, diffusion models have become a representative paradigm in text- to-image generation, leading to models like GLIDE []6] for text-guided image editing, Imagen 117 with cascaded diffusion for high-resolution synthesis, and Stable Diffusion M using latent diffusion for efficient generation. To enhance image generation quality, researchers typically focus on two main directions. One approach involves architectural improvements, with the Diffusion Transformer (DiT) [3] gaining attention for its improved image fidelity, performance, and diversity. Notable advancements include PixArt-c [8]. Hunyuan DiT [19], and SD3 [4] Another approach leverages supervised fine-tuning to refine text-to-image diffusion models. These methods curate datasets by integrating various strategies, such as preference models (2), pre-trained image models (e.g., image captioning models), and expert-assisted data filtering
20 21
23
2.2
Preference-Basee Optimization Methods
In recent years, preference- based optimization has gained traction, refining models through user feedback or ranked preference pairs. In Large Language Models, Reinforcement Learning from Human Feedback (RLHF) leverages human comparisons to train reward model that guides policy learning [25 26]. Alternatively, direct preference optimization (DPO) fine-tunes models directly
on preference data, bypassing the need for an explicit reward model while achieving comparable performance [9]. Subsequently, preference-basee optimization has been applied to image generation. Some methods enhance image quality by increasing rewards for preferred outputs [27 28 22)].
while others use reinforcement learning [29 30). However, training reliable reward models remains challenging and computationally expensive, with over-optimization potentially leading to mode collapse, reducing diversity [3I (28). Similarly, Direct Preference Optimization has been introduced in text-to-image generation, with Diffusion-DPO [10) demonstrating the effectiveness of optimizing on human comparison data to enhance both visual appeal and text alignment. Additionally, direct score preference optimization 32 refines diffusion models through score matching, providing a novel approach to preference learning. Several recent studies have further explored adapting preference learning techniques from large language models to fine-tune diffusion models [33] 34 [35 highlighting the growing interest in aligning generative models with human preferences.
2.3
Self-Supervised Learning
Self- supe-vupe learning has emerged as a pivotal paradigm in machine learning. Among its most widely used approaches are contrastive self-supervised learning and generative self-supervised learn- ing. Contrastive self-supervised learning distinguishes representations by pulling similar instances closer while pushing dissimilar ones apart. MoCo (36) enhances training efficiency through a mo- mentum encoder, while SimCLR (37] simplifies the process with strong data augmentations. CLIP 38 extends contrastive learning to vision-language tasks, aligning images and text in a shared latent space, enabling zero-shot transfer. Generative approaches inherently follow unsupervised or self-supervised learning principles, training without labeled data to model the underlying distributions of the input. GANs [39 utilize adversarial training generate realistic data, while VAEs [40] encode data into a structured latent space for controlled synthesis. VQ- VAE [4 introduces discrete latent representations for high-quality generation. MAE 42 leverages masked image modeling to learn rich visual features. More recently, denoising diffusion models 13 have demonstrated impressive results by iteratively adding and removing noise, learning robust representations in self-supervised manner. Self-supervised learning enables training without manually labeled data, laying the foundation for future advancements in representation learning, generative modeling, and multimodal understanding.
3 Methods
3.1 Preliminary
Diffusion Models. Denoising Diffusion Probabilistic Models [ represent the image generation process Markovian process. Let 2o e Hd be the data point and g[=,|[,;-)) denote the forward
3

---

process, where noise is added to the data at each timestep t. The forward process is defined as:
g(z,|<z,-1 (a,; v 1,
where 5e is schedule that controls the variance of noise added at each timestep, and ((-) denotes the normal distribution. The forward process gradually adds noise, with æT being pure noise after T timesteps. The reverse process aims to learn the distribution (o ( which represents the process of denoising and generating data from pure noise. The reverse process can be parameterized as:
(o [2a) = 1i (2s, t). Eo (zte t));
(2)
where μo (z;; f) and Eg (2,,, t) are the mean and covariance learned by the model each timestep t. The model is trained by minimizing the following objective:
Ldiffusion
Er, |||e ep (a., 0)]
where is the noise added at each timestep, and eœ (2r, t) the model's predicted noise. The model trained to predict this noise accurately at each step, enabling it to reverse the diffusion process and generate high-quality data.
Reinforcement Learning from Human Feedback. For diffusion models, human preferences each diffusion step are modeled using a Bradley- Terry formulation [43], where the probability of preferring "winning" sample x. over "losing" sample x for given prompt is defined as:
pBT( > x//c) (r(c, a²") r(c, x))).
with a, r, representing the sigmoid function, reward model, and prompt, respectively. Subsequently, by conceptualizing the diffusion denoising process as multi-step Markov Decision Process, the gen- erative model is fine-tuned via reinforcement learning. The training objective [10 44 is formulated as:
Lih Re Eys (warr c)
t= r(c, ... - A Dxi. (po (noorrll (c) || Pee((wo.re)),
where Prer Prer(wor7le) (c) is the distribution from the pretrained diffusion model and X controls the influence of the KL divergence regularization term.
Direct Preference Optimization (DPO). DPO streamlines RLHF by using the learning policy's 's log likelihood to imp licitly encode the reward. In text-to-image diffusion models, this leads wise reward defined as:
r(c, x1) = A Alog Po Po 1141,
(6)
Prr x)-1,c)
32 follow this line and adapt it to diffusion models. They optimize the model pe based on the Bradley-Tery model [43]. leading to an objective function:
Recent works [10
- E log log Po(x x.t1,c) c) (re
po(2) x c)
)
Diffussion DPO
A Alog
1/-1: c)
(ae 1
where the winning and losing samples (x)", x/) and the prompt e are drawn from the dataset, and timestep t is uniformly sampled from the diffusion process. This formulation effectively aligns the learning policy with human preference signals embedded in the reference model.
3.2 Self-DPO for Text-to-image Diffusion Models
Inspired by DPO's effective alignment with human preferences at the image level 11 our work aims to extend this formulation into the standard post-training process for diffusion models (e.g., SFT). Unfortunately, direct application of DPO is not feasible because it requires collecting image pairs (typically generated by different models with the same prompt or by using different seeds with the same model-alow along with their associated manual rankings). This approach does not align with the conventional fine-tuning pipeline and incurs significant additional costs. To overcome this limitation, propose Self-DPO, which generates preference image pairs in self-supervised manner.
In each training iteration, we denote the text-image pair as (c, x where the image component is regarded as the wining image " Traditional DPO methods [10 9 require generating image pairs corresponding to the same prompt, followed by manual selection of the preferred (wining) and less preferred (losing) images, denoted x" and x respectivelly Our method removes such cumbersome

---

High quality dataset
Pre-generated peired large dataset
High quality dataset
Annotate
Sample
Sample
Random resample
Preference-ranked high quality dataset
annotator
Text- Text-lmage pair
Sample
Text-image pair
Losing Image
Wining image
Text prompt
Losing image
MSE
DPO Loss
Diffusion Model
Diffuaton Model
Diffusian
Model
SFT
DPO
Different post-training processes. We generate the "losing" images self- self-supervisedlly enabling direct preference optimization without extra collecting and ranking steps. This lightweight procedure eliminates the substantial overhead of conventional DPO while retaining the same data requirements as standard SFT. Best viewed in color with zoom
Figure 2:
steps. Because every image in the curated high-quality dataset already satisfies human preference, each can be regarded as a winner x" We obtain a corresponding self-supervised losing sample by deliberately degrading the winner: x Downgrade(("") Inspired by [10], the self-supervised direct preference loss is:
LSArDDO = log a (C( (1)" 1) e " |< al 1) - =
[ret < ( 6t1 12)))
where x² Downgrade(z")) (8)
where C' and el refer a scale factor and the noise corresponding losing images, and ene is the reference model. In our experiments, the Downgrade operation can be simply performed by randomly selecting images from the training dataset. For each self-generated image pair, the winning sample closely aligns with the prompt, whereas the losing sample fails to correspond to the description. Surprisingly, this simple manner brings significant improvements to the model. also compare different degradation strategies in the experiments. The training process overview is shown Figure The final loss as follows:
L X1 LMSE A2
We empirically set a and A2 to 0.5, 0.5, respectively.
Experiments
4.1 Setup
Implement Details: Following Diffusion-DPOO [10). for the SD1.5 ( experiments, AdamW 47 uti-
lized, while SDXL (2] training is conducted with Adafactor 48 to conserve memory. Following the official implementation in Diffusion-DPe [10), C' in Equation
8 is set to 2500. For SD 1.5, batch size of 2048 pairs (resolution: 512 512) is maintained by training across NVIDIA A100 GPU. Each GPU handles 8 pairs locally with gradient accumulation
Figure 5: Implicit accuracy during the train- ing stage.
5

---

Andraw Huang riding pink fluffy unioorn
hdr 8k resolution highly detailed image fairy by artist "Jammine Beoket Griffith Brian Froud
pixelart a ayberpunk ear
Tall fat man white baokgroundt-
A Great dane dog the style of Unoent Van Gogh
SD1.5
DPO
SFT
Self-DPO
Figure 3: Qualitative comparisons with the SD1.5 base model. All results are generated with the same random seed. Comparing with SFT and DPO [10, the model trained by Self-DPO demonstrates superior text prompt alignment. It also shows more appealing visual quality, especially in layout, colors, and details. Best viewed color.
over 64 steps. For SDXL, considering the resource limitation, we use the total batch size of 96 pairs (resolution: 1024 1024). Training is performed fixed square resolutions. We use learning rate of le-6 coupled with a 25% linear warmup.
Training Dataset: Our training data is sourced from the Pick-a-Pic V2 dataset 455 which keeps the same to Diffusion-DPO [10] It contains pairwise preference annotations for images generated by Dreamlike (a fine-tuned variant of SD1.5), SD2.1, and SDXL. These prompts and preferences were collected from users of the Pick-a-Pic web application. Please note that Self-DPO only uses the preference image and associated text in the dataset, instead of using the whole manually annotated image pairs.
Evaluation: We conduct evaluation on three datasets: Pick-a-Pic V2 45 validation set (contains
500 prompts), PartiPrompts (46) (contains 1632 prompts, including diverse categories and challenge aspects), and HPDv2 24 (contains 3200 prompts, including anime, concept art, paintings and photo). We compare Self-DPO with three different type baselines, i.e., base models (Stable Diffusion 1.5 (SD1.5) and Stable Diffusion XL (SDXL)), SFT models, DPO models, where DPO models are the

---

Prints, Ink, Paper, prints, Woodblook Drinting blooks, Wood blooks, Asia, Sapan, on 1853, Ja Japan, Polyehrome woodblook print, ink and oolor on paper, Metropolitan Museum of
Harry potter eat, pixar style, octane render, HD, high- detail
Cadilla ElDorado de style Bedaass Steampunk dans vieills rue paváa, trending on artatation, sharp focus, studic photo, intrinate de delallo, Nighly de taled, by greg rutkowski
marine iguana surfhoard
eharacter sheat, litle ririig an electris soccter bikee, in beautiful anime Myseskt snowy Tokyo olly with maeotve Alyarakdolouds floating in the blue sky, enchanting anowssapes of the city with bright sunlight, Miyazak''s is slandhoape imagery, Japanese art, 16-9
by Hayao
SDXL
DPO
SFT
Self-DPO
Qualitative comparisons with the SDXL. base model. All results are generated using the same random seed. Please note that the training dataset (Pick-a-Pic V2 [45) used for fine-tuning is obtained from a SD1.5 variant, SD2.1, and SDXL models. Consequently, directly fine-tuning (SFT)
Figure 4:
result in degraded performance (e.g.. as shown the second row). In contrast, DPO [10] optimizes the model by leveraging preference relationships with pre-ranked pairs, thereby avoiding this issue. Self-DPO uses the same data requirements as SIT but yields significant improvements both text prompt alignment and visual quality, demonstraing its effectiveness. Best viewed in color.
on this dataset does not lead to improvements- and may
For evaluation metrics, we employ the popular PickScore 45 Aesthetics 49 CLIP 38 HPS V2 24 and ImageAward (50) scores. For convenience of comparison, we scale the scores to fit a similar range (x 100 for PickScore, CLJP, HPS, and ImageAward, x10 for Aesthetics). PickScore a caption-sensitive scoring model, originally trained on Pick-a-Pic (y1), that estimates the perceived image quality by humans. Aesthetics assesses the visual appeal of an image, considering factors such lighting, color harmony, composition, and overall artistic quality. CLIP measures the semantic alignment between an image and a corresponding text prompt. By computing the cosine similarity between the image and text embeddings, this score evaluates how well the image content matches the provided textual description. Human Preference Score (HPS V2) is a metric designed to align with human judgments of image quality, particularly in the context of text-to-image synthesis. ImageAward quantifies the quality, aesthetic appeal, or alignment of a generated image with respect to desired attributes. is typically derived from reward model trained on human preference data.
publicly released from Diffusion-DPO 10
7

---

SD1.5
SDXL
Datasets
Methods
P.S.
Acs.
CLIP
HPS
I.R.
P.S.
Acs.
CLIP
HPS
L.R.
Base
20.57
53.15
32.58
26.17
-14.81
22.10
60.01
35,86
26.83
50.62
SFT DPO Self-DPO
Avg
21.10
56.35
33.75
27.03
45.03
21.48
57.84
35.67
26.67
30.89
Pick-a-Pic Pic V2
score
20.91
54.07
33.19
26.46
4.13
22.57
59.93
37.30
27.30
81.14
21.23
56.35
34.79
27.33
71.00
22.34
59.97
37.53
27.89
103.96
SFT
Win
75.00
77.20
60.40
90,20
80.00
19,40
31.80
47.00
44,60
42.4
DPO
73.80
60.00
60.00
71.80
61.00
72.60
47.20
63,00
79.80
69.8
Self-DPO
rale
78.60
77.80
68,40
94.20
85,20
60.80
50.80
62.40
93.80
79.2
Basc
21.39
53.13
33.21
26.79
1.48
22.63
57.69
35.77
27.33
69.78
SFT
Avg
21.75
55.31
33,93
27.57
50.75
22.02
56.41
35.31
27.13
47.29
DPO
score
21.61
53,58
33.88
26.98
21.43
22,90
$7.85
36.95
27.73
103.36
PartiPrompts
Self-DPO
21.84
55.09
35.11
27.84
75.66
22.79
58.69
37.00
28.30
117.50
SFT
Win
67.28
70.89
53.43
85.42
73.35
21.38
38.11
45.10
43.75
40.93
DPO
67.10
57.17
56.74
61.83
63.05
63.42
$3.62
62.32
73.10
68.44
Selr-DPO
rate
69.85
68.50
63.24
89,40
81.00
56.19
60.48
60.17
92.16
76.84
Buse
20.84
$4.32
33,96
26.84
-11.79
22.78
61.34
37.68
27.68
78.27
SFT
Avg
21.57
57.41
35.26
27.89
57.74
22.24
60.08
37.39
27.76
66.62
DPO
scure
21.30
55.80
34.68
27.22
13.24
23.18
61.35
38.45
28.14
102.74
HPD V2
Selr-DPO
21.58
57.10
36.30
28.11
76.13
22.98
61.30
38.35
28.77
110.67
SFT
79.53
75.31
59.34
90,10
81.16
23.47
37,28
46,63
58.22
45.81
Win
DPO
75.72
66.28
57.56
72.43
64.69
72.66
50.28
58.69
80.56
69.78
Self-DPO
rute
79.53
74.03
68.47
92.49
85.19
58.78
48.06
55.65
94.97
72.50
Table 2: Quantitative comparisons. We compare different fine-tuning methods (SFT. DPO [00] and Sell-DPO) on SD 1.5 and SDXL base models over three datasets (Pick-a-Pic V2 [5]] PartiPrompts 46). and HPDv2 24 ) "P.S." refers PickScore, "Aes." " is Aestheties, and "I.R." denotes Im- ageAward. For the SD1.5 base model, our method achieves the best performance across most metrics. In contrast, for the SDXL base model, we observe that SFT clearly degrades performance. This is likely due to the fact that the training dataset (Pick-a-pic V2 (55) used for fine-tuning is derived from SD1.5 variant, SD2.1, and SDXL. models. Interestingly, our method still achieves competitive results compared to DPO, which utilizes the full dataset and optimizes the model by leveraging preference relationships with pre-ranked pairs. These results underscore the effectiveness and robustness of our approach.
4.2 Quantitative Results
We provide quantitative comparisons in Table 2 We compare our method with SFT and DPO 10]. For the SD1.5 base model, our method significantly outperforms the alternatives. For example, Self-DPO achieves CLIP score of 34.79 on the Pick-a-Pic V2 dataset-an improve- ment of +2.21 over the base model-whereas SFT and DPO yield improvements of +1.17 and +0.61, respectively. In terms of overall human preference metrics, Self-DPO delivers substantially higher gains, improving the base model from -14881 to 71.00 on the ImageRe- ward metrie, which far exceeds the improvements observed with SFT (45.03) and DPO (4.13). Notably, the win rate of Self-DPO reaches 94.20 on the HPS met- ric. Results on other datasets (PartiPrompts and HPD v2) fur- ther confirm these improvements. In contrast, results on the SDXL base model show a slightly dif- ferent scenario, particularly with SFT. We observe that SFT sub- stantially degrades the perfor- mance of the base model. This degradation is likely due to the fact that, while the training dataset (comprising generations from an SD1.5 variant, SD2.1, and SDXL) is of much higher quality than that used for SD1.5, it does not exhibit clear superiority for SDXL. Nevertheless, the model trained by Self-DPO still demonstrates significantly better perfor- mance compared both SFT and the base model. On the PickScore, Aesthetics, and CLIP metries, Sclf-DPO achieves results comparable to DPO, and it attains superior scores on the HPS and ImageRe-
Methods
P.S.
Aes.
CLIP
HPS
L.R.
Base model DPO
20.57
53.15
32.58
26.17
-14.81
20.91
54.07
33.19
26.46
4.13
w/ Blur w/ Random grid
20.80
55.16
33.22
26.57
4.26
20.85
55.86
32.71
26.64
24.21
w/ Random image
21.23
56.35
34.79
27.33
71.00
Table 3:
Ablation on different downgrade manners. The "Ran- dom image" row achieves most significant improvements, imply- ing that degrading the image quality can be detrimental.
8

---

ward metrics. These findings validate our hypothesis that image-level learning benefits text-to-image diffusion models and highlight the superiority of Self-DPO. We also employ UnifiedReward [5] as the VIM evaluation metric, a ofte-oo-thhearr reward model designed for multimodal understanding and generation tasks. Built upon strong VI.M models [22 53
54 [5], it supports pointwise scoring to align model outputs with human preferences. The results are shown in Table 4.4 We further validate our method's s effectiveness by using a new real-world T2I evaluation setting. Specifically, we conduct experiments a subset of the LAION-COCO dataset containing 400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts. The results are shown in Table 4.4 While naive SFT degrades performance, our Self-DPO method achieves consistent improvements. Please note that we cannot provide DPO baseline it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.
4.3 Qualitative Results
We provide qualitative comparisons in Figure and Figure for for SD1.5 and SDXL, respectively. All quantitative and qualitative results are generated using the same random seed. We observe that Self-DPO consistently improves over other models, particularly in terms of producing more vivid colors and better adherence to text prompts. For example, in the third row of Figure 3 only Self-DPO successfully reveals the concept "cyberpunk". capture the intended meaning of the prompt, but Self-DPO exhibits more appealing visual quality. These quantitative and qualitative results confirm the effectiveness of our approach. We show the implicit win rate to measure the optimization process in Figure Specifically, the implicit win rate during training refers to the probability that the model prefers the winning image over the losing image. In other words, higher implicit win rate indicates stronger preference for the winning image. As training progresses, it steadily increases and eventually reaches around 0.85. This demonstrates that the model gradually learns to distinguish between the wsinning and losing images, and prefers to generate the winning image.
the fourth row of Figure both DPO and Self-DPO
4.4 Ablation
We conduct ablation studies on various downgrade approaches, shown in Table [ In partic- ular, we examine two additional methods-namely, blur (which involves downsampling and up- sampling the image by factor of 4) and random grid (which di- vides the image into an 8 8 grid and randomly swaps two grids). We observe that these two down- grade methods lead to worse performance, indicating that significantly degraded image quality can be harmful. Additionally, we remove the MSE loss and report the results in Table Interestingly, the performance does not drop noticeably and consistently performs better than DPO, further confirming the effectiveness of Self-DPO.
Methods
P.S.
Acs.
CLIP
HPS
L.R.
Base model
20.57
53.15
32.58
26.17
-14.81
DPO
20.91
54.07
33.19
26.46
4.13
DPO w/o MSE Self-DPO
21.26
56.22
35.00
27.31
67.78
21.23
56.35
34.79
27.33
71.00
Table 4: Ablation on MSE loss. When removing MSE loss,
Self-DPO still shows comparable performance and demonstrates the superiority to DPO.
Methods
P.S.
Acs.
CL.IP
HPS
I.R.
Base model
Base
SFT
DPO
Self-DPO
Base
20.57
53.15
32.58
26.17
-14.81
SFT
20.49
52.63
31.83
26.13
-23.95
SDI.5
2.44
2.62
2.53
2.72
SDXI.
2.96
2.73
3.10
3.07
Sell-DPO
20.68
52.87
34.31
26.52
17.11
Table 6: Comparisons on UnifiedReward.
Table 5: Results on a real-world image-text dataset.
5 Limitation and Future Work
In the present implementation, we construct losing samples by randomly sampling images from the training set. This manner can become unreliable when the datasct is small, as the resulting

---

losers may fail to provide sufficiently informative preference signals. Future work therefore can explore more sophisticated self-supervised strategies for synthesizing losing images, such more content-awarr perturbations or adversarial degradations that better challenge the model. Furthermore, our approach built on direct preference optimization, which is an offline reinforcement-learm paradigm. Extending the self-supprrvsed pairing concept to online policy-gradient or actor-critic frameworks represents another promising direction.
Conclusion
In this paper, we propose Self-Supervised Direct Preference Optimization (Self-DPO), fully self- supervised framework that aligns generative models with human preferences without requiring pre-generated image pools manual rankings. By synthesizing win-lose pairs the fly, Self- not only removes costly data-collection steps but also exposes the model to a broader and more diverse set of preference signals. At every training iteration, it constructs its preference pairs through controlled degradations of high-quality images and immediately updates the model via preference learning. Extensive experiments across multiple datasets and base architectures demonstrate that Self-DPO consistently delivers superior performance, validating its effectiveness and versatility.
Acknowledgments
This research was supported by The National Nature Science Foundation of China (Grant Nos: 62402417, 62273302, 22036600,, 61936006), in part by the Key R&D Program of Ningbo (Grant Nos: 2024Z115, 20252035), in part by Yongjiang Talent Introduction Programme (Grant No: 2023A-197- G).
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[2] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint or Xiv:2307.01952 2023.
William Peehles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pages 4195-4205, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Rong-Cheng Tu, Zi-Ao Ma, Tian Lan, Yuehao Zhao, Heyan Huang, and Xian-Ling Mao. Automatic evalu- ation for -o--g--maage generation: Task-decomposed framework, distilled training, and meta-cvaluation benchmark. arXiv preprint arXiv:2411.15444 2024.
[6] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.106 10629, 2024.
[7] Black Forest Labs. Flux, 2024. Black Forest Labs. Flux, 2024.
Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, and Dacheng Tao. Diffusion model-based video editing: survey, arXiv preprint arXiv:2407.00 00111, 2024.
Rafael Rafailov, Archit Sharma, Firi Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. Advances in Neural Information Processing Systems,
2023.
[10] Bram Wallace, Meihua Dang, Rafael Rafailov. Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228 8238, 2024.
10

---

[11] Xiaoliang Dai, Ji Hou, Chih-Yao Ma. Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang. Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint 2rXiv:2309.15807 2023.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 0794,, 2021.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[14] Rong-Cheng Tu, Wenhao Sun, Zhao Jin, Jingyi Liao, Jiaxing Huang, and Dacheng Tao. Spagent: Adap- tive task decomposition and model selection for general video generation and editing. arXiv preprint arXiv:241 I. 18983, 2024.
[15] Wenhao Sun, Long-Cheng Tu, Yifu Ding, Zhao Jin, Jingyi Liao, Shunyu Liu, and Dacheng Tao. Vorta: Efficient vidco diffusion via routing sparse attention. arXiv preprint arXiv:2505. 18809, 2025.
[16] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 1074) 2021.
[17] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[18] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis, arXiv preprint arX0v22310.00426, 2023.
[19] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-rcsolution diffusion transformer with fine mneigraieed chinese understanding. arXiv preprint arXiv:2405.0877 088448, 2024.
[20] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle-3 pdf. 2(3):8, 2023.
[21] Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongeheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal- guided image editing with text-to-image diffusion models. arXiv preprint 2rXiv:2406. 14555, 2024.
[22] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arX1v:2304.0677 2023.
[23] Wenhao Sun, transformers accelcration with asymmetric reduction and restoration. In Forry-second International Conference on Machine Learning.
Tu, Jingyi Liao, Zhao Jin, and Dacheng Tao. Asymmr: Vidco diffusion
[24] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng 7hu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid henchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint Xiv: 09341, 2023.
[25] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
[26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[27] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. /7400, 2023.
[28] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. 2023.

---

[29] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 06:6663336666666668 2023.
[30] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 2023.
[31] Kimin Lec, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadch, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
[32] Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representations. 2025.
[33] Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback fine-tune diffusion models without any reward model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941-8951, 2024.
[34] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. Advances in Neural Information Processing Systems, 37:24897 27:24597-224955. 2024.
[35] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. Advances in Neural Information Processing Systems, 37:73366-73398, 2025.
[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 9729-9738, 2020.
[37] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-16007. PmL., 2020.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[39] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neurul information processing systems, 27, 2014.
[40] Diederik Kingma, Max Welling, al. Auto-encoding variational bayes. 2013.
[41] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[42] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[43] Ralph Allan Bradley and Milton E Temr. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345. 1952.
[44] Ying Fan, Olivia Watkins, Yuging Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning fine-tuning text-to-image diffusion models. In Shirth-eeventh Conference on Neural Information Processing Systems (NeurIPS) 2023. Neural Information Processing Systems Foundation, 2023.
[45] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
[46] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 2(3):5, 2022.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
a7Xiv:171..05101,20 2017.
12

---

[48] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference Muchine Learning, pages 4596-4604, PMLR, 2018.
[49] Christoph Schuhmann. Laion-aesthetics. http://////aa..aaaaa///aaaaaaaaa cessed: 2023-11-10.
2022. Ac-
[50] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances Neural Information Processing Systems, 36:15903 15935, 2023.
[51] Yibin Wang, Yuhang Zang. Hao Li, Cheng Jin, and Jiagi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05236. 2025.
[52] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- languaee model understanding. localization, text reading. and beyond. arXiv preprint arXiv:2308.12444 2023.
[53] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing dision-languae model's pereeption of the 2024.
world any resolution. arXiv preprint
[54] Rong- Cong- Tu, Yatai Ji, Jie Jiang, Weijie Kong. Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang. and Wei Liu. Global and local semantic completion learning for vision-language pre-training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[55] Qwen Team. Qwen2.5-v.. January 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with
NeurlPS Code of Ethics htttpp///paarppp..cco/ooo///ccla
Answer: [Yes]
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
14

---

Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes,
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
15


Reviews:
[{"review_id": "C01yZwuKOA", "reviewer": "Reviewer_viTx", "summary": "The authors propose Self-DPO, a direct alignment algorithm that curates image preference data through image augmentation. Self-DPO addresses the reliance on paired preference data required by Diffusion DPO, enabling direct alignment with high-quality images and their captions. The authors validate the effectiveness of Self-DPO on Stable Diffusion v1.5 and Stable Diffusion XL, demonstrating superior performance compared to Diffusion DPO and SFT across several automatic evaluation metrics.", "questions": "* It seems that the MSE term is doing SFT and did not impact the performance significantly, it would be interesting to see how tuning the weights of MSE and Self-DPO affects the performance.\n* On SDXL, given that the MSE term is essentially doing SFT and could degrade performance, has the authors attempted to remove the MSE term and keep only the Self-DPO loss?\n* The scale of reported Image Reward seems different from the raw model output, did the authors scale it by 100?\n* Could the authors potentially include VLM evaluation on general human preference to complement the reward models?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "* Strengths:\n  * Simple solution to an existing data dependence problem to Diffusion-DPO, easy to implement for practitioners.\n  *  Comprehensive evaluation on automatic models with superior or at least comparable results to DPO without paired data.\n  * Writing is clear and easy to follow.\n* Weakness:\n  * The proposed algorithm seems simply borrows existing negative sample crafting techniques from SSL, the technical contribution is limited.\n * The experiments are limited to SD1.5 and SDXL, limiting the robustness of Self-DPO.\n * The authors only evaluated CLIP-based reward model suites, additional human/ large VLM evaluation could further solidate the results.", "was_revised": true, "final_justification": "As suggested in my comments", "rebuttal": "**W1: The proposed algorithm seems simply borrows existing negative sample crafting techniques from SSL, the technical contribution is limited.**\n\nWe appreciate your feedback and acknowledge that our implementation adopts a straightforward approach. However, we emphasize two key distinctions from SSL. First, our work introduces a novel perspective to direct preference learning by exploring whether preference data pairs can be generated directly from unpaired data. We affirmatively answer this question through a simple yet effective framework, demonstrating that this approach not only achieves good performance but also significantly reduces the annotation cost inherent in traditional DPO methods. Second, task-relevant image degradation plays a critical role, as demonstrated by our ablation study in Table 3 of the main text. Different degradation strategies yield divergent performance outcomes, and we propose a simple yet effective degradation method. Crucially, we show that excessive image quality degradation negatively impacts results, underscoring the need for balanced degradation design.\n\n**W2: The experiments are limited to SD1.5 and SDXL, limiting the robustness of Self-DPO.**\n\nPrior works commonly adopt two widely-used baselines (SD1.5 and SDXL), and we follow this fashion. Our method is model-agnosic and can be employed in most diffusion models.  As an initial attempt, we extend our evaluation to the state-of-the-art FLUX model using the Pick-a-Pic V2 synthetic dataset. Notably, this dataset comprises image generations from SD2.1 and SDXL, which have lower quality compared to FLUX outputs. Despite this, our experiments demonstrate that the method remains effective when applied to modern architectures.\n\n|                 | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n| Flux    | 22.45     | 62.25 | 32.95 | 28.81 | 102.00\n| Flux + DPO  | 22.46      |  **62.38** | 32.88 | 28.86 | 101.41\n| Flux + Self-DPO  | **22.52**     |  61.98 | **33.56** | **28.99** | **108.31**\n\n**W3 & Q4: The authors only evaluated CLIP-based reward model suites, additional human/ large VLM evaluation could further solidate the results; Could the authors potentially include VLM evaluation on general human preference to complement the reward models?**\n\nWe employ UnifiedReward [1] as the VLM evaluation metric, a state-of-the-art reward model designed for multimodal understanding and generation tasks. Built upon Qwen2.5-VL-Instruct, it supports pointwise scoring to align model outputs with human preferences. The experimental results are summarized below:\n\nSD1.5 resluts:\n|                 | SD1.5 | SFT | DPO | Self-DPO  \n|-----------------------|-----------|-----------|-----------|-----------|\n| UnifiedReward  | 2.44    | 2.62 | 2.53 | **2.72**\n\nSDXL resluts:\n|                 | SDXL | SFT | DPO | Self-DPO  \n|-----------------------|-----------|-----------|-----------|-----------|\n| UnifiedReward  | 2.96   | 2.73 | **3.10** | 3.07\n\nFLUX resluts:\n|                 | FLUX  | DPO | Self-DPO  \n|-----------------------|----------|-----------|-----------|\n| UnifiedReward  | 3.20 | 3.23  | **3.28**\n\n\nThe results demonstrate that Self-DPO shows better results over baseline methods under most comparisons. It highlights the effectiveness of our proposed method in aligning model outputs with human preferences.\n\n[1] Unified Reward Model for Multimodal Understanding and Generation. arXiv 2025.\n\n**Q1: It seems that the MSE term is doing SFT and did not impact the performance significantly, it would be interesting to see how tuning the weights of MSE and Self-DPO affects the performance.**\n\nWe present an initial ablation study on the impact of removing the MSE loss in the main text. Here, we provide an expanded analysis comparing different weight ratios between the Self-DPO and MSE components:\n\n|      Self-DPO : MSE         | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  0.00 : 1.00   | 21.10 | 56.35 |33.75 |27.03 |45.03\n|  0.25 : 0.75   |21.19| 56.23 |34.71 |27.29 |66.20\n| 0.50 : 0.50  | 21.23 |56.35 |34.79 |27.33 |71.00 \n| 0.75 : 0.25  | 21.24 | 56.24 |34.75 |27.34| 68.27\n| 1.00: 0.00 |21.26| 56.22 |35.00 |27.31 |67.78\n\nWe have two obervations.\nFirst, when Self-DPO is entirely removed (0.0:1.0), the method reduces to standard SFT, yielding the lowest performance across most metrics.\nSecond, introducing Self-DPO (ratios > 0.0:1.0) consistently improves results, with performance remaining stable across varying weight distributions. This suggests the Self-DPO loss plays the main role during the training process.\nThese findings confirm that our method is insensitive to loss weight tuning, as the Self-DPO component primarily delievers performance gains.\n\n**Q2: On SDXL, given that the MSE term is essentially doing SFT and could degrade performance, has the authors attempted to remove the MSE term and keep only the Self-DPO loss?**\n\nThanks for your advice! We conducted this experiment, and the results are presented below:\n\n|                 | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n| SDXL    | 22.10 | 60.01 | 35.86 | 26.83 | 50.62\n| SDXL + DPO  | 22.57 | 59.93 | 37.30 | 27.30 | 81.14\n| SDXL + Self-DPO  | 22.34 | 59.97 | 37.53 | 27.89 | 103.96\n| SDXL + Self-DPO (No MSE) | 22.42     |  59.78 | 37.52 | 28.07 | 106.37\n\nAs anticipated, eliminating the MSE loss for the SDXL baseline yields marginal improvements. This aligns with our hypothesis that the Self-DPO loss plays the main role, and its standalone application achieves the best overall performance. Thank you for your valuable feedback!\n\n**Q3: The scale of reported Image Reward seems different from the raw model output, did the authors scale it by 100?**\n\nYes. As shown in Line 203-204 in the main text, we scale ImageReward by a factor of 100. It is beacause we aim to adjust the value range approximately 0-100."}, {"review_id": "BqVhlpiiFN", "reviewer": "Reviewer_vbGM", "summary": "The paper proposes a surprisingly simple method to improve upon Diffusion-DPO. Traditionally, DPO requires a preference-pairs dataset, but the proposed method only requires a high-quality dataset. In this sense, the technique can achieve performance gains for free beyond standard SFT.", "questions": "1. The paper lacks a fair comparison with Diffusion-KTO [1], which is an important baseline as it also does not require paired data.\n\n2. The paper does not clearly specify the fine-tuning method used. While using MSE loss is a natural choice, it is conceptually similar to LoRA fine-tuning, where the goal is to keep the tuned model close to the original. More ablation studies are needed to demonstrate the effectiveness of the MSE loss in this context.\n\n3. This paper's method is simple and clean, which is good. The effectiveness of the proposed method needs more evidence. \n\n\n\n[1] Aligning Diffusion Models by Optimizing Human Utility. NeurIPS 2024.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is clear and easy to follow.\n2. It provides fair comparisons with DPO across multiple benchmarks.\n\n**Weaknesses:**\n\n1. The paper lacks a fair comparison with Diffusion-KTO [1], which is an important baseline as it also does not require paired data.\n2. More ablation studies on fine-tuning methods used and its interaction effect with MSE losses are needed to clarify the effectiveness of each component.\n\n\n[1] Aligning Diffusion Models by Optimizing Human Utility. NeurIPS 2024.", "was_revised": true, "final_justification": "I thank the authors for the detailed feedback in the rebuttal. I thus raise my rating to 4.", "rebuttal": "**W1 & Q1:The paper lacks a fair comparison with Diffusion-KTO, which is an important baseline as it also does not require paired data; The paper lacks a fair comparison with Diffusion-KTO, which is an important baseline as it also does not require paired data.**\n\nWe apologize for the omission and provide qualitative comparisons below. The experiments were conducted using the official released model for fair comparison:\n\n|     Models       | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  Diffusion-KTO  | 21.14 | 56.03 |33.81 |27.31 | 52.22\n|  Self-DPO   |21.23 | 56.35 | 34.79 | 27.33 | 71.00\n\nOur method demonstrates superior performance over Diffusion-KTO. Notably, while Diffusion-KTO avoids pairwise ranking, it still requires binary annotations (e.g., \"like/dislike\") for training, which incurs annotation costs similar to traditional preference-based methods.\nThank you for your valuable feedback.\n\n**W2 & Q2:More ablation studies on fine-tuning methods used and its interaction effect with MSE losses are needed to clarify the effectiveness of each component; The paper does not clearly specify the fine-tuning method used.**\n\nIn the main text, all experiments are conducted under full-parameter fine-tuning. Here, we present results using LoRA fine-tuning with rank 64:\n\n\n|   Models    | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  SD1.5 | 20.57 |53.15|32.58 |26.17| -14.81 \n|  SD1.5 + DPO (Full)   |20.91 |54.07| 33.19|26.46 |4.13\n|  SD1.5 + Self-DPO (Full)  |21.23 |56.35| 34.79 |27.33 |71.00 \n|  SD1.5 + DPO (LoRA)   |20.62| 53.34 |32.60 |26.28 | -11.63\n|  SD1.5 + Self-DPO (LoRA)  |20.95| 54.77 |34.38 |26.85 | 23.82\n\nWe observe that both DPO and Self-DPO under LoRA show lower performance than their full-parameter counterparts but outperform the baseline.\nThis can be attributed to the fact that LoRA reduces trainable parameters significantly compared to full fine-tuning. Notably, Self-DPO consistently surpasses DPO in all metrics, demonstrating its robustness even with parameter constraints.\n\n**Q3:This paper's method is simple and clean, which is good. The effectiveness of the proposed method needs more evidence.**\n\nWe further validate our method’s effectiveness by introducing a new real-world T2I evaluation setting. Specifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.\n\n|   Models    | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  SD1.5 | 20.57 |53.15|32.58 |26.17| -14.81 \n|  SD1.5 + SFT   |20.49| 52.63 |31.83 |26.13| -23.95\n|  SD1.5 + Self-DPO  |20.68| 52.87 |34.31 |26.52 | 17.11\n\n\nWhile naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics.\nPlease note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.\n\n**Formatting Concerns: Rows 168 and 169 are too close.**\n\nWe will fix it in the revision. Thank you for your advice."}, {"review_id": "7JiDUXlWjy", "reviewer": "Reviewer_cdb2", "summary": "The paper proposes Self-DPO, a self-supervised approach to Direct Preference Optimization for text-to-image (T2I) diffusion models. Instead of relying on externally curated preference pairs, the method automatically constructs a “winning vs. losing” image pair by degrading the higher-quality sample. This eliminates the costly human-labelled preference data required by conventional DPO pipelines.", "questions": "1. The results show that Self-DPO is even better than the DPO with human annotation datasets. Could the authors explain this? \n\n2. Random replacement often yields completely unrelated images, could the model over-weight low-level aesthetics at the expense of fidelity? A qualitative failure analysis would be valuable.\n\n3. When you draw a random loser from the same minibatch, its prompt is different from the winner’s. Do you ensure different seeds each epoch to avoid memorization?", "limitations": "Please see the Weaknesses.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths  \n\n1. By automatically “downgrading” high-quality images to obtain the losing side of each preference pair, the authors remove the expensive component of DPO—data collection and annotation. This design not only cuts labeling costs, but also enables the method to bootstrap essentially unlimited preference data from any captioned image corpus. As a result, the technique can be scaled to new domains or larger datasets without the bottleneck of crowdsourcing or labeling.\n\n2. Experiments show consistent improvements in CLIP-Score, ImageReward, and related quality metrics—despite using fewer training samples than baseline DPO. The method therefore offers both sample efficiency and better performance.\n\n3. The paper lists the degradation step and key hyperparameters in clear fashion. Readers can replicate the pipeline with minimal code changes to existing diffusion frameworks. The provided code in the supplementary material is also helpful.\n\n### Weaknesses  \n\nMajor:\n\n1. While the paper reports final metrics, it omits learning-curve plots comparing Self-DPO with vanilla DPO. Without these curves, readers cannot assess convergence speed and optimization stability. Such information is important to judge whether the observed gains come from faster learning, better local minima, or merely longer training.\n\n2. The model is trained on Pick-a-Pic, a synthetic preference dataset whose images were generated by diffusion models. Because Self-DPO no longer requires explicit preference pairs, it is crucial to demonstrate effectiveness on a standard text–image dataset. A comparison against conventional supervised fine-tuning (SFT) on such datasets would reveal whether the method’s advantages persist in real-world scenarios.\n\nMinor: \n\n1. Typo: Table 4, second row: extra “s”; Line 161: “wining image”->“winning image”; Figure 4 caption:“demonstraing” ->“demonstrating”\n\n2. How were these hyperparameters (\\lambda_1 and \\lambda_2 in Equ.9) selected, and how sensitive are the results to them?\n\n3. Blur and random-grid corruptions can hurt performance, whereas replacing the winner with a random image helps dramatically. Could you analyze why semantic mismatches outperform perceptual degradations?\n\nOverall, this paper is clearly written and the idea is both simple and interesting, but addressing the weaknesses above—particularly learning curve and more analysis—will significantly strengthen the submission.", "was_revised": true, "final_justification": "My concerns are well-addressed", "rebuttal": "**W1:While the paper reports final metrics, it omits learning-curve plots comparing Self-DPO with vanilla DPO. Without these curves, readers cannot assess convergence speed and optimization stability. Such information is important to judge whether the observed gains come from faster learning, better local minima, or merely longer training.**\n\nAll baseline models are trained for the same number of iterations. We observe that the learning curves can demonstrate consistent improvement over training. Unfortunately, due to the rebuttal policy, we are unable to include visualizations in this rebuttal. The final figures and detailed descriptions will be provided in the revision. We appreciate your feedback.\n\n**W2:It is crucial to demonstrate effectiveness on a standard text–image dataset. A comparison against conventional supervised fine-tuning (SFT) on such datasets would reveal whether the method’s advantages persist in real-world scenarios.**\n\nSpecifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.\n\n|   Models    | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  SD1.5 | 20.57 |53.15|32.58 |26.17| -14.81 \n|  SD1.5 + SFT   |20.49| 52.63 |31.83 |26.13| -23.95\n|  SD1.5 + Self-DPO  |20.68| 52.87 |34.31 |26.52 | 17.11\n\nWhile naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics.\nPlease note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.\n\n**Minor1:Typo.**\n\nWe will fix it in the revision, thank you.\n\n**Minor2: How were these hyperparameters ($\\lambda_1$ and $\\lambda_2$ in Equ.9) selected, and how sensitive are the results to them?**\n\nWe present an initial ablation study on the impact of removing the MSE loss in the main text. Here, we provide an expanded analysis comparing different weight ratios between the Self-DPO and MSE components:\n\n|     $\\lambda_2$ : $\\lambda_1$         | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  0.00 : 1.00   | 21.10 | 56.35 |33.75 |27.03 |45.03\n|  0.25 : 0.75   |21.19| 56.23 |34.71 |27.29 |66.20\n| 0.50 : 0.50  | 21.23 |56.35 |34.79 |27.33 |71.00 \n| 0.75 : 0.25  | 21.24 | 56.24 |34.75 |27.34| 68.27\n| 1.00: 0.00 |21.26| 56.22 |35.00 |27.31 |67.78\n\nWe have two obervations:\nFirst, when Self-DPO is entirely removed (0.0:1.0), the method reduces to standard SFT, yielding the lowest performance across all metrics.\nSecond, introducing Self-DPO (ratios > 0.0:1.0) consistently improves results, with performance remaining stable across varying weight distributions. This suggests the Self-DPO loss plays the main role during the training process.\nThese findings confirm that our method is insensitive to loss weight tuning, as the Self-DPO component primarily delievers performance gains.\n\n**Minor3: Blur and random-grid corruptions can hurt performance, whereas replacing the winner with a random image helps dramatically. Could you analyze why semantic mismatches outperform perceptual degradations?**\n\nThis phenomenon can be attributed to the model's greater ease in learning blur and random-grid corruption patterns. In contrast, identifying random images proves more challenging for the model due to their similar quality, which necessitates heightened sensitivity to prompts, thereby enhancing performance. Future work could explore more diverse and reliable image degradation strategies to further optimize model capabilities.\n\n**Q1:The results show that Self-DPO is even better than the DPO with human annotation datasets. Could the authors explain this?**\n\nThis can be attributed to the fact that conventional DPO relies on pre-generated offline data with limited diversity, whereas our method dynamically generates preference pairs in an online fashion. Online generation introduces greater data diversity by continuously sampling from the model's current policy distribution, addressing the static nature of offline datasets.\n\n**Q2:Random replacement often yields completely unrelated images, could the model over-weight low-level aesthetics at the expense of fidelity? A qualitative failure analysis would be valuable.**\n\nThe model does not sacrifice fidelity during preference optimization. A detailed qualitative failure analysis will be included in the revision. Thanks for your advice.\n\n**Q3:When you draw a random loser from the same minibatch, its prompt is different from the winner’s. Do you ensure different seeds each epoch to avoid memorization?**\n\nYes, each epoch generates different image pairs within the same minibatch."}, {"review_id": "MqOXPYHaNR", "reviewer": "Reviewer_4ghY", "summary": "The paper introduces Self-Supervised Direct Preference Optimization (Self-DPO), a training paradigm for aligning text-to-image diffusion models with human preferences—without requiring any manually ranked data or pre-generated images dataset. Instead, the method dynamically constructs training pairs by identifying a \"good\" image, and a corresponding \"bad\" image by downgrading the winner. In practice, the authors choose to downgrade the winning image by selecting a random image instead, which lacks text faithfulness. Importantly, this approach surpasses the more expensive DPO baseline on several datasets, metrics, and base models. Additionally, the authors experiment with different downgrading operations such as blur, random image, and patch mixing.", "questions": "1. The authors make use of the \"preference image\" in the Pick-a-Pic dataset. However, the claim is that the algorithm does not need annotation. To better support this claim, it can be helpful for the authors to sample between the preferred image and the rejected image. Will the approach work using random images from pick-a-pic?\n2. Will the algorithm work with real images rather than generated images?\n3. Will the approach work for SOTA models such as FLUX? If it is unclear how to perform such an experiment, it can be useful to add this to the limitations section.\n4. For different downgrading operations, are you able to analyze and identify different improvement axis? this can also be answered qualitatively.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The core idea of selecting a winning image and utilizing a simple and efficient heuristic to get a losing image, and then applying DPO is simple and very elegant.\n2. The effectiveness of this approach when compared to DPO, combined with the limited resources required to apply it, make it both appealing and accessible.\n3. The paper is easy to follow.\n4. The experimental setup is extensive and contains multiple models, datasets, metrics, etc.\n\nWeaknesses:\n1. The authors make use of the preference image in the Pick-a-Pic dataset. However, the claim is that the algorithm does not need annotation. To support the claim the authors should use random images from the Pick-a-Pic dataset, rather than the preference image.", "was_revised": true, "final_justification": "Solid paper, see the other sections as for why I gave this score.", "rebuttal": "**W1 & Q1:The authors make use of the preference image in the Pick-a-Pic dataset. However, the claim is that the algorithm does not need annotation. To support the claim the authors should use random images from the Pick-a-Pic dataset, rather than the preference image; Will the approach work using random images from pick-a-pic?**\n\nYes, the method performs effectively when using randomly selected images from the pick-a-pic dataset. As shown follows:\n\n|   Models    | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  SD1.5 | 20.57 |53.15|32.58 |26.17| -14.81 \n|  SD1.5 + DPO   |20.91 |54.07 |33.19 |26.46 | 4.13\n|  SD1.5 + Self-DPO  |21.23 |56.35 |34.79 |27.33 |71.00\n|  SD1.5 + Self-DPO (Random images) |21.20 | 56.25 |34.49 |27.39 | 59.18\n\nThe results demonstrate that the random selection approach achieves comparable performance to the standard Self-DPO method across most metrics. We appreciate your feedback and will incorporate these findings into the revision.\n\n**Q2:Will the algorithm work with real images rather than generated images?**\n\nYes, the method also works with real images. Specifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.\n\n|   Models    | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n|  SD1.5 | 20.57 |53.15|32.58 |26.17| -14.81 \n|  SD1.5 + SFT   |20.49| 52.63 |31.83 |26.13| -23.95\n|  SD1.5 + Self-DPO  |20.68| 52.87 |34.31 |26.52 | 17.11\n\nWhile naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics.\nPlease note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.\n\n**Q3:Will the approach work for SOTA models such as FLUX? If it is unclear how to perform such an experiment, it can be useful to add this to the limitations section.**\n\nYes, Self-DPO works for SOTA models such as FLUX beacuase it is model-agnosic. Ideally it can be employed in most diffusion models. As an initial attempt, we performs this experiment on FLUX using the Pick-a-Pic V2 synthetic dataset. Notably, this dataset comprises image generations from SD2.1 and SDXL, which have lower quality compared to FLUX outputs. Despite this, our experiments demonstrate that the method remains effective when applied to modern architectures.\n\n|                 | PickScore | Aesthetics | CLIP | HPS | ImageReward \n|-----------------------|-----------|-----------|-----------|-----------|-----------|\n| Flux    | 22.45     | 62.25 | 32.95 | 28.81 | 102.00\n| Flux + DPO  | 22.46      |  **62.38** | 32.88 | 28.86 | 101.41\n| Flux + Self-DPO  | **22.52**     |  61.98 | **33.56** | **28.99** | **108.31**\n\n**Q4:For different downgrading operations, are you able to analyze and identify different improvement axis? this can also be answered qualitatively.**\n\nThank you for your feedback. Each image downgrading operation involves distinct optimization dimensions. For random grid downgradtion, it varies in grid configurations and selection numbers. In such scenarios, a enhancement could involve using instance masks to relocate objects within the image, generating more feasible results with minimal fidelity loss. For blur downgradtion, it can be changed by different blur kernels and area. We appreciate your suggestion and will incorporate this discussion into the revision."}]

Author rebuttal:

Meta-review: This paper proposes a refinement/simplification to the Diffusion DPO algorithm that solves the data dependency issue. Rather than relying on human-labeled preference pairs, they generate synthetic pairs by taking high quality images and downgrading them in order to constructing winning/losing pairs. They show that this exceeds the effectiveness of regular DPO.

Reviewers had no major objections, they found the approach elegant. There was some concerns that certain ablations were not done, but the authors seemed to respond effectively. I personally have 2 concerns that I think I take more seriously than the reviewers do:

a. Evaluation was done by (weak) reward models not stronger VQA and Human ratings, and these frequently do not transfer to downstream effectiveness.
b. I don't think this sort of augmentation can completely replace the usual way of doing preference set construction for DPO, we do need to sample a fuller distribution of deltas between winner and loser, there is a chance in fact that "simpler" degradations are exactly the ones that will give gains on CLIP without reflecting in gains or harder evaluations.

However, I think since this paper was universally approved by the reviewers, I still recommend acceptance.
Decision: Accept (poster)