Aligning Text-to-Image Diffusion Models to Human Preference by Classification
Authors: Longquan Dai
Xiaolu Wei
He Wang
Shaomeng Wang
Jinhui Tang
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion Models
Human Preference
Primary area: deep_learning

Abstract: Text-to-image diffusion models are typically trained  on large-scale web data, often resulting in outputs that misalign with human preferences. Inspired by preference learning in large language models, we propose ABC (Alignment by Classification), a simple yet effective framework for aligning diffusion models with human preferences. In contrast to prior DPO-based methods that depend on suboptimal supervised fine-tuned (SFT) reference models, ABC assumes access to an ideal reference model perfectly aligned with human intent and reformulates alignment as a classification problem. Under this view, we recognize that preference data naturally forms a semi-supervised classification setting. To address this, we propose a data augmentation strategy that transforms preference comparisons into fully supervised training signals. We then introduce a classification-based ABC loss to guide alignment. Our alignment by classification approach could effectively steer the diffusion model toward the behavior of the ideal reference. Experiments on various diffusion models show that our ABC consistently outperforms existing baselines, offering a scalable and robust solution for preference-based text-to-image fine-tuning.

Aligning Text-to-lmage Diffusion Models to Human Preference by Classification
Longquan Dai, Xiaolu Wei, He Wang, Shaomeng Wang, and Jinhui Tang' Nanjing University of Science and Technology, Nanjing, China {dailongquan, weixiaolu, wanghe, smw, tangjinhuij] Jonjust edu. cn
Abstract
Text-to-image diffusion models are typically trained on large-scale web data, often resulting in outputs that misalign with human preferences. Inspired by preference learning in large language models, we propose ABC (Alignment by Classifica- tion), simple yet effective framework for aligning diffusion models with human preferences. In contrast to prior supervised fine-tuned (SFT) reference models, ABC assumes access to an ideal ref- erence model perfectly aligned with human intent and reformulates alignment as a classification problem. Under this classification view, we recognize that preference data naturally forms semi-supervised classification setting. To address this, we propose data augmentation strategy that transforms preference comparisons into fully supervised training signals. We then introduce classification based ABC loss to guide alignment. Our alignment by classification approach could effectively steer the diffuusinn model toward the behavior of the ideal reference. Experiments on various diffusion models show that our ABC consistently outperforms existing baselines, offering scalable and robust solution for preference based text-to- tot-t---mmge fine-tuning toningung...... Code available at https:/github..com// https//ggthbb.commmlliogguaaaaa
methods that depend on suboptimal
Introduction
Text-to-image diffusion models [4] have dominated image generation for years, trained web-scale text-image pairs in a single stage. However, this approach may produce images misaligned with human preferences. In contrast, Large Language Models (LLMs) excel generating numan-preferred outputs through a two-stage process: pre-training on web data and fine-tuning on preference data. Applying this fine-tuning strategy text-to-image models could enhance their ability meet diverse user preferences, making them more useful and relevant.
Recent research 12] 9 14 59 61 has focused enhancing diffusion models align with human preferences using Reinforcement Learning from Human Feedback (RLHF) 22]. This RLHF approach involves pretraining reward model [56] to capture human preferences and then optimizing the diffusion models to maximize the reward generated images. However, creating a robust reward model that accurately reflects human preferences is both challenging and computationally costly, and over-optimizing the reward model can lead to significant issues of model collapse [38].
Diffusion-DPOC (51 integrates Direct Preference Optimization (DPO) 40 into the preference learning framework of diffusion models, eliminating the need for a reward model. DPO reparameterizes the reward function in RLHF directly learn model from preference data. In DPO, the implicit reward formulated using the log ratio the likelihood of a response between the current model and the supervised fine-tuned (SFT) model. However, the SFT model is far from ideal model that aligns with human preferences completely. We hypothesize that this discrepancy may lead to suboptimal performance.
CCoresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we propose the ABC (Alignment by Classification) framework, simple yet effective preference optimization algorithm to solve this problem. The core of our algorithm relies on three key insights: (1), DPO with an ideal reference model can be framed as classification problem using a diffusion model. (2), The alignment performance depends on the discriminative ability of the diffusion model. (3), We identify alignment with preference data as semi-supervised learning and propose a data augmentation method to convert into supervised data. (4), We propose a classification-based ABC loss, incorporating augmented preference data, to align the diffusion model, which equivalent to aligning the diffusion model with the ideal reference model. Therefore a reference model is not needed during training which saves a lot of memory.
We conduct empirical evaluations of our ABC framework on state-of- of-e-of-the-as fi-thee-rtt art text-to-image diffusion models, including SD1.5 [2] and SDXL) [37,, comparing it with leading image preference alignment methods. Extensive analysis demonstrates that our ABC method effectively leverages preference data, resulting in more accurate ranking of winning and losing responses.
2 Related Work
Diffusion model alignment can be achieved through fine-tuning [1] 60 63
Recently, methods [I
2 27 448 (59 [6]] based on RLHF have garnered increasing attention. Among them, Wallace al. [51] expanded direct preference optimization [40), which was originally suggested for language models, to diffusion models, aligning them with pairwise preference datasets on top of a frozen reference model. Similarly, Li et al. 26 applies Kahneman-Tvvrskf Optimization (13] from language model alignment to diffusion models to inject preferences into the reference model. Theoretically, using an ideal alignment model as the reference should produce the best results, but all these methods rely on a non-perfect SFT checkpoint as the reference model. In this paper. we disclose that using an ideal alignment model the reference and minimizing the DPO loss will minimize a classification loss. We thus transform the reference-required alignment task into a reference-free classification task.
Diffusion model classification is type of generative classification [64], where class probabilities ply (z) are inferred by modeling the data likelihood p(æt) ly) using generative models. Compared to discriminative classifiers [49], generative classifiers tend to be more robust and better calibrated (31. Zimmermann et al. 665 leverage score-based models to compute the log-likelihood p( ly) via
integration and then apply Bayes' theorem to obtain p(ylar). Other works [16] 21 perform diffusion
in logit space to model the categorical classification distribution. Recent studies [6 17 25 convert diffusion models into generative classifiers, showing that generative networks can be effectively repurposed for discriminative tasks. In this paper, we further reveal a close connection in the reverse direction: discriminative learning, specifically classification, can also be naturally applied a particular generative task diffusiioo model alignment.
Classification loss generally follows two paradigms: learning with class-level labels and learning with pairwise labels. In the first setting, the model is trained to assign each input to its corresponding class using a classification loss, such 1.2-Softmax [41] Large-margin Softmax [29] Angular
Softmax (28), NormFace [52), AM-Softtax (53), and ArcFace [2). In contrast, pairwise-hased
approaches learn directly model similarity or dissimilarity between sample pairs. Representative methods include contrastive loss 7 [15), triplet loss [19 43). Lifted- Liftee-Structtue loss [34], N-pair
loss 46. Histogram loss [50], Angular loss [54]. Margin-baser loss (57), and Multi-Similarity
loss 555 In this paper, we employ Circle loss which unifies the two paradigms, to conduct the
diffusion model alignment task.
3 Background
Diffusion models are certifiably robust classifiers [5]. To provide a classification perspective on the preference alignment of diffusion models, we offer preliminary discussion diffusion models [4]
18 and diffusion classifiers 5 8), as well as the Circle loss [47, a generalized classification loss.
3.1 Diffusion Models
We briefly review denoising diffusion probabilistic models [8] Given an from a real data distribution q(x)) and assuming that the signal-to-noise ratio SNR (t) az/of is monotonically decreasing

---

over time, the forward diffusion process gradually adds Gaussian noise to the data to obtain sequence of noisy samples {ar]==1 according to {ar]==] and {or)f=1 which are designed such that or is nearly an Gaussian distribution ((0 = (ax;; va The reverse process po(#:-1|az) = (= (0 t), T² is defined as Markov chain aimed at approximating ( by gradually denoising from the Gaussian distribution p(æx)) Wxrr; O. I). where μo is generally parameterized by time- conditiodi noise prediction network cø t). Let C' be small constant and w; the weight. The reverse process can be learned by optimizing the variational lower bound on the log-likelihood
log (o) 2 -e,t (u c12) +C
3.2 Classification with Diffusion Models
where each image (1() is associated with label y that belongs to one of K classes, denoted as Y = [ Given a new image æ, the classification objective is to predict the class label ý that has the highest probability of being assigned ø.
Consider dataset Ω y )
argmin p(y(2)) argmin -p(xly) -p(y).
Assuming uniform prior distribution over the classes, i.e. p(yx) & for all k, the prior term becomes constant and can be ignored the maximization process. Thus, the problem reduces to:
argmin - log p(aly).
Clark and Jaini [8] leverage the score function še y) (5). which can be considered good measure of the similarity between the category prompt y and the image æ, approximate log po yy) and convert the text-to-image diffusion model into classifier (4).
ÿ argmin log (o(yy)) = argminyev - (x y), where
(4)
50 50 y) E.,, (sr y)] and sa w, le (Vara o, 1)
(5)
Further, Chen 5 extend this approach by caleulating the class probability po(y(æ)) through
p(aryye) plyc) [=(yx<æ)) Enjeyp(aly,) ply,) Σy, ey exp(logpo(x1yy,) Ey,er exp(so(m.)))
exp(logpo(zrlyx)
exp(so(xx yx))
(6)
3.3 Circle Loss for Classification
Classification involves selecting one target category from K candidate categories. Suppose the scores of x: for binary categories are 2 for simplicity. The binary cross-entropy loss is given by:
LACE Σn [lo (1 exp y2)) + log (1 exp (1( where
(xx, .y Ee, [so(x,y))) E.,, (z)
Sun et al. 47 propose the binary Circle loss Leaete by extending (xx yt, as:
(x,y+,y")== : = (Be, [se(x, y+]] A+) - (E., |s (x, y A-).
(8)
The Circle loss degenerates to AM-Softmax LAM loss [53 an important variant of the binary cross-entropy loss when y .y-) defined as:
(x,y*.y Ex, [so (2 - (E., [1)
4 ABC for Diffusion Models
In this section, we introduce our Alignment by Classification (ABC) framework for diffusion models. We first reformulate alignment classification task. To mitigate the instability from the semi- supervised nature of preference data, we apply data augmentation enable supervised learning. Finally, we present the ABC objective for preference alignment.

---

4.1 The Connection Between Alignment and Classification
In the following sections, we assume each text prompt corresponds to a single aligned image. Let a. denote the image aligned with prompt y, and xy a misaligned one. Here, we provide classification perspective on diffusion model preference alignment [5T], formalized through two theorems.
The first theorem shows that the Diffusion-DPO loss serves as an upper bound on the diffusion classification score O Specifically, the Diffusion-DPO loss [5I] is defined as Equation (TO), where Str( y) w/llc E Gerl, Eerlyaar yayz + o.c. y, t) and Eer is reference diffusion model. Therefore, minimizing this loss corresponds to training diffusion- based classifier.
LDDRO E,, log (1 + exp (- (50(22, y) Szet (2, y))) exp (se(2,) .y) Stel ()))) (10)
Unlike common practices where Exef (Zz, y, t) is set as the SFT checkpoint [40], we consider as the ideal model here for discussion. The reason is that using the ideal alignment model the reference model should produce optimal performance in preference optimization. Thus, it deserves a typical case to discuss in the following theorem.
Theorem 1. (Proof in the supplementary material) We say a diffusion model Cali (mt,Y, 1) is ideal alignment if it satisfies ||eali ( y.t) cl12 = 0 and || V, <l = 5 for any y. Here, Biyy Voraa + oge and Fyy = Voras + ase. When the reference model ert (zt, enti (ar,,, 1) in Equation (10p an ideal alignment model and 8an (an( = welle Eali (Vore +
o,e,y, v,t 12. the AM-Sofimax loss (9) is upper bounded by the Diffusion-DPP loss 10 Specifically, we have
log (1 exp (- (Be,, (2, .)] -6)) exp (Ee, [so(z, v)])
(1I)
I Be,r [log (1 + exp (- (50 y) (2, y))) exp (so(x)) - y) San .)))]
This theorem indicates that the Diffusion-DPOO loss serves an upper bound for the AM-Softmax loss (53). Thus, minimizing the Diffusion-DPO loss with an ideal reference model will also minimize the AM-Softmax loss. This implies that performing the alignment task results in performing the classification task for diffusion models.
The second theorem shows that the predicted noise in a diffusion model is a weighted average of noise estimates across all possible images. To generate an image aligned with a prompt, the model must increase the weight on the noise corresponding to the aligned tange-highlighting that strong classification ability is essential for alignment.
Theorem 2. (Proof in the supplementary material) Let y= (y:) denote N text prompts and D = {axy; be corresponding aligned images. We assume that the prior prompt distribution ply) and image distribution p(x) are uniform. To describe the discriminative ability, define the conditional probability p(æly) as
n N MIN-I
y y - [y.)- where n < N.
p(ylay,))
=
(12)
Then, p(øly) = p(ylaz) and the optimal diffusion model Copt (It. y,t), f), which achieves minimal diffusion loss over both the training set and the test set, over D is given by:
Cope Cope(Dr, (Ir, v, >
Wi
E
(13)
Σ
Ex(eE Wj
exp
Vare
x E {ay)
where E;
-1N-1) N
W/
2
( E D (ay)-
exp
2
Theorem 2 indicates that the predicted noise of the optimal diffusion model is the weighted average of the noise E,, which denotes the exact noise contained in 2, with respect to the clean image ( D. In order to control the diffusion model approximate the image Iy that is aligned with the text y, it has to make the predicted noise approximate to the noise Re-yara This further leads to approximate N, which means we maximize p(ylary). Since diffusion models provide a good estimation for p(yl ar) according to Equation (4), this implies that once the diffusion model an ideal classifier, the diffusion model Eopt (Z,, t) will be an ideal alignment model.

---

Finally, to enhance understanding, we briefly interpret the two theorems. Theorem ! proves that the AM-Softmax loss is upper bounded by the Diffusion-DPO loss. In other words, minimizing the Diffusion-DPO loss for better alignment will also reduce the AM-Softmax loss, leading to improved classification performance. Simply put, better alignment leads to better classification. Conversely, [heorem@]showe that, under certain conditions, improved classification leads to better alignment. Together, these two theorems reveal a strong connection between classification and alignment, forming the theoretical foundation of our approach, which replaces the DPO loss with the ABC loss for alignment tasks. We first establish a connection between alignment and classification.
4.2 The Connection Between Alignment and Semi-Supervised Learning
Diffusion model alignment is form of semi-supervised learning using a human preferences dataset. Specifically, Diffusion-DPO [5]] is fine-tuned on Pick-a-Pic Pic [23], human preference dataset for text-to-image generation. To construct the dataset, an SFT model generates pairs of images (I1, X2) from given p. These pairs then shown to human annotators, who indicate a preference, denoted as 2 > Ty where at and xyy are the preferred and dispreferred images, respectively. Each dataset item is thus a triplet (y, x By ). In this setup, y serves as the correct label for 2y while zy lacks corresponding optimal prompt. As alignment resembles a classification task in which only half the data has labels, naturally fits within semi-supervised classification framework.
Regularization is critical for stable semi-supervised training. When aligning the diffusion model using AM-Softmax-as as suggested in Theorem [ the optimization problem reduces to:
ming E log (1 + cxp (u(Z,, x y))) where ( y) = EE, [s (e(x*t,x .y)] (Be, [so( [v]] - --).
(14)
Since 2. lacks a corresponding prompt, the optimization tends to maximize so(27 yy) which can lead to ||+ - co (araz + o,e,y, 4) l12 becoming arbitrarily large. However, even if a, is less preferred than 2. this reconstruction error should still remain bounded; otherwise, the diffusion model will lose its ability to generate valid images. A practical compromise to prevent the loss from diverging is to select a large A effectively modeling an ideal alignment function with large ( y, 1) -- e el22 during DPO optimization. However, this does not fully resolve the issue, which helps explain why Diffusion-DPO may fail to reliably train diffusion models in some cases.
We regularize classification through data augmentation to mitigate instability caused by missing prompts in half of the user preference dataset. Specifically, we define y* as the original prompt and construct y by appending "The image that aligns less with human preferences" to y. This refor- mulates each preference tuple (y, 2,, .2, into two supervised examples: ly+,zy, and (yT, ,ay ). The task thus becomes binary classification problem between images conditioned on y and y converting the eeni-supereieed objective (14) into a fully supervised one (15), which helps stabilize training by constraining the residual error term.
mine Eyev [log (1 exp (1(xy. Byy, ,y))))))) + log (1 + exp (c(xy) })]] where ((x: " .y) Reg, [so [y)] - (Fe,, [so(x: (2: .y)] -17).
,
(15)
4.3 ABC Loss for Alignment
Theorem I shows that optimizing the Diffusion-DPP loss 10) effectively minimizes the AMM-offtmx loss (9) Theoremp further demonstrates that achieving ideal alignment requires the diffusion model to be discriminative. Section 4.2 attributes training instability of alignment to the semi-supervised nature of the task, which we address through a data augmentation strategy that converts it into supervised classification problem. Together, these results support the feasibility of aligning diffusion models to human preferences via classification.
Alignment by classification imposes an important constraint: the expected score for the less preferred image, E [s0(x= (y)], must be properly bounded. If this value becomes too large, the model is compelled to increase Fr,,, [so(xt,y) y)] accordingly, which may cause the diffusion model to fail in generating coherent images. Conversely, if E,, [se(x .y)] is too small, the model becomes
5

---

insufficiently discriminative, weakening its ability to align with human preferences according to Theorem 2 Ideally, the model should maintain saargin-basee separation:
E. [s .Y)] = B.,, [se [y)] + 8,
(16)
where 5 > O is a fixed positive margin that ensures both discriminability and stability. To enforce this constraint, we adopt the Circle loss (8)- generalized version of the AM-Softmax loss-whidd better accommodates the margin- formulation. Specifically, we introduce the Alignment by Circle (ABC) loss, denoted LABC, defined as follows:
CAnc (0) = Σ yEY [log (1 + exp (1(xy- B y y))) + log (1 + exp (1(xy, 1 by-yy"))) where a(x xx ,y) nly (Be, [so + y)] - - "y (Be,, (so .y)] -A7)
(17)
Here, nyt and My act self-paced weighting factors that adaptively emphasize samples with subopti- mal scores specifically, those far from their ideal values O+ and - to ensure stronger gradients and more effective updates. We define these weights as follows:
my Ee,t [se (xt ,y) 0 fly 0, Ee,, (so ( y)] Since log (1 + exp(x)) is a monotonically increasing function, minimizing the loss is equivalent to minimizing the term u(x , y). Substituting Equation (18) into this term yields:
{ 0,
O - [ ( y)]l + 6, 1 Eg, (so (ay y)]l
(18)
(xx x ,y,,))= = Be, [s .y]] 0 +4y 2 (E, [so(xTT))) O, + 2
(19)
2
2
where sg[-] denotes stop-gradient (i.e., the value is detached from backpropagation). and 6 introduces soft margin to ensure separation between positive and negative scores. The minimizer of this objective can be verified as (0, (s[[,,,ss00z (atls//(.t.)))) .)]l] 5). Accordingly, the loss drives IE,, (st(x,))) y)]
toward O, while encouraging E,,, [se .y)] to approach sg(E,, (so y)]l + precisely the behavior we desire for effective alignment.
5 Experiments
We present both qualitative and quantitative experiments in Section to highlight the alignment advantages of our method. In Section 5.2 provide deeper analysis of why our approach outperforms existing diffusion model alignment techniques. In Section 5.3, we conduct an ablation study examine how the hyperparameters in the ABC loss affect alignment quality.
5.1 Human Preference Alignment Comparison
We present both quantitative and qualitative comparisons for human preference alignment. Our approach builds on the Diffusion-DPe codebase [51]. We train the models using the Adam [30 optimizer for SD1.5, Adafactor 455 optimizer for SDXL on A6000 GPUs, with a batch size of 2, gradient accumulation of 128 steps and a learning rate of 1 x 10 10-8, incorporating a linear warmup schedule. For SD1.5 and SDXL. training, å is set to 0.025. These settings largely follow the original Diffusion-DPO configuration, with minor modifications to enhance training efficiency. We apply our proposed ABC loss to fine-tune both the SD1.5 and SDXL base models, resulting in our SD1.5-ABC and SDXL-ABC variants. For clarity, adopt "Model-Method" naming convention-ee e.g. SDXL-ABC refers to the SDXL model fine-tuned with the ABC loss.
5.1.1 Qualitative Comparison
We present a qualitative comparison of SDXL-ABC with SDXL-Base. SDXL-DPO (51). SDXL- SPO (27). and SDXL-MAPO 2 in Figure D where SDXL-Base represents the original SDXL-1.0 model. As shown in Figure I SDXL-ABC generates images with clear improvements in eext-imaee aligmment. To assist readers identifying mismatches between the text and the generated images, we highlight relevant textual phrases in color and enclose the corresponding image regions in bounding boxes. Our method incorporates human preferences through direct optimization based on user feedback, resulting in more engaging visuals, such vivid color palettes, dramatic lighting, coherent compositions, fine detail, creative elements, consistent color harmony, and structured multi-object arrangements. More important, the generated text-image pairs are also more semantically aligned.

---

Prompts
SDXL-Base
SDXL-DPO
SDXL-SPO
SDXL-MAPO
SDXL-ABC
A capybara wearing sunglasses and a blue cap.
baby red panda wearing cake as hat.
A pineapple with one beer left and two beers right.
Figure 1: Qualitative Comparison for Diffusion Model Alignment. We develop alignment-by- classification approach to align diffusion models with human preferences. Fine-tuned from the SDXL-110 model, method generates images with improved visual appeal and textual alignment compared to other alignment baselines. In our comparisons, SDXL-DPO [51), SDXL-SPO (27) and SDXL-MAPO 20) denote competing aligned variants and SDXL denote the SDXL-1.0 model.
Table 1: Quantitative Win-rate Comparison Using Automated Preference Metrics. We evaluate the alignment performance of diffusion models using prompts from HPS and PartiPrompts across various evaluators. Both SDXL and SD1.5 serve base models. Win rates above 50%-indicating superior performance over the baseline-ane highlighted in bold. We note that MAPO has not released their SD..s-based checkpoint, and KTO has released their -based checkpoint.
PartiPrompta
HPS benchmark
PickScore
HPS
Aesthetics
CLIP
PickScore
HPS
Aesthetics
CLIP
vs. SDI.5-Basc
60.02
81.51
74.27
59.72
74.83
85.75
68.84
59.65
vs. 3DL.5-DPO
55.85
73.02
64.90
44.97
53.46
71.50
64.19
$2.06
vs. SDI.5-SPO
51.16
61.59
47.60
60.02
45.35
$4.99
38.08
64.83
vs. SD1.5-KTH
57.77
44.72
53.90
47.22
52.28
42.88
52.86
53.93
vs. SDXL-Base
74.38
79.26
80.20
52.46
79.35
70.17
72.28
60.38
vs. SDXL-DPO
73.22
72.50
68.25
50.51
77.26
69.54
70.19
$7.06
vs. SDXL-SPO
52.49
40.31
59.93
55.53
51.16
52.41
46.78
59.87
vs. SDXL-MAPO
65.35
81.17
72.10
46.97
68.55
51.14
5.1.2 Quantitative Comparison
We compare our method against existing baselines-inccuddin SD1.5, SDXL, and their DPO, SPO, KTT, and MAPO variants-using both user studies and automated preference metrics. For automated evaluation, we assess Pick Score [23], HPS [58). LAION Acsthetics [44), and CLIP [39], using prompts from the HPS benchmark (58) and PartiPrompts [62]. We report win rates between
method and each baseline under these metrics Table [ while Table 2 presents results on the GenEval benchmark evaluating model performance across 8,000 prompts.
To confirm the method's efficacy, we conducted user study. Specifically, we randomly sampled 100 prompts from the PartiPrompts dataset and another 100 prompts from the HPSv2 benchmark. For each prompt, we generated five images using five different methods. Participants were shown five images per prompt (one from each method) and asked to answer three questions: Q1 Which image is your overall preferred choice? Q2 Which image is more visually attractive? matches the text description? To minimize position bias, the order of images randomized for
Q3 Which image better

---

Table 2: Quantitative comparison on GenEval. We evaluate model performance on 8,000 prompts spanning attribute binding, relationships, numeracy, and complex compositions. Higher scores indicate stronger alignment with the intended composition.
Model
Color (B-VQA)
Shape
Texture (B-VQA)
Numeracy (UniDet)
2D- Spatial (UniDet)
3D- Spatial (UniDet)
Non- Spatial (CLIP)
Complex (3-in-1)
(B-V (B-VAA)
SDI.5-Base
0.3811
0.3395
0.4192
0.4436
0.1460
0.2912
0.3092
0,3002
SDI.5-DPO
0.3943
0,3440
0.4374
0,4523
0.1627
0.3090
0.3091
0,3032
SDL,5-SPO
0.4030
0,4001
0.4152
0,4461
0.1471
0.2958
0.3010
0.3131
SD1.-KTTO
0.4645
0.3815
0.4730
0,4618
0.1919
0.3318
0.3104
0,3514
SDI.5-BCC
0.4647
0.4005
0.4751
0,4570
0.1895
0,3324
0,3106
0.3587
SDXL-Base
0.5708
0,4880
0.5600
0.5591
0.1949
0,3551
0.3065
0,4383
SDXL-DPO
0.6586
0,5358
0.6521
0,5300
0,2376
0.3668
0.3116
0.4923
SDXL-SPO
0.6431
0.5200
0.6496
0.5765
0.2298
0.3513
0.3031
0,4424
SDXL-MAPO
0.6682
0.5104
0,5650
0.5189
0.1700
0.3507
0.3136
0,4401
SDXL-ABC
0.6708
0,5450
0.6866
0.5623
0.2401
0.3697
0.3154
0.5051
SD1.5
SDXL
DPO
SPO
KTO
MAPO
ABC
HPS Parti
SD1.5
14.3%
19.2%
20.7%
17.6%
28.2%
SDXL
13.1%
13.5%
21.6%
18.8%
33.0%
SD1.5
12.2%
17.7%
20.9%
18.3%
30.9%
SDXL
9.8%
11.6%
23.9%
18.2%
36.5%
Figure 2: Quantitative Win-rate Comparison Using User Study, Both SD1.5-ABC and SDXL-ABC outperform the baselines, with the top figure showing results on PartiPrompts and the bottom showing results on the HPS benchmark. The rows for SD1.5 and SDXL indicate that the base diffusion models SD1.5 and SDXL, respectively. Our method consistently generates outputs with higher overall preference across two key dimensions: visual appeal and prompt alignment. We note that MAPO has not released their SD1.5 SD1.5-based ba-based checkpoint, and KTO has not released their SDXL-based checkpoint.
each prompt. Each method's final score was computed as a weighted sum of its win rates under the three criteria, with weights of 30% for general preference, 30% for visual appeal, and 40% for prompt alignment. The study was conducted as a blind evaluation. Annotators were not informed about which method generated each image. We recruited participants from our research group, comprising approximately 100 students, and collected a total of 82 valid responses.
Table reports the win rates of ABC-aligned diffusion models against their respective baselines. Fine-tuning with our ABC loss consistently improves performance for both SD1.5 and SDXL across nearly all metrics and datasets, demonstrating the effectiveness of our approach. Table [ shows the quantitative comparison on GenEval, with ABC achieving competitive or superior performance across various compositional tasks. Figure further illustrates user study results, where our method receives the highest number winning votes terms of general preference and visual appeal. For instance, on the HPS dataset with SDXL, ABC achieves leading win rate of 36.5% in general preference among five competing methods.
5.2 Performance Analysis
Theorem I demonstrates that enhancing model's 's alignment capability improves its classification performance, while Theorem 2 indicates that stronger discriminative ability leads better alignment. In this section, we evaluate the zero-shot classification performance of aligned diffusion models investigate the intrinsic connection between alignment and classification. We also analyze their discriminative strength, providing experimental evidence to support the effectiveness of our approach.
5.2.1 Zero-Shot Classification
To validate the effectiveness of Theorem we evaluate zero-shot classification performance on six benchmark datasets: Food-10 101 3. CIFAR-10 [24], Aircraft [32] Pets [36]. Flowers [3] and STI.-10 10). We adopt prompt templates and class labels from (39]. including refinements to disam- biguate class names (e.g., "crane" - "crane bird") [35]. As shown in Table 3 diffusion classifiers
8

---

Table 3: Zero-Shot Classification Performance. We adopt the Robust Classification via Single Diffusion Model method [6] to evaluate the classification ability of checkpoints produced by dif- ferent alignment methods. The results suggest that alignment generally improves the classification capabilities of diffusion models. Among all alignment approaches, our method achieves the highest classification accuracy.
Food-10) 101
(IFAR-10
Aireraft
Pets
Flowers 102
STI--10
SD1.5-Bas
75.87
83.15
25.47
83.53
50.33
89.44
SD1.5-DPO
78.57
84.79
28.39
87.37
52.96
92.38
SD1.5-SPO
76.04
84.59
27.25
84.64
51.94
91.06
SD1.5-KTO
77.93
83.92
26.71
86.21
51.13
92.03
SD1.5-BBC (Ours)
79.12
85.13
29.11
88.48
53.42
93.75
0.05887
0.00740
Pick-a-Pic
0.05833
0.01003
0.05875
0.00760
DPO
0.05813
0.01141
0.05861
0.00851
SPO
0.05795
0.01167
0.05840
KTO
ABC
Pick-a-Pic
Figure Comparison of Noise Prediction Errors on Pick-a-Pic and HPS. Left: Average noise prediction error Re, (so .y]] for the preferred text-image pair (x+.y) y). Right: Average margin = LE,, [sa(# [y)] - E., [ y)] between the noise prediction errors of the preferred pair (x+ y) and the dispreferred pair (æ (x".y). A lower noise prediction error suggests higher image quality, while larger indicates better discrimination aligned with user preference.
built on aligned SDeess-SD1.5-PPPO SD1.5-SPO, and CA--T----oiistetlllyy outperform the baseline classifier based on the original SD1.5. We attribute this improvement to the fact that the dif- fusion DPO loss serves as an upper bound to the AM-Softmax classification loss, thereby enhancing alignment with discriminative objectives. These results empirically support the theoretical insight from Theorem | which states that better alignment capability improves classification performance. Furthermore, our method outperforms other diffusion-based classifiers, likely due to the explicit use of classification loss to guide alignment.
5.2.2 Discriminative Strength Measured by Prediction Error
Theorem confirms that stronger discriminative ability in diffusion models leads to better alignment. Assuming a uniform prompt distribution p(y). Equations (4) and (5) show that the class probability p(y[æ) (x) is proportional to exp (-E,,t [se (æ. y)]). Therefore, lower prediction error indicates better generation quality and a higher likelihood that æ belongs to class y. further improve discriminative power, it is crucial to maximize the margin between prediction errors of positíve and negative samples. However, directly maximizing exp (-E,, [Se (z, ,)]) can destabilize training when the prediction error is large. more stable approach is to maximize the margin A = Ez, [so (tx(=,))) E., [s y)] ensuring it is large while keeping the generation quality high. Ideally, the noise prediction error E.,, [a + y)] should be as low as possible for preferred pairs, and the margin A should be large possible reinforce alignment with human preferences. As shown in Figure 3 our method achieves the lowest prediction error and the largest margin on both the Pick-a-Pic and HPS datasets.
5.3 Ablation Study
The ABC loss 17 introduces a hyperparameter 6, which defines the separation margin between preferred and dispreferred samples. This margin directly affects the strength of the alignment signal

---

during training, consequently, the quality of the generated results. As shown in Table 4 all metrics follow consistent U-shaped trend, with 8 0.025 achieving the best overall performance.
When 5 is too small, the model strug- gles to distinguish between preferred and dispreferred outputs, This of- leads trivial solutions-such as degrading dispreferred samples to minimize the coss-which ultimately harms both generation fidelity and
Table 4: Ablation Study on 6, which controls the discrimi- nation strength of the diffusion model in the ABC objective. 8-0.025 yields the best performance.
PickScore
HPS f
Aesthetics f
CLIP f
0.005
18.34
24.25
4.55
22.16
0.015
19.97
25.72
4.96
28.79
alignment quality (see the first row of the table). Conversely, when too large, the model enforces an overly strict separation, resulting in high noise prediction errors for dispreferred samples, which negatively impacts generation quality (as shown in the last row). In summary, small 5 limits the discriminative power of the model, weakening alignment, while a large increases prediction error, degrading output quality.
0.025
21.79
27.67
5.65
33.86
0.035
20.08
26.09
29.81
0.055
14.27
19.75
3.32
8.87
Conclusion
In this work, propose method for aligning text-to-image diffusion models with human preferences through classification. We begin by reformulating the alignment task as a classification problem, showing that optimizing the Diffusion-DPo loss effectively minimizes the AM-Softmax loss, and demonstrating that achieving ideal alignment requires the diffusion model to be discriminative. Building on this insight, we introduce the Alignment by Circle (ABC) loss guide diffusion models toward human- buman-aligned outputs, From the classification perspective, we identify that human preference datasets are inherently semi-supervised and propose data augmentation strategy convert them into fully supervised datasets for more stable ABC training. Experimental results show that our method outperforms previous approaches in human preference alignment, highlighting the effectiveness of alignment by classification for fine-tuning diffusion-baste poxt-to-image models.
7 Limitation
In this paper, we reveal the connection between discriminative ability and alignment performance through Theorem I [and 2 However, this connection remains primarily qualitative. A promising direc- tion for future research is to establish quantitative relationship between discriminative strength and alignment effectivvness. Moreover, since preference data are inherently noisy, is often challenging to define a clear criterion for determining whether one image is preferred over another. Although we do not explicitly address this issue in the current work, formulation offers potential path forward: by transforming the alignment task into classification problem, we can leverage the extensive literature classification under label noise. Adapting these techniques noisy alignment scenarios may provide a principled solution, which we leave for future investigation.
Acknowledgment
This work was supported by the National Natural Science Foundation of China (62372237.62332010 and the Major Science and Technology Projects in Jiangsu Province under Grant BG2024042.
10

---

References
[1] Amirabbas Afzali, Boraa khodabandch, Ali Rasckh, Mahyar JafariNodch, Sepehr Kazemi Ranjbar, and Simon Gottschalk. Aligning Visual Contrastive learning models via Preference Optimization. In International Conference on Learning Representations, 2025.
[2] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training Diffusion Models with Reinforcement Learning. In International Conference on Learning Representations. 2024.
[3] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- 1ood-101-Mil Components with Random Forests. In European Conference on Computer Vision, 2014.
- Mining Discriminative
[4] Stanley Chan. Tutorial on Diffusion Models for Imaging and Vision. Foundations and Trends@ in Computer Graphics and Vision, 2024.
[5] Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Diffusion Models are Certifiably Robust Classifiers. In Advances in Neural Information Processing Systems, 2024.
[6] Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust Classification via Single Diffusion Model. In International Conference Machine Learning, 2024.
[7] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a Similarity Metric Discrimina- tively, with Application to Face Verification. In Conference on Computer Vision and Pattern Recognition, 2005.
[8] Kevin Clark and Priyank Jaini. Text-to- Image Diffusion Models are Zero Shot Classifiers, In Advances Neural Information Processing Systems, 2023.
(9] Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly Fine-Tuning Diffusion Models Differentiable Rewards. In International Conference Learning Representations, 2024.
[10] Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In International Conference Artificial Intelligence and Statistics, 2011.
[11] Meihua Dang. Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized Preference Fine-tuning of Diffusion Models. arXiv, 2025.
[12] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive Angular Margin Loss for Deep Face Recognition. In Conference Computer Vision and Pattern Recognition, 2019.
[13] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model Alignment as Prospect Theoretic Optimization. In International Conference on Machine Learning, 2024.
[14] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: Reinforcement Learning for Fine-tuning Text-to-lmage Diffusion Models. In Advances in Neural Information Processing Systems, 2023.
[15] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality Reduction by Learning an Invariant Mapping. In Conference Computer Vision and Pattern Recognition, 2006.
[16] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou CARD: Classification and Regression Diffusion Models. In Advances in Neural Information Processing Systems, 2022.
[17] Xuehai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Basu, William Yang Wang, and Xin Eric Wang. Discffusion: Discriminative Diffusion Models Few-shot Vision and Language Learners. Transactions Machine Learning Research, 2024.
11

---

[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.
[19] Elad Hoffer and Nir Ailon. Deep metric learning using Triplet Network. International Workshop on Similarity Simiearit-Baseed Pattern Recognition, 2015.
[20] Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thoree, and Jongheon Jeong, Margin- aware Preference Optimization for Aligning Diffusion Models without Reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2025.
[21] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. In Advances Neural Information Processing Systems, 2021.
[22] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A Survey of Reinforcement Learning from Human Feedback. arXiv, 2024.
[23] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. In Advances in Neural Information Prucessing Systems, 2023.
[24] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar html, 2010.
[25] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your Diffusion Model is Secretly a Zero-Shot Classifier. In International Conference on Computer Vision, pages 2206-2217, October 2023.
[26] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing Diffusion Models by Optimizing Human Utility. In Advances in Neural Information Processing Systems, 2024.
[27] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic Post-Training Diffusion Models from Generic Preferences with Step- by-step Preference Optimization. In Conference Computer Vision and Pattern Recognition, 2025.
[28] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-Margin Softmax Loss for Convolutional Neural Networks. In International Conference on Machine Learning, 2016.
[29] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. SphereFace: Deep Hypersphere Embedding for Face Recognition. In Conference on Computer Vision and Pattern Recognition, 2017.
[30] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In arXiv, 2017.
[31] Radek Mackowiak, Lynton Ardizzone, Ulirich Kothe, and Carsten Rother. Generative Classifiers as Basis for Trustworthy Image Classification. In Conference on Computer Vision and Pattern Recognition, 2021.
[32] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Grained Visual Classification of Aircraft. arXiv, 2013.
[33] Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Number of Classes. In Indian Conference Computer Vision, Graphics & Image Processing, 2008.
[34] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep Metric Learning via Lifted Structured Feature Embedding. In Conference on Computer Vision and Putlern Recognition, 2016.
[35] OpenAl. Prompts for Datasets, Github, 2021.
[36] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and Dogs. Conference on Computer Vision and Pattern Recognition, 2012.
12

---

[37] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High- Resolussouttion Image Synthesis. In International Conference on Learning Representations, 2024.
[38] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning Text- to- lo-lmage Diffusion Models with Reward Backpropagation. arXiv, 2024.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ila Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, 2021.
[40] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In Advances in Neural Information Processing Systems, 2023.
[41] Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained Softmax Loss for Discriminative Face Verification. arXiv, 2017.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- Resolution Image Synthesis With Latent Diffusion Models. In Conference on Computer Vision and Pattern Recognition, pages 10684-10695, June 2022.
[43] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A Unified Embedding for Face Recognition and Clustering. In Conference on Computer Vision and Pattern Recognition, 2015.
[44] Christoph Schuhmann et al. Laion-aesthetics v2+ dataset. chr Chrrisoppsschhmannn/St htwp:n/iiimmmooiimeeeeeee httpp//ccccaaaaaoooooooeeeeeeeee prepiicorreectoooorrrr 2022. Accessed: 2025-05-12 12.
com/
[45] Noam Shazeer and Mitchell Stem. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In International Conference on Machine Learning, 2018.
[46] Kihyuk Sohn. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In Advances in Neural Information Processing Systems, 2016.
[47] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle Loss: A Unified Perspective of Pair Similarity Optimization. In Conference on Computer Vision and Pattern Recognition, 2020.
[48] Dipesh Tamboli, Souradip Chakraborty, Aditya Malusare, Biplab Banerjee, Amrit Singh Bedi, and Vaneet Aggarwal. BalancedDPO: Adaptive Multi-Metric Alignment. arXiv, 2025.
[49] Iika Ulusoy and Christopher M. Bishop. Comparison of Generative and Discriminative Tech- niques for Object Detection and Classification. In Toward Category-Level Object Recognition. 2006.
[50] Evgeniya Ustinova and Victor S. Lempitsky. Learning Deep Embeddings with Histogram Loss. In Advances in Neural Information Processing Systems, 2016.
[51] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. In Conference on Computer Vision and Pattern Recognition, 2024.
[52] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L.2 Hypersphere Embedding for Face Verification. In ACM International Conference on Multimedia, 2017.
[53] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. CosFace: Large Margin Cosine Loss for Deep Face Recognition. In Conference on Computer Vision and Pattern Recognition, 2018
[54] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep Metrie Learning with Angular Loss. In International Conference on Computer Vision, 2017.
13

---

[55] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-Similanne Loss with General Pair Weighting for Deep Metric Learning. In Conference on Computer Vision and Pattern Recognition, 2019.
[56] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified Reward Model for Multimodal Understanding and Generation. arXiv, 2025.
[57] Chao-Yure Wu, R Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling Matters in Deep Embedding Learning. In International Conference Computer Vision, 2017.
[58] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Texe-to-Image Synthesis. arXiv, 2023.
[59] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. IwageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances Neural Information Processing Systems, 2023.
[60] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model. In Conference on Computer Vision and Pattern Recognition, 2024.
[61] Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A Dense Reward View on Aligning Text-to- Image Diffusion with Preference. In International Conference on Machine Learning, 2024.
[62] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-lmage Generation. Transactions Machine Learning Research, 2022.
[63] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-Play Fine-tuning Diffusion Models for Text-to-image Generation. In Advances in Neural Information Processing Systems, 2024.
[64] Chenyu Zheng, Guoqiang Wu, Fan Bao, Yue Cao, Chongxuan Li, and Jun Zhu, Revisiting Diseriminative vs. Generative Classifiers: Theory and Implications. In International Conference on Machine Learning, 2023.
[65] Roland S. Zimmermann, Lukas Schott, Yang Song, Benjamin A. Dunn, and David A. Klindt. Score-Bare Based Generative Classifiers. arXiv, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: ABC assumes access an ideal reference model perfectly aligned with human intent and reformulates alignment as a classification problem. (see Abstract and the fourth paragraph of the introduction.)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a discussion of limitations in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: All assumptions and derivations are presented in Section 4 and formally proven in the supplementary material, including Theorem and Theorem 2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5.1 provides full details training procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: We are currently reorganizing and simplifying the codebase for better clarity and usability. Although the full implementation is not yet publicly available, we intend to release on GitHub after the paper decision.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
ssiocoolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 provides full details of training parameter settings. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not have error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the training hardware setup at the beginning of the experiments section, which also applies to the inference and evaluation stages.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps://neurrs wwtperiips.c.pppppeeeee httpp////..tttttttccsseeeeeeeeea
Answer: [Yes]
Justification:
We use only publicly available datasets and open-source models.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We propose more effective alignment method which may benefit the broader research and application generative models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer:[NA]
The paper does not release any models or datasets that pose potential misuse risks.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available models (e.g., SD 1.5, SDXL) and datasets with proper citation and license attribution (e.g., HPS dataset, PartiPrompts dataset).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We conducted human evaluation compare generated images from different methods in figure 2. Annotators were volunteer graduate students who were informed of the task and consented participate without compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve human subjects research. The human annotations were limited to scoring the quality of generated images, without collecting any personal or sensitive data.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No large language models were used as part of the core methodology this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) aatto:://eaarrrpccc/////ofeeeeee hctpa///ooooeeeeeeee//////////. for what should or should not be described.
21


Reviews:
[{"review_id": "LdU2hzYa7U", "reviewer": "Reviewer_qFcD", "summary": "This paper proposes a novel loss function for aligning text-to-image diffusion models with human preferences. The authors hypothesize that conventional DPO-type loss functions are limited due to their reliance on a reference model. Since the reference model itself is not aligned with human preferences, comparisons against it may lead to sub-optimal solutions. To address this, the authors introduce a new loss function called ABC (Alignment by Classification) loss, which does not require a reference model. ABC loss resembles a contrastive loss that directly compares positive and negative pairs. The authors first reformulate DPO with an ideal reference model as a classification problem within the diffusion framework and argue that alignment performance depends on the model’s discriminative capability. Based on this insight, they design the ABC loss. Experimental results demonstrate that the proposed approach achieves better alignment with human preferences compared to existing loss functions.", "questions": "please check the weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros\n\n1. The paper proposes a novel loss function for aligning text-to-image diffusion models with human preferences, and it outperforms conventional loss functions.\n\n2. The semi-supervised learning technique introduced in Section 4.2 is interesting and appears effective.\n\n3. The authors provide various qualitative examples and compare their method with recent approaches.\n\nCons\n\n1. The paper is not easy to read, particularly the theorem sections. It would be helpful if the authors could provide informal interpretations or explanations in simpler terms to improve accessibility.\n\n2. Due to the large number of notations, the paper is difficult to follow. Including a table summarizing all notations and their meanings would greatly enhance readability.\n\n3. Additionally, comparisons of training time and analyses of the effect of dataset size are important. More extensive ablation studies exploring different aspects could further strengthen the paper.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns, and I will maintain my positive score. However, my confidence in the evaluation remains low.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each  concerns in the following.\n\n---\n**Q: The paper is not easy to read, particularly the theorem sections. It would be helpful if the authors could provide informal interpretations or explanations in simpler terms to improve accessibility.**\n\nA: We apologize that the theorem sections may have made the paper difficult to follow. We briefly interpret them here. **Theorem 1** proves that the AM-Softmax loss is upper bounded by the Diffusion-DPO loss. In other words, minimizing the Diffusion-DPO loss for better alignment will also reduce the AM-Softmax loss, leading to improved classification performance. Simply put, **better alignment leads to better classification**. Conversely, **Theorem 2** shows that, under certain conditions, improved classification leads to better alignment. Together, these two theorems reveal a strong connection between classification and alignment, forming the theoretical foundation of our approach, which replaces the DPO loss with the ABC loss for alignment tasks.  We first establish a connection between alignment and classification. This idea is conceptually similar to the ICLR 2025 paper \"On a Connection Between Imitation Learning and RLHF,\" which draws a connection between Reinforcement Learning from Human Feedback (RLHF) and Imitation Learning.\n\n---\n**Q: Due to the large number of notations, the paper is difficult to follow. Including a table summarizing all notations and their meanings would greatly enhance readability.**\n\nA: We apologize for the oversight in summarizing the notations. Below is a table that summarizes all the notations used in the paper:\n\n| Symbol        | Description   |\n| :--------  | :-----  |\n| $x_0$ |Clean image sampled from real data distribution $q(x_0)$. |\n| $x_y^±$ |Clean image $x_0$ aligned ($+$) or misaligned ($-$) with prompt $y$ in preference pairs. |\n| $x^±_{t;y}$ |Noisy version of  $x_y^±$ at timestep $t$. |   \n| $x_{y^±}$ |The same image $x$ conditioned on aligned ($+$) and misaligned ($-$) prompts $y^±$. |   \n| $\\epsilon$|Ground-truth noise sampled from $\\mathcal{N}(0, \\mathbf{I})$, added during forward diffusion. |\n| $\\epsilon_\\theta(x,t)$|Noise predicted by the model $\\epsilon_\\theta$ at timestep $t$, given noisy input $x$. |\n| $\\epsilon_\\mathrm{ali}(x,y,t)$ | Noise prediction from the ideal alignment model for input image $x$, prompt $y$ and timestep $t$. |\n| $\\epsilon_\\mathrm{ref}(x,y,t)$ |Noise predicted by the reference diffusion model for input image $x$, prompt $y$ and timestep $t$. |\n| $\\epsilon_\\mathrm{opt}(x,y,t)$ |Optimal noise prediction as a weighted average over clean images, obtained via marginalization of $p(x \\mid y)$. |\n| $s_{\\theta}(x,y)$ | Score predicted by the traing model for image sample $x$ and prompt $y$ |\n| $s_\\mathrm{ref}(x,y)$ |Score predicted by the reference model for image sample $x$ and prompt $y$  |\n| $\\delta$ | Margin used to enforce separation between aligned ($+$) and misaligned ($-$) scores. |\n| $\\Delta^±_{y}$ |Margin offsets applied to the expected scores of aligned ($+$) and misaligned ($-$) images with prompt ${y}$. |\n|$O^±_y$|Ideal score targets for aligned ($+$) and misaligned ($-$) images with prompt ${y}$.|\n| $\\eta^±_{y}$ | Scaling factors that modulate the contributions of aligned ($+$) and misaligned ($-$) images with prompt ${y}$. |\n| $\\iota({x}^-_y, {x}^+_y, {y})$ | Score difference in ABC loss between aligned ($+$) and misaligned ($-$) images with prompt ${y}$. |\n\n\nPS：\n\n1. $x_0$ can refer to any clean image, while $x_y^+$, $x_y^-$ refer specifically to images aligned/misaligned with a prompt $y$.\n2. For simplicity, the timestep $t$ is sometimes omitted in notation when it is clear from context, eg. $\\epsilon_\\mathrm{ali}(x,y)$.\n3. For simplicity, the prompt $y$ is sometimes omitted in notation , eg. $\\iota({x}^-, {x}^+, {y})$.\n\n---\n\n**Q:  Comparisons of training time and analyses of the effect of dataset size are important. More extensive ablation studies exploring different aspects could further strengthen the paper.** \n\nA: Thank you for the suggestion. We believe it will significantly improve our paper. Due to the page limitations of NeurIPS, we initially restricted the training dataset to Pick-a-Pic v2, published in \"Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation,\" which is commonly adopted by current diffusion alignment methods. Additionally, given the current research focus on designing various alignment losses, the training overhead for these methods is minimal. As a result, the training time for different alignment methods remains nearly the same for any given stable diffusion model. This helps explain why we did not include an ablation study comparing training times or analyzing the effect of dataset size.\n\n----\n\n**We sincerely apologize for the inconvenience.  Following is the supplementary response for Reviewer RW4E, provided here due to the rebuttal length limitation.**\n\n----\n\nQ: How would your approach perform with more natural prompts than the ones you used in your evaluation? Would it be possible to consider using T2I-CompBench++ for your evaluation?\n\nA: Following the official setup, we tested on 8,000 prompts in total, covering: attribute binding, relationships, numeracy, and complex compositions. We report the results in the tables below. Higher scores indicate better alignment with the intended composition.\n\n| Model      | Color (B-VQA) |Shape (B-VQA) |Texture (B-VQA) | Numeracy (UniDet) | 2D-Spatial (UniDet) | 3D-Spatial (UniDet) | Non-Spatial (CLIP) | Complex (3-in-1) |\n| ---------- | ---------- | ---------- | ------------ | -------- | ---------- | ---------- | ----------- | ------- |\n| SD1.5-Base | 0.3811     | 0.3395     | 0.4192       | 0.4436   | 0.1460     | 0.2912     | 0.3092      | 0.3002  |\n| SD1.5-DPO  | 0.3943     | 0.3440     | 0.4374       | 0.4523   | 0.1627     | 0.3090     | 0.3091      | 0.3032  |\n| SD1.5-SPO  | 0.4030     | 0.4001     | 0.4152       | 0.4461   | 0.1471     | 0.2958     | 0.3010      | 0.3131  |\n| SD1.5-KTO  | 0.4645     | 0.3815     | 0.4730       | 0.4618   | 0.1919     | 0.3318     | 0.3104      | 0.3514  |\n| SD1.5-ABC  | 0.4647     | 0.4005     | 0.4751       | 0.4570   | 0.1895     | 0.3324     | 0.3106      | 0.3587  |\n|  |  |  |  |  |  |  |  |  |\n| SDXL-Base | 0.5708     | 0.4880     | 0.5600       | 0.5591   | 0.1949     | 0.3551     | 0.3065      | 0.4383  |\n| SDXL-DPO  | 0.6586     | 0.5358     | 0.6521       | 0.5300   | 0.2376     | 0.3668     | 0.3116      | 0.4923  |\n| SDXL-SPO  | 0.6431     | 0.5200     | 0.6496       | 0.5765   | 0.2298     | 0.3513     | 0.3031      | 0.4424  |\n| SDXL-MAPO | 0.6682     | 0.5104     | 0.5650       | 0.5189   | 0.1700     | 0.3507     | 0.3136      | 0.4401  |\n| SDXL-ABC  | 0.6708     | 0.5450     | 0.6866       | 0.5623   | 0.2401     | 0.3697     | 0.3154      | 0.5051  |"}, {"review_id": "VCaFRyCX7M", "reviewer": "Reviewer_PucN", "summary": "This work proposes a novel connection between preference alignment and classification, i.e. being able to discriminate between preferred and unpreferred samples, and presents an alignment objective for aligning text-to-image diffusion models. The objective extends the Circle Loss objective into the setting of text-to-image diffusion models where the score, provided by the diffusion model, is used to approximate p(y|x). Unlike related approaches like Diffusion-DPO (which ABC, their method, is connected to), the provided objective does not require a reference model during training, and thus provides some memory/compute benefits as well. Results show that ABC outperforms prior alignment approaches according to existing reward models and human annotators.", "questions": "- Can the authors report the statistical significance of the quantitative preference alignment experiments? \n\n- For the quantitative preference experiment using off-the-shelf reward models, can the authors report the average score (within some confidence interval) to help quantify the improvement of ABC over other methods?\n\n- Can the authors provide a quantitative ablation studying the effect of the data augmentation strategy?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Proposes novel connection between classification and alignment, that is justified both theoretically and empirically. \n\n- Does not require a reference model during training, saving memory & compute\n\n- Strong, comprehensive results on automated evaluations (e.g. using HPS reward model as judge) and via user study. \n\n**Weaknesses**\n\n- Importance of data augmentation is unclear. The authors note that the training loss may diverge without the use of data augmentation. However, the effect of this augmentation is not well documented, e.g. there is no quantitative ablation on the effect of the augmentation.\n\n- Strong assumptions regarding the nature of preference data, e.g. each text prompt corresponds to one aligned image, or the data augmentation strategy that prefaces the losing image’s prompt with “The image that aligns less with human preferences”. Real-world preference data is quite noisy and its unclear why a formulation dependent on such strong assumptions is able to outperform other methods like Diffusion-KTO, for instance, which have some noise-resistant properties. Additionally, the data augmentation strategy is questionable as even though an image is less-preferred in a single triplet it may generally be more preferred on aggregate. It is unclear why this strategy works with noisy real-world data. \n\n- Statistical significance of results. The authors report win-rates in the quantitative preference alignment experiments. However, win-rate can be quite sensitive to the choice of seed, noise, etc, as in order for a method to win it just needs to at least very slightly outperform the other per the reward model score. To fully validate the performance of ABC, statistical significance of these results should be provided, e.g. via confidence intervals. \n\n- Lack of user study details. The details of the user study are largely omitted in the paper, outside of 2 lines in the checklist. Was this a blind user study? How many annotators were assigned per comparison?\n\n- [minor editorial] Line 75 is missing a citation for triplet loss.", "was_revised": true, "final_justification": "The paper presents an interesting alternative to T2I alignment by formulating it is a classification problem. The results provided in the rebuttal are statistically significant (spec. for SDXL) and show the benefit of this approach. While the loss function is not novel, I believe it's application to this new domain and its performance merits acceptance.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each  concerns in the following.\n\n---\n**Q: Importance of data augmentation is unclear.**\n\nA: We apologize for the confusion regarding data augmentation in the preference data. Each dataset item is a triplet $(\\mathrm{y}, \\pmb{x}^+\\_{\\mathrm{y}}, \\pmb{x}^-\\_{\\mathrm{y}})$, where $\\mathrm{y}$ denotes the prompt, and $\\pmb{x}^+\\_{\\mathrm{y}}$ and $\\pmb{x}^-\\_{\\mathrm{y}}$ represent the positive and negative images corresponding to the prompt $\\mathrm{y}$, respectively.\n\nIn this paper, we propose replacing the alignment loss (DPO loss) with a classification loss (ABC loss). Since the classification loss requires a label for each image, we need to transform the triplet $(\\mathrm{y}, \\pmb{x}^+\\_{\\mathrm{y}}, \\pmb{x}^-\\_{\\mathrm{y}})$, where the two images share the same label $\\mathrm{y}$, into a form where each image has its own label. Specifically, we define $\\mathrm{y}^+$ as the original prompt $\\mathrm{y}$ and construct $\\mathrm{y}^-$ by appending “The image that aligns less with human preferences” to $\\mathrm{y}$. This reformulation transforms each preference tuple $(\\mathrm{y}, \\pmb{x}^+_{\\mathrm{y}}, \\pmb{x}^-\\_{\\mathrm{y}})$ into two supervised examples: $(\\mathrm{y}^+, \\pmb{x}^+\\_{\\mathrm{y}})$ and $(\\mathrm{y}^-, \\pmb{x}^-\\_{\\mathrm{y}})$. This is clarified in lines 173 to 176 of the paper.\n\n---\n**Q: Real-world preference data is quite noisy and its unclear why a formulation dependent on such strong assumptions is able to outperform other methods like Diffusion-KTO. Additionally, the data augmentation strategy is questionable as even though an image is less-preferred in a single triplet it may generally be more preferred on aggregate. It is unclear why this strategy works with noisy real-world data.**\n\nA: Yes, this is a great question. Preference data is inherently noisy, making it difficult to establish a clear principle for determining whether one image is preferred over another. Current diffusion alignment methods do not take this issue into account. To address this problem, we believe it is necessary to build a new dataset that provides preference comparison results across different dimensions. However, current alignment methods are typically trained on datasets like Pick-a-Pic, which assumes that one image is better than another with respect to the given prompt. \n\nWe do not directly address this issue in the current paper, but we present a promising approach to tackle it. Specifically, we show that the alignment task can be transformed into a classification problem, and classification in noisy environments has been extensively studied. It may therefore be possible to adapt existing methods from noisy classification to address the noisy alignment problem.\n\nAs for the noise-resistant properties of our method, they are easy to interpret. The well-known ImageNet classification dataset contains incorrect labels, yet research has shown that simply using cross-entropy loss to train the network still yields state-of-the-art performance, without the need for special handling of mislabeled data. We attribute this noise-resistant property to the classification loss. In this paper, we transform the preference data into a classification form. As a result, even if the data is noisy, the network retains the ability to resist the effects of noise.\n\n---\n**Q: Statistical significance of results. To fully validate the performance of ABC, statistical significance of these results should be provided, e.g. via confidence intervals.**\n\nA: Thank you for raising this important point. We agree that providing statistical significance is important for validating the reported win rates. While our experiments followed the standard protocols used in prior work, we now include win rates along with variance intervals.\n\nSpecifically, we compute these intervals by discarding the top 5% of deviations from the mean, resulting in a range that captures 95% of the scores. This provides a robust estimate of variability and offers a more informative view of our method’s performance. \n\n**Table 1.** Win rate  on PartiPrompts (P) and HPS (H) benchmarks for SD1.5 and SDXL-based models.\n\n| Model      | PickScore (P) | HPS (P)       | Aesth. (P)    | CLIP (P)      | PickScore (H) | HPS (H)       | Aesth. (H)    | CLIP (H)      |\n| ---------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| SD1.5-Base | 60.02 ± 2.40% | 81.51 ± 2.21% | 74.27 ± 2.14% | 59.72 ± 2.45% | 74.83 ± 3.10% | 85.75 ± 2.87% | 68.84 ± 3.31% | 59.65 ± 3.53% |\n| SD1.5-DPOK  |57.15 ± 2.45% | 68.19 ± 2.21% | 62.51 ± 2.42% | 55.28 ± 2.45% | 52.17 ± 3.53% | 68.16 ± 3.46% | 65.24 ± 3.49% | 60.95 ± 3.56% |\n| SD1.5-D3PO  | 58.42 ± 2.43% | 72.75 ± 2.30% | 65.72 ± 2.40% | 53.61 ± 2.45% | 52.33± 3.52% | 73.25 ± 3.46% | 63.89 ± 3.51% | 54.14 ± 3.62%|\n| SD1.5-DPO  | 55.85 ± 2.44% | 73.02 ± 2.38% | 64.90 ± 2.34% | 44.97 ± 2.43% | 53.46 ± 3.51% | 71.50 ± 3.46% | 64.19 ± 3.56% | 52.06 ± 3.57% |\n| SD1.5-SPO  | 51.16 ± 2.41% | 61.59 ± 2.41% | 47.60 ± 2.45% | 60.02 ± 2.36% | 45.35 ± 3.57% | 54.99 ± 3.53% | 38.08 ± 3.47% | 64.83 ± 3.13% |\n| SD1.5-KTO  | 57.77 ± 2.41% | 44.72 ± 2.38% | 53.90 ± 2.44% | 47.22 ± 2.45% | 52.28 ± 3.52% | 42.88 ± 3.48% | 52.86 ± 3.57% | 53.93 ± 3.55% |\n|            |               |               |               |               |               |               |               |               |\n| SDXL-Base  | 74.38 ± 1.84% | 79.26 ± 1.97% | 80.20 ± 1.56% | 52.46 ± 2.04% | 79.35 ± 2.23% | 70.17 ± 2.58% | 72.28 ± 2.86% | 60.38 ± 3.26% |\n| SDXL-DPO   | 73.22 ± 2.18% | 72.50 ± 2.33% | 68.25 ± 1.39% | 50.51 ± 1.88% | 77.26 ± 3.06% | 69.54 ± 3.42% | 70.19 ± 2.39% | 57.06 ± 3.07% |\n| SDXL-SPO   | 52.49 ± 2.34% | 40.31 ± 2.44% | 59.93 ± 2.41% | 55.53 ± 2.44% | 51.16 ± 3.34% | 52.41 ± 3.57% | 46.78 ± 3.53% | 59.87 ± 3.58% |\n| SDXL-MAPO  | 65.35 ± 1.81% | 81.17 ± 2.07% | 72.10 ± 1.82% | 46.97 ± 2.06% | 68.55 ± 2.37% | 64.89 ± 3.11% | 68.18 ± 2.96% | 51.14 ± 3.19% |\n\n\n---\n**Q: Lack of user study details. The details of the user study are largely omitted in the paper, outside of 2 lines in the checklist. Was this a blind user study? How many annotators were assigned per comparison?**\n\nA: Thank you for your question. We conducted a user study to compare the proposed ABC method with several baseline approaches. Specifically, we randomly sampled 100 prompts from the PartiPrompts dataset and another 100 prompts from the HPSv2 benchmark. For each prompt, we generated five images using five different methods.\n\nParticipants were shown five images per prompt (one from each method) and asked to answer three questions:\n\n1. Which image is your overall preferred choice?\n2. Which image is more visually attractive?\n3. Which image better matches the text description?\n\nTo minimize position bias, the order of images was randomized for each prompt. Each method’s final score was computed as a weighted sum of its win rates under the three criteria, with weights of 30% for general preference, 30% for visual appeal, and 40% for prompt alignment.\n\nThe study was conducted as a blind evaluation. Annotators were not informed about which method generated each image. We recruited participants from our research group, comprising approximately 100 students, and collected a total of 82 valid responses.\n\nWe hope this clarification provides a more complete picture of our user study design and evaluation protocol.\n\n---\n**Q: Can the authors report the average score (within some confidence interval) to help quantify the improvement of ABC over other methods?**\n\nA: We thank the reviewer for the valuable suggestion. In addition to win-rate comparisons, we now report the absolute scores along with the setting in the Table 1. \n\n**Table 2.** Absolute scores  on the PartiPrompts (P) and HPS (H) benchmarks for SD1.5 and SDXL-based models.\n\n|Model|PickScore (P)|HPS (P)| Aesth. (P)|CLIP (P)|PickScore (H)|HPS (H)|Aesth. (H)|CLIP (H)|\n|-|-|-|-|-|-|-|-|-|\n|SD1.5-Base|21.25±2.02|26.98±2.85|5.29±1.17|29.57±9.71|21.17±2.45|27.61±3.10|5.45±1.01|35.73±7.38|\n|SD1.5-DPOK|21.57±2.21|27.21±3.03|5.59±1.21|30.01±9.93|21.80±2.42|28.15±3.17|5.60±1.04|36.75±8.12|\n|SD1.5-D3PO|21.41±2.03|27.08±2.97|5.42±1.13|29.87±8.66|21.76±2.58|28.20±3.23|5.54±1.07|36.71±7.87|\n|SD1.5-DPO|21.49±2.19|27.16±3.13|5.36±1.08|29.80±9.46|21.71±2.41|28.23±3.35|5.59±0.99|36.66±7.95|\n|SD1.5-SPO|21.53±2.33|27.33±3.74|5.89±1.04|28.13±9.23|21.99±2.76|28.53±3.69|5.96±1.10|33.14±9.15|\n|SD1.5-KTO|21.46±1.96|27.70±3.21|5.62±0.99|30.78±8.09|21.79±2.53|28.95±3.12|5.62±0.96|37.01±8.01|\n|SD1.5-ABC|21.85±2.04|27.97±2.85|5.93±1.01|31.07±8.14|21.97±2.45|28.86±3.02|5.72±0.87|37.18±7.72|\n||||||||||\n|SDXL-Base|22.76±2.45|28.49±3.59|5.86±1.05|35.76±9.66|23.26±2.50|29.38±3.59|6.08±1.03|37.24±6.74|\n|SDXL-DPO|22.94±2.37|28.93±3.52|6.01±0.98|36.01±8.73|23.59±2.65|29.86±3.40|6.14±1.00|38.34±5.98|\n|SDXL-SPO|23.56±2.64|29.12±3.52|6.26±0.92|33.82±9.84|23.76±2.70|30.30±3.18|6.48±0.86|37.62±7.14|\n|SDXL-MAPO|22.82±2.40|28.62±3.62|5.98±1.04|36.58±9.39|23.60±2.53|29.92±3.52|6.19±0.92|38.61±7.17|\n|SDXL-ABC|23.79±2.27|29.42±3.29|6.35±0.89|36.81±8.51|24.39±2.38|30.67±3.08|6.54±0.86|38.97±6.02|"}, {"review_id": "AQyqtrQfFM", "reviewer": "Reviewer_RW4E", "summary": "This work deals with the alignment of text-to-image generative models. In short, diffusion-based generative models (and other generative models such as flow matching, rectified flows, and so on, by the way) suffer from a number of problems when generating an image, given a prompt. Those issues range from catastrophic neglecting (objects present in the prompt are not generated as part of the final image), attribute binding (textures, colors, and other image characteristics) are not properly respected, spatial binding (positioning of objects as defined by the prompt) is not respected, numbering (e.g. object counts) is not correct, and many more.\n\nThe authors build on fine-tuning methods that were originally devised for LLM alignment, and that have been also recently applied to diffusion models, to derive a new approach that relates alignment to classification accuracy. They amend the existing connection between alignment and semi-supervised learning by introducing a simple form of data augmentation, transforming the problem into a supervised classification task. Leveraging insights on bounds on the AM-Softmax loss and the Diffusion-DPO loss, as well as results indicating that ideal alignment requires the diffusion model to be discriminative, they propose a new loss that is an instance of the Circle Loss, which is used to fine-tune the (conditional) score network based on positive and negative examples, with their associated labels, plus additional parameters to improve the discriminative power of the diffusion model.\n\nExperiments using standard metrics and a user study indicate that the proposed ABC-loss works well and outperforms existing variants of DPO-based alignment methods. Additional experiments serve the purpose of validating the theoretical analysis, and study the impact of one important hyper-parameter of the proposed method.", "questions": "* If recent autoregressive generative models are becoming the new standard for image generation, how relevant do you consider alignment approaches for diffusion models in general, and DPO-based fine-tuning approaches in particular, will be?\n\n* What are the key advantages of DPO-based alignment methods such as ABC when compared to other fine-tuning methods from the literature (e.g., you cite DPOK as reference [14], but do not even refer to it in the main), or other inference-time methods?\n\n* How would your approach perform with more natural prompts than the ones you used in your evaluation? Would it be possible to consider using T2I-CompBench++ (reference [2] above) for your evaluation? \n\n* Could you report absolute metrics rather than percentage of wins in Table 1?\n\n* In sec 4.1, line 116, you rely on a pretty strong assumption. I realize your work is not the only one requiring such an assumption, but it would be interesting to hear your opinion on how a DPO-based framework such as ABC could cope with the more realistic case of multiple images being aligned to a single prompt. Consider a realistic user prompt: natural language is ambiguous, subject to interpretation, thus there might be multiple generated images that could be considered as aligned to it\n\n* In Theorem 2, lines 142-143, you need to assume $p(y)$ and $p(x)$ to be uniform distributions. Also this assumption is pretty strong. Can you comment on what would it be necessary to do if we relax such an assumption?", "limitations": "Yes, limitations are discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths:\n\n  * The authors propose a simple solution to a difficult problem, building on existing work on model alignment through direct preference optimization, through some valid insights connecting generative modeling and classification\n\n  * The article is well written and easy to follow\n\n  * The experimental validation goes beyond a quantitative/qualitative reporting of alignment results, and provides additional insights on the benefits of aligned models when used as classifiers, as well as an ablation study on an important hyper-parameter\n\n* Weaknesses:\n\n  * State-of-the-art on text-to-image alignment is assessed superficially. This works only focuses on DPO-based variants, and neglects many existing work in the domain. On the one hand, we have inference-time methods, which attempt to align T2I models by steering the generative paths through optimization of the attention layers of the score network, such as [1], and many others, on the other, we have fine-tuning methods that do not belong to the DPO family, such as [2] and [3], [4], for example.\n\n  * The quantitative analysis of the performance of the proposed method in Table 1 relies on a protocol that hinders a proper assessment of the absolute performance of compared methods.\n\n  * The metrics used in Table 1 are only a small subset of the many that can be used to assess alignment. See for example [2].\n\n  * The proposed method is only compared to DPO-based alternatives, but not to other existing methods that use a different approach to alignment, such as [1-4] (and many more)\n\n  * The proposed method, in principle, can be applied to other generative models, such as rectified flows. The evaluation is based on “older” models such as SD1.5 and SDXL, whereas the current best practice is to use FLUX or SD3, which use a different generative modeling paradigm (still related to diffusion somehow), and that are known to be more aligned than SD1.5 and SDXL.\n\n  * There is no mention to recent work on autoregressive generative models, which appear to have superior alignment when compared to diffusion models [5, 6].\n\n[1] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or, “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models”, https://arxiv.org/abs/2301.13826\n\n[2] Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, Xihui Liu, “T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation”, https://arxiv.org/abs/2307.06350\n\n[3] Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro Michiardi, “Information Theoretic Text-to-Image Alignment”, https://arxiv.org/abs/2405.20759\n\n[4] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, Kimin Lee, “DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models”, https://arxiv.org/abs/2305.16381\n\n[5] Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, Li Yuan, “GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation”, https://arxiv.org/abs/2504.02782v1\n\n[6] Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, Feng Zhao, “Frequency Autoregressive Image Generation with Continuous Tokens”, https://arxiv.org/abs/2503.05305", "was_revised": true, "final_justification": "I think the authors did a great job in addressing my concerns, and they run additional experiments that broaden the position of their work in the current landscape of T2I alignment methods.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each  concerns in the following.\n\n---\n**Q:  This works only focuses on DPO-based variants, and neglects many existing work in the domain. On the one hand, we have inference-time methods, which attempt to align T2I models by steering the generative paths through optimization of the attention layers of the score network**\n\nA: We believe this concern arises from a misunderstanding of the comparison scope. As discussed in *\"Large Language Models Post-training: Surveying Techniques from Alignment to Reasoning,\"* generative modeling methods can be broadly categorized into **pre-training** and **post-training**. Pre-training focuses on building general-purpose models from large datasets, while post-training adapts these models to specific downstream tasks.\n\nDue to their distinct goals, evaluation strategies for these two categories differ. Pre-training methods are typically compared across architecture types, datasets, or training paradigms. In contrast, post-training methods—like ours—should be evaluated using the same foundation model and training setup, with only the loss objective varying to ensure a fair comparison.\n\nMost works cited by Reviewer RW4E, except [2,3], focus on pre-training. For instance, [6] proposes an autoregressive model as a new foundation model and compares its generative capabilities with other pre-training approaches. By contrast, our ABC loss is a post-training method applied to a fixed foundation model. Analogously, comparing a fine-tuned GPT-3 with GPT-4o would not inform the quality of the fine-tuning method applied to GPT-3.\n\nAdditionally, [2] is an inference-time method without training, and [3] is self-supervised, whereas DPO-based methods—including ours—require supervised training. According to our tests conducted during the rebuttal period, [2] and [3] consistently underperform all DPO-based methods on the same base model. Given their fundamentally different setups, comparing their raw performance to ours is **not meaningful** for assessing the quality of our approach.\n\n---\n**Q: How would your approach perform with more natural prompts than the ones you used in your evaluation? Would it be possible to consider using T2I-CompBench++ for your evaluation?**\n\nA: This is a valuable suggestion. However, implementing it poses several technical challenges. T2I-CompBench++ is specifically designed to test compositional generation—handling multiple objects, attributes, and relationships—which makes it more suited for evaluating foundation models developed through **pre-training**.\n\nIn contrast, **post-training** methods like ours require a well-defined downstream task and a corresponding dataset. The standard procedure involves fine-tuning a pre-trained foundation model using a specific loss and evaluating the resulting checkpoint against either the original model or other post-trained variants.\n\nIn our case, we currently lack a task-specific dataset aligned with the compositional goals of T2I-CompBench++. Therefore, although T2I-CompBench++ is a promising benchmark, it is not directly applicable under our current post-training setup. Currently, we make a comporise which take the checkpoints trained on  Pick-a-Pic to conduct this evaultion. Due to the maximum length limitation, we have included the corresponding data **in our response to Reviewer qFcD.**\n\n---\n**Q: If recent autoregressive generative models are becoming the new standard for image generation, how relevant do you consider alignment approaches for diffusion models in general, and DPO-based fine-tuning approaches in particular, will be?**\n\nA: Both autoregressive generative models and diffusion models are promising approaches for image generation. The DPO (Differentiable Prompt Optimization) method was initially proposed for aligning autoregressive language models, and Diffusion-DPO adapts this technique for use with diffusion models. The primary contribution of our paper is that we are the first to demonstrate that the alignment task can be transformed into a classification task. Reviewer RW4E can verify that this discovery does not depend on the model type. Both autoregressive and diffusion models can benefit from this finding.\n\nSince Reviewer RW4E has a particular interest in the application of the ABC loss to autoregressive generative models, we believe it would be a great idea to prepare a separate paper discussing how to apply the ABC loss to autoregressive models.\n\n---\n**Q: What are the key advantages of DPO-based alignment methods such as ABC when compared to other fine-tuning methods from the literature (e.g., you cite DPOK as reference [14], but do not even refer to it in the main), or other inference-time methods?**\n\nA: Both DPO and SFT methods have demonstrated their value in training large language models. SFT-based methods, such as DPOK, require a reward model to fine-tune the diffusion model. However, training a good reward model requires additional time and dataset resources. In contrast, DPO-based methods bypass the need for a reward model when fine-tuning the diffusion model.\n\nRegarding inference-time methods, the running time for inference is significantly longer compared to SFT and DPO-based methods. In general, these three approaches represent different pathways toward achieving better results, and at this stage, it is difficult to predict which will ultimately prevail. This paper focuses solely on demonstrating that the alignment task can be transformed into a classification task through our ABC loss. We do not address whether DPO-based methods will outperform SFT and inference-based methods, as this is beyond the scope of our current work.\n\n---\n\n**Q: In sec 4.1, line 116, you rely on a pretty strong assumption. I realize your work is not the only one requiring such an assumption, but it would be interesting to hear your opinion on how a DPO-based framework such as ABC could cope with the more realistic case of multiple images being aligned to a single prompt.**\n\nA: Thank you for your question. We acknowledge the limitation of this assumption. However, we could not avoid it in the current environment for two reasons:\n\n1. The current preference datasets are built on this assumption.\n2. The evaluation networks are also designed based on this assumption, as they are trained on the preference dataset.\n\nWithout new preference datasets and evaluation criteria, it is difficult for methods that handle multiple preferences to demonstrate their superiority and gain acceptance in top conferences like NeurIPS.\n\nSince we have established the connection between alignment and classification, the assumption that \"each text prompt corresponds to a single aligned image\" can be interpreted as a multi-class classification task, where each sample is assumed to belong to only one class. In contrast, Reviewer RW4E's preference for a scenario where one prompt corresponds to multiple images can be interpreted as a multi-label classification task, where each sample may belong to multiple classes. In the future, we plan to explore the multi-label classification perspective to address this issue.\n\n---\n\n**Q: In Theorem 2, lines 142-143, this assumption is pretty strong. Can you comment on what would it be necessary to do if we relax such an assumption?**\n\nA: We apologize for any confusion caused to Reviewer RW4E. In our paper, Theorem 2 is used to illustrate that classification performance directly determines alignment performance. To demonstrate this, we assume that we have a total of $N$ text prompts $\\mathrm{y}_i$​, each corresponding to an aligned image $\\pmb{x}\\_{\\mathrm{y}\\_i} $​​. We believe it is reasonable to assume that the prior distribution for both the prompt $\\mathrm{y}_i$​ and the image $\\pmb{x}\\_{\\mathrm{y}\\_i} $​​ follows a uniform distribution. If we were to assume otherwise, we would have to acknowledge that certain labels and images are inherently more special than others. However, we currently do not have a compelling reason to support this assumption.\n\n---\n**Q: The proposed method, in principle, can be applied to other generative models, such as rectified flows. The evaluation is based on “older” models such as SD1.5 and SDXL, whereas the current best practice is to use FLUX or SD3, which use a different generative modeling paradigm (still related to diffusion somehow), and that are known to be more aligned than SD1.5 and SDXL.**\n\nA: Applying DPO-like methods to flow models is not an easy task. Flow models rely on a deterministic generative process based on Ordinary Differential Equations (ODEs), meaning they cannot sample stochastically during inference. In contrast, DPO relies on stochastic sampling to explore the environment, learning by trying different actions and improving based on rewards. This need for stochasticity conflicts with the deterministic nature of flow matching models. To apply DPO to flow models, one would need to conduct an ODE-to-SDE conversion, which transforms a deterministic ODE into an equivalent Stochastic Differential Equation (SDE) that matches the original model’s marginal distribution at all timesteps, thus enabling statistical sampling.\n\nWe recently discovered a concurrent work, \"Flow-GRPO: Training Flow Matching Models via Online RL\", which claims to be the first method to apply GRPO to flow models in order to address this problem. By employing  Flow-GRPO, our approach could potentially be applied to models like FLUX or SD3. However, this paper was released on May 8, 2025, on arXiv, and the code was released on May 15, 2025. As a result, we were not aware of this work before submitting our paper.\n\n---\n**Q: Could you report absolute metrics rather than percentage of wins in Table 1?**\n\nA: Thank you for the suggestion. Due to length limits, we report the absolute metrics in Table 2 of our response to Reviewer PucN, including appended results for DPOK and D3PO."}]

Author rebuttal:

Meta-review: This paper proposes a genuinely new approach to the problem of T2i diffusion model alignment (an important problem with wide applications). It recasts the problem as a certain kind of classification problem, using ideas from circle loss. All reviewers appreciated the novelty and interestingness of this idea. A significant advantage of the method is that a reference model is NOT needed during training which saves a lot of memory, a very significant factor for frontier diffusion models. Objections mainly focused on the robustness of the evaluations, many of which were effectively addressed by the authors with new experiments etc. Overall, a solid work that is likely to influence new ideas and directions.
Decision: Accept (spotlight)