Aligning Diffusion Models by Optimizing Human Utility
Authors: Shufan Li
Konstantinos Kallidromitis
Akash Gokul
Yusuke Kato
Kazuki Kozuka
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-image; diffusion; computer vision;
Primary area: generative_models
TL;DR: We extend the utility maximization framework to the setting of diffusion models and use it to align text-to-image diffusion models with human preferences using only per-image binary preference signals, e.g., likes and dislikes.

Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Unlike previous methods, Diffusion-KTO does not require collecting pairwise preference data nor training a complex reward model. Instead, our objective uses per-image binary feedback signals, e.g. likes or dislikes, to align the model with human preferences. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit improved performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary preference signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Aligning Diffusion Models by Optimizing Human Utility
Shufan Li'+
Konstantinos Kallidromitis
Akash Gokul
Yusuke Kato²
Kazuki Kozuka?
Uniierriity California, Los Angeles
Panasonic AI Research
Salesforce AI Research
+Equal contribution * Work done outside of Salesforce. Correspondence to jacklishufan cs. ucla.eduu
Figure 1: Diffusion-KTV novel framework for aligning text-to-image diffusion models with human preferences using only per-sample binary feedback. Diffusion- KTOO bypasses the need to collect expensive pairwise preference data and avoids training a reward model. As seen above, Diffusion-KTT aligned text-to- image models generate images that better align with human preferences. We display results after fine-tuning Stable Diffusion v1-5 and sampling prompts from HPS v2 [50]. Pick-a-Pic [27], and PartiPrompts [54] datasets.
Abstract
We present Diffusion-KTT KTTO, a novel approach for aligning text- to to-image diffusion models by formulating the alignment objective the maximization of expected human utility. Unlike previous methods, Diffusion-KTT KTO does not require col- lecting pairwise preference data training complex reward model. Instead, our objective uses per-image binary feedback signals, e.g. likes or dislikes, to align the model with human preferences. After fine-tuning using Diffusion-KTO. text-to-image diffusion models exhibit improved performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPOI of human judgment and automatic evaluation metrics such as PickScore [27] and Iewae [52]. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary preference signals and broadens the applicabil-
both in terms
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ity of aligning text- -immage diffusion models with human preferences. Code is
available https////gtttbbooomm/ccchhhhhhht
Introduction
In the rapidly evolving field of generative models, aligning model outputs with human preferences remains a paramount challenge, especially for text-to-image (T2I) models. Large language models (LLMs) have made significant progress in generating text that caters to wide range of human needs, primarily through a two-stage process: first, pretraining on noisy web-scale datasets, then fine-tuning smaller, cerference-speciffe dataset. This fine-tuning process aims to align the generative model's 's outputs with human preferences, without significantly diminishing the capabilities gained from pretraining. Extending this fine-tuning approach to text-to-image models offers the prospect of tailoring image generation to user preferences, a goal that has remained relatively under-explored compared to its counterpart in the language domain.
Recent works have begun to explore aligning text-to-image models with human preferences. These methods either use reward model and reinforcement learning objective [52, 33, 14], or directly fine-tune the T2I model on preference data [48, 53]. However, these methods are restricted to learning from pairwise preference data, which consists of pairs of preferred and unpreferred images generated from the same prompt.
While paired preferences are commonly used in the field, it is not the only type of preference data available. Per-sample feedback a promising alternative to pairwise preference data, as the former is abundantly available on the Internet. Per-sample feedback provides valuable preference signals for aligning models, as it captures information about the users' subjective distribution of desired and undesired generations. For instance, as seen in Fig. 2, given image and its caption, a user can easily say if they like or dislike the image based on criteria such as attention to detail and fidelity to the prompt. While paired preference data provides more information about relative preferences, gathering such data is an expensive and time-consuming process in which annotators must rank images according to their preferences. In contrast, learning from per-sample feedback can utilize the vast amounts of per-sample preference data collected on the web and increases the applicability of aligning models with user preferences at scale. Inspired by these large-scale use cases, we explore how to directly fine-tune T21 models on per-image binary preference data.
To address this gap, we propose Diffusion-KTOO KTO, novel alignment algorithm for T21 models that operates on binary per-sample feedback instead of pairwise preferences. Diffusion- KTO extends the utility maximization framework shown in KTO [18] to the setting of diffusion models. Specifically, KTO bypasses the need to maximize the likelihood from paired preferences and, instead, directly optimizes LLM using utility function that encapsulates the characteristics of human decision- making. While KTO is casy to apply to large language models, we cannot immediately apply to diffusion models it would require sampling across all possible trajectories in the denoising process. Although existing works approximate this likelihood by sampling once through the reverse diffusion process, pack-propagating over all sampling steps is extremely computationally expensive. To overcome this, we present a utility maximization objective that applies to each individual sampling step, circumventing the need for sampling through the entire reverse diffusion process.
Our main contributions are as follows:
We generalize the human utility maximization framework used to align LLMs to the setting of diffusion models (Section 4).
Our method, Diffusion-KTT -TTO, facilitates alignment from per-image binary feedback. Thus, introducing the possibility of learning from human feedback at scale using the abundance of per-sample feedback that has been collected on the Internet.
Through comprehensive evaluations, we demonstrate that generations from Diffusion-KTT aligned models are generally preferred over existing approaches, judged by human evaluators and preference models (Section 5).
In summary, Diffusion- KTO offers a simple yet robust framework for aligning T2I models with human preferences that greatly expands the utility of generative models in real-world applications.

---

Diffusion-DPO/ / RLHF
Diffusion-KTO
BOC
"Man walking in NYC"
"Man walking in NYC"
"Portrait of dog"
Figure 2: Diffusion-KTO aligns text-to-image diffusion models using per-image binary feedback. Existing alignment approaches (Left) are restricted to learning from pairwise preferences. However, Diffusson-Ko -TO (Right) uses per-image preferences which are abundantly available on the Internet. As seen above, the quality of an image can be assessed independent of another generation for the same prompt. More importantly, such per-image preferences provide valuable signals for aligning T2I models, as demonstrated by our results.
2 Related Works
Text-to-Image Generative Models. Text-to-image (T2I) models have demonstrated remarkable success in generating high quality images that maintain high fidelity to the input caption [38, 54, 37, 7, 11, 26, 55, 8). In this work, we focus on diffusion models [43, 44, 24] due to their popularity and open-source availability. While these models are capable of synthesizing complex, high quality images after pretraining, they are generally not well-aligned with the preferences of human users. Thus, they can often generate images with noticeable issues, such as poorly rendered hands and faces. We seek to address these issues by introducing fine-tuning objective that allows text-to-image diffusion models to learn directly from human preference data.
Improving Language Models using Human Feedback. Following web-scale pretraining. large language models are further improved by fine-tuning on a curated set of data (supervised fine-tuning) and then using reinforcement learning to learn from human feedback. Reinforcement learning from human feedback (RLHF) [2. 12, 13], in particular, has been shown to be effective means of aligning these models with user preferences [59, 5, 31, 30, 29, 45, 10, 3, 6. 22]. While this approach has been successful [1, 46], the difficulties in fine-tuning LLM using RLHF [36, 58, 49, 17, 21, 42, 4] has led to the development of alternative fine-tuning objectives [35, 18, 56, 57]. Along these lines, KTO [18] introduces a fine-tuning objective that trains an LLM to maximize the utility of its output according to the Kahneman & Tversky model of human utility [47]. This utility maximization framework does not require pairwise preference data and only needs per-sample binary feedback. In this work, we explore aligning diffusion models given binary feedback data. As a first step in this direction, generalize the utility maximization framework to the setting of diffusion models.
Improving Diffusion Models using Human Feedback. Before the recent developments in aligning T21 models using pairwise preferences, supervised fine-tuning was the popular approach for improving these models. Existing supervised fine-tuning approaches curate dataset using preference models [39, 32], pre-trained image models [41, 8, 16, 51, 50], and/or human experts [15], and fine-tune the model on this dataset. Similarly, many works have explored using reward models fine-tune diffusion models via policy gradient techniques [19, 23, 9, 52, 14, 33, 28, 20] to improve aspects such image-text fidelity. Similar to our work, ReFL [52], DRaFT [14], and AlignProp [33] align T21 diffusion models with human preferences. However, these methods require hack-propagating the reward through the reverse diffusion sampling process, which extremely expensive in memory. As result, these works depend on techniques such as low-rank weight updates [25] and sampling from only a subset of steps the reverse process, thus limiting their ability fine-tune the model. In contrast, the Diffusion-K'TH objective extends to each step in the denoising process, thereby avoiding such memory issues. More broadly, the main drawbacks of these reinforcement learning based approaches are: limited generalization, e.g. closed vocabulary [28, 20), reward hacking [14, 33, 9], and they rely on a potentially biased reward model. Since Diffusion Cirfussio-KKOO trains directly on open- vocabulaaryyyyyoo preference data, we find that it can generalize to an open-vocabulary and avoids issues such as reward hacking. Recently, works such as Diffusion-DPOJ49 and D3PO [53] present extensions of the DPO objective [35] to the setting of diffusion models. Diffusion- KTO shares

---

similarities with Diffusion-DPO and D3PO, as we build upon these works to introduce reward model-free alignment objective. However, unlike these works, Diffusion-KTT does not rely on pairwise preference data and, instead, uses only per-image binary feedback.
3 Background
3.1
Diffusion Models
Denoising Diffusion Probabilistic Models (DDPM) [24] model the image generation process as Markovian process, Given data Zo, the forward process p(=+(==-1) gradually adds noise to an initial image 2g according a variance schedule, until reaches ær (0, I). A generative model can be trained to learn the reverse process go (0(x:-1/zz) using the evidence lower bound (ELBO) objective:
CDDPM E. -e (zz, +)
(1)
where A()) is a time-dependent weighting function and e is the added noise.
3.2 Direct Preference Optimization
RLHF first fits a reward model r(x, c), for generated sample and input prompt to human preference data DD and then maximizes the expected reward of a generative model me while ensuring does not significantly deviate from the initialization point Weet- uses the following objective with a divergence penalty controlled by hyperparameter 3.
max E dewee,eawttel))r(( c)]
(Te((())
(2)
The authors of DPO [35] present equivalent objective (Eq. (3)) using the implicit reward model r(x, c) = 3 log [a(a)) + B log (cc) (=)e)
To( (c) Teet (ar(x=))))
To To Teet (x (c)
B2n of enD o log
B log
where Z(c) is the partition function, (x x²) are pairs of winning and losing samples, and e input conditioning. Through this formulation, the model my can be directly trained in supervised fashion without explicitly fitting reward model.
3.3 Implicit Reward Model of a Diffusion Model
One of the challenges in adapting DPO to the context of diffusion models is that the likelihood (2|c) hard to optimize because each sample x is generated through a multi-step Markovian process. In particular, requires computing the marginal distribution Σx,..rx wo (20, 21.... (e) over all possible path of the diffusion process, where To(#o, I1 II.... (c) = TT(xN) II mo( (21,
D3PO [53] adapted DPO by considering the diffusion process Markov Decision Process (MDP). In this setup, an agent takes an action a at each state of the diffusion process. Instead of directly maximizing r(z,, c). one can maximize (aa, s) which assigns value each possible action a at
given state s in the diffusion process instead of the final outcome. This setup uses the local policy T (a]s), which represents single sampling step. In this setup, D3PO [53] showed that the optimal
xā(a(s) for the optimal policy zg. This leads solution Q* (a, s) satisfies the relation Q" (a, s) 3 log (aa(ae) to the approximate objective:
max Eyddd am ( (Q(a, s)] BKx [mlas
(4)
where dd is the state visitation distribution under policy æ. Concretely, the action is a sampling step, and we can write Q(a, s) as Q( 21, and i[aas) π (2)

---

Binary Feedback
Align Model
Utitty UpX)
Risk Scektng
N(u.o*)
argmmax
Diftusion Model
Up&
Reward
Figure We present Diffusion-KTO, which aligns text-to-image diffusion models by extending the utility maximization framework to the setting of diffusion models. Since this framework aims to maximize the utility of each generation (U)))) independently. it does not require paired preference data. Instead, Diffusion-KTe trains with per-image binary feedback signals, likes and dislikes. Our objective also extends to each step in the diffusion process, thereby avoiding the need back-propagate reward through the entire sampling process.
3.4 Kahneman-Tvvrsky Optimization
In decision theory, the expected utility hypothesis assumes that rational agent makes decisions based on the expected utility of all possible outcomes of an action, instead of using objective measurements such as the value of monetary returns. Formally, given an action a and the set of outcomes O(a), the expected utility is defined as (a) = Loco(a) PA (o) where PA is subjective belief of the probability distribution of the outcomes and the utility function (() is real-valued function.
Prospect theory [47] further augments this model by asserting that the utility function is not defined solely on the outcome (e.g. the absolute gain in dollars), but also with respect to some reference point (e.g. current wealth). In this formulation, the utility function defined as (o, ocet) for a reference outcome Ore-. Based this theory, KTO [18] proposed an alternative objective for aligning LLMs:
(x(xcc) Trer (xlc)
max Ee (D(x)) (3 log TT
EVND [3 (L( (I( (m] (1()))
(5)
where æ is the output of the LLM, is the input prompt, w(x) if æ desirable and w(x) --1 otherwise, and A(x) is a weighting function of samples. The divergence penalty is computed as the expectation of the KI. divergence between the model distribution no (x'(('') and the reference distribution (ee((a'(c [c') over all input prompts e in the dataset. This formulation uses the sigmoid function or (xx) as an approximation for the Kahneman- Tversky utility function which concave in gain and convex in loss. Experiments showed that KTO aligned LLMs were able to outperform DPO
aligned LLMs, and KTO is resilient towards noise in the preference data [18].
Method
4.1 Diffusion-KTO
Here, we propose Diffusion-KTT KTO. Instead of optimizing the expected reward, we incorporate non-linear utility function that calculates the utility of an action based on its value Q(a, with respect to the reference point Que-
max Egr Be a'
(U Que))
where (()) a monotonically increasing value function that maps the implicit reward subjective utility. Practically, the local policy Te (a(as) is sampling step, and can be written as TT( 1x)) Applying the relation Q" (a, 8) from Sec. 3.3, we get the objective
To T (st)
max Sero=-DDDo-LLimmmmiti (U (3 log Ti
Quer]]
(7)
T (I ( (2,))
5

---

Following KTO [18], we can optimize the policy based on whether a given generation is considered "desirable" or "undesirable":
Ta (2t
max Ex De (uttuatooooetto log Teat ( xz= xtt)
Quet))
(8)
X0
where wx(o)) 1 image To is desirable or undesirable. We set Qiel = [ilDx [To(()))
Empirically, this is calculated by computing max : Σ log =o(a"la") over batch of unrelated pairs of (s', d') following the KTO setup [18] in Eq. (5).
4.2 Utility Functions
While we incorporate the reference point aspect of the Gaahneman-Tverssee model, it is unclear if other assumptions about human behavior are applicable. It is also known that different people may exhibit different utility functions. Thus, we explore wide range of utility functions. For presentation purposes, we center all utility functions (xr around 0 by using Veenered (x) = (xr) (0), such that U ceettere (0) O. This does not change the objective in Eq. (8) as the gradient and optimal policy are not affected. We experiment with the following utility functions:
Loss-Averse: We characterize loss-averse utility function as any utility function that is concave (see (ax) plotted in blue in Figure 3). Using this utility function, the Diffusion- KTO objective can be considered as a variant of the Diffusion-DPO objective. While aligning according to this utility function follows a similar form to the Diffusion-DPO objective, our approach does not require paired preference data.
Conversely, we define a risk-seeking utility function as any convex utility function (see (xx plotted in yellow in Figure 3). A typical example of risk-secking utility function is the exponential function. However, its exploding behavior on (0, +oo) makes hard to optimize. Instead, for this case, we consider (xx)
Risk-Seeking:
log (-xr)
prospect theory argues that humans tend to be risk-averse for gains but sceking for losses relative to a reference point. This amounts to function that is concave in (0, -00) and convex in (0, -oo). Following the adaptation proposed in KTO, we employ the sigmoid function U(x) = (xx (see (x) plotted in red in Figure 3). Empirically,
Gahneman-Tversky model: Kahneman-Tversee
find this utility function to perform best.
Under the expected utility hypothesis. the expectation is taken over the subjective belief of the distribution of outcomes, not the objective distribution. In our setup, the dataset consists of unpaired samples that either desirable (w(x) 1) undesirable (w(x) 1). Because we do not have access to additional information, we assume the subjective belief of a sample x is solely dependent on w(z). During training, this translates to biased sampling process where each sample is drawn uniformly from all desirable samples with probability ς and uniformly from all undesirable samples with probability 7.
5 Experiments
We comprehensively evaluate Diffusion-Be K'TO through quantitative and qualitative analyses to demon- strate its effectiveness in aligning text-to-image diffusion models with a preference distribution. Further comparisons, such as the performance when using prompts from different datasets, the results of our ablations, and implementation and evaluation details can be found in the Appendix. Additionally, in the Appendix, we report the results of synthetic experiments which highlight that Diffusion- KTO can be used cater T2I diffusion models to the preferences of a specific user. The code used for this work will be made publicly available and is available in the Supplementary material.
Implementation Details. We fine-tune Stable Diffusion v1-5 (SD v1-5) [39] (CreativeML Open RAIL-M license) with the Diffusion-KTO objective, using the Kahneman- -eeremmn--Tverkyy utility function, on the Pick-a-Pic v2 dataset [27] (MIT license). The Pick-a-Pic dataset consists of paired preferences in the form of (preferred image, non-preferred image, input prompt). Since Diffusion- KTO does not require paired preference data, we partition the images in the training data. If an image is labelled

---

Win Rate %
SD v1.5
Diffusion-KTO
VS SFT
Diffusion-KTO
Dimusion-DDP
Diffusion KTO
Figure 4: User study win-rate (%) comparing Diffusion-KTT KTO (SD v1-5) to SD v1-5, and SFT (SD v1-5) and Diffusion-DPO (SD v1-5). Results of our user study show that Diffusion-Ki Cartssoon--OOO- significantly improves the alignment of the base SD vl-5 model. Moreover, our Diffusion-KTF aligned model also outperforms supervised finetuning (SFT) and the officially released Diffusion-DPO model, as judged by users, despite only training with simple per-image binary feedback. We also include the 95% confidence interval of the win-rate.
Table Automatic win-rate (%) for Diffusion-KTO (SD v1-5) in comparison to existing alignment approaches using prompts from the Pick-a-Piic v2 set. We use off aff-the-sheli models, e.g. preference models such as PickScore, to compare generations and determine a winner based on the method with the higher scoring generation. Diffusion KTO drastically improves the alignment of the base SD v1-5 and demonstrates significant improvements in alignment when compared to existing approaches. Win rates above 50% are bolded.
Method
Aesthetic
PickScore
ImageReward
CLIP
HPS v2
vs. SD v1-5
86.0
85.2
87.2
62.0
62.0
vs. SFT
56.4
72.8
64.8
64.8
54.6
vs. CSFT
50.6
73.6
65.2
62.8
60.4
vs. AlignProp
86.8
96.6
84.4
96.2
90.2
vs. D3PO
68.0
73.6
71.6
56.8
55.6
vs. Diffusion-DPO
74.2
61.8
78.4
53.2
51.6
preferred least once, consider it desirable sample, otherwise consider the sample undesirable. In total, we train with 237,530 desirable samples and 690,538 undesirable samples.
Evaluation Details. We evaluate the effectiveness of Diffusion-Ko KTO by comparing generations from our Diffusion-KTTD aligned model to generations from existing methods using automated preference metrics and user studies. For our results using automated preference metries, present win-rates (how often the metric prefers Diffusion-KTT -TT'ss generations versus another method's generations) using the LAION aesthetics classifier [40] (MIT license), which is trained to predict the aesthetic rating a human would give to the provided image, CLIP [34] (MIT license), which measures image-text alignment, and PickScore [27] (MIT license), HPS v2 [50] (Apache-2.0 license), and ImageReward [52] (Apache-2.0 license) which are caption-aware models that are trained to predict studies compare Diffusion-KTO with existing baselines. In our user study, we ask judges to assess which image they prefer (Which image do you prefer given the prompt?) given an image generated by our Diffusion-K KTO model and an image generated by the other method for the same prompt.
human preference score given an image and its caption. Additionally, we perform
We compare Diffusion-KTO
the following baselines: Stable Diffusion v1-5 (SD v1-5), supervised fine-tuning (SFT), conditional supervised fine-tuning (CSFT), AlignProp [33], D3PO [53] and Diffusion-DPo [48]. Our SFT baseline fine-tunes SD v1-5 the subset of images that are labelled as preferred using the standard denoising objective. Our CSFT baseline, similar to the approach introduced into HPS v1 [51], appends a prefix to each prompt ("good image". "bad image") and fine- tunes SD v1-5 using the standard diffusion objective while training with preferred and non-preferred samples independently. To compare with D3PO (MIT license), we fine-tune SD vI-5 using their officially released codebase. For AlignProp (SD v1-5) (MIT license) and Diffusion-DPP (SD v1-5) (Apache-2.0 license), we compare with their officially released checkpoints.
5.1 Quantitative Results
Table I provides the win-rate, per automated metrics, for Diffusion-KTO aligned SD v1-5 and the related baselines. Diffusion-KT0 markedly improves alignment of SD v1-5, with win-rates of up to 87. 47.2%. Results from our user study (Figure 4) confirm that human evaluators consistently prefer the

---

Diffusion-KTO
Diffusion-DPO
SFT
CSFT
SD v1-5
perched forest, with sunight fitoring through the leaves."
'A cute turtle baby sitting cup, captured an humorous
illuttation with warm colors and night scanery.
"Woman,, perfect hair, perfect nose, perfect face, natural light, raytracing, symmetrical depth of fieid, elegant, highly datalled, octane render. photograph Hasselblad
'A minimalistic geometric design with isometric mountains and forest the background and flying
Figure Side-by-side comparison of images generated by related methods using SD v1-5. Diffusion-KTT demonstrates a significant improvement terms of aesthetic appeal and fidelity to the caption (see Sec. 5.2),
generations of Diffusion-KTOO to that of the base SD v1-5 (75% win-rate in favor of Diffusion-KTT KTTO). Further, Diffusion-KTO aligned models outperform related alignment approaches such as AlignProp, D3PO, and Diffusion-DPO. Diffusion-K'1 such as LAION Aesthetics, PickScore, and HPS v2 while performing comparably in terms of other metrics. We also find that human judges prefer generations from our Diffusion- KTO model (72% win-rate versus SFT and 69% win-rate versus Diffusion-DPO) over that from SFT and Diffusion-DPO. This highlights the effectiveness of our utility maximization objective and shows that not only can Diffusion-KIT learn from per- per-image binary feedback, but it can also outperform models training with pairwise preference data.
significantly outperforms Diffusion-DPO on metrics
5.2 Qualitative Results
In Fig. 5, showcase visual comparison of Diffusion-KTTO with existing approaches for preference alignment. As seen in the first row, most models are misguided by the "sunlight" reference in the prompt and produce in dark image. Diffusion-KTO demonstrates a focus on the bird, which the central object in the caption and provides a better quality result over Diffusion-DPO, which doesn't t include any visual indication for the "sunlight". In the second row of images, our Diffusion-Ke -TT aligned model is able to successfully generate a "turtle baby sitting in a coffee cup". Methods such as Diffusion-DPoG in this example, have an aesthetically pleasing result but ignore key components of the prompt (e.g. "turtle", " "night"), On the other hand, SFT and CSFT follow the prompt but provide less appealing images. For the third prompt, which is detailed description of woman, the output from our Diffusion- Euffusion-KTTO model provides the best anatomical features, symmetry, and pose compared the other approaches. Notably, for this third prompt. the generation of the Diffusion-Ko also generated a background is more aesthetically pleasing. The final row uses difficult prompt that requires lot different objects in niche art style. While models were able to depict the right style, i.e. geometric art, only the Diffusion-KTT generation includes key components such as the "moon," " "flying fish," and "fisherman" objects. These examples demonstrate that Diffusion-KTT significantly increases the visual appeal of generated images while improving image-text alignment.
model

---

Data
Pretraining
SFT
Risk Seeking
DPO
Kahneman-Tverra
Begnate Cata
Dnzcgribee byo
Figure 6: Visualizing the effect of various utility functions. We sample from MLP diffusion models trained using various alignment objectives. We find that using the Kahneman -eeeemm---Terkkyy Tversky utility function Icads the best performance in terms of aligning with the desirable distribution and avoiding the undesirable distribution.
Analysis
To further study the effect different utility functions, we conduct miniature experiments observe their impact. We assume the data is two-dimensional, and the pretraining data follows a Gaussian distribution centered at (0.5, 0.8) with variance 0.04. We sample desirable samples from a Gaussian distribution Pa centered at (0.3, 0.8), sample undesirable samples from Gaussian distribution P. centered at (0.3, 0.6), and the variance of both distributions is 0.01. We pretrain small MLP diffusion models, using the standard diffusion objective on the pretraining data, and then fine-tune using various utility functions. We sample 3500 data points from the trained model. Figure 6 shows that risk-averse utility function (used by Diffusion-DPO) has strong tendency to avoid loss to the point that it deviates from the distribution of desirable samples. The risk-seeking utility function behaves roughly the same the SFT baseline and shows strong preference for desirable samples at the cost of tolerating some undesirable samples. In comparison, our objective achieves good balance.
Limitations
While Diffusion-Ko KTT significantly improves the alignment of text-to-image diffusion models, it suffers from the shortcomings of T21 models and related alignment methods. Specifically, Diffusion- KTO is trained on preference data from the Pick-a-Pic dataset which contains prompts submitted by online users and images generated using off-the-shelf T2I models. As a result, the preference distri- bution in this data may skewed toward inappropriate or otherwise unwanted imagery. Furthermore, in this work, we examined three main models of human utility, from which we have concluded the Kahneman -versky model to perform best based on empirical results. However, we believe that the choice of utility function, as well as the underlying assumptions behind such functions, remains an open question. Additionally, since Diffusion-Ko weaknesses of this model, including generating images that reflect and propagate negative stereotypes. Despite these limitations, Diffusion- KTO presents broader framework for improving and aligning diffusion models from per-image binary feedback.
fine-tunes a pretrained T2I model, it inherits the
8 Conclusion
In this paper, we introduced Diffusion-KTO, a novel approach to aligning text-to-image diffusion models with human preferences using utility maximization framework. This framework avoids the need to collect pairwise preference data, as Diffusion- KTO only requires simple per-image binary feedback, such likes and dislikes. We extend the utility maximization approach, recently introduced to align LLMs, to the setting of diffusion models and explore various utility functions. Diffusion-KTO aligned diffusion models lead demonstrable improvements in image preference and image-text alignment when evaluated by human judges and automated metrics. While our work has empirically found the Kahneman- Tversky model of human utility work best, we believe that the choice of utility functions remains an open question and promising direction for future work.

---

References
[1] J. Achiam, S. Adler, S. Agarwal, I., Ahmad, I. Akkaya, F. L., Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] R. Akrour, M. Schoenauer, and M. Sebag. Preference-basct policy learning. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part 11, pages 12 12-27. Springer, 2011.
[3] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant a laboratory for alignment. arXiv preprint (XXi::2122.00060 2021.
[4] A. Baheti, X. Lu, F. Brahman, R. L. Bras, M. Sap, and M. Riedl. Improving language models with advantage-based offline policy gradients. arXiv preprint urXiv:2305.14711 2023.
[5] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv
preprint arXiv:2204..0 05862, 2022.
[6] M. Bakker, M. Chadwick, H. Sheahan, M. Tessler, L. Campbell Hhipbbbhl-ill J. Balaguer, N. McAlcese, A. Glaese, J. Aslanides, M. Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-388899 2022.
[7] Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211 112111111224 2022.
[8] Betker, G. Goh, L. Jing, T. Brooks, Wang. L. Li, L. Ouyang, J. Zhuang, Lec, Y. Guo, et al. Improving image generation with better captions. Computer Science. httpx://cdn. openai. com/papers/dail-e-3 pdf. 2(3):8, 2023.
[9] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement lcarning. arXiv preprint arXiv:2305.133:5 2023.
[10] F. Böhm, Y. Gao, C. M. Meyer, O. Shapira, I. Dagan, and I. Gurevych. Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:190.00224 2019.
[11] H. Chang, H. Zhang, J. Barber, A. Maschinot. J. Lezama, L. Jiang. M.-H. Yang. K. Murphy, W. T. Freeman, Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
[12] W. Cheng, J. Fürnkranz, E. Hüllermeier, and S.-H. Park. Preference-based policy iteration: Leveraging preference learning for reinforcement learning. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5.9, 2011. Proceedings. Part I 11. pages 112-327. Springer, 2011.
[13] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[14] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 1 17400, 2023.
[15] X. Dai, J. Hou, C.-Y. Ma. S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309. 15807, 2023.
[16] H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXi:22304.06677 2023.
[17] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[18] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.0 00600000..6... 2024.
[19] Y. Fan and K. Lee. Optimizing ddpm sampling with shortcut fine-tuning, arXiv preprint 2331v:2301.13336 2023.
[20] Y. Fan, O. Watkins, Y. Du. Liu, M. Ryu, Boutilier, P. Abbeel, M. Ghavamzadeh, Lee, and K. Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[21] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835 10866. PMLR, 2023.
[22] A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chad- wick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint X20: 14375, 2022.
10

---

[23] Y. Hao, Z. Chi, L. Dong. and F. Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[24] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[25] E. J. Hu, Y. Shen, P. Wallis. Z. Allen- Zhu, Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint S0ti:2206.096888 2021.
[26] M. Kang. J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up gans for text-to- image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[27] Y. Kinstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pie: An open datasct of user preferences for text-to-image generation. Advances Neural Information Processing Systems, 36, 2024.
[28] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 2023.
[29] J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell- Gillingham, G. Irving, al. Teaching language madels support answers with verified quotes. arXiv preprint arXiv:2203. / 147, 2022.
[30] R. Nakano, J. Hilton, S. Balaji, J. Wu, Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, al. Webgpt: Browser-assisted question- -anseernnn with human feedback. arXiv preprint arXiv:2112.09332.. 2021.
[31] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-22774, 2022.
[32] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhom, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high- nigh-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[33] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki, Aligning text-to-image diffusion models with reward hackpropagation. arXiv preprint arXiv:2310.033799 2023.
[34] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[35] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances Neural Information Processing Systems, 36, 2024.
[36] R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage. H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.0124. 2022.
[37] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint trXiv:2204.06122 1())3, 2022.
[38] A. Ramesh, M. Pavlov, G. Goh, Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831, PMLR, 2021.
[39] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] C. Schuhmann. Laion-aesthetics. http:://aaao....aiil///aaaaaaoaa cessed: 2024- - 03 - 06.
2022. Ac-
[41] E. Segalis, D. Valevski, D. Lumen, Y. Matias, and Y. Leviathan. A picture is worth a thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310.1660 2023.
[42] J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. Advancex in Neural Information Processing Systems, 15:9460-9471 2022.
[43] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256 2265. PMLR, 2015.
[44] Y. Song, J. Sohl-Dickstein D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.33450 2020.
11

---

[45] N. Stiemnon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[46] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2rX1v:2307.09288, 2023.
[47] A. Tversky and D. Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323, 1992.
[48] B. Wallace, M. Dang, R. Rafailov, L. Zhou. A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik, Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:231 12,900
2023.
[49] B. Wang, 7heng, I.. Chen, Liu, S. Dou, C. Huang, Shen, S. Jin, E. Zhou, C. Shi, et Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401. .60000, 2024.
[50] x. Wu, Y. Hao, K. Sun, Y. Chen, Zhu, Zhao, and H. Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-lo-image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
[51] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Human preference score: Better aligning text-to-image models with human preference. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096-2101 2005, 2023.
[52] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances Neural Information Processing Systems, 36, 2024.
[53] K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, Q Li, W. Shen, X. Zhu, and X. Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arX1v:2311.13231, 2023.
[54] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint ar 22v: 10789, 2(3):5, 2022.
[55] L. Yu, B. Shi, R. Pasunuru, B. Muller, O. Golovneva, T. Wang, A. Babu, B. Tang, B. Karrer, S. Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint vrXiv:2309.0254/4 2023.
[56] Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 2023.
[57] Y. Zhao, R. Joshi. Liu, M. Khalman, M. Saleh, and J. Liu. Slic-hf: Sequence likelihood calibration with human foedback. arXiv preprint arXiv:2305. 10425, 2023.
[58] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang. Y. Liu. S. Jin, Q. Liu, Y. Zhou, al. Secrets of rihf in large language models part Ppo. arXiv preprint arXiv:2307.04966 2023.
[59] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (oK::::909005994 2019.
12

---

A Experiment Settings
A.1 Implementation Details
We train Stable Diffusion v1-5 (SD v1-5) on 4 NVIDIA A6000 GPUs with a batch size of 2 per GPU using the Adam optimizer. We use a base learning rate of lc-7 with 1000 warm-up steps for a total of 10000 iterations. We set 3 to 5000. We sample from the set of desirable samples according to y 0.8. To make Diffusion-KTPP TT possible on paired preference datasets such Pick-a-Pic [27], we create new sampling strategy where we categorize every image that has been labelled as preferred least once desirable samples and the rest as undesirable samples.
A.2 Evaluation Details
We employ human judges via Amazon Mechanical Turk (MTurk) for our user studies, Judges are given the prompt and a side-by-side image consisting two generations from two different methods (e.g. Diffusion-KTTD and Diffusion-DPO [48]) for the given prompt. We gather prompts by randomly sampling 100 prompts, 25 from each prompt style ("Animation". "Concept-art" "Painting". "Photo"), from the HPS v2 [50] benchmark. We collect a total of 300 human responses for our user study. In the interest of human safety, we opt to use prompts from HPS v2 instead of Pick-a-Pic [27]. as we have found some prompts in the latter to be suggestive or otherwise inappropriate. The authors of HPS incorporate additional filtering steps remove inappropriate prompts. We also inform judges that they may be exposed to explicit content by checking this box in the MTTurk interface and including the disclaimer: "WARNING: This HIT may contain adult content. Worker discretion is advised" in our project description. We gauge human preference by asking annotators which image they prefer given the prompt, i.e. "Which image do you prefer given the prompt?". The instructions given to our judges are provided below. Judges are asked to select "I" or "2", corresponding to which image they prefer (1 refers to the image on the left and 2 refers to the image on the right, these values noted above each image when displayed). To ensure fair comparison, they are not given any information about which methods are being compared and the order methods (left or right) is randomized. MTurk workers compensated in accordance with the minimum wage laws of the authors' country. We follow the guidelines and approval required for such studies by our institution.
Instructions
Both of these images were generated by AI models trained to create an image from text prompt. Which image do you prefer given the associated text?
Example criteria could include: detail, art quality, aesthetics, how well the text prompt is reflected, lack of distortions/iregglariiie (e.g. extra limbs, objects). In general, choose which image you think you would consider to be "better".
For our evaluation using automated metrics, we report win-rates (how often the metric prefers Diffusion- KTO's generations versus another method's generations). Given prompt, we generate one image from the Diffusion-KTO aligned model and one image using another method. The winning method is determined by which method's image has a higher score per the automated metric. To account for the variance in sampling from diffusion models, we generate images per method and report the win-rate using the median scoring images. We evaluate using all the prompts from the test set of Pick-a-Pic, the test set of HPS v2, and the prompts from PartiPrompts.
The human evaluation experiment received exemption for IRB.
13

---

B Additional Quantitative Results
B.1 Performance in terms of Average Score per Preference Models.
In Table 2, we report the average score (and 95% confidence interval of the mean) given by cach metric used in our automated evaluations. For each method (using SD v1-5), we sample generations per prompt, for a total of 2500 generations (i.e. N=2500). As seen below, Diffusion- KTTO exhibits state-of-the-an performance according numerous metrics while performing comparably to the state-of-the-an art in the remaining metrics. These results demonstrate the effectiveness of Diffusion- KTO for aligning T2I diffusion models with human preferences. Additionally, we report the 95% confidence interval of win rate in Table 3
Table 2: Average score according existing models when evaluated
Pick-a-Pic test set. We report the mean score and the 95% confidence interval of the mean when evaluating prompts from the Pick-a-Pic test set. Methods with the highest mean score according to a given metric are bolded.
Method
Aesthetic
PickScore
ImageReward
CLIP
HPS v2
SD v1-5 SFT
5.281 =.022
20.387 #.054
0.333 +.002
31.364 +. 144
0.102 +.042
5,499 =.020
20.664 +.052
0.336.002
31.377 1.141
0.485 #.038
CSFT
5.527 #.020
20.713 #.053
0.335 #.00
31.448 +.147
0.488 #.40
AlignProp
5.106 #.021
19.123 #.054
0.278 #.002
26.932 #.144
0.195 #.02
D3PO
5.326 #.022
20.413 #.055
0.333 #.002
31.350 #.147
0.143 0.42
Diffusion-DPP
5,380 #.022
20.785 #.055
0.339.000
31.673 1.145
0.293 #.04
Diffusion-K KTO
5.527 .020
20.908 +.054
0.342.0000
31.781 +.143
0.623 +.039
Table 3: Confidence Interval of Win Rate (%) on Pick-a-Pic test set. We report the the 95% confidence
interval of the win rate over 2500 samples.
Method
Aesthetic
PickScore
ImageReward
CLIP
HPS v2
vs. SD v1-5
86.0+1.36
85.2+1.39
87.2+1.31
62.0+1.90
62.0+1.90
vs. SFT
56.4+1.94
72.8+1.74
64.8*1.87
64.8+1.87
54.6+1.95
vs. CSFT
1..6
73.6+1.73
65.2+1.87
62.8+1.89
60.4+1.92
vs. AlignProp
86.8+1.33
96.6+0.71
84.4+1.42
96.2+0.75
90.2+1.17
vs. D3PO
68.0+1.83
73.6+1.73
71.6#1.77
56.8+1.94
55.6+1..55
vs. Diffusion-DPO
74.2+1.72
61.8+1.90
78.4+1.61
53.2+1.96
51.6+1.96
B.2 Performance on HPS v2 and PartiPrompts.
In addition to the results on the Pick-a-Pic test set reported in Table 1, we provide additional results on HPS v2 (Apache-2.00 license) and PartiPrompts (Apache-2.0) license) datasets in Table and Table Results show that Diffusion-KTO outperforms existing baselines on diverse set of prompts.
Table 4: Automatic win-rate (%) for Diffusion-K KTO in comparison to existing alignment approaches using prompts from the HPS v2 set. The provided win-rates display how often automated metries prefer Diffusion-KTO generations to that other methods. Win rates above 50% are bolded.
Method
Aesthetic
PickScore
ImageReward
CLIP
HPS v2
vs. SD v1-5
76.2+1.67
77.7*1.63
74.3#1.71
53.5+1.96
53.6+1.95
vs. SFT
60.1+1.92
66.6+1.85
54.5
54.9+1.95
51.9+1.96
vs. CSFT
51.5+1.96
64.7+1.87
52.2+1196
54.5+1.95
55.3+1.95
vs. AlignProp
86.2+1.35
98.0+0.55
81.0+1.54
93.2+0.99
89.7+1.19
vs. D3PO
76.0+1.67
76.9+1.65
75.6*1.68
54.4+1.95
53.9+1.95
vs. Diffusion-DPO
63.9+1.88
60.3+1.92
66.3+1.85
47.3+1.96
48.9+1.96
In addition, we provide a per-style score breakdown using the prompts and their associated styles (Animation, Concept-art, Painting, Photo) in the HPSv2 test set in Appendix B.2. Across these
14

---

Table 5: Automatic win-rate (%) for Diffusion-KTO in comparison to existing alignment approaches using prompts from PartiPrompts. The provided win-rates display how often automated metrics prefer Diffusion-KTO generations to that of other methods. Win rates above 50% are bolded.
Method
Aesthetic
PickScore
ImageReward
CLIP
HPS v2
vs. SD v1-5
74.2+1.72
67.1+1.84
66.9#1.84
53.8
1..5+1.96
vs. SFT
55.0=1.95
65.0+1.87
53.6#1.95
54.4+1.95
53.2+1.96
Vs. CSFT
50.9:1.96
62.3+1.90
53,9#1.95
51.8+1.96
53.0+1.96
Vs. AlignProp
7..6+1166
95.5+0.81
79.0=1.60
91.2+t1.11
86.8+1.33
vs. D3PO
7..1
67.0#1.84
68.9+1.81
51.5+1.96
53.0=1.96
vs. Diffusion-DPO
66.2+1.85
52.7+1.96
56.4#1.94
49.6+1.96
46. 16.95
metrics, our model performs best for "painting" and "concept-art" styles. We attribute this to our training data. Since Pick-a-Pic prompts are written by users, it will reflect their biases, e.g., bias towards artistic content. Such biases are also noted by the authors of HPSv2 who state "However, significant portion of the prompts in the database is biased towards certain styles. For instance, around 15.0% of the prompts in DiffusionDB include the name 'ree Rutkowski", 28.5% include "artstation".
We also observe that different metrics prefer different styles. For example, the "photos" style has the highest PickScore but the lowest ImageReward. With this in mind, would like to underscore that our method, Diffusion-KTO, is agnostic to the preference distribution (as long as feedback per-sample and binary), and training different, less biased preference data could avoid such discrepancies.
Style anime
Aesthetic
PickScore
ImageReward
CLIP
HPS
5,493
21.569
0.716
34.301
0.368
concept-art
5.795
21.011
0.804
33.141
0.359
paintings
5.979
21.065
0.802
33.662
0.360
photo 5.365
21.755
0.471
31.047
0.332
Table 6: Per-style score breakdown for different metrics in the HPSv2 test set.
B.3 Using Stable Diffusion v2-1.
We perform additional experiments, this time using Stable Diffusion v2-1 (SD v2-1) [39] (CreativeML Open RAIL++M license). We fine-tune SD v2-1 using Diffusion-KTO with the same hyperparame- ters and compute listed in Appendix A. In Table v2-1 and Diffusion-DPO (SD v2-1). To compare with Diffusion-DPO. we fine-tune SD v2-1 using the official codebase released by the authors. As in Table 7, Diffusion KTO outperforms the SD v2-1 base model and Diffusion-DPO according to most metrics while performing comparably in others. This highlights the generality of Diffusion-KTO it is architecture agnostic approach to improving the alignment of any text-to-image diffusion model.
compare Diffusion-KTT (SD v2-1) with SD
C Synthetic Experiment: Aligning with a Specific User
Per-image binary feedback data is easy to collect and is abundantly available on the internet in the forms of likes and dislikes. This opens up the possibility of aligning T2I models to the preferences of a specific user. While users may avoid the tedious task of providing pairwise preference, Diffusion- KTO can be used to easily align a diffusion model based on the images that user likes and dislikes. Here, we conduct synthetic experiments to demonstrate that Diffusion-KIT can be used to align models to custom preference heuristics, in attempt to simulate the preference of select user.
experiment using two custom heuristics to mock the preferences of user. These heuristics are: (1) preference for red images (i.e. red filter preference) and (2) preference for images with high aesthetics score. For these experiments, we fine-tune Stable Diffusion v1-5 using the details listed in A.I. For the red filter preference experiment, use (image, caption) pairs from the Pick-a-Pic
15

---

Table 7: Automatic win-rate (%) for Diflusion-KTO when using Stable Diffusion v2-1 (SD v2-1). The provided win-rates display how often automated metrics prefer Diffusion-KTO generations that other methods. Results using Diffusion-DPO (SD v2-1) were produced by training SD v2-1 with the Diffusion-DPO objective, using the official codebase released by the authors. Win rates above 50% are bolded.
Dataset
Diffusion-Kii
Aesthetic
PickScore
ImageReward
CLIP
HPS
vs SD v2-1 Diffusion-DDPO
71.4
70.0
69.2
53.4
49.6
Pick-A-Pic
63.8
67.4
66.2
50.0
44.8
vs SD v2-1 72.2
77.9
71.0
51.6
50.1
HPS v2
vs Diffusion-DPO
65.8
71.4
67.3
49.9
47.4
SD v2-1 69.2
67.2
65.0
50.8
48.0
PartiPrompts
VS Diffusion-DPo
66.0
61.2
63.1
49.2
44.7
Iteration 0
Iteration 1K
lteration 2K
A A
Aesthetics
bestheet score
100
Red Filter
Redness
Figure 7: Aligning text-to-image models with the preferences of a specific user. Since per-image binary feedback is easy-to-collect, we perform synthetic experiments to demonstrate that Diffusion-KTO is effective of aligning models the preferences of specific a text-to-ing text-to-image model with arbitrary user preference using Diffusion KTO Stable Diffusion v1-5. The first row displays generations from a model that is trained to learn a preference for LAION aesthetics score 2 7. As expected. these generations tend to introduce further detail (such as the woman's facial features) and add additional colors and textures. The second row displays images from model that is trained to learn a preference for red images, and Diffusion-KTO learns add this preference for red images while minimally changing the content of the image. We additionally plot the aesthetic score redness score throughout training. The redness score is calculated as the difference between the average intensity of the red channel and the average intensity in all channels.
We show qualitative results for customizing
v2 training set and enhance the red channel values to generate desired samples (original images are considered undesirable). For the aesthetics experiment, we train with images from the Pick-a-Pic v2 training set. We use images with an aesthetics score 2 7 as desirable samples and categorize the remaining images as undesirable. Figure 7 provides visual results depicting how Diffusion-KTT can align with arbitrary user preferences. For the aesthetics preference experiment (Figure 7 row 1), we see that generations contain finer detail and additional colors and textures, in comparison to the baseline image, both of which are characteristics of a high scoring images per the LAION aesthetics classifier. Similarly, Diffusion-KTO also learns the preference for red images in the red filter preference experiment (Figure 7 row 2). While we experiment with simple heuristics, these results show the efficacy of Diffusion-Ko KTO learning arbitrary preferences using only per-sample binary feedback.
16

---

D Ablations
We explored various design choices of Diffusion-KTO this section. We report the mean PickScore on the HPS v2 dataset, which consists of 3500 prompts and is the largest amongst Tick-a-Pic, PartiPrompts, and HPS v2. We show results in Fig.
Serpph
2149
Sumple
21.00
23.29
11.20
11.18
21:33
21.35
15100
2133
71.0
win
Whri wntiiineerdeeaee
0.2
0.5
0.8
1.0
Lous-Aourse KTO Nisk-Seeking
2000
3000
4000
5000
Figure 8: Ablation Studies. We experiment with different data sampling strategy and different utility functions. Results show best comhination is a) use the winning (i.e. images that are always preferred) intermediate (Lee images that are sometimes preferred and sometimes con-preferred) samples, b) use positive ratio of 0.8, c) use the KTO objective function. d) use beta 3 value of 5000.
D.1 Data Partitioning
In converting the Pick-a-Pic dataset, which consists pairwise preferences, to a dataset per-image binary feedback, consider two possible options. The first option is to categorize sample desired is always preferred across all pairwise comparisons (win), with all other samples considered undesirable. The second option additionally incorporate any samples desirable samples if they are labelled as preferred at least one pairwise comparison. Results show that the latter option is better strategy.
D.2
Data Sampling
During the training process, we sample some images from the set of all desired images and some images from the set of all undesired images. This ratio is set to fixed value to account for potentially imbalanced dataset. We find that sampling 80% of the images a minibatch from the set of desired images achieves the optimal result.
D.3 Choice of Beta
We experiment with different values between 2000 and 5000 for B, the parameter controlling deviation from the policy. We show that there is rise in performance with the increase in value but with diminishing returns, indicating an optimal score around 5000.
D.4 Utility function
We explored the effect of various utility function described in the main paper. Particularly, we consider Loss-Averse Uxx) log a (x), KTO U(x) = (x) and Risk-Seeking U(x) (-g) Results show that KTO is the optimal result.
We visualize the utility function and its first order derivative Fig. Intuitively, Loss-Aversion will reduce the update step during the training when reward is sufficiently high, Risk-Seeking will reduce the update step during the training when reward low. KTO will reduce the update step the reward is either sufficiently high or sufficiently low. This makes KTO more robust to noisy and sometimes contradictory preferences.
17

---

U64
duidx
Kahnemaan-Ivenn Aversinn Risk-Seeking
Roward
Reward
Figure 9: Utility functions visualizations. We visualize the utility function and its first order derivative. Intuitively, Loss-Aversion will reduce the update step during the training when reward is sufficiently high, Risk Dick-Sccking will reduce the update step during the training when reward low. KTO will reduce the update step reward either sufficiently high or sufficiently low. The functions are centered by a constant offset so that (0) = for better visibility. The constant offset does not contribute to the gradient and, thus, has no effect to training.
E Additional Qualitative Results
We provide further visual comparisons between Diffusion- KTO aligned SD v1-5 and the off-the-shelf SD v1-5 (Fig. 10). We find that Diffusion-KT KTO aligned models improve various aspects including photorealism, richness of colors, and attention to fine details. We also provide visual examples of failure cases from Diffusion-K' KTO aligned SD v1-5 (Fig. 13).
Safety
To understand the effect of aligning with Pick-a-Pic v2, which is known to contain some NSFW content, we run a CLIP-basee NSFW safety checker on images generated using test prompts from Pick-a-Pic v2 and HPSv2. For Pick-a-Pic prompts, 5.4% of Diffusson-KTT generations NSFW, and 4.4% SDv1-5 generations are marked NSFW. For HPSv2 prompts, which are safer, 1.3% of Diffusion-KTT KTO generations marked NSFW, and 1.0% of SD v1-5 generations marked NSFW. Overall, training on the Pick-a-Pic dataset leads to a marginal increase in NSFW content. We observe similar trends for Diffusion-DPOO which aligns with the same preference distribution (5.8% NSFW on Pick-a-Pick and 1.3% NSFW on HPSv2). We would like to emphasize that our method is agnostic to the choice of preference dataset, long as the data can converted into binary per-sample feedback. We used Pick-a-Pit Pic because of its size and to fairly compare with related works. In general, we encourage fair and responsible use of our algorithm and
marked
G Details of Qualitative Results
In this section, we discuss the sources of the prompts used in Fig. 5. To highlight the advantage of Diffusion-KTO in real-world scenarios, we refer to user prompts shared over the internet. In particular, chose Playground AI (https:/playgrruud.coo alongside their prompts. These prompts reflect typical use cases in real-world scenarios and is generally similar to the "good" samples in the Pick-a-Pic dataset, which is also written by human labelers.
where users share generated images
18

---

Diffusion-KTO (SD v1-5)
SD v1-5
DICE!
Figure 10: Side-by-side comparison of Diffusion-KTTT (SD v1-5) versus Stable Diffusion v1-5. The images were created using the prompts: "A rare animal in its habitat, focusing on its fur texture and eye depth during golden hour." "A magical forest with tall trees, sunlight, and mystical creatures, visualized in detail.", "A city after an apocalypse, showing nature taking over buildings and streets focus on rebirth."
19

---

Diffusion-KTO (SD v1-5)
SD v1-5
Figure 11: Additional side-by-side comparison of Diffusion-KTO (SD v1-5) versus Stable Diffusion v1-5. The images were created using the prompts: "A dramatic space hattle where two starships clash among asteroids, with laser beams lighting up the dark void and explosions sending debris flying. intense and futuristic." "A timeworn portal in the middle of
lake, with glowing edges that ripple with energy, reflecting starry sky its surface. captured in an ultra HDD painting.", peaceful digital painting meadow sunrise, where wildflowers bloom and gentle mist rises from the grass, soft focus."
20

---

Diffusion-KTO (SD v1-5)
SD v1-5
Figure 12: Additional biddeb comparison of Diffusion-KTO (SD v1-5) versus Stable Diffusion v1-5. The images were created using the prompts: "A dramatic scene of two ships caught in a stormy sea, with lightning striking the waves and sailors struggling to steer. 8k resolution." ", "A cinematic black and white portrait of man with weathered face and stubble. soft natural light through window, shallow depth of field. shot a Canon SD Mark III.", "A hyperrealistic close-up cherry blossom branch full bloom, with each petal delicatoly illuminated by the morning sun, 8k resolution."
21

---

"A playful dolphin leaping out of turquoise waves, with backdrop a stunningly colorful coral reef."
"An ancient dragon perched atop mountain, overlooking valley illuminated by the golden light sunrise, with every scale visible in crisp detail."
"Wish you were here
"A Pokémon trainer discovering a valley with wild Pokémon with Butterfree fluttering nearby and group Jigglypuff singing the distance"
WISH WIOULSE WSI WIST
Figure 13: Failures cases of Diffusion-KTO SD v1-5. In the first instance (top-left) the dolphin is correctly shown "leaping out of the water". however, the coral reef the surface of the water not the bottom of the sea. The second image (top-right) shows dragon the top the mountain, however the dragon's body seems merge with stone. For bottom left image, written. The final image depicts Pokémon trainer in a valley accurately, hut is missing a "group of Jigglypuff". We note that Diffusion-K KTO aligned models inherit the limitations of existing text-to-image models.
caption "Wish you were here" which is incorrectly
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in abstract and introduction are supported by experimental results Section (and additionally Appendix B).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not introduce new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We take the following steps to ensure reproducibility: (1) we clearly describe the objective used in our work (Section 4), (2) full details of our experiments are provided in Section 5 and Appendix A. Additionally, our code is available in the Supplementary material and will be publicly released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use publicly available datasets to train (Pick-a-Pic) and evaluate (Pick-a- Pic, HPS v2, PartiPrompts) (see Section 5). Code is provided in the Supplemental material, and will be open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /ttp://nips...e cc/
public/ guides/ Sulomu wwwsssssssiigisooooooiiisiissiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa: / /nips. cc/
wttppcccppcccccccccctccccccccccc public/ healic//giiee gt//uuuee//////// /eddeooeSuulla Submiu as wwwoooooioooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: For details of the data split, see Section 5. For details about hyperparameters, optimizers, etc., see Appendix A. These details are also documented our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
See Appendix A.2 and B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are reported in Appendix A.1 and documented via structured template in our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttls::///uurriee https///c...p.ccc/cccccceeeeeeee
Answer: [Yes]
Justification: We fairly compensate human evaluators and take additional measures to ensure that they are not exposed to NSFW content during their work, further details can be found in Appendix A.2 (Potential Harms Caused by the Research Process). We communicate the impact, including biases that are learned from skewed preference data, of this work in Section of the paper (Societal Impact and Potential Harmful Consequences).
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are discussed in Section 7 and in the Introduction.
26

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We only release training code.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Assets used in this work are cited and the licenses are explicitly mentioned (see Section 5 and Appendix B).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw's thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details regarding our training process are documented in the paper (see Section 5 and Appendix A) and are documented as part of the code in the Supplementary material (see readme.md md and the structured template is given in the implementation details markdown file).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/cooeemmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We perform user study to evaluate the quality SiffuusionKTOO" generations. We provide full details of this evaluation in Appendix A.2 (Evaluation Details).
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, risks are described and disclosed to participants (plus we take additional safety measures to minimize exposure). We also obtain approval from our institution. See Appendix A.2.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "lDFTf7PpYz", "reviewer": "Reviewer_MiLc", "summary": "This paper proposes Diffusion-KTO, which extends the KTO theory to develop a preference optimization algorithm for", "questions": "* Can you include an example without the use of a preference pair dataset? Pick-a-pic is paired, but the main benefit of KTO should be the ability to extend beyond paired data (which might be useful for diffusion models since generation is expensive).", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The derivations are clean, direct, and seem reasonably principled to me.\n\n* The experimental results are good and show a clear improvement over preexisting work. However, I would include more quantitative examples in the main paper.\n\n* Aligning diffusion model methods is a reasonably impactful area, and doing so with novel techniques is nontrivial and thus requires works like this one.", "weaknesses": "* While I enjoyed the paper, the novelty is somewhat limited. In particular, the paper is a relatively direct combination of the Diffusion DPO type methods and novel DPO style losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review! We are excited to hear that you appreciate the importance of our problem statement, the strength of our experimental results, and the presentation of our method. Please find our responses below.\n\n**The novelty is somewhat limited**\n\n\nPlease see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for our response.\n\n**Can you include an example without the use of a preference pair dataset**. \n\nWe provide results when using two protocols for sampling at the population level of our data. We tried a) for each prompt in the dataset, incorporate both winner and loser separately as training data and b) for each prompt in the dataset, keep either the winner or loser as training data. Empirically, we find no significant differences between the performance of these two protocols. This effect is also seen by the authors of the original KTO paper. Specifically, we find the second protocol (b) performs slightly better on Aesthetics (+0.02), PickScore (+0.04), HPS (+0.002), CLIP (+0.13), and ImageReward (+0.05). These differences, apart from ImageReward, are within the error bound of estimates. The second setup (b) could be marginally better than the first setup (a) as it may remove some relative noise within paired preference data. In summary, we did not train with paired preferences and, we find that incorporating both the winner and loser separately or incorporating either the winner or loser into our training data generally leads to no significant difference in performance.\n\nAdditionally, we would also like to point out that we tried two toy experiments using binary preferences in Appendix C. In these experiments, we created two synthetic datasets with binary labels by a) using a red filter and setting red-filtered images as desirable and non-filtered ones as undesirable, and b) thresholding the LAION aesthetic score and labeling samples with high scores as desirable and samples with low scores as undesirable. Results show that Diffusion-KTO can effectively align to binary preferences at the user level.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHkfpOOyOh", "reviewer": "Reviewer_7SUT", "summary": "This paper presents Diffusion-KTO, a novel preference learning algorithm for diffusion models. The proposed preference learning algorithm is based on Kahneman & Tversky Optimization (KTO). Diffusion-KTO enables aligning a diffusion model using only binary pointwise feedbacks, improving data efficiency and robustness. The experiments show promising results.", "questions": "See weakness. I would adjust my rating based on the response regarding the weaknesses of the paper.", "limitations": "The paper addresses its limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Aligning a diffusion model from human feedback is an important problem that receives huge attention from the community.\n* The experimental results are promising and interesting.\n* Overall presentation of the paper is clear, with some parts requiring improvements (see Weakness)", "weaknesses": "* The proposed algorithm is almost a simple application of KTO to diffusion models. The originality of the contribution is, therefore, not very strong. Of course, the combination of diffusion models and KTO is indeed novel; somebody else might have done it soon if this paper hadn't.  I acknowledge the novelty, but the paper would have been stronger with more original ideas.\n* The paper has to be more self-contained. The paper borrows a lot of components from existing works, such as KTO and D3PO, and the borrowed elements should be explained well. Section 3.3 is particularly dissatisfying. There is no rigorous definition of Q, no definition of pi^*, and no explanation of why Equation 4 is an approximate objective. In a similar spirit, Section 4.1 could be more informative, such as providing an explanation for why Q_ref has to be set as proposed.\n* The major concern regarding the experiment is that the number of human subjects and human responses are not disclosed (please correct me if I am wrong). This information is required to judge the uncertainty of the winning rate presented in Figure 4. It would be great if the uncertainty of the winning rate could also be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are happy to hear that you find our problem area important and that you appreciate the presentation of our work and our experimental results. Please see our responses below.\n\n\n**The proposed algorithm is almost a simple application of KTO to diffusion models.**\n\nPlease see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for our response.\n\n\n**The paper has to be more self-contained.**\n\nIn regards to the specific concerns raised (i.e. the substitution of $Q$ and definition of $\\pi^*$, and $Q_{ref}$), we provide a detailed review of our Diffusion-KTO formulation with clarifications for these definitions and choices in the main rebuttal (Clarification of Formulation). We hope these discussions can resolve your concerns. If you have any additional questions, please let us know and we would be happy to answer them in the discussion period. \n\n\n**The number of human subjects and human responses are not disclosed.**\n\nWe are sorry for this oversight. We collected 300 human responses. The 95% confidence interval is 65.6%-74.8% for the win-rate against DPO and 73.6%-82.9% for the win-rate against SDv1-5. Both results are significant, with p-value < 0.001.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "irCV1coFDm", "reviewer": "Reviewer_s9JR", "summary": "This paper (DKTO) combines D3PO and KTO. D3PO let's us apply DPO to Diffusion models using pairwise preferences, and KTO is a way to align generative models (specifically autoregressive LLMs) using pointwise preference. For example, this gives us a way to tune text-conditioned, image generative models from thumbs-up, thumbs down or star ratings type of bandit feedback, without training a secondary reward model.\n\nDKTO works by optimizing the expected utility of the advantage of the new policy versus a reference policy. And empirically, its results on finetuned Stable diffusion are pretty compelling. However, the mathematical derivation of DKTO is a bit messy and not rigorous.", "questions": "See weaknesses.", "limitations": "Yes, authors have addressed the limitations adequately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The results in the paper are compelling and the contributions in the paper seem to be novel. AFAICT relevant baselines were considered for experiments and DKTO seems to win handily against them.", "weaknesses": "Section 4.1 which derives the actual diffusion KTO method is really short and seems wrong on first glance. To go from eq (6) to Eq(7) the relation Q* = β (log π_θ - log π_ref) is substituted but that relation is lifted from the D3PO paper that derived it under the assumption that the policy optimization objective is E[Q*] - β KL[π_θ || π_ref] . But the diffusion KTO objective is different so it's not clear why prop 1 from the DKTO paper still applies.\n\nSecondly, the substitution Q_ref = β KL is not very well motivated. The notation Q_ref suggests that it should not have had any dependence on π_θ at all !! Why is the KL divergence a reasonable proxy for Q_ref ? \n\nOverall, I think the paper in its current form is weak from a theory/conceptual point of view, and the method needs to be motivated a lot more cleanly. Please let me know if I have made a mistake, I'll gladly update my scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are glad that you appreciated the contributions and comprehensive experiments presented in this paper. \n\nRegarding the concerns on specific formulations, we have provided a detailed review of our formulation in the main rebuttal that should address these concerns. The relevant parts are summarized as follows:\n\n**It's not clear why prop 1 from the DKTO paper still applies.**\n\nWe assume you are referring to prop1 from the D3PO (not DKTO) paper. It is sufficient to establish this relation under the policy optimization objective. The “substitution” is not wrong because KTO defines its implicit reward function $r_\\theta(X_0,C)=\\beta\\log(\\pi_{\\theta}(X_0|C)/\\pi_{ref}(X_0|C))$ in an axiomatic approach (Definition 3.4 of KTO paper).\n\nKTO justifies such definition through a “classic prospect theory experiment”, and the fact that $r_\\theta$ is in the same equivalence class as the human-preference-induced reward function in (eq 2) at optimal policy $\\pi^*_\\theta$ under RLHF objective (eq 2). In prospect theory, $r_\\theta$ \"amounts to the dollar amount assigned to each outcome.\" However, the KTO formulation does not imply that the KTO objective is strictly equivalent to the RLHF objective in a similar way as DPO. There is no \"substitution\" in KTO formulation.\n\nBy \"applying the relation $Q^∗(a, s)$\" in sec 4.1, we meant that we adopt the implied quality function definition $Q_\\theta(X_{t-1},X_{t},C)=\\beta\\log(\\pi_{\\theta}(X_{t-1},X_{t},C)/\\pi_{ref}(X_{t-1},X_{t},C)$ based on the relation $Q^*(X_{t-1},X_{t},C)= \\beta\\log(\\pi^*_\\theta(X_{t-1}|X_{t},C)/\\pi_{ref}(X_{t-1}|X_{t},C))$, as well as the connection between $Q^*$ and the  RLHF objective established by D3PO.  We provide further details in the main rebuttal. If you have any additional questions, feel free to let us know and we are happy to answer them in discussion period.\n\n**Secondly, the substitution $Q_{ref}=\\beta D_{KL}$ is not very well motivated.**\n\nThe reference point of KTO is defined as $z_0=E_D[r_\\theta(X_0,C)]$, which is the expected reward over some distribution $D$ of $(X_0,C)$. This definition stems from the assumption that \"rather than having just one dispreferred generation serve as the reference point z0\", \"humans judge the quality of [generation] in relation to all possible input-output pairs.\" \n\nSimilarly, we define $Q_{ref}$ as $E_{D'}[Q_\\theta(X_{t-1},X_{t},C)]$ over some distribution $D'$ of $X_{t-1},X_{t},C,t$. KTO set $D$ to be a uniform distribution over the input dataset, and the reference point simplifies to the expected KL divergence. Following an identical derivation, we can establish that  $Q_{ref}$ in Diffusion-KTO formulation simplifies to the expected KL divergence. \n\nWe provide further details in the main rebuttal. We will add these details in future versions and apologize for any confusion caused by this omission. \n\nNote: In the above discussion, we replace the term $l(y)$, the normalization factor in the definition of $r_\\theta(X_0,C)$, with a constant $\\beta$ following Eq(6) of KTO paper for simplicity. We also replace the notation $x,y$ to $C,X_0$ for consistency with notations of diffusion models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1CTpCmbabU", "reviewer": "Reviewer_YybC", "summary": "This paper introduces Diffusion-KTO, a novel approach for aligning text-to-image diffusion models with human preferences using per-image binary feedback (like likes/dislikes) rather than pairwise preference data. The key contributions include:\n\n1. Extending the human utility maximization framework used to align language models to the domain of diffusion models.\n\n2. Developing an objective that allows training on per-image binary feedback rather than pairwise preferences, enabling the use of abundant internet data like likes/dislikes.\n\n3. Demonstrating through experiments that Diffusion-KTO outperforms existing alignment approaches like Diffusion-DPO, as judged by both automated metrics and human evaluators.\n\n4. Showing that Diffusion-KTO can align models to specific user preferences using synthetic experiments.\n\nThe authors fine-tune Stable Diffusion models using their Diffusion-KTO objective on datasets like Pick-a-Pic. They evaluate the aligned models using automated metrics and human studies, comparing them against baselines like supervised fine-tuning and other alignment methods. The results indicate that Diffusion-KTO produces images that are preferred by humans and score higher on various automated metrics compared to existing approaches, while only requiring simpler per-image feedback data.\n\nThe paper also discusses limitations, broader impacts, and potential misuse of the technology. Overall, Diffusion-KTO presents a new framework for improving text-to-image models using readily available preference signals.", "questions": "Did you use the trainers from the diffusers library to base your model on?\n\nHow does the effectiveness of Diffusion-KTO vary with different types of prompts? Is it more effective for certain categories of images or styles of prompts?\n\nHave you considered extending the method to incorporate other types of feedback beyond binary signals? For instance, could it be adapted to use scalar ratings or textual feedback?", "limitations": "The paper includes a dedicated \"Limitations\" section (Section 7), which addresses several key points:\n1. It acknowledges that Diffusion-KTO inherits limitations of the base text-to-image models.\n2. It notes that the preference data used (Pick-a-Pic dataset) may contain biases or inappropriate content.\n3. The authors recognize that the choice of utility function remains an open question.\n4. They acknowledge that their model may propagate negative stereotypes present in the training data.\n\nThe authors discuss both potential positive and negative societal impacts:\nPositive impacts:\n1. Improved alignment of text-to-image models with human preferences.\n2/ Potential for personalized image generation systems.\n\nNegative impacts:\n1. Risk of propagating biases present in preference data.\n2. Potential for misuse in generating inappropriate or harmful imagery.\n3. Concerns about the model inheriting weaknesses of the base text-to-image model, including generating images that reflect negative stereotypes.\n\nThe authors have made a good-faith effort to address limitations and societal impacts. They've covered key points such as data biases, potential misuse, and ethical considerations in experimentation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates originality in several ways:\n1. It extends the utility maximization framework from language models to diffusion models, representing a novel cross-domain application of ideas.\n2. It introduces a new alignment objective that can work with per-image binary feedback rather than pairwise preferences, opening up new possibilities for using abundant internet data.\n3, The approach allows for customization of text-to-image models to individual user preferences, which is an innovative direction in this field.\n\nIn terms of the quality of the approach and execution:\n1. The authors provide a comprehensive evaluation, using both automated metrics and human studies to assess their method.\n2. They compare against multiple baselines and state-of-the-art methods, demonstrating rigor in their experimental design.\n3. The paper includes ablation studies. \n4. The authors are transparent about limitations and potential negative impacts,\n\nThe paper is generally well-structured and clear:\n1. The methodology is explained in detail, with the objective function presented.\n2. Visual results are provided to illustrate the improvements, aiding in understanding.\n3. The paper includes a detailed appendix with additional results and implementation details, supporting reproducibility.\n\nThe work is significant in how it improves text-to-image models in the setting when there is no preference data, but rather binary good/bad examples.  The ability to customize models to individual preferences could have broad implications for personalized AI systems.", "weaknesses": "The paper acknowledges that the method may learn biases from skewed preference data, but doesn't provide a detailed analysis of this issue. A more thorough examination of how different biases in the feedback data affect the aligned model would be valuable.\n\nIn some sense, the method itself is not novel as it has been just applied to the text-to-image setting where other preference-tuning methods have shown promise and similar methods from the NLP research can potentially be applied to this setting in the future, e.g. BCO or SimPO etc.\n\nThe experiments primarily focus on Stable Diffusion v1-5 and v2-1. Including a wider range of diffusion models, as they are generally available via the diffusers library would better demonstrate the generalizability of the approach. \n\nWhile the paper demonstrates the potential for personalizing models to individual preferences, this is only shown through synthetic experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review! We are excited to hear that you found our method innovative and significant to improving text-to-image models. We are also happy to hear that you appreciated our comprehensive experiments and overall presentation. Please find our responses below.\n\n\n**A more thorough examination of how different biases in the feedback data affect the aligned model would be valuable.**\n\nTo understand the effect of aligning with Pick-a-Pic v2, which is known to contain some NSFW content, we run a CLIP-based NSFW safety checker on images generated using test prompts from Pick-a-Pic v2 and HPSv2. For Pick-a-Pic prompts, 5.4% of Diffusion-KTO generations are marked NSFW, and 4.4% of SDv1-5 generations are marked NSFW. For HPSv2 prompts, which are safer, 1.3% of Diffusion-KTO generations are marked NSFW, and 1.0% of SD v1-5 generations are marked NSFW. Overall, training on the Pick-a-Pic dataset leads to a marginal increase in NSFW content. We observe similar trends for Diffusion-DPO, which aligns with the same preference distribution (5.8% NSFW on Pick-a-Pick & 1.3% NSFW on HPSv2). We would like to emphasize that our method is agnostic to the choice of preference dataset, as long as the data can be converted into binary per-sample feedback. We used Pick-a-Pic because of its size and to fairly compare with related works.\n\n**In some sense, the method itself is not novel... similar methods from the NLP research can potentially be applied to this setting in the future, e.g. BCO or SimPO etc.** \n\nRegarding BCO and SimPO, we agree that these works show promising results in the NLP domain. However we find it infeasible to implement and evaluate these methods for text-to-image generation in such a short rebuttal period. We will investigate the effectiveness of these approaches in future works. Please see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for further details regarding novelty.\n\n**Including a wider range of diffusion models, as they would better demonstrate the generalizability of the approach.** \n\nWe provide results using two different models (SD v1-5 and SD v2-1) in our main paper and Appendix. These results highlight the generality of Diffusion-KTO for various text-to-image diffusion models. We agree that it would be interesting to see how our method works for other state-of-the-art models. However, since recent models use more complicated architectures (e.g., Multi-Modal Diffusion Transformer of SD v3.0) and significantly more parameters, it is not feasible to complete these experiments within a week. We leave these endeavors for future work.\n\n\n**While the paper demonstrates the potential for personalizing models to individual preferences, this is only shown through synthetic experiments.** \n\nWhile binary preferences are easier to collect than paired preferences, curating high-quality personalized preference data is still expensive. We explored the possibility of scraping likes and dislikes from websites such as Artstation, but this does not comply with their terms of service. Additionally, it is hard to evaluate whether the model can effectively align with personalized preferences due to the diverse nature of human interests. In contrast, synthetic experiments are more controllable with measurable quantitative metrics such as aesthetic scores (Appendix C). For these reasons, we left the task of curating a high-quality personalized feedback dataset for future works.\n\n**Did you use the trainers from the diffusers library?** \n\nWe refer the reviewer to our sample code in supplementary material for implementation details. We do not use the trainers.\n\n**How does the effectiveness of Diffusion-KTO vary with different types of prompts?**\n\nBelow, we provide a per-style score breakdown using the prompts and their associated styles (Animation, Concept-art, Painting, Photo) in the HPSv2 test set. Across these metrics, our model performs best for \"painting\" and \"concept-art\" styles. We attribute this to our training data. Since Pick-a-Pic prompts are written by users, it will reflect their biases, e.g., a bias towards artistic content. Such biases are also noted by the authors of HPSv2 who state \"However, a significant portion of the prompts in the database is biased towards certain styles. For instance, around 15.0% of the prompts in DiffusionDB include the name “Greg Rutkowski”, 28.5% include “artstation”.\" We also observe that different metrics prefer different styles. For example, the \"photos\" style has the highest PickScore but the lowest ImageReward. With this in mind, we would like to underscore that our method, Diffusion-KTO, is agnostic to the preference distribution (as long as feedback is per-sample and binary) and, training on different, less biased preference data could avoid such discrepancies.\n\n|    Style    | Aesthetic |  PickScore  | ImageReward |  CLIP  |  HPS  |\n|:-----------:|:---------:|:------:|:------------:|:------:|:-----:|\n|    anime    |   5.493   | 21.569 |     0.716    | 34.301 | 0.368 |\n| concept-art |   5.795   | 21.011 |     0.804    | 33.141 | 0.359 |\n|  paintings  |   5.979   | 21.065 |     0.802    | 33.662 | 0.360 |\n|    photo    |   5.365   | 21.755 |     0.471    | 31.047 | 0.332 |\n\n**Have you considered extending the method to incorporate other types of feedback beyond binary signals?**\n\nWhile we have considered exploring a variety of human feedback, this work, Diffusion-KTO, focuses specifically on binary feedback. For continuous feedback, we show in Appendix C an example where Diffusion-KTO can effectively align with the preference distribution after the continuous feedback signal is thresholded into a binary one. Textual feedback is non-trivial to implement in such a short rebuttal period, and we leave this for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We are glad to hear that the reviewers recognize our strong experimental results (Reviewers MiLc, 7SUT, s9JR, YybC), the importance of learning from binary preference data (Reviewers MiLc, 7SUT, YybC), and the novelty of extending the utility maximization framework to the setting of diffusion models (Reviewers s9JR & YybC).

In this main rebuttal, we address the common concern of novelty and clarify some common confusions on theory formulation. For other specific questions, we refer the reviewers to individual rebuttals.

**Novel Contributions of Diffusion-KTO**

We would like to emphasize the novel contributions of our work. Our work is the first to extend the utility maximization framework to the setting of diffusion models (as noted by Reviewers s9JR & YybC). Unlike the original KTO paper, we explore the effects of various utility functions and provide an analysis of their performance (see Section 6 and Appendix D.4). As we are extending this framework to a novel domain (text-to-image diffusion models), we perform such experiments to avoid naively assuming that the Kahneman & Tversky model would work best. Last but not least, we would like to highlight the potential impact of our method. Diffusion-KTO demonstrates that text-to-image diffusion models can be aligned with human preferences using binary preference data and can even outperform methods that require paired data. This is a new capability that, to our knowledge, was not previously available as the best alignment methods required paired preference data. As a result, this opens a world of possibilities as binary feedback is abundant on the internet and can be collected at scale.


**Clarification of Formulation**

We apologize for these oversights. A diffusion process from an image $X_0$ to random noise $X_T$ induces a conditional distribution $P(X_0|C)=\prod_{t=1}^{T}P(X_{t-1}|X_t)P(X_T|C)$ where C is the prompt. $P(X_T|C)$ can be written as $P(X_T)$ which is an i.i.d. Gaussian distribution. We can induce a global policy $\pi(X_0|C)$ representing the whole sample process, as well as a local policy $\pi(X_{t-1}|X_{t},C)$ representing each sample step. A reward function $r(X_0,C)$ is a real-valued function that encodes human preference. Every possible $r(X_0,C)$ induces an optimal global policy $\pi^*(X_0|C)$ that maximizes this reward. A "quality" function $Q(X_{t-1},X_{t},C)$ is a real-valued function that similarly induces an optimal local policy $\pi^*(X_{t-1}|X_{t},C)$.

Conceptionally, $Q$ can be an arbitrary real-valued function. $Q^*$ is a special choice of $Q$ such that the induced local policy maximizes the expected global reward $r(X_0,C)$.  Hence the (Eqn. 4) is an approximation of RL objective (Eqn. 2).  D3PO shows the relation $Q^*(X_{t-1},X_{t},C)= \beta\log(\pi^*_\theta(X_{t-1}|X_{t},C)/\pi_{ref}(X_{t-1}|X_{t},C))$. Following MDP convention, we can consider ($X_{t},C$) as a state $s$ and $X_{t-1}$ as an action $a$, and use $(s,a)$ in the notation instead. These formulations are elaborated in Sec 4.1 of the D3PO paper, we will incorporate more details in the future version of our paper. 

In Diffusion-KTO, $Q$ and $Q_{ref}$ in (Eqn. 4) follow the definitions proposed in the original KTO paper. KTO adopted an axiomatic approach and defined the implicit reward function as $r_\theta(X_0,c)=\beta\log(\pi_{\theta}(X_0|C)/\pi_{ref}(X_0|C))$. The justification of this definition stems from classic prospect theory experiments, as well as the observation that $r_\theta$ is in the same equivalence class as the human-preference-induced reward function in (Eqn. 2) at optimal policy $\pi^*_\theta$ under RLHF objective (Eqn. 2).  This formulation can be found in Sec 3.2 of KTO paper. Unfortunately, this formulation is intractable as it involves the global policy $\pi(X_0|C)$, and cannot be directly applied. 

Similarly, we can define implicit quality function $Q_\theta(X_{t-1},X_{t},C)=\beta\log(\pi_{\theta}(X_{t-1},X_{t},C)/\pi_{ref}(X_{t-1},X_{t},C)$, since results of D3PO have established the relation of $Q^*$ and the optimal policy. 

The reference point of KTO is defined as $z_0=E_D[r_\theta(X_0,C)]$, which is the expected reward over some distribution $D$ of $(X_0,C)$. This definition stems from the assumption that "rather than having just one dispreferred generation serve as the reference point z0", "humans judge the quality of [generation] in relation to all possible input-output pairs." 

Similarly, we define $Q_{ref}$ as $E_{D'}[Q_\theta(X_{t-1},X_{t},C)]$ over some distribution $D'$ of $X_{t-1},X_{t},C,t$. KTO set $D$ to be a uniform distribution over the input dataset, and the reference point simplifies to the expected KL divergence. Following an identical derivation, we can establish that  $Q_{ref}$ simplifies to the expected KL divergence. We will add these details in future versions and apologize for any confusion caused by this omission. 

Note: In the above discussion, we replace the term $l(y)$, the normalization factor in the definition of $r_\theta(X_0,C)$, with a constant $\beta$ following Eq(6) of KTO paper for simplicity. We also replace the notation $x,y$ to $C,x_0$ for consistency with notations of diffusion models.

Meta-review: The paper introduces Diffusion-KTO, a method largely seen as a straightforward extension of KTO, which was originally applied to fine-tuning language models, now adapted for fine-tuning diffusion models. This approach could also be considered a variant of Diffusion-DPO, with the DPO component replaced by KTO. From a methodological standpoint, the novelty is somewhat limited. However, the paper is notable for being among the few that explore human preference-based optimization of diffusion models. It provides effective results and detailed comparisons with established methods, earning the favor of all four reviewers for acceptance.

During the post-rebuttal discussion phase, there was an in-depth dialogue between the Area Chair and the authors, focusing on areas for improvement. Key issues raised during the discussion included questioning the statistical significance and practical utility of the minor improvements reported in some of the tables, as well as acknowledging potential performance degradation due to domain shifts in the text prompts. The authors are encouraged to carefully consider these points when preparing the camera-ready version of the paper.
Decision: Accept (poster)