Aligning Diffusion Models by Optimizing Human Utility

Authors: shufan  li
konstantinos  kallidromitis
akash  gokul
yusuke  kato
kazuki  kozuka
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 19:51:04

Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Unlike previous methods, Diffusion-KTO does not require collecting pairwise preference data nor training a complex reward model. Instead, our objective uses per-image binary feedback signals, e.g. likes or dislikes, to align the model with human preferences. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit improved performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary preference signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Main paper: 
 introduction: 
 In the rapidly evolving field of generative models, aligning model outputs with human preferences remains a paramount challenge, especially for text-to-image (T2I) models. Large language models (LLMs) have made significant progress in generating text that caters to a wide range of human needs, primarily through a two-stage process: first, pretraining on noisy web-scale datasets, then fine-tuning on a smaller, preference-specific dataset. This fine-tuning process aims to align the generative model's outputs with human preferences, without significantly diminishing the capabilities gained from pretraining. Extending this fine-tuning approach to text-to-image models offers the prospect of tailoring image generation to user preferences, a goal that has remained relatively under-explored compared to its counterpart in the language domain.Recent works have begun to explore aligning text-to-image models with human preferences. These methods either use a reward model and a reinforcement learning objective [52, 33, 14] , or directly fine-tune the T2I model on preference data [48, 53] . However, these methods are restricted to learning from pairwise preference data, which consists of pairs of preferred and unpreferred images generated from the same prompt.While paired preferences are commonly used in the field, it is not the only type of preference data available. Per-sample feedback is a promising alternative to pairwise preference data, as the former is abundantly available on the Internet. Per-sample feedback provides valuable preference signals for aligning models, as it captures information about the users' subjective distribution of desired and undesired generations. For instance, as seen in Fig. 2 , given an image and its caption, a user can easily say if they like or dislike the image based on criteria such as attention to detail and fidelity to the prompt. While paired preference data provides more information about relative preferences, gathering such data is an expensive and time-consuming process in which annotators must rank images according to their preferences. In contrast, learning from per-sample feedback can utilize the vast amounts of per-sample preference data collected on the web and increases the applicability of aligning models with user preferences at scale. Inspired by these large-scale use cases, we explore how to directly fine-tune T2I models on per-image binary preference data.To address this gap, we propose Diffusion-KTO, a novel alignment algorithm for T2I models that operates on binary per-sample feedback instead of pairwise preferences. Diffusion-KTO extends the utility maximization framework shown in KTO [18] to the setting of diffusion models. Specifically, KTO bypasses the need to maximize the likelihood from paired preferences and, instead, directly optimizes an LLM using a utility function that encapsulates the characteristics of human decisionmaking. While KTO is easy to apply to large language models, we cannot immediately apply it to diffusion models as it would require sampling across all possible trajectories in the denoising process. Although existing works approximate this likelihood by sampling once through the reverse diffusion process, back-propagating over all sampling steps is extremely computationally expensive.To overcome this, we present a utility maximization objective that applies to each individual sampling step, circumventing the need for sampling through the entire reverse diffusion process.Our main contributions are as follows:• We generalize the human utility maximization framework used to align LLMs to the setting of diffusion models (Section 4).• Our method, Diffusion-KTO, facilitates alignment from per-image binary feedback. Thus, introducing the possibility of learning from human feedback at scale using the abundance of per-sample feedback that has been collected on the Internet.• Through comprehensive evaluations, we demonstrate that generations from Diffusion-KTO aligned models are generally preferred over existing approaches, as judged by human evaluators and preference models (Section 5).In summary, Diffusion-KTO offers a simple yet robust framework for aligning T2I models with human preferences that greatly expands the utility of generative models in real-world applications."Man walking in NYC"> "Portrait of a dog" "Man walking in NYC"Figure 2 : Diffusion-KTO aligns text-to-image diffusion models using per-image binary feedback. Existing alignment approaches (Left) are restricted to learning from pairwise preferences. However, Diffusion-KTO (Right) uses per-image preferences which are abundantly available on the Internet. As seen above, the quality of an image can be assessed independent of another generation for the same prompt. More importantly, such per-image preferences provide valuable signals for aligning T2I models, as demonstrated by our results. 
related works: 
 Text-to-Image Generative Models. Text-to-image (T2I) models have demonstrated remarkable success in generating high quality images that maintain high fidelity to the input caption [38, 54, 37, 7, 11, 26, 55, 8] . In this work, we focus on diffusion models [43, 44, 24] due to their popularity and open-source availability. While these models are capable of synthesizing complex, high quality images after pretraining, they are generally not well-aligned with the preferences of human users. Thus, they can often generate images with noticeable issues, such as poorly rendered hands and faces. We seek to address these issues by introducing a fine-tuning objective that allows text-to-image diffusion models to learn directly from human preference data.Improving Language Models using Human Feedback. Following web-scale pretraining, large language models are further improved by fine-tuning on a curated set of data (supervised fine-tuning) and then using reinforcement learning to learn from human feedback. Reinforcement learning from human feedback (RLHF) [2, 12, 13] , in particular, has been shown to be an effective means of aligning these models with user preferences [59, 5, 31, 30, 29, 45, 10, 3, 6, 22] . While this approach has been successful [1, 46] , the difficulties in fine-tuning an LLM using RLHF [36, 58, 49, 17, 21, 42, 4] has led to the development of alternative fine-tuning objectives [35, 18, 56, 57] . Along these lines, KTO [18] introduces a fine-tuning objective that trains an LLM to maximize the utility of its output according to the Kahneman & Tversky model of human utility [47] . This utility maximization framework does not require pairwise preference data and only needs per-sample binary feedback. In this work, we explore aligning diffusion models given binary feedback data. As a first step in this direction, we generalize the utility maximization framework to the setting of diffusion models.Improving Diffusion Models using Human Feedback. Before the recent developments in aligning T2I models using pairwise preferences, supervised fine-tuning was the popular approach for improving these models. Existing supervised fine-tuning approaches curate a dataset using preference models [39, 32] , pre-trained image models [41, 8, 16, 51, 50] , and/or human experts [15] , and fine-tune the model on this dataset. Similarly, many works have explored using reward models to fine-tune diffusion models via policy gradient techniques [19, 23, 9, 52, 14, 33, 28, 20] to improve aspects such as image-text fidelity. Similar to our work, ReFL [52] , DRaFT [14] , and AlignProp [33] align T2I diffusion models with human preferences. However, these methods require back-propagating the reward through the reverse diffusion sampling process, which is extremely expensive in memory. As a result, these works depend on techniques such as low-rank weight updates [25] and sampling from only a subset of steps in the reverse process, thus limiting their ability to fine-tune the model. In contrast, the Diffusion-KTO objective extends to each step in the denoising process, thereby avoiding such memory issues. More broadly, the main drawbacks of these reinforcement learning based approaches are: limited generalization, e.g. closed vocabulary [28, 20] , reward hacking [14, 33, 9] , and they rely on a potentially biased reward model. Since Diffusion-KTO trains directly on open-vocabulary preference data, we find that it can generalize to an open-vocabulary and avoids issues such as reward hacking. Recently, works such as Diffusion-DPO [48] and D3PO [53] present extensions of the DPO objective [35] to the setting of diffusion models. Diffusion-KTO shares similarities with Diffusion-DPO and D3PO, as we build upon these works to introduce a reward model-free alignment objective. However, unlike these works, Diffusion-KTO does not rely on pairwise preference data and, instead, uses only per-image binary feedback. 
diffusion models: 
 Denoising Diffusion Probabilistic Models (DDPM) [24] model the image generation process as a Markovian process. Given data x 0 , the forward process p(x t |x t-1 ) gradually adds noise to an initial image x 0 according to a variance schedule, until it reaches x T ∼ N (0, I). A generative model can be trained to learn the reverse process q θ (x t-1 |x t ) using the evidence lower bound (ELBO) objective:EQUATIONwhere λ(t) is a time-dependent weighting function and ϵ is the added noise.RLHF first fits a reward model r(x, c), for a generated sample x and input prompt c, to human preference data D, and then maximizes the expected reward of a generative model π θ while ensuring it does not significantly deviate from the initialization point π ref .It uses the following objective with a divergence penalty controlled by a hyperparameter β.EQUATIONThe authors of DPO [35] present an equivalent objective (Eq. ( 3)) using the implicit reward modelEQUATIONwhere Z(c) is the partition function, (x w , x l ) are pairs of winning and losing samples, and c is the input conditioning. Through this formulation, the model π θ can be directly trained in a supervised fashion without explicitly fitting a reward model.One of the challenges in adapting DPO to the context of diffusion models is that the likelihood π θ (x|c) is hard to optimize because each sample x is generated through a multi-step Markovian process.In particular, it requires computing the marginal distribution x1...x N π θ (x 0 , x 1 ...x N |c) over all possible path of the diffusion process, where π θ (x 0 , x 1 ...x N |c) = π θ (x N ) N i=1 π θ (x i-1 |x i , c). D3PO [53] adapted DPO by considering the diffusion process as a Markov Decision Process (MDP). In this setup, an agent takes an action a at each state s of the diffusion process. Instead of directly maximizing r(x, c), one can maximize Q(a, s) which assigns a value to each possible action a at a given state s in the diffusion process instead of the final outcome. This setup uses the local policy π(a|s), which represents a single sampling step. In this setup, D3PO [53] showed that the optimal solution Q * (a, s) satisfies the relationQ * (a, s) = β log π * θ (a|s)πref(a|s) for the optimal policy π * θ . This leads to the approximate objective:EQUATIONwhere d π is the state visitation distribution under policy π. Concretely, the action is a sampling step, and we can write Q(a, s) as Q(x t-1 , x t , c) and π(a|s) as π(x t-1 |x t , c). In decision theory, the expected utility hypothesis assumes that a rational agent makes decisions based on the expected utility of all possible outcomes of an action, instead of using objective measurements such as the value of monetary returns. Formally, given an action a and the set of outcomes O(a), the expected utility is defined as EU (a) = o∈O(a) p A (o)U (o), where p A is a subjective belief of the probability distribution of the outcomes and the utility function U (o) is a real-valued function.Prospect theory [47] further augments this model by asserting that the utility function is not defined solely on the outcome (e.g. the absolute gain in dollars), but also with respect to some reference point (e.g. current wealth). In this formulation, the utility function is defined asU (o, o ref ) for a reference outcome o ref .Based on this theory, KTO [18] proposed an alternative objective for aligning LLMs:EQUATIONwhere x is the output of the LLM, c is the input prompt, w(x) = 1 if x is desirable and w(x) = -1 otherwise, and λ(x) is a weighting function of samples. The divergence penalty is computed as the expectation of the KL divergence between the model distribution π θ (x ′ |c ′ ) and the reference distribution π ref (x ′ |c ′ ) over all input prompts c ′ in the dataset. This formulation uses the sigmoid function σ(x) as an approximation for the Kahneman-Tversky utility function which is concave in gain and convex in loss. Experiments showed that KTO aligned LLMs were able to outperform DPO aligned LLMs, and KTO is resilient towards noise in the preference data [18] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use publicly available datasets to train (Pick-a-Pic) and evaluate (Pick-a-Pic, HPS v2, PartiPrompts) (see Section 5) . Code is provided in the Supplemental material, and will be open-sourced.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
diffusion-kto: 
 Here, we propose Diffusion-KTO. Instead of optimizing the expected reward, we incorporate a non-linear utility function that calculates the utility of an action based on its value Q(a, s) with respect to the reference point Q ref .maxEQUATIONwhere U (v) is a monotonically increasing value function that maps the implicit reward to subjective utility. Practically, the local policy π θ (a|s) is a sampling step, and can be written as π θ (x t-1 |x t ).Applying the relation Q * (a, s) from Sec. 3.3, we get the objective maxEQUATIONFollowing KTO [18] , we can optimize the policy based on whether a given generation is considered "desirable" or "undesirable":EQUATIONwhere w(x 0 ) = ±1 if image x 0 is desirable or undesirable. We setQ ref = βD KL [π θ (a|s)||π ref (a|s)].Empirically, this is calculated by computing max(0,1 m log π θ (a ′ |s ′ ) π ref(a ′ |s ′ )) over a batch of unrelated pairs of (s ′ , a ′ ) following the KTO setup [18] in Eq. ( 5).While we incorporate the reference point aspect of the Kahneman-Tversky model, it is unclear if other assumptions about human behavior are applicable. It is also known that different people may exhibit different utility functions. Thus, we explore a wide range of utility functions. For presentation purposes, we center all utility functions U (x) around 0 by using U centered (x) = U (x) -U (0), such that U centered (0) = 0. This does not change the objective in Eq. ( 8) as the gradient and optimal policy are not affected. We experiment with the following utility functions:• Loss-Averse: We characterize a loss-averse utility function as any utility function that is concave (see U (x) plotted in blue in Figure 3 ). Using this utility function, the Diffusion-KTO objective can be considered as a variant of the Diffusion-DPO objective. While aligning according to this utility function follows a similar form to the Diffusion-DPO objective, our approach does not require paired preference data. • Risk-Seeking: Conversely, we define a risk-seeking utility function as any convex utility function (see U (x) plotted in yellow in Figure 3 ). A typical example of a risk-seeking utility function is the exponential function. However, its exploding behavior on (0, +∞) makes it hard to optimize. Instead, for this case, we consider U (x) = -log σ(-x). • Kahneman-Tversky model: Kahneman-Tversky's prospect theory argues that humans tend to be risk-averse for gains but risk-seeking for losses relative to a reference point. This amounts to a function that is concave in (0, +∞) and convex in (0, -∞). Following the adaptation proposed in KTO, we employ the sigmoid function U (x) = σ(x) (see U (x) plotted in red in Figure 3 ). Empirically, we find this utility function to perform best.Under the expected utility hypothesis, the expectation is taken over the subjective belief of the distribution of outcomes, not the objective distribution. In our setup, the dataset consists of unpaired samples x that are either desirable (w(x) = 1) or undesirable (w(x) = -1). Because we do not have access to additional information, we assume the subjective belief of a sample x is solely dependent on w(x). During training, this translates to a biased sampling process where each sample is drawn uniformly from all desirable samples with probability γ and uniformly from all undesirable samples with probability 1 -γ. 
experiments: 
 We comprehensively evaluate Diffusion-KTO through quantitative and qualitative analyses to demonstrate its effectiveness in aligning text-to-image diffusion models with a preference distribution.Further comparisons, such as the performance when using prompts from different datasets, the results of our ablations, and implementation and evaluation details can be found in the Appendix. Additionally, in the Appendix, we report the results of synthetic experiments which highlight that Diffusion-KTO can be used to cater T2I diffusion models to the preferences of a specific user. The code used for this work will be made publicly available and is available in the Supplementary material.Implementation Details. We fine-tune Stable Diffusion v1-5 (SD v1-5) [39] (CreativeML Open RAIL-M license) with the Diffusion-KTO objective, using the Kahneman-Tversky utility function, on the Pick-a-Pic v2 dataset [27] (MIT license). The Pick-a-Pic dataset consists of paired preferences in the form of (preferred image, non-preferred image, input prompt). Since Diffusion-KTO does not require paired preference data, we partition the images in the training data. If an image is labelled Table 1 : Automatic win-rate (%) for Diffusion-KTO (SD v1-5) in comparison to existing alignment approaches using prompts from the Pick-a-Pic v2 test set. We use off-the-shelf models, e.g. preference models such as PickScore, to compare generations and determine a winner based on the method with the higher scoring generation. Diffusion-KTO drastically improves the alignment of the base SD v1-5 and demonstrates significant improvements in alignment when compared to existing approaches. Win rates above 50% are bolded. as preferred at least once, we consider it a desirable sample, otherwise we consider the sample undesirable. In total, we train with 237,530 desirable samples and 690,538 undesirable samples.Evaluation Details. We evaluate the effectiveness of Diffusion-KTO by comparing generations from our Diffusion-KTO aligned model to generations from existing methods using automated preference metrics and user studies. For our results using automated preference metrics, we present win-rates (how often the metric prefers Diffusion-KTO's generations versus another method's generations) using the LAION aesthetics classifier [40] (MIT license), which is trained to predict the aesthetic rating a human would give to the provided image, CLIP [34] (MIT license), which measures image-text alignment, and PickScore [27] (MIT license), HPS v2 [50] (Apache-2.0 license), and ImageReward [52] (Apache-2.0 license) which are caption-aware models that are trained to predict a human preference score given an image and its caption. Additionally, we perform user studies to compare Diffusion-KTO with existing baselines. In our user study, we ask judges to assess which image they prefer (Which image do you prefer given the prompt?) given an image generated by our Diffusion-KTO model and an image generated by the other method for the same prompt.We compare Diffusion-KTO to the following baselines: Stable Diffusion v1-5 (SD v1-5), supervised fine-tuning (SFT), conditional supervised fine-tuning (CSFT), AlignProp [33] , D3PO [53] and Diffusion-DPO [48] . Our SFT baseline fine-tunes SD v1-5 on the subset of images that are labelled as preferred using the standard denoising objective. Our CSFT baseline, similar to the approach introduced into HPS v1 [51] , appends a prefix to each prompt ("good image", "bad image") and finetunes SD v1-5 using the standard diffusion objective while training with preferred and non-preferred samples independently. To compare with D3PO (MIT license), we fine-tune SD v1-5 using their officially released codebase. For AlignProp (SD v1-5) (MIT license) and Diffusion-DPO (SD v1-5) (Apache-2.0 license), we compare with their officially released checkpoints.Table 1 provides the win-rate, per automated metrics, for Diffusion-KTO aligned SD v1-5 and the related baselines. Diffusion-KTO markedly improves alignment of SD v1-5, with win-rates of up to 87.2%. Results from our user study (Figure 4 ) confirm that human evaluators consistently prefer the generations of Diffusion-KTO to that of the base SD v1-5 (75% win-rate in favor of Diffusion-KTO). Further, Diffusion-KTO aligned models outperform related alignment approaches such as AlignProp, D3PO, and Diffusion-DPO. Diffusion-KTO significantly outperforms Diffusion-DPO on metrics such as LAION Aesthetics, PickScore, and HPS v2 while performing comparably in terms of other metrics. We also find that human judges prefer generations from our Diffusion-KTO model (72% win-rate versus SFT and 69% win-rate versus Diffusion-DPO) over that from SFT and Diffusion-DPO. This highlights the effectiveness of our utility maximization objective and shows that not only can Diffusion-KTO learn from per-image binary feedback, but it can also outperform models training with pairwise preference data.In Fig. 5 , we showcase a visual comparison of Diffusion-KTO with existing approaches for preference alignment. As seen in the first row, most models are misguided by the "sunlight" reference in the prompt and produce in a dark image. Diffusion-KTO demonstrates a focus on the bird, which is the central object in the caption and provides a better quality result over Diffusion-DPO, which doesn't include any visual indication for the "sunlight". In the second row of images, our Diffusion-KTO aligned model is able to successfully generate a "turtle baby sitting in a coffee cup". Methods such as Diffusion-DPO, in this example, have an aesthetically pleasing result but ignore key components of the prompt (e.g. "turtle","night"). On the other hand, SFT and CSFT follow the prompt but provide less appealing images. For the third prompt, which is a detailed description of a woman, the output from our Diffusion-KTO model provides the best anatomical features, symmetry, and pose compared to the other approaches. Notably, for this third prompt, the generation of the Diffusion-KTO model also generated a background is more aesthetically pleasing. The final row uses a difficult prompt that requires a lot of different objects in a niche art style. While all models were able to depict the right style, i.e. geometric art, only the Diffusion-KTO generation includes key components such as the "moon," "flying fish," and "fisherman" objects. These examples demonstrate that Diffusion-KTO significantly increases the visual appeal of generated images while improving image-text alignment. We sample from MLP diffusion models trained using various alignment objectives. We find that using the Kahneman-Tversky utility function leads to the best performance in terms of aligning with the desirable distribution and avoiding the undesirable distribution. 
analysis: 
 To further study the effect of different utility functions, we conduct miniature experiments to observe their impact. We assume the data is two-dimensional, and the pretraining data follows a Gaussian distribution centered at (0.5, 0.8) with variance 0.04. We sample desirable samples from a Gaussian distribution P d centered at (0.3, 0.8), sample undesirable samples from a Gaussian distribution P u centered at (0.3, 0.6), and the variance of both distributions is 0.01. We pretrain small MLP diffusion models, using the standard diffusion objective on the pretraining data, and then fine-tune using various utility functions. We sample 3500 data points from the trained model. Figure 6 shows that the risk-averse utility function (used by Diffusion-DPO) has a strong tendency to avoid loss to the point that it deviates from the distribution of desirable samples. The risk-seeking utility function behaves roughly the same as the SFT baseline and shows a strong preference for desirable samples at the cost of tolerating some undesirable samples. In comparison, our objective achieves a good balance.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: For details of the data split, see Section 5. For details about hyperparameters, optimizers, etc., see Appendix A.1. These details are also documented in our code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitations: 
 While Diffusion-KTO significantly improves the alignment of text-to-image diffusion models, it suffers from the shortcomings of T2I models and related alignment methods. Specifically, Diffusion-KTO is trained on preference data from the Pick-a-Pic dataset which contains prompts submitted by online users and images generated using off-the-shelf T2I models. As a result, the preference distribution in this data may be skewed toward inappropriate or otherwise unwanted imagery. Furthermore, in this work, we examined three main models of human utility, from which we have concluded the Kahneman-Tversky model to perform best based on empirical results. However, we believe that the choice of utility function, as well as the underlying assumptions behind such functions, remains an open question. Additionally, since Diffusion-KTO fine-tunes a pretrained T2I model, it inherits the weaknesses of this model, including generating images that reflect and propagate negative stereotypes. Despite these limitations, Diffusion-KTO presents a broader framework for improving and aligning diffusion models from per-image binary feedback.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: See Appendix A.2 and B.1.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we introduced Diffusion-KTO, a novel approach to aligning text-to-image diffusion models with human preferences using a utility maximization framework. This framework avoids the need to collect pairwise preference data, as Diffusion-KTO only requires simple per-image binary feedback, such as likes and dislikes. We extend the utility maximization approach, recently introduced to align LLMs, to the setting of diffusion models and explore various utility functions. Diffusion-KTO aligned diffusion models lead to demonstrable improvements in image preference and image-text alignment when evaluated by human judges and automated metrics. While our work has empirically found the Kahneman-Tversky model of human utility to work best, we believe that the choice of utility functions remains an open question and promising direction for future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We only release training code. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Assets used in this work are cited and the licenses are explicitly mentioned (see Section 5 and Appendix B). Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Justification: We perform a user study to evaluate the quality of Diffusion-KTO's generations. We provide full details of this evaluation in Appendix A.2 (Evaluation Details). Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: Yes, risks are described and disclosed to participants (plus we take additional safety measures to minimize exposure). We also obtain approval from our institution. See Appendix A.2. 
 
 Appendix: B.1 Performance in terms of Average Score per Preference Models.In Table 2 , we report the average score (and 95% confidence interval of the mean) given by each metric used in our automated evaluations. For each method (using SD v1-5), we sample 5 generations per prompt, for a total of 2500 generations (i.e. N=2500). As seen below, Diffusion-KTO exhibits state-of-the-art performance according to numerous metrics while performing comparably to the state-of-the-art in the remaining metrics. These results demonstrate the effectiveness of Diffusion-KTO for aligning T2I diffusion models with human preferences. Additionally, we report the 95% confidence interval of win rate in Table 3 In addition to the results on the Pick-a-Pic test set reported in Table 1 , we provide additional results on HPS v2 (Apache-2.0 license) and PartiPrompts (Apache-2.0 license) datasets in Table 4 and Table 5 .Results show that Diffusion-KTO outperforms existing baselines on a diverse set of prompts. We perform additional experiments, this time using Stable Diffusion v2-1 (SD v2-1) [39] (CreativeML Open RAIL++-M license). We fine-tune SD v2-1 using Diffusion-KTO with the same hyperparameters and compute listed in Appendix A. In Table 7 , we compare Diffusion-KTO (SD v2-1) with SD v2-1 and Diffusion-DPO (SD v2-1). To compare with Diffusion-DPO, we fine-tune SD v2-1 using the official codebase released by the authors. As seen in Table 7 , Diffusion-KTO outperforms the SD v2-1 base model and Diffusion-DPO according to most metrics while performing comparably in others. This highlights the generality of Diffusion-KTO, as it is an architecture agnostic approach to improving the alignment of any text-to-image diffusion model.Per-image binary feedback data is easy to collect and is abundantly available on the internet in the forms of likes and dislikes. This opens up the possibility of aligning T2I models to the preferences of a specific user. While users may avoid the tedious task of providing pairwise preference, Diffusion-KTO can be used to easily align a diffusion model based on the images that a user likes and dislikes.Here, we conduct synthetic experiments to demonstrate that Diffusion-KTO can be used to align models to custom preference heuristics, in an attempt to simulate the preference of a select user.We experiment using two custom heuristics to mock the preferences of a user. These heuristics are:(1) preference for red images (i.e. red filter preference) and ( 2) preference for images with high aesthetics score. For these experiments, we fine-tune Stable Diffusion v1-5 using the details listed in A.1. For the red filter preference experiment, we use (image, caption) pairs from the Pick-a-Pic As expected, these generations tend to introduce further detail (such as the woman's facial features) and add additional colors and textures. The second row displays images from a model that is trained to learn a preference for red images, and Diffusion-KTO learns to add this preference for red images while minimally changing the content of the image. We additionally plot the aesthetic score and redness score throughout the training. The redness score is calculated as the difference between the average intensity of the red channel and the average intensity in all channels.v2 training set and enhance the red channel values to generate desired samples (original images are considered undesirable). For the aesthetics experiment, we train with images from the Pick-a-Pic v2 training set. We use images with an aesthetics score ≥ 7 as desirable samples and categorize the remaining images as undesirable. Figure 7 provides visual results depicting how Diffusion-KTO can align with arbitrary user preferences. For the aesthetics preference experiment (Figure 7 row 1), we see that generations contain finer detail and additional colors and textures, in comparison to the baseline image, both of which are characteristics of a high scoring images per the LAION aesthetics classifier. Similarly, Diffusion-KTO also learns the preference for red images in the red filter preference experiment (Figure 7 row 2). While we experiment with simple heuristics, these results show the efficacy of Diffusion-KTO in learning arbitrary preferences using only per-sample binary feedback.We explored various design choices of Diffusion-KTO in this section. We report the mean PickScore on the HPS v2 dataset, which consists of 3500 prompts and is the largest amongst Pick-a-Pic, PartiPrompts, and HPS v2. We show results in Fig. 8 win win+intermediate In converting the Pick-a-Pic dataset, which consists of pairwise preferences, to a dataset of per-image binary feedback, we consider two possible options. The first option is to categorize a sample as desired if it is always preferred across all pairwise comparisons (win), with all other samples considered undesirable. The second option additionally incorporate any samples as desirable samples if they are labelled as preferred in at least one pairwise comparison. Results show that the latter option is a better strategy.During the training process, we sample some images from the set of all desired images and some images from the set of all undesired images. This ratio is set to a fixed value γ to account for potentially imbalanced dataset. We find that sampling 80% of the images in a minibatch from the set of desired images achieves the optimal result.We experiment with different values between 2000 and 5000 for β, the parameter controlling the deviation from the policy. We show that there is a rise in performance with the increase in value but with diminishing returns, indicating an optimal score around 5000.We explored the effect of various utility function described in the main paper. Particularly, we consider Loss-Averse U (x) = log σ(x), KTO U (x) = σ(x) and Risk-Seeking U (x) = -log σ(-x).Results show that KTO is the optimal result.We visualize the utility function and its first order derivative in Fig. 9 . Intuitively, Loss-Aversion will reduce the update step during the training when reward is sufficiently high, Risk-Seeking will reduce the update step during the training when reward is low. KTO will reduce the update step if the reward is either sufficiently high or sufficiently low. This makes KTO more robust to noisy and sometimes contradictory preferences.Kahneman-Tversky Loss-Aversion Risk-Seeking Figure 9 : Utility functions visualizations. We visualize the utility function and its first order derivative. Intuitively, Loss-Aversion will reduce the update step during the training when reward is sufficiently high, Risk-Seeking will reduce the update step during the training when reward is low. KTO will reduce the update step reward is either sufficiently high or sufficiently low. The functions are centered by a constant offset so that U (0) = 0 for better visibility. The constant offset does not contribute to the gradient and, thus, has no effect to training.We provide further visual comparisons between Diffusion-KTO aligned SD v1-5 and the off-the-shelf SD v1-5 (Fig. 10 ). We find that Diffusion-KTO aligned models improve various aspects including photorealism, richness of colors, and attention to fine details. We also provide visual examples of failure cases from Diffusion-KTO aligned SD v1-5 (Fig. 13 ).To understand the effect of aligning with Pick-a-Pic v2, which is known to contain some NSFW content, we run a CLIP-based NSFW safety checker on images generated using test prompts from Pick-a-Pic v2 and HPSv2. For Pick-a-Pic prompts, 5.4% of Diffusion-KTO generations are marked NSFW, and 4.4% of SDv1-5 generations are marked NSFW. For HPSv2 prompts, which are safer, 1.3% of Diffusion-KTO generations are marked NSFW, and 1.0% of SD v1-5 generations are marked NSFW. Overall, training on the Pick-a-Pic dataset leads to a marginal increase in NSFW content. We observe similar trends for Diffusion-DPO, which aligns with the same preference distribution (5.8% NSFW on Pick-a-Pick and 1.3% NSFW on HPSv2). We would like to emphasize that our method is agnostic to the choice of preference dataset, as long as the data can be converted into binary per-sample feedback. We used Pick-a-Pic because of its size and to fairly compare with related works.In general, we encourage fair and responsible use of our algorithm andIn this section, we discuss the sources of the prompts used in Fig. 5 . To highlight the advantage of Diffusion-KTO in real-world scenarios, we refer to user prompts shared over the internet. In particular, we chose Playground AI (https://playground.com), where users share generated images alongside their prompts. These prompts reflect typical use cases in real-world scenarios and is generally similar to the "good" samples in the Pick-a-Pic dataset, which is also written by human labelers.SD v1-5 In the first instance (top-left) the dolphin is correctly shown "leaping out of the water", however, the coral reef is at the surface of the water not at the bottom of the sea. The second image (top-right) shows a dragon at the top of the mountain, however the dragon's body seems to merge with the stone. For the bottom left image, the caption is "Wish you were here" which is incorrectly written. The final image depicts a Pokémon trainer in a valley accurately, but it is missing a "group of Jigglypuff". We note that Diffusion-KTO aligned models inherit the limitations of existing text-to-image models.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper introduces Diffusion-KTO, a method largely seen as a straightforward extension of KTO, which was originally applied to fine-tuning language models, now adapted for fine-tuning diffusion models. This approach could also be considered a variant of Diffusion-DPO, with the DPO component replaced by KTO. From a methodological standpoint, the novelty is somewhat limited. However, the paper is notable for being among the few that explore human preference-based optimization of diffusion models. It provides effective results and detailed comparisons with established methods, earning the favor of all four reviewers for acceptance.

During the post-rebuttal discussion phase, there was an in-depth dialogue between the Area Chair and the authors, focusing on areas for improvement. Key issues raised during the discussion included questioning the statistical significance and practical utility of the minor improvements reported in some of the tables, as well as acknowledging potential performance degradation due to domain shifts in the text prompts. The authors are encouraged to carefully consider these points when preparing the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes Diffusion-KTO, which extends the KTO theory to develop a preference optimization algorithm for
strength_weakness: strength_weakness: * While I enjoyed the paper, the novelty is somewhat limited. In particular, the paper is a relatively direct combination of the Diffusion DPO type methods and novel DPO style losses.
questions: questions: * Can you include an example without the use of a preference pair dataset? Pick-a-pic is paired, but the main benefit of KTO should be the ability to extend beyond paired data (which might be useful for diffusion models since generation is expensive).
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents Diffusion-KTO, a novel preference learning algorithm for diffusion models. The proposed preference learning algorithm is based on Kahneman & Tversky Optimization (KTO). Diffusion-KTO enables aligning a diffusion model using only binary pointwise feedbacks, improving data efficiency and robustness. The experiments show promising results.
strength_weakness: strength_weakness: * The proposed algorithm is almost a simple application of KTO to diffusion models. The originality of the contribution is, therefore, not very strong. Of course, the combination of diffusion models and KTO is indeed novel; somebody else might have done it soon if this paper hadn't.  I acknowledge the novelty, but the paper would have been stronger with more original ideas.
* The paper has to be more self-contained. The paper borrows a lot of components from existing works, such as KTO and D3PO, and the borrowed elements should be explained well. Section 3.3 is particularly dissatisfying. There is no rigorous definition of Q, no definition of pi^*, and no explanation of why Equation 4 is an approximate objective. In a similar spirit, Section 4.1 could be more informative, such as providing an explanation for why Q_ref has to be set as proposed.
* The major concern regarding the experiment is that the number of human subjects and human responses are not disclosed (please correct me if I am wrong). This information is required to judge the uncertainty of the winning rate presented in Figure 4. It would be great if the uncertainty of the winning rate could also be provided.
questions: questions: See weakness. I would adjust my rating based on the response regarding the weaknesses of the paper.
limitations: limitations: The paper addresses its limitations.
paper_summary: paper_summary: This paper (DKTO) combines D3PO and KTO. D3PO let's us apply DPO to Diffusion models using pairwise preferences, and KTO is a way to align generative models (specifically autoregressive LLMs) using pointwise preference. For example, this gives us a way to tune text-conditioned, image generative models from thumbs-up, thumbs down or star ratings type of bandit feedback, without training a secondary reward model.

DKTO works by optimizing the expected utility of the advantage of the new policy versus a reference policy. And empirically, its results on finetuned Stable diffusion are pretty compelling. However, the mathematical derivation of DKTO is a bit messy and not rigorous.
strength_weakness: strength_weakness: Section 4.1 which derives the actual diffusion KTO method is really short and seems wrong on first glance. To go from eq (6) to Eq(7) the relation Q* = β (log π_θ - log π_ref) is substituted but that relation is lifted from the D3PO paper that derived it under the assumption that the policy optimization objective is E[Q*] - β KL[π_θ || π_ref] . But the diffusion KTO objective is different so it's not clear why prop 1 from the DKTO paper still applies.

Secondly, the substitution Q_ref = β KL is not very well motivated. The notation Q_ref suggests that it should not have had any dependence on π_θ at all !! Why is the KL divergence a reasonable proxy for Q_ref ? 

Overall, I think the paper in its current form is weak from a theory/conceptual point of view, and the method needs to be motivated a lot more cleanly. Please let me know if I have made a mistake, I'll gladly update my scores.
questions: questions: See weaknesses.
limitations: limitations: Yes, authors have addressed the limitations adequately.
paper_summary: paper_summary: This paper introduces Diffusion-KTO, a novel approach for aligning text-to-image diffusion models with human preferences using per-image binary feedback (like likes/dislikes) rather than pairwise preference data. The key contributions include:

1. Extending the human utility maximization framework used to align language models to the domain of diffusion models.

2. Developing an objective that allows training on per-image binary feedback rather than pairwise preferences, enabling the use of abundant internet data like likes/dislikes.

3. Demonstrating through experiments that Diffusion-KTO outperforms existing alignment approaches like Diffusion-DPO, as judged by both automated metrics and human evaluators.

4. Showing that Diffusion-KTO can align models to specific user preferences using synthetic experiments.

The authors fine-tune Stable Diffusion models using their Diffusion-KTO objective on datasets like Pick-a-Pic. They evaluate the aligned models using automated metrics and human studies, comparing them against baselines like supervised fine-tuning and other alignment methods. The results indicate that Diffusion-KTO produces images that are preferred by humans and score higher on various automated metrics compared to existing approaches, while only requiring simpler per-image feedback data.

The paper also discusses limitations, broader impacts, and potential misuse of the technology. Overall, Diffusion-KTO presents a new framework for improving text-to-image models using readily available preference signals.
strength_weakness: strength_weakness: The paper acknowledges that the method may learn biases from skewed preference data, but doesn't provide a detailed analysis of this issue. A more thorough examination of how different biases in the feedback data affect the aligned model would be valuable.

In some sense, the method itself is not novel as it has been just applied to the text-to-image setting where other preference-tuning methods have shown promise and similar methods from the NLP research can potentially be applied to this setting in the future, e.g. BCO or SimPO etc.

The experiments primarily focus on Stable Diffusion v1-5 and v2-1. Including a wider range of diffusion models, as they are generally available via the diffusers library would better demonstrate the generalizability of the approach. 

While the paper demonstrates the potential for personalizing models to individual preferences, this is only shown through synthetic experiments.
questions: questions: Did you use the trainers from the diffusers library to base your model on?

How does the effectiveness of Diffusion-KTO vary with different types of prompts? Is it more effective for certain categories of images or styles of prompts?

Have you considered extending the method to incorporate other types of feedback beyond binary signals? For instance, could it be adapted to use scalar ratings or textual feedback?
limitations: limitations: The paper includes a dedicated "Limitations" section (Section 7), which addresses several key points:
1. It acknowledges that Diffusion-KTO inherits limitations of the base text-to-image models.
2. It notes that the preference data used (Pick-a-Pic dataset) may contain biases or inappropriate content.
3. The authors recognize that the choice of utility function remains an open question.
4. They acknowledge that their model may propagate negative stereotypes present in the training data.

The authors discuss both potential positive and negative societal impacts:
Positive impacts:
1. Improved alignment of text-to-image models with human preferences.
2/ Potential for personalized image generation systems.

Negative impacts:
1. Risk of propagating biases present in preference data.
2. Potential for misuse in generating inappropriate or harmful imagery.
3. Concerns about the model inheriting weaknesses of the base text-to-image model, including generating images that reflect negative stereotypes.

The authors have made a good-faith effort to address limitations and societal impacts. They've covered key points such as data biases, potential misuse, and ethical considerations in experimentation.

Comments: title: Answer to AC Question #2 out of 2
Answer to AC Question #1 out of 2
Thanks for reply.
Direct Comparison
Clarification
practical significance vs SFT baseline
novelty
Further Discussion of "Practical Significance"
Response to statistical significance of HPSv2 and Parti prompts.
Could the demonstrated empirical gains overcome the concerns on novelty?
comment: Hi
We thank the AC's constructive feedbacks and will provide these important details (e.g. statistic significance, details of Fig 5) in the camera-ready version. We find these discussions exceptionally helpful and will revise our paper accordingly.
I appreciate the authors' clarification on these subtle yet important points. I strongly encourage you to incorporate these discussions into your camera-ready version, as I believe this will help alleviate any doubts and enhance understanding.
Hi,

We are glad that the AC recognize these advantages. We sincerely apologize for any potential misunderstanding and miscommunications. We might have wrongly assumed the discussion around "practical significance" was based on perceived "substantial differences in numbers" as a criterion in lieu of "statistical significance", a stance we found to be not objectively grounded.  We are glad that this is not the case, and the confusions have been cleared away.

**Q:Further Clarification on Fig 5**

For Figure 5, **the images are not chosen by specific seeds, we just randomly generate an image per prompt**. However, the prompt, which is listed on the left most column, are not randomly selected from a dataset. They are manually written ones based on practical use cases on Playground AI, a website where users share their prompts. Typically, these prompts are highly specific.  Our observations show that given these highly specific prompt, (we have provided an example above), the images generated by each model will have small variations. Hence, the difference between models is fairly consistent across generations. We provided additional analysis of these prompts in the previous comment. Please feel free to let us know if you have any additional questions.

**Q: Reproducibility of Human Evaluation**

We agree that this is a limitation for human evaluations. We hope future works from the community can establish an open and reproducible ranking based on human evaluation like lmsys for LLMs. However, at the moment, we have provided a detailed description of the human evaluation protocol in appendix A.2, alongside designs to mitigate systematic biases (e.g. "not given any information about which methods are being compared and the order of methods (left or right) is randomized").  The statistic confidence which we provided in previous comments and our rebuttal to reviewer 7SUT in this case should be a good measurement for reproducibility.

**Q:It's important to consider that the method's advantages might be less pronounced under domain shifts.**

We agree with this observation. Then, it is critical to consider what is the distribution of prompts that best reflect a "practical use", as this is a key focus of this discussion.

Consider the common use case of generating people as a case study. The **PartiPrompts** "People" category contains prompts such as "a woman with long hair", "a boy going to school".

On **HPS v2**, examples of prompts involving people are "Blond-haired girl depicted in anime style", "A cyberpunk woman on a motorbike drives away down a street while wearing sunglasses"

**Pick-a-Pick** contains a mix of different styles and even in different languages. Examples are "A woman’s wrist with a thin golden bracelet, romantic city lights in the background", "Eighteen year-old girl, pale skin, smooth skin, black jacket, ......  Chris Bachalo, Belén Orteg".

Most prompts on **Playground AI** are similar to longer ones in Pick-a-Pick, such as "Transparent-skinned creature showing complex anatomical structures intertwined with botanical patterns,... Lynn Maycroft's work, cinema." 

Unlike the previous three academic datasets, these are prompts used by actual paid subscriber who are willing to pay 15-35 USD per month for T2I models. The authors believe this distribution best illustrate the "practical usecase", which is why we use this for our qualitative results in Fig5. While we also made our best attempt to provide some quantitative analysis in previous comments, various engineering and legal complications of large-scale web-scrapping prevents us from scaling these analyses to a full-sized benchmark.

In summary, we agree that "advantages might be less pronounced under domain shifts", but we consider this a natural consequence of training on Pick-a-Pick. Given that KTO only requires binary preferences, it is easier to collect such preference data on the intended distribution (e.g. PartiPrompts). The authors hold that results presented in this paper have met the burden of "substantiate the empirical advantages". It can be reasonably expected that KTO should show similar improvements with proper preference data on any distribution of prompts.

The main contribution of Diffusion-KTO is a general and effective optimization algorithm, not the actual model trained on Pick-a-Pick dataset. The latter is an experiment to study the effectiveness of the former. We do not claim this specific model weight has stronger generalization capability on unseen concepts, but rather, it can reasonably generalize to "practical usecases" as discussed above. In fact, we believe it is unrealistic to expect any model (including SFT,DPO) that is trained on Pick-a-Pick prompts to generalize to prompts such as  "element", "concurrent lines" (which are concepts uniquely present in PartiPrompts). It is even debatable if human preferences make sense for such prompts. We appreciate the AC's detailed feedback and discussion.
As the AC for this paper, I have consistently urged you to substantiate the empirical advantages of your approach, especially in light of the novelty concerns of the reviewers I highlighted based on initial reviews. As previously stated, "Given these novelty concerns, the empirical performance improvements presented will be crucial in informing my decision. I have reservations about the significance of these performance gains and would appreciate further insights from both reviewers and authors on this matter."

I appreciate the authors providing additional results and emphasizing that their human evaluations clearly prefer Diffusion-KTO over the baselines. I am glad to accept that advantages, but I'd like to also point out that while human evaluation is invaluable, it also poses challenges in terms of reproducibility and comparative cost.

My primary concerns were initially centered around Tables 3 and 4, which prompted my inquiries about the "statistical significance" of the improvements and whether the differences were substantial enough to have practical significance. I am pleased that the authors are now providing error bars for Tables 3 and 4, indicating that some improvements are statistically significant, although it remains challenging to draw meaningful conclusions for some metrics.

The authors suggest focusing on Table 1, noting that "PickScore is the most significant metric as it is trained to reflect human preferences exhibited in the Pick-a-Pick dataset and is most suitable for evaluating the performance of training algorithms. Other metrics, trained on preference data, may also be relevant, but differences in dataset curation could introduce biases in these reward models." However, it's important to consider that the method's advantages might be less pronounced under domain shifts (shifting from Pick-a-Pick to other prompts).

Regarding Figure 5, the selection criteria for these results remain unclear to me. Were these images chosen by finding a specific seed where Diffusion-KTO outperformed all other models, and then displaying those particular pictures?

Regarding the use of negative labels, I find your response satisfactory. Nevertheless, for the toy example illustrated, I think integrating a relatively straightforward technique, such as classifier guidance, into the diffusion model could significantly enhance the model's ability to steer clear of undesired regions.
As requested, to facilitate the discussion, we listed the error bars of 95% confidence interval as below. We observe that all win rate has a margin of error <2%.


Win rate on Pick-a-Pick Test Set (2500 generations)

|                       | Aesthetic        | PickScore          | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| Ours vs SFT         |56.0$\pm 1.95$  | 73.0$\pm 1.74$   |     60.6 $\pm 1.92$        |   57.6 $\pm 1.94$      |   57.2 $\pm 1.94$     |
| Ours vs Diffusion-DPO | 70.4 $\pm 1.79$    | 59.6   $\pm 1.92$  |   74.2$\pm 1.72$  | 48.4$\pm 1.96$ | 49.2 $\pm 1.96$ |

Win rate on HPS v2 Test Set (16050 generations)

|                         | Aesthetic | PickScore | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| Ours vs SFT         | 54.1 $\pm 0.77$  | 61.2$\pm 0.76$   |      54.0 $\pm 0.77$        |   54.6 $\pm 0.77$      |   54.2 $\pm 0.77$     |
| Ours vs Diffusion-DPO | 58.8 $\pm 0.76$    | 58.6   $\pm 0.76$  |   63.9 $\pm 0.74$          |   48.7 $\pm0.77$   | 48.8  $\pm0.77$   |


Win rate on PartiPrompts (8160 generations)

|                         | Aesthetic | PickScore | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| Ours vs SFT         | 50.3 $\pm 1.1$  | 58.6 $\pm 1.08$  |     54.7 $\pm  1.09$        | 54.2 $\pm 1.09$      |   51.4 $\pm  1.10$     |
| Ours vs Diffusion-DPO | 59.5 $\pm 1.08$    | 48.3 $\pm 1.10$  |   59.3$\pm  1.08$          |   48.6 $\pm  1.10$   | 48.2  $\pm  1.09$  |

**Q:How do you sample prompts for Fig 5**. 
To highlight the advantage of Diffusion-KTO in real-world scenarios, we create prompts based on user prompts shared over the internet. In particular, we heavily referenced Playground.AI, where users share generated images alongside their prompts. While these prompts are manually written, we believe they reflect typical use cases in real-world scenarios and is generally similar to the "good" samples in the Pick-a-Pic dataset, which is also written by human labelers. To further highlight the fairness of our method, we report the automatic win rate of 20 randomly selected prompts and generate 5 images per prompt. The prompts are copied from the front page of playground AI on Aug 10, 2024. An example prompt would be 

```
Photorealistic wet-erase marker portrait depicts Cleopatra with a hybrid aesthetic of a bikini and Galadriel's queenly regalia,....
```

The results are as follows

|            Metric       |  | 
| ----------------------- | --------- | 
| P(Ours > SDv1.5)          |  0.75 |
| P(Ours > SFT)          | 0.75 |
| P(Ours > DPO)   | 0.70 |
| P(Ours > SFT \| Ours > DPO)   | 0.785 |
| P(Ours >  SDv1.5 \| Ours > SFT,Ours > DPO)          | 0.909 |


For simplicity, we adopt an overall win rate in the above calculation. X > Y if and only if amongst 5 metrics, (Aesthetic ,PickScore ,ImageReward , CLIP ,HPS), at least 3 are better. **Contrary to AC's statement, "the likelihood of (Diffusion-KTO) outperforming all others", in the distribution of user-specified prompt sampled from Playground AI, is 49.9%, which is not small.** This is because the win rates are highly correlated with each other and not independent. For example, if Diffusion-KTO beats SFT and DPO on a prompt, it is likely that it will beat SDv1.5 as well. 


**Q: Have you adequately utilized negative-labeled examples in SFT**. We request further clarification on how to adequately utilize negatives in SFT as it is not in the default formulation. We tried to naively add a negative term to the L2 loss to "steer clear of poor-performing regions" using the loss $\mathbb{E}[y(x_t)\lVert \epsilon_\theta(x_t)-\epsilon\rVert^2]$ where $y(x_t)\in\{+1,-1\}$ is the label. The model diverge to infinity as the loss on positive samples has range (0,+inf), but the loss on negative samples has range (-inf,0). This results in the model pushing the loss to -inf.  In fact, it is precisely because this form of naive contrastive SFT does not work, that the community has resorted to alternative approaches such as RL, Inverse RL, and more recently DPO. 

In summary, we want to apologize for the confusion during this discussion and we hope in this response to have cleared some of the questions by providing detailed information.
**Q: Is such a small improvement practically significant?** 

**Response 1: In a hypothetical of 50.5%-53.5, the answer is yes.** We apologize for the confusion in adressing this question. Without any prior literature or community consensus to establish a quantative threshold of "practical significance", we argue that all results that are statistically significant are of "practical use", as it implies "an average improvement". In fact, it is precisely because of the large amount of test data, which leads to low margin of errors, that the ML community consider even 1 or 2 point gain in benchmarks such as ImageNet-1K classification as "significant". We believe, this discussion is outside the scope of our paper and is a broader problem within the ML community in general. To highlight the aspect of "average improvements", we additionally provide average scores on HPS v2 below.

|                         | Aesthetic | PickScore | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| SDv1.5          |5.41    | 20.70      |     0.135       |   32.67 |   0.348 |
| SFT | 5.60     | 21.09      |  0.592          |   32.68  | 0.352   |
|Ours | **5.66**| **21.36**  | **0.749** | **33.07** | **0.355** |



**Response 2: We strongly disagree with the AC's premise of "small" improvements**

We would like to respectfully mention, that we have previously inquired about a clear numerical threshold for "practical significance," but this has not been specified. Without a defined standard, it is unfair to justify dismissing our improvements as insignificant. We would like to point out that **we have also presented human evaluation results comparing Diffusion-KTO and SFT side-by-side and found that human evaluators prefer Diffusion-KTO generations 72.3% of the time.** We believe this 70%+ win-rate is practically significant considering human evaluation is the best measurement of how well models are aligned with human preferences. We kindly request the AC to provide a clear threshold and give justifications for such threshold, otherwise, we stand that our results are significant particulary, practically significant.


We would like to also point out that the win rate is more thoroughly studied in the context of LLM. At the moment, 54% win rate is the win rate between LLama3.1-405B-instruct versus LLama3.1-70B-instruct on lmsys tournament. We recognize that the task is different, however given the lack of related studies in the context of text-to-image generation, we think this information may be useful. 


**Response 3: Discussions on specific "in-significant" numbers**. 

Looking back at the discussion, the AC noted his/her reservations based on some low numbers in Table 3 and Table 4, but we think our strong results against SFT in Table 1 have been largely marginalized. We apologize but as we mentioned above, our stance is that it is unreasonable to expect Diffusion-KTO to exhibit a large numeric gap on all metrics, as such are rare cases for most works. We believe that all results should be looked at holistically, based on the following principles.

1. Human evaluation should outweigh automatic ones, as this is the direct objective of preference optimization and human-utility optimization.
2. Among all numerical metrics, PickScore is the most significant one as it is trained to reflect the human preference exhibited in the Pick-a-Pick dataset, and is most suitable to evaluate the performance of training algorithms. While other metrics are also trained on preference data, differences in dataset curation may lead to different biases in these reward models.

In this regard, we argue that human evaluation and the results on Pick-a-Pick test set in Table 1 are the best measurements to compare the effectiveness of optimization algorithms, which the AC has kindly acknowledged: "clearly higher winning rates based on Pick-a-Pick test prompts".
Thanks for the response. Your answer addresses my questions well, and it is impressive to see the statistical significance of the result.
Great, now we are on the same page. Could you respond to my previous questions:

- Consider that if A outperforms B 52% of the time with a 95% confidence interval of (50.5%, 53.5%), A is statistically better than B. However, is such a small improvement practically significant? This is the type of question the paper needs to address more clearly.

- Note that I don't have doubt that Diffusion-KTO outperforms SD1.5 with practical significance. My concern on "practical significance" was mainly about:

   - Diffusion-KTO vs SFT

   -  Diffusion-KTO vs Diffusion-DPO

More specifically, could you illustrate what the comparisons "Diffusion-KTO vs SFT" and "Diffusion-KTO vs Diffusion-DPO" would look like if you included error bars or a 95% confidence interval in Tables 3 and 4?

Additionally, I have two more questions:

1. Regarding the visual results in Figure 5, could you detail the procedure used to select which random generations to display? Given that the winning rates (from both automatic and human evaluations) suggest the superiority of Diffusion-KTO is not overwhelming, the likelihood of it outperforming all others in a single random generation seems low.

2. Figure 6 is quite informative; however, it raises a question about whether you have adequately utilized negative-labeled examples in SFT to steer clear of poor-performing regions. Further clarification on this point would be helpful.
To reviewer MiLc and AC,
We agree that there has been some miscommunication and appreciate your efforts to clarify your questions and comments. To clarify, **we have provided direct comparison with SFT baselines.**  

1. We have provided automatic win-rates (Diffusion-KTO vs SFT, Diffusion-KTO vs Diffusion-DPO, etc) in Table 1,  Appendix Tables 3 and 4.

2. In the comments above, we provided an additional human evaluation which is a direct comparison between images generated by Diffusion-KTO and the SFT baseline using the same HPSv2 prompts. In this human evaluation against SFT, Diffusion-KTO had a win-rate of 72.3%.

3. Further, we also provided qualitative results in Figure 5, which is a "side by side" comparison of images generated using the same prompts from various baselines.

To our best understanding, we think the discussion is not about whether certain results are missing. Rather, its about how to interpret the results. 

We thanks reviewer MiLc's insight and hope the above information is helpful. If you belive additional forms of "direct comparisons" other than (1) (2) (3) can facilitate the discussion, or there are better metrics to measure the "degree of improvement", we are happy to provide those as well.

Conceptually, we would also point out that a key difference between Diffusion-KTO and SFT is that Diffusion-KTO learns to "avoid" bad generations. While this property may be hard to measure directly, in Figure 6 we show that in the 2D Gaussian example, Diffusion-KTO and SFT are both able to fit the distribution of "desirable data", but Diffusion-KTO uniquely avoids "undesirable data".
Thank you, Reviewer MiLc, for your discussion! Yes, you are 100% correct that I was "asking the authors if they have a direct comparison between Diffusion-KTO model and an SFT model."

The additional information the authors provided was helpful, but was not directly addressing that question.
I've read a bit through this thread, and I think the conversation has largely been hindered by miscommunication. 

I believe the AC is asking the authors if they have a **direct comparison** between Diffusion-KTO model and an SFT model. In particular, this would be a direct win rate of Diffusion-KTO over SFT, rather than a comparison of win rates of these methods versus a base SD 1.5 model (as was just mentioned again). If this is not what the AC is looking for, please correct me.

Can the authors provide this? I believe the images are generated from the same prompts, so it would not be too hard to get both the Diffusion-KTO samples and the SFT samples and then compare them side by side.

Pros of this evaluation strategy: this avoids an issue that results in the comparisons being non transitive. In particular, it is possible that Diffusion-KTO improves **more** images to a lesser degree than SFT.

Cons of this evaluation strategy: this type of pairwise comparison should not be the de facto method for comparing preference optimization strategies for diffusion models for academic papers. While some LLM systems like lmsys do an elo rating, such a system would be impractical for diffusion models (since they are more quite a bit more expensive to sample from). This is **fine** for Diffusion-KTO and SFT since the main benefit of Diffusion-KTO is getting rid of preference pairs, which is semantically similar to SFT and is the key benefit of the method.
**Additional Human Evaluation Results**

First, we would like to highlight that **human evaluation shows Diffusion-KTO has a 70.2% win rate against Diffusion-DPO**. We also conducted **additional human evaluation of Diffusion-KTO vs SFT, which results in a win rate of 72.3% in favor of Diffusion-KTO.**


**Further discussion on Automatic Metric**

In additional to head-to-head comparison provided in Appendix Table 3, we additionally incorporate the SFT baseline in the results (Method X vs SDv1.5) from previous comment on the HPSv2 test set.

|                         | Aesthetic | PickScore | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| Ours vs SDv1.5          | **70.0**     | **74.5**      |      **72.0**       |   53.7   |   **54.2** |
| Diffusion-DPO vs SDv1.5 | 60.3(-9.7)     | 68.1(-6.4)      |   59.0(-13.0)          |   **54.2**(+0.5)   | 54.1(-0.1)    |
|SFT vs SDv1.5 |  66.1(-3.9) | 64.7(-9.8)  |  63.4(-8.6) | 49.8(-3.9) | 50.2(-4.0) |


Compared with SDv1.5, **Diffusion-KTO achieves a significantly higher win rate than the SFT and Diffusion-DPO baselines.** For example, Diffusion-KTO has a lead of +9.7 on Aesthetic, +6.4 on PickScore, and +13.0 on ImageReward when compared with DPO, and a lead of +3.9 on Aesthetic, +9.8 on PickScore, and +8.6 on ImageReward compared with SFT. 

In addition, we would argue **that results presented in Appendix Table 3 and the table above are indeed "practically significant"**. For example, while it may seem that a win rate of 54.0 on ImageReward (Ours vs SFT, appendix Table 3) in a head-to-head comparison is not "significant", comparing (Ours vs SDv1.5) and (SFT vs SDv1.5), we can see that Diffusion-KTO expands the margin of improvement by +8.6. **Comparing with the neutral baseline (0.5 win rate), we expand the gain from (+13.4) of "SFT vs SDv1.5" to a gain of (+22.0) of "Ours vs SDv1.5", or 164% when evaluated by ImageReward.** The gains demonstrated by Diffusion-KTO indicate undoubtedly a significant improvement.

In general, the community has not yet established a clear numerical threshold for what constitutes an automatic win rate of 'practical significance' in text-to-image diffusion models, as these are relatively new metrics. This is why it is important to consider the results provided by human evaluations. In other fields, such as image classification and object detection, **even a 1–2-point gain in Top1-Acc or mAP is considered a meaningful improvement**. If the AC is suggesting that we adopt a specific numerical threshold to determine the 'practical significance' of the improvements presented in Appendix Table 3 and Table 4, we would appreciate clarification on what that threshold should be to facilitate further discussions. Otherwise, we maintain that: **1) human evaluation has demonstrated that our improvement is of 'practical significance,' and 2) Diffusion-KTO provides a more significant margin of improvement (Table above) when compared to SFT or Diffusion-DPO**.

Note: we discovered an editorial oversight in the original manuscript: the columns of the tables are mislabeled. The column label "ImageReward" and "HPS v2" should be swapped for all win rate tables. This has been corrected in the tables we provide above.
I completely agree that methods not requiring paired data are more practical, and both Supervised Fine-Tuning (SFT) and KTO are appropriate for such scenarios. The key issue is whether KTO-based methods can surpass the performance of SFT. If diffusion-KTO fails to clearly outperform SFT—or only achieves marginal improvements without practical significance—I would question its utility.
i think even the Diffusion-DPO work can then be viewed in a similar vein, it just used the method from NLP for the text-to-image model. And to a certain extent, this is a valid concern. I suppose the the KTO method is more practical as it does not require paired data but rather text-image pairs that are deemed good to bad which is more realistic.

There is no redeeming in terms of memory / compute usage either.
Note that I don't have doubt that Diffusion-KTO outperforms SD1.5 with practical significance. My concern on "practical significance" was mainly about:

1. Diffusion-KTO vs SFT 

2. Diffusion-KTO vs Diffusion-DPO
The number of test prompts is **500 for Pick-a-Pick, 3210 in HPS and 1632 in PartiPrompts**. We generate 5 images per prompt. For human evaluation, we collected 300 responses on randomly sampled HPS prompts. 

We agree that the "practical significance" of the improvement is very important, which is why we performed the human evaluation. Since human evaluations show considerable improvement over baselines, we argue that our improvement is "practically significant," because the automatic metrics are mere surrogates for human preference (many of these metrics such as PickScore and ImageReward use reward models trained on human preference data).

To further investigate the significance of our improvements, we compare the automatic win-rate of our method against SD v1.5 versus the automatic win rate of Diffusion-DPO against SD v1.5 on HPS, the largest of three datasets. The results are shown in table below

|                         | Aesthetic | PickScore | ImageReward | CLIP | HPS    |
| ----------------------- | --------- | --------- | ----------- | ---- | --- |
| Ours vs SDv1.5          | **70.0**     | **74.5**      |      **72.0**       |   53.7   |   **54.2** |
| Diffusion-DPO vs SDv1.5 | 60.3     | 68.1      |   59.0          |   **54.2**   | 54.1    |


It has been well established the improvements of Diffusion-DPO are of "practical significance". Compared with Diffusion-DPO, Diffusion-KTO achieves "significantly" higher win-rates against baseline on Aesthetic (+9.7%), PickScore (+6.4%), Image Reward (+13.0%), and is comparable on HPS (+0.1%) and CLIP (-0.5%).  Given these results, we argue that we are our method also achieves an improvement of "practical significance".

In summary, we believe that "practical significance" is best measured by human evaluation, which show decisive advantages of Diffusion-KTO. Additionally, we also note that when compared with baseline methods, Diffusion-KTO achieves a larger gap than Diffusion-DPO, which is known to have significant improvements.

To avoid further confusions, we would like to clarify that all occurrences of the term “significance” in the above discussion refers to “practical significance” mentioned in previous comments from AC. 

If you have additional questions or would like to see additional experiment results, don't hesitate letting us know.
Could you please specify the number of test prompts (and hence the margin of error) used for Pick-a-Pick, HPS, and Parti prompts? 

Consider that if A outperforms B 52% of the time with a 95% confidence interval of (50.5%, 53.5%), A is statistically better than B. However, is such a small improvement practically significant? This is the type of question the paper needs to address more clearly.
Thank you for recognizing our strong results on Pick-a-Pic. Regarding the results on HPS and PartiPrompts, these datasets have significantly more prompts than Pick-a-Pic, which leads to a smaller margin of error. For example, on HPS, all automatic win-rates above 51.4% have a p-value < 0.00024, and on PartiPrompts all automatic win rates above 51.4% have a p-value < 0.00729, both of which are below the threshold for statistical significance (p=0.05). 

We would also like to point out that our human evaluation is conducted on a subset of HPS prompts. The 95% confidence interval is 65.6%-74.8% for the human-evaluated win-rate against DPO and 73.6%-82.9% for the human-evaluated win-rate against SDv1-5. Both results are significant, with p-value < 0.001. These experiments reaffirm a key observation made by the authors of KTO when using KTO for LLMs: "margin between KTO and DPO is even bigger in human evaluations than it is in automated LLM-as-a-judge evaluations", because "maximizing preference likelihood does not mean one is maximizing human utility" [1]. We believe these results provide a key insight for aligning diffusion models with human feedback and are valuable for the community.

If you have any further questions, feel free to let us know in the discussion period. 

[1] Ethayarajh, Kawin, et al. "KTO: Model alignment as prospect theoretic optimization." (2024).
Dear all,

From my assessment of the reviews and the rebuttal, there appears to be a prevailing concern regarding the novelty of the submission:

- The proposed Diffusion-KTO method is largely a straightforward extension of KTO, originally applied to fine-tuning language models, to fine-tuning diffusion models.
- It could also be viewed as a variant of Diffusion-DPO, where the DPO component is substituted with KTO.

Given these novelty concerns, the empirical performance improvements presented will be crucial in informing my decision. I do have reservations about the significance of these performance gains and would appreciate further insights from both reviewers and authors on this matter:

The preference data utilized is derived from the Pick-a-Pick training set. While Table 1 in the main text underscores the benefits of Diffusion-KTO, showing clearly higher winning rates based on Pick-a-Pick test prompts, the advantages are much less clear when examining Tables 3 and 4 in the appendix, which employ HPS v2 and Parti prompts, respectively. Are the differences in these tables still statistically significant? This ambiguity makes it challenging to ascertain whether Diffusion-KTO truly outperforms baseline methods, such as SFT that also does not require pairwise preferences.

Thanks, 

AC