Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation
Authors: Huizhuo Yuan
Zixiang Chen
Kaixuan Ji
Quanquan Gu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models; Stable Diffusion; RLHF; Self-play
Primary area: generative_models

Abstract: Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images (``winner'' and ``loser'' images) for each text prompt.
In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data. Codes are available at \url{https://github.com/uclaml/SPIN-Diffusion/}.

Self-Play Fine- Tuning of Diffusion Models for Text-to-Image Generation
Huizhuo Yuan" Zixiang Chen' Kaixuan Ji Quanquan Gu
Department of Computer Science
University of California, Los Angeles
Los Angeles, CA 90095
(hzyuan,chenzx199 chenzx19, kaixuanji, aguu/ees wwla. edu
Abstract
Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing certain volume data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt. In this paper, we introduce innovative technique called self-play fine-tuning for diffusion models (SPIN- Diffusion), where the diffusion model engages in competition with its carlier versions, facilitating an iterative ielloemprove process. Our approach offers alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments a-Pic dataset reveal that Diff-Diffssion outperforms the existing supervised fine- tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHe-based methods across all metrics, achieving these results with less data. Codes are available at
the Pick-
com http:a////////--0.ffffuuioooo/
Introduction
Diffusion models (Ho al. 2020
Peebles and Xie 2023 Podell et al.) 2023 Nichol et al. 2021 2020al have rapidly emerged critical entities within the realm of generative Als (Creswell et al., 2018 Kingma and Welling 2013). demonstrating exceptional capabilities in generating high-fidelity outputs. Their versatility spans diverse area of applications, ranging from image generation (Rombach et al,, 2022a Podell et al., 2023 Ramesh et al. 2022)
Rombach et al. 2022a] Song et al.
to more complex tasks like structure-baset drug design (Corso al. 2022 Guan et al. 2023)
protein structure prediction [Watson al. (02I), l), text generation (Austin al. 2021 Zheng et al. 2023 Chen et al. (023), and more. Prominent diffusion models in image generation, incl including DALL-E (Ramesh al. 2022), Stable Diffusion (Rombach et al. 2022b), SDXL (Podell et al., 2023).
and Dreamlike, etc., typically undergo fine- tine-tuning process following their initial pre-training phase.
Standard fine-tuning method for diffusion models suffers from low alignment with human preferences and low data efficiency due two main reasons: (1) it does not directly optimize for alignment with human preferences, and (2) only one round of training can be performed. Recently, using Reinforcement Learning (RL) for fine-tuning diffusion models has received increasing attention. [ee et al. (2023) first studied the alignment of text-image diffusion models to human preferences using reward-weighter likelihood maximization with reward function trained on human preference data.
'Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Black et al. (2023) formulated the fine-tuning of diffusion models as RL problem solved by policy gradient optimization. In a concurrent work. Fan et al. (2023) studied a similar formulation but with a KI. regularization. Very recently. Wallace et al. (2023) have bypassed the need for training reward functions by using Direct Preference Optimization (DPO) (Rafailov et al., (023) for fine-tuning diffusion models. Similar approach was proposed in Yang et al. (2023) as well.
While RL fine-tuning of diffusion methods has been proven effective, its dependency on human preference data, often necessitating multiple images per prompt, poses a significant challenge. In many datasets including the community-sourcee ones featuring custom content, it is often the case to have only one image associated with each prompt. This makes RL. fine-tuning infeasible.
In this paper, drawing inspiration from the recently proposed self-play fine-tuning (SPIN) technique (Chen et al.] (024) for large language models (LLM), we introduce a new supervised fine-tuning (SFT) method for diffusion models, eliminating the necessity for human preference data in the fine-tuning process. Central to our method is a general-sum minimax game, where both the participating players, namely the main player and the opponent player, are diffusion models. The main player's goal is to discern between samples drawn from the target data distribution and those generated by the opponent player. The opponent player's 's goal is to garner the highest score possible, as assessed by the main player. A self-play mechanism can be made possible, if and only if the main player and the opponent player have the same structure, and therefore the opponent player can be designed to be previous copies of the main player (Chen et al.) (024). The proposed algorithm SPIN-Diffusion overcomes the drawbacks of both supervised fine-tuning (SFT) and RL fine-tuning, Compared with SFT, our method is more data-efficee eftt-effiiciet,, by repeatedly using the prompts from the SFT dataset to improve the model through self-play. Compared with RL fine-tuning methods, our method does not need external reward models or expensive human-annotater winner/loser pairs.
When applying the self-play fine-tuning technique (Chen et al. 2024) to diffusion models, there are two challenges: (a) an exponential or even infinite number of possible trajectories can lead to the same image. The generator in a diffusion model operates through a sequence of intermediate steps. but the performance of the generator only determined by the quality of the image in the last step; and (b) diffusion models are parameterized by sequence of score functions, which are the gradient of the probabilities rather than probabilities in L.LMs. Our algorithm design effectively surmounts these challenges by (a) designing an objective function that considers all intermediate images generated during the reverse sampling process; and (b) decomposing and approximating the probability function step-by-step into products related to the score function. We also employ the Gaussian reparameterization technique DDIM (Song et al. 2020a) to support the advanced sampling method. All these techniques together lead to an unbiased objective function that can be effectively calculated based on intermediate samples. For computational efficiency, we further propose an approximate objective function, which eliminates the need for intermediate images used in our model.
Contributions. Our contributions are summarized below:
We propose a novel fine-tuning method for diffusion models based on the self-play mechanism, called SPIN-Diffusion. The proposed algorithm iteratively improves upon diffusion model until converging to the target distribution. Theoretically, we prove that the model obtained by SPIN-Diffusion cannot be further improved via standard SFT. Moreover, the stationary point of our self- play mechanism is achieved when the diffusion model aligns with the target distribution.
Empirically. we evaluate the performance of SPIN-Diffusion on text-to-image generation tasks (Ramesh et al., 2022 Rombach al.) 2022a| Saharia al. 2022a) Our experiments on the Pick-a- Pic dataset (Kirstain et al.] (202)). with base model being Stable Diffusion-1155 1,5 (Rom- bach al., 2022b), demonstrate that SPIN-Diffusion surpasses SFT from the very first iteration. Notably, by the second iteration, SPIN-Diffusion outperforms Diffusion-DPO (Wallace et al., 2023) that utilizes additional data from 'loser' samples. By the third iteration, the images produced by SPIN-Diffusion achieve higher PickScore (Kirstain et al. (2023) than the base model SD-1.5 79.8% of the times, and superior Aesthetic score 88. 4% of the times.
SPIN-Diffusion exhibits a remarkable performance improvement over current state-of-the-art fine- tuning algorithms, retaining this advantage even against models trained with more extensive data usage. This highlights its exceptional efficiency in dataset utilization. beneficial for the general public, particularly those with restricted access to datasets containing multiple images per prompt.

---

Notation. We use lowercase letters and lowercase boldface letters to denote scalars and vectors, respectively. We use : ' denote the index set {O..... }). In the function space, let F be the function class. We use the symbol q to denote the real distribution in diffusion process, while Po represents the distribution parameterized by a nueral network during sampling. The Gaussian distribution is represented as W(,, Σ), where 1a and Σ are the mean and covariance matrix, respectively. Lastly, Un,.orm(I,..., T denotes the uniform distribution over the set {I....,T)
2 Related Work
Diffusion- -aseed generative models (Sohl-Dickstein et al. 2015) have recently gained prominence, attributed to their ability to produce high-quality and diverse samples. A popular diffusion model is denoising diffusion probabilistic modeling (DDPM) (Ho et al. 2020 Song et al. (2020a) proposed a denoising diffusion implicit model (DDIM), which extended DDPM to a non- Markov diffusion process, enabling deterministic sampling process and the accelerated generation of high-quality samples. In addition to DDPM and DDIM, diffusion models have also been studied with score-matching probabilistic model using I Langevin dynamics (Song and Ermon 2019 Song et al. 2020b). Diffusion models evolved to encompass guided diffusion models, which are designed generate conditional distributions. When the conditioning input is text and the output is image, these models transform into text-to-image diffusion models (Rombach et al., 2022aa Ramesh et al.
Diffuusion Models.
2022 Ho al. 2022 Saharia et al. 2022b). They bridge the gap between textual descriptions and image synthesis, offering exciting possibililies for content generation. A significant advancement in text-to-image generation the introduction of Stable Diffusion (SD) (Rombach al. 2022a). SD has expanded the potential of diffusion models by integrating latent variables into the generation process. This innovation in latent diffusion models enables the exploration of latent spaces and improves the diversity of generated content. Despite the introduction of latent spaces, generating images with desired content from text prompts remains a significant challenge (Gal et al. 2022, Ruiz et al. 2023). This is due to the difficulty in learning the semantic properties of text prompts with limited high-quality data.
Fine- Sine-Tuning Diffuusin Models. Efforts to improve diffusion models have focused on aligning them more closely with human preferences. Rombach et al. (2022a) fine-tuned a pre-trained model using the COCO dataset (Caesar et al., 2018), demonstrating superior performance compared to generative model directly trained on the same dataset. |Podell et al. (022)) expanded the model size of Stable Diffusion (SD) create the SDXL model, which was fine-tuned on high-quality but private dataset, leading to significant improvement in the aesthetics of the generated images. Dai jet al. (2023) further demonstrated the effectiveness of fine-tuning and highlighted the importance of the supervised
(SFT) dataset. In addition to using datasets with high-quality images, Betker et al. (2023); Segalis al. (2023) found that SFT on a data set with high text fidelity can also improve the performance of the diffusion model. The aforementioned methods only requires a high-quality SFT dataset. Recently, preference datasets have been studied in finetuing diffusion models (Lee et al. 2023). Concurrently, DDPO (Black et al. 2023) and DPOK (Fan et al.) 2023) proposed to use the preference dataset to train a reward model and then fine-tune diffusion models using reinforcement learning. Drawing inspiration from the recent Direct Preference Optimization (DPO) (Rafailov et al., 2023), Diffusion- DPO (Wallace et al.) (2023) and D3PO (Yang et al., 2023) used the implicit reward to fine-tur diffusion models directly when a differentiable reward model is available, Clark et al. (023); Prrabhudesai et al. (2023) applied reward backpropagation for fine-tuning diffusion models. Our SPIN-Diffusion is most related the SFT method, as it only assumes access to high-quality image-text pairs. However, the high-quality image-text dataset can be obtained from various sources, including selecting the winner from preference dataset or identifying high-reward image-text pairs through reward model.
the preference dataset. Furthermore,
3 Problem Setting and Preliminaries
In this section, we introduce basic settings for text-to-image generation by diffusion models and the self-play fine-tuning (SPIN) method.
3.1
Text-to-Image Diffusion Model
Denoising diffusion implicit models (DDIM) (Song et al. 2020a) is generalized framework of denoising diffusion probabilistic models (DDPM) (Sohi-Dickstein et al.] 2015] Ho et al.] 2020). DDIM enables the fast generation of high-quality samples and has been widely used in text-to-image
3

---

diffusion models such as Stable Diffusion (Rombach et al. 2022a). We formulate our method building upon DDIM, which makes more general.
Forwrd Process. Following Saharia al. (2022b), the problem of text-to-image generation can be formulated as conditional diffusion models. We use Xo E Rd to denote the value of image pixels where d is the dimension and use denote the text prompt. Given prompt image X drawn from target data distribution (-Ic). The diffusion process is characterized by the following dynamic parameterized by a positive decreasing sequence [ with aq 1,
(xx (2)) == q(xr xx) II
|xe, Xo),
(3.1)
where gxx;-1 X2, xo) represents Gaussian distribution N WH,, ofI). Here, Me is the mean of Gaussian defined as
M4 1x-1X0 + V 04-1 - oz Xt / VapX0
04
can be derived from 3.1 that (xxx (x) = ( Xo (1 -1-68 04 I) for all (Song et al. 2020a). As
generalized diffusion process of DDPM, 3.1 reduces to DDPM (Ho et al. 2020 with special choice of v(1 01-1)/(1 0+)
ore/ /0///
Generative Process. Given the sequence of {at)T= and {arl==I examples from the generative model follows
Po (Xo:r c)) II pe (00-xx1lxxx xx, c) (xrlc)) pe (x c) - c,,, (3.2)
Here f belongs to the parameter space @ and (X,, c, t) is the estimator of mean μ that can be reparameterized (Ho et al., 2020 Song et al. (020a) as the combination of x, and a neural network (x(x, c, named score function. Please Appendix for more details.
Training Objective. The score function ( c, f) is trained by minimizing the evidence lower bound (ELBO) associated with the diffusion models in (3.1) and (3.2). which is equivalent to minimizing the following denoising score matching objective function LDSM:
(0) E. 7x [o(x, c, - 1e
(3.3)
where x; = Vaxxo vI 1 a and the expectation is computed over the distribution e ~ q(-), Xo Adata (-c), e ~ (0, I), t ~ U T). .... In addition, {re/=1 are pre- pre-specified weights that depends on the sequences {ar]?= and (o.)e=1.
3.2 Self-Play Fine- TunnnTuning
Self-Play mechanism, originating from TD-Gammon Tesauro al. 1995). has achieved great successes in various fields, particularly in strategic games (Silver et al. 2017b a). Central to Self-Play the idea of progressively improving model by competing against previous iteration. This approach has recently been adapted to fine-tuning Large Language Models (LLMs) (Chen et al.) 2024), called self-play fine-tuning (SPIN). Considering an LLM where e the input prompt and xo the response, the goal of SPIN is to fine-tune an LLM agent, denoted by pa p-Ic), based on SFT dataset. Chen et al, (2024) assumed access to a main player and an opponent player at each iteration and takes the following steps iteratively:
1. The main player maximizes the expected value gap between the target data distribution Pdata and the opponent player's distribution Po.:
2. The opponent player generates responses that are indistinguishable from Pdata by the main player.
Instead of alternating optimization, SPIN directly utilizes a closed-form solution of the opponent player, which results in the opponent player at iteration k + 1 to copy parameters 0 and forming an end-to-end training objective:
E Leee(Alos (x(x)))) Aog po, (xplc)
(x(x(())
LEPIX
(3.4)
pa, (x/
Here the expectation is taken over the distribution c q(c), x ~ Palata (x|c), x' po, (x'(c), ((-) is a loss function that both monotonically decreasing and convex, and Notably, (344 only requires the knowledge of demonstrationnSST data, ie.. prompt-response pairs.
a hyperparameter.

---

Algorithm Self-Play Diffusion (SPIN-Diffusion)
Input: ((xo: c))ENN): SFT Dataset, po,: Diffusion Model with parameter Oo. K: Number of iterations.
for k = 0, K- do
for i N....N do
Generate real diffusion trajectories X1.T ((x1:T(X))) (xx). Generate synthetic diffusion trajectories x/,7 ~ po. (-Ic). end for Update 0 = argmingge ISPIN(B 0x), which is the empirical version of (4.8 or 4.99 end for
Output: 0x-
4 Method
In this section, we are going present method for fine-tuning diffusion models with self-play mechanism.
Consider a setting where we are training on a high-quality dataset containing image-text pairs xo) Pdata (Xo ())(C) where с is the text prompt and is the image. Our goal is to fine-tune
a pretrained diffusion model, denoted by pa. to align with the distribution pdata [xo lc.. Instead of directly minimizing the denoising
matching objective function LDSM in B.3
adapt
SPIN to diffusion models. However, applying SPIN to fine-tuning diffusion models presents unique challenges. Specifically, the objective of SPIN (3.4) necessitates access to the marginal probability
LLMs, this is not the case with diffusion models. Given the parameterization of the diffusion model as po Xx.T c), computing the marginal probability po [a|c] c) requires integration over all potential trajectories fx.. Po Xx.r (c)rr/r) which computationally intractable.
While obtaining (Xn c) is straightforward
In the following, we propose a novel SPIN-Diffusion method with a decomposed objective function that only requires the estimation of score function Eg. This is achieved by employing the DDIM formulation discussed Section 3 The key technique self-play mechanism with a focus on the joint distributions of the entire diffusion process, i.e., Pdata (No:T (c) = 4(Xi.T) Xy Pdata (xpl c) and (Xoor c), instead of marginal distributions.
4.1 Differentiating Diffusion Processes
In iteration + 1, we focus on training a function fk+1 to differentiate between the diffusion trajectory Xo:r generated by the diffusion model parameterized by po (xxxrrlc), and the diffusion process PXa.a(Xo.T (c) from the data. Specifically, the training of fx+1 involves minimizing a generalized Integral Probability Metric (IPM) (Müller 1997):
fk+1 argmin [[((f(c. (o.T) f(c, Xo:)))
(4.1)
fEFx
Here, the expectation is taken over the distributions ~ (()), Xo.T Paata(-lc), and x pe, (-lc). F& denotes the class of functions under consideration and {(-) is a monotonically decreasing and convex function that helps stabilize training. The value of reflects the degree of belief that the diffusion process Xo:r given context e originates from the target diffusion process Paata (Xo rather than the diffusion model po (Xp:r (c). We name f the test function.
4.2 Deceiving the Test Function
The opponent player wants to maximize the expected value Fe (ore(())x x x pl-lee (f (c. x)). In addition, to prevent excessive deviation of pew.. from pe, and stabilize the self fine-tuning, we incorporate a Kulbbeck-Leibler (KL. regularization term. Putting these together gives rise the following optimization problem:
teranxeeeeexxoorrrteeeeeee Xp:r)] AE KL (p(lc)))ll (+c)),
(4.2)
p
where À > 0 is the regularization parameter. Notably, (4.2 has a closed-for form solution p(-Ic):
((xur(e) Po. (xoer cc) exp (A fkt1 xo:))
(4.3)
5

---

To ensure that p lies in the diffusion process space {po(-Ic)|0 E )) we utilize the following test function class (Chen et )024):
Po (x(x):)(c)
f o
A log
(4.4)
pe, (x1:1
Given the choice of F& in 44.4 optimizing (4.1 gives f parameterized by 0z+1 the following form:
[N+1(C, Xo.T) = X log Poz+1 (Xo:rlc)
(4.5)
pa. (Xo: c)
Substituting (4.
into 4.. yields p(xxrl(c) = pa.: (Xer (c). other words, 0x+1 learned from
(4.1
is exactly the diffusion parameter for the ideal choice of opponent.
4.3
Decomposed Training Objective
The above steps provide training scheme depending on the full trajectory of Xo:T. Specifically, substituting (4.4) into (4.1) yields the update rule 0 argminnge LSPIn(0, 0s), where LaPIx is defined
LSPIN E ( log e
Po (Xu:r (c) po, (Xo:T|c) c))
X Algg Po Po(X6TT() (c)
(4.6)
pa. (xo:r(0)
Here the expectation is taken over the distributions
q(-), No.T Paata (-cc), c)2 X po, (-lc). To formulate a computationally feasible objective, we decompose log pa (XxoT (c) using the backward process of diffusion models. Substituting (.2) into (.6)), we have that
log (Xu:z (c) = log II (x 1 (xo,))))oo(xle))
lo (xr c) +
Σ
log (po(x:-11xx,,))
Am1
T -
1
CCosstant
(4.7)
20
1-1
where the last equality holds since po px |xc, e) is Gaussian distribution A (xo(xx, c, (..)),fl af according
to (3.2). and (xr (c) is approximately a Gaussian independent of 0. By substituting (4.7) into (4.6) and
introducing reparameterization of = (28)( where &: is a fixed positive value, we obtain
Larm((,,0)) 0z) = E
8: [||x-- mx(x 1)112 ||x Mox (xx, c, 1)11
T
-|[xÍ_1 -o c, t)
+|xi- ||xi= Mo, (x',, c, 1)
(4.8)
Here the expectation is taken over the distributions Note that by considering the main player (reward function) across the full trajectory (3.2). rather than focusing solely the final state in Fan et al. (2023): Black et al. (2023): Wallace et al. (2023).
q(-)) Np:T Pdaca (-Ic), x pe, (-cc).
we are able to formulate an exact objective function up to Equation (4.8). The detailed algorithm
presented in Algorithm 1 (4.8) naturally provides objective function for DDIM with a, >
where controls the determinism of the reverse process (3.2) 4.8) remains valid for deterministic
generation processes a, +
4.4 Approximate Training Objective
While (4.8) the exact ELBO, optimizing it requires storing all intermediate images during the reverse sampling. When the trajectory length 7' is large, it would require an impractical amount of GPU memory when the loss is summed over T. Additionally, the required samples from reverse process are not readily accessible. To address these limitations, we propose an approximate objective function. By applying Jensen's inequality and the convexity of the loss function f, we can give an upper bound of (.8) and thus move the average over / outside the loss function E:
approx (0, 00) = -E(e( (ee(-- 11x - mo(xe, c, t)|1² |x--1 usx (xr, c, } L

---

- -|x22-1 no(x/, c, + ||x nos (x/, c, c,t))))))))
(4.9)
where the expectation is taken over the distributions 1,, (x--1.X0 x ~ pea Po, (x t Uniform(1..... .T). This approxima-
g(c), 1x)
tion directly motivated by the nature of diffusion models, which inherently decouple operations on per-time-step step basis. We provide theoretical justifications for our approximation method in the following sections.
The following lemma shows that Lapprox is upper bound of LSPIX-
LIPINN
Lemma 4.1. Fix 0g θ which serves as the starting point of Algorithm f iteration k + 1. holds that LæFIN (,, 0x) < LPII (0, Os) for all 0 E θ.
eliminates the need to store all intermediate steps, as it only involves two consecutive pling steps t - and t. Since the reverse process po (x Xg. c) approximates the forward process ((x)T|
Ligpi
use the per step forward process q(x" (x)) to approximate po,, (x/-1- x x c). We can further approximate po, x/]e)
I po, (x)- 1, x X c)po, (xg [xlc))xx/ with I x. xx) Pe, (x/) htj//c/xx,, Substituting the corresponding terms in (4.9) with the above approxi-
mation allows us to only compute the expectation of (4.9) over the distribution c ~ g(c), (x,-1, x,)
Paata Pd (x,- xx /c) (x/-1, x,) ~ Po, (x)le)g(x(_11 x, xx)xx
Uniform(1, (1..... T). Further-
more, by incorporating the reparameterization of pág into (4.8) and (4.9). we can express (4.8) and
(4.9 in terms of Eg (X,, C, t). Detailed derivations of 4.8 and 4.9 are provided in Appendix
Main Theory
In this section, we provide a theoretical analysis Algorithm ! Section introduucss two distinct objective functions, defined in (4.8) and (4.9), both of which use the loss function l. Since (4.8) is an exact objective function, its analysis closely follows the framework established by Chen et al. (2024). Consequently, we instead focus on the approximate objective function /.PP Lapprrx defined in (4.9), which is more efficient to optimize and the algorithm we use in our experiments. However, its behavior is more difficult to analyze. We begin with formal assumption regarding the loss function é as follows.
Assumption 5.1. The function ((t) R + and satisfies (0()) O. In addition, f(t) a convex function.
R in 4.9 is monotonically decreasing, i.e., Vt, (t) S 0
Assumption 5.I can be satisfied by various commonly used loss functions in machine learning. This includes the correlation loss {(t) 1 t, the hinge loss {(t) max(0. 1 - t), and the logistic loss ((L) log(1 + exp(-()). In our experiments, we are using the logistic loss.
To understand the behavior SPIN-Diffusion, let us first analyze the gradient of the objective function 4.9.
V approx LPII
E (Bi ( - - 1)
Vel
- Velix/-1-Me
(5.1)
Reweighting
Matching
Pushing
where the expectation is taken over the distributions
q(c), (x,-1,
Pdata (x: (x/_1, xt) ~ Po, 1x x/.c). D.3 can be divided into three parts:
Reweighting: ((-) in the "Reweighting" term is negative and increasing because {() is monotonically decreasing and convex according to Assumption 5.1 Therefore, -Bel
-B,6" 3x [[|x--1 px(x,, c, f)11 -.... + ||x|_ noo c,)]]]))]]]]) is always non-negative. Furthermore, - E, decreases as the argument inside E() increases.
Matching: The "Matching" term matches Me Mo(x, [x, c, to x coming from pairs (x,-1, that are sampled from the target distribution. This increases the likelihood of (x,-1, ~ Pdata (x x,) following the generative process (3.2).
Pushing: Contrary to the "Matching" term, the "Pushing" term pushes po(x), c, away from X[_ coming from pairs (x{_,xX) x,) drawn from the synthetic distribution po,, (x/-1, x(). Therefore, the "Pushing" term decreases the likelihood of these samples following the process in the generative
process 3.2
7

---

The "Matching" term aligns conceptually with the LDSM in SFT, as both aim to maximize the likelihood that the target trajectory Xo.T follows the generative process described in (3.2). The following theorem shows a formal connection, which is pivotal for understanding the optimization dynamics of our method.
Theorem 5.2. Under Assumption 5.1 0% is not the global optima of LDM in (33 there exists an appropriately chosen Sr, such that O₁ is not the global minima of (4.9) and thus 0k+1 # Og.
Theorem 5.2 suggests that the optimization process stops only when 0 reaches global optimality of LDSM- Consequently, the optimal diffusion model 0* found by Algorithm Saarthmmmaaaa cannot be further improved using LDSM- This theoretically supports that SFT with (3.3) cannot improve over SPIN- Diffusion. It is also worth noting that Theorem 5.2 does not assert that every global minimum of
LDSM meets the convergence criterion (i.e.. 0x+1 0x), particularly due to the influence of the "Pushing" " term in D.3). The following theorem provides additional insight into the conditions under which Algorithm converges.
Theorem 5.3. Under Assumption 5.1 if po. (-1x) = PPaata-Ixx), then 03 is the global minimum of 4.9 for any A 2 0.
Theorem 5.3 shows that Algorithm I converges when po (-(x))
x) = Paata (- [x), indicating the efficacy of SifN-Diifusionn in aligning with the target data distribution. In addition, while Theorems 5.2 5 are directly applicable to (4.), the analogous conclusion can be drawn for (4.8) as well (see Appendix D for a detailed discussion).
Experiments
In this section, we conduct extensive experiments to demonstrate the effectiveness of SPIN-Diffusion. Our results show that SPIN-Diffusion outperforms other baseline fine-tuning methods including SFT and Diffusion-DPo
6.1 Experiment Setup
Models, Datasets and Baselines. We use the stable diffusion v1.5 (SD-1.5) (Rombach et al. 2022a) as our base model. While adopting the original network structure, we use its Huggingface pretrained version which is trained LAION-5B (Schuhmann al. 2022) dataset, text-image pair dataset containing approximately 5.85 billion CLiPPfiltered image-text pairs. We use the Pick-a-Pic dataset (Kirstain et al. 2023) for fine-tuning. Pick-a-Pic is a dataset with pairs of images generated by Dreamlike (a fine-tuned version of SD-1.5) and SDXL-beta (Podell et al. 2023). where each pair corresponds human preference label. We also train SD-1.5 with SFT and Diffusion-DPO (Wallace) ct al. (023) as the baselines. For SFT, we train the model to fit the winner images in the Pick-a- -a- Pic ( (Kirstain al., 2203) trainset. In addition the Diffusion-DDD DPO checkpoint provided by WWllace ct al. (2023 (denoted by Diffusion-DPO), we also fine-tune an SD-1.5 using Diffusion-DPP DPO and denote by "Diffusion-DPO (reproduced)".
Evaluation. We the Pick-a-Pic test set, PartiPrompts YYu et al. 2022) and HPSv2 (Wu et al. 2023) as our evaluation benchmarks. We defer the detailed introduction and results PartiPrompts and HPSv2 to Appendix AA.3] Our evaluation rubric contains dimensions, human preference alignment and visual appeal. For visual appeal assessment, we follow Wallace et al. (2023): Lee
al. (2024) and use Aesthetic score. For human-prefeeeecee alignment, we employ reward models
including PickScore (Kirstain et al. 2023). ImageReward (Xu et al., 2023) and HPS (Wu et al. 2023).
Luce All these reward models are trained according to the Fradley-Terry Luce (Bradley and Terry 1952)
preference datasets. For each prompt, we generate 5 images and choose the image with highest average score over those four metrics (best out of 5). We report the average of HPS, PickScore, IeageReward and Aesthetic scores over all the prompts. To investigate how the scores align with human preference, we further compare the accuracy of these reward models on a small portion of the Pick-a-Pic training set. It is worth noticing that PickScore is most aligned with human preference according to the experiments conducted by Kirstain et al. (2023).
model on different human-
e hhtpss//dtreamliik..n// btttss//hogginggacc.comm
8

---

Table The results on the Pick-a-Pic test set. We report the mean of PickScore, HPS, ImageReward and Aesthetic over the whole test set. We also report the average score over the three evaluation metrics. SPIN-Diffusion outperforms all the baselines in terms four metrics. For this and following tables, we use blue background to indicate our method, bold numbers to denote the best and underlined for the second best.
Model
HPS
Aesthetic 1
ImageReward 1
PickScore f
Average t
SD-1.5
0.2699
5.7691
0.8159
21.1983
7.0133
SFT (reproduced) Diffusion- DPO
0.2749
5,9451
1.1051
21,4542
7.1948
0.2724
5.8635
0.9625
21.5919
7.1726
Diffusion-DPP (reproduced)
0.2753
5,8918
1.0495
21.8866
7.2758
SPIN-Diffusion-tter
0.2728
6,1206
1.0131
21.6651
7.2679
OPIN-Diffusion-tter2
0.2751
6,2399
1.1086
21.9567
7.3951
SPIN-Diffusion-les
0.2759
6.2481
1.1239
22.0024
7.4126
Main Results
In this subsection, provide empirical evidence demonstrating the superiority of our SPIN-Diffusion model over previous fine-tuning baselines based on the network structure of SD1.5.
Comparison in Terms of Average Score. The results are presented in Table While all fine-tuning algorithms yield improvements over the SD1.5 baseline, at iteration 1, our SPIN-Diffusion not only exceeds the original DPO checkpoint but also surpasses SFT both Aesthetic score and PickScore.
At iteration 2, the superiority of our model becomes even more pronounced, particularly terms of Aesthetic score, where it consistently outperforms other fine-tuning methods, indicating a dominant performance in visual quality. Furthermore, iteration 3, our model's HPSv2 score surpasses all competing models, highlighting the effectiveness and robustness of the SPIN-Diffusion approach. Specifically, on the Pick-a-Pic dataset, while SFT achieves a PickScore of 21.45, and Diffusion- achieves 22.00 at iteration 3, showing a total
has slightly higher score of 21.45, improvement of 0.80 over the original SD1.5 checkpoint. Furthermore, -PIN-Diffssion demonstrates exceptional performance in terms of Aesthetic score, achieving 6.25 at iteration 3, which significantly surpasses 5.86 achieved by Diffusion-Di DPO and 5. by SD1.5.
100
SPIN-Diffusion-ter
100
Winning Rate over 5D-1.5
Diffusion-DFO Diffusion-DPO (reproduced)
SPIN-Diffusoontter
SPIN-Diffusion-Iter
((eproduced)
SD-1.5
SPIN-Diffuionniter
Diffusionnittrrr SPIN-Diffution-tter
Diffusion-DPO
80
60
Winning
40
60
HPS
Aesthetic
Pickšcore
Average
HPS
Aesthotic
PickScore
Average
(a) Compared to SD-1.5
(b) Compared to Diffusion-DPOC (reproduced)
Figure 1: Left: winning rate in percentage of SFT, Diffusion-DPo Diffusion-DPe (reproduced) and SPIN-Diffusion over SD1.5 checkpoint. Right: winning rate in percentage of SFT, Diffusion- DPO, Diffusion-DPO (reproduced) and SPIN-Diffusion over SD1.5 checkpoint. SPIN-Diffusion shows a much higher winning rate than SFT and Diffusion-DPO tuned models.
Comparison Terms of Winning Rate. We further validate our claim by a comparative analysis of the winning rate for our trained model. The winning rate is defined as the proportion of prompts for which model's generated images exceed the quality of those produced by another model. This experiment is conducted on the Pick-a-Pic test set. We show both the winning rate over SD-1.5, as well as the winning rate over Diffusion- DPO (reproduced)
Figure [ The complete results are detailed in Tables and in Appendix A.2 We observe that throughout fine-tuning, our SPIN- Diffusion tremendously beats the baselines. When competing with SD-1.5, SPIN-Diffusin Diffusioon achieves an impressive winning rate of 90.0% at iteration 2, which further increases to 91 1.6% at iteration This winning rate surpasses 73.2% achieved by SFT and 84.8% achieved by Diffusion-DPO (reproduced). When competing with Diffusion-DPO (reproduced), iteration 3, SPIN-Diffusian

---

We show the images generated by different models. The prompts are "a very cute boy, looking at audience, silver hair, in his room, wearing hoodie, at daytime, ai language model, 3d art, c4d, blender, pop mart, blind box, clay material, pixar trend, animation lighting, depth of field, ultra detailed", "painting of a castle in the distance" and "red and green eagle". The models are: SD-1.5, SFT, Diffusion-DPo (reproduced), SPIN-Diffusiie Diffusion-Iterl SPIN- Diffusion-Iter2, UPIN-Diffusion- Iter3 from left to right. SPIN-Diffusion demonstrates notable improvement in image quality, The quantitative evaluation of the aesthetic score of the above images is in Table
Figure 2:
achieves a winning rate of 56.2% on HPS, 86.8% on Aesthetic, 62.4% on PickScore, 55.8% on Image Reward, and has an overall winning rate of 70.2%.
6.3 Qualitative Analysis
We illustrate the qualitative performance of our model on three prompts coming from the Pick-a-Pic test dataset. We prompt SD-11., SFT, Diffusion-DPO (reproduced), and SPIN-Diffusion at iteration 1 to and present the generated images in Figure 2 Compared to the baseline methods, SPIN-Diffusion
demonstrates notable improvement in image quality, even apparent than the improvements in scores. This is especially evident in aspects such as aligning, shading, visual appeal, and the intricacy of details within each image. This qualitative assessment underscores the effectiveness of SPIN-Diffusion in producing images that are not only contextually accurate but also visually superior those generated by other existing models.
7
Conclusion
This paper presents SPIN-Diffusion, an innovative fine-tuning approach tailored for diffuusion models, particularly effective in scenarios where only a single image is available per text prompt. By employing self-play mechanism, SPIN-Diffusion iteratively refines the model's performance. converging towards the target data distribution. Theoretical evidence underpins the superiority of SPIN-Diffusion, demonstrating that traditional supervised fine-tuning cannot surpass its stationary point, achievable at the target data distribution. Empirical evaluations highlight SPIN-Diffusion 's remarkable success text-to-image generation tasks, surpassing the state-of-the-art fine-tuning methods even without the need for additional data. This underscores SPIN-Diffusion's 's potential to revolutionize the practice of diffusion model fine-tuning, leveraging solely demonstration data to achieve unprecedented performance levels.
Limitations While our theoretical analysis ensures that 0g is the only global optimum of our objective function, it relies on the assumption that the data distribution can be adequately represented by the parameterized family. Additionally, algorithm, it cannot, in principle, exceed the performance the underlying data distribution. Finally, although SPIN-DiS Diif--iffuion is data-efficient, it requires additional sampling overhead. The high sampling cost can alleviated by software-level upgrades such as larger batch size, and memory-efficien attention backends. On the algorithm level, advanced sampling acceleration techniques also offer promising improvements. These techniques are orthogonal to our efforts improving the performance of fine-tuning diffusion models, and therefore decide to explore them as a future work.
our methodology is fundamentally distribution matching
10

---

Acknowledgement
We thank the anonymous reviewers and area chair for their helpful comments. HY, ZC, KJ, and QG supported in part by the National Science Foundation CAREER Award 1906169, IIS-200898 and the Sloan Research Fellowship. The views and conclusions contained in this paper are those of the authors and should be interpreted as representing any funding agencies.
References
AUSTIN, J., JOHNSON, D. D., Ho, J., TARLOW, D. and VAN DEN BERG, R. (2021). Structured denoising difffusion models in discrete state-spaces. Advances in Neural Information Processing Systems 34 17981-17993.
BETKER, J., GOH, G., JING, I.., BROOKS, T., WANG, J., LI, L.., OUYANG, L.,, ZHUANG, J., LEF., J. GUO, Y. ET AL. (2023). Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/tallle-3. pdf 3.
BLACK, K., JANNER, M., DU, Y., KOSTRIKOV, I. and LEVINE, S. (2023). Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301
BRADLEY, R. A. and TERRY, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39 324-345.
CAESAR, H., UULINGS, J. and FERRARI, V. (2018). Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition.
CHEN, Z.,, DRNG, Y., YUAN, H., JI, K. and GC, Q. (2024). Self-play fine-tuning converts weak language models strong language models. arXiv preprint arXiv:2401.01335
CHEN, Z.,, YUAN, H., LI, Y., KOU, Y., ZHANG, and GU, Q. (2023). Fast sampling via de- randomization for discrete diffusion models. arXiv preprint arXiv:2312.09199
CLARK, K., VICOL, P., SWERSKY, K. and FLEET, D. J. (2023). Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400
CORSO, G.. STÄRK, H., JING, B., BARZILAY, R. and JAAKKOLA, T. (2022). Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776.
CRESWELL, A., WHITE, T., DUMOULIN, V., ARULKUMARAN, K., SENGUPTA, B. and BHARATH, A. A. (2018). Generative adversarial networks: An overview. IEEE signal processing magazine 35 53-65.
DAI, X., HOU, J., MA, C.-Y., TSAI, S., WANG, J., WANG, R., ZHANG, P., VANDENHENDE, S., WANG, X., DUBEY, A. ET AL. (2023). Emu: Enhancing image generation models using photogenic needles a haystack. arXiv preprint arXiv:2309. 15807
FAN, Y., WATKINS, O., DU, Y., LIU, H., RYU, M., BOUTILIER, C., ABBEEL, P., GHAVAMZADEH, M., LEE, K. and LEF, K. (2023). Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305. 1638/
GAL, R., ALALUF, Y., ATZMON, Y., PATASHNIK, O., BERMANO, A. H., CHECHIK, G. and COHEN-OR D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618
GUAN, J.. ZHOU, X., YANG, Y., BAO, Y., PENG, J.. MA, J., LIU, Q., WANG, L. and Gu, Q. (2023). Decompdiff: Diffusson models with decomposed priors for structure-bri based drug design.
Ho, J., JAIN, A. and ABBEEL, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems 33 6840-6851.
Ho, J.. SAHARIA, C., CHAN, W., FLEET, D. J., NOROUZI, M. and SALIMANS, T. (2022). Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research 23 2249-2281

---

KINGMA, D. P. and WELLING, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
KIRSTAIN, Y., POLYAK, A., SINGER, U., MATIANA, S., PENNA, and LEVY, O. (2023). Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprini
arXiv:2305.01569.
LEE, K., LIU, H., RYU, M., WATKINS, O., DU, Y., BOUTILIER, C., ABBEEL, P., GHAVAMZADEI, M. and Gu, S. S. (2023). Aligning text-to-image models using human feedback. arXiv preprint 23X2:2332.12192
LEE, S. H., LI, Y., KE, J., YOO, I., ZHANG, H., YU, J., WANG, Q. DENG, F,, ENTIS, G., HE, J., LI, G., KIM, S., ESSA, I. and YANG, F. (2024). Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-image generation.
LIN, T.-Y., MAIRE, M., BELONGIE, S., HAYS, J., PERONA, P., RAMANAN, D., DOLLÁR, P. and ZITNICK, C. L. (2014). Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer.
MÜLLER, A. (1997). Integral probability metries and their generating classes of functions. Advances in applied probability 29
NICHOL, A., DHARIWAL. P., RAMESH, A., SHYAM, P., MISHKIN, P., MCGREW, B., SUTSKEVER, and CHEN, M. (2021). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini arXiv:21 /2.
PEEBLES, W. and XIE, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
PODELL, D., ENGLISH, Z., LACEY, K., BLATTMANN, A., DOCKHORN, T., MULLER, J., PENNA, J. and ROMBACH, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.
PRABHUDESAL, M., GOYAL, A., PATHAK, D. and FRAGKIADAKI, K. (2023). Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739.
RAFAILOV, R., SHARMA, A., MITCHELL, E., ERMON, S., MANNING, C. D. and FINN, C. (2023). Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305. 182990
RAMESH, A., DHARIWAL, P., NICHOL, A., CHU, C. and CHEN, M. (2022). Hierarchical text- conditional image generation with clip latents. arXiv preprint 204v:2204.00/25 1 3.
ROMBACH, R., BLATTMANN, A., LORENZ, D., ESSER, P. and OMMER, B. (2022a). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
ROMBACH, R., BLATTMANN, A., LORENZ, D., ESSER, P. and OMMER, B. (2022b). High- resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF Confer- on Computer Vision and Pattern Recognition (CVPR).
RUTZ, N., LI, Y., JAMPANI, V., PRITCH, Y., RUBINSTEIN, M. and ABERMAN, K. (2023). Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
SAHARIA C.. CHAN, W., SAXENA. S., LI, L.. WHANG, J., DENTON, E. L., GHASEMIPOUR, K., GONTUO LOPES, R., KARAGOL AYAN, B., SALIMANS, T. ET AL. (2022a). Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35 14479-36494
SAHARIA, C., Ho, J., CHAN, W., SALIMANS, T., FLEET,, D. and NOROUZI, M. (2022b). Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 4713 -726.
12

---

SCHUHMANN, C., BEAUMONT, R., VENCU, R.. GORDON, C., WIGHTMAN, R., CHERTI, M., CоOMBES, T., KATTA, A., MULLIS, C., WORTSMAN, M. AL. (2022). Laion-5b; An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 25278-25294
SEGALIS, E., VALEVSKI, D., LUMEN, D., MATIAS, Y. and LEVIATHAN, Y. (2023). A picture is worth thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310.1665
SILVER, D., HUBERT, T., SCHRITTWIESER, J., ANTONOGLOU, I., LAI, M., GUEZ, A., LANCTOT, M., SIFRE, L., KUMARAN, D., GRAEPEL, T. ET (2017a). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprini arXiv: 1712.0811..
SILVER, D., SCHRITTWIESER, J., SIMONYAN, K., ANTONOGLOU, I., HUANG, A., GUEZ, A., HUBERT, T., BAKER, L., LAI, M., BOLTON, A. AL. (2017b). Mastering the game of go without human knowledge. nature 550 354-359.
SOHLLDICKSTEIN, J., WEISS, E., MAHESWARANATIANN, N. and GANGULI, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. PMLR.
SONG, J., MENG, C. and ERMON, S. (2020a). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
SONG, Y. and ERMON, S. (2019). Generative modeling by estimating gradients of the data distribu- tion. Advances in neural information processing systems 32.
SONG, Y., DOHL-DICKSTEIN, J., KINGMA, D. P., KUMAR, A., ERMON, S. and POOLE, B. (2020b). Score-based generative modeling through stochastic differential equations. arXiv preprint vrXiv:2011.13455
TESAURO, G. ET AL. (1995), Temporal difference learning and td- id-gammon. Communications of the ACM 38 58 68.
WALLACE, B., DANG, M., RAFAILOV, R., ZHOu, L., LoU, A., PURUSHWALKAM, S., ERMON, S., XIONG, C.,, JOTY, S. and NAIK, N. (2023). Diffusion model alignment using direct preference optimization. arXiv preprini arXiv:2311.12908
WATSON, D.. Ho. J., NOROUZI, M. and CHAN, W. (2021). Learning efficiently sample from diffusion probabilistic models, arXiv preprint arXiv:2106.03800
WU, X., HAO, Y., SUN, K., CHEN, Y., ZHU, F., ZHAO, R. and LI, H. (2023). Human preference score v2: solid benehmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09344
Xu, J., LIU, X., WU, Y., TONG, Y., LI, Q., DING, M.. TANG, J. and DONG, Y. (2023). Imagere- ward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977.
YANG, K., TAo, J., LYU, J., GE, C., CHEN, J., LI, Q., SHEN, W., ZHU, X. and LI, X. (2023). Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arXiv:2311.13231.
Yu, J., Xu, Y., KOH, J. Y., LUONG, T., BAID, G., WANG, Z., VASUDEVAN, V., KU, A., YANG, Y., AYAN, B. K. ET AL. (2022). Scaling autoregressive models for content-rich text-to-image generation, arXiv preprint arXiv:2206. 10789 25.
ZHENG, L., YUAN, J., YU, L. and KONG, L. (2023), A reparameterized discrete diffusion model for text generation. arXiv preprint 27X77:2302.05737.
13

---

Broader Impact
This approach enhances model performance across diverse benchmarks without the need for supervi- sion from more advanced models, facilitating better alignment of Al with human preferences. This improvement bolsters the reliability and safety of AI systems in the field of text-to-image generation. provides a more effective and scalable method for model fine-tum theettnnng resulting in cost reductions and the expedited deployment of models that more accurately reflect human aesthetic and content preferences.
However, there exists the potential for overfitting, which may not lead to genuine improvements in real-world applications. Adhering too closely in creative fields such as text-to-image generation could inadvertently perpetuate existing societal biases in the generated imagery. Furthermore, the capability to finely tune alignment with human preferences could be exploited for unethical ends, such as crafting tailored and manipulative content or disseminating false information.
A
Additional Details for Experiments
A.1 Hyperparameters
We train the SPIN-Diffusio Diffusion, we use the AdamW optimizer with a weight decay factor of le - 2. The images are processed at 512 x 512 resolution. The batch size is set to 8 locally, alongside a gradient accumula- tion of 32. For the learning rate, we use a schedule starting with 200 warm-up steps, followed by linear decay. We conduct grid search the learning rate, coefficient 3,, and number of training steps, and choose the hyperparameters that perform the best on the validation set. We set the learning rate 2.0e for the initial coefficient 3e is chosen as 2000 for the first iteration, increasing to 5000 for the subsequent second and third iterations. The trend in different learning rate and 8t choices reveals that later iterations typically benefit from more conservative updates. Training steps are 50 for the first iteration, 500 for the second, and 200 for the third. In training the DPO model, we employ the same Adam W optimizer and maintain a batch size of and a gradient accumulation of 32. The learning rate is set to 2.0e 5, and 9, is NVIDIA A6000 GPUs We use the AdamW optimizer with weight decay of 0.01. The local batch size is set to 32 and the global batch size is set to 512. Our learning rate is 1e-5, with linear warmup for 500 steps with no learning rate decay. We save checkpoints every 500 steps and evaluate the checkpoints on Pick-a-Pic validation. We select the best checkpoint, trained after 2000 steps as our SFT checkpoint.
8 NVIDIA A100 GPUs with 80G memory. In training the SPIN-
iterations, reducing it to 5.0e 8 for the third iteration. The
2000. The total number of training steps for DPO is 350. In SFT training, we use
During generation, we use a guidance scale of 7.5, and fixed the random seed as 5775709.
A.2 Additional Results
In this section, we first illustrate the main results shown in Table 1 by Figure 3 and radar plot Figure
We present the median scores of baselines and SPIN-Diffusion on Pick-a-Pic testset in Table The results consistent to the results in Table [ We present the detailed winning rate of baselines and
SPIN-Diffusion over SD-1.5 in Table and the winning rate over Diffusion-DPP in Table 4 We present the aesthetic scores of the images in Figure in Table 5
A.3 Additional Ablation Study
We conduct ablation study to investigate several aspects in the performance of SPIN-Diffusion.
Continual Training for More Epochs. We further study the training behavior of UPIN-Diffusion by continual training within iteration 1. Both iteration I and iteration 2 commence training from the same checkpoint. However, for subsequent epochs in iteration 1, images generated by SD-1.5 are used, with SD-1.5 also serving as the opponent player. In contrast, during iteration 2, both the generated images and the opponent player originate from the iteration checkpoint. The results shown in Figure [5] are reported on the 500 prompts validation set of Pick-a-Pic. We observe that in terms of PickScore, HPS, and average score, continual training on iteration even results in a performance decay. Even in terms of Aesthetic score, continual training cannot guarantee consistent improvement. Compared to training for more epochs iteration 1, iteration has much more ideal performance. These results show the key role in updating the opponent.
14

---

22.0
6.2
21.8
6.1
Resthelic
21.7
21.6
5.9
21.4
SFT
Difi
Diffusion-DPO
5.8
Diflusion-DPO (ours)
21.3
Diffusion DFO (ours)
Diffusion
21.2
SPIN-Diffusson
Number of SPIN-Difflsion iterationss
Number of SPIN-Diflusion Iterations
(a) Aesthetic
(h) PickScore
0.276
1.10
0.275
1.05
1.00
HPS
0.273
0.95
0.272
SFT
0.90
0.271
Iffusion-DFO
Diffusion-DPO
Diffusion-DPO tours)
Diffusion DPO (ouns)
0.270
SPIN Dirfusion
Number of SPIN-DiMusion Iterations
Number of SPIN-Diffusion Iterations
(c) HPS
(d) ImageReward
Figure 3: Comparison between SPIN-Diffusion at different iterations with SD-1.5, SFT and Diffusion- DPO. SPIN-Diffusion outperforms SFT at iteration 1, and outperforms all the baselines after itera- tion 2. In the legend, Diffusion-DPr (ours) denotes reproduced version of Diffusion-DPO.
Aesthetic
SD-1.5
SFT
Diffusion DPO (ours)
PickScore
SPIN- Mif.-Diffsioo-tteer
SPIN- MiIN-Diffusion-teer
SPIN-Difusion-tter3
HPS
Image Reward
Average
Figure 4: The main result is presented in radar chart. The scores are adjusted to be shown on the same scale. Compared with the baselines, SPIN achieves higher scores in all the four metrics and the average score by a large margin. the legend, Diffusion-DPO (ours) denotes our reproduced version of Diffusion-DPO.
Evaluation on Other Benchmarks We also conduct experiment on PartiPrompts YYu al., 2022) and HPSv2 (Wu et al. 2023). PartiPrompts consist of 1632 prompts that contains wide range of categories and difficulties that beyond daily scenarios and natural objects. HPSv2 is a text-image prefence dataset, where the prompts come from DiffusionDB and MSCOCO (Lin et al. 2014) dataset.
15

---

Table 2: The results of median scores on Pick-a-Pic test set. We report the median of PickScore, HPSv2, IeageReward and Aesthetic over the whole test set. We also report the average score over the four evaluation metric. SPIN-Diffusion outperforms all the baselines regarding HPS, Aesthetic, PickScore and the average score, which agrees with the results of mean scores.
Model
HPS 1
Aesthetic t
ImageReward
PickScore
Average
SD-1.5
0.2705
5,7726
0.9184
21.1813
7.0357
SFT (reproduced)
0.2750
5,9331
1.3161
21.4159
7.2350
Diffusion-DPO
0.2729
5,8837
1.1361
21.6064
7.2248
Diffusion-DRP (reproduced)
0.2756
5,8895
1.2219
21.8995
7.3216
SPIN-Diffusion-tterl
0.2739
6.1297
1.1366
21.6464
7.2967
0.2751
6,2385
1.3059
22.0101
7.4574
Iter3
0.2761
6,2769
1.3073
22.0703
7.4827
Table 3: The winning rate over SD-1.5 Pick-a-Pic testset. SPIN-Diffusion shows the highest winning rate over SD-1.5 among all the baselines.
Model
PickScore f
HPS
ImageReward
Aesthetic f
Average 1
SFT (reproduced)
62.4
82.0
75.0
70.8
73.2
Diffusion-DPOO
78.4
75.8
65.0
65.4
79.8
Diffusion-DPP (reproduced)
83.8
81.2
71.2
69.0
84.8
SPIN-Diffusion-tterl
75.4
70.0
65.8
86.0
80.8
SPIN-Difflssionlter22 SPIN-Diffussio-Iter:
86.6
82.6
72.6
92.2
90.0
87.0
86.2
77.0
93.8
91.6
Table 4: The winning rate over Diffusion DPO (reproduced) on Pick-a-Pic testset. SPIN-Diffusion shows the highest winning rate over Diffusion DPO (reproduced) among all the baselines.
Model
PickScore t
HPS
ImageReward
Aesthetic f
Average t
SD-1.5
16.2
20.8
28.8
31.0
15.2
SFT (reproduced)
26.8
48.2
51.4
52.8
35.2
Diffusion-DPO
30.6
29.4
36.8
45.2
30.4
PPIN-Diffusion-
37.2
35.6
40.6
74.8
47.4
OPIN-Diffusion-I1
56.8
49.0
52.6
86.6
68.2
SPIN-Diffusion-leeee
62.4
56.2
55.8
86.8
70.2
Table 5: Aesthetic scores of pictures in Figure
SD-1.5
SFT
Diffusion-DPP (reproduced)
SPIN- Diffusion Iterl
Iter2
Iter3
Boy
6.171
6.096
6.072
6.158
6.407
6.831
Castle
6.180
6.346
5.995
6.886
6.993
6,940
Eagle
4.927
5,428
5,289
5.601
6.103
6.189
Table The size of benchmark datasets in our evaluation
Benchmarks
Pick-a-Pic
PartiPrompts
HPSv2
# Prompts
500
1630
3200
In our experiment, we use the prompts from test set, which contains 3200 prompts. We use same evaluation metrics before and the results are shown in Table and The results show that, on both PartiPrompts and HPSv2, SPIN-Diffusion achieves comparable performance with Diffusion DPO (reproduced) and surpasses other bascline models the first iteration. SPIN-Diffusin further
16

---

21.3
2.975
SP N4
212
21.1
5.925
5.900
5.50
23.0
5225
25.7
SPM Difusion
20.6
Epoch
(a) Aesthetic
(b) PickScore
9.15
0.69
SP N- Mudin Rer2 SP N4 Mudon mer
Diffusion
0267
9.00
8.09
0265
(c) HPS
(d) Average Score
The evaluation results on Pick-a-Pic validation set of continual training within SPIN- Diffusion iteration I, and SPIN-Diffusion iteration 2. The x-axis is the number of epochs. Consecutive epochs in iteration reach their limit quickly while switching to iteration 2 boosts the performance.
Figure 5:
Table 7: The results of mean scores PartiPrompts. We report the mean and median of PickScore, HPS, ImwaeReward and Aesthetic score over the whole dataset. We also report the average score over the four evaluation metrics. SPIN-Diffusion outperforms all the baselines in terms of four metrics.
Model
HPS 1
Aesthetic
ImwgeReward f
PickScore
Average f
SD-1.5
0.2769
5.6721
0.9196
21.8926
7.1903
SFT (reproduced)
0.2814
5,8568
1.1559
21.9719
7.3165
Diffusion-DPO
0.2815
5.7758
1.1495
22.2723
7.3698
PPIN-Diffusion-Iter
0,2783
5.9073
0.9952
22.1221
7.3257
SPIN-Diffusion-Iter2 0.2804
6.0533
1.0845
22.3122
7.4326
SPIN-Diffusion-Iterf 0.2813
6.0534
1.0893
22.3435
7.4419
reaches an average score of 7.4326 and 7.5244 on PartiPrompts and HPSv2 dataset respectively at second iteration, which outpuerforms all other baselines by large margin. These results consolidate our statement that SPIN shows a superior performance over both SFT and DPO. We also conduct qualitative result PartiPrompts and the results shown in Figure
Remarks on LoRA fine-tuning While LoRA is parameter-efficient fine-tuning method that focuses on reducing trainable parameters under resource constraints, orthogonal to SPIN-Diffusion, which utilizes self-play mechanism for fine-tuning. We also provide SFT (LoRA) fine-tuning results in Figure TT We can see that full fine-tuning generally surpasses the performance of LoRA fine-tuning. Therefore, we leave the exploration of LoRA version of SPIN-Diffusion Diffusion to future work.
17

---

Table 8: The results of median scores on PartiPrompts. We report the mean and median of PickScore, HPS, ImageReward and Aesthetic score over the whole dataset. We also report the average score over the four evaluation metries. SPIN-Diffusion outperforms all the baselines in terms of four metrics.
Model
HPS f
Aesthetic f
ImageReward t
PickScore
Average f
SD-1.5
0.2781
5,6823
1.1247
21,9339
7.2548 7.2548
SFT (reproduced)
0.2781
5,6823
1.1247
21.9339
Diffusion-DPO
0.2822
5.7820
1.3823
22.3251
7.4429
DiNN-iiffusoon-ttel
0.2793
5,8926
1.1906
22.1632
7.3814
SPIN-Diffusion-Iter2 0,2810
6,0400
1.2857
22.2998
7.4766
SPIN-Difftsion-lter 0.2825
6.0480
1.3095
22.3361
7.4940
Table 9: The results of mean scores on HPSv2. We report the mean and median of PickScore, HPS, ImageReward and Aesthetic score over the whole dataset. We also report the average score over the four evaluation metrics. SPIN-Diffusion outperforms all the baselines in terms of four metrics.
Model
HPS f
Aesthetic 1
ImageReward f
PickScore t
Average f
SD-1.5
0.2783
5,9017
0.8548
21.4978
7.1332
SFT (reproduced)
0.2846
6.0378
1.1547
21.8549
7.333
Diffusion-DPO
0.2843
6.0306
1.1391
22.3012
7.4388
SPIN-Diffusion-Iter
0.2804
6. 6944
1,0133
21.8778
7.3415
SPIN-Diffuusin-ltee2 0.2838
6.3403
1.1145
22.2994
7.5095
SPIN-Diffusion-Iree 0.2849
6.342
1.1292
22.3415
7.5244
Table 10: The results of median scores on HPSv2. We report the mean and median of PickScore, HPS, ImageReward and Aesthetic score over the whole dataset. We also report the average score over the four evaluation metrics. SPIN-Diffusion outperforms all the baselines in terms of four metrics.
Model
HPS f
Aesthetic 1
ImageReward f
PickScore
Average f
SD-1.5
0.2781
5,8529
0.9324
21.4825
7.1365
SFT (reproduced)
0.2847
6.0057
1.308
21.8211
7.3549
Diffusion-DPO
0.2847
5,9878
1.3085
22.2854
7.4666
SPIN-Diffusion-lterl
0.2803
6,1519
1.1331
21,858
7.3558
SPIN-Diffusion-Iter2
0.2839
6.3401
1.2711
22.2577
7.5382
SPIN-Diffusion-Iterr
0.2849
6.3296
1.2853
22.3029
7.5507
Table 11:
The results of LoRA fine-tuning vs. full fine-tuning.
Method
HPS
Aesthetic
ImageReward
PickScore
Average
SFT (full)
0.2749
5.9451
1.1051
21.4542
7.1948
SFT (LoRA)
0.2745
5,8573
1.1393
21.4121
7.1708
A.4 Training Dynamics of SFT and DPO
We first study the training dynamic of SPIN-Diffusion in comparison with SFT and Diffusion-DPO. and we plot the results in Figure We observe that after training with about 50k data, the performance of SFT stop improving and maintains at about 20.8 8 in PickScore, 0.270 in HPS, 5.6 in Aesthetic and 8.9 in average score. These results is significantly inferior to those achieved by SPIN-Diffusion, which achieves 21.2 in PickScore, 0.272 in HPS, 5.9 in Aesthetic and 9.1 in average score. Compared Diffusion-DPer SPIN-Diffusion achieves a superior performance without the loser image. These results demonstrate that self-play fine-tuning plays key role in SPIN- DiNf-uiffnii'' 's performance.
18

---

Figure 6: We show the images generated by different models based on prompts from PartiPrompts. The prompts are "a photo of francisco's golden gate bridge", "an aerial view of the Great Wall" and "Face of an orange frog in cartoon style". The models are: SD-1.5, SFT, Diffusion-Dii Diffusion-Dii (reproduced), FFIN-Diffusion-teer from left to right. SPIN-Diffusion demonstrates a notable improvement in image quality
B Additional Qualitative Results
In this section, provide extensive qualitative results to further support findings. We first demonstrate the impact of different random seeds on model comparisons and include wider range of visual examples support the qualitative results. We also provide image gallary in Figure [2
Effect different random seeds Random seeds sometimes influence the results produced by image generation models. Figures [ and demonstrate this effect, showcasing outputs of multiple models for the same prompt across four different random seeds. SPIN-Diffusion consistently generates higher-quality images across these variations.
More examples on Partiprompts To further showcase SPIN-Diffusion's capabilities handle wide range of styles and subjects, we present results on 10 additional prompts from the PartiPrompts dataset (totaling 1630 prompts). These examples highlight the model's ability to handle a wide range of styles and subjects. Figure 10 showcases results for of these prompts, while Figure TT [highlighhs UPIN-Diffusion ability generating cartoon-style images with additional prompts specifically containing the word 'cartoon'.
Additional Details for SPIN-Diffusion C
C.1 Additional Details DDIM.
Given a prompt c, image Xp, sequence {a+)7=1 S (0, 1] and {ar) S [0, +00), the forward diffusion process defined in [.1
(x1:T120) == g(xr(x)) II g(x; 1 Xo).
t=2
where g(xx)xx)) (xo) N(Vaxxo (1 Irr and q(xx-1 xo) admits the following distribution,
N(Vai-1x0 V 11- a1-111
a,-1 - oz v1 aj .0,1)
Vaixo
(C.1)
Here (ax) is a decreasing sequence with oo and or approximately zero. By Bayesian rule, show that this diffusion process ensures that x(xx _N(Va (1
19

---

5.9
21.2
5.8
21.0
Aesthetic
SFT
PickScore
5.7
Diffusion- SPIN-Diffusion
20.8
20.6
SFT
Diffusion-DPO
5.5
20.4
SPIN-Diffusion
200
400
600
800
200
400
600
800
Training Samples (k)
Training Samples (k)
(a) Aesthetic
(b) PickScore
9.1
0.270
Average Score
9.0
HPS
0.268
8.9
0.266
SFT
8.8
SFT
Diffusion-DPO
Diffuslon-DPO SPIN-Diffusion
0.264
SPIN-Diffusion
200
400
600
800
800
Training Samples (k)
Training Samples (k)
(c) HPS
(d) Average Score
Figure 7: The evaluation results on the Pick-a-Pic validation set of SFT, Diffusion-DPO and SPIN- Diffusion. The x-axis is the number of training data. SFT reaches its limit quickly, while Diffusion- DPO and SPIN-Diffusion continue to improve after training with over 800k data.
I) for all and reduces to DDPM (Ho et al. 2020 with special choice of v(1 a--- / 0
Given noise schedule and o,, examples from the generative model follows
Po (x
II (x (x, Xx, c) (x(lx))),
1m1
(x x,, = -No c, c,t,
Here 0 belongs to the parameter space O and ue
p c) is the mean of the Gaussian that can be
parameterized (Ho et al. 2020 Song et al. (0200a as
no(x,, c,t) = Vai-1
X;
axco c,
c0(x,, c, t), V a: - of a
(C.2)
Var
where (co(xi, C, TT are score functions that approximate noise. Compare C.2 and C.
we can
see that x -I -a--T-oiaaxx.e.. approximates Xg. and Eg approximates the noise E /1
yas
(O.
C.2 Decoupling Technique
In Section 4 we demonstrate that the objective function defined in 4.8) can be simplified to the form in (4.9). This reformulation only requires considering two consecutive sampling steps, - and t, rather than involving all intermediate steps. Now, provide detailed derivation.
Proof of Lemma 4.1
10, 0k)
20

---

Figure 8: We show the figures generated by different models based prompt from Pick-A-Pic test set. The prompt used "a picture of the sea on which a boat sails in a storm and sways in the sea". The models are SD-1.5, SFT, Diffusion-DPO (reproduced), Diffusion-Iterf and SPIN-Diffusion-Iter:. displayed from left to right. Each row shows the results for different random seed. SPIN-Diffusion demonstrates notable improvement in image quality
Iterl, SPIN-
-Eerg(() Paat
e( -
Σ
Be T
x - (x,( -o -|xx-- - cea, (xi, c, 1)
-1|xi-1 uo(xi, 1) - ||xi-1
Mo. (xi, +)]12]
T Σ Bz [||x-= (x,, c, 10112 ||x no (x1, ex
(11)
-1|x/-1 no no(x,, t)][ ||x no, (x', c, t)]12]
-Eongl (en hit (I.... Ee - 3r |||x-- -ox c, t))12
- -|xx--1 noo (xe, c, 1) ||x|_ 1o 1) + ||x|_1 no (2c/, c, c.) 1)]
Re 1x 1m p (x) (=(le))ewuiifomm( [e Bo [[]x--1 (111:x-1--o(xe
11s0-1-Mo0xee mo(x,, c, c,11
||xx-1 nro (xe, c, 1) ||x|_1 co
||x|_ no (x/ (2//, (,,,))
Tppppro (0, 0z):
where the first inequality is by Jensen's 's inequality and the convexity of the function É, the second equality is by integrating the average Σ into the expectation via ~ ( ., and the third inequality holds because the argument inside the expectation is only depend of sampling step é 1 and t.
21

---

Figure 9: We show the figures generated by different models based prompt from Pick-A-Pic test set. The prompt used "A cute hedgehog holding flowers". The models are SD-1.5, SFT, Diffusion-DPO (reproduced). SPIN-Diffusion-Ieet Iterl, SPIN- CeiN--iiffiiieeeee and SPIN-Diffusion-leet displayed from left to right. Each row shows the results for a different random seed. SPIN-Diffusion demonstrates notable improvement in image quality
C.3 Objective Function of SPIN-Diffusion
We look deep into the term this section.
X oso 1 (xe; c, 1) and ||x|_ (x), c, of (4.8 and 4.9 in
When X Follows Forward Process. have that
We have that Xo:T ~ Pdatal and by (C.1 and C.2 wc
vaxx0
Xt-1 = 101-1X0- + V a11 - of VI- at
+ +
Xt vI area(xx, + V1 a1--a-1--0
no c,t) = var-1
of -1-00-1-1-
co(xx,
Vai
where 2s ~ NNO. I). Therefore, ||x,-1 (x c, 1) can be simplified to
/a(x0
h
e0( c,t) + (a./he) ē
(C.3)
vI /1-ay
where h4 = V a:-1 0/ Vat-1/ 04-1 and -a-vaix No I) following a
vT-m.
Gaussian distribution. When 0. becomes h cc Ce--epl co c,t) c,( Vai-/atyI-- a1-1 and
with h4 [VI a6-1
Xa (0.
When X Follows the Backward Process. We have that X pa, (1cc) and
x 140 (x,,,,4, + ozE2
- -ot-1 X v1 CHED, (X', C2)) v 11-00-1- - of co, cx,, + o
aa
22

---

Figure 10: We show the figures generated by different models based prompts from PartiPrompts The prompts are "a old-tim tidt car with a large front grille", "a full moon rising above a mountain at night". "a young badger delicately sniffing yellow rose, richly textured oil painting". "a cartoon of a man standing under a tree" and "a prop plane flying low over the Great Walf". The models are: SD-1.5, SFT, Diffusion-DPO, Diffusion-DPO (reproduced), SPIN-Diffusion-tter from left to right, all utilizing the same random seed for fair comparison
X VI (nco(x), + 11- a:11 - of co(x), c,t),
uo(x(, Vau-i
Va
where e ~ NO,, Therefore, ||x|-1 - (x c, t) can be simplified to
h co. (x,, t) - cx, c, (at/hi) E112
(C.4)
where he [V a2-1 - of Var-/aa 1a--1/aaT--
When or + 0. C.4 becomes
b3 Ea, (x(, c) (x,, c, 1)1 with h, [VI- a,-1
Simple Decoupled SPIN-Diffluson Objective Function. and applying a yields,
Substituting (C.3 and CC44 into 4.9
(0, 0v) = E (ee(--Bo -B ||er - co(x,, c, t)1l ||e, Eo. (Xe, c, 1)
L
approx
23

---

HAN
BAY
Hary
Figure 11: We show the figures generated by different models based prompts from Parti Promiss The prompts are "A cartoon house with red roof", "a cartoon of an angry shark", "a cartoon of a bear birthday party", "a cartoon of house mountain" and "a cartoon of boy playing with tiger". The models are: SI)-1.5, SFT, biffusion-DPO Diffusion-DPe (reproduced), SPIN-Diffusion-lter from left to right, all utilizing the same random seed for fair comparison
- |eo, (x,, c, co c, t) )
(C.5)
where hr VI a,-1 - 04-1, X₁ a:xo (1 az)ec, and the expectation
computed over the distribution,c ~ q(c), Xo ~ Pdata (xolc, X ~ pe, (x/cc), E, ~N((0, I) and
t Uniform[1,... .,, T). (C.5) still need the intermediate steps x/, discussed below (4.9 in Section 4
we can approximate the backward process with the forward process and obtain
approxx (0,0,)
E (e 3.h? [| - c, ||e, - co. (x
cx,, c,2
||4 co co(x,, c,t
+ e en. (Xt, c, (01)))
where h, = /1 04-1 Var-ia//ve ar 0x + (1 (1- x =axx (1 - ). and the expectation computed over the distribution,c ((c), Xy Panta P (x)(c), x/ ~ po. (x6le). E, ~ NV, ef I) and é ~ Uniform(1.....
24

---

Figure 12: Image galary generated by SPIN-Diffusion, a self-play fine-tuning algorithm for diffusion models. The results are fine-tuned from Stable Diffusion v1.5 on the winner images of the Pick-a-Pic dataset. The prompts used for generating the above images are chosen from the Pick-a-Pic test set. The generated images demonstrate superior performance in terms of overall visual attractiveness and coherence with the prompts. SPIN-Diffusion is featured by its independence from paired human preference data, offering useful tool for fine-tuning custom datasets with only single image per text prompt provided.
D Proof of Theorems in Section
Proof of Theorem 5.2 Plugging
CC33 and C.4 into 4.9
yields the following loss parameterized
with Eg.
(0,0,) = EE e ( Beh? ||c co(x,, c, t) + (o./ha) cs x1|2 - -Ile-coo co. (x,, + (ae/ha) all
approxx LSPIN
+ (10///t)) <ill2 - ||a,, (x1, e, 0) - c (x e, () + (a./1k)) <112] (D.1)
where c,, e ~ (O, I). When 0, - 0, D.I can be simplified to C.5) In this proof, we will stick
to the formula (DT) to provide the proof for all a 2 O.
Since O2 is not the global optimum of LdM by condition, there exists 0" such that LDM( S
LDM (0k) which gives that
Te Tell Eo- (xr, c) - S S
1 =
(x) - -el2]]
(D.2)
where the expectation is computed over the distribution q(.),
(-)), (O,
t Uniform(1, (1,..,, T). Define g(s) = T.grIx (0", 0c) with Br = 8 /hū as follows,
((s)= rfe( = ( Sch || c (x,,,, (a./hx) Es 1112 - |||- co. (xe, e, + (a./hr) <all2
+ |(a///e) - <ill2 - ||co (x,, co- (x t) + (a./he) <e1?]
B ( - 87 ||e c c, + (at/hi)) <a|l2 || co, (xr, 1) + (a./hi) eell2
c (x/( c, t) + (a./he) <11l]] )
+ (0t///)) - ||,o, (x), -
25

---

Then we have that g(0) = and
dg ds
(0) - IE - ((()) ||e Ea- (x,, 60-
+ - - -|ee -0x (X,, + (or/hs) eall2
sall
+ (((oe/ha)) <bll2 ||E0, (x,,0, E (x,, c, + (01/he)) haa E (2)
= -4 [TTel
E (x + (0z/h4) Eili2 ETile Ea, (x,, c, (or/he) E
+ +E: <bl1 Ey: Eo, (X,, e, E0- (x 1) + (os/hh) <i1b)
(D.3)
where X2 V -I are and the expectation is computed over the distribution ~ q(-), X Idata (-Ic), -~ (0, I), ~ Uniform(1..... T} and c,. cj ~ (O Since e, e, follows standard Multivariate normal distribution and independent with x,, X Xp, we can simplify the the terms in (D3) as follows,
Ex//= Ea. (xe, c, (at/h.) 6x
= =Emll Eg (xt, e, 10 1 (o. l
(D.4)
Ey|le E Ea, c, + (a. 6x 1
co, = =Eyil co. (x,, e, 1) Ey (ot (1.) 6 1
(D.5)
Ex|l (X4, e, ca. C, + (o./ha)
1
= =Erl|coo (x,, c, - co. (x4, c, t) 1| (x)((o./hh)) </1l2
(D.6)
where we apply the property of standard normal distribution that Ee{e] = [[e[]] = 0. Plugging (D.4 D.5). (.6) into (D.3) gives that
dg (0) (0) [Eya co. (xt, ByillE Eo, (x., e, (x,, 111
ds
BBilleo (x))) (x),
ca- (X cx,
< <0,
where the last inequality is by D.2 Here
0n v orE and the expectation is computed over the distribution e (()), Xo Adata (-Ic), NO0 I), I (1..o
Therefore, there exist a Ao such that for all 0 X An we have g(A) < (()). So for those
3t sm/ sh/h? with 0 < X < λo, we have that
aapprox (0" 0x) g()) ( LSPIx Oz, 0x). LSPIN
where the inequality holds due to the choice of X. Therefore, we conclude that 02 is not the global optimum (4.9
Proof of Theorem 5.3 By 4.9 we have that,
(0.0 0z) E LE(e( Br ||xx-1 mx(xe, c, 1011 ||x-- no, (Xe, c, 1)
Lapprox
||x/_1 mo(xx e, + -ooki--or (x, (xc,,, c,,)
where the expectationis computed over the distribution Paata Paata(Xxlc))a (xn)c)((x,-1)) x, x, X (x)-
g(c), (x1-1,
Po, (x1 [ (x)lc))(x/__) X² x )xx6 Uniform(1,.... T). Since (-atal-lc) po, (-lc), we can conclude that (x--1, and (x/_1, x() are independent and identically distributed random variable. Therefore, by symmetry property of (X--1,X), xx) and (x/_1, x(), we have for any 0 e θ that
26

---

L (0,0)) LIPI
2 2 e B ||X-_ 10 c, t)112 ||xx--1 Mo, (xe,, e, t)ll2
1
|||[_ c, ||x|_ 1o. (x4, (x4,,,, c.
+ e 8. [|xx (x(x ||x|_1 Mo, (x/ c, t)ll
| ,,( ||xx-1 no. (xe, (xx,,,,)) 2) }
E
8c
2 || -
mo(x,, c,
t)111 ||,,- nbx (xe,, t)
|| X nx c, 1) ||x-_| uo, (X/, c, 01]1
31
|| no(x c, c; -1|x'-1 po MoL c, c,
2
||x ,o(x ||x no (XL, c,
E(O),
where the inequality is due to Jensen's inequality (recalling that is convex in As- sumption 5.1 and the expectation is computed over the distribution (x(-1.X)) Paata ( (x)lc)g(Xx-1 X2 )x))xx, (x-11xx)) ~ J pe, x t Uniform ............. T). Therefore, we have that
((c).
D (0, 0&) 2 (()) LSPINN
(Ok, 0k),
which means that 0₂ is the global optimum of
4.9 As consequence, 0k+1 = O
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We validate our claim through theoretical analysis and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations after our conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
The assumptions and proof are fully presented.
28

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide a detailed descriptions on model, data, pipeline and parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
29

---

Justification:
We are not able to reorganize the code at the time of submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ tiaaps.cc/ubllic/uuddees /Coeee/mmddebbeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training methods detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not involved in our paper. However, provide the qualitative results of different random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
30

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computing resource with experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neuripsh http:///........bccctccccccaceaa
Question:
Answer: [Yes]
Justification: Ethics.
The research conducted in this paper conform with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We place our broader impact section the start of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper proposes fine-tuning methodology that is generally applicable to any pretrained language model and preference model, and poses no particular such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that produced the dataset and base models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "kzf0z73blD", "reviewer": "Reviewer_K3rU", "summary": "- This paper introduces SPIN-Diffusion, a new self-play fine-tuning technique for diffusion models that improves iteratively by competing with previous versions.\n- They show that SPIN-Diffusion outperforms existing supervised and reinforcement learning fine-tuning methods in aligning with human preferences and enhancing visual appeal starting from the first iteration.\n- This method is more data-efficient, achieving superior results with fewer data, which is beneficial in cases where there are limited images available per text prompt.\n- The paper uses a competitive setup between two iterations of the model to generate and evaluate images, considering all generated images in the evaluation process, not just the final product.\n- Their experiments on the \"Pick-a-Pic\" dataset demonstrate that SPIN-Diffusion consistently surpasses other methods in multiple performance metrics through successive iterations.\n- The approach is cost-effective and offers a practical solution for improving diffusion models, particularly useful in environments with restricted data access.", "questions": "- I would like to see a practical comparison of computation time in GPU hours between SPIN-Diffusion and other methods.", "limitations": "- The sampling overhead is significant, requiring 5-10 times more training time.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The extension of SPIN to diffusion models is well-formulated for the problem at hand.\n- Theoretical explanations are detailed and well-supported.\n- The design of an approximate version of the objective function, considering computational efficiency, appears practical.\n- The method shows practical utility by outperforming previous methods that required \"loser\" samples within a few iterations.", "weaknesses": "- The sampling overhead is significant, requiring 5-10 times more training time.\n- There is insufficient explanation regarding the assignment of the hyperparameter \\( \\beta_t \\) and its variation across iterations.\n- If a stronger approximation is applied and the sampling overhead is reduced by focusing on trajectories rather than all time steps, it might be possible to compare improvements with significantly reduced training time. Showing improvements within a setup that grants no more than twice the training time compared to SFT could have demonstrated the efficacy of SPIN-Diffusion more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The sampling overhead is significant, requiring 5-10 times more training time.\n\n**A1**: Thank you for highlighting this concern. Since this is a common concern among all reviewers, during the rebuttal period, we have worked on practical solutions addressing the sampling overhead problem, and listed the results in the general response. In summary, the current sampling time can be reduced to 7 hours vs. originally 43 hours per iteration. Further improvements are possible with more software/hardware/algorithm level designs.\n\n---\n\n**Q2**: There is insufficient explanation regarding the assignment of the hyperparameter $\\beta_t$ and its variation across iterations.\n\n**A2**: Thank you for your insightful comment regarding the hyperparameter $\\beta_t$. Due to space constraints, we deferred the detailed discussion of hyperparameters to Section B.1 of the appendix. To summarize:\n\n- $\\beta_t$ values: First iteration: 2000, Second and third iterations: 5000\n\n- Selection process: We conducted a grid search over {2000, 5000, 10000} to determine these values.\n\n- Rationale: Our experiments revealed that later iterations typically benefit from more conservative updates, hence the larger $\\beta_t$ value (5000) for the second and third iterations.\n\n---\n\n**Q3**: If a stronger approximation is applied and the sampling overhead is reduced by focusing on trajectories rather than all time steps, it might be possible to compare improvements with significantly reduced training time. Showing improvements within a setup that grants no more than twice the training time compared to SFT could have demonstrated the efficacy of SPIN-Diffusion more clearly.\n\n\n**A3**: \nThank you for your suggestion. We would like to clarify that the current training time of our method is already no more than twice that of SFT.  In Section 3.4, in addition to formulating the problem with respect to uniformly sampled $t$, we further approximate the reverse timesteps in the expectation to the forward process timesteps. This allows the application of the same training mechanism of SFT, i.e., the inputs of the model are two images (real vs. generated), a batch of data consisting of noisy samples $x_t$’s from randomly sampled $t$’s. In addition, in (3.1) and (3.2) when we formulate the problem, we have already used a trajectory-wise characterization of the IPM and reward function. The trajectory-wise derivation and the approximation in Section 3.4 together ensure that the training time of our algorithm is about twice that of SFT.\n\nIn addition, as we mentioned before, we have managed to reduce the sampling time by 83% during our rebuttal, and now the sampling time is less than the training time.\n\n---\n\n**Q4**: I would like to see a practical comparison of computation time in GPU hours between SPIN-Diffusion and other methods.\n\n**A4**: Following your suggestion, we have provided detailed statistics on the overhead and dataset requirements during each stage of training/sampling. The results were all obtained using 8 A100 GPUs (80 GB memory). From the table below, we can see that by applying fast sampling acceleration techniques, as detailed in our general response, we can control the total overhead of our algorithm within a reasonable amount.\n\n| Method                                | Sampling Time (Per 2048 Images) | Training Time (Per 2048 Images) | Sampling Time (Iter 1) | Training Time (Iter 1) | Sampling Time (Iter 2) | Training Time (Iter 2) | Sampling Time (Iter 3) | Training Time (Iter 3) | Total |\n|---------------------------------------|---------------------------------|---------------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|-------|\n| SPIN-Diffusion (Initial Submission)   | 342s                            | 73s                             | 43h                    | 1h                     | 43h                    | 10h                    | 43h                    | 6h                     | 146h  |\n| SPIN-Diffusion (Fast Sampling)        | 56s                             | 73s                             | 7h                     | 1h                     | 7h                     | 10h                    | 7h                     | 6h                     | 38h   |\n| SFT                                   | -                               | 37s                             | -                      | 20h                    | -                      | -                      | -                      | -                      | 20h   |\n---\nNote: The training times listed depend on the number of steps trained during each iteration, which are selected by the validation results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6xNfhqumX7", "reviewer": "Reviewer_CNjW", "summary": "This paper introduces a method called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the model engages in a competitive process with its earlier versions, driving iterative self-improvement. This method presents an alternative to conventional supervised fine-tuning and RL strategies. Experimental results on the Pick-a-Pic dataset show that SPIN-Diffusion outperforms previous supervised fine-tuning in human preference alignment and visual appeal. Key contributions of this work include the introduction of SPIN-Diffusion and its empirical validation demonstrating superior performance compared to existing fine-tuning methods.", "questions": "1. The \"(ours)\" notations in the tables are a bit confusing, may misleading the audience considering them as a method proposed in this work.\n\n---\n\nAfter rebuttal: Some of my initial concerns have been addressed, while I still hold concerns on the method insight, time efficiency, and quantitative improvements. By comprehensively considering the cons and pros of this paper, my final raiting would be borderline accept.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper for the first time applies SPIN to diffusion model to my knowledge.\n\n2. Theoretical analysis shows that the proposed approximate SPIN loss is an upper bound of exact SPIN loss.", "weaknesses": "1. The main contribution of this work is an approximate SPIN loss compared to the previously proposed exact SPIN loss. The major modification is moving the average over sampling steps $t$ outside the loss function, resulting in an upper bound. I kindly argue this improvement is straightforward when transferring SPIN from LLM to diffusion model, without much insights into diffusion model itself.\n\n2. Since the approximate SPIN loss is proposed for data/memory/time efficiency, I didn't see any computing efficiency comparisons of the approximate and exact SPIN loss in the main paper or supplements. As noted by the authors, \"it requires additional sampling overhead, approximately 10 times of the training time when using traditional DDIM sampling.\" This additional computing cost significantly limits the method in practice.\n\n3. Assumption 4.1 assumes the loss $l(t)$ being monotonically decreasing and convex. It seems like the SPIN loss proposed in Equations. 3.8 and 3.9 do not meet this strong assumption. It makes the analysis results less convincing.\n\n4. Quantitative performance improvement (Table 1) is mild.\n\n5. It's hard to tell which method generates the best pictures in qualitative comparison (Figure 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The main contribution of this work is an approximate SPIN loss compared to the previously proposed exact SPIN loss. The major modification is moving the average over sampling steps  outside the loss function, resulting in an upper bound. I kindly argue this improvement is straightforward when transferring SPIN from LLM to diffusion model, without much insights into diffusion model itself.\n\n**A1**:\nWe appreciate your feedback, but we would like to emphasize that the derivation of SPIN loss for diffusion models is highly non-trivial. One of the novelty of our approach lies in adapting the SPIN objective for diffusion models by considering the main player (reward function) across the full trajectory $x_{0:T}$ (as in Equation 3.2), rather than focusing solely on the final state $x_0$ as done in prior work (e.g., Fan et al., 2023; Black et al., 2023; Wallace et al., 2023). This modification allows us to formulate an exact objective function up to Equation (3.8), which departs from Wallace et al., 2023 that only considers the end state for rewards or preferences.\n\nMoreover, there seems to be a misunderstanding about our approximate training objective. The exact computation of Equation (3.8) is impractical due to two primary constraints: the large trajectory length $T$, which would require an impractical amount of GPU memory when the loss is summed over $T$. Additionally, the required samples from a reverse process are not readily accessible. Consequently, our approximation strategy includes adopting an upper bound and using samples from the forward process as practical surrogates for the unavailable backward process samples.\n\nThis approximation is directly motivated by the nature of diffusion models, which inherently decouple operations on a per-time-step basis. We provide a theoretical justification for our approximation method in Section 4, ensuring that our approximations are both practical and theoretically sound.\n\nThank you again for your suggestion. In the revision, we will highlight the insights from our derivation related to diffusion models.\n\n---\n\n**Q2**: Since the approximate SPIN loss is proposed for data/memory/time efficiency, I didn't see any computing efficiency comparisons of the approximate and exact SPIN loss in the main paper or supplements. As noted by the authors, \"it requires additional sampling overhead, approximately 10 times of the training time when using traditional DDIM sampling.\" This additional computing cost significantly limits the method in practice.\n\n**A2**:\nIn terms of training time, our approximate SPIN loss is approximately 2 times the training time of SFT loss. The additional sampling overhead, as in our discussion of limitations, primarily arises from the generation of synthetic data. This inference process, apart from training, is flexible, parallelizable, and distributable across various computing engines, from large GPU clusters to standard home laptops.\nTo address the sampling overhead, we have explored advanced sampling algorithms and optimizations at both the software and hardware levels. These efforts have successfully reduced the sampling time by 83%. Further details can be found in the general responses to all reviewers.\n\n---\n\n**Q3**: Assumption 4.1 assumes the loss being monotonically decreasing and convex. It seems like the SPIN loss proposed in Equations. 3.8 and 3.9 do not meet this strong assumption. It makes the analysis results less convincing\n\n**A3**: We believe this is a misunderstanding. Assumption 4.1 is made on $\\ell$, rather than the SPIN-Diffusion loss in Equations 3.8 and 3.9. More specifically, we choose $\\ell$ to be the logistic loss (in SPIN-Diffusion experiments), which is monotonically decreasing and convex. Other losses such as correlation loss and hinge loss also satisfy this assumption. So our theoretical analysis indeed holds for SPIN-Diffusion. We will clarify it in the revision.\n\n---\n\n**Q4**: Quantitative performance improvement (Table 1) is mild.\n\n**A4**: There might be a misinterpretation regarding the performance. The SPIN-Diffusion model in fact significantly outperforms baselines SD-1.5 and Diffusion-DPO, with aesthetic scores improving from 5.7691 to 6.2481 and PickScores reaching 22.0024, even exceeding winner images from the Pick-a-Pic test set, which have an aesthetic score of 5.985 and a PickScore of 21.87.\n\nThe improvement is more obvious in Figure 1 and Tables 3, 4, where the winning rate over SD-1.5 reaches 91.6%. \n\n---\n\n**Q5**: It's hard to tell which method generates the best pictures in qualitative comparison (Figure 2).\n\n**A5**: Images generated by SPIN-Diffusion iterations are generally more aesthetically pleasing. To offer a more objective comparison, we assessed the aesthetic scores of the 3 images in Figure 2. The results indicate that SPIN-Diffusion consistently outperforms SD-1.5, SFT, and Diffusion-DPO in visual quality:\n\n|        | SD-1.5 | SFT   | Diffusion-DPO | SPIN-Diffusion Iter1 | Iter2 | Iter3 |\n| ------ | ------ | ----- | ------------- | ----- | ----- | ----- |\n| Boy    | 6.171  | 6.096 | 6.072         | 6.158 | 6.407 | 6.831 |\n| Castle | 6.180  | 6.346 | 5.995         | 6.886 | 6.993 | 6.940 |\n| Eagle  | 4.927  | 5.428 | 5.289         | 5.601 | 6.103 | 6.189 |\n\nFor additional qualitative comparisons, please refer to Figures 8, 9, 10, and 11 in the appendix.\n\n---\n\n**Q6**: The \"(ours)\" notations in the tables are a bit confusing, may misleading the audience considering them as a method proposed in this work.\n\n**A6**: Thank you for your suggestion. We've replaced \"(ours)\" in tables and figures with “(reproduced)” to clearly distinguish between existing checkpoints and our reproductions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNfWEQZnr8", "reviewer": "Reviewer_WzS9", "summary": "The paper introduces a novel method called SPIN-Diffusion for fine-tuning text-to-image diffusion models. SPIN-Diffusion uses a self-play mechanism where the model competes against its earlier versions to iteratively improve its performance. This approach eliminates the need for human preference data, which is a significant requirement for traditional reinforcement learning-based fine-tuning methods. The experiments on the various dataset demonstrate that SPIN-Diffusion outperforms existing supervised fine-tuning methods and reinforcement learning-based approaches, achieving higher human preference alignment and visual appeal with less data.", "questions": "- Why should fine-tuning the DMs use RL methods? Could you please discuss the advantages of fine-tuning DMs with RL compared to traditional methods? \n\n- As the authors claimed, \"In many datasets including the community-sourced ones featuring custom content, it is often the case to have only one image associated with each prompt. This makes RL fine-tuning infeasible.\"I do not understand why this approach is considered infeasible. In my prespective, one image and its prompt can be considered as the observation of agents. The other processes are similar to the standard RL paradigm. Therefore, it would be greater to discuss more.", "limitations": "Please refer to weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper presents an innovative self-play fine-tuning method that does not rely on human preference data, addressing a significant limitation in current fine-tuning approaches.\n\nS2: Extensive experiments are conducted, showing that SPIN-Diffusion outperforms both supervised fine-tuning and reinforcement learning-based methods in terms of human preference alignment and visual appeal.\n\nS3: The theoretical analysis provides a strong foundation for the proposed method, demonstrating its convergence and superiority over traditional supervised fine-tuning methods.\n\nS4: The paper effectively communicates the technical challenges and solutions, making the methodology accessible to readers.", "weaknesses": "W1: The paper lacks a comparison with traditional fine-tuning methods for diffusion models, e.g., LoRA.\n\nW2: The computational overhead of the self-play mechanism is high, requiring 5-10 times more training time compared to baselines, which might limit its practical application.\n\nW3: The motivation of this method is unclear to me. It would paint a more holistic picture of the problem and solution.\n\nW4: The paper assumes that the data distribution can be adequately represented by the parameterized family, which may not hold in all practical scenarios.\n\nW5: The evaluation is primarily focused on a single dataset (Pick-a-Pic), and additional benchmarks could strengthen the generalizability of the results.\n\nW6: The font size of Figure 1 is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The paper lacks a comparison with traditional fine-tuning methods for diffusion models, e.g., LoRA\n\n**A1**: While LoRA is a parameter-efficient fine-tuning method that focuses on reducing trainable parameters under resource constraints, it is orthogonal to SPIN-Diffusion, which utilizes a self-play mechanism for fine-tuning. According to your suggestion, we have provided SFT (LoRA) fine-tuning results as follows. We can see that full fine-tuning generally surpasses the performance of LoRA fine-tuning. \n| Method     | HPS    | Aesthetic | ImageReward | PickScore | Average |\n| ---------- | ------ | --------- | ----------- | --------- | ------- |\n| SFT (full) | 0.2749 | 5.9451    | 1.1051      | 21.4542   | 7.1948  |\n| SFT (LoRA) | 0.2745 | 5.8573    | 1.1393      | 21.4121   | 7.1708  |\nWe will add this additional experiment result to the revision.\n\n---\n\n**Q2**: The computational overhead of the self-play mechanism is high\n\n**A2**:  As this is a common concern raised by all reviewers, we have provided a general response regarding the sampling overhead. In summary, we have implemented some of the “advanced sampling acceleration techniques” and reduced the sampling overhead by 83%.\n\n---\n\n**Q3**: The motivation of this method is unclear to me\n\n**A3**: Thank you for your comments. Here is the motivation for our work. The standard SFT method for diffusion models suffers from low alignment with human preferences and low data efficiency due to two main reasons: (1) it does not directly optimize for alignment with human preferences, and (2) only one round of training can be performed. To address this limitation, prior works (Fan et al., 2023; Black et al., 2023; Wallace et al., 2023) have proposed to use RL fine-tuning (RLHF) to directly align the diffusion model with human preferences. However, RL fine-tuning methods also have limitations: they either require an external reward function trained on additional data  (Fan et al., 2023; Black et al., 2023) or rely on expensive human-annotated winner/loser paired images for each prompt (Wallace et al.). In order to overcome the limitations of existing fine-tuning methods, we propose SPIN-Diffusion, which overcomes the drawbacks of both SFT and RLHF through a self-play mechanism. Compared with SFT, our method is more data-efficient, by repeatedly using the prompts from the SFT dataset to improve the model through self-play. Compared with RLHF methods, our method does not need external reward models or expensive human-annotated winner/loser pairs. We will highlight the motivation of our work in the revision.\n\n---\n\n**Q4**: The paper assumes that the data distribution can be adequately represented by the parameterized family, which may not hold in all practical scenarios\n\n**A4**: Thank you for your suggestion. It is indeed a common assumption in diffusion model algorithms that the parameterized family is expressive enough.For example, DDPO (Black et al., 2023), DPOK (Fan et al., 2023) and Diffusion-DPO are all built upon the assumption of an expressive reward model to capture data preferences accurately. To our knowledge, our work provides the first convergence guarantee along this line of research. While this assumption is solely introduced for theoretical analysis, we believe that the expressiveness of large neural networks generally satisfies this assumption in practice.\n\n---\n\n**Q5**: The evaluation is primarily focused on a single dataset\n\n**A5**: We believe this is a misunderstanding. Our evaluation is in fact performed on three datasets. In Section 5.1 “We use the Pick-a-Pic test set, PartiPrompts (Yu et al., 2022) and HPSv2 (Wu et al.,2023) as our evaluation benchmarks.”. Due to space limit of the main text, additional results for PartiPrompts and HPSv2 are provided in Section B.3 “Evaluation on Other Benchmarks” of our appendix.\n\n---\n\n**Q6**: The font size of Figure 1 is too small\n\n**A6**: Thank you for your feedback. We have increased the font size in Figure 1 for better readability in our revision, which is now available in the uploaded PDF.\n\n---\n\n**Q7**: Could you please discuss the advantages of fine-tuning DMs with RL compared to traditional methods? \n\n**A7**: We do not view our method as an RL fine-tuning method. Here we will highlight the advantage of self-play fine-tuning for diffusion models over both SFT and RL fine-tuning. Standard SFT of diffusion models maximizes the log-likelihood of the training data. However, it cannot directly optimize the diffusion model’s performance in terms of quality indicators, as log-likelihood is not directly related to any human-perceived quality indicators. RL fine-tuning, on the other hand, overcomes this limitation by utilizing a reward function to maximize or by using human-annotated winner/loser image pairs as feedback, thereby optimizing the model to generate images with higher rewards (e.g., aesthetic score, image quality, etc.).  In contrast, self-play fine-tuning uses an implicitly defined reward function integrated within the diffusion model's training process, allowing it to fully utilize the SFT dataset without requiring an additional reward model or human feedback.\n\n---\n\n**Q8**: I do not understand why this approach is considered infeasible\n\n**A8**: We would like to clarify that “RL fine-tuning is infeasible” because of the lack of reward information or human feedback. As you mentioned, the prompt and the image can be considered as observations, but they do not directly provide a reward. As previously explained, RL fine-tuning methods rely on a reward function to optimize performance or on the human-annotated winner/loser image pairs as feedback. In contrast, SPIN/SPIN-Diffusion uses a self-play mechanism, which only requires a high-quality SFT dataset, removing the need to train a reward model that accurately reflects human preferences or human-label winner/loser pairs. Theoretically, we prove that SPIN-Diffusion performs distribution-level matching to the target data distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for the constructive feedback! To address some common concerns, we summarize the improvements on sampling overhead that we have done during rebuttal period as follows:

**Sampling Overhead**: By using batching, torch precompiling, and DPMSolver, we reduced the sampling time from 43 hours to 7 hours, achieving an 83% reduction in sampling time.

The improvements are summarized in the table below:

|                  | **Sampling Time Per 2048 Images** | **Training Time Per 2048 Images** | **Sampling Time Per Iteration** |
|---------------------------------|-----------------------------------|-----------------------------------|---------------------------------|
| Initial Submission              | 342s                              | 73s                               | 43h                             |
| Revision (Batch + Precompile)   | 136s                              | 73s                               | 17h                             |
| Revision (Batch + Precompile + DPMSolver) | 56s                  | 73s                               | 7h                              |

---

These results were obtained using a machine with 8 × A100 GPUs (80G memory per GPU), with samples distributed to the GPUs via data parallelism. The specific optimizations are explained as follows:
- **Batching**: Allows the diffusion score network’s input to be a batch of samples instead of a single sample. In our revision, we use a batch size of 64.
- **Torch Compile** [1]: A feature offered by PyTorch 2.0 that saves time during repeated inference by precompiling the code for efficient execution.
- **DPMSolver** [2,3]: A high-order diffusion ODE solver introduced in recent research. From our experiments, DPMSolver with 20 steps of reverse sampling exceeds the performance of PNDMSolver (default Stable Diffusion Scheduler) with 50 steps.


There are other techniques that could potentially further reduce sampling overhead. On the software level, approaches such as memory-efficient attention backend, Nvidia TensorRT, and DeepSpeed Inference can be explored. On the algorithm level, methods like UniPC, EDM, and DEIS offer promising improvements. These techniques are orthogonal to our efforts in improving the performance of fine-tuning diffusion models, and therefore we decide to explore them as a future work.

---

[1] Von Platen et al. Diffusers: State-of-the-art diffusion models.

[2] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2022). Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.

[3] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2022). Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.

Meta-review: The paper proposes an approach for fine-tuning image diffusion models, based on the SPIN (self-play) approach previously proposed for language models. Modifying the method for diffusion requires non-trivial technical innovations. The method is evaluated on several datasets and is shown to outperform alternative fine-tuning methods like SFT and DPO.

After considering the reviews, rebuttals, discussion, and the paper itself, here is a brief list of key pros and cons.

Pros:
1. Interesting technical contribution
2. Good empirical results on several datasets according to "in-silico" metrics

Cons:
1. The method is substantially slower than other fine-tuning approaches -> this was largely addressed in the rebuttal/discussion
2. Image quality improvements do not appear to be breathtaking - neither qualitative nor quantitative
3. No human evaluation
4. Only tested with one - and fairly old - base model

Overall, even though the evaluation could be more thorough and human evaluation would help a lot, the paper has its merit, presents an interesting method, a clear story, and a fairly thorough evaluation, so I recommend acceptance
Decision: Accept (poster)