Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization
Authors: Zichen Miao
Zhengyuan Yang
Kevin Lin
Ze Wang
Zicheng Liu
Lijuan Wang
Qiang Qiu
Venue: ICLR 2025 Poster (2025)
Keywords: Distilled Diffusion Models; Fine-tuning
Primary area: generative models

Abstract: Recent advancements in timestep-distilled diffusion models have enabled high-quality image generation that rivals non-distilled multi-step models, but with significantly fewer inference steps. While such models are attractive for applications due to the low inference cost and latency, fine-tuning them with a naive diffusion objective would result in degraded and blurry outputs. An intuitive alternative is to repeat the diffusion distillation process with a fine-tuned teacher model, which produces good results but is cumbersome and computationally intensive: the distillation training usually requires magnitude higher of training compute compared to fine-tuning for specific image styles. In this paper, we present an algorithm named pairwise sample optimization (PSO), which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model. PSO introduces additional reference images sampled from the current time-step distilled model, and increases the relative likelihood margin between the training images and reference images. This enables the model to retain its few-step generation ability, while allowing for fine-tuning of its output distribution. We also demonstrate that PSO is a generalized formulation which be flexible extended to both offline-sampled and online-sampled pairwise data, covering various popular objectives for diffusion model preference optimization. We evaluate PSO in both preference optimization and other fine-tuning tasks, including style transfer and concept customization. We show that PSO can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data. PSO also demonstrates effectiveness in style transfer and concept customization by directly tuning timestep-distilled diffusion models.

Published as a conference paper at ICLR 2025
TUNING TIMESTEP-DISTILLE DIFFUSION MODEL USING PAIRWISE SAMPLE OPTIMIZATION
Zichen Miao', Zhengyuan Yang", Kevin Lin", Ze Wang
Zicheng Liu Lijuan Wang', Qiang Qiu Purdue University, Microsoft, SAMD
ABSTRACT
Recent advancements in timestep-distilled diffusion models have enabled high- quality image generation that rivals non-distilled multi-step models, but with sig- nificantly fewer inference steps. While such models attractive for applications due to the low inference cost and latency, fine-tuning them with a naive diffusion objective would result in degraded and blurry outputs. An intuitive alternative is to repeat the diffusion distillation process with a fine-tuned teacher model, which produces good results but is cumbersome and computationally intensive: the dis- tillation training usually requires magnitude higher of training compute compared fine-tuning for specific image styles. In this paper, we present an algorithm named pairwise sample optimization (PSO), which enables the direct fine-tuning diffusion model. PSO introduces additional refer- ence images sampled from the current time-step distilled model, and increases the relative likelihood margin between the training images and reference images. This enables the model to retain its few-step generation ability, while allowing for fine- tuning of its output distribution. We also demonstrate that PSO a generalized formulation which can be flexibly extended to both pllline-sample and online- sampled pairwise data, covering various popular objectives for diffusion model preference optimization. We evaluate PSO in both preference optimization and other fine-tuning tasks, including style transfer and concept customization. We show that PSO can directly adapt distilled models human-preferred generation with both offline and moline-generater pairwise preference image data. PSO also demonstrates effectiveness style transfer and concept customization by directly tuning timestep-distilled diffusion models. The code is provided at: https: 7/github. com/ hww.:o////iiiceeaaaaaeeeeeaa ieaaaiiwisss Sampe__aale____eeeeeeeeeeeeeeeee imtiiiiiio
of an arbitrary timestep-distilee
INTRODUCTION
Diffusion models Ho al.) 2020: Song et al. 2020 Nichol & Dhariwal 2021 Ho & Salimans 2022 Karras et al.) 2022 Rombach et al. 2022) have shown strong capabilities in generating high- fidelity images, marking significant advancement in the field of generative modeling. Despite their impressive performance, a notable challenge is the high inference cost due to its iterative denoising nature. To address this issue, various methods are proposed to accelerate the sampling process of diffusion models, including improving the efficiency of samplers (Karras et al.) 2022 Lu et al.)
2022a b Liu al. 2022; Zhao al., (024) and employing model distillation techniques (Song
et al. 2023; Luo et al.) 2023] Sauer et al. 2023 Xu et al. 2023; Lin et al.) 2024, Sauer et al., 2024
Yin al. 2024b Ren et 2024 Kohler al. 2024) to reduce the number of inference
steps. Recent advancements in trajectory distillation methods and distribution matching techniques,
often enhanced by adversarial learning at scale, have shown considerable promise generating high-fidelity images in extremely low steps such as to four steps.
Despite significant advancements in timestep-distilled diffusion models, it remains unclear how to effectively fine-tune customize such distilled models. Naively tuning the distilled model with dif- fusion loss will make the generation results blurry, as shown in Figure I (b). An alternative approach is to fine-tune or customize the original diffusion model, and then repeat the diffusion distillation process to create a distilled model variant. However, the large computation cost of diffusion distil- lation, when compared with the customization training used for applications (cf., 3840 A 1100 GPU

---

Published as a conference paper at ICLR 2025
Human Preference Tuning with SDXL-DMDD
Pokemon Style Transfer with SDXL-Turbo
(a) Original: Pretrained Timestep-Diste Distilled Diffusion Models.
(b) Fine-tuned with vanilla diffusion objective.
(c) PSO : Fine-tuned with our pairwise sample optimization.
Figure 1: Illustration of images sampled from: (a) original timestep-distilled diffusion models, (b) Fine-tuned distilled model with diffusion objective (Ho et al. (020). and (c) Fine-tuned distilled models with our PSO, and can be seen that simply tuning distilled models with the vanilla diffu- sion loss leads to blurry, degraded generation, while our method can steer the distilled model toward better alignment with human preference prompts, and style-transferred generation. Prompt from left to right: A Pirate in Pirateship.// a woman with long hair next to a luminescent bird.// Photo- graph of a wall along a city street with watercolor mural of foxes in a jazz band// A stern-faced, crown and piercing red eyes.// cute rabbit.
brown-featheres owl Pokémon with
hours for SDXL-DMD2 Yin al., 2024a) and 0.25 A A100 GPU hours for Dreambooth concept customization (Ruiz et al. 2023)). often makes such distilled model tuning approach less feasible.
In this paper, we present an effective algorithm named pairwise sample optimization (PSO), de- signed to directly tuning a timestep-distilled diffusion model for different user preferences or cus- tomizing it towards new image domains. PSO maximizes the likelihood ratio between a given pair of target and reference images, where the target image is sampled from the distribution tune the distilled model towards, and the reference image is sampled from the generative distribution of the tuned distilled models on-the-fly. By optimizing this relative likelihood objective, PSO effec- tively aligns the timestep-distileed models with target distributions while best preserving its original few-step generation ability learned through diffusion distillation. Our formulation of PSO can be flexibly adapted to both offline and online settings, and unifies various prior works (Wallace et al.) 2024 Yang et al. 2024a) as special cases to our framework.
wish to
We validate the effectiveness of the proposed method across broad range of tasks, including human preference tuming, style transfer, and concept customization. For preference tuning and style transfer, construct or directly utilize target-reference images with the given prompts, e.g., win-lose pairs from the user preference data (Kirstain et al.) 2023). and adopt our offline PSO objective tune the distilled models. We also examine online objective in preference tuning, where sample pairs of images from distilled model and use a reward mode to assign the label of target or reference. For concept customization, from the tuned model. In all tasks and settings, PSO enables the lightweight fine- mee-tuning of arbitrary distilled models while preserving their few-step generation capability, shown in Figure (c).
use the given concept images target and sample reference images
We evaluate our method on various datasets and benchmarks. For human preference tuning, we benchmark PSO the standard Pick-a-Pic (Kirstain et al. 2023) and PartiPrompts Yu et al. 2022)
2

---

Published as a conference paper at ICLR 2025
Timestep-Distille Diffusion Model Distribution Pilgre
R
to
Rögra
Xts
X
Xts
X1o
Leso T
Páato (Ka
Noise
Noise Xr
Clean Data Xg
Data Distribution Paata
Figure 2: Demonstration of the proposed pairwise sample optimization. To tune the generative distribution pe to Pdata. we sample a pair of images together with their trajectories of the same prompt, where adopt Markov Decision Process (MDP) formulation for the timestep-distilles diffusion model to efficiently sample the backward denoising trajectories {.f,, }. while sampling {.T,) from data via the forward diffusion process. The sampled trajectories are then to the final objective to move the generation trajectory aligned with the forward process from Pdota-
datasets. We also evaluate distilled model fine-tuning by experimenting the image style transfer task with the Pokemon dataset (Pinkney) (222) and the concept customization task with the excamples in Dreambooth (Ruiz et al., 2023). We observe that PSO shows comparable performance to the oracle results of fine-tuning new teacher model and conduct the diffusion distillation training, while being much less computationally expensive. Our approach also significantly outperforms other designed baselines achieved via the use of LoRA parameters.
Our contributions are summarized as follows.
We present pairwise sample optimization (PSO) that can directly tune cenestep-distillee diffusion models toward a new output distribution for various customization applications.
We show that PSO a generalized formulation that covers both offline and online prefer- ence optimization scenarios, encompassing previous works, such as DiffusionDPO (Wal- lace et al.) 2024) and D3PO (Yang et al.) 2024a) as special cases, which can be flexibly adapted to various settings and tasks.
Extensive of experiments demonstrate the effectiveness of PSO in human preference tun- ing, as well as other domain transfer tasks such as style transfer and concept customization.
METHOD
In this section, we present method of pair-wise sample optimization (PSO) for directly fine- tuning timestep-distille diffusion models. We first show the PSO formulation with both reference and data trajectories, and then extend it to both online and offline settings.
2.1 PAIRWISE SAMPLE OPTIMIZATION FOR DISTILLEE DIFFUSION MODELS
Consider timestep-distilles diffusion model pa (2olc) that samples clean images To conditioned on the prompt from initial noise øT (0, 1) with very few timesteps N, N 1 4, we denote its sampling trajectory as {IeN Ftx 11 .... Zt,, }. }. where tN T, = O. Our target is to fine- tune the model towards the given target distribution Pdata (25 c). i... maximizing the log likelihood log Po (17]c)) where zí ~ Pdata Directly adopting the DDPM (Ho et al., 2020) formulation and minimizing the diffusion objective ||<o(=] t) c leads to blurry, degraded generation. To mit- igate this issue, we introduce the reference sample zf from the current model pe and recast the optimization as maximizing the relative likelihood between the target and reference samples. In- (a(mlcc)
tuitively, by maximizing log
no (=]]e)" we can steer the generative distribution pa towards the data distribution Pdota- Meanwhile, we avoid directly minimizing the diffusion objective, which can potentially help preserve its ability few-step generation.
3

---

Published as a conference paper at ICLR 2025
Formally, given pair of positive and negative images with the prompt c, zn Diota(25 c). 2 ~ po(= c), we take inspiration from direct preference optimization (Rafailov et al., 2024 and maximize the margin between log Po trained timestep-distileed model Pre..
(o c) and log Po(zf(c), regularized by the pre-
[Tog
Po (o(())
po po(2))c) Glgg
Pere (2 e) )]
BBog
Pere (2(()
where a denotes the sigmoid function, / is the regularization weight, Pure = p0 denotes the pre- trained model.
In diffusion models, obtaining Po(zolc) requires integrating over intermediate states which is costly. Instead, we consider the whole sampling trajectory (It,, je)- and maximize the margin between their trajectory joint likelihoods. Specifically, as shown in Figure 2 we sample the data trajectory (d e) with diffusion forward process, and sample the reference trajectory {= [c) with the generative reverse process.
Pdata (F Pdatt (1T (c) II g( [ot,
(2)
Po (a = Po (2l Ic) II po 1sf
Substitute the trajectory formulation into Eq. 1 and obtain our PSO objective below,
B Σ log po(27, 1007_,,07,,-00 1,,7,
Po(r, Ppro (2r 127,, Ppre (of
12%
Cpso -B Foe
log o
log
n
-BBio Gre .n)
log
8 E
DKL (xl (o ) Ppec (2rT [
log Po(=f lagf c) Ppre (3 100
- -Dx (Pxla(1T,, (o
pe (0 lot c))
(3)
The derivation is provided in Appendix
To expand the joint likelihood terms, we need to define the transition kernel Po (o 1 . c). In- spired by previous works (Black al. 2023 Yang al., 2024a), we formulate the denoising sam- pling process of dimeste--distilled models as Markov Decision Process (MDP).
MDP Formulation for Timestep-Distilled Diffusion Models. Let co In, c) denote the timestep-distilled diffusion model, which predicts Zy as fo f0 fn. c) = f(co (3t,, fn. c), a t,) given the corresponding noisy latent Fr, (Song et al. [2023] Yin al.) 2224bb a). the iterative few-step sampling process, the distilled model first predicts To at ln. and then add noise back to noise-level 1
Vtt, So (n, + V1- at,, 17,2 N),,T),
(4)
where ä's are the forward process coefficients. The Markov Decision Process for distilled models can be then formulated as,
8n (It.. tn); an = Itn-11 P(sntilsn, an) 812
(a sn) N N( (T Ins c), ....1) N (Mo bns c),
where Sn, a, denote the state and action, P(sn-118m, an) denotes the transition kernel, 8 is the Dirac function, and To is the policy. For the last timestep, the distilled model directly predict the clean image without noise added. This makes the final timestep deterministic transition, pe paxxxss 6(f(Ie, ti, c)). so we remove the last time step from the training. With this formulation, we have pa(os, (ort..) = (a (-

---

Published as a conference paper at ICLR 2025
Final PSO Loss. By plugging the MDP action-state conditional distribution, obtain the objec- tive for our pairwise sample optimization.
Cpso -E
log a
Σ (Ile" - - bu c)]12-11e- l|e" Epro Ens c)l12)
(5)
1
(Iaffnn Hol in c)1 1| Mpre (a En e)l?
20%
The derivation is provided in Appendix A
We provide Alg. [ for better illustration.
2.2 OFFLINE AND ONLINE EXTENSION
Our pairwise sample optimization can be extended to online and offline settings as follows,
Offine PSO. In practice, may have pre-sampled {af,) from models that have similar gener- ation distribution with po(-lc), which removes the need for reverse sampling {arf,, from po again.
In this case, extend our PSO formulation by approximating po | e) with the forward trajectory g (z()) Specifically, approximate the per-step transition kernel po |x c) with f p( (c)g |2t, F1,), and we obtain our offline PSO objective.
Cpso.onine E log - Σ (1Ee co(oT tuo c), -Ile" Spre tni c)ll
n=
(6)
||ee colo, tn, 2)1 + l|ce Cpro en e)1²²
which can be seen as a variant of the offline Diffusion-DPO (Wallace et al. 2024). We provide Alg. for better illustration.
Online PSO. There are also cases where we decide the data and the reference distribution in an online manner, e.g., with reward model. In this case, we can substitute the forward trajectory in Eq. 3 with the generative trajectory,
NN Σ n=2
3
(117 - Ho (o tn, c)11 ||27,-, - Appeett tn c)l
CpsO.Onlin Online = E log o
207
||ll - (o tu c)1 + 11 Mpre tny c)112)
which can be seen variant of the online DPO for diffusion models (Yang al. 2024a). We provide Alg. for better illustration.
EXPERIMENTS
In this section, we validate our method with multiple tasks and experiments. First, we demonstrate the effectiveness of our proposed PSO in the human-preference tuning task. We then present sults on other general fine-tuning tasks for timestep-distilled diffusion models with PSO, including style transfer and concept customization. Our experiments cover various main-stream timestep- distillation methods, including Distribution Matching (DMD) (Yin et al. 2024a b), Adversarial Dis-
tillation (ADD) (Sauer et al. 2023), and Latent Consistency Models (LCM) (Luo et al. 2023). We
5

---

Published as a conference paper at ICLR 2025
(a) SDXL-DMD2
(b) SDXL-DPO DMD2 LoRA
(c) (Ours) Offline PSO finetuned SDXL-DMD2
(d) (Ours) Online PSO finetuned SDXL-DMD2
Figure 3: Human preference tuning Results with PSO 4-step SDXL-DMD2. Compared with baseline SDXL-DMD2 (sub-figure (a)), SDXL-DPO with DMD2 LoRA (sub-figure (b)) exhibits slightly degraded generation quality. Rather, SDXL-DMD2 with both our offline and online PSO objectives figufiiuues (c) and (d) respectively) demonstrates substantial improvement in visual appeal, prompt following, and details generation. Prompts from left to right: The official portrait of an auhoritarian president of alternate America in 1960, in the style of pan mansion.J/ A profile picture of an anime boy, half robot, brown hair//On the Mid-Autumn Festival, the bright full moon hangs in the night sky. A quaint pavilion is illuminated by dim lights, resembling a beautiful scenery in painting. Camera type: close-up. Camera lenstype: telephoto. Time of day: night. Film type: ancient style. HD.
advertisements, looking up. jet age A curious exploring haunted
use the distilled versions of SDXL (Pdell et al. 2023) with these methods as our base models, i.e.. SDXL-DMD2 (4 step) (Yin et al. 2024a). SDXLoTurbo (Sauer al. 2023), and SDXL-LCM (Luo
et al. 2023).
3.1
PSO FOR
FINE-TUNING
Human Preference Tuning with Offline PSO. We first examine the effectiveness of PSO in numan-preferee fine-tuning in the offline setting. Specifically, consider pre-sampled reference image samples {x[lc) as discussed Sec. 2.2 which are negative samples the human preference dataset in this case, and the data samples { (c) are the preferred images in the dataset. We train all

---

Published as a conference paper at ICLR 2025
Table 1: Human preference tuning Results of SDXL-DMD2.
Method
Steps
Score
ImageReward
Score
SDXX
0.3713
0.8566
6,080
SDXL-DPO Wallace et al. 2021
50
22.60
0.3787
1,0075
6,010
SDXL-SFT
50
22.25
0.3693
0.9665
5.921
0.3723
6,012
SDXL-SPO
Libe uF 2024
50
22.70
0.3527
0,9417
6,283
SDXL
50
22.50
0.3735
0.9481
6.170
Pickapic Test
SDXL-DMM2
22.35
0.3679
0.9363
5.937
SDXI--PPO DMD2-LLRA
22.20
0,3673
0.9287
5.759
Offline-PSO
5.994
Online-P9 w/ SUXL-DMD22
22.73
0,3671
0,9773
6.077
SDXL
50
22.77
0.3607
0.9142
5.750
SDXL-DPO
50
22.92
0.3674
1.1180
5.795
0.3610
0.8565
5.675
SDXL-RPO
Gast ml 2024
50
22.98
0.3670
1.0770
5.872
SDXL-SPO
50
23.27
0.3428
1,0668
6.083
-MPU Fingg
0,3661
1,0315
5,912
Parti-Prompts
SDXI-DDMD2
22.99
0,3607
1,0713
5,671
SDXL-DPO
22.76
0.3644
1,0638
OfflincPSOO w/ SDXL-DMD2
4
23.07
0.3649
1,0964
5.715
Onlinc-PSO SDXL-DMD2
23,29
0,3634
1,1702
5.836
the base timestep-distilled diffusion models with the Pick-a-Picv2 (Kirstain et al. (023) dataset. Af- ter removing the pairs with tied preference labels, we end up with 851,295 pairs, with 58, 58,960 unique prompts. We adopt the PSO-Offline objective in Eq. O and fine-tune all our base timestep-distiiee distilled models. The training details and hyperparameters are provided in Appendix B.I
Human Preference Tuning with Online PSO. We also validate the proposed method with the on- human-prefeerence preference tuning task, where we decide sampled data belonged to Pdata or pe through-
line human- out the training based the human preference model PickScore (Kirstain al. 2023). ). We tune all the base distilled models with the PSO objective in Eq. 7 with a subset of training prompts from
Pick-a-Pic v2 Kirstain et al. 2023). All the other training details are provided in Appendix B.1
Evaluation and Benchmarks. We use the Pick-a-Pic test set and PartiPrompts (Yu et al. 2022) as the evaluation benchmarks, where we fix the random seed for all models and generate images for all prompts in the dataset. As for the evaluation metrics, we adopt PickScore (Kirstain et al. 2023), CLIP score (Radford et al., 202I), ImageReward (Xu et al., 2024), and Aesthetic score following Rafailov et al. (2024). We benchmark distilled models fine-tuned with PSO with various benchmarks and settings. For DMD (Yin et al.) 2024a) distillation method, We compare our fine-tuned SDXL- DMD2 with un-distilled multi-step(2. steps) SDXL models, SDXL with offline human preference tuning, SDXL-DPO (Rafailov et al. 2024). We also benchmark with umestep-distillee SDXL-DMD2. SDXL with DMD2 LoRA applied, and SDXL-DPO with DMD2 LoRA applied. We adopt the similar evaluation settings for SDXL-LCM, as detailed in Appendix B. For SDXL- Turbo, we fine-tune the model with steps (N 4). and we evaluate the results on both 1-step and 4-step settings to show the timestep generalization ability of the proposed method.
models,
Results. Table I shows the experiment results with DMD2 Yin et al. 2024a) as the base model. Directly applying the DMD2 LoRA to human prefereeeee fine-tuned SDXL-DPO (row with SDXL-
Table 2: Human preference tuning results of SDXL-Turbo.
Dataset
Method
Inferenee Steps
PickScore
CLIP Seare
InageReward
Acsthetic Score
SDXL-Tubbe TXXL-Turbo-4teey
4 22.22
0.3610
0.9300
5.987
Offine-PSO SDXL-Tubbo-sste
22.40
0.9695
Pickapic Test
Onling-PSO w/ SDXL--Tubb-bbtt Turbo-Astep
4
22.71
0.3647
0.9882
6.157
SDXL-Turbo Turbo--stepp
22.29
0.3642
0,8830
6.061
Offline-PSO SDXI--Turtoo-ssep
22.40
Online-PSO w/ SDXL- Turbo- 1step
1
22.62
0.3661
0.9113
6.137
SDXL-Turbo-4ssig
22.88
LD1733
OIftine-PSO w/ SDXI -oilo
22.96
0.3642
1.0599
5.746
Parti-Prompts
Online-PSO w/ Torb-Tubb-
4
23.23
0.3632
1,0893
5,837
BOXL--Turbo--ster
22.78
Offiine-PSe w/ SDXL Nurbooubo-ooo Turbo- Istep
22.86
0.3631
0.9664
5.755
BDXL-Turbo-Jae
22.96
0.9622

---

Published as a conference paper at ICLR 2025
Original Generation
Fine-tuning steps
Target Pokemon Style
Figure 4: Experiments on tuning SDXL-TTurbe for style transfer with PSO. The proposed method effectively tunes the distilled model to generate images following the targeted style. Prompts from top to bottom row: stern-faced, brown-feathered owl Pokémon with a leaf-shaped crown and piercing red eyes stands ready for battle./ robotic cat with wings.// A cute bunny rabbit.
DPO + DMD2 LoRA) achieves slightly inferior results compared with the SDXL-DMD2 baseline. It suffers from drop in PickScore (drops from 22.35 to 22.20 in PickaPic-test, and 22.99 to 22.77 in Parti-Prompts) and Aesthetic Score (drops from 5.937 to 5.759 in PickaPic-test, and 5.671 to 5.513 in Parti-Prompts). The performance drop indicates the need redo the diffusion distilla- tion from SDXL-DPO to obtain a distilled model better aligned with human preference, which is costly (~ 4000 A100 GPU hours (Yin al. 2024a)) for each fine-tuning. In contrast, our method achieves superior results compared with the SDXL-DMD2 baseline. For both Pick-a-Pic test set and Parti-Pre Parti-Prompts, our Offline-PSO fine-tuned model achieves higher results in all metrics compared with baselines, showing its dominance in visual quality and human preference. For online-PSO where both the data and reference sets are updated together with the model, it achieves the best re- sults in PickScore, ImageReward, and Aesthetic Score among all distilled models. Furthermore, its performance can even surpass the performance of multi-step fine-tuned SDXL-DPO, which further highlights the effectiveness of our method. For instance, and 23.29 in two datasets, while SDXL-DPO has scores of 22.60 and 22.92. Qualitatively, as shown in Figure 3 both PSO-Offline and PSO-Online fine-tuned models demonstrate better improvement in image quality in terms of prompt following, visual appeal, and the intricacy of details within each image. Moreover, Online PSO fine-tuned SDXL-DMD2 shows notable improvement over other baselines in terms of aesthetic appeal and high-frequency details, underscores the effectiveness of the proposed method.
method achieves PickScore of 22.73
Table 2 reports the results on tuning SDXL-Turbo (auerr et al. 2023). We observe similar trend as in Table where both Offline-PSO and Online-PSS achieves superior results of all metrics 4- step generation compared with the base distilled SDXL-Turbo. Moreover, our fine-tuned model also achieves superior results in 1-step generation, showcasing the generalization ability of the proposed method. We provide qualitative results of SDXI.-Turbo, along with the results on SDXL--ICCM in Appendix [B.]] where the proposed consistently surpass baselines terms of visual appeal and intricate detail generation.
3.2 PSO FINE-TUNING FOR STYLE TRANSFER
In this section, we demonstrate that our PSO can steer the timestep-distilled models towards spe- cific generation style. We select the Pokemon dataset (Pinkney) 2022) that contains Pokemon-style image-text pairs, and we utilize the proposed PSO to have the model generate images following the Pokemon style in the dataset. Specifically, we are interested distilled models that has no equiva- lent distillation LoRA, so select SDXL-Turbo (Sauer et al. 2023). We adopt the PSO objective in Eq. 33 and sample the reverse generative trajectories in each step. Other tarining details and hy- perparameters provided in Appendix B.2 We show the generation results of the Offline-PSO
8

---

Published as a conference paper at ICLR 2025
Inpen Images
A [V] dog chef ourn
A [V] dog in Tirefigter outth
[V] dog
A [V] dog wearing pink glasses
A of grass sonflowers.
teapet cobblestone street
with [ leapet Eiffel backgruind Tower in the bacliground
Input Images
Experiments of tuning SDXL-Turbo for concept customization with PSO. The proposed method can effectively tune SDXL-Turbo to generate images that contain the given objects.
Figure 5:
We observe that along the fine-tuning progress, the gener- ated images gradually shift from the original generative distribution to match the style of the given Pokemon style.
fine-tuned SDXL-Turbo in Figure 4
3.3 PSO FINE-TUNING FOR CONCEPT CUSTOMIZATION
In this section, we validate the PSO fine-tuning method with concept customization with SDXL- Turbo (Sauer et al. 2023). Following Dreambooth (Ruiz et al. 2023), we aim to tune the model to associate the special token with the given object concept represented in few (around 5) images, and make it generalize on unseen environments. To minimize the fine-tuning costs, we adopt the offline- setting. We also observe that in omitting the reference model and add prior preservation loss leads to performance gain in concept custonization. Training details and hyperparameters are given Appendix B We show the qualitative results in Figure 5 where we can see the PSO fine- tuned SDXL-Turbo generate images that contain the exact the same object shown in the training input images. This concept customization results further demonstrate the effectiveness of PSO and its generalization among different tasks. More results are provided in Appendix [.3] We also pro- vide quantitative results in Table 4 where our method shows superior results than vanilla tuning of SDXL-Turbo following RRiiiinggguuieeee Ruiz al. (203)) and comparable results with the multi-step customization
baseline (Dreambooth w/ SDXL).
RELATED WORKS
4.1 DIFFUSION MODEL DISTILLATION
Knowledge distillation, initially proposed for neural network compression (Hinton 2015 Cho &
Hariharan 2019 Patel et al. 2023). has become powerful technique for transferring knowledge from complex teacher model to simpler student model while maintaining comparable perfor- mance. To accelerate the inference speed of the diffusion models, various timestep distillation meth- ods have been proposed to transfer the generative distribution from a multi-step teacher model to few-step student model. This achieved by either distilling the PF-ODE from the pre-trained multi-
step diffusion model (Song et al.) 2023] [Salimans & Ho 2022 Kim et al. 2023), or matching the denoising distribution with an explicit analytical distribution matching loss (Yin et al. 2024b a) or
by learning an implicit discriminator (Sauer et al. 2023 Xu al. 2023 Sauer et al. 2024 Lin al, 2024). There are also recent works that combine those approaches to further boost the generation quality (Ren al. [2024] Chadebec al. 2024). However, the timestep distilled models generally lose the ability to predict the local score, indicating that they can not be directly fine-tuned with the original score-matching loss towards target distribution, as shown in Figure In this work,

---

Published as a conference paper at ICLR 2025
tackle this problem with the incorporation of the reference samples, and formulate the pairwise sample optimization objective to directly fine-tune timestep- diffusion models.
4.2 PREFERENCE OPTIMIZATION FOR DIFFUSION MODELS
To control the generation behavior of the diffusion models, researchers propose the markov de- cision process (MDP) formulation and utilize reinforcement learning (RL) to tune the multi-step diffusion model<Black et al. (2023); Fan et al. (023) to generate images that have higher aesthetic values, better prompt-following ability, alignment with human preference, and diversity (Kirstain) et al. 2023 Xu et al., [024] Liu et al. 2024a] Miao et al. (022)). Inspired by Rafailov et al. (2024), researchers have further derived direct preference optimization (DPO) from the RL formulation for diffusion models with both online (Yang et al. 2202aa b) Liang et al. 2024) and offline settings (Wal-) lace al. 2024). Yuan et al. (2024) takes step further by using SFT data as the preferred images while sampling negative imgges from the model progressively, However, all those methods de- signed for tuning base multi-step diffusion models, which are not applicable to distilled few-step models, and they only focus human preference tuning. In this work, formulate the MDP for timestep-distilled models for direct fine-tuning, and show the effectiveness of the proposed method in fine-tuning tasks including style transfer and concept customization.
4.3 DIFFUSION MODELS TUNING FOR STYLE TRANSFER AND CUSTOMIZATION
Given the base steti-step diffusion models, large corpus of works has focused on fine-tuning the models toward specific style (Huang et al. 2022: Wang et al. 2023 Sohn et al. 2023 Zhang al. 2023), and customizing the models with a given concept (Ruiz et al. [023 Xiong al. 2024 Song et al. 2024b Kumari et al. 2023). However, those methods are specifically designed for the corresponding task, and are general limited for multi-step base diffusion models. our work, we target at directly tuning timestep-distilleee diffusion models with unified objective that can be also applied to human prefereeeeeeence tuning.
CONCLUSION
In this work, we presented pairwise sample optimization (PSO) for directly fine-tuning timestep- distilled diffusion models. By constructing pairs of data and reference images and utilizing pref- erence optimization to increase the likelihood margin between them, PSO enables manipulation of the model's generative distribution while preserving its few-step generation capability. We re- formulated preference optimization for timestep-distilled models and demonstrated effectiveness in aligning model outputs with human preferences using both offline and online geeiinnee pairwise preference data. Furthermore, we extended PSO to style transfer and concept customization tasks by constructing pairwise source-target image pairs. Our approach provides efficient solution for adapting timestep-distilled diffusion models to downstream applications without the need for costly re-distilaation. Our work can be further strengthened with enhanced fine-tuning results in personal- ized generation, and including experiments in controllable image generation.
Limitations. While our method achieves initial promising results in style transfer and concept customization, further explorations on loss design and regularization are required to achieve more competitive results with enhanced image quality.
ETHICAL STATEMENT
Our research on fine-tuning üimeste--distilled diffusion models with the proposed PSO generally does not have ethical concerns, except the usage of human preference data. We acknowledge the potential for introducing or amplifying biases present in human judgments, which could lead to unfair discriminatory outputs. To address this, commit to implementing diverse sampling strategies for both annotators and image datasets, and developing faimess metrics to assess and mitigate potential biases.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 2023.
Clement Chadebec. Onur Tasar, Eyal Benaroche, and Benjamin Aubin. Flash diffusion: Ac- celerating any conditional diffusion model for few steps image generation. arXiv preprint arXiv:2406.02347, 2024.
Wei Chen, Zichen Miao, and Qiang Qiu. Inner product-based based neural network similarity. Advances in Neural Information Processing Systems, 36, 2024a.
Wei Chen, Zichen Miao, and Qiang Qiu. Parameter-efficient tuning of large convolutional models. arXiv preprint arXiv:2403.0000 002.9,02,,,22 2024b.
Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank learnable local filters. In International Conference Learning Representations, 2021.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4794-4802, 2019.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lec. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305.1633) 2023.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning dif- fusion models through relative preference optimization. arXiv preprint M.XXx:2406006388 2024.
Ligong Han, Yınxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. Proceedings of the IEEF/CVE Interna- tional Conference on Computer Vision, pp. 7323-7334, 2023.
Liu He and Daniel Aliaga. Globalmapper: Arbitrary-shaeed urban layout generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 454-464, 2023.
Liu He and Daniel Aliaga. Coho: Context-sensitive city-scale hierarchical urban layout generation. In European Conference Computer Vision, pp. 1-18. Springer, 2024.
Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang. Diffusion-barn based document layout generation. In International Conference on Document Analysis and Recognition, pp. 361-378.
Springer, 2023.
Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint 2015.
Jonathan Ho and Tim Salimans. arXiv:2207.12596 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 6851, 2020.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thoree, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. arXiv preprint wrXiv:2406.06424,, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0963 2021.
Nisha Huang, Yuxin Zhang, Fan Tang, Chongyang Ma, Haibin Huang, Yong Zhang, Weiming Dong, and Changsheng Xu. Diffstyler: Controllable dual diffusion for text-driven image stylization. arXiv preprint arXiv:22/ (..::2//1..000 106682, 2022.
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurlPS, 35:26565 15526655-26677,,, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hs Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. arXiv preprini 10.00277, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Va- jda, and Ali Thabet. Imagine flash: Accelerating emu diffusion models with backward distillation. arXiv preprint arXiv:2405.05222 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Tianchun Li, Chengxiang Wu, Pengyi Shi, and Xiaoqian Wang. Cumulative difference learning vae for sime-series with temporally correlated inflow-outflow, Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13619-13621 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint xrXiv22406.04314, 2024.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint (rXXi:2402.13922 2024.
Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xie. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint arXiv:2409.07253, 2024a.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint 297707202.09778 2022.
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.0933. 09355, 2024b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.00000 1.01095, 2022b.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. 2023.
Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Continual learning with filter atom swapping. In International Conference Learning Representations, 2021a.
Zichen Miao, Ze Wang, Xiuyuan Cheng, and Qiang Qiu. Spatiotemporal joint filter decomposition in 3d convolutional neural networks. Advances in Neural information Processing Systems, 34: 3376-3388 2021b.
Zichen Miao, Jiang Wang. Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10844-10853, 2024.
12

---

Published as a conference paper at ICLR 2025
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pp. 8162-8171, 8171. PMLR, 2021.
Gaurav Patel, Konda Reddy Mopuri, and Qiang Qiu. Learning to retain while acquiring: Combating distribution-shif in adversarial data-free knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7786-7794, 2023.
Gaurav Patel, Christopher Sandino, Behrooz Mahasseni, Ellen I. 7ippi, Erdrin Azemi, Ali Moin, and Juri Minxha. Efficient source-free free time-scries adaptation via parameter subspace disentanglement. arXiv preprint arXiv:2410.02147, 2024.
Justin NM Pinkney. Pokemon blip captions. Pokemon blip captions, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. Defnet: Deep neural network with decomposed convolutional filters. In International Conference on Machine Learning, pp. 4198 4207. PMLR, 2018.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 16:79320-79362. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual 2021.
models from natural language supervision. arXiv preprint
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404. 240864444..66, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pallern Recognition, pp. 10684-10695 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning rext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint XrXiv:2202.00512 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:211..1744,, 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffuusion distillation. arXiv preprint arXiv:2403. 12015, 2024.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crow- son, et al. Laion-Sb; An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
13

---

Published as a conference paper at ICLR 2025
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation any style. arXiv preprint arXhv:2306.0090 2023.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 2023.
Yizhi Song, Liu He, Zhifei Zhang, Soo Ye Kim, He Zhang, Wei Xiong, Zhe Lin, Brian Price, Scott Cohen, Jianming Zhang, et al. Refine-by-align: Reference-guided artifacts refinement through semantic alignment. arXiv preprint arXiv:2412.00306, 2024a.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0048-8058, 2024b.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. $228-8238, 2024.
Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. arXiv preprint arXiv:2406.1077 10777. 2024.
Ze Wang. Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. arXiv preprint 1909v 2019.
Ze Wang, Seunghyun Hwang, Zichen Miao, and Qiang Qiu. Image generation using continuous filter atoms. Advances in Neural Information Processing Systems, 34: 17826-17838 2021a.
Ze Wang. Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per-pixel dynamic filter atom. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 12302-12311, 2021b.
Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7677-7689, 2023.
Enze Xie, Lewei Yao. Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In Proceedings of the IEFFJCVF International Conference on Computer Vision, pp- 4239, 2023.
Zhexiao Xiong, Wei Xiong, Jing Shi, He Zhang, Yizhi Song. and Nathan Jacobs. Groundinghooth: Grounding text-to-image customization. arXiv preprint arXiv:2409.08520, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:2311.0925/ 2023.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1941-8955 2024a.
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. arXiv preprint arXiv:2402.08266 2024b.
14

---

Published as a conference paper at ICLR 2025
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint ar 47Xiv22405.14867, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 6613-6623, 2024b.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. arXiv preprint arXiv:2402.102100 2024.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion- based style transfer with diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10146-10156, 2023.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear parameter--fffiient adaptation of pre-trained models. arXiv preprint 4rXiv22410001800, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A DERIVATIONS
A.1 PSO DERIVATION
In this section, we present the derivation for Eq. 3 in Sec. 2.1 Given the forward sampled data trajectory = Ic and reverse sampled reference trajectory Tippt (c, we represent our PSO objective as increasing the likelihood gap between two trajectories,
Pe Po (c) Ppre (2T cc)
Do (aa le) Perc ( faatyyly (c) )
C = - Ee (olgut Porty ())
log
(og
3 log
(8)
With the formulation Eq.
we can write the joint likelihood in the above equation as,
L -
(ogary "Paccta (P d. p log
(9)
log (17 11,, Ppr 10,,,, log
Po le6, cc)
Σ
Prre ( ( 11
n=1
For the first log- likelihood ratio in the bracket, it can be further derived as,
L1 - Enggey Pautt (o g
log (3
log po (x), 1 Σ
> log Pere (17 1,0, :
n=1
-Bai updaca Je( kmeo)
N log (B3 log
Σ
> 1=
log (2 1 ei)
(10)
Given that each sampling step in the forward process be written as Ft, ((x - and know the marginal q(s+,, (mn), Bayes equation. In this way, the sub-objective above can be further derived as,
sample = q(x7,_, 1 wo) given ot o with the
po(=T, c)
Ppre 1,7, :c) log Σ
L - log (B
log q( 2()
q( 12% z/)
n=1
-yg log ((DKLlo lot (xT 1z,, c)] DKLl9(17 (oT Po(oT, |,7, cIl)]
-EEg log (oggo--Bun (-Bur, Bur, (Ile* c0(27 tnn c)ll ||e" Epre (a7, tn knne))
(11)
where the third equation adopts the reparameterization in the original diffusion objective deriva- tion (Black et al. 2023), and Wt,, the reweighting which is usually set to constant practice.
Meanwhile, the second log- ligellilliood ratio in Eq. 9 be further simplified with Eq. as,
Pe pe(aff (ef,,
log
20 (112 110 In e)l1 - || Mlpre (vf In c) (2) (12)
By substituting Eq. and Eq. 12 into Eq. gand removing the last sampling step f,, we got the PSO objective in Eq.
Cpso
E
log
-
(Ile - c0(2z,, 1,, c)12 l|e" Epre Enn =)
n=2
(13)
u0l tn. c)112 1- 11 More (a tni c)ll²)
207
16

---

Published as a conference paper at ICLR 2025
Algorithm Pairwise Sample Optimization (PSO)
Require: Pre-trained timestep-distilled model pre. prompt number of timesteps N Learning rate y, regularization weight 8
Require:
1: while not converged do
2: Sample z ~ Pdaza(1lo)
D Data sample Reference sample
3: Sample z po(-(c)
4: Get forward trajectory {2T, Jn=1 via q( 10
5: Get reverse trajectory {=f,, n==1 via po(s 1
6: Update using loss Lrso in Equation
7: end while
Ensure:
Fine-tuned timestep-distilled model
Algorithm Online Pairwise Sample Optimization (Online-PSO)
Require: Pre-trained timestep-distileed model Opre. prompt c, number timesteps N
Require: Learning rate 1). regularization weight 3, reward model R(-)
1: while not converged do
2: Sample two trajectories (1:,,, }WV 1m { | via (-(c)c)
3: Score trajectories: 81 R(x))) 82 R(x3)
4: Assign { = {ar ). {xe{ = {x7, if 82
5: Otherwise, { } = {x7, (2f {=), }
6: Update using loss Lpso-Online Equation
7:
end while
Ensure:
Fine-tuned timestep-distilled model
A.2 MDP FORMULATION FOR SDXL-TURBO
The MDP formulation presented in Sec. 2.1 is applicable to DMD Yin et al. 2024a) and LCM (Luo al. 2023). Here we present the MDP formulation for SDXL-Turbo (Sauer et al. (023). The Euler ancestral sampler can be defined as,
To Ton, + s0(To, .n) (ōn - on) + Ōn 2~ 1),
(14)
where on (o -- Gn
o - (2, (son Sol Rn c))/am
and on is the designated noise levels in Euler ancestral sampler used in SDXL-Turbo which has one-to-one mapping tn.
The MDP formulation can then be represented as, The Markov Decision Process for distilled models can be then formulated as,
Sn (Ions σn), On Ion-rs P(sn+1|8ns an) = -6(2on1 Cn- To Ta (8)) N (I solNo,, Fn) (o on),
B SUPPLEMENTARY EXPERIMENTAL SETTINGS AND RESULTS
B.1 HUMAN PREFERENCE TUNING
We adopt the Pick-a-Pic v2 (Kirstain et al. 2023) dataset for the offline human pref- erence tuning task following Wallace et al. (2024). It consists of pairwise preferences for images generated by SDXL-beta and Dreamlike, a fine-tuned version of SD1.5 (Rombach et al., (002)). The prompts and preferences were collected from users of the Pick-a-Pic web application. After removing ~ 12% pairs with tied preference, we obtain 850K win-lose pairs, which are used as data-reference pairs within our Offline-PSO objective as in Eq. 6 For the online human preference setting, use subset of 4K prompts from Pick-a- Pic training prompts the training prompts, and we sample pair of images given a prompt, assign the target & reference labels based on the PickScore (Kirstain al. 2023) as discriminator.
Datasets.
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Offline Pairwise Sample Optimization (Offline-PSO)
Require: Pre-trained timestep-distilled model Pre. prompt number of timesteps N
Require: Learning rate y, regularization weight 3, pre-sampled reference set
1: while not converged do
2: Sample zá ~ (-cm
b Data sample Reference sample
3: Sample = R
4: Get forward trajectories {27,, JNy Ji {=f S. J.v
via g(-++)
5: Update using loss Lpso-offine in Equation
6: end while
Ensure:
Fine-tumed timestep- distileed model
Parameter Efficient Fine-tuning. Diffusion models have been repurposed for various tasks with effective tuning methods (He et al.) 2023 He & Aliaga 2024) Song et al. [202aa] He & Aliaga 2023 Ruiz et al. 2023). As we aim to design a lightweight method for fine-tuning distilled dilfu- sion models, it's natural for us to adopt the parameter-efficiet fine-tuning methods which majorly are based on matrix low-rank decomposition. LoRA (Hu al., (021) and following works (Liu) al. [0244b] Dettmers et al. 2024) decompose the weight matrix along the channel dimension into two low-rank matrices, which induces much lower costs in fine-tuning compared with tuning the original full-rank weight. Meanwhile, (Chen et al. 2024b) proposes the filter subspace decompo- sition for weight matrices. The filter subspace decomposition method (Qiu et al. 2018)) has shown effectiveness in continual learning (Miao et al.
2021 la. Chen al. 2024a), video representation learning (Miao et al., 2021b), graph learning (Cheng et al. (202), ), and generative tasks (Wang et al.)
2021a 2019 2021b Li al. 2024). There some other works fine-tuning the SVD decom-
position of weights Han et al. 2023). Kronecker decomposition (Patel et al.) 2024), sparsity (Wang
al. 2024), non-linearity (Zhong al. (024) fine-tunig bias parameters (Xie et al. 2023). Con-
sidering the implementation and the integration with other codebases, we choose LoRA (Hu et al.,
(021] to fine-tune timestep-distillee distilled diffusion models.
Training Details. We use LoRA (Hu et al. 2021) fine-tune all the distilled diffusion mod- els efficiently, we set LoRA rank r = 16 for SDXL-DMD2 and SDXL-Turbo, and r 32 for SDXL-LCM in both online and offline human preference tuning experiments. As for other training hyperparameters, we set the number of training distilled steps N 4, which is the same as the number of sampling steps of these distilled models, and we set the regularization weight / 50 for Offline preference tuning, and / = for Online preference tuning. We set the batch size to 64 and the learning rate to 5 for offline experiments, and train for 5k steps. For online preference tuning, we first sample 128 pairs of images with 128 training prompts, which are further labeled as reference and target with PickScore (Kirstain et al. (202)) with a batch size of 64, and then we conduct online PSO on the sampled pairs for epoch with a batch size of 32 and learning rate le - and train for 20k steps.
Table 3: Results of SDXL-LCM
Dataset
Method
PickScore
CLIP Score
ImageReward
Aesthetic Score
SDXL-LCM SDXL LCM-LORA
21,9072
0.3572
0.6717
5.8642
21,8520
0,3553
0.6840
5,9447
Pickapic Test
SDXL-DPO + LCM-LORA
22.3178
0.3633
0.7883
5.9286
Offline-PSO w/ SDXL-LCM
22,3953
0.3668
0.8129
5.9949
OnLine-PSO w/ SDXL-LCM
22.6353
0.3613
0,8343
6.0117
SDXL--LCM SDXL LCM-LORA
22.4761
0,3492
0.6738
5,5867
22,4332
0,3481
0,6571
5.5141
SDXL-DPO LCM-LoRA
22.7854
0.3551
0.7778
5.7090
Parti-Prompts
Offline-PSI w/ SDXL-LCM
22,8131
0.3592
0.8131
5.8367
Online-PSO w/ SDXL-LCM
22.9451
0.3581
0.8351
5.8572
18

---

Published as a conference paper at ICLR 2025
SDXL-DMD2
RCO CARRITT
D ONT
SDXL-DPO DMD2 LoRA
EARIT FARRT
DO: ART
Offline PSO tuned SDXL-DMD2
FARFT
Online SDXL-DMD2
DONT FART
Figure 6: Human Preference Tuning with SDXL-DMD2 addidional results, Prompts from left to right: Sign that says DON'T FART// Warrior with an axe on a exotic animal with long hair style loisel// Movie Still of The Joker wielding red Lightsaber, Darth Joker sinister evil clown prince of crime, HD Photograph// A swirling, multicolored portal emerges from the depths of an ocean of coffee, with waves of the rich liquid gently rippling outward. The portal engulfs a coffee cup, which serves as a gateway to fantastical dimension. The surrounding digital art landscape reflects the colors the portal, creating an alluring scene of endless possibilities.
Evaluation. For all baseline distilled models and PSO fine-tuned models, conduct testing on Pickapic (Kirstain et al. 2023) test and PartiPrompts (Yu et al. 2022). They contain 500 and 1632 prompts respectively. We sample for each prompt testing image with 4 sampling steps (we also conduct 1-step sampling test for SDXL-Turbo, which is trained under 4-step schedule). Each sampled image is then evaluated by PickScore (Kirstain et al., 2023), CLIP Score (Radford al.] 2021 RewaeReward (Xu et al., 2024), and Aesthetic Score (Schuhmann et al. 2022) We report the average scores in the tables.
Additional Results. We provide qualitative results for SDXL-DMD2 human preference ing in Figure 6 7 As shown in those figures, images generated by our Online-PSO and Offline- PSO tuned models demonstrate better aesthetic values, prompt following, and detailed accuracy. For SDXL-Turbo, we provide qualitative illustrations in Figure 8a for 1-step evaluation and Figure 3b Again, we see that our PSO-tuned models generate more visual-appeal images in both 1-step and 4-step generations. Specifically, we observe better visual detail generation, for instance, human body, objects, and object consistency images generated by our method. And finally, we provide quantitative results for SDXL-LCM (Luo et al. 2023) in Table 3 As shown in the table, our method achieves higher results in all metrics compared with baselines. Specifi- cally, method achieves better results compared with SDXL-DPO LCM-LoRA, which is the preference-tuned multi-step SDXL equipped with LCM LoRA, which further justify and validate method.
for
19

---

Published as a conference paper at ICLR 2025
SDXL-DMD2
HELP
HELLO
SDXL-DPO DMD2 LoRA
Heellgo
HELLo
Offline PSO tuned SDXL-DMD2
HELLOO)
Online PSO luned SDXL-DMD2
Hotloo
Human Preference Tuning with SDXL-DMD2 addidional results. Prompts from left to right: a red hair girl look at you, distant view.// dog with sign that reads "Hello" // anime pencil concept style.// A galaxy-colored figurine floating over the sea at sunset, photorcalistic.
Figure 7:
Table 4: Quantitative evaluation of PSO tinettunin on concept customization task.
Model
Inference Steps
CLIP-T
DINO
CLIP-I
Dreambooth w/ SDXL
50
0.256
0.610
0.741
Dreambooth w/ SDXL-Turbo
0.267
0.151
0.326
(Ours) PSO w/ SDXL-Turbo
4
0.271
0.593
0.733
B.2 PSO FOR STYLE TRANSFER
We adopt SDXL-Turbo Sauer et al. 2023 for this experiment with the Pokemon dataset (Pinkney) 2022 The reference images are sampled using the training prompts in the pokemon dataset. We add LoRA of rank r 16 to the attention layers and set B 50. Then, adopt the objective of PSO for fine-tuning. We use the batch size 8, learning rate of 1e-5, and train for 3k steps.
B.3 PSO FOR CONCEPT CUSTOMIZATION
We also use SDXL-Turbo for this task with the Dreambooth dataset (Ruiz al.) 2023). We adopt the Offline-PSO objective in Eq. 6 in the way that we use the given input images the targeted set, and sample reference images from SDXL-Turbo with the prompt A photo [V] concept'. mentioned previously, we further train the model with variant of the Eq. 6 where we remove the reference model Epre and add prior preservation loss with images sampled from the initial model following Ruiz et al. 2203);; Qiu al. (2023). We set lora rank r = 16, B 5, learning rate of 5e-5 and train for 1000 steps with batch size of 4.
We provide additional results in Figure with the given animal or object.
Our PSO can effectively customize the distilled model
20

---

Published as a conference paper at ICLR 2025
SuXbo-Turbo
Offline finetuned SDXL-TTubo (Ours)
Online PSO finetuned SDXL-Turbo (Ours)
(a) Preference Fine-tuning Results with PSO on Istep SDXL-Turbo.
SDXL-Turbo
Offline PSO finetuned SDXL-Turbo (Ours)
Online PSO finetuned SDXI-Turbo (Ours)
(b) Preference Fine-tuning Results with PSO 4step SDXL-Turbo
Figure 8: Experiments of Tuning SDXL-Turbo for human preference tuning.
21

---

Published as a conference paper at ICLR 2025
Input Images
[V] cat in police outfit
A [V] wearing pink Glasses
A [V] pink fubric
[V] cat wearing
Inpur Images
A dog in chef outfit
A [V] dog wearing red
[V] dog
A [V] dog with city the background
Input
A [V] cut chef outfit
[V] cat wearing red hat
A [V] cat the snow
A [V] cat wearing yellow shirt
Input Images
A on top of'a wooden floor
A red [V] can
A can on the beach
a [V] can on a cobblestone street
Figure 9: Experiments of Tuning Surbo-Turbo for personalized generation with PSO. The proposed method can effectively tnne SDXL-TTuro to generate images that contain the given objects.
22


Reviews:
[{"review_id": "DJ8MuLHtRF", "reviewer": "Reviewer_B3DX", "summary": "This paper proposes a new and important issue about diffusion models, namely direct finetuning a timestep-distilled diffusion model without apparent performance degradation. Since finetuning a timestep-distilled diffusion model with a naive diffusion loss will result in degraded generated images, this paper proposes a so-called pairwise sample optimization algorithm that increases a relative likelihood margin between the training images and reference images. The authors conduct several experiments on different acceleration methods, including DMD2, Turbo, and LCM on different finetune tasks, i.e., humen preference tuning, style transfer, and customized image generation. Experimental results can demonstrate the effectiveness of the PSO", "questions": "Please refer to Weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. This paper addresses a very important and practical issue in the diffusion-model field.\n2. The proposed pairwise sample optimization (PSO) algorithm is novel and has thereotical derivation to increase its rigorism。\n3. This paper considers both offline and online setting, whichi can unifies various prior works.\n4. Experimental studies are conducted on different acceleration methods and applied on several downstream finetune tasks, which make this PSO more persuasive.", "weaknesses": "1. The analysis and description of the proposed PSO are slightly incomplete. For instance:\n- This paper doesn't explain why \"directly adopting the DDPM  formulation and minimizing the diffusion objective $||\\epsilon\\theta(x_{t_n}^\\tau) − ε||^2$ leads to blurry.\"\n- Why you choose to \"recast the optimization as maximizing the relative likelihood between the target and reference samples?'' I think this is the most crucial insight of this paper.\n- How to chooose the \"pre-sampled reference image samples''?\n2. The experimental results in Table 1 are a little confusing. Why do you choose \"SDXL-DPO + DMD2-LoRA'' as a compared baseline? Use DMD2-LoRA based on a SDXL-DPO model? Why not compare with \"SDXL-DMD2-LoRA  + DPO'' by using the distilled SDXL-DMD2 as base model?\n3. Some typos: Figure 1(b) -> Figure 1(c) in Line 107; the extend it to -> then extend it to in Line 148; no explanation for $p_{pre}$ in Eq (1) (maybe $p_{\\theta_{pre}}$)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yaEAIDbpJv", "reviewer": "Reviewer_fXQz", "summary": "This paper focuses on fine-tuning distilled diffusion models. The authors introduce a pairwise optimization approach, which combines preference optimization over positive-negative pairs and alignment with a frozen distilled few-step diffusion model. They present both offline and online training versions of this loss function. Experiments on tasks of human-preference tuning, style transfer, and concept customization show it outperforms the original DPO.", "questions": "Why is DPO used in all the tables instead of your method, PSO?\n\nIn Table 1, why are your results bolded even when they perform worse than SDXL-DPO?\n\nHow should we interpret that the online update version shows improvements in almost all metrics but generally performs worse than the offline version on the CLIP score?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Fine-tuning time-distilled diffusion models is very useful.\n\nThe proposed method generalizes well across different tasks and base models.", "weaknesses": "Although the experiments cover many tasks and base models, there aren’t many comparisons with other methods. The authors only compare their method to the original DPO in Table 1 and Table 3, and there are no comparisons in Table 2. Including comparisons with more fine-tuning methods would make the results more comprehensive and highlight the advantages of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0MYttKKT1k", "reviewer": "Reviewer_i4cT", "summary": "The motivation behind this work stems from the observation that models distilled over multiple timesteps can suffer from blurriness and reduced quality if distilled further using a naive diffusion objective. A straightforward solution would be to use a teacher model and repeat the distillation process, but this approach is overly cumbersome.\nTo address this, the authors propose an optimization method that fine-tunes the distilled model without requiring a teacher model. PSO introduces additional self-generated images and optimizes the relative likelihood margin, preserving the model's ability to generate high-quality images in fewer steps while still allowing fine-tuning.", "questions": "This paper is good to me. I hope author can provide an algorithm for complete method for better readability.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This problem is both novel and relevant, as many time-distilled diffusion models incorporate multiple objectives, such as adversarial training, making efficient fine-tuning essential.\n\n- The proposed method’s use of pair optimization is innovative, providing a critical solution to eliminate the need for the cumbersome teacher model distillation process.\n\n- Extensive experiments underscore the effectiveness of this approach.", "weaknesses": "- The notation used in the paper lacks clarity and can be confusing for readers, particularly with symbols like ρ and p.\n\n- Additionally, Figure 2 is somewhat abstract. Including an algorithm for the complete method would improve readability and help readers understand the approach more efficiently.\n\n- it would be beneficial to discuss how this method compares to current DiT-like image generation models, such as Flux.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tC0Z3W8YjQ", "reviewer": "Reviewer_bSbN", "summary": "This paper proposes PSO, an objective for finetuning timestep-distilled diffusion models like SDXL-DMD2, SDXL-Turbo, and SDXL-LCM. These models, trained with GAN loss or consistency distillation loss, can no longer be trained with traditional diffusion loss. The proposed PSO is a modification of DPO, which is widely used in human preference optimization for LLMs and diffusion models. Unlike DPO, PSO uses negative samples generated by timestep-distilled models and supports both offline and online generation. PSO shows effectiveness in human preference tuning, style transfer, and concept customization for these models.", "questions": "- Why was offline PSO used for style transfer and online PSO for concept customization? Could the reverse approach also work?\n\n- In Tables 1 and 2, is there a specific reason why \"Offline-DPO\" and \"Online-DPO\" are listed instead of \"Offline-PSO\" and \"Online-PSO\"?\n\n- What reward model was used for online PSO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper addresses the important problem of finetuning timestep-distilled diffusion models.\n\n- PSO, while similar to DPO, introduces key differences, notably the ability to support both offline and online training.\n\n- Finetuning timestep-distilled models with PSO proves more effective than attaching timestep-distilled LoRA to finetuned diffusion models, and the experiments demonstrate the potential for style transfer and customization.", "weaknesses": "- The paper primarily demonstrates the effectiveness of PSO in human preference tuning. However, direct finetuning of timestep-distilled models holds more significance in concept customization rather than human preference tuning. As the authors mention in the introduction, customization methods like Dreambooth require significantly less time compared to timestep distillation methods like DMD. On the other hand, human preference tuning, such as DPO, still requires lengthy training and may not offer much advantage over timestep distillation. To further highlight the necessity of PSO in human preference tuning, it would be beneficial to compare the computational cost of timestep distillation versus human preference tuning.\n\n- For the reasons mentioned above, PSO should further showcase its capabilities in concept customization and style transfer on datasets from Dreambooth (Ruiz et al.) and StyleDrop (Sohn et al). Specifically, following these works, text-image alignment could be compared using CLIP, and image-reference similarity could be assessed using DINO. Additionally, results in Figures 5 and 9 appear blurry, so a quantitative comparison using CLIP and DINO is essential.\n\n- PSO uses generated negative samples, which is similar to SPIN [A]. A discussion on the differences between PSO and SPIN would be beneficial. A discussion on the differences between PSO and SPIN could be added to the related works section.\n\n[A] Chen et al., Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)