Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization

Authors: Zichen Miao, Zhengyuan Yang, Kevin Lin, Ze Wang, Zicheng Liu, Lijuan Wang, Qiang Qiu
Primary area: generative models
Keywords: Distilled Diffusion Models; Fine-tuning

Abstract:
Recent advancements in timestep-distilled diffusion models have enabled high-quality image generation that rivals non-distilled multi-step models, but with significantly fewer inference steps. While such models are attractive for applications due to the low inference cost and latency, fine-tuning them with a naive diffusion objective would result in degraded and blurry outputs. An intuitive alternative is to repeat the diffusion distillation process with a fine-tuned teacher model, which produces good results but is cumbersome and computationally intensive: the distillation training usually requires magnitude higher of training compute compared to fine-tuning for specific image styles. In this paper, we present an algorithm named pairwise sample optimization (PSO), which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model. PSO introduces additional reference images sampled from the current time-step distilled model, and increases the relative likelihood margin between the training images and reference images. This enables the model to retain its few-step generation ability, while allowing for fine-tuning of its output distribution. We also demonstrate that PSO is a generalized formulation which be flexible extended to both offline-sampled and online-sampled pairwise data, covering various popular objectives for diffusion model preference optimization. We evaluate PSO in both preference optimization and other fine-tuning tasks, including style transfer and concept customization. We show that PSO can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data. PSO also demonstrates effectiveness in style transfer and concept customization by directly tuning timestep-distilled diffusion models.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729654232911

thread_type:
official_review

<thread_object>
creator: reviewer_bsbn

<content>
summary:
This paper proposes PSO, an objective for finetuning timestep-distilled diffusion models like SDXL-DMD2, SDXL-Turbo, and SDXL-LCM. These models, trained with GAN loss or consistency distillation loss, can no longer be trained with traditional diffusion loss. The proposed PSO is a modification of DPO, which is widely used in human preference optimization for LLMs and diffusion models. Unlike DPO, PSO uses negative samples generated by timestep-distilled models and supports both offline and online generation. PSO shows effectiveness in human preference tuning, style transfer, and concept customization for these models.

strengths:
- This paper addresses the important problem of finetuning timestep-distilled diffusion models.

- PSO, while similar to DPO, introduces key differences, notably the ability to support both offline and online training.

- Finetuning timestep-distilled models with PSO proves more effective than attaching timestep-distilled LoRA to finetuned diffusion models, and the experiments demonstrate the potential for style transfer and customization.

weaknesses:
- The paper primarily demonstrates the effectiveness of PSO in human preference tuning. However, direct finetuning of timestep-distilled models holds more significance in concept customization rather than human preference tuning. As the authors mention in the introduction, customization methods like Dreambooth require significantly less time compared to timestep distillation methods like DMD. On the other hand, human preference tuning, such as DPO, still requires lengthy training and may not offer much advantage over timestep distillation. To further highlight the necessity of PSO in human preference tuning, it would be beneficial to compare the computational cost of timestep distillation versus human preference tuning.

- For the reasons mentioned above, PSO should further showcase its capabilities in concept customization and style transfer on datasets from Dreambooth (Ruiz et al.) and StyleDrop (Sohn et al). Specifically, following these works, text-image alignment could be compared using CLIP, and image-reference similarity could be assessed using DINO. Additionally, results in Figures 5 and 9 appear blurry, so a quantitative comparison using CLIP and DINO is essential.

- PSO uses generated negative samples, which is similar to SPIN [A]. A discussion on the differences between PSO and SPIN would be beneficial. A discussion on the differences between PSO and SPIN could be added to the related works section.

[A] Chen et al., Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024.

questions:
- Why was offline PSO used for style transfer and online PSO for concept customization? Could the reverse approach also work?

- In Tables 1 and 2, is there a specific reason why "Offline-DPO" and "Online-DPO" are listed instead of "Offline-PSO" and "Online-PSO"?

- What reward model was used for online PSO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374594849

thread_type:
official_review

<thread_object>
creator: reviewer_i4ct

<content>
summary:
The motivation behind this work stems from the observation that models distilled over multiple timesteps can suffer from blurriness and reduced quality if distilled further using a naive diffusion objective. A straightforward solution would be to use a teacher model and repeat the distillation process, but this approach is overly cumbersome.
To address this, the authors propose an optimization method that fine-tunes the distilled model without requiring a teacher model. PSO introduces additional self-generated images and optimizes the relative likelihood margin, preserving the model's ability to generate high-quality images in fewer steps while still allowing fine-tuning.

strengths:
- This problem is both novel and relevant, as many time-distilled diffusion models incorporate multiple objectives, such as adversarial training, making efficient fine-tuning essential.

- The proposed method’s use of pair optimization is innovative, providing a critical solution to eliminate the need for the cumbersome teacher model distillation process.

- Extensive experiments underscore the effectiveness of this approach.

weaknesses:
- The notation used in the paper lacks clarity and can be confusing for readers, particularly with symbols like ρ and p.

- Additionally, Figure 2 is somewhat abstract. Including an algorithm for the complete method would improve readability and help readers understand the approach more efficiently.

- it would be beneficial to discuss how this method compares to current DiT-like image generation models, such as Flux.

questions:
This paper is good to me. I hope author can provide an algorithm for complete method for better readability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730462592097

thread_type:
official_review

<thread_object>
creator: reviewer_fxqz

<content>
summary:
This paper focuses on fine-tuning distilled diffusion models. The authors introduce a pairwise optimization approach, which combines preference optimization over positive-negative pairs and alignment with a frozen distilled few-step diffusion model. They present both offline and online training versions of this loss function. Experiments on tasks of human-preference tuning, style transfer, and concept customization show it outperforms the original DPO.

strengths:
Fine-tuning time-distilled diffusion models is very useful.

The proposed method generalizes well across different tasks and base models.

weaknesses:
Although the experiments cover many tasks and base models, there aren’t many comparisons with other methods. The authors only compare their method to the original DPO in Table 1 and Table 3, and there are no comparisons in Table 2. Including comparisons with more fine-tuning methods would make the results more comprehensive and highlight the advantages of the proposed approach.

questions:
Why is DPO used in all the tables instead of your method, PSO?

In Table 1, why are your results bolded even when they perform worse than SDXL-DPO?

How should we interpret that the online update version shows improvements in almost all metrics but generally performs worse than the offline version on the CLIP score?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730984512014

thread_type:
official_review

<thread_object>
creator: reviewer_b3dx

<content>
summary:
This paper proposes a new and important issue about diffusion models, namely direct finetuning a timestep-distilled diffusion model without apparent performance degradation. Since finetuning a timestep-distilled diffusion model with a naive diffusion loss will result in degraded generated images, this paper proposes a so-called pairwise sample optimization algorithm that increases a relative likelihood margin between the training images and reference images. The authors conduct several experiments on different acceleration methods, including DMD2, Turbo, and LCM on different finetune tasks, i.e., humen preference tuning, style transfer, and customized image generation. Experimental results can demonstrate the effectiveness of the PSO

strengths:
1. This paper addresses a very important and practical issue in the diffusion-model field.
2. The proposed pairwise sample optimization (PSO) algorithm is novel and has thereotical derivation to increase its rigorism。
3. This paper considers both offline and online setting, whichi can unifies various prior works.
4. Experimental studies are conducted on different acceleration methods and applied on several downstream finetune tasks, which make this PSO more persuasive.

weaknesses:
1. The analysis and description of the proposed PSO are slightly incomplete. For instance:
- This paper doesn't explain why "directly adopting the DDPM  formulation and minimizing the diffusion objective $||\epsilon\theta(x_{t_n}^\tau) − ε||^2$ leads to blurry."
- Why you choose to "recast the optimization as maximizing the relative likelihood between the target and reference samples?'' I think this is the most crucial insight of this paper.
- How to chooose the "pre-sampled reference image samples''?
2. The experimental results in Table 1 are a little confusing. Why do you choose "SDXL-DPO + DMD2-LoRA'' as a compared baseline? Use DMD2-LoRA based on a SDXL-DPO model? Why not compare with "SDXL-DMD2-LoRA  + DPO'' by using the distilled SDXL-DMD2 as base model?
3. Some typos: Figure 1(b) -> Figure 1(c) in Line 107; the extend it to -> then extend it to in Line 148; no explanation for $p_{pre}$ in Eq (1) (maybe $p_{\theta_{pre}}$)

questions:
Please refer to Weakness
</content>
</thread_object>

</thread>


Metareview:
metareview:
All reviewers agree to accept the paper. Reviewers appreciate the novel task, the innovative method, and extensive experiments. Please be sure to address the reviewers' comments in the final version.