Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting

Authors: jincheng  zhong
xingzhuo  guo
jiaxiang  dong
mingsheng  long
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 11:14:36

Abstract: Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models.
  In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side.
  We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet.
  Diff-Tuning achieves a 24.6% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning. Code
  is available at this repository: https://github.com/thuml/Diffusion-Tuning.

Main paper: 
 introduction: 
 Diffusion models [45, 17, 47] are leading the revolution in modern generative modeling, achieving remarkable successes across various domains such as image [11, 39, 12] , video [43, 19, 55] , 3D shape [34] , audio generation [25] , etc. Despite these advances, training an applicable diffusion model from scratch often demands a substantial computational budget, exemplified by the thousands of TPUs needed, as reported by [55] . Consequently, fine-tuning well pre-trained, large-scale models for specific tasks has become increasingly crucial in practice [54, 60, 56] .During the past years, the deep learning community has concentrated on how to transfer knowledge from large-scale pre-trained models with minimal computational and memory demands, a process known as parameter-efficient fine-tuning (PEFT) [20, 59, 54, 7, 21, 32] . The central insight of these approaches is to update as few parameters as possible while avoiding performance decline. However, the intrinsic transfer properties of diffusion models have remained largely unexplored, with scant attention paid to effectively fine-tuning from a pre-trained diffusion model.Previous studies on neural network transferability, such as those by [33, 57] , have demonstrated that lower-level features are generally more transferable than higher-level features. In the context of diffusion models, which transform noise into data through a reverse process, it is logical to assume that the initial stages, which are responsible for shaping high-level objects, differ in transferability from later stages that refine details. This differential transferability across the denoising stages presents an opportunity to enhance the efficacy of fine-tuning.In this work, we investigate the transferability within the reverse process of diffusion models. Firstly, we propose that a pre-trained model can act as a universal denoiser for lightly corrupted data, capable of recognizing and refining subtle distortions (see Figure 1 ). This ability leads to improved generation quality when we directly replace the fine-tuned model with the original pre-trained one under low distortion. The suboptimality observed with fine-tuned models suggests potential overfitting, mode collapse, or undesirable forgetting. Then we extend the experiments by gradually increasing the denoising steps replaced, to cover higher-level noised data, observing the boundaries of zero-shot generalization capability. This indicates that the fine-tuning objective should prioritize high-level shaping, associated with domain-specific characteristics. We term this gradual loss of adaptability the chain of forgetting, which tends to retain low-level denoising skills while forgetting high-level, domain-specific characteristics during the transfer of the pre-trained model. We further provide novel theoretical insights to reveal the principles behind the chain of forgetting.Since the chain of forgetting suggests different denoising stages lead to different forgetting preferences, it is reasonable to develop a transfer strategy that balances the degrees of forgetting and retention. Technically, based on the above motivation, we propose Diff-Tuning, a frustratingly simple but general fine-tuning approach for diffusion models. Diff-Tuning extends the conventional finetuning objectives by integrating two specific aims: 1) knowledge retention, which retains general denoising knowledge; 2) knowledge reconsolidation, which tailors high-level shaping characteristics to specific downstream domains. Diff-Tuning leverages the chain of forgetting to balance these two complementary objectives throughout the reverse process.Experimentally, Diff-Tuning achieves significant performance improvements over standard finetuning in two mainstream fine-tuning scenarios: conditional generation and controllable generation with ControlNet [60] . Our contributions can be summarized as follows:• Motivated by the transferable features of deep neural networks, we explore the transferability of diffusion models through the reverse process and observe a chain of forgetting tendency. We provide a novel theoretical perspective to elucidate the underlying principles of this phenomenon for diffusion models. • We introduce Diff-Tuning, a frustratingly simple yet effective transfer learning method that integrates two key objectives: knowledge retention and knowledge reconsolidation. Diff-Tuning harmonizes these two complementary goals by leveraging the chain of forgetting. • As a general transfer approach, Diff-Tuning achieves significant improvements over its standard fine-tuning counterparts in conditional generation across eight datasets and controllable generation using ControlNet under five distinct conditions. Notably, Diff-Tuning enhances the transferability of the current PEFT approaches, demonstrating the generality.2 Related Work 
diffusion models: 
 Diffusion models [17] and their variants [47, 48, 23] represent the state-of-the-art in generative modeling [12, 3] , capable of progressively generating samples from random noise through a chain of denoising processes. Researchers have developed large-scale foundation diffusion models across a broad range of domains, including image synthesis [17] , video generation [19] , and cross-modal generation [43, 42] . Typically, training diffusion models involves learning a parametrized function f to distinguish the noise signal from a disturbed sample, as formalized below:EQUATIONwhere x 0 ∼ X represents real samples, ϵ ∼ N (0, I) denotes the noise signal, and x t = √ α t x 0 + √ 1 -α t ϵ is the disturbed sample at timestep t. Sampling from diffusion models follows a Markov chain by iteratively denoising from x T ∼ N (0, I) to x 0 .Previous research on diffusion models primarily focuses on noise schedules [36, 23] , training objectives [44, 23] , efficient sampling [46] , and model architectures [39] . In contrast to these existing works, our method investigates the transferability of diffusion models across different denoising stages and enhances the transfer efficacy in a novel and intrinsic way.Transfer learning [38] is an important machine learning paradigm that aims to improve the performance of target tasks by leveraging knowledge from source domains. Transferring from pre-trained models, commonly known as fine-tuning, has been widely proved effective in practice, especially for the advanced large-scale models [5, 1, 12] . However, directly fine-tuning a pre-trained model can cause overfitting, mode collapse, and catastrophic forgetting [24] . Extensive prior work has focused on overcoming these challenges to ultimately enhance the utilization of knowledge from pre-trained models [2, 8, 61] . However, effective transfer of diffusion models has received scant attention.Parameter-Efficient Fine-tuning (PEFT) With significant advancements in the development of large-scale models [10, 5, 1, 12] , research in transfer learning has increasingly concentrated on PEFT methods that minimize the number of learnable parameters. The primary goal of PEFT is to reduce time and memory costs associated with adapting large-scale pre-trained models. Techniques such as incorporating extra adapters [20, 60, 35] and learning partial or re-parameterized parameters [59, 21, 22, 14] are employed for their effectiveness in reducing computational demands. Nevertheless, the reliance on deep model architectures and the necessity of carefully selecting optimal placements present substantial challenges. Intuitively, PEFT approaches could potentially mitigate catastrophic forgetting by preserving most parameters unchanged; for a detailed discussion, refer to Section 4.3.Mitigating Catastrophic Forgetting Catastrophic forgetting is a long-standing challenge in the context of continual learning, lifelong learning, and transfer learning, referring to the tendency of neural networks to forget previously acquired knowledge when fine-tuning on new tasks. Recent exploration in parameter regularization approaches [24, 28, 27, 8] have gained prominence. Approaches such as [58, 29, 52] propose the data-based regularization, which involves distilling pre-trained knowledge into a knowledge bank. However, efforts to mitigate forgetting within the framework of diffusion models remain notably scarce. 
chain of forgetting: 
 Compared with one-way models, diffusion models specify in a manner of multi-step denoising and step-independent training objectives. Inspired by prior studies on the transferability of deep neural features [57, 33] , we first explore how the transferability of diffusion models varies along the denoising steps.Pre-trained Model Serves as a Zero-Shot Denoiser Modern large-scale models are pre-trained with a large training corpus, emerging powerful zero-shot generalization capabilities. We begin by analyzing whether the pre-trained diffusion models hold similar zero-shot denoising capabilities. In particular, we utilize a popular pre-trained Diffusion Transformer (DiT) model [39] as our testbed. We fine-tune the DiT model on a downstream dataset. When the reverse process comes to the the last 10% steps, we switch and continue the remaining denoising steps with the fine-tuned model, the original pre-trained model, and our Diff-Tuning model respectively. We visualize a case study in Figure 1 (left) with corresponding replacement setups. Surprisingly, the results reveal that replacement by the pre-trained model achieves competitive quality, even slightly better than the fine-tuned one, indicating that the pre-trained diffusion model indeed holds the zero-shot denoising skills. On the other side, some undesirable overfitting and forgetting occur when fine-tuning diffusion models.Forgetting Trend Next, we delve deeper into investigating the boundary of generalization capabilities for the pre-trained model. Figure 1 (right) illustrates the performance trend when we gradually increase the percentage of denoising steps replaced from 0 to 100%. Initially, this naive replacement yields better generation when applied towards the end of the reverse process. However, as more steps are replaced, performance begins to decline due to domain mismatch. This trend suggests the fine-tuned model may overfit the downstream task and forget some of the fundamental denoising knowledge initially possessed by the pre-trained model when t is small. Conversely, as t increases, the objects desirable in the new domain are distorted by the pre-trained model, resulting in a performance drop. Based on these observations, we conceptually separate the reverse process into two stages: (1) domain-specific shaping, and (2) general noise refining. We claim that the general noise refining stage is more transferable and can be reused across various domains. In contrast, the domain-specific shaping stage requires the fine-tuned model to forget the characteristics of the original domain and relearn from the new domains.Theoretic Insights Beyond empirical observations, we provide a novel theoretical perspective of the transfer preference for the pre-trained diffusion model. Following the objectives of diffusion models, a denoiser F (an x 0 -reparameterization [23] of f in Eq. ( 1)) is to approximate the posterior expectation of real data over distribution D. This is formalized by:EQUATIONwhere p D (x 0 ) represents the distribution of real data from D, and N denotes the Gaussian distributions determined by the forward process. Notably, a larger variance of Gaussian distribution indicates a more uniform distribution. Through a detailed investigation of these Gaussian distributions under varying timesteps t, we derive the following theorem. All proofs and derivations are provided in Appendix A. ), where D new denotes the dataset of the new domain. This theorem highlights the necessity for further adaptation in the new domain.Based on the above observations and theoretical insights, we introduce Diff-Tuning, which incorporates two complementary strategies to leverage the chain of forgetting in the reverse process: 1) knowledge retention, and 2) knowledge reconsolidation. Diff-Tuning aims to retain general denoising skills from the pre-trained model while discarding its redundant, domain-specific shaping knowledge. This enables the model to adapt more effectively to the specific characteristics of downstream tasks. Diff-Tuning harmonizes the retention and reconsolidation via the chain of forgetting tendency. Without loss of generality, we present Diff-Tuning under the standard DDPM objective, omitting conditions in the formulations. The general conditional generation setup will be discussed later.Knowledge Retention As discussed earlier, retaining pre-trained knowledge during the latter general noising refining proves beneficial. However, the classic parameter-regularization-based approaches [24, 8, 27] mitigate forgetting uniformly across the reverse process, primarily due to the parameter-sharing design inherent in diffusion models. To address this, Diff-Tuning constructs an augmented dataset X s = { x s , • • • }, pre-sampled from the pre-trained model. This dataset acts as a repository of the retained knowledge of the pre-trained model. We define the auxiliary training objective, L retention , as follows:EQUATIONwhere ξ(t) is the retention coefficient. In accordance with the principles of the chain of forgetting, ξ(t) decreases monotonically with increasing t, promoting the retention of knowledge associated with small t values and the discarding of knowledge related to large t values. Knowledge Retention shares a similar formulation with the pre-training objective but without the reliance on the original pre-training dataset.It is important to note that the concept of Knowledge Retention in Diff-Tuning is anchored in the principle of the chain of forgetting. This approach encourages the model to recall how to denoise samples with low levels of disturbance, as reflected by the retention coefficient ξ(t). While the proposed augmented dataset serves as an easy-to-implement example of this concept, given the flexibility of Diff-Tuning, various other methods can also effectively facilitate knowledge retention. An alternative approach involving knowledge distillation is detailed in Appendix C.In contrast to knowledge retention, knowledge reconsolidation focuses on adapting pre-trained knowledge to new domains. The intuition behind knowledge reconsolidation is to diminish the conflict between forgetting and adaptation by emphasizing the tuning of knowledge associated with large t. This adaptation is formalized as follows:EQUATIONwhere ψ(t) is the reconsolidation coefficient, a monotonic increasing function within the range [0, 1], reflecting increased emphasis on domain-specific adaptation as t increases. Similar to Knowledge Retention, the principle behind Knowledge Reconsolidation involves adapting the model to effectively handle samples that are heavily disturbed, which are more significantly influenced by the distance between the pre-trained and target distributions. Reweighting by ψ(t) serves as one of the simplest implementations of this concept.A Frustratingly Simple Approach Overall, we reach Diff-Tuning, a general and flexible finetuning framework for effective transferring pre-trained diffusion models to downstream generations, the overall objective is as follows:EQUATIONwhere L retention (θ) and L adaptation (θ) are described before, θ represents the set of tunable parameters. Notably, Diff-Tuning is architecture-agnostic and seamlessly integrates with existing PEFT methods. Further details are discussed in Section 4.3.Choices of ξ(t) and ψ(t) For clarity and simplicity, we define ξ(t) = 1 -ψ(t), ensuring equal weighting for each t, following the original DDPM configuration. This complementary design excludes the influences of recent studies on the t-reweighting techniques [23, 12, 9] . From the above discussion, we can choose any monotonically increasing function whose range falls in [0, 1]. In this work, we scale the variable t to the interval [0, 1], and apply a simple power function group ψ(t) = t τ for practical implementation. In our experiments, we report the main results with τ = 1, and the variations of the choice are explored in Section 4.4.Conditional Generation Classifier-free guidance (CFG) [18] forms the basis for large-scale conditional diffusion models. To facilitate sampling with CFG, advanced diffusion models such as DiT [39] and Stable Diffusion [12] are primarily trained conditionally. CFG is formulated as ϵ = (1 + w)ϵ c -wϵ u , where w, ϵ c , ϵ u are the CFG weight, conditional output, and unconditional output. As a general approach, Diff-Tuning inherits the conditional training and sampling setup to support a wide range of transfer tasks. Due to the mismatch between the pre-training domain and downstream tasks in the conditional space, we apply knowledge retention L retention on the unconditional branch and knowledge reconsolidation L adaptation on both unconditional and conditional branches. 
experiments: 
 To fully verify the effectiveness of Diff-Tuning, we extensively conduct experiments across two mainstream fine-tuning scenarios: 1) Class-conditional generation, which involves eight well-established fine-grained downstream datasets, and 2) Controllable generation using the recently popular Control-Net [60] , which includes five distinct control conditions.Setups Class-conditioned generation is a fundamental application of diffusion models. To fully evaluate transfer efficiency, we adhere to the benchmarks with a resolution of 256 × 256 as used in DiffFit [54] , including datasets such as Food101 [4] , SUN397 [53] , DF20-Mini [40] , Caltech101 [13] , CUB-200-2011 [50] , ArtBench-10 [30], Oxford Flowers [37] , and Stanford Cars [26] . Our base model, the DiT-XL-2-256x256 [39] , is pre-trained on ImageNet at 256 × 256 resolution, achieving a Fréchet Inception Distance (FID) [16] of 2.27 2 . The FID is calculated by measuring the distance between the generated images and a test set, serving as a widely used metric for evaluating generative image models' quality. We adhere to the default generation protocol as specified in [54] , generating 10K instances with 50 sampling steps (FID-10K). β cfg weight is set to 1.5 for evaluation. For the implemented DiffFit baseline, we follow the optimal settings in [54] , which involve enlarging the learning rate ×10 and carefully placing the scale factor to 1 to 14 blocks. For each result, we fine-tune 24K iterations with a batch size of 32 for standard fine-tuning and Diff-Tuning, and a batch size of 64 for DiffFit, on one NVIDIA A100 40G GPU. For each benchmark, we recorded the Relative Promotio of FID between Diff-Tuning and Full Fine-tuning ( Diff-Tuning-Full Fine-tuning Full Fine-tuning) to highlight the effectiveness of our method. More implementation details can be found in Appendix B. Setups Controlling diffusion models enables personalization, customization, or task-specific image generation. In this section, we evaluate Diff-Tuning on the popular ControlNet [60] , a state-ofthe-art controlling technique for diffusion models, which can be viewed as fine-tuning the stable diffusion model with conditional adapters at a high level. We test Diff-Tuning under various imagebased conditions provided by ControlNet3 , including Sketch [51] , Edge [6] , Normal Map [49] , Depth Map [41] , and Segmentation on the COCO [31] and ADE20k [62] datasets at a resolution of 512 × 512. We fine-tune ControlNet for 15k iterations for each condition except 5k for Sketch and 20k for Segmentation on ADE20k, using a batch size of 4 on one NVIDIA A100 40G GPU. For more specific training and inference parameters, refer to Appendix B.Due to the absence of a robust quantitative metric for evaluating fine-tuning approaches with ControlNet, we propose a novel metric based on the sudden convergence steps. In the sudden convergence phenomenon, as reported in [60] , ControlNet tends not to learn control conditions gradually but instead abruptly gains the capability to synthesize images according to these conditions after reaching a sudden convergence point. This phenomenon is observable in the showcases presented in Figure 4 throughout the tuning process. We propose measuring the (dis-)similarity between the original controlling conditions and the post-annotated conditions of the corresponding controlled generated samples. As depicted in Figure 3 , a distinct "leap" occurs along the training process, providing a clear threshold to determine whether sudden convergence has occurred. We manually select this threshold, combined with human assessment, to identify the occurrence of sudden convergence. The detailed setup of this metric is discussed in Appendix F. ControlNet, indicating a consistent enhancement in the transfer efficiency. In Figure 4 , we display showcases from the training process both with and without Diff-Tuning. It is observed that Diff-Tuning achieves sudden convergence significantly faster, enabling the generation of well-controlled samples more quickly. By comparing the images from the final converged model at the same step, it is evident that our proposed Diff-Tuning achieves superior image generation quality.The initial motivation behind adapter-based approaches in continual learning is to prevent catastrophic forgetting by maintaining the original model unchanged [20] . These methods conceptually preserve a separate checkpoint for each arriving task, reverting to the appropriate weights as needed during inference. This strategy ensures that knowledge from previously learned tasks is not overwritten. In transfer learning, however, the objective shifts to adapting a pre-trained model for new, downstream tasks. This adaptation often presents unique challenges. Prior studies indicate that PEFT methods struggle to match the performance of full model fine-tuning unless modifications are carefully implemented. Such modifications include significantly increasing learning rates, sometimes by more than tenfold, and strategically placing tunable parameters within suitable blocks [21, 7, 54] . Consider the state-of-the-art method, DiffFit, which updates only the bias terms in networks, merely 0.12% of the parameters in DiT equating to approximately 0.83 million parameters. While this might seem efficient, such a small proportion of tunable parameters is enough to risk overfitting downstream tasks. Increasing the learning rate to compensate for the limited number of trainable parameters can inadvertently distort the underlying pre-trained knowledge, raising the risk of training instability and potentially causing a sudden and complete degradation of the pre-trained knowledge, as observed in studies like [54] .Elastic Weight Consolidation (EWC) [24] is a classic parameter-regularized approach to preserve knowledge in a neural network. We calculate the L 2 -EWC values, which are defined as EWC = ∥θ -θ 0 ∥ 2 , for the tunable parameters in the evaluated approaches [28] . The EWC value quantifies how far the fine-tuned model deviates from the pre-trained model, indicating the degree of knowledge forgetting from the perspective of parameter space.Figure 5 (b) reveals that DiffFit leads to EWC values that are 2.42 times larger with only 0.12% tunable parameters, indicating heavy distortion of the pre-trained knowledge. Figure 5 (c) illustrates the averaged EWC over tunable parameters, showing that each tunable bias term contributes significantly more to the EWC. In contrast, Diff-Tuning achieves lower EWC values. Diff-Tuning does not explicitly focus on avoiding forgetting in the parameter space but rather harmonizes the chain of forgetting in the parameter-sharing diffusion model and only retains knowledge associated with small t rather than the entire reverse process.Diff-Tuning can be directly applied to current PEFT approaches, and the comparison results in Figure 5 demonstrate that Diff-Tuning can enhance the transfer capability of DiffFit and significantly improve converged performance.Fine-tuning Convergence Analysis To analyze converging speed, we present a concrete study on the convergence of the FID scores for standard fine-tuning, DiffFit, Diff-Tuning, and Diff-Tuning * (DiffFit equipped with Diff-Tuning) every 1,500 iterations in the SUN 397 dataset, as shown in Figure 6 (a). Compared to standard fine-tuning and DiffFit, Diff-Tuning effectively leverages the chain of forgetting, achieving a balance between forgetting and retaining. This leads to faster convergence and superior results. Furthermore, the result of Diff-Tuning * indicates that PEFT methods such like DiffFit still struggle with forgetting and overfitting. These methods can benefit from Diff-Tuning.Ablation Study We explore the efficacy of each module within Diff-Tuning, specifically focusing on knowledge retention and knowledge reconsolidation. We assess Diff-Tuning against its variants where: (1) Only reconsolidation is applied, setting ξ(t) ≡ 0 and ψ(t) = t; and (2) Only retention is employed, setting ξ(t) = 1 -t and ψ(t) ≡ 1. The results, illustrated in Figure 6 (b) demonstrate that both knowledge retention and knowledge reconsolidation effectively leverage the chain of forgetting to enhance fine-tuning performance. The FID scores reported on the DF20M dataset clearly show that combining these strategies leads to more efficient learning adaptations. Tradeoff the Forgetting and Retraining with the Chain of Forgetting For simplicity and ease of implementation, Diff-Tuning adopts a power function, ψ(t) = t, as the default reconsolidation coefficient. To explore sensitivity to hyperparameters, we conduct experiments using various coefficient functions ψ(t) = t τ with τ values from the set {0, 0.3, 0.5, 0.7, 1, 1.5}, and a signal-to-noise ratio (SNR) based function ψ(t) = 1/(1 + SNR(t)) [9] . Results on the Stanford Car dataset, shown in Figure 6 (c), a carefully tuned coefficient can yield slightly better results. To keep the simplicity, we keep the default setting τ = 1. Notably, when τ = 0, Diff-Tuning reduces to the standard fine-tuning baseline.Analysis on Knowledge Retention In Diff-Tuning, knowledge is retained using pre-sampled data from pre-trained diffusion models before fine-tuning. We evaluate the impact of varying sample sizes (5K, 50K, 100K, 200K, and the entire source dataset) on the performance of the DiT model on the Stanford Car dataset, as illustrated in Figure 6 (d). Notably, using the entire source dataset, which comprises 1.2M ImageNet images, results in suboptimal outcomes. This observation underscores that pre-sampled data serve as a more precise distillation of pre-trained knowledge, aligning with our goal of retraining knowledge rather than merely introducing extra training data.Beyond the size of the augmented dataset, the quality of the samples can also influence knowledge retention. Therefore, we further analyze the impact of different sampling steps during the presampling stage, as depicted in Figure 7 (a). The findings demonstrate that Diff-Tuning exhibits consistent performance, indicating that the knowledge retention process is robust to variations in the sampling process of the augmented dataset.We evaluate Diff-Tuning using various sampling parameters, specifically settings of 25, 50, 100, and 500 steps. As depicted in Figure 7 (b), there is a consistent improvement in performance across these configurations. Notably, Diff-Tuning shows a more significant enhancement with fewer steps, suggesting that it builds a more precise denoising model compared to standard fine-tuning.In dynamic target domains like online systems that continually collect new data, continual learning is essential as models must undergo iterative fine-tuning to adapt to evolving datasets. To evaluate the adaptability of Diff-Tuning in this context, we extend out method and conduct experiments on the Evolving Image Search dataset [63] , which includes images from 10 categories collected over three phases: 2009-2012, 2013-2016, and 2017-2020. We apply transfer learning sequentially across these temporal splits, and measure FID on the cumulative test set. For Diff-Tuning, we retain 40% of the initially sampled augmented data and generated the remaining 60% using the fine-tuned model before proceeding to the next phase. As shown in Figure 7 (c), the results demonstrate that Diff-Tuning consistently improves and sustains robust performance as the dataset evolves. 
conclusion: 
 In this paper, we explore the transferability of diffusion models and provide both empirical observations and novel theoretical insights regarding the transfer preferences in their reverse processes, which we term the chain of forgetting. We present Diff-Tuning, a frustratingly simple but general transfer learning approach designed for pre-trained diffusion models, leveraging the identified trend of the chain of forgetting. Diff-Tuning effectively enhances transfer performance by integrating knowledge retention and knowledge reconsolidation techniques. Experimentally, Diff-Tuning shows great generality and performance in advanced diffusion models, including conditional generation and controllable synthesis. Additionally, Diff-Tuning is compatible with existing parameter-efficient fine-tuning methods.A Proofs of Theoretical Explanation in Section 3.1In this section we provide the formal definitions and proofs for theoretical explanation of chain of forgetting. From the setup of diffusion model training and previous works [23] , we suppose the dataset consists of finite bounded samples D = {xEQUATION0 , ..., x(n) 0 }, and f is the denoiser to minimize L(θ) as in Eq. ( 1) under ϵ-parameterization. For convenience, we first convert the denoiser into x 0 -parameterization byF (x t ) = xt- √ 1-αtf (xt) √ αtand the objective becomesEQUATION)Ideal Denoiser in Eq. ( 6). An ideal denoiser F should minimize the value F (x t ) for all t, x t almost surely, implying an objective for F (x t ):EQUATION)By taking a derivative, it holds thatEQUATIONand finally,EQUATIONEQUATIONRemark. This ideal denoiser is exactly the same one as [23] under DDPM-style definition.Case when t → 0. When t → 0, α t → 1. For simplicity suppose the closest sample to x t is unique. LetEQUATIONEQUATIONthenEQUATION)EQUATIONEQUATION)EQUATIONEQUATIONand thus F (x t ) → x closest 0 , which completes the proof. Notably, when there are multiple closest samples, through similar analysis it is clear that F (x t ) converges to their average. Case when t → T . When t → T , α t → 0, and thus N x t ; √ α t x 0 , (1 -α t )I → N x t ; 0, I , a constant for varying x 0 . Bringing this back to Eq. ( 14) and it holds thatEQUATIONwhich completes the proof. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No new data or pretrained models are proposed.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All creators of datasets are properly credited by citations.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We provide the details of our experiment configuration in this section. All experiments are implemented by Pytorch and conducted on NVIDIA A100 40G GPUs.This section describes the benchmarks utilized in our experiments.Food101 [4] The dataset consists of 101 food categories with a total of 101,000 images. For each class, 750 training images preserving some amount of noise and 250 manually reviewed test images are provided. All images were rescaled to have a maximum side length of 512 pixels.SUN397 [53] The SUN397 benchmark contains 108,753 images of 397 well-sampled categories from the origin Scene UNderstanding (SUN) database. The number of images varies across categories, but there are at least 100 images per category. We evaluate the methods on a random partition of the whole dataset with 76,128 training images, 10,875 validation images and 21,750 test images.DF20M [40] DF20 is a new fine-grained dataset and benchmark featuring highly accurate class labels based on the taxonomy of observations submitted to the Danish Fungal Atlas. The dataset has a well-defined class hierarchy and a rich observational metadata. It is characterized by a highly imbalanced long-tailed class distribution and a negligible error rate. Importantly, DF20 has no intersection with ImageNet, ensuring unbiased comparison of models fine-tuned from ImageNet checkpoints.Caltech101 [50] The Caltech 101 dataset comprises photos of objects within 101 distinct categories, with roughly 40 to 800 images allocated to each category. The majority of the categories have around 50 images. Each image is approximately 300×200 pixels in size. Artbench10 [30] ArtBench-10 is a class-balanced, standardized dataset comprising 60,000 highquality images of artwork annotated with clean and precise labels. It offers several advantages over previous artwork datasets including balanced class distribution, high-quality images, and standardized data collection and pre-processing procedures. It contains 5,000 training images and 1,000 testing images per style.Oxford Flowers [37] The Oxford 102 Flowers Dataset contains high quality images of 102 commonly occurring flower categories in the United Kingdom. The number of images per category range between 40 and 258. This extensive dataset provides an excellent resource for various computer vision applications, especially those focused on flower recognition and classification. COCO [31] The MS COCO dataset is a large-scale object detection, segmentation, key-point detection, and captioning dataset. We adopt the version of 2017 where 164k images are split into 118k/5k/41k for training/validation/test. For each image, we randomly select one of its corresponding captions, and use detectors of Canny edge, (binarized) HED sketch, MIDAS depth/normal and Uniformer segmentation implemented in ControlNet to obtain the annotation control, and final construct the dataset of image-text-control pairs for training and evaluation. All controling condition has channel 1 except normal map and segmentation mask with channel 3.Ade20k [62] The Ade20k dataset is a semantic segmentation dataset containing nearly 21k/2k training/validation images annotated with pixel-wise segmentation mask of 149 categories of stuff and objects. For each image, we use the "default prompt" "a high-quality, detailed, and professional image" as adopted in ControlNet, and use Uniformer implemented in ControlNet to obtain the segmentation mask as the control to obtain the dataset of image-text-control pairs.Detailed Algorithm Process For all experiments, we first generate images of memory bank with the pre-trained model. We then construct conditioned dataset from memory bank with default condition (unconditional for class-conditional generation task and "default prompt" with generated control for controllable generation task). Considering instability of weighted training loss with ξ(t) and ψ(t), for each iteration, we instead first uniformly sample timestep t according to the property ξ(t) + ψ(t) = 1, then determine whether sample x 0 from the downstream dataset or the augmented dataset, and finally calculate the diffusion loss L(θ) with the simple form. The pseudo-code of the overall algorithm process is shown in Algorithm 1.Input: Downstream dataset X , pre-trained model parameter θ0 Output: Fine-tuned parameter θ.Collect pre-sampled data X s for knowledge reconsolidation using θ0.Initialize θ ← θ0.while not converged do Initialize mini-batch X = {}. for i in {1, 2, ..., B} do Sample t (i) from uniform distribution U ({1, 2, ..., T }). if with probability ξ(t(i) ) then Sample x (i) 0 ∼ X s . {Retention loss Lretention} else Sample x (i) 0 ∼ X . {Adaptation loss Ladaption} end if Add (x (i) 0 , t (i) ) to mini-batch X. end for Calculate L(θ) ← 1 B B i=1 Eϵ ϵ -f θ √ α t (i) x (i) 0 + 1 -α t (i) ϵ, t (i) 2Update θ according to ∇ θ L(θ). end while return θHyperparameters We list all hyperparameters in our experiments in Table 3 . There exist many possible methods that can achieve a similar effect to Knowledge Retention, provided they adhere to the principle of the chain of forgetting. In this section, we introduce a variant that utilizes knowledge distillation (KD), which avoids the need to pre-sample an augmented dataset and presents a more elegant solution. The KD variant can be formalized as follows:EQUATION)where f θ0 represents the original pre-trained model. The results, depicted in Table 4 , indicate that both techniques should be effective within the framework of Diff-Tuning. The practical decision can be informed by the specific scenario. Below, we summarize reasons why KD was not our initial choice:• GPU Memory: KD requires maintaining a copy of the pre-trained model alongside the fine-tuning model. For large models, this significantly increases memory costs. For example, we can run Diff-Tuning with a batch size of 32 on a single A100 40GB GPU, whereas the KD variant decreases to a batch size of 24.• Computational Cost: KD doubles the forward computation cost by necessitating the matching of output distributions between two models. Notably, pre-computing the KD labels is not feasible due to the inherent noise in diffusion training. For instance, for a batch size of 24, we achieve 2.1 training steps per second, compared to 1.34 for the KD variant.• Training Instability: Transferring a pre-trained model to a domain significantly different from its training data can introduce out-of-distribution corruption during distillation, potentially causing instability in the fine-tuning process. We invested considerable effort to tune a suitable trade-off for the KD loss (0.05), and sometimes the training is easily disrupted due to an unsuitable KD loss setting.• Implementation Difficulty: An elegant KD implementation requires users to be familiar with the code framework and introduces a large design space. In contrast, an augmented replay buffer introduces only a small set of extra source data and changes the training data sampled related to t, which is considerably easier to implement across various fine-tuning scenarios. As mentioned above, we ensure ψ(t) + ξ(t) = 1 in our Diff-Tuning, which maintains the overall loss in Eq. ( 5) with the default DDPM weight (uniform weight across timesteps). In our experimental implementation, we uniformly sample timesteps and then sample training data from either the memory buffer or target dataset according to ψ(t) and ξ(t), respectively. Unlike [9, 15] , which tailor weights or develop separate models for each timestep. We do not alter the weights of individual timesteps in Diff-Tuning, which is significantly different from existing works.To further underscore that the weighting strategies from [9] and [15] are orthogonal to our Diff-Tuning approach, we have implemented these strategies alongside our method. The results demonstrate that while these existing methods can be applied to Diff-Tuning, their integration shows varied impacts on performance.These results in Table 5 indicate that while existing methods can be adapted to our Diff-Tuning, the performance varies, especially with MIN-SNR strategies, which may not align with the principles of the chain of forgetting, thereby potentially undermining the transfer learning efficacy.E Fine-tuning EDM with Diff-TuningIn the main expeirments, we have fine-tuned the DiT-XL-2 model (pre-trained using a VP-SDE approach [48] ) and Stable Diffusion. We employed the same diffusion training strategy used for the pre-trained models. Additionally, we have evaluated EDM on publicly available repositories 4 .Since EDM incoporates continuous σ instead of discrete t in the training state, we extend to use ψ(σ) = cdf(σ) and ξ(σ) = 1 -ψ(σ), aligning our standard Diff-Tuning. Results are shown below: Sudden convergence is a phenomenon observed when tuning ControlNet [60] due to its specific zero-convolution design. As demonstrated in Figure 4 , ControlNet does not gradually learn to adhere to control conditions. Instead, it abruptly begins to follow the input conditions. To identify a simple signal indicating sudden convergence, we pre-collected a validation set of real images and compared the (dis-)similarity between their annotations and the corresponding generated controlled images. Figure 9 illustrates a noticeable "leap" during the training process, providing a clear indicator of sudden convergence. We established thresholds for quantitative evaluation based on test annotation similarity curves and combined them with qualitative human assessment to determine the number of steps as a metric for all tasks.To efficiently and generally compare the (dis-)similarity between the original controlling conditions and the post-annotated conditions of the corresponding controlled generated samples, we employ the simplest reasonable metrics for each vision annotation, irrespective of task specificities such as label imbalance or semantic similarity. Specifically, we use Structural Similarity Index (SSIM) with Gaussian blurring for sparse classification (Edge and Sketch), mean-square error (MSE) for dense regression (Depth and Normal), and accuracy for dense classification (Segmentation). Detailed settings of the metrics and thresholds are provided in Table 7 . We present more case studies to validate the steps for convergence. By generating samples throughout the training process using a consistent random seed, we focus on identifying when and how these samples converge to their corresponding control conditions. As shown in Figure 10 , our selected thresholds provide reasonable convergence discrimination across all six tasks, with Diff-Tuning consistently outperforming standard fine-tuning.Similarities Alone Are Imperfect It is important to note that our proposed similarity metric serves only to indicate the occurrence of convergence and does not accurately reflect sample quality or the degree of control, especially for converged samples. For example, Figure 11 compares generated samples from standard fine-tuning and our Diff-Tuning approach with edge controlling. The generated samples from Diff-Tuning are richer in detail but may score lower on edge similarity metrics, highlighting the limitations of similarity metrics and underscoring the necessity of human assessment.Diff-Tuning consistently outperforms standard fine-tuning and parameter-efficient methods, demonstrating its efficacy across various downstream datasets and tasks. This section also discusses some limitations of Diff-Tuning and explores potential future directions to address these limitations.Necessity of a Pre-sampled Dataset Diff-Tuning involves constructing a pre-sampled augmented dataset for knowledge reconsolidation, which requires additional computational resources. As discussed in Section 4.4, the impact of the sample size indicates that even the smallest set of generated images outperforms baseline methods and the original source data, underscoring the value of this approach. Future work could focus on developing large-scale open-source generated samples and creating more sample-efficient augmented datasets.Extra Hyperparameters Diff-Tuning introduces additional hyperparameters, ξ(t) and ψ(t), as loss weighting coefficients based on the principle of the chain of forgetting. These introduce extra hyperparameter spaces. As detailed in Section 4.4, our analysis shows that simple designs for these coefficients perform well and robustly. Future research could aim to design more effective weighting coefficients to further enhance performance. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The paper focuses mainly on the machine learning intrinsicities within views of transferrability, and no societal impacts are newly involved. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://dl.fbaipublicfiles.com/DiT/models/DiT-XL-2-256x256.pthttps://github.com/lllyasviel/ControlNethttps://github.com/NVlabs/edm 


Decision: This paper introduces Diffusion Tuning to transfer learning for diffusion models. By leveraging a new concept termed the "chain of forgetting," the paper proposes an approach to encourage pre-trained diffusion models to retain essential knowledge while discarding noise-related information during fine-tuning. Experiments demonstrate a 26% improvement over conventional fine-tuning across multiple tasks and accelerates ControlNet's convergence by 24%.

The reviewers identify several strengths, including the novelty, clarity and presentation, and experimental results. There are also concerns by some reviewers, including theoretical clarifications and comparisons with existing literature on timestep weighting strategies, comparing with knowledge distillation, and lack of experiment verification to validate the robustness of the method. All the concerns have been discussed and addressed by the authors thorough rebuttals, and the reviewers are positive about the rebuttal.

Given the conceptual novelty, the theoretical foundations, and comprehensive experimental validation, I recommend the paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper focuses on transfer learning methods for diffusion models. It experimentally demonstrates and provides theoretical insights into how the forgetting trend varies with the diffusion timestep. Based on this observation, they proposes Diff-Tuning. The proposed method introduces objectives for knowledge retention and reconsolidation, ensuring that they are affected decreasingly and increasingly by the diffusion timestep, respectively. The superiority of the proposed method is demonstrated experimentally.
strength_weakness: strength_weakness: * More discussion on how to set the hyperparameters $\xi(t)$ and $\psi(t)$ would be helpful. For example, these parameters perform an weighted sum of the retention and reconsolidation objectives, and combining this with a maximum likelihood analysis could potentially help find hyperparameter selection.

* I am curious if applying the datasets used for pre-training to retention would be better (or worse) than using a pre-sampled dataset and the reason.

* It would be beneficial to discuss how the method can be extended in scenarios where transfer learning is applied multiple times or where the data distribution changes online.
questions: questions: Please see the Weaknesses part.
limitations: limitations: They provided in the Supplementary section D.
paper_summary: paper_summary: The paper proposes a new method to fine-tune the pretrained large-scale diffusion models for new tasks. It finds that different time steps of the denoising process of diffusion models have varying transferability. Specifically, the paper finds that "low-noise" time steps close to the end of the denoising process have better transferability. In contrast, the "high-noise" steps responsible for generating the image layout and containing domain knowledge are less transferable. Based on these observations, the paper proposes a novel fine-tuning objective demonstrating better convergence speed than baselines.
strength_weakness: strength_weakness: 1. I could not follow the motivation behind the scheme that the paper claims to perform Knowledge Retention with. The main argument of the paper is that "the time steps of the denoising process close to the end of the sampling chain of diffusion models are more transferable." Then, why do we need to sample an augmented dataset to do the Knowledge Retention? A straightforward way can be using knowledge distillation from the original model to the fine-tuned one in the low noise time-steps. 

2. Following the previous point, distillation may be better than an augmented dataset considering that the fine-tuning data may have characteristics that are different from the original domain. For instance, (this is just a hypothetical example) the fine-tuning dataset may be on flowers that are mostly red, but the pretaining dataset (ImageNet in the experiments) may not have red flowers. Therefore, for the fine-tuning process, we care more about red flowers, not other types. However, using an augmented dataset can bias the fine-tuning process and lower the model's quality on red flowers. In contrast, distillation on the low-noise time steps may readily preserve the prior knowledge of the model without biasing the fine-tuning process. 

3. I appreciate the analysis in Fig. 1, and the formal experimentation of it is a novelty. Yet, some similar phenomena have been shown in the literature before that the paper does not discuss them. [1] shows that a diffusion model usually generates a high-level layout of the image until SNR gets between [0.01, 1], and then, starts to fill in the details. Similarly, eDiff-I shows that changing the text prompt at the low-noise stages of the denoising process process does not change the output image. These experiments in Fig. 1 also show that if we use the original model in the low-noise time steps, the performance does not decrease, which is similar to these papers. I think the paper should include these references in the related work section. 

[1] Perception Prioritized Training of Diffusion Models, CVPR 2022. 

[2] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023.
questions: questions: I have the following questions from the paper:

1. When fine-tuning the ImageNet pretrained model on the new task, do you train the class embeddings (used for cross-attention) for the new task as well? 

2. Regarding points 1 and 2 in the Weaknesses section, as far as I know, training on images generated by the generative models may degrade the model's performance [1]. Yet, using the whole ImageNet dataset performs worse than using 5k images that are sampled from the pretrained model. Is there any intuition about why this happens?

[1] Self-consuming generative models go mad, ICLR 2024.
limitations: limitations: The authors have pointed to using an augmented dataset as a potential limitation, but as I mentioned in the weakness section, the motivation and intuition of why it should work is not clear.
paper_summary: paper_summary: This paper explores the fundamental transfer characteristics of diffusion models and observes the monotonous chain of forgetting trend of transferability of diffusion models in the reverse process. It then proposes a simple but effective transfer approach to make the fine-tuned model retain the denoising ability of the pre-trained model close to the generated image, while using domain-specific denoising ability at the beginning of the denoising process. Experimentally, Diff-Tuning can achieve effective improvement on some standard fine-tuning tasks, and also improve the convergence speed of ControlNet.
strength_weakness: strength_weakness: 1. There are too few metrics on class-conditional generation, while FID is used to extract features with models trained on top of ImageNet, authors use DiT pre-trained on ImageNet to generate samples, which are then added to the training process, it is difficult to say the lower FID actually translates to improved image quality. can you use more metric to show your results?
questions: questions: 1. the authors use a pre-trained diffusion model to preserve denoising ability, would it be useful to use pre-trained domains, i.e. images from ImageNet, instead of augmented images? Although I know that in most cases we can't fully know the pre-trained diffusion model's pre-training data
2. when using a pre-trained diffusion model to generate augmented data, do different sampling methods achieve similar results, and does the choice of sampling steps make a difference?
3. would using a pre-trained diffusion model on ImageNet and then fine-tuning it on small data achieve better results than a diffusion model trained directly on small data? For example, pre-training on ImageNet and then fine-tuning on CIFAR10/100.
limitations: limitations: yes
paper_summary: paper_summary: This paper proposes Diff-Tuning method to encourage the fine-tuned model to retain the pre-trained knowledge. In this method, both pre-trained data and downstream data are used to train the diffusion model. Compared to standard fine-tuning methods, Diff-Tuning enhances the convergence speed and improves the performance. Diff-Tuning can also be used in Conditional Generation. Additionally, Diff-Tuning is compatible with existing parameter-efficient fine-tuning methods.
strength_weakness: strength_weakness: + The novelty of this paper is limited. Many previous works [1,2] find that gradients exist in conflict for diffusion models across timesteps even in the same dataset. But none of them are compared.  **Since the core conclusions of this paper are very similar to them, I think the author needs to highlight the differences.**
1. Efficient Diffusion Training via Min-SNR Weighting Strategy ICCV 2023
2. Addressing Negative Transfer in Diffusion Models. NeurIPS 2023
+ The robustness of Diff-Tuning to sampling algorithms has not been verified, including DDIM [1], DPM-Solver [2], and others. Meanwhile, More diffusion models need to be validated, including VP-SDE [3], Flow Matching [4], EDM [5], etc.
1. Denoising Diffusion Implicit Models ICLR 2021
2. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps Neurips 2022 O
3. Score-Based Generative Modeling through Stochastic Differential Equations ICLR2021
4.  Flow Matching for Generative Modeling ICLR 2023
5. Elucidating the Design Space of Diffusion-Based Generative Models NeurIPS 2022

+ In the training of the diffusion model, there are different time schedules (Linear, Cosine, etc.), so do I need to keep the same schedule in the Diff-Tuning?
questions: questions: Please see Weakness. If my question is answered, I will raise my score.
limitations: limitations: Please see Weakness. If my question is answered, I will raise my score.

Comments: title: Official Comment by Reviewer WW5v
comment: I appreciate the author's response. Most of my concerns have been addressed, and I raise the score.
I appreciate the authors' rebuttal. The rebuttal addressed most of my concerns, and I raised my score to 5. I did not use a higher score as I think the design choice of using an augmented dataset is not well-motivated. The rebuttal did not provide a compelling answer to my second comment in the weakness section of my initial review. Still, I raised my score as the framework is flexible such that one can replace the augmented dataset with distillation, as the provided experimental results in the rebuttal suggest.
Thank you for your reply, authors. This has addressed my main concern. I remain inclined to accept and maintain my score.
I thank the author for his reply, which resolved my main confusion. Consequently, I have raised my score.