Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting
Authors: Jincheng Zhong
Xingzhuo Guo
Jiaxiang Dong
Mingsheng Long
Venue: NeurIPS 2024 poster (2024)
Keywords: Transfer Learning
Fine-tuning
Diffusion Model
Generative Model
Primary area: diffusion_based_models
TL;DR: This paper introduces Diff-Tuning, a method to adapt pre-trained diffusion models, showing significant improvements over standard fine-tuning across eight tasks and  five conditions with ControlNet.

Abstract: Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models.
  In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side.
  We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet.
  Diff-Tuning achieves a 24.6% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning. Code
  is available at this repository: https://github.com/thuml/Diffusion-Tuning.

Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting
Jincheng Zhong, Xingzhuo Guo," Jiaxiang Dong, Mingsheng Long
School of Software, BNRist, Tsinghua University, China
(zjc22,gxze) 2maais..0mmmm....mmmm.mmmmmmmlm. tsinghua.eene edu. cn, mingshengghhsinguua.edi
Abstract
Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus parameter-efficiet transfer learning but overlook the fundamental transfer characteristics of diffusion models. this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetring trend of transferability along the reverse process. Based this observation and novel theoretical insights, present Tuming, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuming encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side. We conduct comprehensive experiments evaluate Diff- Tuningn including the transfer of pre-trained Diffusion Transformer models eight downstream generations and the adaptation of Stable Diffusion five control conditions with ControlNet. Diff-Tuning achieves a 24.6% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, ceaumetr--fffcieer transfer learning techniques for diffusion models can also benefit from Diff- Diff-Tuning. Code is available at this repository: https://githuu: cww.cooo//tii/ee wwwpl//ifffoooooooo Tuning.
Introduction
Diffusion models 45 are leading the revolution in modern generative modeling, achieving remarkable successes across various domains such as image [IT] 39 [2], video [43 19 55), 3D
shape (34], audio generation 25 etc. Despite these advances, training an applicable diffusion model from scratch often demands a substantial computational budget, exemplified by the thousands of TPUs needed, as reported by [55]. Consequently, finetuning well ore-trained, large-scale models for specific tasks has become increasingly crucial in practice [54] 60 56].
During the past years, the deep learning community has concentrated how transfer knowledge from large-scale pre-trained models with minimal computational and memory demands, a process known parameter-efficient fine-tuning (PEFT) [20 59 54
21 32] The central insight of these approaches to update as few parameters as possible while avoiding performance decline. However, the intrinsic transfer properties of diffusion models have remained largely unexplored, with scant attention paid to effectively fine-tuning from a pre-trained diffusson model.
7
Previous studies neural network transferability, such those by B3 57 have demonstrated that lower-level features are generally more transferable than ligher-level features. the context of diffusion models, which transform noise into data through reverse process, it is logical to assume
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that the initial stages, which are responsible for shaping high-level objects, differ in transferability from later stages that refine details. This differential transferability across the denoising stages presents an opportunity to enhance the efficacy of fine-tuning.
In this work, we investigate the transferability within the reverse process of diffusion models. Firstly, we propose that a pre-trained model can act as universal denoiser for lightly corrupted data, capable of recognizing and refining subtle distortions (see Figure D). This ability leads improved generation quality when we directly replace the fine-tuned model with the original pre-trained one under low distortion. The suboptimality observed with fine-tuned models suggests potential overfitting, mode collapse, or undesirable forgetting. Then we extend the experiments by gradually increasing the denoising steps replaced, cover higher-level noised data, observing the boundaries of zero-shot generalization capability. This indicates that the fine-tuning objective should prioritize high-leve shaping, associated with domain-specific characteristics. We term this gradual loss of adaptability the chain of forgetting, which tends to retain low-level denoising skills while forgetting high-level, comain-speciife characteristies during the transfer of the pre-trained model. We further provide novel theoretical insights to reveal the principles behind the chain of forgetting.
Since the chain of forgetting suggests different denoising stages lead different forgetting preferences, it is reasonable to develop a transfer strategy that balances the degrees of forgetting and retention. Technically, based on the above motivation, we propose Piff-Tuning, a frustratingly simple but general fine-tuning approach for diffusion models. Diff- Tuning extends the conventional fine- tuning objectives by integrating two specific aims: 1) knowledge retention, which retains general denoising knowledge; 2) knowledge reconsolidation, which tailors high-level shaping characteristics to specific downstream domains. Diff- Tuffnn leverages the chain of forgetting balance these two complementary objectives throughout the reverse process.
Experimentally, Diff-Tuning achieves significant performance improvements over standard fine- tuning in two mainstream fine-tuning scenarios: conditional generation and controllable generation with ControlNet 660. Our contributions can be summarized as follows:
Motivated by the transferable features of deep neural networks, we explore the transferability of diffusion models through the reverse process and observe chain of forgetting tendency, We provide a novel theoretical perspective to elucidate the underlying principles of this phenomenon for diffusion models.
We introduce Diff-Tuning, frustratingly simple yet effective transfer learning method that integrates two key objectives: knowledge retention and knowledge reconsolidation. Diff- Tuning harmonizes these two complementary goals by leveraging the chain of forgetting.
As a general transfer approach, Diff- Diff-Tuning achieves significant improvements over its stan- dard fine-tuning counterparts in conditional generation across eight datasets and controllable generation using ControlNet under five distinct conditions. Notably, Diff-Tuning enhances the transferability of the current PEFT approaches, demonstrating the generality.
2 Related Work
2.1 Diffusion Models
represent the state-of-the-art in generative modeling 12 capable of progressively generating samples from random noise through a chain of denoising processes. Researchers have developed large-scale foundation diffusion models across a broad range of domains, including image synthesis [17]. video generation [19], and cross-modal generation [3] 42). Typically, training diffusion models involves learning a parametrized function f to distinguish the noise signal from disturbed sample, as formalized below:
Diffusion models 17 and their variants 47
48 23
L(0) Ex, [||e - So (Vaixo + v1 a.e, t){("
where Xg ~ X represents real samples, e~ (N,, I) denotes the noise signal, and X; = /arXi vI- aye is the disturbed sample at timestep t. Sampling from diffusion models follows a Markoy chain by iteratively denoising from x ~ ~N(0,, to xo.
Previous research on diffusion models primarily focuses on noise schedules 36 2
training
objectives 44 23), efficient sampling [46), and model architectures [39]. In contrast to these existing

---

Reletive Cain (PID)
Performance drops domain specific choroctenggies
Improve by zarovihok from pre-sramne knowledg!
Fineetumeed
aindr
Diff-Tuning
Fine-tured
Pre-traimed
Diff-Turing
Replaced Steps
Figure I: Case study of directly replacing the denoiser with the original pre-trained model on lightly disturbed data (left). The changes in Fréchet Inception Distance (FID) as the denoising steps incrementally replaced by the original pre-trained model (right).
works, our method investigates the transferability of diffusion models across different denoising stages and enhances the transfer efficacy novel and intrinsic way.
2.2 Transfer Learning
Transfer learning (38) is an important machine learning paradigm that aims to improve the perfor- mance of target tasks by leveraging knowledge from source domains. Transferring from pre-trained models, commonly known as fine-tuning, has been widely proved effective in practice, especially for the advanced large-scale models [5] I 12]. However, directly tinettuiing a pre-trained model can cause overfitting, mode collapse, and catastrophic forgetting (24]. Extensive prior work has focused on overcoming these challenges ultimately enhance the utilization knowledge from pre- trained However, effective transfer of diffusion models has received scant attention.
models 2 g
Parameter-Elficieet Fine-tuning (PEFT) With significant advancements in the development of large-scale models [TO, 5 II [2]. research in transfer learning has increasingly concentrated on PEFT methods that minimize the number of learnable parameters. The primary goal of PEFT is to reduce time and memory costs associated with adapting large-scale pre- pre-trained models. Techniques such as incorporating extra adapters [20 60 351 and learning partial or re- re- parameterized parameters
are employed for their effectiveness in reducing computational demands. Nevertheless, the reliance on deep model architectures and the necessity of carefully selecting optimal placements present substantial challenges. Intuitively, PEFT approaches could potentially mitigate catastrophic forgetting by preserving most parameters unchanged; for a detailed discussion, refer to Section 4.3
559
21 12
Mitigating Catastrophic Forgetting Catastrophic forgetting is long- standigan challenge in the context of continual learning, lifelong learning, and transfer learning, referring to the tendency of neural networks to forget previously acquired knowledge when fine-tuning new tasks. Recent ex- ploration in parameter regularization approaches (24 28
have gained prominence. Approaches such as [58 52 propose the data-based regularization, which involves distilling pre-trained knowledge into a knowledge bank. However, efforts to mitigate forgetting within the framework of diffusion models remain notably scarce.
3 Method
3.1
Chain of Forgetting
Compared with one-way models, diffusion models specify in manner of multi-step denoising and step-independent training objectives. Inspired by prior studies on the transferability of deep neural features (57 (33,, we first explore how the transferability of diffusion models varies along the denoising steps.
Pre-trained Model Serves as a Zero-Shot Denoiser Modern large-scale models are pre-trained with large training corpus, emerging powerful zero-shot generalization capabilities. We begin by analyzing whether the pre-trained diffusion models hold similar zero-shot denoising capabilities. In particular, we utilize popular pre-trained Diffusion Transformer (DiT) model [39] as our testbed. We fine-tune the DiT model on a downstream dataset. When the reverse process comes to the the last 10%

---

steps, we switch and continue the remaining denoising steps with the fine-tuned model, the original pre-trained model, and our Diff-Tuning model respectively. We visualize a case study in Figure [ (left) with corresponding replacement setups. Surprisingly, the results reveal that replacement by the pre-trained model achieves competitive quality, even slightly better than the fine-tuned one, indicating that the pre-trained diffusion model indeed holds the zero-shot denoising skills. On the other side, some undesirable overfitting and forgetting occur when fine-tuning diffusion models.
Forgetting Trend Next, we delve deeper into investigating the boundary of generalization capabili- ties for the pre-trained model. Figure (right) illustrates the performance trend when we gradually increase the percentage of denoising steps replaced from 0 to 100%. Initially, this naive replacement yields better generation when applied towards the end of the reverse process. However, as more steps are replaced, performance begins to decline due to domain mismatch. This trend suggests the fine-tuned model may overfit the downstream task and forget some of the fundamental denoising knowledge initially possessed by the pre-trained model when t is small. Conversely, f increases, the objects desirable the new domain are distorted by the pre-trained model, resulting in a performance drop. Based on these observations, we conceptually separate the reverse process into two stages: (1) domain-specinf shaping, and (2) general noise refining. We claim that the general noise refining stage is more transferable and can be reused across various domains. In contrast, the domain-specific shaping stage requires the fine-tuned model to forget the characteristics of the original domain and relearn from the new domains.
Theoretic Insights
Beyond empirical observations, we provide novel theoretical perspective of the transfer preference for the pre-trained diffusion model. Following the objectives of diffusion models, a denoiser F (an Xu reperaattteriiitooonn [23] of f in Eq. (T) is to approximate the posterior expectation of real data over distribution D. This is formalized by:
Sx, N ((x:: / /0,xo (1 a1 Xo PD (x
F F(x:) Exxx-p(xalxx xx) [xo]
(2)
Sx (xxi Vaix, (1 - av, pD (xo)xx
where PD (xo) represents the distribution of real data from D, and N denotes the Gaussian distribu- tions determined by the forward process. Notably, a larger variance of Gaussian distribution indicates more uniform distribution. Through detailed investigation of these Gaussian distributions under varying timesteps f, we derive the following theorem. All proofs and derivations are provided in Appendix A
Theorem (Chain of Forgetting) Suppose diffusion model with lim as 1 and lim a 0 over finite samples, then the ideal denoiser F satisfies
(-T
1. lim F(xx)= = argminn[>x0, Xx [|]. i.e., the closest sample in dataset.
1-0
p(xo)>0
2. lim F(x)) = Exx (xx) [x0], i.e., the mean of data distribution.
b+T
Theorem D elucidates the mechanism behind the chain of forgetting. On one hand, when & + 0. a model optimized on a training dataset D can perform zero-shot denoising within the vicinity of the support set supp( (D). As the training dataset scale expands, so does the coverage of (Dpp(D). enabling diffusion models to act as general zero-shot denoisers for data associated with small On the other hand, as t + T, the model's 's generalization is significantly influenced by the distribution distance (Es [ E... [x)""), where Drew denotes the dataset of the new domain. This theorem highlights the necessity for further adaptation in the new domain.
3.2 Diff-Tuning
Based on the above observations and theoretical insights, we introduce Diff- Diff-Tuning, which incor- porates two complementary strategies to leverage the chain of forgetting in the reverse process: 1) knowledge retention, and 2) knowledge reconsolidation. Diff-Tuning aims to retain general denoising skills from the pre-trained model while discarding its redundant, domain-specific shaping knowl- edge. This enables the model to adapt more effectively to the specific characteristies of downstream tasks. Diff- Diff-Tuning harmonizes the retention and reconsolidation via the chain of forgetting tendency.

---

Tune
Chain of Forgetcing
Retain
(a) Build Knowledge
XT
X
x--1
X
0
Knowledge
Retention
Noise
Reverse Process
Data
Knowledge Reconsolidation
(b) Diff- Tuninu
Domain-Specific Shaping
General Noise Refining
The conceptual illustration of the chain of forgetting (Left). The increasing forgetting tendency as & grows. (a) Build a knowledge bank for the pre-trained model before fine-tuning. (b)
Figure 2:
Diff-Tuning leverages knowledge retention and reconsolidation, via the chain of forgetting.
Without loss of generality, we present Diff-Tuning under the standard DDPM objective, omitting conditions in the formulations. The general conditional generation setup will be discussed later.
Knowledge Retention As discussed earlier, retaining pre-trained knowledge during the latter general noising refining proves beneficial. However, the classic parameter-reguularzztioo based approaches [24] 27 mitigate forgetting uniformly across the reverse process, primarily due to parameter-sharing design inherent in diffusion models. To address this, Diff-Tuning constructs an augmented dataset X
(x*,...) pre-sampled from the pre-trained model. This dataset acts as a repository the retained knowledge of the pre-trained model. We define the auxiliary training objective, Lietention follows:
Lxeuetton (0) B, [[(t)|c- ( + -V-A:-+---- vic t)]].
where {(t) is the retention coefficient. In accordance with the principles of the chain of forgetting, E(t) decreases monotonically with increasing t. promoting the retention knowledge associated with small values and the discarding of knowledge related to large t values. Knowledge Retention shares a similar formulation with the pre-training objective but without the reliance on the original pre-training dataset.
is important to note that the concept of Knowledge Retention in Diff-Tuning is anchored in the principle of the chain of forgetting. This approach encourages the model to recall how to denoise samples with low levels of disturbance, as reflected by the retention coefficient E(t). While the proposed augmented dataset serves flexibility of Diff-Tuning various other methods can also effectively facilitate knowledge retention. An alternative approach involving knowledge distillation is detailed in Appendix C
casy-to-implement example of this concept, given the
Knowledge Reconsolidation In contrast to knowledge retention, knowledge reconsolidation fo- cuses adapting pre-trained knowledge new domains. The intuition behind knowledge reconsoli- dation to diminish conflict between forgetting and adaptation by emphasizing the tuning of knowledge associated with large t. This adaptation is formalized as follows:
Ladaptaion (0) Ex,,,x--x
(() |--- So (Vaixo + /1 (),,
(4)
where (b()) is the reconsolidation coefficient, a monotonic increasing function within the range [0. 1], reflecting increased emphasis on domain- sppeiffcc adaptation increases. Similar to Knowledge Retention, the principle behind Knowledge Reconsolidation involves adapting the model to effectively

---

handle samples that are heavily disturbed, which are more sigmificantly influenced by the distance between the pre-trained and target distributions. Reweighting by (t) serves as one of the simplest implementations of this concept.
Frustratingly Simple Approach Overall, we reach Diff- Diff-Tuning, a general and flexible fine- tuning framework for effective transferring pre-trained diffusion models to downstream generations, the overall objective is as follows:
min L recention (0) + addpttation (0),
(5)
where Lecsention (0) and Ladaptation ( are described before, 0 represents the set of tunable parameters. Notably, Dining is architecture- and seamlessly integrates with existing PEFT methods. Further details are discussed in Section
Choices of {tt) and (t) For clarity and simplicity, we define {(t) 1 (t), ensuring equal weighting for each t, following the original DDPM configuration. This complementary design
excludes the influences of recent studies on the t-reweighting techniques [23 12 9 From the above discussion, we can choose any monotonically increasing function whose range falls in [0, 1). In this work, scale the variable f to the interval [0, 1), and apply a simple power function group (b()) t for practical implementation. In our experiments, we report the main results with r = 1, and the variations of the choice are explored in Section 4.4
Conditional Generation Classifier-free guidance (CFG) [8] forms the basis for large-scale con- ditional diffusion models. To facilitate sampling with CFG, advanced diffusion models such as DiT 339 and Stable Diffusion [12] are primarily trained conditionally. CFG is formulated as = (1 w)E. we.. where w. E,, Eu are the CFG weight, conditional output, and unconditional output. As general approach, Diff-Tuning inherits the conditional training and sampling setup to support a wide range of transfer tasks. Due to the mismatch between the pre-training domain and downstream tasks in the conditional space, apply knowledge retention Leetention on the un- conditional branch and knowledge reconsolidation Lxtuptation on both unconditional and conditional branches.
4 Experiments
To fully verify the effectiveness of Diff-Tuning, we extensively conduct experiments across main- stream fine-tuning scenarios: 1) Class-conditional generation, which involves eight well-established fine-grained downstream datasets, and 2) Controllable generation using the recently popular Control- Net [60], which includes five distinct control conditions.
4.1 Transfer to Class-conditional Generation
Setups Class-conditioned generation is a fundamental application of diffusion models. To fully
evaluate transfer efficiency, adhere to the benchmarks with resolution of 256 x 256 as used in
DiffFit (54) including datasets such as Food101 41, SUN397 (53), DF20-Mini 40). Caltech101 [3..
CUB-200-201 [50], ArtBench- 10
[30]. Oxford Flowers 37,, and Stanford Cars (26). Our base
model, the 2-T-XL-2-256x256 [39], is trainrd on ImageNet at 256 x 256 resolution, achieving a
Fréchet Inception Distance (FID) [6 of 2.27 The FID is caleulated by measuring the distance between the generated images and a test set, serving as a widely used metric for evaluating generative image models' quality. We adhere the default generation protocol as specified in (54), generating 10K instances with 50 sampling steps (FID-10K). Seg weight is set to 1.5 for evaluation. For the implemented DiffFit baseline, we follow the optimal settings in (54), which involve enlarging the learning rate x 10 and carefully placing the scale factor to to 14 blocks. For each result, we fine-tune 24K iterations with batch size of 32 for standard fine-tuning and Diff-Tuning, and batch size of 64 for DiffFit, on one NVIDIA A 100 40G GPU. For each benchmark, we recorded the Relative Promotio of FID between Diff- Biff-Tuning and Full Fine-tuning Diff-Tuning Fuil Fine-juning) to highlight the
Full Fine luning
effectiveness of our method. More implementation details can be found Appendix B

---

Table I: Comparisons downstream tasks with pre-trained DTT-XL-2-256x255. Methods with "t" are reported from the original Table 1 of [54] Parameter-efficient methods are denoted by "*"
Datasel
Oxford Flowers
Standard Cars
Average
Food
SUN
DF-20M Caltech CUB-Bird ArtBench
Method
FID
Full Fine-tuning
10.93
14.13
15.29
23.84
5.37
19.94
16.67
6.32
14.06
AdaptFormer"
11.93
10.68
19.01
34.17
7.00
35.04
21.36
10.45
18.70
BitFit [59
9.17
9.11
17.78
34.21
8.81
24.53
20.31
10.64
16.82
VPT-Deep 22
18.47
14.54
32.89
42.78
17.29
40.74
25.59
22.12
26.80
LoRA
21
33.75
32.53
120.25
86.05
56.03
80.99
164.13
76.24
81.25
DifiFit
7.80
10.36
15.24
23.79
4.98
16.40
14.02
5.81
12.21
Diff- Duninng
6.05
9.01
13.64
23.69
3.50
13.85
12.63
5.37
11.08
Relative Promotion 44.6%
36.2%
10.8%
0.6%
34.8%
30.5%
24.2%
15.0%
24.6%
Results Comprehensive results are presented in Table 1 with the best in bold and the second underlined. Compared with other baselines, our Diff-Tuning consistently exhibits the lowest FID across all benchmarks, outperforming the standard fine-tuning by significant margin (relative 24.6% overpass), In contrast, some PEFT techniques do not yield improved results compared to standard fine-tuning. detailed comparison with DiffFit discussed in subsequent sections.
4.2 Transfer to Controllable Generation
Setups Controlling diffusion models enables per- sonalization, customization, or task-specific im- age generation. In this section, evaluate Diff- Tuning on the popular ControlNet [60], a state-of- the-art controlling technique for diffusion models, which can be viewed as fine-tuning the stable dif- high
0.10
Mche S Mom / Ar
Baseline Diff- Diff-Tunnin
0.09
fusion model with conditional adapters level. We test Diff-Tuning under various image- based conditions provided by ControlNe includ-
MSE
Surddon convergence
picked shreshold
ing Sketch [51], Edge [6), Normal Map [49 Depth Map [41], and Segmentation on the COCO [1 and ADE20k (62) datasets at resolution of 512 x 512. We fine-tune ControlNet for 155k iterations for each condition except 5k for Sketch and 20k for Segmen- tation on ADE20k, using batch size of on one NVIDIA A100 40G GPU. For more specific training and inference parameters, refer to Appendix B
0.05
0.04
0
3k
6k
9k
12k
15k
Training Steps
Figure 3: An example of evaluating dissimilari- ties between conditions (the Normal condition) to infer the occurrence of sudden convergence.
Evaluation through Sudden Convergence Steps Due the absence of a robust quantitative metric for evaluating fine-tuning approaches with ControlNet, we propose a novel metric based on the sudden convergence steps. In the sudden convergence phenomenon, as reported in [60], ControlNet tends not to learn control conditions gradually but instead abruptly gains the capability to synthesize images according to these conditions after reaching sudden convergence point. This phenomenon is observable in the showcases presented in Figure 4 throughout the tuning process. We propose measuring the (dis-) )similarity between the original controlling conditions and the post-annotated conditions of the corresponding controlled generated samples. As depicted in Figure 3] a distinct "leap" occurs along the training process, providing clear threshold determine whether sudden convergence has occurred. We manually select this threshold, combined with human assessment, to identify the occurrence of sudden convergence. The detailed setup of this metric discussed in Appendix
Results As demonstrated in Table 2 Diff-Tuning consistently requires significantly fewer steps to reach sudden convergence across all controlling conditions compared standard fine-tuning of
7

---

Sandard ControlNes
Diff- Tuning ControiNes
Jonaryy
profincional
demileed, Atsh.
quality smaage
Edge
2k
4k
5.3k
6k
6.7k
8k
10k
Standard ControiNet
Ieshhow "e profocsiona deturived ligh- quality
Diff-uning Controliics
Seg.
6k
8.9k
12k
13.9k
15k
20k
Figure Qualitative compare Diff-Tuning to the standard ControlNet. occurence of "sudden convergence".
Red boxes refer to the
Table 2: Sudden convergence steps on controlling Stable Diffusion with conditions.
Method
Sketch
Normal Depth
Edge
Seg. (COCO) Seg. (ADE20k)
Average
ControlNet 60
3.8k
10.3k
9.9k
6.7k
9.2k
13.9k
9.0k
ControlNet +Diff- Boff--unning
3.2k
7.8k
8.8k
5.3k
6.3k
8.3k
6.6k
Relative Promotion
15.8%
24.3%
11.1%
20.9%
31.5%
40.3%
24.0%
NontrolNet, indicating consistent enhancement in the transfer efficiency. In Figure we display showcases from the training process both with and without Dini-nuning. is observed that Diff- Tuning achieves sudden convergence significantly faster, enabling the generation of well-controlled samples more quickly. By comparing the images from the final converged model at the same step. is evident that proposed Diff- Diff-Tuning achieves superior image generation quality.
4.3 Discussion on Parameter-Efficionn Transfer Learning
The initial motivation behind adapter-based approaches in continual learning is to prevent catastrophic forgetting by maintaining the original model unchanged [20). These methods conceptually preserve a separate checkpoint for each arriving task, reverting to the appropriate weights as needed during inference. This strategy ensures that knowledge from previously learned tasks not overwritten. In transfer learning, however, the objective shifts to adapting pre-trained model for new, downstream tasks. This adaptation often presents unique challenges. Prior studies indicate that PEFT methods struggle match the performance of full model fine-tuning unless modifications are carefully implemented. Such modifications include significantly increasing learning rates, sometimes by more than tenfold, and strategically placing tunable parameters within suitable blocks [21] 54). Consider the oate-of-the-an art method, DifiFit, which updates only the bias terms in networks, merely 0.12% of the parameters in DiT equating approximately 0.83 million parameters. While this might seem efficient, such small proportion tunable parameters is enough to risk overfitting downstream tasks. Increasing the learning rate to compensate for the limited number of trainable parameters can inadvertently distort the underlying pre-trained knowledge, raising the risk training instability and potentially causing sudden and complete degradation of the pre-trained knowledge, observed in studies like [54].
Elastic Weight Consolidation (EWC) 24 is a classic parameter-regularize approach to preserve knowledge in a neural network. We calculate the EWC values, which are defined as EWC ||0 0 2 for the tunable parameters in the evaluated approaches (28). The EWC value quantifies

---

4414107
tuing DufiTuning
1950
4.50
30
Diff-Tuning
824
g0h
EWC valves
RD3
426
4bc
>>*
53
5.:10-
SUN
DF 2011
Ca lock
CUB-Brl
Avarege RD
Preoonn St-lorgg Tume
Tewraringg Tuing Dife DiFTurg
Difi-Tunin equipped with DifFit
EWC Summation Food101
EWC Average Food101
Figure 5: The compatibility of Diff-Tuning with PEFT (a), and catastrophic forgetting analysis (b-c).
how far the fine-tuned model deviates from the pre- pre-trained model, indicating the degree of knowledge forgetting from the perspective of parameter space.
Figure ((b) reveals that DiffFit leads to EWC values that are 2.42 times larger with only 0.12% tunable parameters, indicating heavy distortion of the pre-trained knowledge. Figure [ic) illustrates the averaged EWC over tunable parameters, showing that each tunable bias term contributes significantly more to the EWC, In contrast, Diff- Tuninn achieves lower EWC values. Diff-Tuning does not explicitly focus avoiding forgetting in the parameter space but rather harmonizes the chain of forgetting in the parameter-sharing diffusion model and only retains knowledge associated with small t rather than the entire reverse process.
Diff-Tunine can be directly applied current PEFT approaches, and the comparison results in Figure 5 demonstrate that Diff-Tuning can enhance the transfer capability of DiffFit and significantly improve converged performance.
4.4 Analysis and Ablation
Fine-tuning Convergence Analysis To analyze converging speed, we present concrete study on the convergence of the FID scores for standard fine-tuning suee-tuning DiffFit, Diff-Tuning and Diff-Tuning Tuning" (DifiFit equipped with Diff-Tuning) every 1,500 iterations the SUN 397 dataset, as shown in Figure a(a). Compared to standard fine-tuning and DifiFit, Diff- Tuning effectively leverages the chain of forgetting, achieving balance between forgetting and retaining. This leads to faster convergence and superior results. Furthermore, the result of Diff-Tu indicates that PEFT methods such like DiffFit still struggle with forgetting and overfitting. These methods can benefit from Diff-Tuning.
Ablation Study We explore the efficacy of each module within Diff- Tuming, specifically focusing on knowledge retention and knowledge reconsolidation. We assess Diff Diff-Tuning against its variants where: (1) Only reconsolidation is applied, setting
{(t) O and (t) f; and (2) Only retention is employed, setting {(t) = 1 and (t( = The results, illustrated in Figure ((b) demonstrate that both knowledge retention and knowledge reconsolidation effectively leverage the chain of forgetting to enhance fine-tuning performance. The FID scores reported on the DF20M dataset clearly show
that combining these strategies leads to more efficient learning adaptations.
655
5.00
Thooaaiing
5.0
6.15
$36
2.54
5.31
5.50
zo.
550
Turkng
14:5
1433
575
555
550
1364
5.40
477
7.3
Frestaeng
00 Comeergence analysss on 397
(11 Ablodon souly DF20M
(0 The sensibblty of 9/0) Car
(6 Analysk on the toomples memory on Car
Figure 6: Transfer convergence analysis (a), ablation study (b), and sensitivity analysis (c-d).
Tradeoff the Forgetting and Retraining with the Chain of Forgetting For simplicity and ease of implementation. Diff-Tuning adopts power function, (b() = t, as the default reconsolidation coeffi- cient. To explore sensitivity to hyperparameters, we conduct experiments using various coefficient functions (t)) l" with values from the set (0, 0.3, 0.5, 0.7, 1.5). and signal-to-noise ratio (SNR) based function (t) = 1/(1 + SNR(()) [9] Results on the Stanford Car dataset, shown in Figure T c), carefully tuned coefficient yield slightly better results. To keep the simplicity,

---

25
5.38
5.23
2144
Fine-Turing Food. Dif: Turing
CUB-Bird,
Fioo-uiing
CUB Bird, Duii
13:06
12.60
12.04
15-
12
12.28
3.44
10.11
10299
1.25
1.50
3413
280
189
00
219
103
26
55
100
500
50
120
500
100
2009-2016
2009-2310
Aue HD
Sampling steps the sugmented dataset
The sensidhity of sampling sceps
Continual leamning with evolving datasets
Figure The influence of the quality of the augmented dataset (a), sensitivity with respect different sampling steps (b), and application of Diff-Tuning in a continual learning setup (c).
keep the default setting т 1. Notably, when T = 0, Diff-Tuming reduces to the standard fine-tuning baseline.
Analysis on Knowledge Retention In Diff-Tunine knowledge is retained using pre-sampled data from pre-trained diffusion models before fine-tuning. We evaluate the impact of varying sample sizes (5K, 50K, 100K, 200K, and the entire source dataset) on the performance of the DiT model on the Stanford Car dataset, illustrated in Figure dd). Notably, using the entire source dataset, which comprises 1.2M ImageNet images, results in suboptimal outcomes. This observation underscores that pre-sampled data serve more precise distillation of pre-trained knowledge, aligning with goal of retraining knowledge rather than merely introducing extra training data.
Beyond the size of the augmented dataset, the quality of the samples can also influence knowledge retention. Therefore, we further analyze the impact of different sampling steps during pre- sampling stage, as depicted in Figure aa). The findings demonstrate that Diff-Tuning exhibits consistent performance, indicating that the knowledge retention process is robust to variations in the sampling process of the augmented dataset.
Sensitivity to Sampling Steps We evaluate Diff-Tuning using various sampling parameters, specif- ically settings of 25, 50, 100, and 500 steps. As depicted in Figure (T))), there is consistent improvement in performance across these configurations. Notably, Difi-Tuning shows a more sig- nificant enhancement with fewer steps, suggesting that it builds more precise denoising model compared to standard fine-tuning.
4.5 Extend Diff-Tuning to Continual Learning
In dynamic target domains like online systems that continually collect new data, continual learning is essential as models must undergo iterative fine-tuning to adapt to evolving datasets. To evaluate the adaptability of Diff-Tuning in this context, we extend out method and conduct experiments on the Evolving Image Search dataset 633. which includes images from 10 categories collected over three phases: 2009-2012, 2013-2016, and 2017-2020 We apply transfer learning sequentially across these temporal splits, and measure FID on the cumulative test set. For Diff-Tuning, we retain 40% of the initially sampled augmented data and generated the remaining 60% using the fine-tuned model before proceeding to the next phase. As shown in Figure (c), the results demonstrate that Diff- Diff-Tuning consistently improves and sustains robust performance as the dataset evolves.
5
Conclusion
In this paper, we explore the transfcrability of diffusion models and provide both empirical obser- vations and novel theoretical insights regarding the transfer preferences in their reverse processes, which we term the chain of forgetting. We present Diff-Tuning frustratingly simple but general transfer learning approach designed for pre- pre-trained diffusion models, leveraging the identified trend of the chain of forgetting. Diff- Tuning effectively enhances transfer performance by integrating knowledge retention and knowledge reconsolidation techniques. Experimentally, Diff-Tuning shows great generality and performance in advanced diffusion models, including conditional generation and controllable synthesis. Additionally, Diff- foff-Tuning is compatible with existing parameter-efffcient fine-tuning methods.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, and the National Engineering Research Center for Big Data Software. We would like to thank our friend Yuchen Zhang for valuable discussions and support for this work.
References
[1] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23X1::3303.08774,, 2023.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine- min-ttning. arXiv preprint arXiv:2012.13255 2020.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2111.5117,, 2023.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. mood-101-mining discriminative components with random forests. In ECCV, 2014.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ar0iv:2005. 14165, 2020.
[6] John Canny, A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679 698, 1986.
[7] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang. and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022.
[8] Xinyang Chen, Sinan Wang. Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NewrlPS, 2019.
[9] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In CVPR, 2022.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
(11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03200 2024.
[13] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-25t object category dataset. 2007.
[14] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning. In ICCV, 2023.
[15] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In ICCV, pages 7441-7451, 2023.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two siael update rule converge to local nash equilibrium. NewrlPS, 2017.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 2022.
[19] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NewrIPS, 2022.

---

[20] Neil Houlsby. Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parämeler-efficient transfer learning for nlp. In ICML, 2019.
[21] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation large language models. arXiv preprint MxXiv2210600966 2021.
[22] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim, Visual prompt tuning. In ECCV, 2022.
[23] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinssee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):551-33526, 2017.
[25] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint 2007:20009,0966,, 2020.
[26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
[27] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan, Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint arXiv: /0ti::/90000999, 2019.
[28] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer leamning with convolutional networks. In Jennifer G. Dy and Andreas Krause, editors, ICML, 2018.
[29] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pottern analysis and machine intelligence, 40(12):2935 2947, 2017.
[30] Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint Xiv::2206..11 /404, 2022.
[31] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014.
[32] Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. Late prompt tuning: A late prompt could better than many prompts. arXiv preprint arXiv:22 10. 1292, 2022.
[33] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learingg transferable features with deep adaptation networks. In JCML, 2015.
[34] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
[35] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-imaae diffusion models. In AAAI, 2024.
[36] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[37] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
[38] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. Trunsactions on knowledge and data engineering, 22(10)) 2009.
[39] William Peebles and Saining Xie. Scalahle diffusion models with transformers. In ICCV, 2023.
[40] Lukáš Picek, Milan Sule, Jif Matas, Thomas Jeppesen, Jacob
Thomas Læsspe, and Tobias Frøslev. Danish fungi 2020-not just another image recognition dataset. In WACV, 2022.
[41] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dttaset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623 1111111233611111-111111 2020.
12

---

[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resoltiion image synthesis with latent diffuusio models. In CVPR, 2022.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurIPS, 2022.
[44] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021.
[45] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
[46] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2Xiv:2000002502.. 2020.
[47] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
[48] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2020.
[49] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv: /008::0900000666 2019.
[50] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- birds-200-2011 dataset. 2011.
[51] Tengfei Wang. Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205. /2952, 2022.
[52] Chenshen Wu, Luis Herranz, Xialei Liu, Joost Van De Weijer. Bogdan Raducanu, al. Memory replay gans: Learning to generate new categories without forgetting. In NeurlPS, 2018.
[53] Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun datahase: Large- scale scene recognition from abbey to zoo. In CVPR, 2010.
[54] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficien fine-tuning. In ICCV, 2023.
[55] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson. Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06/ 14, 2023.
[56] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 2rXiv:2308.06721, 2023.
[57] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurlPS, 2014.
[58] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. In NeurlPS, 2020.
[59] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldherg. Bitfit: Simple parfmeter-effiiiet fine-tuning for transformer-base masked language-models. arXiv preprint arXiv:2106. 10199, 2021.
[60] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In /CCV, 2023.
[61] Jincheng Zhong, Haoyu Ma, Ximei Wang, Zhi Kou, and Mingsheng Long. Bi-tuning: Efficient transfer from pre-trained models. In ECML-PKDD, 2023.
[62] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba Scene parsing through ade20k dataset. In CVPR, 2017.
[63] Shiji Zhou, Lianzhe Wang, Shanghang Zhang, Zhi Wang, and Wenwu Zhu. Active gradual domain adaptation: Datasct and approach. IEEE Transactions Multimedia, 24:1210- 1220, 2022.
13

---

A Proofs of Theoretical Explanation in Section 3.1
In this section we provide the formal definitions and proofs for theoretical explanation of chain of forgetting. From the setup of diffusion model training and previous works [23]. we suppose
the dataset consists of finite bounded samples D {x0
(1) X
(2) x (m)). and f is the denoiser to
minimize [(0) as in Eq. ① under e-parameterrzatiinn For convenience, we first convert the denoiser
into X parameteriizaionn by F(x:) ---VT-iif(x and the objective becomes
val
L = E x
Xx -(x))
(6)
Ideal Denoiser in Eq. 6 An ideal denoiser F should minimize the value F(xx) for all t, X almost surely, implying an objective for F(x:):
Lix, (F(x:)) Exoxp(xolx.)
||x0- - Fxx))
By taking derivative, it holds that
V((x) Le (F(x)) -xxxp(xalxx)) 1-2 F(x:))],
(8)
and finally,
F F(x:) Ex (xa) [xo]
(9)
S xo p(xo (x))dxx ddxo
(10)
xo
[x Xo PD xx (xo))((((xe (x+ x dxxo
(11)
(x()
Sx xo pD (x ((( )xxo
(12)
Sxo PD PDD x (x)(x)p(x) X, xx ddxo
Sx, -N(xii Vaixo, (1 a)) PD (xo)dxx
(13)
Sx N N(xx5 vaixo, (1-aa))) a1-a,)1) PD (x)
N ((xi: /apxo (1 - II) Xo
(14)
Nx vaixo, (1 - aI
Remark, This ideal denoiser is exactly the same one 2 under DDPM-style definition.
Case when O. When 0, ar + For simplicity suppose the closest sample to x; is unique.
Let
x elossest arggmn I Vixx x1112
(15)
d = min IIva, x111= SnEDi(
ll a. /a
closest x,113 0,
(16)
then
Ex Ex VaiXo, (1- (x))] x0
0
N (xi: / Vaixp dlosess (1 (I))
x0
x
(17)
Σ
1 / (1-aa)
exp
-IIVa, x + Va,
closest X
(18)
2(1 (42)
xnE D) [2drmor)
Σ
22T(1 - aaa)
1
exp
2 ax))
d
+0
(19)
xoEDY (2dnnn)
as 0r + 11 i.c.
ExSED (xxri vanzo, (1 - ar.) a1) X0 closest
Nxxi var clasest (1 - ax,) I)
x
(20)
14

---

Similarly
ExeED (x5s vapx, (1-
+ 1,
(21)
(xx: vaixi Clmeu (1- (1-ax)1)
Xolesss which completes the proof. Notably, when there multiple closest samples, through similar analysis it is clear that (xx)) converges to their average.
and thus F(x,)
x
Case when f - T. When t T. ar O, and thus N N(x; VayXo, (1 a,)) -x(xx;;; constant for varying Xo. Bringing this back to Eq. (14) and it holds that
0,T;
F(x,)
Σ
X0:
(22)
x
which completes the proof.
B
Implementation Details
We provide the details of our experiment configuration in this section. All experiments are imple- mented by Pytorch and conducted on NVIDIA A100 40G GPUs.
B.1
Benchmark Descriptions
This section describes the benchmarks utilized in our experiments.
B.1.1
Class-conditional Generation Tasks
Food101 The dataset consists of 101 food categories with total of 101,000 images. For each class, 750 training images preserving some amount of noise and 250 manually reviewed test images are provided. All images were rescaled to have maximum side length of 512 pixels.
The SUN397 benchmark contains 108,753 images of 397 well-sampled categories from the origin Scene UNderstanding (SUN) database. The number of images varies across categories, but there are at least 100 images per category. We evaluate the methods on random partition of the whole dataset with 76, 128 training images, 10,875 validation images and 21,750 test images.
SUN397 53
DF20M 40
DF20 is a new fine-grained dataset and benchmark featuring highly accurate class labels based on the taxonomy of observations submitted to the Danish Fungal Atlas. The dataset has a well-defined class hierarchy and a rich observational metadata. It is characterized by a highly imbalanced long-tailed class distribution and negligible error rate. Importantly, DF20 has no intersection with ImageNet, ensuring unbiased comparison of models finettuned from ImageNet checkpoints.
Caltech101 [50] The Caltech 101 dataset comprises photos objects within 101 distinct categories, with roughly 40 to 800 images allocated to each category. The majority of the categories have around 50 images. Each image is approximately 300x200 pixels in size.
CUB-200- 11B--20-2011 (Caltech- Birds- 2adss2200-2011 (01) 1) is an expansion of the CUB- 200 dataset by approximately doubling the number of images per category and adding new annotations for part locations. The dataset consists of 11,788 images divided into 200 categories.
CUB-200-201 50
Artbench10 [30] ArtBench- 10 is class-balanced standardized dataset comprising 60,000 high- quality images artwork annotated with clean and precise labels. It offers several advantages over previous artwork datasets including balanced class distribution, high-quality images, and standardized data collection and pre- pre-processing procedures. It contains 5,000 training images and 1,000 testing images per style.
The Oxford 102 Flowers Dataset contains high quality images of 102 commonly occurring flower categories in the United Kingdom. The number of images per category range between 40 and 258. This extensive dataset provides an excellent resource for various computer vision applications, especially those focused on flower recognition and classification.
Oxford Flowers 37
15

---

Food 01
SUN397
Chocee
Conerel
Miad
Dar
Purm Opea Oriberstroo
Deer
Ione
DF20M
1
Hosplooe Delerte
Aand-rroa Kavin
Nouta
Frund
CUB-200-201
Artbench
Unyve Reraseen Bot beprttaamenn Boogge
Boeck sellww qpiiter
Figure 8: Samples show of different datasets.
Stanford Cars 26
In the Stanford Cars dataset, there are 16,185 images that display 196 distinct classes of cars. These images are divided into a training and testing set: 8,144 images for training and 8,041 images for testing. The distribution of samples among classes is almost balanced. Each class represents specific make, model, and year combination, e.g., the 2012 Tesla Model S or 2012 BMW M3 coupe.
B.1.2 Controllable Generation
The MS COCO dataset is large-scale object detection, segmentation, key-point detection, and captioning dataset. We adopt the version of 2017 where 164k images are split into 118k/5k/41k for wraining/validationnesst For each image, randomly select one of its corresponding captions, and use detectors of Canny edge, (binarized) HED sketch, MIDAS depth/normal and Uniformer segmentation implemented in ControlNet obtain the annotation control, and final construct the dataset of image-text-contro) pairs for training and evaluation. All controling condition has channel except normal map and segmentation mask with channel 3.
COCO [31]
The Ade20k dataset is semantic segmentation dataset containing nearly 21k/2k training/validation images annotated with pixel-wise segmentation mask of 149 categories stuff and objects. For each image, we use the "default prompt" "a high-quality, detailed, and professional image" as adopted in ControlNet, and use Uniformer implemented in ControlNet to obtain the segmentation mask as the control to obtain the dataset of image-text-contrel pairs.
Ade20k 62
16

---

B.2
Experiment Details
For all experiments, we first generate images of memory bank with the pre- pre-trained model. We then construct conditioned dataset from memory bank with default condition (unconditional for class-conditional generation task and "default prompt" with generated control for controllable generation task). Considering instability of weighted training loss with {(t) and ()),), for each iteration, we instead first uniformly sample timestep according the property E(t) (b(() 1, then determine whether sample Xo from the downstream dataset the augmented dataset, and finally caleulate the diffusion loss L(0) with the simple form. The pseudo-code of the overall algorithm process is shown in Algorithm
Detailed Algorithm Process
Algorithm 1 Pseudo-code Diff- Tuning
Input:
Downstream dataset X, pre-trained model parameter Oo
Output:
Fine-tuned parameter 0.
Collect pre-sampled data X for knowledge reconsolidation using Oo- Initialize 0 0u.
while not converged do Initialize mini-batch X = (}. for in 2, 2,., B} do
Sample (( from uniform distribution 2/ (f1, ..., T}).
with prohahility E(flt) then
Sample x X=. (Retention loss Lretenion }
else
Sample x6" ~ X. |Adaptation loss Ladaption )
end if
Add (xo ,,()) mini-batch
end
Calculate
B
L(
Σ
E.
| fo (varoix()) +VI 20))
B
Update according to VaL(O). end while return
Hyperparameters
We list all hyperparameters in our experiments in Table 3
Table 3: Hyperparameters of experiments.
Class-conditional
Controlled
Backhone
DIT
Stable-diffusion v1.5
Image Size
256
512
Batch Size
32
Learning Rate
le-4
le-5
Optimizer
Adam
Adam
Training Steps
24,000
15,000
Validation Interval
24,000
100
Sampling Steps
50
50
Augmented Dataset Size
200,000
30,000
C Knowledge Distillation Implementation of Knowledge Rentention
There exist many possible methods that can achieve similar effect Knowledge Retention, provided they adhere to the principle of the chain of forgetting. In this section, we introduce a variant that utilizes knowledge distillation (KD), which avoids the need to pre-sample an augmented dataset and presents more elegant solution. The KD variant can be formalized as follows:
KD sssention (0) = xx,,,xxx ((() foo (Vaix + VI aaee, 1) So (Vaix + vI- VT-aee 119 (23)
17

---

where fa, represents the original pre- pre-trained model. The results, depicted in Table 4 indicate that both techniques should be effective within the framework of Diff-Tuning Tuning. The practical decision can be informed by the specific scenario. Below, we summarize reasons why KD was not our initial choice:
GPU Memory: KD requires maintaining copy of the pre-trained model alongside the fine-tuning model. For large models, this significantly increases memory costs. For example, run Diff-Tuning with batch size of 32 on single A 100 40GB GPU, whereas the KD variant decreases to a batch size of 24.
Computational Cost: KD doubles the forward computation cost by necessitating the matching of output distributions between two models. Notably, pre-computing the KD labels is not feasible due to the inherent noise in diffusion training. For instance, for a batch size of 24, we achieve 2.1 training steps per second, compared 1.34 for the KD variant.
Training Instability: Transferring a pre-trained traindd model domain significantly different from its training data can introduce out-of citt-o-ddisributtio corruption during distillation, poten- tially causing instability the fine-tuning process. We invested considerable effort to tune a suitable trade-off for the KD) loss (0.05), and sometimes the training is easily disrupted due to an unsuitable KD loss setting.
Implementation Difficulty:
An elegant KD implementation requires users to be familiar with the code framework and introduces large design space. In contrast, augmented replay buffer introduces only a small set of extra source data and changes the training data sampled related to 6, which is considerably easier to implement across various fine-tuning scenarios.
Table 4: The FID results comparison of the KD varient. KD loss is trade-off by 0.05.
Methods
CUB-Bird
Standard Cars
Vanilla Fine-tuning
5.37
6.32
Diff-Tuning with augmented data
3.50
5.37
Diff-Tuning with KD
3.75
4.97
D
Discussion on Time-Reweighting Techniques
Table 5: Comparison with existing timestep weighting strategies.
Methods
CUB-Bird
Standard Cars
Vanilla Fine-tuning
5.37
6.32
P2
4.68
9.31
MIN-SNR-L
7.12
7.29
MIN-SNR-5
9.44
9.31
Diff-Tuning
3.50
5.37
Duff-Tuning+22
3.56
4.95
Bit-TTuiing-MN-SNN--
3.76
5.56
DiffTTunig--MIN-SNRR
5.84
6.50
As mentioned above, we ensure (t) + E(t) = in our Diff-Tuning, which maintains the overall loss in Eq. (5) with the default DDPM weight (uniform weight across timesteps). In our experimental implementation, we uniformly sample timesteps and then sample training data from either the memory buffer target dataset according to (t()) and {(t), respectively. Unlike [5). which tailor weights or develop separate models for each timestep. We do not alter the weights of individual timesteps in Diff-Tuning, which is significantly different from existing works.
To further underscore that the weighting strategies from [9] and [5] are orthogonal our Diff- -ffiinunig approach, we have implemented these strategies alongside our method. The results demonstrate that
18

---

while these existing methods can be applied to Diff-Tuning their integration shows varied impacts on performance.
These results in indicatee
indicate that while existing methods can be adapted to our Diff- Diff-Tuning, the performance varies, especially with MIN-SNR strategies, which may not align with the principles of the chain of forgetting, thereby potentially undermining the transfer learning efficacy.
E Fine-tuning EDM with Diff-Tuning
In the main expeirments, we have fine-tuned the DiT-XL-2 model (pre-trained using a VP-SDE approach ( and Stable Diffusion. We employed the same diffusion training strategy used for the pre-trained models. Additionally, we have evaluated EDM on publicly available repositories Since EDM incoporates continuous a instead of discrete / in the training state, we extend to use (b(or) (d) and {(o) = (o),, aligning our standard Diff- Diff-Tuning. Results are shown below:
Table 6: Results of fine-tuning EDM
Method
EDM: ImageNet 64x64 - CIFAR-10 64x64
Vanilla
14.75
Diff-Tuning
6.04
Sudden Convergence of Controlled Generation
Baselin DiT-Tunig
Baselre
Iie Dett-Tuiing
0.60
0.50
Difi-Tuingg
0.12
0.10
0.75
0.00
0.50
0.70
0.15-
0.02
3k
12k
15%
24
0
12x
15k
Training Scops
Training Soapa
Training Scops
(a) Edge
(b) Sketch
(c) Depth
0.10-
Bascine D-TTunin
Dark- Dark-Tunng
Basdine
0.6
DiKTTniing
Bussiie
6.
Ficel Accurstt
0.5
0.08
0.4
0.7.
0.55
3k
dc
13k
15c
3k
%6c Traiinn Scaps
12k
15c
8cc
12c
16k.
28k
Training Scops
Training Scops
(d) Normal
(e) Seg. (COCO)
(f) Seg. (ADE20K)
Figure 9: Validation metrics on each task. For Depth and Normal, lower indicates better, and conversely for other tasks.
Sudden convergence is phenomenon observed when tuning ControlNet [60] due to its specific zero-convolution design. As demonstrated in Figure 4 ControlNet does not gradually learn to adhere to control conditions. Instead, it abruptly begins to follow the input conditions. To identify simple signal indicating sudden convergence, we pre-collected a validation set of real images and compared the (dis-)similarity between their annotations and the corresponding generated controlled images. Figure / illustrates a noticeable "leap" during the training process, providing a clear indicator of sudden convergence. We established thresholds for quantitative evaluation based on test annotation
"hapss//gitubb.co////aaaeeee
19

---

similarity curves and combined them with qualitative human assessment to determine the number of steps as a metric for all tasks.
F.1 Quantitative Metries' Details
To efficiently and generally compare the (dis-)similarity between the original controlling conditions and the post-annotated conditions of the corresponding controlled generated samples, we employ the simplest reasonable metrics for each vision annotation, irrespective of task specificities such as label imbalance or semantic similarity. Specifically, we use Structural Similarity Index (SSIM) with Gaussian blurring for sparse classification (Edge and Sketch), mean-square error (MSE) for dense regression (Depth and Normal), and accuracy for dense classification (Segmentation). Detailed settings of the metrics and thresholds are provided in Table 7
Table 7: Detailed setting of quantitative metries for controlled generation tasks.
Edge
Sketch
Depth
Normal
Seg. (COCO)
Seg. (ADE20k)
Metric
SSIM w/ Blurring (t)
MSE (1)
Pixel Accuracy (t)
Threshold
0.55
0.75
0.04
0.06
0.5
0.4
ControlNet
6.7k
3.8k 3.2k
9.9k
10.3k
9.2k
13.9k
ControlNet+DifffTTnnii
5.3k
8.8k
7.8k
6.3k
8.3k
F.2
More Qualitative Analysis for Human Assessment
We present case studies validate the steps for convergence. By generating samples throughout the training process using a consistent random seed, we focus on identifying when and how these samples converge to their corresponding control conditions. As shown in Figure [00, our selected thresholds provide reasonable convergence discrimination across all six tasks, with Diff-Tuming consistently outperforming standard fine-tuning.
Similarities Alone Are Imperfect It is important to note that our proposed similarity metric serves only to indicate the occurrence of convergence and does not accurately reflect sample quality or the degree of control, especially for converged samples. For example, Figure [[ compares generated samples from standard fine-tuming and our Diff-Tuning approach with edge controlling. The generated samples from Diff-Tuning are richer in detail but may score lower on edge similarity metrics, highlighting the limitations of similarity metrics and underscoring the necessity of human assessment.
G Limitations and Future Works
Tiff-Tuning consistently outperforms standard fine-tuning and parameter-efficient methods, demon- strating its efficacy across various downstream datasets and tasks. This section also discusses some limitations of fiff-Tuning and explores potential future directions to address these limitations.
Necessity of Pre-sa Dataset Diff- Diff-Tuning involves constructing pre-sampled augmented dataset for knowledge reconsolidation, which requires additional computational resources. As discussed in Section 4.4 the impact of the sample size indicates that even the smallest set of generated images outperforms baseline methods and the original source data, underscoring the value this approach. Future work could focus on developing large-scale open-source generated samples and creating more sample-efficiem augmented datasets.
Extra Hyperparameters Diff Diff-Tuning introduces additional hyperparameters, {()) and (()., as loss weighting coefficients based on the principle of the chain of forgetting. These introduce extra hyperparameter spaces. As detailed in Section 4.4 our analysis shows that simple designs for these coefficients perform well and robustly. Future research could aim to design more effective weighting coefficients further enhance performance.
20

---

ControlNee
Sandard
"Lrgee Baoleyy
Dift Turingg
ControNNet
sy
bullding
Depth
4k
8.8k
9.9k
12.5k
15k
Standard ControINee
Three yourur wonsic are Anylig to catck fifiiae
Diff-Tuning ControlNet
Normal
3k
7.8k
10.3k
15k
Standard ControlNet
"Al wstnoll volk awshooadd
Dirk Tuning ControlNes
staraling oe beack."
Edge
2k
4k
5.3k
6k
6.7k
8k
10k
Standard ControlNet
Dirk Tunnnng ControlNer
Skerch
1.5k
3k
3.2k
3.5k
3.8k
4.2k
Sk
ControiNee ConpolNot
Scandard
profectional
Diff- Tunnnn
denatied Atge-
poaloy umsge?
Seg.
6k
7.8k
12k
13.9k
15k
Figure 10: Qualitative analysis. Red boxes refer to the occurrence of "sudden convergence"
ControlNet
Standard
(0.82)
wilk
Dift funing ControlNet
(0.76)
demected Algo- quattty
Figure 11: Generated images using standard ControlNet (SSIM 0.82) and Diff-Tuning ControlNet (SSIM of 0.76). Analyzing these cases, Diff-Tuning generates images with higher quality and more details hence results in a lower similarity, indicating the limitation of similarity metrics and the necessity of human assessment.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section of main text
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Please refer to Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The method design have been fully described in the main and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: Code is available at this repository:
http://giiihbbooommuuu///////fio We also provide the pseudo-code Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.crt cc/
public/guidess Sostess//ooSSSuubssssooooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ioweeeomiisiiiioooooo
//nips.c//pp wwwppbblccgguiicuceccccccoeooope
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.1 of main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All researches work around open-source data or models and no ethic issues are involved.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses mainly on the machine learning intrinsicities within views of transferrability, and no societal impacts are newly involved.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No new data or pretrained models are proposed.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators of datasets are properly credited by citations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "ciwkT3GEcb", "reviewer": "Reviewer_a4Zm", "summary": "This paper focuses on transfer learning methods for diffusion models. It experimentally demonstrates and provides theoretical insights into how the forgetting trend varies with the diffusion timestep. Based on this observation, they proposes Diff-Tuning. The proposed method introduces objectives for knowledge retention and reconsolidation, ensuring that they are affected decreasingly and increasingly by the diffusion timestep, respectively. The superiority of the proposed method is demonstrated experimentally.", "questions": "Please see the Weaknesses part.", "limitations": "They provided in the Supplementary section D.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, and the method is well-motivated both experimentally and theoretically.", "weaknesses": "* More discussion on how to set the hyperparameters $\\xi(t)$ and $\\psi(t)$ would be helpful. For example, these parameters perform an weighted sum of the retention and reconsolidation objectives, and combining this with a maximum likelihood analysis could potentially help find hyperparameter selection.\n\n* I am curious if applying the datasets used for pre-training to retention would be better (or worse) than using a pre-sampled dataset and the reason.\n\n* It would be beneficial to discuss how the method can be extended in scenarios where transfer learning is applied multiple times or where the data distribution changes online.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the careful review and insightful suggestions. Our responses to the concerns raised are outlined below:\n\n---\n> **W1: More discussion on how to set the hyperparameter $\\xi(t)$ and $\\psi(t)$.**\n\nThank you for highlighting the importance of hyperparameter selection for $\\xi(t)$ and $\\psi(t)$. To clarify, **as detailed in Lines 194-196, we set $\\xi(t) + \\psi(t) = 1$ to ensure uniform weighting across timesteps.** This design simplifies implementation and emphasizes the trade-off between retention and adaptation, without the complexity of a detailed scheduling mechanism. Our approach diverges from conventional MLE-based weighting schedules but can be adapted to include them by modifying $\\xi(t)$ and $\\psi(t)$ to align with a predefined weight function $w(t)= \\frac{\\beta_t}{(1-\\beta_t)(1-\\alpha_t)}$ as discussed in DDPM [1].\n\n- **Robustness of $\\xi(t)$ and $\\psi(t)$:** As discussed in** Section 4.4 Lines 314-321 and shown in Figure 6(c)**, our results indicate that Diff-Tuning is robust to variations in the design of $\\xi(t)$ and $\\psi(t)$. This robustness highlights the primary importance of managing the chain of forgetting tendency over specific scheduling details. Our choice, linear to $t$, while simple, proves effective in achieving the desired behavior.\n- **Difference with (MLE-based) weighting schedules:** We recognize that we missed an emphasis in the method details and cause confusion. We here clarify that, as stated **in Lines 194-196, we set $\\xi(t) + \\psi(t) = 1$ to maintain a consistent weight across all timesteps**, thereby the weighting strategy of $\\xi(t)$ and $\\psi(t)$ focuses on the trade-off between retention and adaptation, instead of the reweighting across all timestep $t$.\n- **Applicability for any (MLE-based) weighting schedules:** We agree that a maximum likelihood analysis or other sophisticated weighting schedules could potentially aid in the optimal selection of hyperparameters, and present how our Diff-Tuning strategy can be potentially combined with (MLE-based) weighting schedules: Suppose a weighting schedule is defined by the weight function $w(t)$ on the timestep $t$ (such as the MLE-based weighting schedule $w(t)= \\frac{\\beta_t}{(1-\\beta_t)(1-\\alpha_t)}$, referring to the original derivation in DDPM [1]), then one can set\n$$\n\\xi’(t)=w(t)\\xi(t),\\quad \\psi’(t)=w(t)\\psi(t),\n$$\nand replace $\\xi(t),\\psi(t)$ in Eq.(3)(4) with $\\xi’(t),\\psi’(t)$ to satisfy $\\xi’(t)+\\psi’(t)=w(t)$, i.e., in usage of the corresponding weighting schedule. For simplicity, we have retained the most widely used weighting of $w(t)=1$ and do not alter the weight or the distribution of $t$ for fair comparison.\n\n> **W2: Whether using the datasets applied for pre-training during retention would be better (or worse).**\n\nThe primary reason we do not use the pre-training data is the typical unavailability in most fine-tuning scenarios. However, using the original pre-training data is indeed feasible with Diff-Tuning. **As discussed in Section 4.4 Lines 325-328 and shown in Figure 6(d), using the entire pre-training dataset may not yield optimal results.** We believe this is because the fine-tuning stage usually involves fewer training steps and utilizes only a small subset of the pre-training data (about 1/3 epoch here), whereas the samples generated from a learned model can be more representative than sampling from a large empirical dataset.\n\n>  **W3: how the method can be extended to scenarios where transfer learning is applied multiple times or where the data distribution changes online.**\n\nThe scenario you mentioned aligns with the setting known as continual learning, which is a meaningful extension of Diff-Tuning. It is feasible to extend the replay buffer to collect generated data (or a subset of training data) as the data distribution evolves over time. We conducted experiments on the continual learning dataset Evolving Image Search [2], which consists of images with 10 categories collected in recent years, and splitted into 3 folds: 2009-2012, 2013-2016, and 2017-2020. We perform transfer learning sequentially across these splits, and calculate the FID value in the the accumulated test set. For Diff-Tuning, we maintain $60\\%$ of the pre-sampled augmented data and collect other $40\\%$ from the fine-tuned model before fine-tuning to the next split. The experimental results are shown below:\n\n**Table B: Continual learning with multiple fine-tuning.**\n| Evolving Dataset    | 2009-2012 (FID) | 2009-2016 (FID) | 2009-2020 (FID) |\n| :------------------ | :-------------: | :-------------: | :-------------: |\n| Vanilla Fine-tuning |      12.71      |     12.04       |    13.06        |\n| Diff-Tuning         |      10.31      |     10.29       |     10.72       |\n\nFrom the results, is observed that Diff-Tuning demonstrated significant improvements, underscoring the importance of addressing the forgetting phenomenon in continual learning scenarios. \n\n---\n\n**References**\n\n[1] Denoising Diffusion Probabilistic Models, Neurips 2020. \\\n[2] Active Gradual Domain Adaptation: Dataset and Approach, IEEE Transactions on Multimedia 2022.\n\n---\n\nWe hope that our additional clarifications and discussion address your questions and concerns. Please let us know if you have any further concerns!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WwvcpjWUbY", "reviewer": "Reviewer_WW5v", "summary": "The paper proposes a new method to fine-tune the pretrained large-scale diffusion models for new tasks. It finds that different time steps of the denoising process of diffusion models have varying transferability. Specifically, the paper finds that \"low-noise\" time steps close to the end of the denoising process have better transferability. In contrast, the \"high-noise\" steps responsible for generating the image layout and containing domain knowledge are less transferable. Based on these observations, the paper proposes a novel fine-tuning objective demonstrating better convergence speed than baselines.", "questions": "I have the following questions from the paper:\n\n1. When fine-tuning the ImageNet pretrained model on the new task, do you train the class embeddings (used for cross-attention) for the new task as well? \n\n2. Regarding points 1 and 2 in the Weaknesses section, as far as I know, training on images generated by the generative models may degrade the model's performance [1]. Yet, using the whole ImageNet dataset performs worse than using 5k images that are sampled from the pretrained model. Is there any intuition about why this happens?\n\n[1] Self-consuming generative models go mad, ICLR 2024.", "limitations": "The authors have pointed to using an augmented dataset as a potential limitation, but as I mentioned in the weakness section, the motivation and intuition of why it should work is not clear.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple without numerous hyperparameters or tricks to work. \n\n- The proposed idea is orthogonal to the main focus of the recent papers that aim to do parameter-efficient fine-tuning, and it can be combined with them as the paper demonstrates in the experiments.", "weaknesses": "1. I could not follow the motivation behind the scheme that the paper claims to perform Knowledge Retention with. The main argument of the paper is that \"the time steps of the denoising process close to the end of the sampling chain of diffusion models are more transferable.\" Then, why do we need to sample an augmented dataset to do the Knowledge Retention? A straightforward way can be using knowledge distillation from the original model to the fine-tuned one in the low noise time-steps. \n\n2. Following the previous point, distillation may be better than an augmented dataset considering that the fine-tuning data may have characteristics that are different from the original domain. For instance, (this is just a hypothetical example) the fine-tuning dataset may be on flowers that are mostly red, but the pretaining dataset (ImageNet in the experiments) may not have red flowers. Therefore, for the fine-tuning process, we care more about red flowers, not other types. However, using an augmented dataset can bias the fine-tuning process and lower the model's quality on red flowers. In contrast, distillation on the low-noise time steps may readily preserve the prior knowledge of the model without biasing the fine-tuning process. \n\n3. I appreciate the analysis in Fig. 1, and the formal experimentation of it is a novelty. Yet, some similar phenomena have been shown in the literature before that the paper does not discuss them. [1] shows that a diffusion model usually generates a high-level layout of the image until SNR gets between [0.01, 1], and then, starts to fill in the details. Similarly, eDiff-I shows that changing the text prompt at the low-noise stages of the denoising process process does not change the output image. These experiments in Fig. 1 also show that if we use the original model in the low-noise time steps, the performance does not decrease, which is similar to these papers. I think the paper should include these references in the related work section. \n\n[1] Perception Prioritized Training of Diffusion Models, CVPR 2022. \n\n[2] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the careful review and insightful suggestions provided by the reviewer. Our responses to the concerns raised are detailed below:\n\n---\n> **W1&2: The motivation of knowledge retention and a potential varient of Diff-Tuning with knowledge distillation**\n\nThank you for your insightful comments on the motivation behind knowledge retention and the concerns regarding knowledge distillation (KD). Firstly, it is crucial to emphasize that a novel and critical contribution of our work is the unveiling of the chain of forgetting tendencies, both experimentally and theoretically,  **the theoretical analysis presented in Theorem 1 (Lines 151-162) is never discussed before.** The chain of forgetting tendencies provides a guideline for designing transfer methods.\n\nUtilizing an augmented data buffer in a generative model has been a rational practice since [1], and **KD is a viable implementation only if it adheres to the principle of chain of forgetting**. Both techniques should be effective within the framework of Diff-Tuning. Below, we summarize reasons why KD is not our initial choice:\n\n- **GPU Memory**: KD requires maintaining a copy of the pre-trained model alongside the fine-tuning model. For large models, this significantly increases memory costs. For example, we can run Diff-Tuning with a batch size of 32 on a single A100-40GB GPU, whereas the KD variant decreases to a batch size of 24.\n- **Computational Cost**: KD doubles the forward computation cost by necessitating the matching of output distributions between two models. Notably, pre-computing the KD labels is not feasible due to the inherent noise in diffusion training. For instance, for a batch size of 24, we achieve 2.1 training steps per second, compared to 1.34 for the KD variant.\n- **Training Instability**: Transferring a pre-trained model to a domain significantly different from its training data can introduce out-of-distribution corruption during distillation, potentially causing instability in the fine-tuning process. We invested considerable effort to tune a suitable trade-off for the KD loss (0.05), and sometimes the training is easily disrupted due to an unsuitable KD loss setting.\n- **Implementation difficulty**: An elegant KD implementation requires users to be familiar with the code framework and introduces a large design space. In contrast, an augmented replay buffer introduces only a small set of extra source data and changes the training data sampled related to $t$, which is considerably easier to implement across various fine-tuning scenarios.\n\nConsidering scenarios where KD might be preferred over an augmented data buffer, we have also implemented a new variant of knowledge retention incorporating KD. Due to time constraints, we have tuned a preliminary implementation and present the KD results below for a quick overview during the rebuttal phase, and we will revise the methodology section to include this discussion accordingly.\n\nIn light of potential settings to prefer KD over an augmented data buffer, we also implement a new varient of knowledge retention incorporating KD. Due to time constraints, we have tuned a preliminary implementation and present the KD results below for a quick overview during the rebuttal phase, and we will revise the methodology section to include this discussion accordingly.\n\n**Table C: The FID results comparison of the  KD varient. (KD loss is trade-off by 0.05)**\n|             Methods             | cub-200 | car  |\n| :-----------------------------: | :-----: | :--: |\n|       Vanilla Fine-tuning       |  5.32   | 6.05 |\n| Diff-Tuning with augmented data |  3.50   | 5.37 |\n|       Diff-Tuning with KD       |  3.75   | 4.97 |\n\n\n> **W3: Similar phenomena noted in previous literature [2,3].**\n\nSee Common Concern #1 in Author Rebuttal.\n\n\n> **Q1: Does Diff-Tuning train the class embeddings?**\n\nYes, we fine-tune the class embeddings in Diff-Tuning using the same methods as in the pre-training stage. To further note, we observed that either reinitializing the class embeddings or directly updating existing ones shows similar performance.\n\n> **Q2: Why using the whole ImageNet dataset performs worse than 5k Images and the intuition behind.**\n\nThe slight difference in performance between using the entire ImageNet dataset and 5k augmented images (5.54 vs. 5.52, with vanilla fine-tuning at 6.04) is not statistically significant. As discussed in Section 4.4, one potential explanation is the size of the dataset relative to the fine-tuning duration. Given that fine-tuning only requires 24k steps, a large source dataset is underutilized, where only a fraction of images are iterated (about 1/3 of an epoch). In this context, data sampled from a learned model acts as a form of knowledge distillation, representing the original distribution more effectively than direct sampling from a vast dataset.\n\n[4] explores how synthetic training loops affect generative model performance when the real dataset is fixed. However, this does not directly apply to fine-tuning diffusion models to new domains with new data.\n\nWe sincerely thank you for your insightful comments. We will expand the discussion in Section 4.4 in the revised paper accordingly.\n\n---\n**References**\\\n[1] Transferring GANs: generating images from limited data ECCV 2018.\\\n[2] Perception Prioritized Training of Diffusion Models, CVPR 2022.\\\n[3] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023. \\\n[4] Self-consuming generative models go mad, ICLR 2024.\n\n----\n\nWe hope that our responses adequately address your queries and clarify our methodologies. We welcome any further questions or feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3fLODP9P8", "reviewer": "Reviewer_y5ux", "summary": "This paper explores the fundamental transfer characteristics of diffusion models and observes the monotonous chain of forgetting trend of transferability of diffusion models in the reverse process. It then proposes a simple but effective transfer approach to make the fine-tuned model retain the denoising ability of the pre-trained model close to the generated image, while using domain-specific denoising ability at the beginning of the denoising process. Experimentally, Diff-Tuning can achieve effective improvement on some standard fine-tuning tasks, and also improve the convergence speed of ControlNet.", "questions": "1. the authors use a pre-trained diffusion model to preserve denoising ability, would it be useful to use pre-trained domains, i.e. images from ImageNet, instead of augmented images? Although I know that in most cases we can't fully know the pre-trained diffusion model's pre-training data\n2. when using a pre-trained diffusion model to generate augmented data, do different sampling methods achieve similar results, and does the choice of sampling steps make a difference?\n3. would using a pre-trained diffusion model on ImageNet and then fine-tuning it on small data achieve better results than a diffusion model trained directly on small data? For example, pre-training on ImageNet and then fine-tuning on CIFAR10/100.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Diff-Tuning is simple and easy to follow, and experimentally well realized.\n2. the motivation is clear and it explores the transfer capability of the most popular diffusion model, which is novel.\n3. this article should make me deepen my understanding of the diffusion model and notice its interesting phenomena in the reverse process.", "weaknesses": "1. There are too few metrics on class-conditional generation, while FID is used to extract features with models trained on top of ImageNet, authors use DiT pre-trained on ImageNet to generate samples, which are then added to the training process, it is difficult to say the lower FID actually translates to improved image quality. can you use more metric to show your results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have taken to review our manuscript and for your constructive feedback. We will address the concerns you raised and revise the paper accordingly, as your comments provide valuable insights for improving our work.\n\n\n---\n> **W1: Use more metric to show the results of class-conditional generation.**\n\nWhile FID is based on features from models trained on ImageNet, the calculation between the downstream dataset and generated data does not directly reflect any bias from ImageNet training data. In fact, intuitively, using ImageNet might even be perceived as potentially detrimental to the FID value.\n\nDespite FID's widespread use as a metric for evaluating modern generative models, we acknowledge the need for a broader set of evaluations. We agree that additional metrics, such as sFID [1], Precision, and Recall [2], provide a more comprehensive comparison. During the rebuttal phase, we have evaluated Diff-Tuning with these metrics, and the results are as follows:\n\n**Table D: More class-conditional generation metrics evaluated on Stanford Car**\n| Method | FID $\\downarrow$ | sFID $\\downarrow$ | Precision $\\uparrow$ | Recall $\\uparrow$  |\n| :-----------| :------: | :-----: | :--: | :--: |\n| Vanilla     | 6.04 | 9.45 | 0.5852 | 0.5873 |\n| Diff-Tuning | 5.37 | 5.43 | 0.6062 | 0.5723 |\n\nThese results demonstrate that Diff-Tuning enhances the quality of generation with comparable precision and recall. We also provide showcases under the same random seeds. Please refer to Figure 1 in the attached PDF file.\n\n> **Q1: Would it be useful to use pre-trained domains?**\n\nOn the one-hand, using the original pre-training data is indeed applicable to Diff-Tuning. By comparing **Table 1 and Figure 6(d), using the entire pre-training dataset outperforms vanilla fine-tuning, showing the validity of using pre-trained domains.** On the other hand, note that using pre-training dataset may not yield optimal results compared with using generated samples as illustrated in Figure 6(d). We believe this is because that, as discussed in Section 4.4, the fine-tuning stage typically involves fewer training steps and utilizes only a small subset of the pre-training data, whereas the samples generated from a learned model can be more representative than sampling from a large empirical dataset. Hence using pre-trained data, even with accessibility, is sub-optimal.\n\n> **Q2: Do augmented data generated by different sampling methods achieve similar results**\n\nThis is an important concern regarding the robustness of Diff-Tuning to different augmented datasets. In our original experiments, the augmented data are generated using the default evaluation protocol provided by the pre-trained model (e.g., the pre-trained DiT-XL-2 was evaluated using DDIM with 250 steps and a CFG of 1.5, so we maintained the same strategy). To further investigate the impact of different sampling methods, we analyze the results with 50 and 500 DDIM steps. These results, in conjunction with Section 4.4 and Figure 6(d), should address your concerns about the impact of different sampling methods on performance.\n\n**Table E: Analysis on the different sampling steps of augmented data**\n| Dataset             | cub-200 | car  |\n| :------------------ | :-----: | :--: |\n| 50 steps            |  3.53   |  5.38|\n| 250 steps (default) |  3.50   |  5.37|\n| 500 steps           |  3.44   |  5.23|\n\n\n> **Q3: Would fine-tuning a pre-trained model achieves better results than training from scratch on small dataset**\n\nThe pre-training and fine-tuning paradigm generally outperforms training from scratch, especially for small datasets. Fine-tuning offers numerous advantages, including higher-quality results, lower computational costs, reduced training data requirements, and more efficient training processes. \n\n---\n\n**References**\\\n[1] Generating Images with Sparse Representations, ICML 2021\\\n[2] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019\n\n---\n\nWe hope that our additional clarifications and discussion address your questions and concerns. Please let us know if you have any further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vUztlENRVt", "reviewer": "Reviewer_x2jk", "summary": "This paper proposes Diff-Tuning method to encourage the fine-tuned model to retain the pre-trained knowledge. In this method, both pre-trained data and downstream data are used to train the diffusion model. Compared to standard fine-tuning methods, Diff-Tuning enhances the convergence speed and improves the performance. Diff-Tuning can also be used in Conditional Generation. Additionally, Diff-Tuning is compatible with existing parameter-efficient fine-tuning methods.", "questions": "Please see Weakness. If my question is answered, I will raise my score.", "limitations": "Please see Weakness. If my question is answered, I will raise my score.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The idea of using the pre-trained model to act as a universal denoiser for lightly corrupted data is interesting.\n+ Diff-tuning achieved faster training and better performance than standard fine-tuning.\n+ This paper provides novel theoretical insights to reveal the principle behind the chain of forgetting. Therefore, this paper is solid.", "weaknesses": "+ The novelty of this paper is limited. Many previous works [1,2] find that gradients exist in conflict for diffusion models across timesteps even in the same dataset. But none of them are compared.  **Since the core conclusions of this paper are very similar to them, I think the author needs to highlight the differences.**\n1. Efficient Diffusion Training via Min-SNR Weighting Strategy ICCV 2023\n2. Addressing Negative Transfer in Diffusion Models. NeurIPS 2023\n+ The robustness of Diff-Tuning to sampling algorithms has not been verified, including DDIM [1], DPM-Solver [2], and others. Meanwhile, More diffusion models need to be validated, including VP-SDE [3], Flow Matching [4], EDM [5], etc.\n1. Denoising Diffusion Implicit Models ICLR 2021\n2. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps Neurips 2022 O\n3. Score-Based Generative Modeling through Stochastic Differential Equations ICLR2021\n4.  Flow Matching for Generative Modeling ICLR 2023\n5. Elucidating the Design Space of Diffusion-Based Generative Models NeurIPS 2022\n\n+ In the training of the diffusion model, there are different time schedules (Linear, Cosine, etc.), so do I need to keep the same schedule in the Diff-Tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n> **W1: Concern about the novelty and difference from [1,2].**\n\nThank you for your insightful thoughts regarding the distinction from existing works. We recognize there may have been some misunderstanding due to the weighted forms in Eq(3) and Eq(4). Our work is distinctively focused on the transferability of pre-trained diffusion models which is significantly different from the approaches taken in [1,2], both in analytical and technical aspects:\n- **Differences in Analysis**: Our work discusses model transferrability from **both theoretical and empirical analysis**, **as detailed in Section 3.1**, which is **never** explored in previous works. [1,2], though also concentrate on varying timesteps $t$, focus on general learning of diffusion models and address the issues of gradient conflicts in multi-task learning.\n- **Differences in Technique Details**: Technically, our proposed Diff-Tuning aims to trade-off **not varying timesteps $t$ but the retention and adaptation**. As stated in Line 194-196, we ensure $\\psi(t) + \\xi(t) = 1$ in our Diff-Tuning, which **maintains the overall loss in Eq(5) with the default DDPM weight (uniform weight across timesteps)**. And in our experimental implementation, we uniformly sample timesteps and then sample training data from either the memory buffer or target dataset according to $\\psi(t)$ and $\\xi(t)$, respectively. This contrasts sharply with [1,2], which assign carefully designed weights to different diffusion timesteps $t$ to manage gradient conflicts. From the gradient conflict perspective, we do not alter the weights of individual timesteps; the improvements arise from the transfer preferences we introduce.\n\nAdditionally, we present supplementary experiments combining Diff-Tuning with MIN-SNR-$lambda$ ($\\lambda = 1,5$) [1].\n\n**Table A': Comparison with MIN-SNR**\n|Methods|cub-200|car|\n|:--:|:--:|:--:|\n|vanilla Fine-tuning|5.32|6.04|\n|MIN-SNR-1 [1]|7.12|7.29|\n|MIN-SNR-5 [1]|9.44|9.31|\n|Diff-Tuning|3.50|5.37|\n|Diff-Tuning+MIN-SNR-1 [1]|3.76|5.56|\n|Diff-Tuning+MIN-SNR-5 [1]|5.84|6.50|\n\nThe concerns raised by Reviewer WW5v discuss other existing works, we hope Common Concern #1 in Author rebuttal also helps to address your concerns. \n\n***From these results, we observe that MIN-SNR [1] does not perform well in transfer learning, since MIN-SNR lacks consideration on transferrability, and its weigting strategy is completely incompatible to the chain of forgetting principle.***\n\n---\n>**W2: Robustness to sampling algorithms and additional diffusion models**\n\nWe agree that evaluating our method with more sampling algorithms and diffusion models would strengthen our findings. To point out, our selected pre-trained models, DiT [5] and Stable Diffusion [6], are the **largest and most representative foundation models** in the diffusion model family, which **ensures the effectiveness** of our methods. Meanwhile, we always follow their default sampling strategies to **ensure the validity** of our experiments. As complement, we further conduct the following experiments to demonstrate the robustness of our method:\n\n- In the original class-conditional experiments, the main results were obtained using 50 uniform DDIM[3] steps. We have extended these experiments to include 100, 250, and 500 DDIM steps, as well as DPM-solver[4]. The results are summarized below:\n\n**Table F: Analysis on the different sampling algorithms**\n| Sampling Algorithm | DDIM-25 | DDIM-50 (default)  | DDIM-100 | DDIM-500 | DPM-solver-20* |\n| :----------------- | :-----: | :------: | :------: | :------: | :-----------: |\n| food-101 (vanilla) |21.4|10.68|5.75|5.52|36.19|\n| food-101 (ours)    |12.28|6.05|4.91|3.86|26.81|\n| cub-200 (vanilla)  |9.86|5.32|4.32|3.71|23.47|\n| cub-200 (ours)     |4.25|3.50|3.41|3.33|14.66|\n\n*Due to the time limit, we do not carefully tune the hyperparameters.\n\n- In the original manuscript, we have fine-tuned the DiT-XL-2 model (pre-trained using a VP-SDE approach [7]) and Stable Diffusion. We employed the same diffusion training strategy used for the pre-trained models. Additionally, we have evaluated EDM on publicly available repositories [9]. Since EDM incoporates continuous $\\sigma$ instead of discrete $t$ in the training state, we extend to use $\\psi(\\sigma)=\\text{cdf}(\\sigma)$ and $\\xi(\\sigma)=1-\\psi(\\sigma)$, aligning our standard Diff-Tuning. Results are shown below:\n\n**Table G: Results of fine-tuning EDM**\n|Method|EDM:ImageNet 64x64$\\rightarrow$CIFAR-10 64x64|\n|:---------:|:---------------------------------------------------:|\n|Vanilla|14.75|\n|Diff-Tuning|6.04|\n\n---\n>**W3: Need to keep the same schedule in the Diff-Tuning?**\n\nDiff-Tuning does not assume any specific scheduling strategy. However, when fine-tuning a model from a pre-trained checkpoint, it is generally preferable to maintain the same schedule as the pre-trained model to avoid potential mismatches. In our experiments, we adhered to this principle by fine-tuning DiT with a linear schedule and Stable Diffusion with its default schedule.\n\n\n---\n**References** \n\n[1] Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023. \\\n[2] Addressing Negative Transfer in Diffusion Models. NeurIPS 2023.\\\n[3] Denoising Diffusion Implicit Models, ICLR 2021\\\n[4] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Neurips 2022\\\n[5] Scalable diffusion models with transformers, ICCV 2023 \\\n[6] Stability. Stable diffusion v1.5 model card, https://huggingface.co/runwayml/stable-diffusion-v1-5, 2022. \\\n[7] Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021\\\n[8] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022 \\\n[9] EDM repo URL: https://github.com/NVlabs/edm\n\n---\n\nWe hope that our additional clarifications and discussion address your questions and concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> **Common Concern #1 (raised by WW5v and x2jk): The comparison to existing literatures [1-4]**

Reviewer WW5v concerns about some similar phenomena have been shown in the existing literatures [1,2]. and Revewer x2jk also points out that [3,4] find gradients conflict across timesteps even in the same dataset. We acknowledge the importance of this issue and will expand the discussion in the related works section of our revised manuscript to highlight our unique contributions further.

It appears there has been a misunderstanding, as the studies cited only superficially resemble our findings. Here are the significant distinctions:

- [1] empirically determines that diffusion models construct a high-level perceptual layout within the SNR range [0.01, 1], and designs a loss weighting strategy based on this range. While [1] focuses on enhancing the efficiency of training diffusion models, it overlooks aspects of transfer learning, which is central to our research.
- [2] finds that diffusion models increasingly rely on the text prompt at higher noise levels. [2] then improves text-conditional generation by deploying a suite of denoisers, each tailored to specific noise levels. However, this approach does not incorporate transfer learning considerations.
- [3,4] focus on addressing gradient conflicts across timesteps $t$ in a general learning context, particularly in the multi-task learning perspective. Their findings and methodologies are **orthogonal** to Diff-Tuning.
- **Our work discusses model transferability from both theoretical and empirical analysis (Section 3.1, Lines 151-162), which has never been explored in previous works.**
- **Technically, our proposed Diff-Tuning aims to trade-off the retention and adaptation rather than strategically weighting varying timesteps $t$**. As stated in Line 194-196, we ensure $\psi(t) + \xi(t) = 1$ in our Diff-Tuning, which maintains the overall loss in Eq(5) with the default DDPM weight (uniform weight across timesteps). In our experimental implementation, we uniformly sample timesteps and then sample training data from either the memory buffer or target dataset according to $\psi(t)$ and $\xi(t)$, respectively. Unlike [1-4], which tailor weights or develop separate models for each timestep. We do not alter the weights of individual timesteps in Diff-Tuning, which is significantly different from existing works.

To further underscore that the weighting strategies from [1] and [3] are orthogonal to our Diff-Tuning approach, we have implemented these strategies alongside our method. The results demonstrate that while these existing methods can be applied to Diff-Tuning, their integration shows varied impacts on performance.

**Table A: Comparison with Existing Timestep Weighting Strategies**
|Methods|cub-200|car |
|:----------------------:|:-----:|:--:|
|Vanilla Fine-tuning|5.32|6.04|
|P2 [1]|4.68|9.31|
|MIN-SNR-1[3]|7.12| 7.29 |
|MIN-SNR-5[3]|9.44| 9.31 |
|**Diff-Tuning**|**3.50**|5.37|
|*Diff-Tuning+P2*[1]|3.56|**4.95** |
|*Diff-Tuning+MIN-SNR-1*[3]|3.76| 5.56 |
|*Diff-Tuning+MIN-SNR-5*[3]|5.84| 6.50 |

These results indicate that while existing methods can be adapted to our Diff-Tuning, the performance varies, especially with MIN-SNR strategies, which may not align with the principles of the chain of forgetting, thereby potentially undermining the transfer learning efficacy.

> **Explanation to the Attached *PDF* File**

In the attached pdf file, we organize all the tables and figures of experiment results and showcases during the whole rebuttal period as a supplement for the reviewers. Here we offer a concise explanation of all the concents.

- **[Table A] (from Reviewer WW5v and x2jk)**: We compare Diff-Tuning with loss weighting strategies, P2 [1] and MIN-SNR [3].
- **[Table B] (from Reviewer a4Zm)**: We extend Diff-Tuning to continual learning scenarios with the Evolving Image Search dataset [5]. We split the dataset into 3 folds by the year of images. We sequentially fine-tune on the 3 folds, and calculate the FID with the accumulated dataset. For Diff-Tuning, we maintain a replay buffer for the augmented data.
- **[Table C] (from Reviewer WW5v)**: We incoporate KD to replace the reconsolidation loss within our Diff-Tuning method. The trade-off for the KD loss is set to 0.05 to avoid instability.
- **[Table D] (from Reviewer y5ux)**: We evaluate with more metrics including sFID [6], Precision and Recall [7] for class-conditional generation tasks.
- **[Table E] (from Reviewer y5ux)**: We use different sampling steps to generate the augmented data.
- **[Table F] (from Reviewer x2jk)**: We use different sampling algorithm (DDIM [8] and DPM-Solver [9]) and steps to sample from the fine-tuned models.
- **[Table G] (from Reviewer x2jk)**: We evaluate EDM [10] pre-trained on ImageNet 64x64 by fine-tuning on CIFAR-10 64x64. For Diff-Tuning, we use $\psi(\sigma)=\mathop{\mathrm{cdf}}(\sigma)$ and $\xi(\sigma)=1-\psi(\sigma)$ to fit continuous $\sigma$ instead of discrete $t$.
- **[Figure A]**: We provide showcases on Standford Cars.


**References**

[1] Perception Prioritized Training of Diffusion Models, CVPR 2022.\
[2] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023. \
[3] Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023. \
[4] Addressing Negative Transfer in Diffusion Models. NeurIPS 2023. \
[5] Active Gradual Domain Adaptation: Dataset and Approach, IEEE Transactions on Multimedia 2022. \
[6] Generating Images with Sparse Representations, ICML 2021. \
[7] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019. \
[8] Denoising Diffusion Implicit Models, ICLR 2021. \
[9] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Neurips 2022. \
[10] EDM repo URL: https://github.com/NVlabs/edm.

Meta-review: This paper introduces Diffusion Tuning to transfer learning for diffusion models. By leveraging a new concept termed the "chain of forgetting," the paper proposes an approach to encourage pre-trained diffusion models to retain essential knowledge while discarding noise-related information during fine-tuning. Experiments demonstrate a 26% improvement over conventional fine-tuning across multiple tasks and accelerates ControlNet's convergence by 24%.

The reviewers identify several strengths, including the novelty, clarity and presentation, and experimental results. There are also concerns by some reviewers, including theoretical clarifications and comparisons with existing literature on timestep weighting strategies, comparing with knowledge distillation, and lack of experiment verification to validate the robustness of the method. All the concerns have been discussed and addressed by the authors thorough rebuttals, and the reviewers are positive about the rebuttal.

Given the conceptual novelty, the theoretical foundations, and comprehensive experimental validation, I recommend the paper for acceptance.
Decision: Accept (poster)