Transfer Learning for Diffusion Models
Authors: Yidong Ouyang
Liyan Xie
Hongyuan Zha
Guang Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Transfer Learning
Primary area: diffusion_based_models

Abstract: Diffusion models, a specific type of generative model, have achieved unprecedented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently, various finetuning and regularization approaches have been proposed to transfer knowledge from existing pre-trained models to specific target domains with limited data. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinct from conventional finetuning and regularization methods. 
We prove that the optimal diffusion model for the target domain integrates pre-trained diffusion models on the source domain with additional guidance from a domain classifier. 
We further extend TGDP to a conditional version for modeling the joint distribution of data and its corresponding labels, together with two additional regularization terms to enhance the model performance. We validate the effectiveness of TGDP on both simulated and real-world datasets.

Transfer Learning for Diffusion Models
Yidong Ouyang', Liyan Xie?; Hongyuan Zha?, Guang Cheng'
'Dpparmment of Statistics and Data Science, University of California, Los Angeles
*Department of Industrial and Systems Engineering, University of Minnesota Twin Cities
Schooo Data Science, Chinese University of Hong Kong, Shenzhen yidongouyangeg ucla. edu; liyanxie@umn, edu; zhahy@cuhk, edu. guangchengéucla.com edu
Abstract
Diffusion models, specific type of generative model, have achieved unprece- dented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently, various finetuming and regularization approaches have been proposed to transfer knowledge from existing pre-trained models specific target domains with limited data. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinet from conventional finetuning and regularization methods. We prove that the optimal diffusion model for the target domain integrates pre- pre-trained diffusion models on the source domain with additional guidance from domain classifier. We further extend TGDP to a conditional version for modeling the joint distribution of data and its corresponding labels, together with two additional regularization terms to enhance the model performance. We validate the effectiveness of TGDP both simulated and real-world datasets.
Introduction
Diffusion models have achieved remarkable success in modeling data distributions and generating
various types of synthetic data, such as images [13] 36 I7]. videos [14, vision language [32] 33 30), and time series (39]. However, their success heavily relies the availability of large number of training samples. In real-world applications, acquiring ample samples for specific tasks can be challenging due the high costs associated with data collection or labeling, the potential risks involved. Therefore, an important research question is how to effectively transfer knowledge from a pre-trained generative model the source domain (using existing large-scale datasets) target domain (for specific tasks) where data is limited.
Training generative model directly finetuming a pre-trained generative model on limited data from the target domain often results in significant performance degradation due to overfitting and memorization. To address these issues, numerous studies have proposed methods in generative domain
adaptation, including the GAN-based models [46 45 49
I
51 28 48
15
43
22
50), diffusion-
be divided into two categories: finetuning lightweight adapters 25 and finetuning with regularization (52]. Approaches involving finetuning lightweight adapters focus adjusting only subset of parameters in a pre-trained model. The primary challenge here is identifying which parameters to finetune. This process is typically heuristic and requires preliminary experiments identify the most efficient parameters for adjustment. Additionally, the specific parameters to be finetuned can vary across
based model [25] 52 44 etc. Specifically, approaches using diffusion models
"Conrespondence to: Liyan Xie, liyanxie@umm edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

different neural network architectures. On the other hand, the challenge in incorporating regularization during the finetuning process is the heuristic design of the regularization term, which can significantly alter the optimization landscape. We refer to Appendix A for more detailed discussion of existing literature.
In this work, we introduce a new approach, termed Transfer Guided Diffusion Process (TGDP), to transfer knowledge in the source domain generative model to the target domain with limited samples. Unlike finetuning-based methods that primarily use the pre-trained model as an initialization point, TGDP leverages the pre-trained model as a plug-and -ud--aay prior. We show that the score function for the diffusion model on the target domain is the score function on the source domain (which can be pre- trained) with additional guidance as shown by Theorem 3. and Theorem 3.3) The guidance network is related to the density ratio of the target and source domain data distributions. Consequently, we convert the original optimization problem for a diffusion model on the target domain into estimating the density ratio.
We utilize a domain classifier (binary classifier) along with samples from both domains efficiently estimate the density ratio. Furthermore, we introduce two additional regularization terms for better training and calibration of the guidance network. These regularization terms are equivalent forms that the optimal guidance network should satisfy, ensuring they do not alter the original optimization problem. We validate the effectiveness of our approach through experiments on Gaussian mixture simulations and real electrocardiogram (ECG) data. Under both fidelity and utility evaluation criteria, TGDP consistently outperforms finetuning- based methods.
Our contributions can be summarized as follows.
We introduce new framework, the Transfer Guided Diffusion Process (TGDP), for transferring a pre-trained diffusion model from the source domain to the target domain.
We extend TGDP conditional version for modeling the joint distribution of data and its corresponding labels, along with two additional regularization terms, which are important for practical applications and downstream tasks.
TGDP demonstrates superior performance over finetuning- bbased methods on Gaussian mixture simulations and on benchmark electrocardiogram (ECG) data.
The rest of the paper is organized as follows. Section 2 reviews the setup of generative domain adaptation and the diffusion model. Section 3 introduces the proposed method and theoretically characterizes its effectiveness. Numerical results are given in Section[4] We conclude the paper in Section All proof' and additional numerical experiments are deferred to the Appendix.
2 Problem Formulation and Preliminaries
2.1 Transfer Learning Problem Setup
Let X' denote the data space and Y the label space. A domain corresponds to joint distribution over X' and y, denoted as pxy for the source domain and gxy for the target domain. The marginal distribution of data in the source and target domains are px and qx, respectively. Suppose we have access to m (labeled) samples from the source domain S = (x); H ~ pxy and n (labeled) samples from the target domain T [(x', v4) ~ gxy. Typically, the source domain contains significantly more samples than the target domain, i.e., < m. This setup reflects the common scenario where there is limited data available for specific tasks in the target domain, while abundant data is readily accessible and stored in the source domain.
The problem of interest as follows. Given a pre-trained generative model po for the data distribution Px in the source domain, and a relatively small number of samples from the target domain, generative domain adaptation approaches aim to obtain a generative model that can generate synthetic samples following the target data distribution 9x. We will focus on diffusion generative models, given their great success in synthetic data generation. We first present the key idea of a carefully designed guidance network for the generation of x values only. Then, we extend the method to facilitate conditional generations so that we can generate paired samples with labels, (x, y), and can incorporate downstream classification tasks on the target domain.

---

2.2 Preliminaries of Diffusion Model
Diffusion models are characterized by their forward and backward processes. For illustrative purposes, we discuss the diffusion model trained on the source domain. The forward process involves perturbing the data distribution px (x) by injecting Gaussian noise, as described by the following continuous-tine equation [36]:
dx,- = f(x, t)dt g())dw t [0,
where w the standard Brownian motion, f(-,(): Rd Rdd a drift coefficient, and g(.) R I is diffusion coefficient. The marginal distribution of X, at time / is denoted as pa px(X)), and po is the distribution of the initial value xo, which equals the true data distribution px (x). For notational simplicity and provided it does not cause further confusion, in the following, and we define p(x; Xx). Vs, t, as the conditional distribution x; given the value X,- Similarly, for initial value x following the target domain distribution, we denote the corresponding probability measure induced by the above diffusion process T as q.
will refer to this diffusion process p
Then, we can reverse the forward process I for generation, defined as:
dx,- [( - Vx log (x)] dt + o(t)dw:
where W is standard Brownian motion when time flows backwards from T to O, and di is an infinitesimal negative time step. The key of the backward process to estimate the score function of each marginal distribution, Vx log (x). then the generation can be performed by discretizations
of 2 13 36). Score Matching [16] (40, 335 are proposed to train a neural network s (x,, t) (parameterized by Φ) to estimate the score:
6¹ arg min E, (())) [x,, 6) Vx, log Pe (xx)) }.
where X(1) [0, 7] + o is positive weighting function, / is uniformly sampled over [0, 0.. One commonly adopted forward process is choosing affine f(x, t)
(()) and g(t) v ((t,
which yields the Gaussian transition distribution p(x; (x.] = .N(x;; v I ((t)x.,,(ttI), t > s.
with B(t) [0 > (0, 1) as variance schedule. This is the Variance Preserving Stochastic Diffcrential Equation (VP SDE) that we use in the numerical Section
Several works on image generation Conditional Score Matching, i.e.,
5 and inverse problem extends Score Matching to
Φ" = arg min E. (x())))))x.))) [|
t) - Vx, log (x)l())
(4)
where (x (y) is the conditional distribution of perturbed data x, given corresponding label y.
3 Transfer Guided Diffusion Process
In this section, we introduce the proposed Transfer Guided Diffusion Process (TGDP) that leverages pre-trained diffusson model - trained on the source domain data to generate data in the target domain. The proposed approach is orthogonal to and different from the existing fine-tuning type methods. We introduce the additional guidance in Section 3.1 The methods for calculating the guidance are provided Section 3.2 We extend our framework to the conditional diffusion model in Section 3.3 and propose regularization terms for enhancing the performance of our method in Section
All proofs are deferred to Appendix
3.1 Methodology Formulation
This subsection outlines the process of transferring knowledge from a diffusion generative model pre- trained using the source domain data S for generating samples that match the underlying distribution of target domain sample T. The simplest non-transfer type approach involves directly training a diffusion model on samples T from the target domain by denoising Score Matching as described by Eq (3) or Eq (4). However, since we assume only a limited amount of data is accessible on the target domain, directly learning from the target domain is unlikely yield effective generative model.
3

---

Several studies propose to finetune the pre-trained diffusion model to alleviate the challenges caused by limited data and make use of acquired knowledge [25 42 533. These methods typically design different strategies, such as adapters, to avoid finetuning all weights in a pre-trained model. However, these approaches generally use the pre-trained diffusion model from the source domain only as initial weights. Our method offers different way for better utilization of the acquired knowledge.
Our proposed method is inspired by the key observation detailed in the following Theorem 3.1 Intuitively, the score function Vx, log qe (x(x)) for the target domain differs from the score function Xx, log Pr (x²) of the source domain by term related to the density ratio function 9x / /px. We refer to this differing term as a guidance term in the following Theorem.
Theorem 3.1. Consider two diffusion models on the source and target domain, denoted as p and 9. respectively. Let the forward process on the target domain be identical to that on the source domain, (xx((x0) = >>xx((x)) and SΦr (xa, 1) is the score estimator in the target domain:
o² arg min Et {x(t)
||5
t) Vxi log q: (x(()))
(5)
then we have
(xx))
SΦ (x,,t Xx, log (x(e))
Vx, log E Bx(xo(x)) (xe)
(6)
p(xo)
pre-trained model on sourcce
guidance
Based on Eq G instead of solving sΦ from the limited training samples on the target domain, we construct Sø by combing the pre-training score estimator and the guidance based on binary classifier of source and target domain samples (detailed in Section 3.2). We comment on some potential advantages of this simple yet effective idea. First of all, we do not need to fine-tune the pre-trained diffusion model on the source domain, with the corresponding computation shifted to training the guidance network which is essentially classifier. Second, the guidance network can be effectively estimated by a domain classifier using data from both the source and target domains. There is also great flexibility in constructing this guidance network due to the extensive literature on classification problems and density ratio estimation approaches. Additionally, the sample complexity for training a generative model could be much larger than a discriminative model, since the generative model needs to recover the full spectrum of target data distribution, while a domain classifier only needs to distinguish whether the sample is from the source or target distribution.
3.2 Learning Guidance Network
We calculate the guidance for the diffusion model on the target domain as defined in the second term of Eq 66) via two steps. In the first step, we estimate the density ratio ((xo)/ by training a
classifier Cw (x) : X' -> [0, 1] to distinguish samples from the source and target domains. We adopt
the typical logistic loss as follows:
Σ log cw (xi) Ximp
Σ log(1 - (w x/-gg
w arg min
(7)
m
Then, the density ratio g(xo)/p hppppp((xp/) (x)) can be estimated as (1 Cw- (xo))/cw- (xo). and it can be shown that the optimal solution to the population counterpart of Eq (7) is exactly the true likelihood ratio 338. is worthwhile mentioning that we may only use a subset of source domain samples to learn the classifier ew to alleviate the unbalanced sample sizes, and we could also adopt modern density ratio estimators to improve the accuracy [31]. After learning the density ratio q q(x0) /p(())), the second step is to calculate the expectation Ep(xo x, (x(xx))/////) ///(x/(x)) using Monte Carlo simulation. Since it is hard to sample from g(xo Xx), we use the following equivalent formulation to get the value instead. This trick has also been used previous work such as the Appendix
(23..
Lemma 3.2. For a neural network bw (xx,t) t) parameterized by sb define the objective
((x0)
Eguidince (ab) = Eppxooxx)) Iawp (xx,, p(xo)
(8)
then its minimizer 36" = arg min Lquidaanc (1b) satisfies: 1
hep (xe, t) Expxalxe) ((03))/((xo]]

---

By Lemma 3.2 we estimate the value Ep(x)/x.) ( ppx)) using the guidance network hys solved by minimizing the objective function Lguidance (Tb), which can be approximated by easy sampling from the joint distribution p(xo.Xx X²). Combine the above steps together, the estimated score function for the diffusion generative model on target domain can be calculated as follows:
SΦ (x,,
Vx. log logp(x;) + Vx, log Rug (xe,
pre trainerinddd model on source
guidance network
3.3 Extension the Conditional Version
The approach outlined above is for generating the sample x in the target domain. In this section, we extend the idea to the conditional generation task. Such extension is essential when the label sets in the source and target domain are different since, in such cases, we usually rely on the conditional diffusion model for sampling 18 21. We first present the following theorem, which is an analog to Theorem 3.1 within the context of conditional score matching.
Theorem 3.3. Assume x, and y are conditional independent given xo in the forward process, i.e., p(Xe [2o y) p(x, Xo), Vt E [0, T. and let the forward process on the target domain be identical to that on the source domain q(x, [x)) = p(x, Xo), and Ø" is the optimal solution for the conditional diffusion model trained target domain q(xo; v). i.e.
Φ² = -arg min Es { (EE) (X..V)
[|sa(Xe. v, - Vx. log gr (x (xcly) 1v) }
(10)
then
xxxo: y) p(xo, w)
SΦ (x,, y. ))= = xx lx y)
+ Vx log Ep(xo x..
(11)
pre-trained conditional model on source
conditional guidonce
The key difference is we need estimate the joint density ratio between the source and target domain. We can extend the density ratio estimator in Section [32 for estimating joint density ratio, ie., also feed the label y into the classifier (x. y). The corresponding Lemma and its proof for the conditional version of Lemma 3.2 can be found in Appendix C.3 We further provide detailed discussion about how extend this conditional guidance to text-to-image generation tasks and when the source and target domain contain different class labels in Appendix
3.4 Additional Regularizations in Practical Implementations
In this subsection, provide two additional regularization terms in our final objective function, to enhance the performance of the proposed scheme.
Cycle Regularization In the approaches described above, after obtaining the classi- fier network (wr, calculation of the additional guidance Vx, log E (x.) (x(x)))(((x)))(( Xx, log Ex x..) (x(x, y) v)/p(xx. y)] for conditional generation) only utilizes the data from source domain. In this section, we provide an enhancement in which the limited data from the target domain can also be utilized to improve the training of the guidance network hag
Notice that (with detailed derivation given in Appendix C.5
g p(xu)
qo (xx)
Epxxo (xc)
Ex (xc)
(12)
pr [x:)x
where recall Pr (x₂) and qi (x₂) are the marginal distribution at time t for source and target distributions, respectively. A similar idea to hheorem(3.2impplle that we can learn the guidance network by solving the following optimization problem as well:
4 arg min Leyck Ex(xx.))) | (xx,))
(x,, (x(
(13)
Moreover, order to estimate the density ratio for marginal distributions at time between the target and source data distribution, train dineddennnentt classifier ew (x, f) to distinguish samples from
5

---

source domain (x) and target domain gc(x) by the logistic loss as follow:
w² arg min
m E
Σ
log cwo (Xe,
Σ
Σ
log(1 cw (x,, (xx;
1))
12
Xomp X No
Xo
x ix
where m, are the number of training samples in source and target domains. The density ratio Ye (x,))) can then be calculated by (1 cw- (x,: (Cw (x,,
Consistency Regularization Motivated by the fact that an optimal guidance network should recover the score in the target domain, further use score matching in the target domain as the Consistency Regularization Connssstonce to learn the guidance network better.
2b" = arg min Comsistence
4
= E. { ((0) [1Vx, log ((x)())))))) + Vx, log he (xx,t) Vx, log ((x;
(14)
Combining these two additional regularization terms together with the original guidance loss (I9). the final learning objective for the guidance network can be described as follows:
10" arg min {Eguidance + 7/, Leyele + 1)2 Leonsistence);
(15)
where 77 2 0 are hyperparameters that control the strength of additional regularization, which also enhances the flexibility of our solution scheme. We summarize the Algorithm of TGDP in Appendix [.I We provide the ablation studies that demonstrate the effectiveness of these two regularizations in Appendix D.2 and we also empirically show only adopt Coonsistence to optimize the guidance network is not good enough because of the limited data from the target distribution.
Remark 3, 3.4 (Discussion about related guidance). Classifier guidance has become common trick in recent research 36
6 We highlight that, under the transfer learning framework, the guidance proposed in our work is the optimal guidance since the resulting score function matches the oracle score on the target domain. On the contrary, vanilla versions of classifier guidance utilizing a domain classifier cannot generate samples that exactly follow target distribution. Indeed, for pre- pre-trained domain classifier Cu. vanilla domain classifier guidance formulates the source for generation as follows:
SΦ (X,,t) = Vx, logp(x)) Vx, Ex(x)
[log(1 cw (xo))
((x)
Vx, logp(x;) + Ux, log E Ex(x)/x))
(correct form proven in Theorem 3.1
p
cw (xo))
Vx, logg(xx)) + Vx, log Ex(x))x))
Co (Xo)
4 Experiments
In this section, we present empirical evidence demonstrating the efficacy of the proposed Transfer Guided Diffusion Process (TGDP) on limited data from target domain. In Section 4.1 conduct proof-of-concept experiments using a Gaussian mixture model to showcase that the guidance network of TGDP can successfully steer the pre-trained diffusion model toward the target domain. In Section we illustrate the effectiveness of TGDP using a real-world electrocardiogram (ECG) dataset.
4.1 Simulation Results
Experimental setup We begin with a Gaussian mixture model where X' Ed and y = (-1, 1}. On both domains, the marginal distribution for label y uniform y, and the conditional distribution of features is xly ~ N(Ju, III)), where μ E Rdd is non-zero, and I is the d dimensional identity covariance matrix. We let f1 #s on the source domain and f fT on the target domain, with (Mg)T μT 0. Under such case, the marginal distribution of x on the source domain px is a Gaussian mixture, for convenience we denote as ( o²I) ))) + 0.55 o²I), and the marginal feature of target distribution qx is 0.5. (HTT: 2 + 0.55 ( We let d = 2, Ms [0.5, 0.5). pT = [0.5. -0.5), o² 0. and draw m 10000 samples from source domain px, and 10, 100, 1000 samples from target domain qx, respectively.

---

Implementation details and Baselines We adopt the default Variance Preserving (VP) SDE in
36 with a linear schedule, i.e., q(x, Xo) p(x, (xo) N [x of of with a, and o being:
Or 81 Bo
30 2 L of o= v oz
4
with 3o 0,1, = 20. We adopt 5-layer MLP with hidden sizes of [512, 512, 1112,512 512, 256] and SiLU activation function as the diffusion model. We train the diffusion model on data from the source domain for 100 epochs using the Adam optimizer with a learning rate of 1e- and batch size of 4096. The guidance network is a 4-layer MLP with 512 hidden units and SiLU activation function. We train the guidance network 20 epochs for our TGDP and train vanilla diffusion model or finetune the diffusion model target domain 50 epochs. For generation, we adopt DPM-Solver [24] with a second-order sampler and diffusion step of 25. We compare TGDP with the following baseline methods: 1) Vanilla Diffusion: directly training from target domain: 2) Finetune Diffusion: finetuning all weights of pre-trained diffusion model target distribution
Experimental results We first demonstrate the effectiveness guidance in Figure I under the above setup. Figure (a) plots the source samples, while [i shows the target samples under different sample sizes n 10, 100, 1000. Figure [ (c-e) illustrates the generated target samples different methods, respectively. It can be seen that the samples generated via the proposed TGDP approach share similar patterns with the target distribution and two mixture components are more obvious compared with other baseline methods. Furthermore, since the true data distribution of the target domain is known, we calculate the average likelihood samples generated by each method as demonstrated in Table I for quantitative evaluation and comparison.
n=10
1=0
Source
1=1000
(b) Target
(c) Vanilla Diffusson
(d)
Figure An illustration of the effectiveness of TGDP on simulations with 10/100/1000 target samples, respectively.
Table 1: Quantitative evaluation of TGDP on simulations. Training 10K samples from the source domain and n = 10, 100, 1000 numbers on the target domain, respectively. TGDP achieves the highest average likelihood under target distribution.
Average likelihood
n=10
n=100
n=1000
Vanilla Diffusion
0.145
0.253
0.328
Finetune
0.290
0.329
0.335
TGDP
0.417
0.627
0.673
Fit is worthwhile mentioning that reason we not compared with works that finetune partial weights in a pre-trained diffusion model (44) is their results are usually worse or comparable with method that finetunes all weights, the implementation of 25 are not available, and the regularization proposed by [52 is only valid for image data.

---

As sanity check, we also look at the sensitivity of the learned density ratio estimator (through the classifier network (7) regarding different sizes of target samples. As shown in Figure even with only 10 samples from the target domain (and 10 samples from the source domain for class balance sampling), we can accurately estimate the landscape of density ratio (although the magnitude of the estimated ratio is not entirely accurate when the number of target samples equal 10).
(a) Oracle
(b) 10 target
(c) 100 target
(d) 1000 target
Figure 2: An ablation study the sensitivity of density ratio estimator.
4.2
ECG Data
In this section, we demonstrate the effectiveness of the proposed guidance on the benchmark of electrocardiogram (ECG) data. We first provide the standard synthetic quality and diversity evaluation in Section 4.2.1 Then, we utilize downstream classification tasks to further evaluate the effectiveness of TGDP Section 4.2.2 We follow the setup of existing benchmarks on biomedical signal processing 37 that regard PTB-XI. dataset [41] as the source domain and ICBEB2018 8 dataset [27] as the target domain. PTB-XI. dataset contains 21,837 clinical 12-lead ECG recordings 10 seconds length from 18,885 unique patients. A 12-lead ECG refers to the 12 different perspectives of the heart's electrical activity that are recorded. Moreover, the PTB-XI. dataset is multi-label dataset with 71 different statements (label). ICBEB2018 dataset [27] comprises 6877 12-lead ECGs lasting between and 60 seconds. Each ECG record is categorized into one nine classes, which a subset of labels in the PTB-XL dataset. We randomly select 10% samples limited target distribution by stratified sampling preserving the overall label distribution in each fold following [41]. We use the data from PTB-XL dataset and ICBEB2018 dataset sampling frequency of 100 Hz, which means 100 samples per second. We include more implementation details in Appendix D.3
4.2.1 Synthetic Quality and Diversity Evaluation
Baseline method We compare TGDP with the following baseline methods to demonstrate the effectiveness of TGDP. 1) Learn a generative model directly (Vanilla Diffusion): The vanilla way is to learn generative model directly on limited samples from the target domain. 2) Leveraging the pre-trained generative model from source domain (Finetune Generator): Since the label set of the target domain is subset of that in the source domain, preliminary solution is to utilize the pre-trained diffusion model to generate samples with labels in the target domain.
Experimental results In Table using two metrics. The first criterion is the widely used Frechet Inception Distance (FID) [12 to evaluate the quality of synthetic data, which calculates the Wasserstein-2 distance between the real data and the synthetic data on the feature space. We use the pre-trained classifier on the target domain the feature extractor, i.e., xresnet1 1500 [37] The second metric is the coverage (26) that evaluates the diversity of the synthetic data. It is defined as the ratio of real records that have at least one fake (synthetic) record in its sphere. The higher the coverage is, the more diverse the synthetic data are.
compare the generation performance on the target domain
From Table we see that TGDP achieves better performance than baseline methods on two criteria, which demonstrates the effectiveness of TGDP on generative transfer learning in scenarios with limited data. Moreover, TGDP has fewer parameters to trained and less training time. We also demonstrate the T-SNE of the generated ECG data in Figure
4.2.2 TGDP for Downstream Task
In Section 4.2.T we illustrate that TGDP capable generating samples that adhere to the joint distribution of data and labels in the target domain and is diverse enough. In this subsection,

---

(a) Vanilla Diffusion
(b) Finetune Generator
(c) TGDP
(d) Real
Figure 3: T-SNE of the generated ECG data.
Table 2: The effectiveness of TGDP ECG benchmark under synthetic quality and diversity criteria.
Method
Diversity (1)
FID (1)
Number of Parameters
Training Time
Vanilla Diffusion
0.37
11.01
50.2M
Ih
Finetune Generator
0.47
12.26
50.2M
40min
TGDP
0.53
10.46
2.8M
30min
further investigate whether utilizing TGDP acquire generative model for the target domain yields superior performance compared to existing transfer learning pipelines.
Baseline method First of all, generate sufficient samples. Incorporated with the original limited sample from the target domain, we train the classifier, which still denoted Vanilla Diffusion, Finetune Generator, and TGDP, respectively. Moreover, we have the following baseline methods. Directly train a classifier on target domain (Vanilla Classifier): Utilizing the limited data from the target domain, vanilla classifier can be obtained. Finetune pre-trained classifier (Finetune Classifier): Instead of training classifier from seratch on the target domain, the parameters of the classifier trained the source domain adjusted by using the limited data from the target domain. To verify the effectiveness of the generative model, we demonstrate that it improves the performance of the learned classifier in the following.
utilize the generative model learned in Section 4.2. to
Experimental results We adopt the same evaluation criteria as ECG benchmark 37 i.e., Macro-
averaged area under the receiver operating characteristic curve (AUC), Macro-averager F3-score
(14.82) )....)).. TP
(3 = 2), where Fg (1+321-17 +.84 EN FF'. and Macro-averaged G -cose with / 2, where TP G TP+ B In Table 3 TGDP outperforms baseline methods across three evaluation criteria, showcasing effectiveness transfer for diffusion model with limited data.
Table 3: The effectiveness of TGDP on ECG benchmark for downstream task. We provide 95% confidence intervals via empirical bootstrapping used by (37] 0.906(03) stands for 0.906 + 0.003.
Method
AUC
F3=2
G3=2
Vanilla Classifier
0.906(03)
0.674(06)
0.433(06)
Finetune Classifier
0.941(05)
0.747(08)
0.521
Vanilla Diffusion
0.932(05)
0.718(09)
0.464(09)
Finetune Generator
0.941(04)
0.761(10)
0.528(12)
TGDP
0.953(05)
0.773(11)
0.534(11)
5 Conclusion
In this work, we propose a novel framework, Transfer Guided Diffusion Process (TGDP), for transferring source-domain diffusion model to the target domain which consists of limited data. Instead reducing the finetuning parameters or adding regularization for finetuning, TGDP proves the optimal diffusion model on the target domain is the pre-trained diffusion model on the source domain with additional guidance. TGDP outperforms existing methods on Gaussian mixture simulations and clectrocardiogram (ECG) data benchmarks.

---

Limitations and broader impact Overall, this research presents a promising direction for lever- aging pre-trained diffusion models tackle new tasks. The proposed method, TGDP, has potential applications in wide range of tasks where domain shift exists. A limitation of this study is the lack of empirical validation regarding TGDP's performance on language vision tasks, which we have earmarked for future exploration. Since we propose generic algorithm for transferring knowledge to new tasks, this technique could enable people to train Deepfakes for disinformation better. Our approach hinges on the efficacy of detection methods in mitigating negative societal consequences.
6
Acknowledgement
Hongyuan Zha was supported in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low-Carbon Energy Network (No. ZDSYS20220606 002).. Guang Cheng was partially sponsored by NSF SCALE MoDL (2134209), NSF CNS (2247795), Office of Naval Research (ONR N00014-22-1-2688) and CISCO and Optum Al Research Grants.
References
[1] Aibek Alanov, Vadim Titov, and Dmitry P. Vetrov. Hyperdomainnet: Universal domain adapta- tion for generative adversarial networks. ArXiv, abs/ 10ss/2210.08884, 2022.
[2] Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-basee conditional ecg generation with structured state space models. Computers in biology and medicine, 163:107115, 2023.
[3] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 843- 852, 2023.
[4] Georgios Batzolis, Jan Stanczuk, Bisoil-Bibiine Schonlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. ArXiv, abs/2111. 2021.
[5] Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Ch Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun Yi Lee. Denoising likelihood score matching for conditional score-based data generation. In ICLR, 2022.
[6] Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Mare Louis Klasky, and C. Ye. Diffusion posterior sampling for general noisy inverse problems. ArXiv, abs/2209. 000s/2209.1687,, 2022.
[7] Agnimitra Dasgupta, Javier Sirgoitio-Esandee Deep Ray, and Assad Oberai. Conditional score-based generative models for solving physics-based inverse problems. In NIPS workshop. Deep Learning and Inverse Problems, 2023.
[8] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis. ArXiv, 105.0523,, 2021.
[9] Yuxuan Duan, Li Niu, Y. Hong, and Liqing Zhang. Weditgan: Few-shot image generation via latent space relocation. ArXiv, abs/2305.06671, 2023.
[10] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. ArXiv, abs/2305. 16381, 2023.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ArXiv, abs/2208. 0016118,11 2022.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.
10

---

[14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. ArXiv, abs/2204.03458. 2022.
[15] Xingzhong Hou, B. Liu, Shuai Zhang, Lulin Shi, Zite Jiang, and Haihang You. Dynamic weighted semantic correspondence for few-shot image generative adaptation. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[16] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695-709, 2005.
[17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. ArXiv, abs/2206.0055 00364, 2022.
[18] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-ze Text-to-image diffusion models shot video generators. ArXiv, abs/2303.13439, 2023.
[19] Nupur Kumari, Bin Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. 2023 IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), pages 1931-1941, 2022.
[20] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuging Du, Craig Boutilier, P. Abbeel,, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. ArXiv, abs/2302.12192, 2023.
[21] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis L Brown, and Deepak Pathak. Your diffusion model secretly a zero-shot classifier. ArXiv, abs/2303. 16203, 2023.
[22] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman, Few-shot image generation with elastic weight consolidation. ArXiv, abs/2012.08 2.02780, 2020.
[23] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guide diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, 2023.
[24] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li. and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. ArXiv, abs/2206.00927, 2022.
[25] Tachong Moon, Moonseok Choi, Gayoung Lee, Jung- Woo Ha, Juho Lee, AI Kaist, Naver AI Lab, and Aitrics. Fine-tuning diffusion models with limited data. 2022.
(26) Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. ArXiv, abs/2002.00979, 2020.
[27] Eddie Y. K. Ng, Feifci Liu, Chengyu Liu, Lina Zhao, X. Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei. Zhiqiang He, and Jianqing Li. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 2018.
[28] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross domain correspondence. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 00738-10747, 2021.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06113 125,, 2022.
[31] Benjamin Rhodes, Kai Xu, and Michael U. Gutmann. Telescoping density-ratio estimation. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4905-4916 Curran Associates, Inc., 2020.
11

---

[32] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to to-image diffusion models for subject-driven generation. 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22500- 22510, 2022.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes. Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11487, 2022.
[34] Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris N. Metaxas, and A. Elgammal. Diffusion guided domain adaptation of image generators. ArXiv, abs/2212.04473. 2022.
[35] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Conference on Uncertainty in Artificial Intelligence, 2019.
[36] Yang Song, Jascha Narain Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011 133456, 2021.
[37] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojeiech Samek. Deep learning for ecg analysis: Benchmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health Informatics, 25:1519-15288 1528. 2020.
[38] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012.
[39] Yusuke Tashiro, Jiaming Song. Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. ArXiv, abs/2107.03502, 2021.
[40] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23: 6666666666 1-6744, 2011.
[41] Patrick Wagner, Nils Strodthoff, Ralf Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific Data, 7, 2020.
[42] Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. A closer look at parameter- efficient tuning in diffusion models. ArXiv, abs/2303. 18181, 2023.
[43] Jiayu Xiao, Liang Li, Chaofei Wang, Zhengjun Zha, and Qingming Huang. Few shot generative model adaption via relaxed spatial structural alignment. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1 11203, 2022.
[44] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4207-4216, 2023.
[45] Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolei Zhou. One-shot generative domain adaptation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7699-7708. 2021.
[46] Mengping Yang and Zhe Wang. Image synthesis under limited data: A survey and taxonomy. ArXiv, abs/2307. 16879, 2023.
[47] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3813 3824, 2023.
[48] Yabo Zhang, Mingshuai Yao, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, and Wangmeng Zuo. Towards diverse and faithful one-shot adaption of generative adversarial networks. ArXiv, abs/2207.087:: 2022.
12

---

[49] Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, and Tao Mei. General- ized one-shot domain adaptation of generative adversarial networks. In Neural Information Processing Systems, 2022.
[50] Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. International Conference on Machine Learning. 2020.
[51] Yunqing Zhao, Henghui Ding, Houjing Huang, and Ngai-Man Cheung. A closer look at few-shot image generation. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9130-9140, 100, 2022.
[52] Jin Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. Few-shot image generation with diffusion models. ArXiv, abs/2211 1..0264,, 2022.
[53] Jin Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. Domainstudio: Fine-tuning diffusion models for dorveendriven image generation using limited data. ArXiv, abs/2306. 14153, 2023.
13

---

A More Discussion on Related Work
Finetune diffusion model on limited data Directly finetuning the pre-trained generative model on limited data from the target domain may suffer from overfitting and diversity degradation. Moon et. al. [25] introduce a time-aware adapter inside the attention block. Since the attention modules take about 10% of parameters in the entire diffusion model, they significantly reduced the turning parameters and alleviated the overfitting. While in 44, the authors only finetune specific parameters related to bias, class embedding, normalization, and scale factor. Zhu et. al. [52) found out the images generated by directly finetuned diffusion models share similar features like facial expressions and lack ample high-frequency details. Therefore, they introduce two regularization terms, pairwise similarity loss for diversity and high-frreggn fegh-frequency components loss to enhance the high-frequenn feature.
The main drawback of finetuning the pre-trained diffusion model is the sample complexity is larger compared with training a classifier since modeling the distribution a tougher task. In our work, we decompose the diffusion model on the target domain as the diffusion model on the source domain plus a guidance network. Since training guidance network (essential as classifier demonstrated in section 3) requires smaller sample complexity, we believe this novel framework might provide a new way for diffusion-basee domain adaptation.
Text-to-image difflusion model and learning with human feedback Numerous studies Text-to- Image diffusion models focus on optimizing the diffusion model to align with human preferences and personalize its performance for specific tasks. These endeavors commonly involve strategies such or finetuning diffusion model (or its adaptor) through
as text-guit sutddg zero- zero-shot finetuning 34
29
10] We acknowledge the significant potential in these approaches, given that language models inherently encapsulate rich semantic information, thereby endowing text-to-image diffusion models with zero-shot transferability. However, it is noteworthy that in domains lacking a substantial amount of paired data for learning semantic mappings, such as biomedical signal processing and electrocardiogram (ECG) data, we refrain from considering these methods as the primary benchmarks in our comparative analysis.
rewgrd-weighted objectives 332
19
20
Non-diffusion based approaches in generative domain adaptation Numerous works in genera- tive domain adaptation (or few-shot generative adaptation) study how to improve the transferability of the generative model on limited data from the target domain. Since we mainly focus on the diffusion model, we summarize the primary GAN-based domain adaptation there. They mainly propose to add
different kinds of regularization to avoid model collapse [28] 51 49
48
15 43) or finetune subset
of the parameter (adaptor) [1 45 22 50)
B More Discussion on the Potential of the Proposed Method
In this section, we demonstrate the proposed framework is general enough to deal with text-to-image generation tasks and heterogeneous transfer learning.
B.1 Text-to-Image Generation Tasks
Given a source distribution (Ir, c:) ~ p. where et denotes text prompts by using the terminology from 47 pre-trained diffusion model can be trained on the source distribution. Given target distribution (z, ct, c:) ~ q. where cy denotes a task-specific condition, Zhang et al. 47 can fine-tune the pre-traine pre-trained model by noise matching objective,
C EEm,tt.c...e.N.N..... [l]e Eg (x,, t, c,, ct) 12]
Our method can directly estimate Vx, log Ep(x))xx,c)) xxxx....o))
p(x).ez) rather than fine-tuning the pre-
trained diffusion model. Domain classifier (w. y) can still be used for estimating the density ratio, where y denotes the embedding of the condition. Moreover, directly fine-tuning the diffusion model on data from the target domain used by (7 is similar to the consistency regularization proposed in our work, while they have more in-depth design for the architecture and have great results on vision-language tasks. However, with limited data from the target distribution, direct fine-tuning may not achieve good enough performance, which is verified in the two-dimensional Gaussian
14

---

setting. In 1471, Zhang et al. propose to use zero convolution layers, i.e., x 1 convolution layer with both weight and bias initialized zeros, which alleviates the instability of fine-tuning process. This is very different from our methodology which relies on the smaller sample complexity of the classifier/density ratio estimator.
B.2 Heterogeneous Transfer Learning
When the source and target domain contain different class labels, our framework is still applicable, i.e, when y/ + ys:
q(xo, M/e
Sen (xt, We.
(xe,Be,()) (),
Xx, log (x
+
V
xr log Pep(x (e.
p(xo, U/5)
target source
pre-trained conditional model on source
conditional guidance
To generate an umseen class yr, the key problem here is to choose particular class from the source domain y/ such that we can borrow useful information from the source domain to generate this unseen class from the target domain. The coupling between Y/ and ys can be learned by solving a static optimal transport problem. More in-depth design, e.g. coupling solved by static optimal transport, can be left to future work.
C Theoretical Details for Section 3
C.1 Proof of Theorem 3.1
Proof. To prove Eq 6 we first build the connection between Score Matching on the target domain and Importance Weighted Denoising Score Matching the source domain in the following Lemma.
Lemma C.1. Score Matching the target domain is equivalent Importance Weighted Denoising Score Matching on the source domain, i.e.,
o² arg min 18, ((t)
[[sp(Xe,)) Vx, log (x
(16)
g(xx)
arg min E. (p()) (xpe
[1x((x)))) Vx, (x, (x))] p(xo)
Proof of Lemma C.1
Φ We first connect Score Matching objective in the target domain to Denoising Score Matching objective in target distribution, which proven by 440 i.e.,
= arg min E, ((t)), [|s (xe, t) - Vx, log (xx)
arg min E, (xx)) [1so((),)) t) Vx, log (xx/)x))))
Then, we split the mean squared error Denoising Score Matching objective on target distribution into three terms follows:
Ey(xo) Eytxe xo) ||ا0| (x),,) Vx, log q(x, xxo)
=B -By(xoxxx)
[ (x,, 1)12 2xx(x))))) (s t), Xx, log (xx((()))))) + Ci,
(17)
where Ct Ex(xo.x.)
||Vx, log (xxx//xx)))))))) |xx is constant independent with Φ. We similarly split the objective function in the right-hand side (RHS) of Eq (16) as follows:
[1sp(xx..)) () Vx, logp(xx ((x0)
Ep(xa) Eptxilan (xo)
p(xo)
((xo)
2E (S 1), Vx, logp(x((xo)) ((xp)
-Ep(xo))) ( 11 p(xo)
p(xu) + C2. (18)
15

---

where C2 is a constant independent with Φ. It is easy to show that the first term in Eq 17 is equal to the first term in Eq (8)) i.e.,
[Is t)ll 9(x0) xx(x),xx)
p(xo)p(xxe)x) ||S@ (xe: t)ll g(xo)
dx
p(xo)
((xo)
p(xx))((x(2x)) || (x,, t) 1 (xx))
dx
p(xo)
((x0,xx) x1) (s t)ll dx
(x((xx)))) [1s0((x, t)113]
where the equality (7) is due to ((xx(xx) = p(xe|x())
Next, we prove the second terms Eq 17) and Eq 18) are also equivalent:
(sxs(x,,)) 1), Vx, logp(x:(xo)) (xx))
xx)x xx)
p(xo)
Vx, Xn), ((x0) dx
p(xo, (1)5 1),
p(x, (x0)
p(xo)
V gxxe xx0 ((xo)
p(xo)p(xx (x.,,
dx
p(xx (x0)
p(xo)
(sa(xi,)) 1), Vx (x))
(so(x1,,) t), Vx.
(xxdxx
Bxxxxxxt) ((
f), Xx, log (x,, xn))],
where the equality (i) is again due to g(xx) (x)) p(x//xo). Thereby we prove Eq 16
According to Lemma
C.T
[Iso( Vx, log 1(x xo)) 9(x0)
Φ- arg min E, MEtx xx)
p(xo)
Based on this, we may use Importance Weighted Denoising Score Matching the source domain to get the analytic form of s as follows:
Ep(x)(x))
Xx, log ((x (xo) {(80) p(xo)
Søb (Xe, 1)
a(Xo) p(xo)
Ex(xo/De))
The RHS Eq (6) can be rewritten as follows:
g(xo)
Vx, E Exxxxs (x+)
x(xo p(xo)
RHS Vx, (xx))(xx)) + Vx, log Exxxxo/xx (x))
Vxx (xx.)((x))
p(xo)
E Ex((x xxs)
(250)
((xo)
Ex(xo(xa) (xa)
2(xx)) plxo)
Vx. log p(xo (x,)
Xx, logpe(
g(xa) p(xa)
Since
V log (xo
Vx, log + Vx, log p(xo) Vx. logpe(())
Vx, log (og((xx((0)) Vx, log (xt);
16

---

we can further rewrite the RHS of Eq (6 as follows:
E pp(x)(x))
g(Xo] p(Ne)
Vx, x. log p(x, (xo)
RHS V x, log (x(e))
Vx, log (x.)
Ep(xp) xx))
x(xo)
p(xa)
E px)x
Vx. log p(x, (xo)
g(x0)
p(xo)
Ep(xolxe (xc)
e(xo) pixo)
- (X2, t)-
Therehy we complete the proof.
C.2 Proof of Lemma 3.2
Proof. The proof is straightforward and include it below for completeness. Note that the objective function can be rewritten as
Lguidance (ab) : : Ep(xoxx))
now (xe, p(xo)
((xo)
= fx { x0 p(xo)>x) naw (xe, t)
((x0) //2
122 dxo pl (xx, )dxx
p(x0)
= S., { (xes( 1)1 2( b), xx
g(x0)) dxo) J p(xx)dxx + C
p(xolx))
p(xo)
(((x))
N t) - Ex(xa(x))
d(xx)
p(xn)
Xx
where C' is constant independent of vb. Thus we have the minimizer 30" arg min Cguidance (1b) satisfies hap (xe, t) Ep(xo x.) (g(xo)/p(xo)]
C.3 Conditional version for Lemma 3.2
Lemma C.2. For a neural network h. (xe, y, parameterized by sb. define the objective
27
g(xo 9)
Cuuistace (1b) = Ep(xo.x..) Naw (Xes v.
(1/)
(19)
pl No, v)
then its minimizer 3p arg min guuidance (bb) scttssfie::
(ab)
hag (xe, y, (p( [g(xo, w)/p(xx0 v)].
Proof of Lemma
Proof. The proof is straightforward and we include it below for completeness. Note that the objective function can be rewritten
Cguidance (xb)
[ (xe, v.
(x0. y)
-Ep((xoxx..)
p(xo. v)
(xx. v) - dxo p(xe (elll)yyddss
p(xooxxe: w)
hap (xx, 8)
p(xo, x)
Xo
2(h (xe, yr t),, / p(xolxx, v) g(x0 v) p(xo. v)
dxoi } p(xxll)p(u)
|/ (Xe, M. t)11
/
qxx.. v) p(xo, v)
Map (x,, Ep(xo (ee.e))
p(xxls)p(y)ddddzz

---

where C' is constant independent of 2/. Thus we have the minimizer x0" = arg min Lguidance (ab)
satisfies has- (xt, V, t) Bp(xo(xraa) [g v)/p(x))pp))))p 2)]-
C.4 Proof of Theorem 3.3
Proof. The proof is similar to the proof of Theorem 3.1 To prove Eq [I we first build the connection
between the Conditional Score Matching on the target domain and Importance Weighted Conditional Denoising Score Matching on the source domain in the following Lemma:
Lemma C.3. Conditional Score Matching on the target domain is equivalent to Importance Weighted Denoising Score Matching on the source domain, i.e.,
Φ arg min E, ((t)) [[ssp Vs Vx. log qe (xxx (x)]
((x. v) [Is v. Vx, log ((x) p(xo, y)
arg min E,
We first connect the Conditional Score Matching objective in the target domain to the Conditional Denoising Score Matching objective in target distribution, which is proven by 44 Theorem i...
Proof of Lemma C.3
6- aarg min E. ((t)) [1sp(Xe. V. Vxx log qe (x((el)))) [2]
arg min E. ((t) Bx(x) (ott)tt)))
[|sg(X(.. v. Xx, log q [x0)]]]
Then we split the mean squared error of the Conditional Denoising Score Matching objective on target distribution into three terms as follows:
Eg((o Bx(xx (xo)
[X y, - Vx, log xx) [13]
-By((xo x)))) |1 (x+, U. 1112]
2 Xex) (x,, v, (), Vx, log [x))] + C1,
(20)
where Ci xx))
| log ((x, xx))ll is constant independent with Φ. and q(x, X y)
(X, (x0) because of conditional independent of x, and y given x by assumption. We can similarly split the mean squared error of Denoising Score Matching on the source domain into three terms as follows:
Ep(xo..) Epixe rx)) [ (x.,,)))) v. - Vx, log p(x, xx) 1g (xx. y)
p(xu. v)
[150((2000 v. 1)12 q(xo. w)) p(xo, v)
((x0 3) p(xo- y)
-Ep(xo.Xe.))
2F 2(xxx))))
(8 Vs, Vx, (xxp(x)(x))
+ Cz.
(21)
where C22 a constant independent with d.
It is obvious to show that the first term in Eq 20 is equal to the first term in Eq 21 i.e.,
[| v. 1 4(x. y)
Bp(xoixe11)
p(xo. y)
f xa Xx
((xo.v)((x(xx))) || (xe, v. 1 (xx. v)
dxndx,
p(xo. v)
9(x0, v)
p(xo. ()g(x(xx))) (xe, y. +)1? p(xo: v)
dx
[
q(x0, x., v) (x v. 1) dx dy
Ey(xxxx..)) [|10 (xe: v. 1112
18

---

And the second term is also equivalent:
Vx, log (xx((x))x))) g(x0: 21
xp(xo.x..1))
p(xo.1 1)
Vx, Xx, (x)
g(x0, 1) dxodx,dy
p p(xo, Xx, x), H. 1,
p(x, (x))
p(xo, y)
Xg
Vx. 9 xx))
((50: y) dxodxxdy
p(x, (x))
p(xo, v)
(x.: Vx (xx W)
(s (x.. v, V log (xx y)xxoxx,dy
- x.1)) (18((x)))))) t), Vx. log (xx
According to Lemma
C.3 the optimal solution satisfies
[|s y1, - Vx. log 1og)(xx|xo)11 ((xo: v)
Φ- = arg min E, ((1) Ep(x.1x0) (x))
(xx. v)
Then, we use Importance Weighted Conditional Denoising Score Matching on the source domain to get the analytic form of sø- follows:
E px(xolxx
V log pa (xx (x)) 2(x0.9) p(xu.9)
S (x1, y,
g(xo.3/) p(Xo.M)
xx(xo(x11)) ()))
Moreover, the RHS of Eq be rewritten
(xx. M/
RHS V x. logpr(x(lv)) + V x. log Ey(xo (x.1))
p(xo-
V xx (lpppx (ce.g)
((x0-21) p(x0-3r)
Vx log logl)(x<lv))
Exp(xo xe.i))
((x0-2/)
p(xo-211
Ep(xo bes.g)
Vx, log p x (x,, m)
Vx. logpe(xxll)) +
pl xo.3)
Epixo|xe.w)
(xx0.0)
p(Xa.3)
Since
Vx, x, (xg) = Vx, log p Xx No. v) + Xx, log pl (xoll)) Vx, log (x<el())elv)
Xx. log ((x, Xo. y) Xx log Pr (xxly),
Vx, ( Xx) Xx log (x y).
we can further simplify the RHS Eq as follows:
E Ex(xo x..1)
((Xo.v) ((XoMM)
Vx. log p (x0)
RHS Vx, log (x:l)))
Vx, log pe (xel9)
E Ex(x0 xx)1))
g(xo.v) p(xo.1)
Vx, log pl X xxxx/xx)) ((xa.g)
p(xo.9))
Eppxooxc..
=S (Xt, 1). Thereby, finish the proof.
19

---

C.5 Proof for Cycle Regularization
Proof of Eq 12
(x5) p(xo)
g(x0)
p(xx
(xo))
g(200)
Ep(xolxe)) (xc)
p(xo (x,)
dxo
dxo
p(x0)
(xe)
p(xo)
(xxe(x0)p1 g(x0)
g(xo) dxo
dxo g(xx|xxx) xx))
pe (x+)
p(xo)
pr(x+)
/ gxxo) (xxx))) (Xe) g(Xn)
dxco = / ((>xo|xx))
9: dx0
(xx(x))
g(xo)
Pc(xx)
(xx)
ge gx()x)
[x(x))(xs) xx,)
(x.))
where P((x)) = ((x0))((x)))) Xx d and (x,) =
are the marginal distribu-
tions time source and target distributions, respectively.
D More Details on Experiments
D.1 Algorithms for TGDP
TGDP adopts Algorithm 1 and for training domain classifier and Algorithm 3 and 4 for training the guidance network.
Algorithm 1 Training domain classifier
Require: Samples from the marginal distribution of the source domain p(x) and target domain g(x), and initial weights of domain classifier w.
1: repeat
2: Sample mini-batch data from source distribution and target distribution respectively with batch
size b.
Take gradient descent step on
V
[lgg cw (x:)]
Σ x/E9
[log(1 cw (x(()))
b
4: until converged.
5: return weights of domain classifier w.
Algorithm 2 Training Line-dependen domain classifier
Require: Samples from the marginal distribution of the source domain p(x) and target domain g(x), pre-defined forward transition pxx,l Xx)), and initial weights of domain classifier w.
1: repeat
2: Sample mini-batch data from source distribution and target distribution respectively with batch size b.
3: Sample time t Uniform((1,, T}) and perturb xo by forward transition p(x, (xo).
4: Take gradient descent step on
Σ
Dog (x:,
Σ
Σ
[log(1 ew (xt,
b
Xi up \x
Xor-q xi Xxo
5: until converged.
6: return weights of time-dependent domain classifier w.
20

---

Algorithm 3 Training a guidance network (without regularization)
Require: Samples from the marginal distribution of the source domain p(x), pre-defined forward
transition p(x)(Xo), pre-trained domain classifier cw, and initial weights of guidance network rb.
repeat
2: Sample mini- bani-batch data from source distribution Xy with batch size b.
3: Sample time Uniform([1, T)) and perturb Xp by forward transition p(x,, xx))
4: Take gradient descent step
I1le (xx, - (1 - cw (x0))) // (x00)l }
Σ XoiXt
V
5: until converged.
6: return weights of guidance network 2b.
Algorithm 4 Training a guidance network (with regularization)
Require: Samples from the marginal distribution of the source domain p(x) and target domain g(x), pre-trained diffusion model on source distribution Ssource (X,, t), pre-defined forward transition q(x, ((x]|xo), p(x; (ox/xx/xx),.ooo)>)>>>)) pre-trained domain classifier (xo) and time dependent domain classifier d (Xo, t), hyperparameter 7/,, 72, and initial weights guidance network rb.
1: repeat
2: Sample mini-batch data from source distribution Xy with batch size b.
3: Perturb Xg by forward transition p(x, xx))
4: Lguiddace (xb) 5 Σ 11 (xe, () (1 cw (xo)))
Xo.Xi,t
5: Sample mini-batch data from target distribution x/ with batch size b.
6: Sample time & ~ Uniform({ ((rrmm((1........ ,.}.....))))))....... and perturb x by forward transition (xx(1x)).
[lle (x(,)) (1 e (x(: t)) /4/ ( (X6: 1
7: Laycte vb) = Σ
x/xx/.e
8:
Leumsistence (1b) = Σ
Ssaunce (x), + Vx; log hop (x,, t)
Vx; log (x{((x)()
9: Take gradient descent step
V { (guidancc + Leydle + 1/2 Leonsistence
10: until converged.
return weights of guidance network 4b.
D.2 Ablation Studies on simulations
In Figure D.2 demonstrate the ablation studies on simulations. We can that only using the consistency regularization term (Figure [D.2 (b)) is not able to recover the true distribution in the target domain. Our guidance loss together with cycle regularization can learn good approximation of target distribution while adding consistency regularization can achieve better performance.
D.3 Implementation details for ECG Benchmark
For TGDP and all of the baseline methods, we utilize the same architecture as the conditional generative models for ECG data, SSSM-ECG 22. For Vanilla Diffusiio, train the diffusion model for 100k iterations by Adam optimizer with learning rate 24 For Finetune Generator, we finetune the pre-trained diffusion model for 50k iterations by Adam optimizer with learning rate -c-5 For TGDP, we adopt a 4-layer MLP with 512 hidden units and SiLU activation function as the backbone of the guidance network. We train the guidance network for 50k iterations by Adam optimizer with learning rate 2e For utility evaluation, we adopt the same architecture, xresnet d50 [37]. as the backbone. We train the classifier from sketch for 50 epochs with with learning rate le-2. For Finetune Classifier, we finetune pre-trained classifier for 30 epochs with with learning rate le-3.
21

---

63
05
0.5
50
00
0.
-05
-05
-05
-10
-15
(a) Source diffusion
(b) Only Luuidince
(c) Lguidiance Leyele
1.5
1.5
1.0
10
-13
-15
-10
-05
-15
-1.0
-55
-13
(d) Lguidunee
Leonsistance
(e) Lquidunce Leyole Loonsistnnce
(l) Target
Figure 4: An illustration of the effectiveness of cycle regularization and consistency regularization proposed in Section 3.4
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes].
We summarize the contributions and scope in Abstract as "we prove the optimal diffusion model for the target domain integrates pre-trained diffusion models with additional guidance" and we also summarize the main contribution of our paper in Introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
We summarize the limitations of this paper in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specificatiin, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
22

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recogmition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
Justification: We carefully state the assumptions in the statement of all theorems in this paper and all the theorems are proven the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We provide all of the details reproduce the experiments in this paper and also release our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
23

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the partieular way they provide for repro- ducibility. In the case of closed-source models, it may be that access to the model limited in some way (e.g., to registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes].
Justification:
We provide the code for all of the experiments together with clear instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cd
Pabllic/guidesssCddddSubmm sesiiisssoooocyyoii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.co. http:cccccpbccccceecceeccccccccc
issions Policy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: Experimental Setting/Details can be found in our paper and code.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: The standard deviation of the experiments in ECG Benchmark has been provided in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes].
Justification: We utilized a computing cluster equipped with 6 NVIDIA GeForce 3090 GPUs with memory 24268MiB and Intel(R) Xeon(R) Platinum 8352Y CPUs @ 2.20GHz. The computational costs for each of the individual experimental runs can be found the following table. The total computational time of all experiments in this paper is around 200 GPU hours.
Experiments
Memory-Usaa
Running Time
Experiments Table
1281MiB
5min
Training Vanilla Diffusion
19815MiB
1h 40min
Finetune Generator
1981
Training TGDP
19597MiB
40min
Sampling
9535MiB
21h
Experiments Table
6075MB
10min
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics httppe://earrrgg.ccc///b////eebb
Question:
Answer:
[Yes].
Justification:
We review the NeurIPS Code of Ethics and ensure our compliance with its requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there a special consid-
cration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We summarize the potential negative societal impacts of this paper together with the corresponding solutions to mitigate the negative impacts in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
26

---

Answer: [NA].
Justification: The main contribution of our paper is an efficient way for transferring a pre- trained model target distribution. We do not release data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: The license of the dataset and code used in this paper can be found in the following table.
Assets
License
Link
PTB-XI.
CC-BY 4.0
ICHEB2018
COO: Public Domain
hnttpa://wwe http://///.///////////////////aa 1067rabreeddee
SSSD-ECG
MIT License
ptwp:///ooooooaooooooooooroooao fy-file
FCG Renchmarks
CC-BY 4.0
httpa///ggthubb...o///h//t////// http:///wwweeeoootooooe.eecrcate banchrarking
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, phcors thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification:
The code provided in the supplementary material follows the CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not release new assets.
27

---

Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]-
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "ykIh0QeNss", "reviewer": "Reviewer_9nph", "summary": "introduces a novel approach called Transfer Guided Diffusion Process (TGDP) for transferring knowledge from a pre-trained diffusion model in the source domain to a target domain with limited data. The authors present the methodology for transferring knowledge, including the formulation of the guidance network and its learning process. They also extend TGDP to a conditional version for joint data and label distribution modeling.", "questions": "In what scenarios does the proposed framework work best?", "limitations": "Authors have discussed limitations well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TGDP offers a new perspective on transferring knowledge from pre-trained diffusion models to new domains with limited data. The whole framework is innovative and reasonable.\n2. The paper provides a solid theoretical basis for TGDP, including proofs for the optimality of the approach.", "weaknesses": "1. More experiments on real-world datasets are required to further validate the effectiveness of the proposed framework.\n2. The computational cost of training the guidance network and the diffusion model should be discussed.\n3. The paper primarily validates TGDP on Gaussian mixture simulations and ECG datasets. Its performance on other types of data or domains is not empirically tested or discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided. \n\n**Q**: *The paper primarily validates TGDP on Gaussian mixture simulations and ECG datasets. Its performance on other types of data or domains is not empirically tested or discussed. More experiments on real-world datasets are required to further validate the effectiveness of the proposed framework.*\n\n**A**: Thank you very much for your comments. We refer to the \"general\" response and Table 1 in the attached file.\n\n\n**Q**: *The computational cost of training the guidance network and the diffusion model should be discussed.*\n\n**A**: Thank you very much for this question. We provide the computation cost of training in Table 2 on page 9. The number of parameters in the guidance network (domain classifier) is much less than that of the diffusion model, which reduces the computational costs against fine-tuning based methods.\n\n\n**Q**: *In what scenarios does the proposed framework work best?*\n\n**A**: Thank you very much for this question. In this work, we prove that using the guidance $\\nabla _{\\mathbf{x} _t} \\log \\mathbb{E} _{p(\\mathbf{x}_0|\\mathbf{x}_t)}\\left[q(\\mathbf{x}_0)/p(\\mathbf{x}_0)\\right]$ (or $\\nabla _{\\mathbf{x}_t} \\log \\mathbb{E} _{p(\\mathbf{x}_0|\\mathbf{x}_t, y)}\\left[q(\\mathbf{x}_0, y)/p(\\mathbf{x}_0, y)\\right]$ for conditional generative model), we can generate the samples from target distribution with the source pre-trained generative model. Therefore, The performance of our method depends significantly on the accuracy of the estimated guidance network, particularly the estimated density ratio between the target and source distributions. When this density ratio is accurately estimated (especially when the relative magnitudes are accurately captured), our method achieves optimal performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2CmALtxV1", "reviewer": "Reviewer_53Zj", "summary": "The paper proposes an approach for transfer learning based on score-based generative models and density-ratio estimation. The authors show that in order to transfer a trained diffusion model from a source to a target domain, all that is needed is the expectation of density ratios of source and target domains. This expectation can be learned using neural networks reducing the problem of learning a generative model on the target domain to a regression/classification task.", "questions": "- Figure 1 shows the data as scatterplot which makes it impossible to see the learned density. Since, the true densities are known, a representation using kernel density estimates would be better to see if the learned model aligns with the truth.\n- For readers who do not know what a \"12-lead ECG data\" is, the supplement should contain this information (at least a rough overview). How does 12-lead ECG data look? Are they one-dimensional time series? At what rate are they sampled? \n- Task 4.2.2 is a bit unclear. The authors should be more detailed what the goal is and what they are exactly doing (even though one can infer it from the text).\n- It would have been nice to see the ablation for all combinations that contain $\\mathcal{L}\\_{\\text{guidance}}$, and in particular $\\mathcal{L}\\_{\\text{guidance}}$ alone (and not  $\\mathcal{L}\\_{\\text{consistence}}$). In this state, it is not clear what, e.g.,  $\\mathcal{L}\\_{\\text{cycle}}$ or $\\mathcal{L}\\_{\\text{consistance}}$ contribute separately.\n- How are the hyper-parameter $\\eta$ chosen?", "limitations": "- A statement on limitations regarding empirical evaluation exist.\n- Possible negative societal impact, e.g., regarding generation of deepfakes using transfer learning, is acknowledged.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- As far as I can judge, the proposed method for transfer learning is both original and significant. It should make a good contribution to the research community.\n- In addition to the core of the method, the proposed regularizers that improve efficiency are in my opinion conceptually convincing.\n- The paper is well written and easy to follow.\n- The experimental evaluation on the ECG data is in my opinion well executed and convincing.", "weaknesses": "- While the method is in my opinion theoretically convincing, the experimental section and ablations are a bit lackluster which decreases confidence in the method (even though I acknowledge the complexity of the task itself). \n- The first experiment on a 2D Gaussian illustrates the performance of the method in comparison to baselines decently, but it is in my opinion ultimately too trivial to be really meaningful (since the density ratios of the 2 Gaussian mixtures can be easily learned, because they are well separated, I would expect that a method based on learning density ratios outperforms baselines here). The ablation in Figure~4 in the appendix is conducted too superficially. It consists of scatterplots, no real quantitative analysis, and doesn't evaluate all relevant loss terms of the model.\n- The evaluation on the ECG data is not fully clear, since meta-information, such as sampling rate, and other details are not given (as far as I can tell).\n- In my opinion, a quantitative evaluation on at least 2-3 more high-dimensional real-world data sets should have been conducted in addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided. \n\n\n**Q**: *The first experiment on a 2D Gaussian illustrates the performance of the method in comparison to baselines decently, but it is in my opinion ultimately too trivial to be really meaningful (since the density ratios of the 2 Gaussian mixtures can be easily learned, because they are well separated, I would expect that a method based on learning density ratios outperforms baselines here).*\n\n**A**: Thank you very much for this question. We agree that the two-dimensional Gaussian mixture setup is a relatively easy task and serves as a proof-of-concept numerical example to validate the proposed method. To further demonstrate the effectiveness of our method, we have conducted experiments on real-world ECG datasets, and on image datasets in the \"general\" response above and Table 1 in the attached file therein. Additionally, we would like to highlight a challenge associated with the two-dimensional Gaussian example: when two distributions differ significantly, the density ratio estimator struggles to accurately estimate the density ratio. Due to the well-separated nature of the distributions, the density ratio at some points can be as large as $10^5$. One key takeaway from our work is that our method is robust to the magnitude of the estimated density ratio.\n\n**Q**: *The ablation in Figure~4 in the appendix is conducted too superficially. It consists of scatterplots, no real quantitative analysis, and doesn't evaluate all relevant loss terms of the model. It would have been nice to see the ablation for all combinations that contain $\\mathcal{L} _{\\text{guidance}}$, and in particular $\\mathcal{L} _{\\text{guidance}}$ alone (and not $\\mathcal{L} _{\\text{consistence}}$). In this state, it is not clear what, e.g., $\\mathcal{L} _{\\text{cycle}}$ or $\\mathcal{L} _{\\text{consistence}}$ contribute separately.*\n\n**A**: Thank you very much for this suggestion. We refer to the “global” response and Figure 1 in the attached file.\n\n**Q**: *The evaluation on the ECG data is not fully clear, since meta-information, such as sampling rate, and other details are not given (as far as I can tell). For readers who do not know what a \"12-lead ECG data\" is, the supplement should contain this information (at least a rough overview). How does 12-lead ECG data look? Are they one-dimensional time series? At what rate are they sampled?*\n\n**A**: Thank you very much for this suggestion. A 12-lead ECG (electrocardiogram) refers to the 12 different perspectives of the heart's electrical activity that are recorded, which is 12-dim time series data. We use the data from PTB-XL dataset and ICBEB2018 dataset at a sampling frequency of 100 Hz, which means 100 samples per second. We include this necessary information in the revised version in section 4.2. \n\n**Q**: *How are the hyper-parameter $\\eta$ chosen?*\n\n**A**: Thank you very much for this question. The guidance term is calculated based on data from the source distribution, while two regularization terms are calculated based on data from the target distribution. To choose a good $\\eta_1$ and $\\eta_2$, we must take the number of samples in the source $m$ and target distribution $n$ into consideration. Therefore, we initially set  $\\eta_1= \\eta_2 = \\sqrt{n/m}$ and rely on the grid search to determine the $\\eta_1$ and $\\eta_2$.\n\n**Q**: *A quantitative evaluation on at least 2-3 more high-dimensional real-world data sets should have been conducted in addition.*\n\n**A**: Thank you very much for this suggestion. We refer to the \"general\" response and Table 1 in the attached file.\n\n**Q**: *Task 4.2.2 is a bit unclear. The authors should be more detailed what the goal is and what they are exactly doing (even though one can infer it from the text).*\n\n**A**: Thank you very much for this suggestion. We include a paragraph in section 4.2.2 to describe the ECG classification task in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DcvjhyGAxp", "reviewer": "Reviewer_UUwS", "summary": "This paper addresses the transfer learning problem for diffusion models, specifically adapting pre-trained diffusion models to downstream datasets, particularly when the data size is small. Traditional parameter-efficient fine-tuning methods often use pre-trained models as parameter initialization, selectively updating parameters based on prior knowledge. These methods, however, can be suboptimal and not robust across different downstream datasets. The authors propose a novel approach called Transfer Guided Diffusion Process (TGDP), which aims to transfer pre-trained diffusion models more effectively. Instead of fine-tuning, TGDP treats the knowledge transfer process as guided by the pre-trained diffusion model. This method involves learning a domain classifier and using its gradient to guide the estimated score function from the source to the target domain, complemented by additional regularizations in practical implementation. Experimental results demonstrate that TGDP outperforms traditional fine-tuning methods, achieving state-of-the-art performance on both synthetic and real-world datasets.", "questions": "1. **Consistency Regularization:**\n   - The consistency regularization requires optimization of the gradient term $\\nabla_{x_t}\\log h_\\psi(x_t,t)$, but the paper lacks details on how to optimize this term, especially when second-order gradients are involved. Could the authors elaborate on the optimization process for this term?\n\n2. **Effectiveness of TGDP:**\n   - In Table 1, although TGDP significantly outperforms baselines, its performance is highly dependent on the target data size $n$. This seems contradictory to two points mentioned by the authors: (a) the training of the domain classifier $c$ is not significantly affected by $n$ (Figure 2), and (b) the training of the guidance network $h$ with the main term $\\mathcal{L}_{\\text{guidance}}$ does not require target data (Equation 8). How do the authors explain this discrepancy?\n\n3. **Explanation of Figure 3:**\n   - Figure 3 shows that the visualizations of *Finetune Generator* and *TGDP* appear similar, yet their performance differs significantly. Could the authors provide more insights and explanations about the figure and the reasons behind this performance difference?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Provides a novel perspective on knowledge transfer for pre-trained diffusion models.\n2. Theoretical foundation of the proposed method is well-constructed.\n3. The paper is well-organized with clear presentation.", "weaknesses": "1. **Scope of Title:**\n   - The title \"Transfer Learning\" is too broad. The paper focuses solely on (few-shot and supervised) domain adaptation, where the upstream and downstream tasks are similar in nature. General transfer learning encompasses a wider range of tasks, including those with different downstream tasks from the upstream one. For instance, transferring pre-trained text-to-image models to controllable generation or text-to-video generation are broader and more complex tasks that fall under transfer learning.\n   - The paper does not address transfer learning across different label spaces. Even within the same generation tasks, domain adaptation requires identical label spaces for source and target domains, which is not always practical. For example, transferring pre-trained conditional generation models on ImageNet to other fine-grained datasets with different label spaces presents a more challenging problem than the domain adaptation addressed in this paper.\n\n2. **Empirical Results:**\n   - The benchmarks used are insufficient. The authors conduct experiments primarily on synthetic datasets and a single real-world dataset (ETG). This limited scope is inadequate for demonstrating the method's effectiveness. More experiments on various modalities and datasets, such as the DomainNet dataset, which is a benchmark for domain adaptation, are necessary to showcase the generalization ability of the proposed method.\n   - The provided analyses are insufficient. While several ablation studies on simulations are included in Appendix C.2, they are not comprehensive. Essential missing analyses include (1) examining the main term $\\mathcal{L}\\_{\\text{guidance}}$ alone and (2) combining $\\mathcal{L}\\_{\\text{guidance}}$ with $\\mathcal{L}\\_{\\text{cycle}}$. These studies are crucial for understanding each term's effectiveness. Additionally, ablation studies should also be conducted on real-world datasets, not just synthetic ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided. \n\n**Q**: *The title \"Transfer Learning\" is too broad. The paper focuses solely on (few-shot and supervised) domain adaptation, where the upstream and downstream tasks are similar in nature. General transfer learning encompasses a wider range of tasks, including those with different downstream tasks from the upstream one. For instance, transferring pre-trained text-to-image models to controllable generation or text-to-video generation are broader and more complex tasks that fall under transfer learning.*\n\n**A**: Thank you very much for this question. Actually, our proposed framework is general enough to deal with more complex tasks, e.g., fine-tuning text-to-image models or text-to-video generation. These models can be viewed as the conditional generative model with text embedding as the condition. \n\n\n**Q**: *The paper does not address transfer learning across different label spaces. Even within the same generation tasks, domain adaptation requires identical label spaces for source and target domains, which is not always practical. For example, transferring pre-trained conditional generation models on ImageNet to other fine-grained datasets with different label spaces presents a more challenging problem than the domain adaptation addressed in this paper.*\n\n**A**: Thank you very much for this question. Our framework (conditional guidance) does not assume identical label spaces for source and target domains. In the ECG experiments, the label set of target domain is the subset of the source domain. When the source and target domain contain different class labels, our framework is still applicable, i.e., when $y_t\\neq y_s$, $$\\underbrace{\\mathbf{s} _{\\boldsymbol{\\phi}^*}(\\mathbf{x} _t, y _{t}, t)} _{\\substack{\\text {target source}}} = \\underbrace{\\nabla _{\\mathbf{x} _t} \\log p_t(\\mathbf{x}_t |  y_s)} _{\\text {pre-trained conditional model on source}}+ \\underbrace{\\nabla _{\\mathbf{x}_t} \\log \\mathbb{E} _{p(\\mathbf{x}_0|\\mathbf{x}_t, y_s)}\\left[\\frac{q(\\mathbf{x}_0, y_t)}{p(\\mathbf{x}_0, y_s)}\\right]} _{\\text {conditional guidance}}.$$ To generate an unseen class $y_t$, the key problem here is to choose a particular class from the source domain $y_s$ such that we can borrow more information from the source domain to generate this unseen class from the target domain. The coupling between $y_t$ and $y_s$ can be learned by solving a static optimal transport problem. \n\nWe show our framework is general enough to deal with homogeneous transfer learning together with heterogeneous transfer learning. More in-depth design (e.g. coupling solved by static optimal transport) can be left to future work.\n\n**Q**: *More experiments on various modalities and datasets are necessary to showcase the generalization ability of the proposed method.*\n\n**A**: Thank you very much for this suggestion. We refer to the \"general\" response and Table 1 in the attached file.\n\n\n**Q**: *While several ablation studies on simulations are included in Appendix C.2, they are not comprehensive.*\n\n**A**: Thank you very much for this suggestion. We refer to the “global” response and Figure 1 in the attached file.\n\n\n**Q**: *The consistency regularization requires optimization of the gradient term $\\nabla _{\\mathbf{x}_t} \\log h _{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, t\\right)$. Could the authors elaborate on the optimization process for this term?*\n\n**A**: Thank you very much for this question. Optimization of the gradient term is widely used by the meta-learning method. There is only one-line difference in practice to implement it, i.e., \"torch.autograd.grad($\\log h_{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, t\\right)$, $\\mathbf{x}_t$, retain\\_graph=True)\" to retain the computational graph for the backpropagation of the weights of guidance network $\\boldsymbol{\\psi}$, which can be found at line 249 in density\\_ratio\\_guidance.py. We will add additional clarifications to the revised version. \n\n**Q**: *Effectiveness of TGDP.*\n\n**A**: Thank you very much for this question. Both of the two claims are correct. (The training of the domain classifier is not significantly affected by the number of samples in the target domain since the domain classifier can achieve more than 90\\% accuracy learning from only 10 samples in the target domain. And the training of the guidance network with $\\mathcal{L}_{\\text{guidance}}$ does not require target data.) The key problems are: 1) The density ratio estimator may not be accurate in some regions since the coverage will not be large enough when the number of training samples is 10; \n2) The $\\mathcal{L} _{\\text{cycle}}$ and $\\mathcal{L} _{\\text{consistence}}$ depends on the number of samples in the target domain. \n\n**Q**: *Explanation of Figure 3.*\n\n**A**: Thank you very much for your question. From Table 2, we demonstrate improvements in the diversity and fidelity of TGDP against two baseline methods. \nFor each sub-figure in Fig 3, we get the embedding for T-SNE by using the same encoder from learning directly on target samples. If we change the encoder to the learned classifier by each method, we may see a big difference between Finetune Generator and TGDP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8rXTlZtNX", "reviewer": "Reviewer_brAM", "summary": "This paper introduces a new framework, the Transfer Guided Diffusion Process (TGDP), for transferring a pre-trained diffusion model from the source domain to the target domain. They connect the score function for the target domain and the score function of the source domain with a  guidance term related to the density ratio. They use a classifier to estimate the density ratio and extend TGDP to a conditional version. Besides, Cycle Regularization term and Consistency Regularization term are proposed to improve the performance.", "questions": "1. Is there a writing mistake in Line.162? It is hard to sample from p(x0|xt) rather than q(x0|xt)?\n2. What is the Lamma 3.2 in the conditional version?\n3. Do you find out other articles related to transfer learning for diffusion models and can you compare other methods with TGDP?", "limitations": "A limitation of this study is the lack of empirical validation regarding TGDP’s performance on language vision tasks.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduce a novel training paradigm for transferring a pre-trained diffusion model from the source domain to the target domain in a more efficient and effective way. The proposed guidance term proposed in this paper is a technique worthy of reference and further in-depth study by researchers. \n2. The paper is well-written and easy to understand. The theoretical analysis of the robustness is interesting and well described, which helps to understand the proposed techniques.\n3. Experimental results show the effectiveness of the proposed method.", "weaknesses": "1. The part of Extension to the Conditional Version could be more detailed. The authors could discuss what is the Lemma 3.2 in the conditional version.\n2. This paper is missing a comparison with articles related to transfer learning for diffusion models. In experiment part, the authors only compare the TGDP with vanilla diffusion model and finetune generator.\n3. Except Gaussian mixture simulations and benchmark electrocardiogram (ECG) data, the authors could provide more experiment results on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below, we address your concerns and comments. \n\n**Q**: *The authors could discuss what is the Lemma 3.2 in the conditional version. What is the Lamma 3.2 in the conditional version?*\n\n**A**: Thank you very much for this kind reminder. Yes, the key idea behind Lemma 3.2 is that the conditional expectation is the optimal solution to the least regression problem. It can be easily extended to conditional generation, i.e., $h _{\\psi^*}\\left(\\mathbf{x}_t, y, t\\right) = \\mathbb{E} _{p\\left(\\mathbf{x}_0 \\mid \\mathbf{x}_t, y\\right)}\\left[q\\left(\\mathbf{x}_0, y\\right)/p\\left(\\mathbf{x}_0, y\\right)\\right]$. We give the Lemma and proof here for the sake of completeness, and will add them to the revised version.\n\nLemma: For a neural network $h _{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, y, t\\right)$ parameterized by $\\boldsymbol{\\psi}$, define the objective  $\\mathcal{L} _{\\text{guidance}}(\\boldsymbol{\\psi}) :=\\mathbb{E} _{p(\\mathbf{x}_0, \\mathbf{x}_t, y)}\\left[\\left\\|h _{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, y, t\\right)-\\frac{q(\\mathbf{x}_0, y)}{p(\\mathbf{x}_0, y)}\\right\\|_2^2\\right],$\nthen its minimizer $\\boldsymbol{\\psi}^* = \\underset{\\boldsymbol{\\psi}}{\\arg \\min } \\ \\mathcal{L} _{\\text{guidance}}(\\boldsymbol{\\psi})$ satisfies:\n$h _{\\boldsymbol{\\psi}^*}\\left(\\mathbf{x}_t, y, t\\right)=\\mathbb{E} _{p(\\mathbf{x}_0 |\\mathbf{x}_t, y)}\\left[{q(\\mathbf{x}_0, y)}/{p(\\mathbf{x}_0, y)}\\right].$\n\nThe proof is straightforward and very similar to the unconditional version. Note that the objective function can be rewritten as \n$$ \\begin{aligned}\n\\mathcal{L} _{\\text{guidance}}(\\boldsymbol{\\psi}) :&=  \\mathbb{E} _{p(\\mathbf{x}_0, \\mathbf{x}_t, y)}\\left[\\left\\|h _{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, y, t\\right)-\\frac{q(\\mathbf{x}_0, y)}{p(\\mathbf{x}_0, y)}\\right\\|_2^2\\right] \\\\\\\\\n   & =   \\int _{\\mathbf{x}_t} \\int _{y} \\{\\int _{\\mathbf{x}_0}p(\\mathbf{x}_0|\\mathbf{x}_t,y) \\left\\|h _{\\boldsymbol{\\psi}}\\left(\\mathbf{x}_t, y, t\\right) -  \\frac{q(\\mathbf{x}_0, y)}{p(\\mathbf{x}_0, y)}\\right\\|_2^2 d\\mathbf{x}_0 \\} p(\\mathbf{x}_t|y) p(y) dy d\\mathbf{x}_t  \\\\\\\\\n   & = \\int _{\\mathbf{x}_t} \\int _{y} \\{ \\left\\|h _{\\boldsymbol{\\psi}}(\\mathbf{x}_t, y, t)\\right\\|_2^2  -  2 \\langle h _{\\boldsymbol{\\psi}}(\\mathbf{x}_t, y, t),  \\int _{\\mathbf{x}_0}p(\\mathbf{x}_0|\\mathbf{x}_t, y)  \\frac{q(\\mathbf{x}_0,y)}{p(\\mathbf{x}_0,y)}    d\\mathbf{x}_0 \\rangle \\} p(\\mathbf{x}_t|y) p(y) dyd\\mathbf{x}_t + C \\\\\\\\\n  &  =   \\int _{\\mathbf{x} _t} \\int _{y} \\left\\|h _{\\boldsymbol{\\psi}}(\\mathbf{x}_t, y, t) - \\mathbb{E} _{p(\\mathbf{x}_0 |\\mathbf{x}_t, y)}\\left[\\frac{q(\\mathbf{x}_0, y)}{p(\\mathbf{x}_0, y)}\\right] \n    \\right\\|_2^2 p(\\mathbf{x}_t|y) p(y) dy d\\mathbf{x}_t,\n\\end{aligned}$$\nwhere $C$ is a constant independent of $\\boldsymbol{\\psi}$. Thus we have the minimizer $\\boldsymbol{\\psi}^* = \\underset{\\boldsymbol{\\psi}}{\\arg \\min } \\ \\mathcal{L} _{\\text{guidance}}(\\boldsymbol{\\psi})$ satisfies $h _{\\boldsymbol{\\psi}^*}\\left(\\mathbf{x}_t, y, t\\right)=\\mathbb{E} _{p(\\mathbf{x}_0|\\mathbf{x}_t, y)}\\left[{q(\\mathbf{x}_0, y)}/{p(\\mathbf{x}_0, y)}\\right]$.\n\n**Q**: *This paper is missing a comparison with articles related to transfer learning for diffusion models. Do you find out other articles related to transfer learning for diffusion models and can you compare other methods with TGDP?*\n\n**A**: Thank you very much for this question. As far as we know, [1,2,3] explores approaches to fine-tuning diffusion models. They focus on methods that either significantly reduce the number of tunable parameters or introduce regularization terms to alleviate the overfitting for image data. Since fine-tuning only a subset of weights in diffusion models often yields results that are worse or comparable with methods that finetune all weights, we compared our method to full weights fine-tuning, which we believe serves as a strong baseline.\n\n[1] Moon, T., Choi, M., Lee, G., Ha, J., Lee, J., Kaist, A. Fine-tuning Diffusion Models with Limited Data. NeurIPS 2022 Workshop on Score-Based Methods.\n\n[2] Xie, E., Yao, L., Shi, H., Liu, Z., Zhou, D., Liu, Z., Li, J., Li, Z. DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 4207-4216.\n\n[3] Zhu, J., Ma, H., Chen, J., Yuan, J. (2023). DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data. ArXiv, abs/2306.14153.\n\n**Q**: *Except Gaussian mixture simulations and benchmark electrocardiogram (ECG) data, the authors could provide more experiment results on other datasets.*\n\n**A**: Thank you very much for this question. We refer to the \"general\" response and Table 1 in the attached file.\n\n**Q**: *Typo in line 162.*\n\n**A**: Thank you very much for your kind reminder. We have corrected it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewer for the valuable comments and suggestions. We appreciate the time you spent on the paper. We summarize the positive feedback that we received as follows:

Motivation and Novelty: The whole framework is innovative and reasonable (9nph); the proposed method for transfer learning is both original and significant. It should make a good contribution to the research community (53Zj); Provides a novel perspective on knowledge transfer for pre-trained diffusion models (UUwS); this paper introduce a novel training paradigm and the proposed guidance term proposed in this paper is a technique worthy of reference and further in-depth study by researchers (brAM).

Theoretical solid: The paper provides a solid theoretical basis for TGDP, including proofs for the optimality of the approach (9nph); the proposed regularizers that improve efficiency are in my opinion conceptually convincing (53Zj); theoretical foundation of the proposed method is well-constructed (UUwS); The theoretical analysis of the robustness is interesting and well described, which helps to understand the proposed techniques (brAM).

Writing Quality: The paper is well written and easy to follow (53Zj); the paper is well-organized with clear presentation (UUwS); The paper is well-written and easy to understand (brAM).

Effectiveness: The experimental evaluation on the ECG data is in my opinion well executed and convincing (53Zj); Experimental results show the effectiveness of the proposed method (brAM).


Secondly, we aim to address the common weaknesses and concerns raised by the reviewers. 

Firstly, all reviewers suggest providing additional experimental results on various data types and real-world datasets beyond the electrocardiogram (ECG) benchmark. Therefore, we conducted a preliminary experiment using an image dataset to verify the effectiveness of the proposed method. Given the time constraints and the computational cost associated with training diffusion models on large datasets (e.g., DomainNet) or fine-tuning/evaluating conditional diffusion models on language-vision tasks, we selected the MNIST to one-channel Street View House Numbers (SVHN) dataset for a preliminary verification. Our method achieves a lower Fréchet inception distance (fid) than the two baseline methods, which can be found in Table 1 in the attached file. Although the digit dataset is relatively simple, we believe these experimental results can still demonstrate the effectiveness of the proposed method.

Moreover, Reviewer 53Zj and UUwS point out that the ablation study on the effectiveness of two regularization terms is not comprehensive enough. The key reason we do not provide the ablation for all combinations involving $\mathcal{L} _{\text{guidance}}$ is that we considered $\mathcal{L} _{\text{consistence}}$ as an intuitive solution for the transfer setting, while $\mathcal{L} _{\text{guidance}}$ and $\mathcal{L} _{\text{cycle}}$ represent the key contributions of our proposed methods. Therefore, in the ablation study, we verify that using $\mathcal{L} _{\text{consistence}}$ alone does not yield a sufficiently effective solution. To better illustrate the contributions of the three terms, we provide the ablation studies, $\mathcal{L} _{\text{guidance}}$ and $\mathcal{L} _{\text{guidance}} + \mathcal{L} _{\text{cycle}}$, in Table 1 in the attached file.

Finally, we summarize our individual responses to each reviewer below. We appreciate the questions raised by the reviewers, which have helped improve the clarity of our paper.

Reviewer 9nph:
We clarified the computational advantages of training the guidance network compared to fine-tuning the diffusion model. Additionally, we discussed the scenarios in which our method performs well.

Reviewer 53Zj:
We enhanced the clarity of our paper by including detailed information about the 12-lead ECG data, the experimental setup, and the preliminary theoretical insights behind choosing the hyperparameters for the regularization terms.

Reviewer UUwS:
We clarified the applicability of the proposed framework, demonstrating its use for transferring text-to-image/video diffusion models and transferring knowledge across different label spaces. We also provided additional insights on Table 1 and Figure 3.

Reviewer brAM:
We improved the clarity of the conditional version of our framework by providing the necessary lemma along with the proof.

We believe these revisions address the concerns raised by the reviewers and have improved the overall clarity and strength of our paper. Should you have any further questions, please do not hesitate to let us know.

Sincerely, 

TGDP Authors.

Meta-review: Thanks to the authors and reviewers for engaging in discussion. Overall, the four reviewers leaned toward acceptance (1 accept, 2 weak accept, 1 borderline accept), where some lingering concerns involved making the experimental results more thorough (reviewer 53Zj) and having the discussion be more thorough (reviewer UUwS). In their response, the authors have provided some additional experimental results, although these appear preliminary and a bit limited in scope. I would encourage the authors to try to make the experiments even more thorough (now not having to worry about the tight turn around time from the author/reviewer discussion period), and to--as well as possible--address reviewer UUwS's remaining concerns within the manuscript text.
Decision: Accept (poster)