Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning
Authors: Ziheng Cheng
Tianyu Xie
Shiyue Zhang
Cheng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: conditional diffusion model
transfer learning
sample complexity
Primary area: theory
TL;DR: We provide the first theoretical sample-efficiency results of conditional diffusion models via transfer learning

Abstract: While conditional diffusion models have achieved remarkable success in various applications, they require abundant data to train from scratch, which is often infeasible in practice. To address this issue, transfer learning has emerged as an essential paradigm in small data regimes. Despite its empirical success, the theoretical underpinnings of transfer learning conditional diffusion models remain unexplored. In this paper, we take the first step towards understanding the sample efficiency of transfer learning conditional diffusion models through the lens of representation learning. Inspired by practical training procedures, we assume that there exists a low-dimensional representation of conditions shared across all tasks. Our analysis shows that with a well-learned representation from source tasks, the sample complexity of target tasks can be reduced substantially. Numerical experiments are also conducted to verify our results.

Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning
Ziheng Cheng', Tianyu Xie2, Shiyue Zhang', Cheng Zhang 2,3,+
1 Department of Industrial Engineering and Operations Research, University of California, Berkeley
School of Mathematical Sciences, Peking University Center for Statistical Science, Peking University edu, tianyuxie@pku edu, en, zhangshiyue@stuu pku. edu.en, chengzhang@math. pku. edu. cn
Abstract
While conditional diffusion models have achieved remarkable success in various applications, they require abundant data to train from scratch, which is often infeasible in practice. To address this issue, transfer learning has emerged as an essential paradigm in small data regimes. Despite its empirical success, the theoretical underpinnings of transfer learning conditional diffusion models remain unexplored. In this paper, we take the first step towards understanding the sample efficiency of transfer learning conditional diffusion models through the lens of representation learning. Inspired by practical training procedures, we assume that there exists a low-dimensional representation of conditions shared across all tasks. Our analysis shows that with well-learned representation from source tasks, the sample complexity target tasks can be reduced substantially. Numerical experiments also conducted verify our results.
Introduction
Conditional diffusion models (CDMs) utilize a user-defined condition guide the generative process of diffusion models (DMs) to sample from the desired conditional distribution. In recent years, CDMs have achieved groundbreaking success in various generative tasks, including text-to-image generation [Ho et al., 2020, Song et al., 2020, Ho and Salimans, 2022, Rombach et al., 2022], reinforcement learning [Janner et al., 2022, Chi al., 2023, Wang et al., 2022, Reuss al., 2023], time series [Tashiro et al., 2021, Rasul et al., 2021], and life science [Song et al., 2021, Watson et al., 2022, Gruver al., 2024, Guo et al., 2024].
Training CDM from scratch requires large amount of data to achieve good generalization. However, often have access only limited data for the target distribution due cost
in practical scenarios, or risk concerns, making the model prone to over-fitting. In such small data regime, transfer learning has emerged as predominant paradigm [Moon et al., 2022, Ruiz et al., 2023, Xie al., 2023, Han et al., 2023]. By leveraging knowledge acquired during pre-training on large source datasets, transfer learning enhances the performance of fine-tuning on target tasks, facilitating few-shot learning and significantly improving practicality.
Among the successful applications of transfer learning CDMs, the conditions typically high- dimensional vectors with embedded low-dimensional representations (features) that encapsulate all the information required for inference. In addition, these representations are likely be task-agnostic,
Conresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Tasks
Backbone Score Network
Condition Encoder
Texl-to-lmage [Esser et al., 2024]
2-8B
4.7B
Text-to-Audio [Liu et al., 2024]
350-750M
750M
Robotic Control [Chi et al., 2023]
9M
20-45M
Table 1: Comparing the number of parameters of different parts in CDMs.
enabling effective knowledge transfer. For example, in text-to- image generation, the text input inherently in nigh-dimensional space, but contains now-dimensional semantic information such as object attributes, spatial relationships, despite the differences of styles or contents in different image distributions. To take advantage of this structure, condition encoders are often frozen in the fine-tuning stage [Rombach et al., 2022, Esser et al., 2024], which typically constitutes significant portion of the overall model (see Table 1).
While this paradigm has demonstrated remarkable empirical success, its theoretical underpinnings remain largely unexplored. The following fundamental question is still open:
Can transfer learning CDMs improve the sample efficiency of target tasks by leveraging the representation of conditions learned from source tasks?
There some recent works attempting study the theoretical underpinnings of CDMs [Fu et al., 2024, Jiao et al., 2024, Hu et al., 2024], but focus on single task training. Notably, Yang et al. [2024] investigates transfer learning DMS under the assumption that the data a linear transformation of low-dimensional latent variable following the same distribution across all tasks. However, fine-tuning merely the data encoder is not widely adopted training approach in practice.
In this paper, we take the first step towards addressing the above question. Our key assumption is that there exists generic owwddimensional representation of conditions shared across all distributions. Then we show that, with a well-learned representation from source tasks, the sample complexity of target tasks can be reduced substantially by training only the score network. The main contributions are summarized as follows:
establish the first generalization guarantee for transferring score matching error in CDMs, showing that transfer learning can reduce the sample complexity for learning condition encoder in the target task. This is aligned with existing transfer learning theory in supervised learning. Specifically, we present two results in Theorem 3.4 and Theorem 3.6, under the settings of task diversity assumption and meta-learning", respectively. On the technical side, we develop a novel approach to tackle Lipschitz continuity under weaker assumptions on data distribution in Lemma 3.1, which may be of independent interest for the analysis of even single-task diffusion models.
In Section
In Section 4, we provide an end-to-end distribution estimation error bound in transfer learning CDMs. To obtain an 2 accurate conditional score estimator, we construct universal approx- imation theory using deep ReLU neural networks in Theorem 4.1. Then by combining both generalization еттог and approximation error, Theorem 4.2 and 4.3 provide sample complexity bounds for estimating conditional distribution. Notably, our results are the state of the art even when reduced to single-task learning setting.
In Section A, we further utilize our results to establish statistical guarantees in practical applications of CDMs. In particular, investigate amortized variational inference (Theorem A. and behavior cloning (Theorem A.2), and present guarantees in terms of posterior estimation and optimality gap, laying the theoretical foundations of transfer learning CDMs in practice. We also conduct numerical experiments in Section to verify our results.
II practice, the terms such transfer learning, meta- meta-learning, learning to-lean,, etc., often refer to the same training paradigm. i.e., to fine-tune tune on target tasks with limited data using knowledge from source tasks. However, in the theoretical framework, we use meta- learnin to emphasize that target tasks and source tasks are randomly sampled from meta distribution [Baxter, 2000], whereas in transfer learning, the tasks are fixed.

---

1.1 Related Works
Score Approximation and Distribution Estimation Recently, some works analyze the score approximation theory via deep neural networks and corresponding sample complexity bounds for diffusion models. Oko et al. [2023] considers distributions with density in Besov space and supported on bounded domain, Chen et al. [2023b] assumes the data distribution lies in low-dimensional linear subspace and obtains improved rates only depending on intrinsic dimension. Fu et al. [2024] studies conditional diffusion models for Hölder densities and Hu et al. [2024] further extends the framework to more advanced neural network architectures, e.g., diffusion transformers. Wibisono et al. [2024] establishes a minimax optimal rate to estimate Lipschitz score by kernel methods. With an L accurate score estimator, several works provide the convergence rate of discrete samplers for diffusion models [Chen et al., 2022b, 2023a, Lee et al., 2023, Chen et al., 2024]. Combining score matching error and convergence of samplers, one can obtain an end-to-ent distribution estimation error bound.
Transfer Learning and Meta-learning Theory in Supervised Learning The remarkable empirical success of transfer learning, meta-learning, and multi-task learning across wide range of machine learning applications has been accompanied by gradual progress their theoretical foundations, especially from the perspective of representation learning. To the best of our knowledge, Baxter [2000] is the first theoretical work on letannearnin. It assumes a universal environment generate tasks with some shared features. Following this setting, Maurer et al. [2016] provides sample complexity bound for general supervised learning problem and Aliakbarpour et al. [2024] studies very few samples per task regime. Another line of research replaces the environment assumption and instead establishes connections between source tasks and target tasks through various notions of task diversity (Tripuraneni et al., 2020, Du et al., 2020, Tripuraneni et al., 2021, Watkins et al., 2023, Chua et al., 2021]. However, theoretical understandings of transfer learning for unsupervised learning are much more limited.
Few-shot Fine-Tuning Diffusion Models Adapting pre- see-trained conditional diffusion models to specific tasks with limited data remains a challenge in varied application scenarios. Few-shot fine-tuning aims to bridge this gap by leveraging various techniques to adapt those models to a novel task with minimal data requirements [Ruiz et al., 2023, Giannone et al., 2022]. A promising paradigm use transfer (meta) learning by constructing a representation for conditions in all the tasks, which has been widely applied in image generation [Rombach et al., 2022, Ramesh et al., 2022, Sinha et al., 2021], reinforcement learning [He et al., 2023, Ni et al., 2023], inverse problem [TTwwrr et al., 2023, Chung et al., 2023], etc. Another work Yang et al. [2024] is closely related to this paper, proving that few-shot diffusion models can escape the curse of dimensionality by fine-tuning a linear encoder.
2 Preliminaries and Problem Setup
We use x and y to denote the data and conditions, respectively. The blackboard bold letter P represents the joint distribution of (Ir y). while the lowercase p denotes its density function. The superscript k indicates the task index, and the subscript means the sample index. The norm ||- || refers the f2-norm for vectors and the spectral norm for matrices. For the hypothesis class F. we use FOK refer its K-fold Cartesian product. For any a, R. b min(a, b} and a V = max(a, 6}. Finally, we use standard ((-), ((-) to omit constant factors.
Notations
2.1 Conditional Diffusion Models
Let de denote the data space and [o. 1 denote the condition space. Let IP be any joint distribution over dx x [0, 1] Pa with density p and P(-ly) be the conditional distribution with density p(-ly). As in diffusion models, the forward process is defined as an Orastein-Uhlenben (OU) process,
dX, = -X,dt + /2dW,,X, ~ IP(-l))
(2.1)
where (We) is standard Wiener process. We denote the distribution of X, P.(-ly). Note that the limiting distribution (-1g) is a standard Gaussian (O,
To generate new samples, we can reverse the forward process (2.1) from any T 0.
dx; (xF + 2V Pog lt) + /2aW, 1, X5 Pr(+))), t S T.
(2.2)
3

---

where { PosesT is a time-reversed Wiener process. Unfortunately, we don't have access to the exact conditional score function V Vlgg pr-+ and need estimate through neural networks. For any (xc, y) ~ IP and score estimator define the individual denoising score matching objective [Vincent, 2011]an as
[ E (1s) V log (x(al()))))
f(r,y. 21 = T T To
(2.3)
where (x) - of a, e- o = the transition kernel of x4 (20 = x.
And the population error of score matching is
I.F (8) == ((aay) [| y, 1) - V logre(<<ly)11 = ((es>~=(((,,)) y, ((,, 5. (2.4)
Here s.P denotes the true score function and t Unif( (To T). We also define (1r, y, s) ==
f(z, s) f(r, y, ). In practice, with a score estimator 3, the generative process is to simulate
dx.- = -8;- 20 : y. T 1))d + v2W 1, 86 ~N(0 5/,,)),,STT, To-
(2.5)
Here To > 0 is the early-stopping time. And the distribution of X is written P(-ly)
Note that we don't apply the commonly used classifier-fre: free guidance [Ho and Salimans, 2022] which has a tunable guidance strength since we mainly concentrate on sampling from conditional distribution instead of optimizing other objectives.
2.2 Transfer Diffusion Models via Learning Representation
Consider K source distributions over dd x 1,, pl. Pl...... IKK and a target distribution po. Suppose that for each source distribution pk 1 K, we have n i.idd samples {(=", (f))" Fd, and i...d. samples {<x] ) Ip are available for the target distribution, where typically m < n. In transfer (meta) learning setup, we assume there exists a shared nonlinear representation of the condition y for all distributions, i.e., the conditional distribution 1 = Pp a/A>. (v) for some
he : [0, 1) - [0, 1 (see also Assumption 3.2). Note that due to the shared features, the score of P (-1)) also has the form of V log P (x<ly) (x,, he (y), 8) for some fz.
Similar to Tripuraneni et al. [2020], our transfer learning procedures consist of two phases. In the pre- pre-tinining phase, the goal is to learn representation map h, through nK samples from K source distributions. Then during the fine-tuning phase, we learn the target distribution via m new samples and the representation map learned in the pre- pre-training phase.
Formally, let F, Further let F C F be the hypothesis class of score network in fine-tuning phase. In the pre-training phase, we solve the following Empirical Risk Minimization (ERM),
the hypothesis classes of score networks and representation maps, respectively.
K
= arg min JEFOR NEw nk
(2.6)
Then for the fine-tuning task, we solve
80. = arg min
Σ Cl Ul s..)).
(2.7)
feF
Here (z, = f(z, ((y), t) for f de 1] x To. T7 + t and h: 10, 00, 1 and l is defined (2.3).
In the meta-learning setting, we further assume that all the distributions {P*} are i.i.d. sampled from meta distribution PP Pmeet- Here Pmeta can be interpreted as universal environment [Baxter, 2000, Maurer et al., 2016]. In this case, we posit the existence of a shared representation map that holds for all P Pmeta- And the performance benchmark is then defined as the expected error on the target distribution pD IP PPsta-

---

2.3 Deep ReLU Neural Network Family
We use feedforward neural networks to approximate the score function and representation map. Let (x) = max(x, 0} be the ReLU activation. Define the score network family
B, R) == (f(r,,,) = (ALa()) + bL) o ... (A1(z' .ww , + b,)
A. E R ,b, E Rdtt,, dr+I max: S W, I. S M, Σ (11A:ll + [16ello) 5
S. max a V libllloo S B.llf(z, f(x, wt)l S 7/|w wlloo, V |10 S R, S T))
and encoder network NN W, = {h(v) = (ALO(.) + bL) ... (Aiy + b1)
A, E Edixd+t, by E Rd+1, :d++ dy max d; I W. W.llhlll==00011))) & 1, > A111 +
[bb:llo) I S, |A V l|billc S BY. Throughout this paper, we let F = (Lg, My,
Sy. By, Ry, T5) and H. = (Ln, Wh, Sp, Br) unless otherwise specified.
Remark 1. In practice, J C may (and typically will) depend on I for parameter efficient fine-tuning (PEFT), e.g, LoRA [Hu et al., 2021]. This will substantially reduce the complexity of F and further improve sample efficiency. The analysis of PEFT beyond the scope of this paper.
3 Statistical Guarantees for Transferring Score Matching Error
In this section, we present our main theoretical results, statistical theory of transferring the con- ditional score matching loss. We provide two upper bounds of the score matching loss on target distribution, based on whether task diversity [TTriprraneni et al., 2020] is explicitly assumed. Our analysis introduces novel techniques to address the smoothness properties of the noised data distribu- tion-a challenge that remains nontrivial even in single-task settings. Additionally, we extend the classical theory of local Rademacher complexity to quantify the empirical estimation error.
Throughout this paper, we make the following standard and mild regularity assumptions ('Tripuraneni al., 2020, Chen et al., 2023b] on the initial data distribution and the representation map h..
Assumption 3.1 (Sub-gaussian tail). For any source or target distribution TP, P is supported on Rd= x [0, 1| and admits continuous density p(z,8) y) C² x |0, 1) Moreover, the conditional distribution p(*ly) & C₁ exp(-C2l|æ) [= ()) for some constant C1 C2.
Assumption 3.2 (Shared con-dimeniional representation). There exists an L-Lipschitz function h, 1) [o, 1j with dy S Dy. such that for any source and target distribution P. the conditional density p(r y) g. (z. h. hy)) for some g.¹ E CR x [0, 11dx).
Equivalently, h.w (v) is a sufficient statistic for x, which indicates that Pr pr(xly) = pr(2//h (v)). There- fore, with a little abuse of notation, for any w E [0, 1) we define p(æ; w) p(x]h, (2) = w) (z, w). Also note that by definition, for any x, y. we have p(æ; he (y)) = p(z)/h, (3)) = p(zly). Assumption 3.3 (Lipschitz score). For any source and target distributioo IP and its density function p. the conditional score V, log p(æly) Va log g (x, hv(y)). The score function V lol g. (z, w) L-Lipschitz in x and w. And || log g. (0, w)ll S B for some constant B and any w.
3.1 Tackling Lipschitz Continuity under Weaker Assumptions
Notice that we only impose smoothness assumption on the original data distribution p(-ly), instead of the entire trajectory pe(-lg) in forward process. This is substantially weaker than the Lipschitzness assumption required in Chen et al. [2023b, 2022b], Yuan et al. [2024], Yang et al. [2024]. However, Lipschitzness of loss function l and class J is a crucial hypothesis in theoretical analysis of transfer learning [Tripuraneni et al., 2020, Chua et al., 2021]. The intuition is that without Lipschitz conti- nuity of the score network f. it is generally impossible to characterize the error from imperfect representation map h. Hence is inevitable to show the smoothness of pe (-1g) to some extent.
Fortunately, even with assumptions merely on the initial data distribution, we are still able to prove smoothness of the forward process in any bounded region, as shown in the following lemma. The proof can be found in Appendix B.I.
5

---

Lemma 3.1. Under Assumption 3.1, 3.2, 3.3, for any w E [0.1]4, denote the conditional score forward process V, log Pr w) by fa w.t). There exist constants Cx Cxx, such that for any R > O, the function (z, w, is +Cx+ -Lipshiit I, (Cx + Cx ()) -ipschitt in w, the domain BR x [0, 1] 1 [0, T. Here BR denotes the ball with radius centered at the origin.
3.2 Results under Task Diversity: Sample-Efficient Transfer Learning
In the literature of transfer learning, task diversity is an important assumption that connects target tasks with source tasks (Tripurameni et al., 2020, Du al., 2020, Chua et al., 2021]. In the context of conditional diffusion models, we state the formal definition as follows.
Definition 3.1 (Task diversity). Given hypothesis classes F, H. we say the source distributions pll P!.... IP are (v, divivers over target distribution p0, if for any representation H.
(sport) - inf E
inf
(sjeN)
(3.1)
v SETON
Here L is defined in (2.4). This notion of diversity ensures that the representation error on the target task caused by h can be controlled by the error on the source tasks, thereby establishing certain relationships in between. More detailed discussions are deferred to Appendix B.5.
We first present the generalization guarantee for each phase respectively.
Proposition 3.2 (Fine-tuning phase generalization). Under Assumption 3.1, 3.2, 3.3, for any E H. the population loss of jo can be bounded by
(( (2.) _ (z,8, 13.8pi sp.
inf Po (2, log" (m)r.: (3.2)
JEF
No and log Ny is some complexity measures of F.
where va
Proposition 3.3 (Pre-training phase generalization). Under Assumption 3.1. 3.2, 3.3, if Ry 2 logă (nKM,/8), with probability no less than 1 8, the population loss can be bounded by
K
K
log(1/0) nK
Σ Amm
B(a e"
(I, (I,91
inf JEFaK NEH K
Eam (x, (fon://
(3.3)
K log log NH nK
where T2
and log Ny, log No are some complexity measures of F, H.
Combining these two propositions with the notion task diversity in Definition 3.1, we are able to show the statistical rate of transfer learning as follows.
Theorem 3.4. Under Assumption 3. 3.2, 3.3, suppose ".....,,P" are (v, A)-diverse over target distribution po given F, H. If R2 Z log (mKM,/0) / then with probability no less than 1 5,
K
E (( ((ag))) / (2 (e.u.sp,
S inf hEN
inf
[e (I,Y. A (aw))
K
fef
#=1
log (m) log N
lng)())))) K log No log(Nzz/)) +
m
vnk
(3.4)
where
log Ns = M1 log W loM(/)))
(3.5)
log Ngg: = S, log MKLL,W,(BB, V 1)M log(1/6)).
The formal statements and proofs are provided in Appendix B.2.

---

K
Let Eapprox
inf NEH
Σ
inf fer
(n 1e (z, U. 8f.)l
the approximation error. The lead-
K
ing terms can be simplified to ① Eappeix + K loggN log
logNF where log and
nk
mn
logAA capture the complexity of the hypothesis classes.
Improving Sample Efficiency Theorem 3.4 demonstrates the sample efficiency transfer learn- ing. Compared to naively training the full CDM for target distribution, which has error of Sapprox + No + log NH transfer learning saves the complexity of learning H and thus the
performance much better when is relatively small to n, K (i.e., in few-shot learning setting).
3.3 Results without Task Diversity: Meta-Learning Perspective
The results in previous section heavily depend on the task diversity assumption, which is hard to verify in practice. An alternative to consider meta-leariing setting, where all source and target distributions are sampled from the same enviromment, i.e., meta distribution.
For any C((0, P=; ; [0, 11() and distribution over Rd x [0, 1)",, define the representation error
C(P, h) = inf (( (z, (z, s(h,) 2 0.
(3.6)
JEF
We characterize the generalization bound of source tasks on the entire meta distribution as follows.
Proposition 3.5 (Generalization on meta distribution). Under Assumption 3.1, 3.2, 3.3, there exists constant Cp such that for =k=1
itd. P
Pmera with probability no less than - 6,
2 K
C(P*, ,h) - Cp rpp log(1/8)
RewPan ((P,h)
S
(3.7)
K
K C(Pk,
h) 2E2 C(P,h) Cpp (rp+ log(1/6)
(3.8)
holds for any h E H, where rp = expe Mxxx + Sn log (KLaWa(B, V 1)M
K
Theorem 3.6. Under Assumption 3.1, 3.2, 3.3, Ry Z log* (nK ( (5), then with probability no less than 8, the expected population loss of new task can be bounded by
Epo (( ~ (x.,)) P0 18t (v, 87,
< inf Ep inf E ((x))== (z.f(x..)) (I,,, + log (m) log log"(nK/o) log
log(WH/0)
hEHt
feF
m
n
K
(3.9)
where Nog,N,, Nu are defined in (3.5).
The formal statements and proofs are provided in Appendix B.3.
Let aapprox Ep-Fran inf ((x (e (z, y, 8 s.N) be the approximation error in meta- veta-learning. JEJ
bEH
log log.Nx
Different from transfer learning bound in Theorem 3.4, the leading term decays only in K and not in This is because that without task diversity assumption, the connection between source distributions and target distributions can only be constructed through meta distribution. And according to Proposition 3.5, the source distributions p1 ..... Ip collectively form K-shot empirical estimation of Pretax leading to an estimation error (11/K). Despite this, Theorem 3.6 still demonstrates the sample efficiency of meta-learning compared to naive training method when m is small and K sufficient large.
The results above can be further simplified to δ Tapprox m. An +
K
7

---

End-to-End Distribution Estimation via Deep Neural Network
Section provides statistical guarantee for transferring score matching. In this section, we establish an approximation theory using deep neural network to quantify the misspecification error. Combining both results we are able to obtain an end-to-end distribution estimation error bound for transfer learning diffusion models.
4.1 Score Neural Network Approximation
The following theorem provides guarantee for the ability of deep ReLU neural networks to approxi- mate score and representation. The proof is provided in Appendix C.I.
Theorem 4.1. Under Assumption 3.1, 3.2, 3.3, to achieve R₃ 2 log (nKM, and
K Σ k=1
inf bEH
inf (x.g)== 1et (x. 8f,)) = (I (nK((=)))e") (transfer learning) (4.1)
K
FEF
inf EewPons hER
inf (ax)) 1 (z, U. sfn) O (nog))n(((s))))))) (meta-learning)
(4.2)
fer
the configuration of F and H. should satisfy
[1og(k/((0)) ))
log (og"(d=++d))/2 (nKK))()))
b O log
od, editd,t1T
E
8 log 1oggld+ (oog (nKK)// (nK/(66)) (1022//n//(66)))
B, 00 =B (Tlog (nK((c5))
(4.3)
+1
E
R O (log*) (nK (nK/(=)))) , (log" (nK((EE)))) .78 (log(nK(<30)).
log(1/=)) Sh. 0 (5 (E-D: (1/E) B, = 0(1. (4.4) Here (-) hides all the polynomial factors of d, dy. Dy, C₁, C2, B.
LA O (log(1/E)), W, =
Universal approximation of deep ReLU neural networks a bounded region has been widely studied [Yarotsky, 2017, Schmidt-Hieber 2020]. However, we have to deal with an unbounded domain here, hence more refined analysis is required, e.g. truncation arguments.
In addition, traditional approximation theories typically cannot provide Lipschitz continuity guaran- tees, which crucial in transfer learning analysis. Following the constructions Chen al. [2023b]. the Lipschitzness restriction doesn't ''t compromise the approximation ability of neural networks, while ensuring validity of the generalization analysis Section 3.
4.2 Distribution Estimation Error Bound
Given the approximation and generalization results, we are in the position of bounding the distribution estimation error of our transfer (meta) learning procedures. The formal statements and proofs be found in Appendix C.2.
Theorem 4.2 (Transfer learning). Under Assumption 3.1, 3.2, 3.3 and (v, Aidvieersit with proper configuration of neural network family and T, To. it holds that with probability at least 1 8,
log (nK/5) log looglllg/m///// ((m/v) An)
log" (nK/5)
E (( (m (
PO P
+VA.
vト ((m/v) A n) Ze 40y
1ト (n()) Dz4z
(4.5)
Theorem 4.3 (Meta-leaming)) Under Assumption 3.1, 3.2, 3.3 and meta-learnin setting, with proper configuration of neural network family and T, To. it holds that with probability at least 5,
log (nK/0) log (m An) log" (nKK()/))
EaporPPn ((( Ppo Byywp TV pO ( S
(m A n) -.
K 2yya
(4.6)

---

100
fine tuning
14.47
3.68
2.45
1.82
1,9
0.91
fine- tuning
6.14
2.65
1.61
1,08
0.96
0.45
friis-from-seaacc
21.99
10.61
2.38
1.04
train from scratch
24.41
20.62
18.67
13.49
7.03
1.23
Table 2: MSEs for 90 5.5.
Table 3: MSES for 30 15.
Theorem 4.2 and 4.3 again unveil the benefits of transfer (meta) learning for conditional diffusion models, with a rate of ((m A A 4.4dy+5 + (nK) -y+z or Ö(m n) d.-dy + 2y4z
To compare, naively learning the target distribution in isolation will yield Om -.+ When the condition dimension D3 is much larger than feature dimension dy. transfer (meta) learning can substantially improve sample efficiency on target tasks, thanks to representation learning.
Comparison with Existing Complexity Bounds of CDMs Fu et al. [2024] studies conditional diffusion model for sub- gaussian distributions with 3-Hölder density, Since the Lipschitzness of score is analogous to the requirement of twice differentiability of density [Wibisono et al., 2024], reasonable to let 33 2 for a fair comparison.
this case, the TV distance is bounded
by Ö( Mda Py12) ) with sample size m according to Fu al. [2024], which worse than our naive bound Ö(m -x+ due to the inefficiency of score approximation. We also aware of another work [Jiao et al., 2024] that assumes Lipschitz density and score, obtaining a rate of Õm Fds T Bi12 12y31)
Relation to Yang et al. [2024] Unlike our setup. Yang al. [2024] considers transfer learning unconditional diffusion models with only one source task, i.e. D, dy 0, K = 1. The unconditional distribution is assumed to be supported in low-dimensional linear subspace, where the source task and the target task have the same latent variable distribution. Hence, only a linear encoder is trained for fine-tuning instead of the full score network. In this case, Yang et al. [2024] w(m) Ag+
is able to bound the TV distance by Ö(m - + target task. However, the assumption on shared latent variable distribution is stringent and we believe our analysis methods can be extended to this setting well.
) escaping the curse of dimensionality for
5
Experiments
Our theoretical results can be readily applied in various real world settings. In Appendix A, we investigate amortized variational inference and behavior cloning utilizing our theories, providing statistical guarantees of practical applications of CDMs. In addition, we conduct experiments on both synthetic and real world data numerically verify the sample efficiency of transfer learning.
Conditioned Diffusion The first numerical example is the high-dimensional conditioned diffusion [Cui et al., 2016, Yu et al., 2023] arising from the following Langevin SDE
du, = B1, (1 dd + dw,, uy =0 0,
(5.1)
where 3 >> 0 and w, is one-dimensional standard Brownian motion. The SDE (5.1) is discretized by the Euler- Euler-Maruyama scheme with step size of 0.02, which defines the prior distribution P. (1 for the (discretized) trajectory x = (ua.02, uo.04, u1.00) E R-50 We consider a conditional Gaussian likelihood function, p(y] projection matrix. Given a set of pre-selected (B:; < k < K} with 8y = k and K = 10, the k-th joint source distribution is given by (x y) = Ps, (z)).) The target distribution P0 (z, y) is given by 3g 5.5 (in-domain) or 3o 15 (fut-of domaain... More details are found Appendix 1.
((Mr, I /100/ 14). where M E R 100x50 is pre-defined
We report the MSFs of the estimated posterior mean of PP (xly) on the test samples in Table and 3. We see that across different values of and m, the fine-tuned models can provide significantly more accurate posterior mean estimations in most cases, suggesting the effectiveness of the representation map h learned in the pre-training phase. Notably, as the number of fine-tuning samples m increases, the performance gaps between fine-tuned models and rrain-from-scrachh models get smaller, since more training samples yield more generalization benefits and thus less dependence the pre-trained

---

m
10
20
30
40
50
100
fine-tuning
0.3799
0.2846
0.2544
0.2406
0.2404
0.2268
0.4409
0.3180
0.2746
0.2551
0.2501
0.2344
Table MSEs on the image restoration task.
model. This is aligned with our theoretical results. We also notice large variance among the results of different replicates, and attribute the slightly worse performance of fine-tuned models at m 50, So 5.5 to the potential randomness.
Image Restoration For real data experiment, consider the image restoration task on MNIST. We have K 9 source tasks with p (x, v) = pa (r)p(v())) where the prior (xr) is the data distribution of the digit & in the MNIST data set (1 S S K) and p(y|æ) = N (z. I78//4) 4). The target task is TP" (x, y/ = po (=)))p())z) where po(x) is the data distribution of the digit O. We use the full MNIST 1-9 data for pre-training which corresponds to n 5000. For the finetuning phase, we consider m = 10, 20, 30, 40, 50, 100 training samples and 100 test samples from IP' (xx y). More details can be found in Appendix E.2.
We report the MSEs between estimated posterior mean of P y) = (y) ()) and the ground truth sample on the test samples in Table 4. We see that for all fine- viee-tunigg sample sizes the results obtained by fine-tuning consistently outperform those obtained by training from scratch, indicating the benefits of transfer learning. Similarly to the experiment on conditioned diffusion, we also observe a reduced performance gap as m increases.
6 Conclusion and Discussion
In this paper, we take the first step towards understanding the sample efficiency of transfer learning conditional diffusion models from the perspective of representation learning. We provide generaliza- tion guarantee for transferring score matching in CDMs in different settings. We further establish an cod-to-end distribution estimation error bound using deep neural networks. Two practical applications are investigated based on our theoretical results. We hope this work can motivate future theoretical study on the popular transfer learning paradigm in generative Als.
Although this work provides the first statistical guarantee for transfer learning in CDMs, it has several limitations that we plan to address in future research. First, our theoretical results heavily rely on the task diversity notion introduced in Section 3.1, which can be challenging to verify in practice. While we provide some preliminary empirical evidence in Appendix B.5, a more fine-grained theoretical and empirical analysis will be essential for deeper understanding of CDMs. Second, our analysis focuses on the ERM estimator, whereas in practice, fine-tur line-tuning typically starts from pre-trained model and may employ techniques such as LoRA. Incorporating these settings would allow for an optimization based perspective on the sample efficiency of transfer learning. Finally, in our current formulation, the sample efficiency gain arises from reducing the complexity associated with learning the conditional encoder. Consequently, our results primarily apply to CDMs in which the conditional encoder constitutes substantial part of the overall model. Extending the theory to settings where this assumption does not hold is an important direction for future work.
Acknowledgements
This work was supported by National Natural Science Foundation of China (grant no. 12201014, grant no. 12292980 and grant no. 12292983). The research of Cheng Zhang was support in part by National Engineering Laboratory for Big Data Analysis and Applications, the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQFF) of Peking University. The authors are grateful for the computational resources provided by the hegh-peeffrrance Computing Platform of Peking University. The authors appreciate the anonymous NeurlPS reviewers for their constructive feedback.
10

---

References
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
Maryam Aliakbarpour, Konstantina Bairaktari, Gavin Brown, Adam Smith, Nathan Srebro, and Jonathan Ullman. Metalearning with very few samples per task. In The Thirty Seventh Annual Conference on Learning Theory, pages 46-93, PMLR, 2024.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Olivier Bousquet. Concentration inequalities and empirical processes theory applied to the analysis of learning algorithms. PhD thesis, ficole Polytechnique: Department of Applied Mathematics Paris, France, 2002.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763. PMLR, 2023a.
Minshuo Chen, Wenjing Liao, Hongyuan Zha, and Tuo Zhao. Distribution approximation and statisti- cal estimation guarantees of generative adversarial networks. arXiv preprint arXiv:2002.0399 2020.
Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Nonparametric regression low- dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: Journal of the IMA, 11(4): 2022a.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on oo-ddimesiiona data. International Conference on Machine Learning, pages 4672-4 4712. PMLR, 2023b.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint 0rXiv:2209.11215 2022b.
Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode provably fast. Advances Neurul Information Processing Systems, 36, 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal Robotics Research, page 027836492412736668 2023.
Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34:8871-8884, 2021.
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22542-22551, 2023.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
Tiangang Cui, Kody JH Law, and Youssef M Marzouk. Dimenssinn-independent likelihood-informees meme. Journal of Computational Physics, 104:
Simon Du, Wei Hu, Sham Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint 2:Xh::2002.094444, 2020.

---

Patrick Esser, Sumith Kulal, Andreas Blatimann, Rahim Entezari, Jonas Müller. Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory, arXiv preprint arXiv:2403.1 2024.
Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15563, 2022.
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances neural information processing systems, 36, 2024.
Zhiye Guo, Jian Liu, Yanli Wang, Mengrui Chen, Duolin Wang, Dong Xu, and Jianlin Cheng. Diffusion models in bioinformatics and computational biology. Nature reviews bioengineering. 12):136-1544 2024.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems. 36:64896-6444 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0962. 2021.
Jerry Yao-Chieh Hu, Weimin Wu, Yi-Chen Lee, Yu-Chao Huang, Minshuo Chen, and Han Liu. On statistical rates conditional diffusion transformers: Approximation, estimation and minimax optimality, arXiv preprint arXiv:241 1...:2211,.7522,, 2024.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 1XXi:2205.0099) 2022.
Yuling Jiao, Lican Kang, Jin Liu, Heng Peng, and Heng Zuo. Model free prediction with uncertainty assessment. arXiv preprint 240.::2405..284 2024.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of corre-bsed generative modeling for general data distributions. In International Conference Algorithmic Learning Theory, pages 946-985. PMLR, 2023.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Pascal Massart. About the constants in talagrand's concentration inequalities for empirical processes. The Annals of Probability, 28(2):863-88) 2000.
Andreas Maurer, Massimiliano Pontil, and Bernardino Pamerr-Parreds.. The benefit of multitask
representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
12

---

Tachong Moon, Moonseok Choi, Gayoung Lee, Jung- Woo Ha, and Juho Lee. Fine-tuning diffusion models with limited data. In NeurlPS 2022 Workshop on Score- Based Methods, 2022.
Fei Ni, Jianye Hao, Yao MU, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pages 26087-26105. PMLR, 2023.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators, In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023,
Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offiine meta-reinnfrrcement learning with online self-supervision. In International Conference on Machine Learning, pages I 17811-17823 PMLR, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference Machine Learning, pages 8857-888888888 8868. PMLR, 2021.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint wrXiv:2304.02532, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pages 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject- generation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-225 10, 2023.
Johannes Schmidt-Hieher Nomparametric regression using deep neural networks with ReLU activa- tion function. The Annals of Statistics,, 48(4):1875 1897, 2020. doi: 10. 10.1214/9-A001877 URL https://doi.org 10. 1214/19-A0S1875
Abhishek Sinha. Jiaming Song, Chenlin Meng, and Stefano Emnon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356. 2020.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffu- sion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-2453 2021.
Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Durand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems, 36: 12349-12362, 2023.
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852-7862. 2020.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning, pages 10434-10443, PMLR, 2021.
Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University). 2(3):2-3, 2014.
13

---

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2367):1611-1674, 2011.
Martin Wainwright. /ligh-dimensional statistics; non-asymptotic viewpoint, volume 48, Cam- bridge university press, 2019.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061193 2022.
Austin Watkins, Enayat Ullah, Thanh Nguyen-Tang and Raman Arora. Optimistic rates for multi-task representation learning. Advances Neural Information Processing Systems, 36:2207-2251, 2023.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen E Eisenach, Woody Abern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. BioRxiv, pages 1022-12,-0 2022.
Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv preprint ExXv:2402.07744: 2024.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Diffii:: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4230-4239, 2023.
Ruofeng Yang. Bo Jiang. Cheng Chen, Ruinan Jin, Baoxiang Wang, and Shuai Li. Few-shot diffusion models escape the curse of dimensionality. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL JrraNaaZm5.
net/formm?id
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural networks, 94: 103-114, 2017.
Longlin Yu, Tianyu Xie, Yu Zhu, Tong Yang, Xiangyu Zhang, and Cheng Zhang. Hierarchical semi-implicit variational inference with application to diffusion model acceleration. Advances in Neural Information Processing Systems, 36, 2023.
Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang, Reward- conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36, 2024.
14

---

A
Applications
We explore two applications of transfer learning for conditional diffusion models, supported by theoretical guarantees derived from our earlier results. In particular, we study amortized variational inference and behavior cloning. These real-world use cases not only validate the applicability of our theoretical findings but also lay the foundations of transferring diffusion models in practice.
A.1 Amortized Variational Inference
Diffusion models have exhibited groundbreaking success in probabilistic inference, especially latent variable models. We study a simple amortized variational inference model, where the observation given latent variable is distributed according to an exponential family Fi with density
(v(cx) - (v) exp (vr (v)) - (x)))
(A.1)
where v E Vr is non-nege non-negative and supported on [0. 1]_ and h. (v) E 00 11. Note that we also have d d in this case. The prior distribution of variable x is denoted as Pe for some E $ Let 0 (1b σ) and we aim to sample from the posterior distribution of po (xly) X Po (x) exp (z, (v)) - As (x)). Due to the special structure, the posterior pa (zly) only depends on the low-dimensional feature h. (v), shared across all 0 O V x Φ. This formulation encompasses various applications including independent component analysis [Comon, 1994], inverse problem [Song et al., 2021, Ajay et al., 2022] and variational Bayesian inference [Kingma, 2013].
(v) oc
Consider source tasks consisting of 0¹ ..... , E O, and for each 6 we have n ii.d samples {(*7 ..t. )/-)- For the target task 8", we only have m samples {(x] ... "/ We conduct our transfer learning procedures to train conditional diffusion models Poo (-ly). For theoretical analysis, we further impose some assumptions on the probabilistic model as follows.
Assumption A.1. The prior distribution satisfies po(x) S C1 exp(-Ca||=|l*) and V po is L-Lipshcitz in x, V log P& (0)ll S B for any 6 E T. The representation h, is I-Lipschitz. The integral f (())dy E [1/C,C] any vb
Theorem A.1. Suppose Assumption A.1 holds. Then under meta-learning setting, we have with probability no less than
8, 6,
Ego (o2 Ey- TV (oo( S log (nK/8) log (m An)
log (nK/o)
(m A n) 2019
K 2012
(A.2)
If (v, A)-diversity holds, then we have with probability no less than
[(ie w()m Eny- o0 (TV (+l)), Poo log² (nK/5) log"
((m/v) log" A An) log((n//))
+VA.
ad ((m/v) n)zzTo ž
vz(KK) Diz
(A.3)
The proof is deferred to Appendix D.1. We show that under mild assumptions, transfer (meta) learning diffusion models can improve the sample efficiency for target task in the context of amortized variational inference. This error bound can be further extended to establish guarantees for statistical inference such as moment prediction, uncertainty assessment, etc.
A.2 Behavior Cloning via Meta-Diffusion Policy
Although originally developed for image generation tasks, diffusion models have recently been extended reinforcement learning (RL) [Janner al., 2022, Chi al., 2023, Wang al., 2022], enabling the modeling of complex distributions of dynamics and policies. In the context of meta-RL, some works have further utilized diffusion models for planning and synthesis tasks [Ni et al., 2023, He et al., 2023]. In this application, we focus on a popular framework of behavior cloning, diffusion policy [Chi et al., 2023], which uses conditional diffusion models to learn multi-modal expert policies in high-dimensional state spaces. In such settings, the state often corresponds to visual observations of the robot's surroundings, such as high resolution images, and thus typically share low-dimensional underlying representation.
15

---

Let M be the space of decision- decision-making environments, where each M E M is an infinite horizon Markov Decision Process (MDP) sharing the same state space S, action space A, discount factor ? and initial distribution / (S)) And each M E M has its own transition kernel TM S x A - (SS), and reward function r' S A + [0, 1]. The policy is defined as a map - A(A). The value function of MDP M under policy is
r'rm(se, an) oor ~ x(-16)). (Tr(())),a)),
(T, 80) E
(A.4)
t=0
(T)(T) == Exo (TT,
00
Denote the visitation measure as dīr (s, a) == (1 (1- 2))
Σ''(ss PP(s 50
1=0
Suppose there are K source tasks M "..... M K M, and the expert policy of each task is denoted as T.. In behavior cloning, for each source task M have pairs of {(s a it dt d d The goal is to imitate the expert policy of target task M E EM, of which the sample size only YT <
To unify the notation, let x a. y s and assume
R S [0. and representation
space [0, 1j Our meta diffusion-policy framework aims to learn a state encoder h S + [0, 1jd. during pre-training. which acts shared representation map in different MDPs and consequently enhances sample efficiency on fine-tuning tasks. Let @0 be the learned policy in fine-tuning phase.
The following theorem shows the optimality gap between the learned policy and the expert policy.
Theorem A.2. Suppose the expert policy &² satisfies Assumption 3.1, 3.2, 3.3. Then under meta- learning setting, it holds that with probability no less than 1 6,
log 2 (nK/0) log (m A An)
log (nK/0)
E E ((ff))))) (TT) Vo (Ta)
7)²
(m A n) 3. +2,+
K 1r5+z
(A.5)
we further assume ".. , ग. are (v, A)-diverse over nh then the gap can be improved by
1
log (nK/o) (ogK//)lo"(m////1 (m//v) An)
log log2(n (nk/())
(ao (Vo VM (7
VA
(1 (1~1)² 7)2
uà (mm/v)) A An) Fe
v2(KK) D54z (A.6)
The proof can be found in Appendix D.2. This provides the first statistical guarantee of diffusion policy in behavior cloning. Notably, in both cases, the number of source tasks K has an exponential dependence D,, further suggesting the importance of data coverage when tackling distribution shift in offline meta-RL [Pong et al., 2022].
B Proofs in Section 3
B.1 Preliminaries
Lemma B.I. xf ~ p(zoly), the density of forward process Pr can be written as
exp (- || 110=020011)
o2toll3)
pe(xly)) - (ollooo)pooooa
(1(x)(o)) (2TO?) 4
(B.1)
20
Besides, the score function has the form of
V log (x = / Va log dr (xc(oo))))
de (=( drg (z)
(B.2)
dr (2(2o)) dxo. (o [
V Vo log p(gol)))))
(B.3)
0x
16

---

Proof. (B.1) can be directly implied by the definition of forward process. And it yields
Vx logp:(<ll)) = Vape(aly)
pr(xly)
(B.4)
( ((
v. log or s) (m)) dr (1(((2)))((01))) dro.
which is (B.2). Moreover, noticing that V (x parts,
2 (x( then by integration by
a
J V Vacoellooooltooooddd
1 (100
J de (2) g)d
0t I (o yr
(ar(2r) ((ro I [ drd
(B.5)
at
04 /
2 logp(soly) (c(r wv) dro-
Hence (B.3) is proved.
Lemma B.2. [Lem. 3.1] For any E [0, 1] denote the conditional score of forward process
V log (x;;; w) by f-(x, W. t). Then there exist constants Cx C'x, such that for any R > O, the function Ja (ar, w. 1) is (Cx + C'x L-ipsshit in x, (Cx + C'x Rip-iihhhiizz in w, the domain BR x [0, 11.1 x [0, T). Here BR denotes the ball with radius R centered at the origin.
Proof. Define density function (xola, w) x <0(x)<o)p((0: w). Our proof strategy will depend on
whether t 2 2(L 1)
When 2 2(L+1)) according to (B.2), we have
u,, V3 log (c w)
(xol
V log ( + Var (ar log
(B.6)
002000
Var (20
of
o
For any R 0, we have
(a:t0
4)
x
/
0+J0 =1
oe (1(2oo)))(z)ll) dæø
Var (mola,w)
of
Te
0:z0- = 112 exp
|| 110120-2113)
p(ao; w)dxo
R2
0120-x
ER
at
207
01
of f exp
plzui w)dro
20
1 exp(-
R²a >p<oo w wddor
R12
+
o7
1
|SR/2
exp(
Rc )p (xo; w)dx
(B.7)
2||0:|| 2Cb
No: : 01500 - -*1| < R/2] includes {oo ||o0|| 1 Co
Let R
then the domain
indicating
/
p( w wddry= 2
w)dw 2 1 - 2 Coxpp-C''C 2'
-1000 |
(B.8)
1 2
piao; w) ddo S
plzo; w) w)don S
114001 91
12
110011200
17

---

and
1 VV, w, t)ll $ of
a,30 - 2
R
2 of
|||=||
808
+ 20
Var
(B.9)
o;
of
Similarly, for have
Vwla(2, w.
Cov
Vv log (ar((r)) V logp(ro;;
(B.10)
Cov (xoli))))
V log log
of
which implies
(r, we t)1l < B Var (wolor
oî
(B.11)
[2 + 2Co + 1)
de
ai
When / 1)
have of
and
2T.
(x, 2 Pe;;e(
(ip((a;
Va De (z)
(((x;w)
1 S [
V w w)dro
V (e
af
(e w)
1
Villo; u))
E (2alaaa)))
Va log (V, (V.
af
p(zu; w)
1
E (20l
w)) - V logp(z0; Vox ]]
V, log Pa
V ([.
(B.3)
E (xolu.))))))
logr(so; + a?
Varg (V. log p(2o; w)).
V
(B.
a
|a:00 - x11 20
Note that when o² <
the distribution (= w) & exp
pl p(zo: w) is L-
2L
strongly log-concave, and thus satisfies the Poincare inequality with constant L [Chen et al., 2023a].
Var (x (V log p(ro w)) I. E V
log (oggg( s L.
(B.13)
And thus
21
(z, t)ll s
(B.14)
at
Analogously,
V Vwfa(r,w.k w. =
E (nt mqso)
V V ((zup w)] + Cov (zol.a)) (V log
w), V p(zo: w))
at
a1
L
B
(mg
log w))
at
a,
L + BVL
at
(B.15)
Combine all the arguments in
and complete the proof.
Lemma B.3 (Lemma 7, Chen et al. [2022a]). The covering number of F
NN, W,, Sy By, Ty) be bounded by
log,N(F11) || 1ell:Illx((Rppp ([ 12y. S log (B
(B.16)
8
18

---

The covering number of H. = NN Wh. Sn. Bb) can be bounded by
LA (B V
log (H, II (1o.))))))) log
(B.17)
8
B.2 Proof's of Transfer Learning
Proposition B.4 (Prop. 3.2). Under Assumption 3.1, 3.3, there exists some constant Cxy such Ltd
that the following holds. For any and (æ1, v1), ". (Ims 9m) minimizer
P. define the empirical
m
arg min mig JEF
f(r Us. sy,n).
(B.18)
m
The population loss of f can be bounded by
E (Cr (z. y, 7, S 4 inf (l.,)) sg,11l + log (m)rz. (B.19)
FEF
where v MJ log (mL, W V
m
Proof. Consider the truncated function class defined R x [0, (),
Φ ((x, v) (x, y, f) = (C(2,, y, 88,)) {(x,y y, )) 1l - FF. where the truncation radius less than 1 2m exp(-CC R²), it holds that || I R for all 1 i S m. Hence by definition, the empirical minimizer also satisfies f arg min
(B.20)
will be defined later. It is easy to show that with probability no
1
m
Ala, Vs. f). Below we reason conditioned on m
JEF
this event and verify the conditions required in Lemma B.11.
Step To bound the individual loss Ü(z, V. f) S Ee,
V. (o M2 + de (log(1/To)
+ +1).
T T To
(B.21)
And by Lemma B.10,
log(1/To)
-((z, f) & Er,,, in (I.. Va log (1))((=)))11 1 SR 2 C² + dg
+1).
T
(B.22)
Let M = CGR + M3 + do log(1/To)
+1)) and thus f((z, (le.v.f) f) $ M.
T
Step 2. To bound the second order moment, we have
E Ewww
y, 81,)) (f(s,,
E(x, +.)
| (E,,,, Atf....llss.aa (Ie, v, - v log (r (1.,( Vz V=
< E(
[1 1 sn (E ( - vx,
y, 1e (Ze, + (x,, 2V log (xx
< 4MB [1 [ SR (E (1 t) u...
4M 4M. (E(x, y, 8F - f(z, U/ x
(r. (1x y, f)] + 8M2 exp( - C2 R2).
(B.23)
19

---

Step 3. To bound the local Rademacher complexity, note that
mm Σ
oflla,,3i,
fi)
o o,l Vi f2) fe)l S <41e(. -
:.(:;:,2)1. f2)1 (P,..))
m 2=1
(B.24)
where P
(ar.)))) Define D, := 10 E +
p(2 - r and it is Σ
easy to show that diam (pr, ((.))...)) S 2vr. By Dudley's bound [Van Handel, 2014, Wainwright, 2019), there exists absolute constant Co such that for any 0,
0 /
20
log y, |l (P.) de
Rm (1,) S Co
(B.25)
m
Since |,,|| S R.
m
((0, Ub: f1)
E(xy Vc. f2))
(la, Vos 81 (( Uc. s2,h))
m
m
[E |1Ii S21l [Be 1I1 + S2 - 2V log 11
4M
E 2 (1 (())), - f2( h(ys):
(B.26)
Let R1 2R. Since Zele, ~ W(z axt, of we have P(]|re|lo > Ri) S d,PP((N((,,)) S R) < 2d exp( exp(-C(1t") for some absolute constant C. Therefore, 1=: hyx)),) t) h(,,,
$ L in [1 || Il IIll h(v)), - fe(1,, t) Sd.M3 exp(C
< f1 f2ll (10Rg)
(1xR)) - 8d Md exp(-
(B.27)
where QR. = |- R1, R1] d== x [0, 1j x To: Tj. Plug the bound above,
m
Σ ((0,, E(2, Ui, f2)) < 4M ffliffll==
+ +8d M M exp(-Ck
/2).
m
(B.28)
For any 2 16d3 M /222 according B.3,
1og.N(2,r111 ())))) log N(F, |l (0,,
(B.29)
C. log
Plug in (B.25) and let = 16dè M exp(- (R
247
CaSyThy log
(,W,(BB
Ry (o,) < Co
0
de
m
C,S,Ly
<
16d M exp(-C,,
=: (r)
(B.30)
20

---

Combine the three steps above, by Lemma B.11 with Bu 8.M2 - == 4M,b- M, holds that with probability no less than 1 - 2m emexp(-CC
for any f
2 m m
log(log(m)/0)
((a,)==(0x, v.
Σ
far Vs. f) + CsM
r rm+
m
(B.31)
M log(()g((m)00)
+ Os
exp(-CiRR3)
m
(ri log(log(m)/0)
(a, Mc. f) (s.s)) (L +GM
m7
m
(B.32)
M log(log(m)/0)
+ CoV
exp(-C(R*)
where im is the largest fixed point of Rm, and it can be bounded as
Ty log (ml.,W, (B)T(((TM))
nm S Cs
(B.33)
m
for some absolute constant Cs. Moreover, we have
(l y, sy,) t(x, y, s.)l E Blw 2M exp( exp(-Cr
(B.34)
Combine this with (B.31),(..32.
(( E(x, y. 50) S
clri, Vi, 85.) - ((xi, yir 8:)l
log(log(m)/0) + exp((CC(T1)
+ CrM (ri
m
(B.35)
(L, Vi, 8111 Ex,, yo, 50 (())))))(((x. yxx,y.55))
772
+ CCs (rim log(log(m)10) + +exp(-
m
(B.36)
Plug in the definition of M c cCC'RR +M3++M3- dz 1og(1/To) +1)) and let R
C' C'log (m8 for some large constant C. Hence (B.35) and (B.36) reduce to
E ((ay))r( (x. < (xx, Mi, s1.n) - l(x,, Vi, v.. s.)1 CyMf log (1th + log log(l)gm))/6)
(B.37)
1 m
> Ve, Sy. f(x,, yss s3)l 2E, (B) (.,,, + lo2 (xt log(log(m)/6)
m
m
(B.38)
SyL log (mL, (BL V1TTM log
where rl
m
21

---

Therefore, we obtain that with probability no less than 1 - 5, the population loss of the empirical minimizer can be bounded by
2
(a Vi, S7,1) Mi. Vs. s. + log (m/8) (rhh + log(1/5)
[( (2. S 87
i=
2 (1 ys. 81,) f(z M1. s511 + 20,M3 log (m/6) (nt. t + log(1/6)
< inf JEF m
m
Pew (1, 8,,)) + 6C² My log (m/0) (th log(1/0) FEF
I 4 inf
m
(B.39)
P(X 2 x)dr and plugging in the bound We conclude the proof by noticing that E[X] = /0 above.
Proposition B.S (Prop. 3.3). There exists some constant Cz, CR such that the following holds. For k uid. pk for any le and (s)), ds are all independent. Consider the
any Pl..... Pkk let at empirical minimizer
f
arg min FEFOR MEH nk
ΣΣ la syr spp
(B.40)
For any 5 E (0, if the configuration o satisfies R 2 Cn log (nKM,/6) then with probability no less than - 8, the population loss of f. can be bounded by
K
E (a e** (1.y.8p.h)
inf 4
E (ft..) log (nK/8) rz r=+ log(1/8)
JEFAK NEH K
nK
(B.41)
bog 1 + Ln log (nKLaW, V M log(1/6))
where
nK
Proof. Throughout the proof, we will use = (k, xr y) to denote the tuple of task index h and data
(x., 3)- With a little abuse of notation, we will also let a o.th Consider the function class defined on [K] x gd [0, Py
{ = (k, v. v) + f1) = (f(x, l(x, (.)) 1 f FOR h H
(B.42)
Rj
will be specified later. It is easy to show that with probability no less than 1 2nK exp(-C, Rº), it holds that ||o I R for all i, Hence by definition, the empirical minimizer also satisfies
where 1 < R <
2
K n
f.h=
arg min JEFSK NEW nK
Σ M f.
(B.43)
where 2/ = (k, att ). Below we reason conditioned on this event and verify the conditions in Lemma 12
Following Step and in Proposition B.4, we have for any FOK h E H,
(C'10 M3 + de log([/To) + +1))
(z,f,h) M M: = C
(B.44)
ELey ((( .f,))
4M Σ
E
Blaynee 10(2* ((**,f.h) f.h)] 8M2 exp(-C(R*)) (B.45)
1=1
22

---

For the local Rademacher complexity bound, note that
K
c( f1.
of ( f2, ha) ma)ll
S f1,h1) El, f2. h2) (P()))))
Vnk
Vnk
k=
i=1
(B.46)
K n
where P(K)
d2t and diam (P,, |l (((0))))))) S 2vr. By Dudley's bound [Van Han-
nk
i=1 del, 2014, Wainwright, 2019], there exists absolute constant Co such that for any A 0,
(0 0
20F logN((B,,) |1- ( c)
Rk,n(*,) Co
de
(B.47)
nk
Since ||=; S R.
K
(1, mi) 12,
nk
i=1
K
Σ
(lla vi m) e(x, vi $ n22)
nK
K
B In 11st - 1/1 B 11 + 12 2V log log 11
nk
(B.48)
K
4M
h1 E v. I (n (x))),)) - (1r, h2(t))))))
n
k=
8M
hn (20, (utt))))))))
la
nk
8M
la 11f2 (xr, k1())),)) - ft (2x, h2(1t),
nk
ue
Let R1 -2R. Since 22l2 Nax a,,,,, have Po R1) d.P(WN(),)) 1) s
R) S 2d, (sp(-C(RR) for some absolute constant Cg. Therefore.
Be,,,1 11 (Ir, hi ).
hi hi
< | [1 | (1 hy (v )) (2 hu (u)),, ( 8d.
$ 11 fb 11. (RxR)
+ 8d, M2 R))))
(B.49)
where QR. = - R1, Rıj dh [0, 114
[b. T]. Moreover, notice that R₁ 2 2R R1,
E, ln 11.22 (z,, hr (2. ), t) 1 (z,, he t))
< 1 i | (2,, (4
(ae, h2 ()1 M3 exp(-C,
< / he hell2 + 8d, M3 exp( Rxp(-c(RR)
(B.50)
Plug in the bound above,
K
Celet. h1) - f(2tff2... f2. h2))2
nk
k=li=1
<8M1
maxxl.f - fEll2=(0RR)) 7s /1/1 hall halle~(0..100)) + gpet M exp(-Co /2).
(B.51)
23

---

For any & 32d M exp(-C)pp /2), according to Lemma B.3,
log.N ogg.V2r,11 ))
- <olog ||
+ log.V [II=((g)): hbgpa//---paaaaa//oooo/oopp////p (1
V 1)(RRT)M + C4Shh log
(Bp V 11MM71
S log
8
(B.52)
Plug in (B.25) and let 0 32d M exp(- /2),
0+ 22/
CsKSyLy log
[,W,
+ L4 log
L. LB V1)MM3
RK,((H,) Co
nk
Cj [KS log
+ Sh log
LaWnnBnvV)MM3 )}
I Co1
nK
Co32di M exp(-Cilt*
=: Rr.(r).
(B.53)
Combine the arguments above, by Lemma B.12 with Bo 8M2 exp( exp( 4M, M,
holds that with probability no less than 2nK exp( exp(-CC, 8/2, for any f E JSK E H.
K
log(log(n (nK)/o)
E (0, f.)) S
f, fh) + Cs.
rk,n+ +
nk
nk
A=
(B.54)
M log(log(nK)/8)
+ +Gb
exp( exp(-C'RR)
nK
K
log(log(nK)/0)
El2 k h,h) < 21E
Fre [L(z, f. h)] + C&M
rk.n
n
nk
k=1 t=1
(B.55)
2 log(log(nkK)8)
+ Cb
exp(-
nK
where r,,, is the largest fixed point of Rx,.. and can be bounded by
rk s Co ( /2) KS log W V ()(VVT)MM) + Sn log (nKL V v1)M?1)
nK
(B.56)
for some absolute constant Cg. Moreover, we have
Σ (f e(x,,, (+)) - E B |.)]]] < 2M exp(-C,
(B.57)
Combine this with (B.54).(..55)
K
B(aaa)a ((2,,, ((2,
nK Σ
2
Ma sf (a), st))
K
+ CsM (xxn + log(log(nK)/0) + exp(pp exp(-C
nK
(B.58)
24

---

K
2 K K
Σ
-
-
Σ
(1 (c(,))- (1x, y/s s.)1
T.K
log(log(nK)/0) + expl-cci
+ CgM
rx +
nk
(B.59)
Plug in the definition of M = c o( + M2 + de
[og((1/To)
T + +1) and define
C" log 10 for some large constant C". Hence (B.58) and (B.59) reduce
K
K
B(z,)) ((2, g0,
2 nk
Σ
[( spx l(x, vi s.
K
CMM log (nK/0) (.km + log(log(nK)/0)
+
nk
(B.60)
K
2
K
xt)l 1 -
E (( ((x, s.)1
n.K
K
(1 (rkoo log(log(nK)/0
Cy CyM3 log log (nK/8)
nK
(B.61)
g log (11 S. log (nKLAW,, V 1)M log11//))
where r Knnn"
nK
Therefore, we obtain that with probability no less than 6, the population loss of the empirical minimizer f. h can be bounded by
K
E B(sa)))- 1e (z, sp.
K
1m
2
m
(a vi 8.,N) e(x,, vt st) + 2og (rkm log(1/0) nk
nK
2
K
n
Ma ui sym elx, sb) 20y log (nK/0) (rkm log(1/2
2 inf
fErON HeH nk
nk
k=li=1
4
K
B(aay (6e (r, 60, log (nK/8) (>kmm log(1/8)
< inf
JEH K
nk
k=1
(B.62)
which concludes the proof.
Theorem B.6 (Thm. 3.4). Under Assumption 3.1, 3.2, 3.3, suppose Pl_ Pl.... .IP are (v, diveieess over larget distribution To given F,H. H. There exists some constant C, Cn such that the following holds. Define the empirical minimizer of training task and new task as
K
f = arg min
Eaa (
(B.63)
JEJOK dEn KK
1=1
m
7 = arg min
El Be s1K))
(B.64)
JEF
m
i=1
25

---

If Ry 2 CR loglo32nn (n 15),, then with probability no less than 1 6, the expected population loss of new task can be bounded by
K
((1 B(n [1 (z. y. 830 Jull 2 inf
1
Σ
inf E Be (z, J. (...
v NEW K
jeF
[log (m)) log log (nk(///))(H log + log(NH/6))
+ C
vnk
(B.65)
where
log,N == I log
(B.66)
logNNx S, log (nKL,,,,BB, 1)M y log(/
(B.67)
Proof.
E [(na)))) Pe (2=0,8p0,1))
inf (la (E (I. Czy log (m)rz FEF
1
inf E feF
(2:)) 11th (x. g + + Czy log)mm)>s
E pa 1e (1,21 + + Cay Img))o) (m)ra
K
E B(aa)) (2,0, 1 Cz log log (nK/8) (1= log(1/5)
inf fEjOR MEN K
nK
+A+ + Czu log" (m)r..
(B.68)
Here we apply Proposition B.4 in the first inequality, task diversity in the second inequality, and Proposition B.5 in the fourth. Plug in the definition of rs, Tr and log logNNe and we complete the proof.
B.3 Proofs of Meta-Learning
Proposition B.7 (Prop. 3.5). There exists some constants C₁, Cr, such for IP..... with probability no less than we have for any h E H.
pk ltd. Pmeta
2 Ep-P.n C(P,))
log(1/6)
C(P h) + Cp rpp
(B.69)
K
K
K
(pp log(1/6)
C(Pk h) < 21E ((P,h) Cp
(B.70)
K
K
k=1
S. log (KL (BAV V1)Mg1)
where rp M² exp(- - R?) +
K
Proof. Given p2 .... p Lid
IP metaa we define the empirical Rademacher complexity of a function class O defined on the of distribution ((Rd, x [0, 1y)
K Σ
1 K
R& (D) := Eg sup
(
Unif([-1, l
(B.71)
k=1
26

---

1 K
For any r > 0, let Hr = { h E H:
Σ
(C(PP, sr and 2, = (6(;;)) : : H,). Note
K
that for any P1, P22 4r.
K
K Σ
K
JkPi (IPA)
xxP2 (pt)
11 (1Pk) (2(()))
42
K
(B.72)
4 -1 422 (
where p(K)
Σ 6pa and diam (r |l (12a()())))))) S 22F. Then by Dudley's bound [Van Han-
K
del, 2014, Wainwright, 2019], there exists absolute constant Co such that for any 0 2 0.
0+ 2
log.N(#r.
)((((()))))))) de
Rx S Co
(B.73)
K
For any PP and h1, 22 E Hy. denote the minimizer of (3.6) in F as f2, respectively, Without loss of generality, suppose [(P,h1) 2 C(P, h2). Then
((P,))) - ((2,h2) s By
h1)), V. - - V logp( By [11 lu (V), - f20,, (x))
x (w)) f2 (v)) 2V, (o
V BL,.y [1l (v). t) (2 h2 + 8(M, c1/3)
(B.74)
In the last inequality we apply ||A My and Moreover,
Cr. by Lemma B.9.
((
(if( fe(2n, (1)),, +]]
/ (12(z1,hil)). 8) f2(x,, h2(()))))))))))))))) ball
$ E.y
$ E.y
[1,2(2 (1), f2 h2 t) (ll (xcl))d + 4M3TP( > Rylu)
< 13 [1/hn - h2(2))1)111111 8.1
73 1111 - -22l? (1.1)))))))) + 8M1 exp( exp(
(B.75)
Therefore, let C3 = 32 + -//2 M S 64M3 and we have
|C [(P, C3 11 - halle- + exp((CC113))
(B.76)
which implies that when 2 203 exp((C( R?), by Lemma B.3,
( |1. -1111112ppp)) k
< log. log.N(Hr. |l [L==(o.ppx))/(((C5T))))
Ln V (1) 1)C331)
(B.77)
I C4S5Lp log
8
27

---

Plug in (B.73) and let 0 = 20's exp( exp(-Ci
2/r
C4Shun log
(x(2,)
0
de
K
L (.WWlBBy1))Mn
(B.78)
Lj log
.
S Co 2C's exp( Cxp(
K
Rx(r).
According to Lemma B.11 (by setting B, = 0, = b = Cr.), for some absolute constant C5. with probability no less than 6, we have for any E H.
2 Ep-P.ne mm ((P,))
(og(log(K//o)
C(P h) + CsCCL
rk +
(B.79)
K
K
k
log log(log( (log(())))))) (6))
L(Pk,M
S 2Pppp L(P, + C3C1
(
(B.80)
K
K
k=1
where r the unique fixed point of Rx. And is easy to show that for some absolute constant Cs.
C3 Cceep((-Cr Sn log (KL, Wo (Br v x)M
r Co
(B.81)
K
which concludes the proof.
Theorem B.8 (Thm. 3.6). Under Assumption 3.1, 3.2, 3.3, there exists some constant C, Cr such that the following holds. Define the empirical minimizer of training task and new task as
K
fhh = arg min
Aaa U. sp),
(B.82)
JEFOK NEN nk
1=1
j arg min
(T Vis 8
(B.83)
JEF
m
If RR 2 Cn Cnlog' Cn ( then with probability no less than 1 8, the expected population loss new task can be bounded by
Ep-Far E (( ae E( (z, (z, y, s.p.
inf Ep inf Bea (I,,, +C log (m) log
(ng(())/)) log.N=
log(Nzz10)
NEW
JEF
m
K
(B.84)
where
log,> = MG log (B
(B.85)
logNN = SnLn. log W, V 1)M log (1/8))
(B.86)
28

---

Proof.
EppP E (I (2, p
inf E ((..))=ll (I, (. + log" Bropan
JeF
inf (() 11 (2, (2,,, Op (rD+ 1o8(1/0)) + Cay log (m)r> JEF
K
K
B (a. 1e (p, + +Cp (pp -p+-0/ 1o8(1/01) + Czy log (m)r
K
km
1
Σ E ( 11 (z, (g C2 logg01//0) (r=+ log(1/6)
inf
FEFOK MEH K
nk
Cp rp 1og(1/0) Cay (m)rz
I inf EprPons inf E(zg)/ole' (z. y. + Cz (oKg/6) (r=+ log(1/8)
NEH
fEF
nK
Op rp 1og(1/8) + Cay loggloggnnnn (m)r..
K
(B.87) Here we apply Proposition B.4 in the first inequality, Proposition B.7 in the second and last inequality. Proposition B.5 in the fourth. Plugging in the definition of TP, and log.Nr, log log.N and
noticing that R2 2 On log (nKd,M,/00) 2 C₁ log² M, K
we have with probability no less Nog
than 1.
Ep-Faee E (xxc.)))) -N Be (z. v, spr
1log (ogg(()) log No log
(nkK)) Nog,N>
log(N (o(NN118)
x inf Ep inf Beagwell (I, (r. 8.,و + C KEH
JEF
K
(B.88)
B.4 Auxiliary Lemmas
Lemma B.9. There exists some constant Ct, such that for any h IP,
C(P,)) I B,,. (x, (x.1v)112 < Cz.
(B.89)
Proof. Note that
(x, (eall E (2r, hw)), V log p. (2> [
Va log
(B.90)
and 0 E F, it suffices to show that Er,,, V log (2r [2) is uniformly bounded for any I,,
According to (B.2),
Er, V. log pr (2, 1 < Ex,,y Ez / log (x)
Ezo, Er, lmo ||| V log (=
(B.91)
d,
01
1 - c 21
29

---

On the other hand, by (B.3) and Assumption 3.3,
Bm lll. log (xal))l1 v Eu,, e lo
Exo.y |mo V |g 622
Emo.yl |ll V. logp(zolw))1//
(B.92)
S Exo.g((B
+ -||x001)2
s Chea
Therefore, we have
C(P,h) < E .. V log log
dz
< E, - A C
(B.93)
2 dz) CL.
Lemma B.10. There exists some constant 8'x such that for any 1 E [0, and x E Rdd y E [0, 1)
Ex V lo (1 + 1).
(B.94)
Proof. Note that 21/2 a 1) and by Lemma B.2.
V Pogg 1,,12 | (o + (Cx + C'x 1xel 1
(B.95)
Let (c(oo 3) x (( (l) Since (0/zo) x exp (- (-c3lla |=]1=|11
a
is decreasing | by
20
Fortuin Kastinn-KKtteen-Giiinii inequality,
E (2.00000 1oll < (P(col))|<<01 S Co-
(B.96)
According to (B.2),
a² V log 10g2:
Coag
B (2 10 -
(B.97)
of
By (B.3), we also have
2(B + L'Cc)
| log (0ly))1
< -
[B
oj
ai
a²
(B.98)
Combine the two incqualities,
| (Olv)) < (B² (1,+
(
A
1 2 + +1. 11.2+10 1)CCo)
(B.99)
of af
Plug in (B.95) and we obtain for some constant Cx
Ex,1 V. Po < 2 + 1x +2 + + +1CG)
(B.
< C' +1).
Lemma B.11. Let & be class of functions on domain and a probability distribution over Ω. Suppose that for any ip &. [|<]) b. Ep[i] O. and Eplp²) < BEplp]] + for some
b. B, Bo Let x1, In itd. TP and &n be positive, non-decreasing and sub-root function such that
x oxp(xx)) s (n(())
R2 (Φ,) = Eg sup
(B.101)
n
30

---

where 4, = ( +
1
Σ (o(x;))) sr). Define the largest fixed point of on Fw. Then for
some absolute constant C", with probability no less than 1 - 8, it holds that for any ip E 4,
2 Σ I n2
r log ((log n)) //// +C'
Bo log (loo n)/6)
Ep|p]
+
(B.102)
1
Σ
((x)) s 27ls (BVV))
r log (logn) C'
Bo log ((lgg /))
n
(B.103)
We follow the procedures in Bousquet [2002]. Let Cj b2 i and consider sequence of classes
Proof.
Φb) (pp 1 :
Ep [ج|| e
(B.104)
Note that $ = U)2006 and for y E Φ()), Ep Beg + By Let ju llog n.. Then by Bousquet [2002, Lemma 6.1], it holds that with probability no less than 1 - 6, for any j S ju and $6))
(Bey + Bo) log (log(b/(;)/o) blog (log(b/c))/8)
Σ p(x)) Eple| 2 R., (D())) +
(B.105)
(B + Bo) log (log(b/e))/0)
8a log log (og(bb/,)/0)
Σ (p(x)))) Be(0? -Ber>>))BBB(oo)1 S 1R,, (+b))) +
(B.106)
Besides, for E Uk Φ(11 =: $(ob.)
Σ ((x,) Ex(Ø]] $ Rn Rn((Booooo (Tliot) [Bejo + Bo) log (log(n)/o)
b log ((log n)/6)
n
n
(B.107)
From now on reason on the conjunction of (B.105), (B. and (B.107). Define
BB + Bo) log (log(b/<;)// b log ( (log(b/e))/o)
U, Beg Bo (T(()) +V
n
n
(B.108)
and thus for any &o $b1), we have
1
Σ (x(x:))) CU, for some absolute constant C' by (B.106),
indicating that Rr (b(i) & on (CU,) < VCOn (U,). For any j S ju- 26 log (logn)/1
2(cc Bo) bVCo. +
(B.109)
n
Since on is non-decreasing and sub-root, the inequality above implies that 1a log (logn)/0) =n(ne)))
U b BE, Bo
(B.110)
Therefore, for any & $(1), , ju. by (B.105).
+ p(.))) - Er|0| s (n(((()) +
(Bej Bo) log ((og m)/5)
log ((og /
n
F.(e)
(B.111)
31

---

Noticing that Eplp] c S 2IE; [p], it reduces to
Σ ((x,) - Bels]] S (Eelle)).
(B.112)
Hence we have by noting that Fn is also a non-decreasing sub-root function,
2 Σ Ep|p] <
log (log m)/5) + CVV
Bo log ((log n)/6)
p(x))
+ C
+
1=1
(B.113)
(I;; & 21p C((Vb) (12 + log (1ogg))8)
1
Bo log (logg)) n)/6)
+ C"
(B.114)
Here C' is absolute constant. Moreover, when &o E g(6) for j > Jo. we have Er(p] v n'
and
according to 10.107)
Σ ((;)) EelØl s (5)))))
(B.115)
Hence the same bounds apply, which completes the proof.
Lemma B.12. Let T be class functions domain Q P .P be probability distributions over D and [p(K))
1
opc. Suppose that for any D, [|o]| S b, Eioos [sp] 2 0, and
K
Epucy 102] < BE [pp] Bo for some B, Bo O. Let af, ah iid pk for any k and all (x Ni are independent. Let ΦKn be positive, non-decreasing and sub-root function such that
(k,n(Dp) = Eø sup ped,
of d(s(aa)) s (K,(())
(B.116)
nK
n
where ; = { (
(x(x)))) Er). Define the largest fixed point of ΦR as r.
nK
k=
Then for some absolute constant C², with probability no less than it holds that for any ip E 4
2
K
((x)) + (BVb) rx,n+ + log ((log nK)/8)
Bo Bolog nkog)
Epixi [sp] <
nK + +C'
nk
nK
k=1
(B.117)
K
+ (BVBB) ( rk,n +
log (loggn))/0
Bo log ((lo nk)/ 1/5) nK
((x))) < 2E
+ +0'1
nk
nk
(B.118)
We follow the procedures in Bousquet [2002]. Let e = b2 and consider a sequence of classes
Proof.
460) (0 & e/41
Egcx [p] E
(B.119)
32

---

Note that + = U)20 and for & E Φl1, Bppx>lo²]= S Bey + Bo. Let ju = [loga (nK) ). Then by Massart (2000, Theorem 31, with probability no less than ô, for any j S jo and yp o(),
K
n Σ
Eaon [p] x Rk,n((0)))) (Be, Bo) log (log(b/e))/0) b blog (log(b/c))/6) ((a)))
+
nk
nk
nk
im1
(B.120)
K n
62 (Be,, Bo) log (log(b/e))/0)
v log ( (log(b/ //)
Σ
Rpo [4 2 (BR,((Bl)) Σ (+(x))))=
nk
nK
nK
(B.121)
Besides, for any yp Ui>jo Tl1) o(oo))
K n
- & RK (Blio))))) (Bej, Bo) log ((log nK)/6)
b blg ((log (k)
Σ
Σ ((
+
nk
nK
nK
k=
(B.122)
From now on we reason on the conjunction (B.120), (B.121) and (B.122). Define
6 (Be, Bo) log (log(b/e))/8)
6a log (log(b/e,)/8)
U, = Bej + Bo + bR (8(5) +
nk
nK
(B.123)
K
and thus for any E g(5), we have
(x(x))) = &CU, for some absolute constant C by k=
nK
i=1
(B.121), indicating that RK, (1))) S ΦK (CU,) S (COk,, For any S jo.
U, S((e, Bo) bV (U₂) 26 log ((log nk) nK
(B.124)
Since ΦK, is non-decreasing and sub-root, the inequality above implies that Therefore, for any $(1), jj S jo. by (B.120),
62 log (log nk)
U3 S b + BEj + Bo +
(E)))
(B.125)
nK
K
(Be, + Bo) log ((log nK)/8)
blog ((log nK)/8) nk
p(a - E Bux0 R [s]
S (x.((Tk,((s))
nk
nK
=: Fx,((())))
(B.126)
Noticing that Epoo [s] Sij (jj_ S P (sp,, it reduces to
K
p(xt) - Egcc S FK. (Egao) (p)))
(B.127)
nK
k=
Hence we have by noting that Fx. is also a non-decreasing sub-root function,
2
log (log nk)/6)
Bo log ((log nK) nK
Epcry [sp] <
plat) (BBBbb) rk +
CVV
nk
nK
(B.
1
K
log ((log (lognK)) (oggnKK/oj
Bo log (oogg nK) nK
((att < 2R (BVB)
r +
+ C'1
n.
nK
1=1
(B.129)
33

---

Here C" is absolute constant. Moreover, when Φhi) for j > Jo. we have E 5 b and according to (B.122),
nk
1 ((
Euo [p]] & Fk,n(E
(B.130)
nk
k=1
Hence the same bounds apply, which completes the proof.
B.5 Verifying Task Diversity Assumption
When F is linear function class, Tripuraneni et al. [2020] provides explicit bound on (v, A). However, general, performing fine-grained analysis is challenging, especially for complex function classes such neural networks. In the following proposition, we present very pessimistic bound for (v, A) based on density ratio, which is independent of the specific choice of hypothesis classes F and H.
Proposition B.13. Suppose F conv and inf p² (x.y),
2 du for any s &o S K. Let
pT v)
K
As. Then P!, .!,.... ... are (i,, A)iverree over pd with vV A/(2K),
1=1
K d logr/ - v log (x
À = ((
(B.131)
We mention that the only requirement is F is convex hull of itself, which can be easily satisfied by hypothesis classes such as neural networks. More refined analysis on specific neural network class is an interesting future work.
Proof. For any E H. let 1& F be the corresponding minimizer for 1 S & S K. Further define
K
d and j0 =
Aast conv(F) E F. Then we have
1, (o70,)) )= Epp [1.7 ((y)), logg (or [2)11
K
< 21Ep0
(()))(()))))
Σ k=1
A
v log log (xcl)))1 -11 Σ
A v log (orll( V Vlog (o
1=1
2
EsoAE h(2)), v Vlogp((<<lv)1 A
2
Ryza (11, V (log)((a>l))111 +
" (sp+,))
inf JEPSK
K
k=1
(B.132)
We conclude the proof by noticing that inf t (sgn) < L (s7o,))
fer
34

---

C Proofs in Section 4
C.1 Proofs of Score Network Approximation
Theorem C.1 (Thm. 4.1). Under Assumption 3.1, 3.2, 3.3, to achieve Ry 2 Cn Cgl (nKM,/6)
and
K
1
inf bEH K
Σ
inf R ((. (p.0.8,,)) o (log))))))) (nK((()))))) (transfer learning) (C.1) FEF
inf Bp-PPan bE7
inf (.. fer
(x, o (Iog((n/ (meta-learning)
(C.2)
W,, the configuration of ,, satisfy
By. B R5),
WNN.Ln. Wh. Sn. Bb) should
log(nK/(E6)) )) W , 0 log 3(, +d,)/2 +dd))/) (nK/(EK)
1 =
log
Ed2 cdxto,t177
log (nK/(E8))
Tlog3 (nK/(E8))
(C.3)
, 0
BB, =0
ed.tatt T3
R₁ =
(log* (nK/(s8))) MM 0 (log" (nK/(e(K)) .75 - =0 (Ooo((k(((8)))
01og
L = O (log(1/E)) W₁ = 0 log(1/E))
(C.4)
S o (c D, log (1/E)) Bn 0(1).
Here (() hides all the polynomial of dz, dyy Dy, C1 C2, I,, B.
K
1
Proof. With little abuse of notation, transfer learning setting, we define Pimeta =
8ps and
K
it directly reduces meta-learning case. Therefore, we only focus the proof in meta-learning.
We first decompose the misspecification error into two components: representation error and score approximation eπror.
JeF inf Ep-Fan inf ((aa))= (z. s.,)
bEH
inf Ep-ras
inf E ((x,))= E (())) h. h.,,), ()1
FEF
< inf Ep-raa
inf E (ay) E, 2 [x] M()) s(x, h. (v), + [1, - 01)
bER JEF
(C.5)
Further note that for any f E
E( (Il, h(y). f(z,, (v) $ E. h(y), f(x,, h- (v), |
+ exp-
() h.) 8M1 exp(-cppCCcpppp
(C.6)
35

---

where Og, = =I-Ry. Ry x 10, 10, x [To, 27. By Proposition C.2, C.3,
inf Bp-ifan NER
inf E ((z,))~ (ed (1,1,,),,))
SEF
< inf Ep-eann B (22) ( - h. - -x inf 2 f f No (v), - (2+, (v),
hEH
+ Bew
/EF
I 2 inf 7/1lh - h..1111 L ((u,))b)) 1 + 16M3 exp( -C-
(C.7)
NCW
inf + 2EpxPann
(1) (z,, hx(v)))()))))))))))) t)1?
FEF
< (log" (nK/ (nK/(E6))d +z)) =² 5²
= (log (nK)(s0))e*)
Proposition C.2. To achieve R 2 CR log (nKM,/1 1 /0) and
inf B( Ioll h. (v). 1) -- JEF
f (xt, h. (v), t)]1 S dze?
(C.8)
the configuration of F= WNNsLL,,,,,M,, By, {1) should satisfy
1. = log (1o((n((60))) )) ,, 0 (dg)(d=+d +d.) (nK/
od td, +1,
1oggldu+da)/2+t (nn) B, =0 =0
[Tlog (nK/
(C.9)
, o
editd,t1ng
R O (log) (nK/(E6))) = (log (nK/(E8))) .75 O (log(nK(<6))).
Here (1) hides all the polynomial factors of de, dy, Dy, C₁, C2 L.
Proof. For notation simplicity, we will f. = f. throughout the proof. Our procedures consist of two main steps. The first is to clip the whole input space to a bounded set UR := [-Ry, R2 dd= x
[0, 1] [To, T) thanks to the light tail property P. Then approximate f.² on 12m.-
By Lemma B.2 and C.6, f. ?1 Lipschitz in x, -ipschhe in w, and 73- tiphhipseetz in a bounded
C.R
domain Qny where 71 = Cx + C' R2, 12 = Cx + Cx Ry, 73 = T3
We first rescale the input domain by x' x
+
w' = w,t' t'
t/T, which be implemented
2R, 2'
by single ReLU layer. Denote e (a', w/ t'). We only need to approximate g(v) fR, 1), w/, Tt') defined on Ω [0, 10, +d [To/T, 1]. And g 7z 271 RypLisshiizz in a', Tw == Liipsshhtt in w' and := Top--ipschitz in t'. We will approximate each coordinate of ) separately and then concatenate them together.
9 =
........ 0=
Now we partition the domain Ω into non-overlapping regions. For the first d,, + d, dimensions, the space [0, 1jd++d, is uniformly divided into hypercubes with edge length of e₁. For the last dimension, the interval [To/T, 1] is divided into subintervals of length é2, where the values of ei and e2 will be specified later. Let the number of intervals in each partition be N₃ [1/e]] and N₂ = [1/e2] respectively.
Let [u,, ... Ud, +a,1 E {0, (O.....A. NI - 1edetdy be a multi-index. Define
g: w/, (') Σ (u/NN j w/ (t').
(C.10)
36

---

where v/ is the coordinate-wee wiie product of trapezoid function:
d
by
Vus Vasla', = (3N2-11((''N-11110 - j/N2a)
II
v (3N. - u
II (3N - (o+4d/Ni)),
(C.11)
1.
|a| <
v(a) =
2 |a|,
1 S la| <
(C.12)
0,
|o| >>>
We claim that 9e is an approximation to 9i since for any o' (x', w') [0, 10 E [To/T.1].
sup o',"'
() g.((' $ sup d'e
Σ
E(ga(
gi t)) M u. (d'))
N1
N2
u.j
S sup of
Σ
gil Ni N₂
No21
j
a:| Mi
-015
22
1s 2N2
2%
2
3N1
3N2
(C.13)
Below we construct a ReI.U neural network to approximate 9.. Let a be ReL.U activation and r(a) (a) 4a 0.5) + 2a 1) for any scalar E [0, 1]. Define
o square (a) = 2 2-2m mm
(mm(a)) (a), Im = ro. in compositions
(C.14)
m=1
o'ma (a, Q aquare
a +
d
squuree
a b
(C.15)
2
2
According to Yarotsky [2017].
Hlma b) abl - 2--21-2 Va, 10.
(C.16)
Then we approximate Vusi by recursively apply d'mmi
Vngla', w/ (/) = d'mm
- (/Na())))))) mmom (3(3 u/ uI/N))),,..)) ...))
(C.17)
And we construct the final neural network approximation as
C, w/, t') = Σ
/ood//N..//N2 w,,t).
(C.18)
The approximation error of 9 can bounded by
l - gill gell
< 00 9e gillem + 9 gill gill==(0)
< 2d, +d 119:ll sup 1Tnj V (uille=(0)) + 2% + d,y)²
27:
+
n.j
3N1
3N2
$ (d2 + dy + d ||: -
(21+2) 27 (d, + dyy
27
+
3N1
3N2
(C.19)
Besides, by Chen et al. [2020, Lemma 15), for こ d. + dy and Va', w', w". t'
(x'2', w/, t') - ai(a", w". f')l S (dz + dy) (nw + N₁ 11 t: (0)) 2 4tdd+dd |lwl - w loce (C.20)
Let o d + dly + log To (1lgg (0)) +
( +11 N₁ 00 (2m) .N²= 0 (2) Then
10 w (1', w" ,()) S (d, +d) do))l - ww' w"lloo. (C.21) [6 - 9 S
37

---

1 g,w.tf w.t/T) Then the approximation error
Define 9 == 19..... -d, and f(, w. : 2Rg + of f in RR can be bounded by I I - 1 - - S e and Ma, w.() = 0, ||]| > R₃- (C.22)
Therefore, when R2 2 CR ((M3 + Ct)/e), the overall approximation error is
((x,))==BB,, (t (~ h. (w), 0)1 - Bt,,,,yl/ h(), - f(x he (1), - 1
+ 4(M + C1. exp(-Ci
| f 4(M} ex.)exp((C3
dgs².
(C.23)
Now we characterize the configuration of neural network f(xx, w. t). For boundedness, by Lemma B.
[1, w, I)ll S ([,llle-((nn))) + 20% Rí My.
(C.24)
Hence we can let R2 - O log in Theorem B.8. For Lipschitzness, by (C.21),
nk eδ
to ensure the lower bound of R2 mentioned above and
[I w. - A(x w. 1)ll 2 To (de + dy)llw wi|lx
(C.25)
(Cx + Cx (xx + dr)llw T1
Hence "y = 00 Cx d.)) o log
nk
For the size of neural network, for each coordinate, by the construction in (C.18), the neu- ral network 9. consists of N Adaadyy N2 parallel subnetworks, i.e., 9 (u/NN, L P.)l).). By definition in (C.17), the subnetwork consists of O (dz +dy + dy + log Rg the width bounded by Oldz + dz). Therefore, the whole neural network Ti can imple- mented by O ((dz + (d2 + dy + LoogRR,/5550 layers with width (Nrletd, Nz(d, + +dy)) = Ry (2((d=ddz) cd2+dy IT
layers and
3(d, +d,) log(R)/E)
Re
and the number of parameter is bounded by o edetdx+ 1
Com-
bine these arguments together, we can claim that the size of neural network f is
(og(nk/(=6)) ))
0 ((d2 + d,,) + dy log((R//)))) = log
E
R R((dd+dd))
W o
td, 1T3
ods cdstdytly
(C.26)
==0 (ds + dyy 12 a(d, + log log(Rr/e)
(o8(R)/e) =0 (
3ld (nK/(e0)) ed++dy+17g
td
To bound of the neural network parameters, note that the trapezoid function & is rescaled by 3Ni or 3N, and the weight parameter of i is bounded by constant. Moreover, the input of J first rescaled by R, or T. Hence we have
R&T
0 Tlog= (nK/(Ed))
B O NoT) =0
(C.27)
which concludes the proof.
38

---

Proposition C.3. To achieve
inf || h hol11=(100119)) S Vdye,
(C.28)
DEA
the configuration of N (L,, Wh, Sh. Bb) should satisfy
Ln = O (log ()))) Wh = (s Da log(1/E))
(C.29)
S = 0s log (1/E)) Bn O(1).
Here O()) hides all the polynomial factors of d2 dy: L.
Proof. The main idea replicates Yarotsky [2017, Theorem 1]. We approximate each coordinate of h [h-1.-.. ,had,] respectively and then concatenate all them together. By Yarotsky [2017, Theorem 1]. h4s can be approximated up to by network h4 with O (log(1/=)) layers and O (s log(1/e)) width. Besides, the range of all the parameters are bounded by some constant, and the number of parameters is (e-Dx log" (1/e)). Then we concatenate all the subnetworks to get h ===(o [., [hd,) and [h h. ((03)))(0)))))) S dys.
C.2 Proofs of Distribution Estimation
Theorem C.4 (Thm. 4.2). Suppose Assumption 3.1, 3.2, 3.3 hold. For sufficiently large integers
K, and 6 > 0, further suppose that Pp_ .P_.... ,PK are (v, diverres over target distribution po with proper configuration of neural network family and T, To- It holds that with probability no less than 1 6,
log (nK/o) log ((m/v) An) a ((m/v) n)) T0-
log (nkK(//))
E ((x,01))1 By-py TV z/y p
2z (nK) Tot + V..
(C.30)
Proof. Combine Theorem C.I and Theorem B.6 and plug in the configuration of F, H. we have with probability no less than 8
E ((( Fo
B 1 (z, y. s30 (2
side sido+dylt10 (nK/ed) log log log'(1/)) ((ng/))))(()/((())
log
S log" (nK)(=8))e +
(m (un))edstdy+117
vnKeD,
(C.31)
By Lemma C.7,
TV aly: popy)) S V log (1/Ta) +T- + Epo ( (1.
(C.32)
Taking expectation of f P. we have
datt 2 (1/To) c + + loggoo((KK((E))) VA E ((x.,)))) Ey-pg T T øly: D S / To log
log
(nk
log (T)
log log ( nK
(m (mn)
dxtdy T.
(unK)+E
(C.33) = O(lo((1/5o)); Oleo/ log(nK for some
Let To o (03/ 1o
small Eo > 0 defined later. Then it reduces to
Eo + (( By TVV FD 1
log
aid, +ddytdy+11
nk
200
log
log
Dv log
log
nK
Egd
da dz
Dy (unK) (C.34)
(m A (un))
t
tE0
39

---

log (nK/6) log" ((n/v) An) 1og2(n1/0 (nK/6)
Let eo C max
and we can conclude that
((m/v) A n)) 4 (nK) 2y4z
((x,0))) Byure [V flyy P -لyا] log (nK/o) log ((m/))mm/)))))) (m A
log (nk/6)
2t vz(KK) potz + VA.
vz((mvV) A n) ao-2044
(C.35)
Theorem C.5 (Thm. 4.3). Suppose Assumption 3.1, 3.2, 3.3 hold. For sufficiently large integers n, K, m and 5 > O, with proper configuration of neural network family and T, To, holds that with probability no less than ō,
log (nK/8) log (m A n)
log (nKK()))))
kly: PxlU)] S Ep-F E8 (Z..):)) ( P [V
(m A n) 2=+dut3
K Py44
(C.36)
Proof. Combine Theorem C.1 and Theorem B.8 and plug the configuration of F, H. we have with probability no less than 1 8
Ep-Pam (( ~F B( (l 8pp1))
s(dx
$ (ngg(nn(/E6))) + log 2 (nK/Ed) (T/T))
(1/=( (og[KK(o((((/(6)
(m (man)
KeD
(C.37)
By Lemma C.7,
Pall S TT log dxtt (1/T6) TT + Pf
(C.38)
Taking expectation of f., P. we have
Ep-Fam E (= (TV EIV, P2)O)]
Vo log
det1 (1/To) + - +log(nK((Eo))
log
3(d=+4,)+15
ak 88
log
+
log² ( log ( nk
+
Idy
K K Dy (C.39)
(m A n2)
To
Let To o (e0/ l1 (1/=o) , = O(lo((//so)) = O(Eo/ lo (((no)()) for some
small 5o 0 defined later. Then it reduces to
log
M(du+dy))-1
nK
log
3d
Ep-P.., ( WTV (P Payy S +
(m A n)) 26
(C.40)
log
log D
20
Dv
K3 15000
log (nK/8) (m An) log (nK())
Let 5o = C' max
and we can conclude that
(m A 12) 2o+dy+
K Dy+z
log (nK/0) log 3 (m A n)
+ log (nKK(n
ER ~Byyr (P Paly
(m A n) a.tdy-1
K 22
(C.41) D
40

---

C.3 Auxiliary Lemmas Lemma C.6. Let RR = R, Ry) x [0, 10. x (To, foror some R₃ 2 1. Then there exists some
C.R
-ipscchiz with respect to RRy constant Cs such that the score function f. (ax w. is
T
Proof. According to (B.2),
t)) = x oi of
ai
/ 2o J (=([))))((; w)dz dæg.
(x( w)
+
(C.42)
Define density function te (2 ar w) x te ( w). Then
i का
(x,
2a72 x of
+ at Cov
Cov (zolxa))
as + az) log de (1(
Be x, [co]
of
zo.
o
of
(C.43)
Note that
log di (11
|| - ar2ol
CoVa.
ro. a
= Cov (xolaa))))
30: O2
20²
arlæ arto) 1
2a/110
a,30 1|2
Cov (e0ol,)))
w0.
of
0
(C.44)
Hence for any E [-R,, Ryy E [0, 11d,,
|
| f (z, w, w,
8
of Rs
+ E (zolz,))
1 0420
3
a1 + az)
E (no (m) [oloo]
of
oî
of
20
(C.45)
2R + 2Co
Let R
We have
Tt
a,20 - x
0430 - 2
(x(x) xo (200))((()) (v) dro
Eg (wo (x,
of
0t
[ 2 z)p(2 (yy)dz
| 1|2 exp (- ||a
p( w dzo
R
|2
207
+
of J exp
|1a 111 207
p(uo; w)dz
|2R R exp(
- )p(())((i cro w ddro
R3
0 aten "||5//2 exp(
R²
8 )p())) )p(zo: w)dro
(C.46)
The domain
To : a13o - x
S R/2]
includes (oo: |ool| S Ca indicating
1
| 1000- 1/R/2
p(zui w)drr >
p(zo: wdr - 2 2exp(-Ci -C, 2 2'
110 IsCo
(C.47)
p(ro; w wdd 1
p(ao;w)dzg - 2
11a011200
Therefore, for any (x, w. 1) E Any
i 1h u.,)
R2
R +
C3
Ry
Co
R
(C.48)
07
of
T
Lemma C.7. Suppose KL(TA (W 7)) < CKI. some constant CKL. Then
T fly: pOp))= S VTolog dx
(1/To) -T + Eeg 1 (x.8.87ū)).
(C.49)
41

---

Proof. With little abuse of notation, we will use pa(aalv) to denote the conditional density of 24 under sly Consider the following two backward processes
dē; (0g + 2V log + 22d,, To -(0.1),0011
(C.50)
dæ c log lt) 2dW,, To ~ Pr. 0 To.
(C.51)
Denote the distribution of 5. Pr-t- And note that F. Pr-t by classic reverse-time SDE results [Anderson, 1982]. Then by Fu et al. [2024, Lemma D.5].
TV Po) S To log (1/To).
(C.52)
At the same time, we apply Data Processing inequality and Pinsker's ' inequality to get
(V Pr.)) & (V()), & (V((, & ((,
(C.53)
Again according to Pinsker's incquality and Oko et al. [2023, Proposition D.1].
Tv.),Pr)) S V (L(TT, (P) z /Bululf (z. U.
(C.54)
Combine three inequalities above and we complete the proof.
D Proofs in Section A
D.1 Proof of Theorem A.1
Due the structure of exponential family, Assumption 3.2 holds obviously. To apply previous results, we only need to verify Assumption 3.1 and 3.3. Recall that a basic property of exponential family
Proof.
vzAp(z) = E (vlx (v)l E 1]% 0 3 (z) =
(D.1)
(v. (h)) = I.
(D.2)
Hence by Assumption A.1, (x S Ag + ||-||] S log (ʃ (/ww)dy)) + || s log + ||=||1-
And Ap (x) (0) ||>|||
log C || ||- Further that the posterior density po po(p))
(a) exp( (r, (v)) Ax)x)) where the normalizing constant Ze (v) is lower bounded by
Za
Zo(y) = / Po(x) exp)))x)( - (x))dr
> / po(x) exp((2))z11))//ar
(D.3)
2
2 (xx()))-xxx(pCC))(())) =: Co.
where the second inequality we apply (1 2 R) < 2exp( -C and let R R= 1
1/vC/ to
get Co. Therefore, by Assumption A.
(x(p)) < C1 exp(-C2ll=dp 2| + log C)/C/ < C expl exp(-C(1=1)
(D.4)
and thus Assumption 3.1 holds. At the same time, ley w = hy), then the score function is
V log = V, log = V, lo
Az).
(D.5)
Since V, (x) L-Lipschitz, (4 also I-Lipschitz, the score function V, log po (z, w)
is (L + L)--scchtttt in r and 1-Lipschitz in w. And | log pa (0, w) < || log (0) + 2vd =
B + 2vd, indicating that Assumption 3.3 holds with L,' L+ 1, B' B 2vd.
We conclude the proof by applying Theorem 4.3 under meta-learning setting Theorem 4.2 under
(v, A) -diverssiyy
D
42

---

D.2 Proof of Theorem A.2
Proof. Let (ir (ir(s,a) ()) be the advantage function of policy x. Note that the reward function rM E [0, 1/, we have (4% - 1- for any M, म. According to performance difference lemma,
Varo (TT) Viso (T))
y E ()s LA (s, (s)
E Baway( [4 a)] E Ba (1)
(D.6)
E Eondy [TV (-1s))
7)2
Hence in meta-learning setting, we plug in Theorem 4.3 to obtain
log (nK/6) log" n)
log 2 (nK/ö)
Exro (a (5)) (1 7)²
(m A n) 2n42,45
K DzFz
(D.7)
If we further assume (v, -diversityy holds, then we plug in Theorem 4.2,
1
log (nK/0) (om//(m/V)))))))))mmmm//m)) An)
log (nkg()/)) VA
(((of,o))) "0)","aa (T.) (5)) 1 (1 7)2
+
(2 A An) 20
ud (nK) 1x+2
(D.8)
E Experiment Details
E.1 Conditioned Diffusion
Each 1 and 0 are implemented as 2-layer MLP with 128 internal channels and 60 input channels. The representation map h is implemented as 5-layer MLP with 512 internal channels and 10 output channels. We have {10, 20, 30, 40,50, 100} fine-tuning samples from the target distribution TP. We run Langevin Monte Carlo for sufficiently long time to obtain 100 test samples from the target distribution PD' for evaluating the test error of different models. In the pre- pre-training phase, the (f" 1 k < K} } and h are trained on the K 10 source distributions with 400K iterations and batch size of 5 52.. In the fine-tuning phase, the pre-trained representation map Te is fixed, and the j trained on the target distribution with 200K iterations and a batch size of m. As an important baseline, we also consider jointly training and fo on the target distribution from scratch, using the same fine-tuning samples.
1000 pre-training samples from each source distribution Tp*
E.2 Image Restoration on MNIST
Each f and 1 are implemented a 3-layer MLP with 512 internal channels and 784 input channels. The representation map h is implemented as a 5-layer MLP with 512 internal channels and 64 output channels. We have n = 5000 pre-training samples from each source distribution T& and m (10, 20, 30, 40, 50, 100} fine-tuning samples from the target distribution IPO For evaluation, directly compute the mean squared error between the posterior samples and the ground truth images, based on 100 test samples from Idd In the pre-training phase, the the {f*; < k < K = 9} and are 2K epochs and a batch size 512. The initial learning rate 0.0003 and annealed according to a cosine annealing schedule. In the fine-tuning phase, the pre-trained representation map h is fixed, and the fo is trained on the target distribution with 20K iterations and a batch size of m. As an important baseline, we also consider jointly training and fd the target distribution from scratch. using the same fine-tuning samples.
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions, i.e., proposing data-efficient training method for machine learning models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
44

---

Justification:
The paper provides the full set of assumptions and complete (and correct)
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
45

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will provide complete codes upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wttpcccppppppeeeeeeeccecccccceet gww///uiisee//ee Seteee//ooeSuumieeeeeeee wwwsssssiiiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports experimental results based on the average of independent random trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
46

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
47

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
48

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets currently.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
49

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ //tpr///eeu..peeeee Ctteurrps.cc/oonfiiie 22LM/LLM for what should or should be described.
50


Reviews:
[{"review_id": "tEXdzHfRXF", "reviewer": "Reviewer_JJrR", "summary": "This paper examines the generalization error of conditional diffusion models obtained through transfer learning with a smaller dataset than the source training sample. Under the assumption that a low-dimensional representation exists that is shared across all tasks, the generalization guarantee of conditional diffusion models trained using the score matching error is provided in Section 3 for settings where task diversity is both known and unknown. In Section 4, the authors apply the result to the estimation error bound of neural networks by evaluating their model complexity. Their results show that, under the assumption of low-dimensional representation, the curse of dimensionality is mitigated in lower dimensions.", "questions": "- While this paper focuses on the Lipschitz score, can the results be extended to other function spaces, such as the Holder space?\n- Could the authors provide the lower bound on the estimation error trained through this setting? Specifically, I am interested in how the dependency on $m$ can be theoretically improved in transfer learning.", "limitations": "The authors properly address the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- Exploring the theoretical understanding of conditional diffusion models is a significant topic in the deep learning literature, and this paper addresses this issue.\n- This paper provides a general framework for evaluating the generalization guarantee of the conditional diffusion model. The results in Section 3 are model-agnostic and provide substantial theoretical insights into the conditional diffusion model under the transfer learning procedure.\n- This paper is clearly written. All theoretical results seem solid.\n\n**Weakness**\n- My primary concern is that the upper bound derived in Section 4 consists of $O(m^{-\\frac{1}{d_x+d_y+9}})$, which is much worse than $O(m^{-\\frac14})$ exhibited in [Yang et al., 2024], as the authors also refer to in the paper. Since $m\\ll n$ typically holds in transfer learning, the exponent of $m$ substantially affects the convergence performance of the trained models.  \n- Additional numerical experiments will support the theoretical statements. For example, I would like to know he shared low-dimensional representation actually improves the generalization performance of trained models in artificial datasets.\n- The definition of $R_f$ (in Sections 3 and 4) is lacking in the paper.", "was_revised": true, "final_justification": "I would like to maintain my original score.", "rebuttal": "Thanks for your careful review and constructive feedback! We address your concerns as follows.\n\n**W1**: The primary concern is that the upper bound derived in Section 4 consists of $O(m^{-\\frac{1}{d_x+d_y+9}})$, which is much worse than $O(m^{-\\frac{1}{4}})$ exhibited in [1].\n\n**A1**: We would like to point out that [1] considers a completely different setting from ours. In fact, [1] considers unconditional diffusion models and the unconditional distribution is assumed to be supported in a low-dimensional linear subspace, where the source task and the target task have the same latent variable distribution. Hence, only a linear encoder is trained for fine-tuning instead of the full score network. The complexity of the linear encoder function class is much lower than that of the neural network family in our setting and hence [1] is able to achieve better rates. We believe that our methods for CDMs can be extended to this setting and get similar results as well.\n\n**W2**: Does the shared low-dimensional representation actually improve the generalization performance of trained models in artificial datasets?\n\n**A2**: We conduct an additional experiment following the setting in Section 5 except that we use distinct operators $M_k$ for $p_k(y|x)=\\mathcal{N}(M_kx, I_{100}/4)$ and $M_k$ are independently generated for different tasks. In this case, source and target tasks will not share identical representation and the MSEs on target task are shown below. The performance of fine-tuning is even worse than train-from-scratch, indicating that transfer learning fails without shared representation. We remark that $M_k$ varies from each other substantially in this setting leading to highly variable low-dimensional representations for each task. However, if $M_k$'s are similar (not necessarily identical), fine-tuning can still outperform train-from-scratch. \n\n|m|10|20|30|40|50|100|\n|---|---|---|---|---|---|---|\n|fine-tuning|943.08|215.6|184.94|81.17|69.1|48.38|\n|train-from-scratch|21.99|10.61|5.71|2.38|1.77|1.04|\n\n\n**W3**: The definition of $R_f$ (in Sections 3 and 4) is lacking in the paper.\n\n**A3**: $R_f$ is defined in Section 2.3 (see line 161), which quantifies the region in which the score network $f$ is Lipschitz. For generalization bounds in Section 3, we assume this parameter has some lower bound. In Section 4, we explicitly present the value of $R_f$ in Eq (4.3). We'll make cleaner statements in the revised version.\n\n**Q1**: While this paper focuses on the Lipschitz score, can the results be extended to other function spaces, such as the Holder space?\n\n**A4**: Both Lipschitz score [1,2,3] and Holder density [4,5] are commonly used assumptions in the literatures of diffusion models. And either of two assumptions are not able to indicate the other. In our case, the Lipschitz continuity of score function is crucial since it enables a Lipschitz neural network estimator, which is essential in transfer learning theories. Expansion of our methods to the Holder density space is an interesting future work.\n\n**Q2**: Could the authors provide the lower bound on the estimation error trained through this setting, especially on the dependence of $m$?\n\n**A5**: [2] studies the lower bound of score matching loss with Lipschitz score and achieves the min-max optimal rate of $O(m^{-\\frac{2}{d+4}})$ (Thm. 3). If we apply this result to diffusion models, the min-max optimal rate of TV distance should be $O(m^{-\\frac{1}{d+4}})$ as indicated in [2] (Coro. 1). This is aligned with our results when reduced to unconditional DMs, i.e. $d_y=D_y=0$. To rigorously establish a lower bound in CDMs and our settings is also an interesting future work. \n\n[1] Yang, Ruofeng, et al. \"Few-shot diffusion models escape the curse of dimensionality.\" Advances in Neural Information Processing Systems 37 (2024): 68528-68558.\n\n[2] Wibisono, Andre, Yihong Wu, and Kaylee Yingxi Yang. \"Optimal score estimation via empirical bayes smoothing.\" The Thirty Seventh Annual Conference on Learning Theory. PMLR, 2024.\n\n[3] Chen, Minshuo, et al. \"Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data.\" International Conference on Machine Learning. PMLR, 2023.\n\n[4] Oko, Kazusato, Shunta Akiyama, and Taiji Suzuki. \"Diffusion models are minimax optimal distribution estimators.\" International Conference on Machine Learning. PMLR, 2023.\n\n[5] Fu, Hengyu, et al. \"Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory.\" arXiv preprint arXiv:2403.11968 (2024)."}, {"review_id": "ivhdNYB7t3", "reviewer": "Reviewer_CoEJ", "summary": "As in other areas of machine learning, transfer learning has proven useful in the context of diffusion modeling: one can train on a large data set, and then fine-tune the model on some other data set, instead of training a model from scratch. On the other hand, it is not completely clear why one should expect transfer learning to work, and some theoretical guidance or guarantees would be helpful here. The authors endeavor to find such guarantees, essentially assuming models learn low-dimensional representations which are useful for multiple tasks. They present a large number of formal results on this topic, mainly on how well one expects a model to generalize on task X, given that it was trained on tasks A, B, C, D, ... from the same task distribution.", "questions": "1. How tight are the generalization error bounds? How tight are they in practice, when the assumptions may not exactly apply?\n2. Can the authors provide explicit examples of the kinds of transfer learning tasks to which their theory applies? Are there examples of situations commonly viewed as 'diffusion model transfer learning' that, for one reason or another, the theory doesn't really apply to? Or is it hard to say? Lines 232-233 (\"hard to verify in practice\") sort of speak to this.", "limitations": "There is no \"limitations\" section. I think it may be worth at least gesturing at the question of whether these bounds approximately hold in real settings, which is not addressed by the experiments currently in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths.**\n- The motivation is clear, and although the paper is highly technical, the writing is also fairly clear. I especially liked how the authors compared their work to related formal work on this topic, and explicitly related the bounds they derive to other bounds in the literature (see, e.g., the paragraph starting on line 285). This is helpful for someone not directly working on this topic.\n- The math is very high quality and appears carefully done.\n- The results suggest particular scalings of generalization error with various parameters (e.g., the number of previous tasks), including in the transfer learning setting.\n\n**Weaknesses.**\n- The \"experiments\" section is kind of underwhelming. One thing I'd be interested in is whether the scalings that follow from the math can be shown empirically, or if they are somewhat different in practice (see also the aforementioned discussion around line 285). But this may require more/different experiments than the authors have an appetite for.\n- The way the authors assume different tasks are similar is a bit specific and abstract. It may be helpful to provide some examples of the kind of transfer learning tasks that their task similarity notion applies to, even just for the sake of exposition.", "was_revised": true, "final_justification": "I maintain my original positive score. The paper has some limitations, mostly with relating its theoretical claims to more realistic settings, and the authors acknowledge this. This didn't change during the rebuttal period, but in fairness to the authors, it is generally hard to cleanly link this kind of theory to empirical experiments. Still, I think the math here is valuable for the community, and was very carefully done. I think this paper should be accepted.", "rebuttal": "Thanks for your careful review and constructive feedback! We address your concerns as follows.\n\n**W1**: The \"experiments\" section is kind of underwhelming. Can the scalings that follow from the math be shown empirically, or if they are somewhat different in practice (see also the aforementioned discussion around line 285). But this may require more/different experiments than the authors have an appetite for.\n\n**A1**: We have conducted additional experiments on real image data (A3 for Reviewer Qqzq) and ablations on non-identical representations (A2 for Reviewer JJrR). \nIn practice, it is difficult in general to show the scaling shown in theoretical results. First, the theory only provides an upper bound for TV distance instead of an exact characterization. Besides, the TV distance between two distributions is typically hard to compute especially in high dimensional cases. Moreover, the theory is only about statistical rates while in practice, optimization error is non-negligible. These factors together make it difficult to empirically validate the theoretical scaling.\n\n**W2**: The way the authors assume different tasks are similar is a bit specific and abstract. It may be helpful to provide some examples of the kind of transfer learning tasks that their task similarity notion applies to, even just for the sake of exposition.\n\n**A2**: In Appendix B.5, we verify the task similarity assumptions and provide some naive bounds. A straightforward example is our experiment in Section 6. The conditional distribution $p_{k}(x|y)\\propto p_{\\beta_k}(x)p(y|x)$, where $p_{\\beta_k}(x)$ is the prior distribution of the dynamics determined by a parameter $\\beta_k$. The trajectory Eq (5.1) is very similar across all tasks except that the drift term involves different parameters. In this case, we believe that all tasks have similar structures and thus the notion in Eq (3.1) is reasonable.\n\n**Q1**: How tight are the generalization error bounds? How tight are they in practice, when the assumptions may not exactly apply?\n\n**A3**: To the best of our knowledge, there is no result on the lower bound of sample complexity in transfer learning setting, but our results match the state of the arts in existing literature. See more discussion in lines 285-292. In practice when the assumptions may not exactly apply (e.g., real images), our empirical results on image restoration for MNIST (A3 for Reviewer Qqzq) still indicate the sample efficiency of transfer learning. This is aligned with our theories.\n\n**Q2**: Can the authors provide explicit examples of the kinds of transfer learning tasks to which their theory applies? Are there examples of situations commonly viewed as 'diffusion model transfer learning' that, for one reason or another, the theory doesn't really apply to? Or is it hard to say?\n\n**A4**: We provide concrete examples in Appendix A. Specifically, Table 3 lists some real-world application of CDMs and the number of parameters (complexity) of backbone score networks and conditional encoders. In these applications, the conditional encoders are much more complex than the backbone score networks, indicating that the assumption $N_\\mathcal{H}\\gg N_{\\mathcal{F}}$ in our theoretical setting is very common in practice. In Section A.1 and A.2, we discuss two specific applications of transfer learning CDMs, including amortized variational inference and behavior cloning via diffusion policy, which basically satisfy our assumptions. We acknowledge that it is indeed hard to quantify if representation is shared or not in general cases. However, our framework is motivated by the prevalent intuition that shared representations naturally emerge in standard pretraining–finetuning paradigms, which are widespread in diffusion model transfer learning.\n\nLimitations:\n\nThanks for your valuable suggestions and we will add limitation and discussion section in the final version."}, {"review_id": "Wq8EiWdA8p", "reviewer": "Reviewer_Qqzq", "summary": "This paper investigates the theoretical underpinnings of transfer learning for Conditional Diffusion Models (CDMs). The core setting involves a source and target domain that share a common, low-dimensional representation of the condition, captured by a function $h(\\cdot)$. The proposed transfer learning scheme first learns both a score model and the condition representation on the source tasks. Subsequently, for a new target task with limited data, the condition encoder is reused (frozen), and only the score model is fine-tuned.\n\nThe main theoretical contributions provide generalization bounds for this process. The authors first analyze the Lipschitz properties of the score function with respect to the data and the condition representation, establishing bounds that depend only on the initial data distribution. They then connect the source and target tasks through two established theoretical frameworks: task diversity (Theorem 3.4) and meta-learning (Theorem 3.6). In both settings, the analysis demonstrates that transfer learning can yield a tighter generalization bound compared to training from scratch. This improvement stems from leveraging the nK source samples to reduce the error associated with learning the ground-truth representation $h_*$ from its hypothesis class $\\mathcal{H}$.\n\nFinally, the authors present auxiliary results on the approximation error of the score function using deep ReLU networks and compare their bounds to prior work, arguing for improvements in terms of tightness, generality, and the mildness of the required assumptions. The central thesis is that learning a shared, low-dimensional condition representation is a key mechanism that enables the success of transfer learning with CDMs.", "questions": "In Thm 3.4, the error reduces in terms of $\\log N_\\mathcal{H}$ but can increase in terms of $\\log N_\\mathcal{F}$, how to interpret this ?", "limitations": "Please see the weaknesses section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper introduces a clear and tractable framework for analyzing transfer learning in CDMs. The central assumption of a shared, low-dimensional representation (Assumption 3.2) is intuitive and provides a solid foundation for the theoretical development. \n* The analysis provides a novel approach to bounding the Lipschitz continuity of the score function (Lemma 3.1) under weaker assumptions than some prior works.\n\nWeaknesses:\n* A primary concern is the gap between the theoretical setup and common practices like Parameter-Efficient Fine-Tuning (PEFT). As noted in Remark 1, the analysis assumes the target score network $f^0$ is trained from scratch, independent of the pre-trained source models $f^K$.\n* While Assumption 3.2 is critical for the analysis, it is a strong idealization. The existence of a single, perfectly shared representation $h_*$ across diverse tasks is a significant simplification.\n* The experiments serve as a proof of concept but are somewhat limited. The core claim is only validated on a synthetic dataset.", "was_revised": true, "final_justification": "This paper has solid theoretical derivations, and it does offer justifications for the success of PEFT on diffusion models. Although I prefer theoretical studies with tighter connections and more insights into real-world applications, this bias should not be a major factor in a fair judgment of this paper. Also, the extra verification on the MNIST restoration serves as a good example, So I am increasing my final score to 4.", "rebuttal": "Thanks for your careful review and constructive feedback! We address your concerns as follows.\n\nW1. A primary concern is the gap between the theoretical setup and common practices like Parameter-Efficient Fine-Tuning (PEFT). As noted in Remark 1, the analysis assumes the target score network is trained from scratch, independent of the pre-trained source models.\n\n**A1**: The main contribution of our work is to take the first step towards understanding the sample efficiency of transfer learning CDMs and show that practical training procedures (pretraining and fine-tuning) is able to reduce sample complexity on target task. In practice, full-finetuning is widely used (e.g. T2I models, see more applications in Table 3 in Appendix A) and thus our theoretical setting is very close to practice. Although PEFT is also a popular training paradigm, its theoretical analysis is much more challenging and no existing theoretical paper has tackled with PEFT for neural networks to our knowledge. It is beyond the core idea of this paper and can be an interesting future work.\n\n**W2**. While Assumption 3.2 is critical for the analysis, it is a strong idealization. The existence of a single, perfectly shared representation across diverse tasks is a significant simplification.\n\n**A2**: Assumption 3.2 is a standard assumption in theoretical analysis of transfer learning [1,2,3], without which one cannot establish connection between source tasks and target tasks. In addition, in many applications such as the setting in our experiments and in Appendix A (e.g. T2I generation), there indeed exists a shared representation across tasks. Therefore, Assumption 3.2, although as an idealization, is very close to practice. \n\n**W3**. The experiments serve as a proof of concept but are somewhat limited. The core claim is only validated on a synthetic dataset.\n\n**A3**: For real data experiments, we consider the image restoration task on MNIST, where we have $K=9$ source tasks with $P^k(x,y)=p(y|x)p_k(x)$. Here the prior $p_k(x)$ is the data distribution of the digit $k$ in the MNIST dataset and $p(y|x)=\\mathcal{N}(x, I_{784}/4)$.\nThe target task is $P^0(x,y)=p(y|x)p_0(x)$ where $p_0$ is the data distribution of the digit 0.\nWe use the complete MNIST 1-9 data for pre-training, which corresponds to $n=5000$.\nFor the finetuning phase, we consider $m=10,20,30,40,50,100$ training samples from $P^0(x,y)$, and 100 test samples from $P^0(x,y)$.\nBelow are the MSEs of fine-tuning models and train-from-scratch models and it's obvious fine-tuning is consistently better than train-from-scratch, indicating the benefits of transfer learning.\n\n|m|10|20|30|40|50|100|\n|---|---|---|---|---|---|---|\n|fine-tuning|0.3799|0.2846|0.2544|0.2406|0.2404|0.2268|\n|train-from-scratch|0.4409|0.3180|0.2746|0.2551|0.2501|0.2344|\n\n**Q1**: In Thm 3.4, the error reduces in terms of $\\log N_{\\mathcal{H}}$ but can increase in terms of $\\log N_{\\mathcal{F}}$, how to interpret this?\n\n**A4**: In Eq (3.4), the RHS increases in terms of both $\\log N_{\\mathcal{H}}$ and $\\log N_{\\mathcal{F}}$.\n\n[1] Tripuraneni, Nilesh, Michael Jordan, and Chi Jin. \"On the theory of transfer learning: The importance of task diversity.\" Advances in neural information processing systems 33 (2020): 7852-7862.\n\n[2] Chua, Kurtland, Qi Lei, and Jason D. Lee. \"How fine-tuning allows for effective meta-learning.\" Advances in Neural Information Processing Systems 34 (2021): 8871-8884.\n\n[3] Du, Simon S., et al. \"Few-shot learning via learning the representation, provably.\" arXiv preprint arXiv:2002.09434 (2020)."}, {"review_id": "oPyJIPnndn", "reviewer": "Reviewer_8krS", "summary": "This paper provides rigorous statistical analysis on how transfer learning reduces the sample complexity for training diffusion models. The main idea is to assume distributions of different conditions share certain common information, modeled as $h(y)$. Intuitively, finetuning phase saves the computational cost of learning $h(\\cdot)$, as it is assumed such optimal latent representation is already learnt during pretraining.", "questions": "1. How tight is the bound in (3.7) and (3.8)? How large is $C_P$?\n\n2. Can you verify the task diversity assumption in practice? What is the relationship between $N_{F}$ and $N_{H}$? What if $N_{F}$ dominates?", "limitations": "1. The core assumption, task diversity is strong and not verified. The relationship between $N_{F}$ and $N_{H}$ is unclear.\n\n2. The task-diversity assumption already bears most of the theoretical burden; although the subsequent theorems supply extra bounds on sample diversity, the insights they yield are, in my view, predictable and add little novelty.\n\n3. The inequalities in (3.7) and (3.8) need further discussion. It is not clear how tight they are.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow.\n\n2. The study is comprehensive, considered several settings, such as with or without task diversity assumption. The approximation error brought by the deep network is also analyzed.\n\nWeakness:\n\n1. The core assumption, i.e., task diversity, assumption is quite strong and is not empirically verified. Moreover, the theorem in section 3.2 seems a little bit trivial given the task diversity assumption, as this assumption alone already implies pretraining bounds the error of finetuning phase.\n\n2. The bound in (3.7) and (3.8) needs further discussion. How tight are they? How large is $C_P$?\n\n3. The relationship between $N_{F}$ and $N_{H}$ is not clear. Currently the improved sample complexity comes from $N_{H}$. But what if $N_{F}$>>$N_{H}$, hence dominate the loss? Can you somehow quantify their relationship empirically on real-world dataset such as natural images?", "was_revised": true, "final_justification": "This is a theoretical paper and the contents are rigorous, having no obvious flaw. Hence I think it could be accepted. The limitation is that some conclusions are hard to be validated on real world practical models.", "rebuttal": "Thanks for your careful review and constructive feedback! We address your concerns as follows.\n\n**W1**: The task diversity, assumption is quite strong and is not empirically verified. Moreover, the theorem in section 3.2 seems a little bit trivial given the task diversity assumption, as this assumption alone already implies pretraining bounds the error of finetuning phase.\n\n**A1**: The task diversity is a standard assumption in theoretical studies of transfer learning [1,2,3], which establishes the connection between source tasks and target tasks. Thm 3.4 is indeed a straightforward corollary based on Prop 3.2-3.3 and task diversity. However, Prop 3.2-3.3 is definitely non-trivial and requires novel analysis methods for pretraining and finetuning in CDMs. In addition, we provide some naive analysis to verify the task diversity assumption in Appendix B.5. Conducting a more fine-grained analysis is an interesting future work.\n\n**W2**: The bound in (3.7) and (3.8) needs further discussion. How tight are they? How large is $C_P$?\n\n**A2**: The bound in Eq (3.7) and (3.8) can be interpreted as generalization bounds for *empirical risk minimization over samples of tasks*. Intuitively, standard generalization results based on (local) Rademacher complexity [4] should apply. In fact, one of our technical contributions is to extend the framework in [4] to general *samples* as formalized in Lemma B.11. In addition, note that Eq (3.7), (3.8) are key lemmas for Thm 3.6, which provides the SOTA sample complexity for meta-learning as $\\mathcal{O}(\\frac{1}{m}+\\frac{1}{K})$ dependence when $n$ is large. In contrast, existing work [3; Thm 2] can only achieve $\\mathcal{O}(\\frac{1}{\\sqrt{m}}+\\frac{1}{\\sqrt{K}})$ for supervised meta-learning due to less sharp techniques. \n\nRegarding $C_P$, it is a constant with polynomial dependence on the parameters in Assumptions 3.1-3.3, and its definition can be found in the proofs in Appendix B.3.\n\n**W3**: Currently the improved sample complexity comes from $N_\\mathcal{H}$. But what if $N_\\mathcal{F}\\gg N_\\mathcal{H}$, hence dominate the loss? Can you quantify their relationship empirically on real-world dataset?\n\n**A3**: In this work, we focus on the setting when the conditions are of high dimension and the condition encoder in CDM is harder to learn than the backbone score network, i.e.,  $N_\\mathcal{H}>N_\\mathcal{F}$. In Table 3 in Appendix A, we list some real world applications of CDMs including text-to-image, reinforcement learning, etc, where the number of parameters of condition encoder is larger than or at least comparable to that of backbone score network, indicating that our setting is aligned with many real world cases. Of course there are also examples such that the class of condition encoder is very small and $N_\\mathcal{F}\\gg N_\\mathcal{H}$. To understand sample efficiency of transfer learning in such settings is beyond the scope of this paper.\n\nThanks for your valuable feedback and we will add a section for limitation and discussion in the final version.\n\n\n[1] Tripuraneni, Nilesh, Michael Jordan, and Chi Jin. \"On the theory of transfer learning: The importance of task diversity.\" Advances in neural information processing systems 33 (2020): 7852-7862.\n\n[2] Chua, Kurtland, Qi Lei, and Jason D. Lee. \"How fine-tuning allows for effective meta-learning.\" Advances in Neural Information Processing Systems 34 (2021): 8871-8884.\n\n[3] Maurer, Andreas, Massimiliano Pontil, and Bernardino Romera-Paredes. \"The benefit of multitask representation learning.\" Journal of Machine Learning Research 17.81 (2016): 1-32.\n\n[4] Bartlett, Peter L., Olivier Bousquet, and Shahar Mendelson. \"Local rademacher complexities.\" (2005): 1497-1537."}]

Author rebuttal:

Meta-review: This paper studies transfer learning for training conditional diffusion models. Assuming that the distributions of different conditions share certain common information, the authors derive generalization error bounds for conditional diffusion models, showing how transfer learning can reduce sample complexity.

This is a theoretical paper. Overall, the reviewers agreed that the paper is well written, the analysis is rigorous, and the technical contribution is clear. Reviewers raise concerns regarding the connection of the theoretical claims to realistic settings, which are partly addressed in the rebuttal by some additional experimetns. The authors are encouraged to address this by including additional experiments and further discussion in the final version.
Decision: Accept (poster)