Few-Shot Diffusion Models Escape the Curse of Dimensionality
Authors: Ruofeng Yang
Bo Jiang
Cheng Chen
Ruinan Jin
Baoxiang Wang
Shuai Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models
Few-shot learning
Primary area: generative_models

Abstract: While diffusion models have demonstrated impressive performance, there is a growing need for generating samples tailored to specific user-defined concepts. The customized requirements promote the development of few-shot diffusion models, which use limited $n_{ta}$ target samples to fine-tune a pre-trained diffusion model trained on $n_s$ source samples. Despite the empirical success, no theoretical work specifically analyzes few-shot diffusion models. Moreover, the existing results for diffusion models without a fine-tuning phase can not explain why few-shot models generate great samples due to the curse of dimensionality. In this work, we analyze few-shot diffusion models under a linear structure distribution with a latent dimension $d$. From the approximation perspective, we prove that few-shot models have a $\widetilde{O}(n_s^{-2/d}+n_{ta}^{-1/2})$ bound to approximate the target score function, which is better than $n_{ta}^{-2/d}$ results. From the optimization perspective, we consider a latent Gaussian special case and prove that the optimization problem has a closed-form minimizer. This means few-shot models can directly obtain an approximated minimizer without a complex optimization process. Furthermore, we also provide the accuracy bound $\widetilde{O}(1/n_{ta}+1/\sqrt{n_s})$ for the empirical solution, which still has better dependence on $n_{ta}$ compared to $n_s$. The results of the real-world experiments also show that the models obtained by only fine-tuning the encoder and decoder specific to the target distribution can produce novel images with the target feature, which supports our theoretical results.

Few-Shot Diffusion Models Escape the Curse of Dimensionality
Ruofeng Yang', Jiang', Cheng Chen', Ruinan Jin
Baoxiang Wang"1, Shuai Li+1
1 John Hopcroft Center for Computer Science, Shanghai Jiao Tong University
East China Normal University
3 The Chinese University of Hong Kong, Shenzhen Vector Institute
(vanshuiyin, bjiang, shuail18){ssj tu. edu. en, chchen@sei ecnu. edu. en, {jinruinan, bxiangwang @cuhk edu.
Abstract
While diffusion models have demonstrated impressive performance, there is growing need for generating samples tailored to specific user-defined concepts. The customized requirements promote the development of few-shot diffusion models, which use limited Tit target samples to fine-tune pre-trained diffusion model trained on ni, source samples. Despite the empirical success, no theoretical work specifically analyzes few-shot diffusion models. Moreover, the existing results for diffusion models without fine-tuning phase can not explain why few- shot models generate great samples due to the curse of dimensionality. In this work, we analyze few-shot diffusion models under linear structure distribution with latent dimension d. From the approximation perspective, we prove that
few-shot models have Ö(n 2/d + nla 1/3) bound approximate the target score function, which is better than nia 2/d results. From the optimization perspective, we consider latent Gaussian special case and prove that the optimization problem has a closed-form minimizer. This means few-shot models can directly obtain an approximated minimizer without a complex optimization process. Furthermore, we also provide the accuracy bound ((1/nta + 1/ for the empirical solution, which still has better dependence fl compared n2,, The results of the real- world experiments also show that the models obtained by only fine-tuning the encoder and decoder specific to the target distribution can produce novel images with the target feature, which supports our theoretical results.
Introduction
In recent years, diffusion models have shown an excellent ability generate diverse, high-quality samples and show state-of-the-art performance in many areas with large-scale, standard datasets
(Rombach al. 2022 Ho et al. 2022 Li al., 2024a Blattmann al.) 2023 Li al.) 2023b
Liu et al. 2024 Li et al. 2024b). However, users often desire to generate samples that resemble the ones they provide, such as images related their families, daily lives, or specific items. These user- user-prove ner--proviee samples are typically limited in number and do not appear frequently in large-scale datasets. Consequently, training diffusion model from scratch using such limited, personalized samples often results in poor performance. To cater the customized requirements of users, few-shot diffusion models attract much attention. Few-shot diffusion models aim to fine-tune me-ttrineed diffusion model using limited amount of data (5 ~ 10 samples), and they have recently delivered
"Corresponinng author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

impressive results in various domains, including image generation (Ruiz et al. 2023 Han et al. 2023
Zhu al. 2023). video generation (Chen al. 2023bj. and the medical domain (Dutt al. 2023).
Before the fine-tuning phase, we first need to train a diffusion model on the large source dataset (=, as the pre-trained model. A diffusion model consists of a forward process and reverse process (Song et al. (020). The forward process gradually converts the data distribution into Gaussian noise. The reverse process sequentially removes the noise in the data to generate samples, which relies on the gradient of logarithmic forward process density (a.k.a. score function). To run the reverse process, diffusion models use a neural network to approximate the unknown score function.
With a pre-trained diffusion model, the paradigm to obtain feotshot diffusion model is to fine-tune the model using a limited target dataset (Xta. 1 In earlier times, fully fine-tuned methods, such as DreamBooth (Ruiz et al., 2023), provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters. Furthermore, a fully fine-tuned method is both memory and time inefficient (Xiang al., 2023). To avoid the above problems, many works freeze most parameters and fine-tune some key parameters, such as cross-attention layers (Kumari et al. 2023 Moon al. 2022). , some concept neurons (Liu et al. 2023) or text-embedding (Gal et al. 2022), to approximate the ground-truth target score function. These works not only preserve the prior information but also have lower requirement for the target dataset size, which more practical for applications. Hence, we aim to explain the great performance of these models in this work.
Despite the empirical success, no existing theoretical work specifically analyzes the approximation bound for few-shot diffusion models, and the following question remains open:
Do few-shot diffusion models with fine-tuning phase enjoy a small approximation error with limited target dataset?
For the approximation error bound, some works currently analyze diffusion models without fine- tuning phase (Oko al., 2023 Chen al. 2023c Yuan al. 2023 Li et al. 2023c). Importantly, when analyzing general. .al. bounded data, these works suffer from the curse of dimensionality. More
specifically, Oko et al. (2023) analyze bounded distribution and show the n, = approximation bound, where is is the data dimension of X,, Chen et al. (2023c) analyze linear structure distribution
X,- A, Z with subgaussian latent variable Z E 12d and achieve n, 2/dd results. Since the source dataset size is large enough, the influence of dimension is tolerable. However, for the limited target dataset, if trivially using the above technique, the bound is nto -1/D nta 22/d which is large and can not explain why few-shot diffusion models efficiently approximate the target score function.
In this work, for the first time, we propose the approximation bound specifically to few-shot diffusion models with a fine-tuning phase and prove that the few-shot diffusion model can escape the curse of dimensionality. More specifically, we show that when assuming (1) linear structure data and (2) the source and the target data share latent distribution, the few-shot diffusion models with a fine-tuning phase achieve (n, 22d + nza 1/2) approximation error bound, which makes the first step to explain why few-shot difffusion models have great performance in the application. Generally speaking, due to -1/2 1/2 the few-shot diffusion only needs a few target samples to achieve the same the component na
bound compared to n. -2/d To support our augmentation, we calculate the requirement of nua to obtain an accurate enough approximated target score function in popular datasets. Table shows that the requirement of 14a is about 5 10 samples, matching the customized diffusion model requirement. We also do experiments on the real-world datasets and show that 10 target images are enough for few-shot models to generate novel images with the target feature (see Section (6.
After directly using the property of the minimizer to obtain an approximation bound, we analyze how to optimize the few-shot diffusion models to obtain minimizer. Since the score-matching objective function is highly non-convex, only a few works analyze the optimization problem of diffusion
models (Shah et al. 2023 Bruno et al. 2023 Cui al. 2023 Li al. 2023c). Furthermore, these works either require (1) an exponential size neural network (Li et al. 2023c) or (2) distribution determined by one variable (Shah et al., 2023) Bruno et al. 2023 Cui et al., (023) to simplify the
optimization problem. This work proves that few-shot diffusion models can simplity the optimzzation
problem without these requirements When analyzing the optimization problem, we focus on a
2

---

Gaussian latent variable special case Then, we prove that the expected few-shot objective function has closed-form minimizer, which means the empirical solution can be directly obtained without a complex optimization process. We also prove the accuracy bound O(1/nta + 1/ / nt) of empirical closed-form solution, which still has better dependence the target dataset. In conclusion, accomplish the following results for few-shot diffusion models under linear structure distribution:
For the approximation bound, we consider a subgaussian latent variable and prove Ö(n 2/d_ -1/2) bound for few-shot models, which is better than Nza Pta
-2/d result without fine-tuning.
For the optimization problem, we consider a latent Gaussian special case and prove that the expected few-shot objective function has a closed-form minimizer. Furthermore, prove the accuracy bound O(L/nea" + 1/ nT.) for the empirical closed-form solution.
To support our theoretical results, we do real-world experiments and show that the models obtained by only fine-tuning specific encoder and decoder only 10 target images to generate novel images with the target feature.
2
Related Work
The approximation error bound. Recently, some works analyze the approximation error bound of diffusion models without a fine-tuning phase. Oko et al. (2023) analyze s' -order bounded deriva- tives distribution and show the approximation error bound is n2
Chen et al. (2023c) analyze distribution with linear structure and subgaussian latent variable and show that the nts 2/d result. The approximation error bound of the above works suffers the curse of (latent) dimensionality. To avoid this phenomenon, some works analyze special data distributions. [Shah et al. (2023) and Cui et al. (2023) analyze the mixture of Gaussian (MOG) with known variance and achieve 1/n, approximation bound. Yuan et al. (023) analyze linear structure distribution with Gaussian latent variable and achieve 1/ Vn, result. Mei & Wu (2023) analyze Ising models and prove that the term corresponds to n, is 1/ Vn, However, the remainng terms do not converge to 0 when n, goes to
+00. For general bounded data distribution, Li et al. (023c) provide a nss -2/5
approximation error bound. However, they use a 2-layer random feature network and only allow the second linear layer to be trainable. Hence, the network size is exp (n,)) compared to Poly(n,) size of all previous works.
The optimization of diffusion models. Since the score matching objective function is highly non-convex, only few works analyze how to optimize it to obtain a minimizer (Shah al. 2003 Cui et al. 2023; [runo et al. 2023 Li et al. 2023c). These works either make assumptions about data distribution or network size to guarantee only one optimization variable, leading a simpler optimization problem. For special data distributions, [Buno et al. (2023) and Cui et al. (2023) analyze Gaussian with fixed variance and 2-mode mixture of Gaussian (MOG) with equal, trainable mean and fixed variance, respectively. Shah et al. (202)) analyze multi-mode MOG with a fixed variance and prove local convergence guarantce. Since they assume the distance between any two modes is large enough and good enough initialization. the optimization problem is similar to optimizing Gaussian distribution. For the large neural network size, [Li et al. (202cc) analyze a general, bounded distribution with 2-layer NN. Note that they require exp (n,) hidden neurons and only allow the linear layer to be trainable, which also leads the optimization problem convex optimization.
3 The Introduction of Few-shot Diffusion Models
With pre-trained models, the paradigm to obtain a few-shot diffusion model is to freeze most parameters and fine-tune some key parameters corresponding to the target data distribution. Since the analysis of few-shot diffusion models relies heavily on the pre-trained model, this section first provides concise overview of the fundamental concepts and notations associated with diffusion models. Then, we introduce the paradigm of few-shot diffusion models in Section 3.2
TThough it is special case, the previous analysis can not be used since it is determined by two components.
3

---

3.1 The Forward and Reverse Process
Let qu be the data distribution. Given Xo ~ E RR, non non-decreasing function f(X,, and g()), the forward process defined by:
dX; f(X,, g(t)dB;,
where {Bo [ejor is a D-dimensional Brownian motion. In this work, we choose f(X,,t) -1/2X, and g 1. which corresponds to variance preserving (VP) forward process and is widely used in practice Shah et al. 2023; Song et al. 2020). Let qt be the density function of X,. Once a forward process is chosen, the conditional distribution of X₂ Xo is qe (X,(XX)) N (m, TD), where m, e-t/2 of = e- Note that when t goes + + qi converges (0, TD), which is helpful choosing the initial distribution for the sampling process.
To generate samples, diffusion models reverse the forward SDE and run the reverse process. Since the reverse process contains the gradient of forward logarithmic density V llog ge (-) (a.k.a. score function), the model approximates it by using neural network s(: 1) and the score matching objective function (see Section 3.2). After that, diffusion models discretize the continuous reverse process to obtain an implementable algorithm. Let to t1 forward time and hę = ty: - be the k-th stepsize. When considering the reverse time, we define t/ = T T fK- k- In this work, we choose the exponential integrator (EI) discretization scheme, which has great performance (Zhang & Chen 2022). The EI discretization freezes the approximated score at to2 and runs the following process in the reverse time:
tx T be the discretization points in the
di. = [PPC, - ( gTT - (t) (2) dt - t)dB., t U1 (erl
where Po ~ N' In) due to the stationary distribution of the forward process.
While the discretization complexity K has been well-studied with an accurate enough score function (Benton et al.] 2023 Li et al.) (023a). there is lack of analysis for the score-matching process. Therefore, this work focuses on the score approximation and the optimization problem of the few-shot score-matching objective function.
3.2 The Score Matching Objective Function
In this work, we specifically analyze few-shot diffusion models, which involve two datasets: (1) the source dataset (X.. -== (2) the target dataset {Xto, 1i The approach involves first training pre-trained diffusion model on the source dataset and then freezing the backbone network to fine-tune the diffusion models on the target dataset.
For data distributions, we assume that the source distribution q" and the target distribution gda are both supported on a fow-dimensional linear subspace. The low-dimensional structures have been discovered on many popular image datasets (Pope et al., 2021 Gong et al.) 2019 Tenenbaum et al.. 2000) due to the locally connected and symmetrical property, and it is crucial for diffusion models. For image generation, current popular diffusion models, such as Stable Diffusion (Rombach et al.. 2022), transform images latent space and run diffusion models in the latent space. Except for the image generation, Chen et al. (2024) recently show the latent dimension plays an important role in diffusion models to work well in self-supervised learning, and linear subspace is enough.
We further assume that the source and target data share the same latent distribution. Note that this is a common assumption in few-shot learning. In particular, previous theoretical works in the context of supervised few-shot learning often assume that the source and target distributions have common latent representation (Du et al., 2020 Chua et al. 2021 Meunier et al. 2023).
The source datapoints X, and target datapoint Xte admit a low dimensional linear structure and share the same latent distribution X, = A,Z and Xta = AlaZ where Ag, Ato E R Dxd with orthonormal columns and Z 9= dR.d
Assumption 3.1.
'Our analysis can be extended to f().,t) 1/228xX, and g(t) = Vp., where (8e is non non con-decressing and bounded sequence.

---

As mentioned in Chen et al. (2023c), when assuming linear distribution, the ground-trute score function is decomposed into the latent score function V log 9: (Z") and linear encoder and decoder:
log (X) A. oo LD (A
of ( A.AA) x,
where o. ( q (Z Z) 9: (Z)dZ and q (-12) = (m, oa This form indicates that the diffusion process happens in the latent subspace. A conceptual way to approximate the score function is to minimize the following loss on function class SNN:
{' w w(t) V log of (X,) - (X,, f)l
min sESNN
where w({) is a weight function. However, the above objective function is intractable since V log qe (-) is unknown. Vincent (2011) propose the following implementable loss:
(s)
w (l) Ex,,x V log of (X,lXo) - (X2, dt.
Due to the forward process, V log qi (x) has an analytical form and is equal to -(x, m4)o)/af |Vineet (2011) also prove that this objective function only has constant difference compared to the above one. The empirical loss with the source datasets (X is defined by:
Σ {" (s(s,)))= (T 58) i=1 66
min 8v.0ESNN
(v
where
E (X.,ii) -xx, X Vogg (XilXo) t)1
and
SNN = (sv,o(X,t)= = 01
Vfg (V x,t) Ve f X . R Dxd with orthonormal columns, of
fg R [0,7] - R ReLU network }.
Note that we take w=t) = 1/(T 5) for simplicity, where is the early stopping parameter to avoid the blow-up phenomenon score functions at the end reverse process. Furthermore, we take the integral over the forward time instead of discretizing the timeline since X, is easy to generate.
The linear encoder and decoder structure and the shortcut connection in SNN is due the form of the ground-truth score function. The specific parameters for fg. such as its length and width, are identical those used Chen al. (2023c). Generally, with a given network accuracy parameters e, the network size is Poly (1/e). We show the parameter of neural network in Appendix A
The diffusion models minimize the empirical loss to obtain pre-trained approximated score function. Let the minimizer of Equation [ be (Va, 0). Chen et al. 2023c) show that (V,, 0) leads a n, -2dd approximation error bound. If trivially replacing nı, with tea we obtain a Vita -2/d bound for the target dataset without the fine-tuning phase. Note that this bound suffers from the influence of the latent dimension d, which is still large in popular datasets (Table [I]. In the next paragraph, we introduce the few-shot diffusion models with a fine-tuning phase and show that the dependence on Aio. is n 1/2 in the error bound (Theorem 4.3)
The Few shot Diffusion Models with a Fine-tuning Phase. Since the source and target distribution share the same latent data distribution, we freeze î and only fine-tune the low-rank linear encoder and decoder layer Via. This method can significantly reduce the fine-tuning parameters and is similar to LoRA (Hu et al. 202T), which also fine-tunes two low-rank matrices and is widely used fine-tuning the stable diffusion (Rombach et al. 2022).
Let ela be the loss function of the target dataset at time which has similar definition compared et. The optimization problem for the target dataset is
{' T
Era (Sv..)) nea (T -8) 1=1
1
Σ
ee (Xra.iSv.n.)) de,
min
Sv (Qxx(()
5

---

Table 1: The requirement of nta in popular datasets. We use latent dimension in Pope et al. (2021
Dataset
CIFAR-I0
CIFAR-I00
CelebA
MS-COCO
ImageNet
Dataset Size
x 10
6 10*
2 x 10
3.3
10"
1.2 x 10"
Latent Dimension
25
22
24
37
43
The Requirement of Rea
8
5
where
QNN(O) = (sv, = (V x,1) -. x: of
x: Ve E Dxd with orthonormal columns. }, d;
Similarly, define the minimizer of the few-shot objective function (Via, 0).
Notations. We denote by In the D-dimensional identity matrix. For X E RD and A E RDddd we denote by || the Euclidean norm for vector and ||A|| the Frobenius norm for matrix. We denote by ||_X] 1 (q) the expectation of X in L2 norm Ex~q X
4 Fho- Diffusion Models Enjoy Better Approximation Error Bound
In this section, we show that few-shot diffusion models with fine-tuning phase escape the curse of latent dimensionality and have Ö(, 2/4 + m approximation bound This result makes the first step explain why few-shot models have great performance with limited target dataset.
Before showing our results, we first introduce standard assumptions on the latent distribution and the on-support ground-truth score function. We first assume that Z has subgaussian tail and the minimum eigenvalue of Z is lower bound by co. also used in Chen et al. (2023c).
Assumption 4.1. 9= > is twice continuously differentiable, Amin (E ZZ 2 Co and E 2 < Cz. Moreover, there exist positive constants B, C1, C2 such that when || B, (Z) S
(2mr (2m)- exp (-C2 1B/2..
Assumption 4.2. The on-support ground truth score A, V log qt (Z) and A.a V log ot (Z) is L-pischiitz Z E Fdd for any [].
Note that different from previous works directly assume V log (-) is Lipschitz (Chen et al. 2022 2023d), the 3-Lipschitz on-support score function assumption does not conflict with the blow-up phenomenon when goes to 0 due to the existence of (TD AA X//o With these assumptions, we prove the approximation bound for few-shot models with a fine-tuning phase.
Theorem 4.3. Let ((())=
dog log log n
(d+ and network parameter e - 1210 -1/2 Assume 6
Assumption 3.1 3.1 4.1 4.2 and nia
ol 2 ng. Then, with probability 1 1-8,, the following inequality
holds (hiding logarithmic factors)
T T- /.¹ T ||
V Vlog of ( (2(E)") dt ( Dds ( /nio
(1
2-20
1
Fri
log
The dependence of is due to the blow-up property of the score function. Note that when n, is sufficiently large, (<(n.)) ) is negligible. Then, the approximation error bound for few-shot diffusion models is Ō(n-2/4 + nie 1/2), Compared to the approximation error bound Tita -2/d without a few-shot phase, it is clear that few-shot diffusion models escape the curse of the (latent) dimensionality.
Remark 4.4 (The discussion on the coefficient Theorem 4.3) The goal of the fine-tuning phase is to achieve the same order eтror bound compared with the pre-trained models, which means that we consider the relative relationship between n,a and n.. Hence, if the coefficient of Thea and n, has
the same order, we can only consider 1/ VTta and nx 2//d To support the above augmentation, we calculate the coefficient of n. and Rta detail. The dominated term of coefficient for T4a and n,, are D / and d3 /(62co). respectively. The classic choice for the early stopping parameter 8 and
Here, the approximation means the matching with finite source and target datasets.

---

forward time 7 are 10 3 and 10, respectively (Karras et al. 2022). Then, with D = 256 x 256 x 3 as an example f Dd / d' 20 x 10 and d' d 10 /co. which has the same order. Hence, we consider the relative relationship between 1/vTta and na 2/d
4.1 Discussion on the Approximation Bound
The relationship to empirical phenomenon. In applications, current few-shot diffusion models only require ~ 10 target images to achieve great performance. Theorem (4.3 makes the first step to explain why the few-shot diffusion models have great performance with limited target Thra More specifically, with known source dataset size ft and the corresponding latent dimension d, we can
calculate the inequality
401 401-0(ma)) 2 Yts obtain the requirement of nıe achieve the same ataa
accuracy compared to the pre-trained diftusion models. Combined with the latent dimension of popular datasets (Pope al. 202T), Table Soooee[shooe shows the requirement of nlt-- It is clear that we only need less than 10 target images to obtain an accurate enough few-shot diffusion model that matches the performance reality. The real-world experiments also support our discussion (Section (6)
Table T shows that the requirement of nua is heavily influenced by the latent dimension d. When d is large (e.g. ImageNet), the approximation bound of trained models is influenced by latent dimension and has a large approximation error even with large-size source data. We only need a few target data achieve the same error in this setting. When d is small (e.g. CIFAR-10), pre-trained models have a small approximation error. We need slightly larger target data size.
The approximation error of the fully fine-tuned method. As shown in our real experiment Section and DrcamBooth (Ruiz et al. 2023), when fine-tuning all parameters with a small target dataset, models tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use 0 learned
by the pre-trained model and achieves a nt -2/d approximation error bound, which suffers from the curse of dimensionality. From intuitive perspective, the probability density function (PDF) of distribution learned by an overfitting model only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to large error term. We also note that it is possible to avoid this phenomenon by using a specific loss (Ruiz et al.) (023) or carefully choosing the optimization epochs (Li et al. 2023c). We leave them as interesting future works.
Proof sketch. The first step is to prove that in @NN(0) there exists solution (Veas 0) has the following inequality (only focusing on ny and (iaa))
T T- | || V Voog t (x)
2-20 (m2) a4t
))))
1
dt s 2
log 51
LP(()))
To prove the above inequality, we first do the following decomposition:
|s ol, t) - V log of (-12+ + || .|, (t(:)) t) SV...(. 1)
where (Via.0) SNN is constructed solution. The first term is due to the accuracy of the constructive neural network with network accuracy parameter e. For the second term, since the latent score function is shared and few-shot diffusion models directly use 0, it is control by the approximation bound of the pre-trained diffusion models. Then, by using the inequality
Em (sv...) S Eio (sv...)
inf VE. (
we build the bridge between Sp. T and s
The second step is using the concentration to control the error between empirical D.n and expected Lia Roughly speaking, we have that
Lea (Sp...)) Lra (Vv...)
log (N (1/mia. (N) 181)
5 Since smaller more friendly to Meay our discussion holds for all datasets Table
TThi also indicates the requirement of nes in Theorem 4.3 is easy to satisfy.
7

---

where (11mmto (0) 11 l12) is the covering number of QNN(() in L2 norm. Since only V E Dxd can be optimized and 0 is fixed in QNN(O),
log N (1/nta: QN)(O), 1l 112)) =
Then, we balance different terms and achieve the final bound by choosing c² = 1/ T
The Few-shot Diffusion Model Have Closed-form Minimizer
This section focuses on how to optimize the few-shot diffusion model. When considering the opti- mization problem, we assume the shared latent distribution admits an isotropic Gaussian distribution 4: (0, A2Ia) with X² > O, which indicates the score function has the following formulation:
1 of
V log att (X)
X2 A
(ID Ata X.
Note that though 9. = (0, La) is a special case of Assumption 4.1, we still need to know X² and Ato generate samples, which indicates the previous optimization analysis for diffusion models without a fine-tuning phase can not be used.
We fix a E [5, T] for the few-shot objective function since the matrix A.a is independent of time More specifically, with approximated latent distribution T= (0, E.), where E Xa the expected few-shot objective function at fixed time is
(8 +t) E Extnngto 1go (Xeais
min
V 2 @NN(ES)
where
1
= =
Vfs (V X,t) - og 0f
x: X: E Dxd with rank(V) = d.}.
oi
In this case, f2( = (Ia - of where E. = oa The constraint rank(V) d used to guarantee that the few-shot diffusion models learn meaningful subspace instead of 0 Note that rank(V) = d is a weaker constraint than V = Ia since the pre-trained diffusion model has already learned the length information. This weaker constraint means we need less prior knowledge compared to ((0), which is more user-friendly. Let V.o be a minimizer of the above expected few-shot objective function. We show that Vea has a closed form and good property.
Lemma 5.1. Assume Assumption and q= expected data covariance mairix. Then, Vna has closed form:
(T)). Let C Ex XT]] be the
m mx + of (C+o(ln)~
Vra VT to
Lemma 5.I indicates that few-shot diffusion models can directly obtain an approximation of the minimizer without complex optimization process. Furthermore, this minimizer has good properties and exactly recovers the subspace spanned by Ata- More specifically, the expected minimizer indicates Vra ViT Ara A AT - 0 when n, and nıa are infinite. However, the source datasets n, and target datasets Uia finite, we analyze the empirical closed-form solution
T
mf + o (m3C + -
X
where C = 1 Ene XXT i is the empirical covariance matrix. Pic
Theorem 5.2. Assume Assumption 3.1 and q= ((0,) Let qo be the latent distribution 8 (d(x)) /Ddlog (Ddn,) ) (d2 V D).
generated by the pre-trained models with (Via, Σ) and M Then, with probability 82. have that for any E [5,
TT Viato V.
Iō (l
M
+ d (m + 033') 02y²)
to - Ala
F
m2 + o
d8 d
nso
8

---

(a) Target dataset
(b) The results of fine-tuning all parameters
(c) The results of fine-tuning encoder and decoder
Figure 1: The experiments on CelebA64 dataset
The above result indicates that the few-shot diffusion models can still recover the true subspace with finite n, and Vla- Note that when the latent distribution is Gaussian distribution, the approximation enror bound for the source dataset is n, 1/2 instead of n, 22/ (Yuan et al. 2023). Hence, n, in Theorem [.2 do not depend on latent dimension d.
Remark 5.3. The bound of ||V - AA 12 3 only guarantees the subspace spanned by V and A close, which still holds after an orthogonal transformation on V. Hence, this bound does not indicate I Ali is small. Since all previous works (Chen et al. 2023c, Yuan et al. (023) consider ||VV - AA [| we also use this metric to measure the subspace recovery. However, our results are stronger due to the closed-form solution, where previous works do not consider how to obtain VVT.
5.1 Discussion on the Closed-form Minimizer
Better dependence on Tita, ō and d. Note that Theorem 5.2 has better 1/TTaa dependence on the target dataset compared to 1/ /ug dependence on the source dataset. Furthermore, the coefficient of ng term is dependent on the early stopping parameter ö and D. This is due to the ó and D dependence of the approximation bound, which is used in generating go. However, the nua term only has dependence. Hence, even in the latent Gaussian setting, we still need larger source dataset than the target dataset to obtain a sufficiently accurate closed-form solution.
The relationship with principal component analysis (PCA). The expected few-shot score matching objective can be simplified to
min 1/ VIVV...VV
m.
V. (
where = Ta- problem is similar to PCA. This suggests that few-shot models implicitly optimize an objective function akin to PCA. However, few-shot models extend beyond traditional PCA. More specifically, when X is large, classical PCA suffers from the influence of X², In contrast, due to (mī 2 - at)/ /nto term, few-shot models can select a large to mitigate the impact of X² and achieve a 1/Tia-
Note that when ignoring 1/0 and choosing O, the above minimization
Experiments
To corroborate theoretical findings, conducted experiments utilizing real-world datasets. These experiments show that the new model obtained by only tne-tuming appropriate encoder and decoder layers target datasets can produce novel images with the target feature, which shows the effectiveness of the methods and supports our theoretical results.
Datasets and benchmark. Note that human face images tend to exhibit similarity in their latent space, primarily due to shared facial features, while differing in specific features. Hence, we initially pre-train model using the CelebA64 dataset, focusing distinct hairstyle features the goal for the fine-tuning phase. For the source data, we construct a large dataset (6400 images) with different hairstyles (without the bald feature). For the target data, we choose the bald feature the target feature and select 10 images with this feature to constitute the target dataset, which are much smaller than the size of target dataset (Figure [ (a)). To show the effectiveness of our methods, we also fine-tune all parameters of the pre-trained models as the benchmark.
Discussion. As shown in Figure the results obtained by only fine-tuning the encoder and decoder layers can generate novel face images with the bald feature. Conversely, when fine-tuning all parameters, the models suffer from memory phenomenon and can only generate images that slightly

---

modify the brightness and angle of the target dataset. This phenomenon indicates that only fine-tuning the appropriate encoder and decoder will result in a model with generalization property.
We note that these experiments aim to verify the effectiveness of the methods instead of achieving state-of-the-an performance since previous works carefully select specific parameters, such specific cross-attention layers (Kumari et al. (2023) or special neurons (Liu et al. (023), to fine-tune pre- trained models. However, we simply fine-tune all encoder and decoder layers simultaneously. There are more experiments on cat faces and more discussion on why Assumption 3.1 is satisfied in our experiments. We refer to Appendix for more details.
7 Conclusion
This work aims to provide a deeper understanding of few-shot diffusion models from a theoretical perspective. Our analysis is conducted from two key perspectives: the approximation and optimization aspects, all under linear structure distribution and shared latent space assumptions.
From the approximation error bound, we consider general subgaussian latent variable and prove that few-shot models have a (n=2//d + 1/22)) approximation bound, which is better than Fta -2/a results of diffusion models without a fine-tuning phase and escape the curse of dimensionality. This result also makes the first step to explain why few-shot diffusion models only require 5 ~ 10 images generate great samples. The experiments on the real-world dataset also show that the fine-tuning phase only requires 10 images to generate novel images with the target feature.
When analyzing the optimization process, consider more special, shared Gaussian latent variable and prove that the expected score matching has a closed- minimizer, which indicates that the few-shot diffusion models can simplify the optimization problem. Furthermore, we prove that the empirical closed-form solution has a O (1/mta + 1/ (8))))) accuracy bound, which still has better 1/nra target data dependence compared to 1/ (6v/Ts) dependence on the source data.
Future work and limitation. When considering the approximation bound, we assume distribution with linear structure. Though has been supported by much empirical evidence, it is not as general as bounded distribution. After that, we plan to consider a general, bounded distribution and show the advantage of few-shot diffusion models. One possible way is to analyze the mixture of low-rank Gaussian (Wang al. 2024), which is more general than the linear subspace assumption.
We focus on a special Gaussian latent distribution when considering the optimization problem. As next step. we plan to consider more general latent distribution, such as log-concave distribution. In this setting, we can not directly obtain the closed-form solution. However, due the shared information and simplified landscape, it is still possible to use some optimization algorithms, such as gradient descent, to optimize the few-shot objective function achieve the convergence guarantee.
Broader Impact. This paper presents work whose goal is to understand few-shot diffusion models from the theoretical perspective. noteworthy societal impact that few-shot diffusion models may be used to imitate the style of artists and generate fake images, thereby infringing on the rights of artists (Mirsky & Lee T).T). We recommend adding watermarks to images to determine whether the image was generated by generative model (Fernandez et al. 2023). The other societal impact is the same as general generative models (Mishkin al. 2022).
Acknowledgments and Disclosure of Funding
The author Bo Jiang is supported by National Natural Science Foundation of China (62072302). The author Baoxiang Wang is partially supported by the National Natural Science Foundation of China (62106213, 72394361), an extended support project from the Shenzhen Science and Technology Program, and support from Longgang District Key Laboratory of Intelligent Digital Economy Security.
References
Benton, J., De Bortoli, V., Doucet, A., and Deligiannidis, Linear convergence bounds for diffusion models via stochastic localization. arXiv preprint arX8v:2308.0366 2023.
10

---

Blattmann, A., Dockhorn, T., Kulal, S.. Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprini arXiv:2311. 1.1111111122,,...111 2023.
Bruno, S., Zhang, Y., Lim, D.-Y., Akyildiz, Ö. D., and Sabanis, S. On diffusion- based generative models and their error bounds: The log-concave case with full convergence estimates. arXiv preprint arXiv:231 I. 13584, 2023.
Chen, H., Lee, H., and Lu, J. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. International Conference on Machine Learning. pp. 4735-4763, PMLR, 2023a.
Chen, H., Wang, X., Zeng. G., Zhang, Y., Zhou, Y., Han, F,, and Zhu, W. Videodreamer: Customized multi-subject text-to-video generation with disen-mix finetuning. arXiv preprint 1rX19:231..00990, 2023b.
Chen, M., Huang. K., Zhao, T., and Wang, M. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.077 01022.07/44 2023c.
Chen, S., Chewi, S., Li, J., Li, Y. Salim, A., and Zhang, A. R. Sampling as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint urXiv:2209.11215 2022.
Chen, S., Daras, G., and Dimakis, A. G. Restoration-degraaation beyond linear diffusions: non-asymptotic analysis for ddim-type samplers. arXiv preprint arXiv:2303.0334 003384444, 2023d.
Chen, X., Liu, 7., Xie, S., and He, K. Deconstructing denoising diffusion models for self-super learning. 2024.
Chua, K., Lei, Q., and Lee, J. D. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 01.8871-88884 2021.
Cui, H., Krzakala, F., Vanden-Eijnden, E., and Zdeborová, L. Analysis of learning flow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.0357,, 2023.
Du, S. S., Hu, W., Kakade, S. M., Lee, D., and Lei, Q. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.094433 2020.
Dutt, R., Ericsson, L.,, Sanchez, P., Tsaftaris, S. A., and Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. arXiv preprint arXiv:2305.0820 088522 2023.
Fernandez, P., Couairon, G., Jégou, H., Douze, M., and Furon, T. The stable signature: Rooting watermarks in latent diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22466-22477, 2023.
Gal, R., Alaluf, Y., Atzmon, Y., Patashník, O., Bermano, A. H., Chechik, G., and Cohen-Or, image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611
2022.
Gong, S., Boddeti, V. N., and Jain, A. K. On the intrinsic dimensionality image representations. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 3987-3996, 2019.
Han, L.. Li, Y., Zhang, H., Milanfar, P., Metaxas, D., and Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning, arXiv preprint arXiv:2303. 1305, 2023.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. arXiv preprint arXiv:2204.03455 2022.
Hu, E. J., Shen, Y. Wallis, P., Allen-Zhu, Z.., Li, Y., Wang, S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0968) 2021.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.

---

Karras, T., Aittala, M.. Aila, T., and Laine, S. Elucidating the design space of diffusion-basse generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931-1941. 2023.
Li, G., Wei, Y., Chen, Y., and Chi, Y. Towards faster non-asymptotic convergence for diffusion-based generative models. arXiv preprint arXiv:2306.09251, 2023a.
Li, H., Ma, L., Liao, Y., Cheng, L., Hao, Y., and Zhou, P. 3d-goi: 3d gan omni-inversion for multifaceted and multi-object editing. arXiv preprint arXiv:2311.12055 2023b.
Li, H., Shi, H., Zhang. W., Wu, W., Liao, Y., Wang, L., Lee, L.-h., and Zhou, P. Dreamscene: 3d gaussian-based text-to-3d scene generation via formation pattern sampling. arXiv preprint arXiv:2404.03575, 2024a.
Li, H., Tian, Y., Liao, Y., Wang, L., Wang. Y., and Zhou, P. Enhanced 3d generation by 2d editing. arXiv preprint arXiv:2412.05929 2024b.
Li, Li, Z., Zhang. H., and Bian, On the generalization properties of diffusion models. arXiv preprint ar arXiv:2311.0170 2023c.
Liu, M., Shi, R., Chen, L.., Zhang, Z., Xu, C.,, Wei, X., Chen, H., Zeng, C., Gu, J., and Su, H. One-2- 3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10072-10083, 2024.
Liu, Z., Feng, R., Zhu, K., Zhang, Y., Zheng, K., Liu, Y., Zhao, D., Zhou, J., and Cao, Y. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125, 2023.
Mei, and Wu, Y. Deep networks as denoising algorithms: Sample-efficient learning diffusion models in high-dimensional graphical models. arXiv preprint arXiv:2309. 11420, 2023.
Meunier, D., Li, Z., Gretton, A., and Kpotufe, S. Nonlinear meta-learing can guarantee faster rates. arXiv preprint 1.XXv:3307.10777 2023.
Mirsky, Y. and Lee, W. The creation and detection of deepfakes: A survey, ACM computing surveys (CSUR), 54(1):1-41, 2021.
Mishkin, P., Ahmad, L., Brundage, M., Krueger, G., and Sastry, G. Dall 2 preview-risks and limitations. Noudettu, 28:2022, 2022.
Moon, T., Choi, M., Lee, G., Ha, J.-W., and Lee, J. Fine-tuning diffusion models with limited data. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
Oko, K., Akiyama, S., and Suzuki, T. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303.0/861, 2023.
Pope. P., Zhu, C., Abdelkader, A.. Goldblum, M., and Goldstein, T. The intrinsic dimension of images and impact on learning. arXiv preprint arXiv:2104.0888 088944 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ruiz, N., Li, Y.. Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusson models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500-22510 2023.
Shah, K., Chen, S., and Klivans, A. Learning mixtures of gaussians using the ddpm objective. arXiv preprint ar arXiv:2307.011 01 2023.
12

---

Song. Y., Sohl- Sobk-Dickttein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arX1v:2011.13456. 2020.
Tenenbaum, J. B., Silva, V. d., and Langford. J. C. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500);2319-255 2-23,,, 2000.
Vincent, P. connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674, 2011. 2011.
Wang, P., Zhang, H., Zhang, Z., Chen, S., Ma, and Qu, Q. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02226, 2024.
Xiang, C., Bao, F., Li, C., Su, H., and Zhu, J. A closer look at parameter-efficien tuning in diffusion models. arXiv preprint arXiv:2303. 18181, 2023.
Yuan, H., Huang, K., Ni, C., Chen, M., and Wang, M. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. arXiv preprint arXiv:2307.07055, 2023.
Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
Zhu, J.. Ma, H., Chen, J., and Yuan, J. Domainstudio: Fine-tuning diffusion models for domain-driven image generation using limited data, arXiv preprint arXiv:2306. 2an.:23336.1115,,. 2023.
13

---

Appendix
A
The Neural Network Structure
In this section, introduce the multi-layer ReLU network E NN (L, MM JJ, 7. in SNN- We note that the following setting is exactly the same as the one in Chen et al. (2023c), and we show the structure for completeness. We denote by NN (L, M', J, K1, Kg Y, (+) the following neural network:
NN (L, M', J, K1,K, 7. (f(Z.t) W7.0 (... Wi [z] .t] + bb) (b.)..) b, network width bounded by M M', sup [f[(z, f)]l2 S K1. max [l|b;]l 1|W,ll - k for I.....I. L,
Σ (IlWallo + ||b: - J 7=1
|f (21, t) - f (Z2, 1) S 7 11 - 12ll2 for any E [0, [), || (Z, - (2,12) 7 11 (2l for any Z}
where is the ReLU activation function. Given network accuracy by
0, the parameters is defined
L - log + K1 -
2d log
8E
d
M' = 0
(1 Td ((d+1) log
102
4/2
)
5E
J = 0
(1 (1+ Td // (d++1)
(d+1) log 4/2
(d)
log
+ +aa):
+
6E
d
7)
O max 2(1 1 dlog
NTTT
TT
8e
7 10d(1 + 8), 10T,
where 7 = A suPte],,T] sup ||llo /dlg z
1
[o, log d. (x) l1z
B The Proof of the Approximation Error Bound
Let (Va,0) be the minimizer the pre-trained objective function. The few-shot diffusion model freezes the bottleneck network and fine-tunes Vta E Dxd to obtain the minimizer (Vi.. 0) of the few-shot objective function. As the first step, show that with the bottleneck parameterized by 0. there also exists solution (Vta; 0) E 2N achieve the e2 + nis -2/d
error bound.
1-utne)
Lemma B.1. Ife S nl
where a(n)
logn then with probability 8, there exists log n
solution (Via, 0) QNN(0) such that
[" Ex-ei" |1 Voog g: (x) dt <o
d e
(T ---07 --0)(d C2)
2-20()
log( 1 11))
ns
8200
Proof.
As shown in Theorem 1 of Chen et (023c), there exists a solution (V., 0) in SNN such that
Vd+1 of e, Vt [8,7].
|s - V Vlogg(())
14

---

Hence, we do the following decomposition:
(...o(:; V Vlog ol ()) - 10. log o1 ())1 + b(:,)( -s (,s - 1)
For the encoder and decoder layer, we choose Via = Ata- The first term is bounded due to the construction of the neural network. We first show that sv... is e-close to the true score function v of" Since the encoder and decoder have been chosen, we only need to focus on the latent bottleneck. For the latent bottleneck, we need to use f Ztt) to approximate ground-truth function Zz, 1) oiV log "t (2) Z for Z E R. Chen et al. 2023c) show that for any latent variable Z' E I2 with subgaussian tail, we have that ||A (Z', t)
fo (Z', ( S (Vd++ 1)c.
Then, have that
|| (1.1) V log tt ())2 og
For the second term, we know that with probability 1 1 ő1:
Ex~gi
51,,0(,, t) 2
<
E Ei
[[[g(Z) 11 dt
8
((T 5) d 7² d3) + 72 22.Cz )ns
2- 2atinal
1
0
443
log
82
co
co
81
where a(n) =
The first inequality follows Ata is a matrix with orthonormal columns. 1-alea) Since we assume S ni
dlloglog log
the network has good enough ability to obtain an accurate enough
0. Hence, directly the true matrix VT, A.a instead of the approximate Vza. we do not need orthogonal transformation and can choose U = Ia in the Appendix C.4 [Chnn et al. (202cc). Then, we complete our proof.
Appendix C.4 of Chen al. (023c) to obtain the second inequality. Since
To prove Theorem 4.3 we need to do the following decomposition for the population loss of the target datasets
Lia (SP...)
Lia (sV...0) (1 + a) (sp...) + (1 Lro (st...)
Ctrune
(8Pro.)) (1 + a) Drune () + fea (Spmn
pirune
(sVr..) +(1 + a)
inf
Eno (Svin))
"veu a EQQ(0)
(a)
(b)
where E (0, 1) and Ciune is defined as
Cirne (V Exngo
[etrune
(w: }] Ex [eea (x: 1 1l[ll112 < R) dt
tta
In this section, we take
( (d/dloow log d + log K₁ + log uia 8,
0
0
to
guarantee
P Po (1)Xaa,ll2 I R for all i= -,....,, Rea)
81 where Ki is defined in Appendix A
Term (a). Similar
Chen et al. (023c). we define a function class (( Atrunc (::yy)) : Vv E (0) which is induced by 2(0). For the upper bound of G(Ø),
directly use the augmentation Chen et al. (202cc) to obtain that
K? + R²
ptrunc ta
x (X:sy;6)
0
for any Sv.0 E QNN
(T 8)
15

---

Then, by using Lemma D.I we know that with probability 1 81, term (a) is bounded by
o (1 + 3/a) ((1 + 3) log de + log
(1+ nea (T - 5)
Rea 5
N T1, (()), |l l1xe) +
log
To bound the above term, need to calculate the covering number of ( which related to i-covering of QNv( Suppose that given Vv, and Sy2,0 with |v (M, - Syg, 1)/12 grame
S we need to bound
(::5vi.)
(::Sv2.0)
By using the same calculation compared Term (A) of of et al. (2023c). we know that
(mme (::8y.6)
étrme
(::8ve.e)l S so (
(Ki + R) log
T +
4Ki(K - R)
(R/D) D-2 exp
(-
(T -8) 5)
The above inequality indicates that i-covering of QNN(() in L2 norm leads a 7-1 (K2 log (K,,K (R/ D-
exp (- R²) covering of of G( in Tox norm.
8(T 8)
By taking R = o ( dlog d log K + log 01 know that
K1 = O (2d log (z)),
2 DT -T- we
< d²
log( log( d log(
Tha
de
which indicates with probability - 51, term (a) is bounded by
(1 + 3/a) ((1 + (2 log d log mia) dc 5 (T - 8)
N
(nnol[--o] QNN(0),11-1)
+ d2
T log(
d log( meo
log
log( 8
81
nea
be
After that, we need to determine the covering number of QNN with truncated X to bound term (a).
Lemma B.2. The logarithmic covering number of QN || s 3R + VDlo D, [8.
log. (c, (NN, =o (2Dd log (1+ 1+ 6K>vd(33 Dog log D)
Proof.
Suppose that there exists two orthonormal column matrix V1, V2 such that 1| -Vllp S 6z. then we have
sup
(xX,,
[3 (.ET
1/ (V,Tx; x,t) Vifa (V,T x,t) + V1. (V22 a,
Vgfo (V₂ x,
oi
sup 1
1 oo
$ (,5.Vo(3R+ / Dlog D) + 8,KK)
For (V E Rxd [|]| 1}. the dz-covering number is (1 + 2 200
Dd
Then we know that
log (1, SNN, 1l (12) = o (2nd log 1+ 6K 6K>vd(3R + VDlog D)
51.
16

---

Term (b).
For the term (b), the proof of Theorem 2 in Chen et al. 2023c) shows that
Term (b)
nea (T -6)
Term (c).
For the term (c), we know that it is bounded by the constructed solution (Via, 0):
inf
Lea (Svi.)) S Lra (sii..) - (1 + tume
(V... +(1 + a) Lrr (Svi..
SVa (E(()
(C:)
(Ca)
For the term (C.1), since sy... is fixed function, we directly use the results of (Chen et al.
2023ch:
Term(C1) 00 (1 (1+6/o 6/a) ((1 + 3) log de + log 7)
1
log
nead (T d)
with probability 1 - 81. For the term (C.2), we know that
1
6 L
Vlog aha(((() ()) L((()))
Cetune (Sv...0) S L (8Vr..)
dt
+ L ( (spm))
T δ 1/
/s 66: - V log dt (-) 1e6
( di
As we show Section 3.2, the two terms in & are both score matching objective function and have a constant different E, which is independent of the trainable parameters (V, 0). We denote by this difference E and F 8 bounded by
(d"
With probability 1-6, 61, Lemma B. shows that term (C.2)
o
2
Fn,
2-20( 445
log
1 1 ) + E.
6(T - 8)
51
After bounding these three terms, we prove Theorem
Theorem 4.3. Let c(())=
d log log
F
3d 8
and network parameter
n2, = -1/2 Assume
2+5
Assumption 3./ 4.1 4.2 and nea
2 ng. Then, with probability 1 1- -,, the following inequality
holds (hiding logarithmic factors)
T 8 [/
1
|0) V Flog loggi" ()) LAGEE di = = (()(((1) (1+3))DDt (((((11-))
2-20(mx)
1
log Fris
( 8) \nnz
Proof. Equipped with the bound of the term (a), (b), and (c) and hiding the logarithmic term (except for the covering number term), with probability 1 - 81, we have that
Lia (sv...) (1 (1+a)2E+0) (3) log de + log Uea
N
Ueao 5J. 2NN @NN(0),)1
-11)
+ d2
log
ad (T 8) nıa
81
n
1 + 5(T 8)
d 2 + Fns
Buimal 243
log -
51 )
neo 8[T 8)
17

---

Since Lio (Sv...)- E 7-8 So [van V log (-)
we have that the following
[,(())))
inequality when choosing a = :
()] 12(())
V log loggi (1)
T 8 8
N
Puu((T (m QNN
(1 + .)2
d
df
2-20((a)
0
log
2
+
Fns
d+3
log
26 (T -8) 1za.
51
5(T 6)
Rea
(1 + 1+13)2 log
1+ 6K 6Kvva(3R + Vllog D) Dinao - -5)
d
d2
+
2
8 (T 6) neae?
61
5(T 6)
nea
2a(me)
+ Fns
4+3
log
81
where the second inequality follows the convering number of QNN(O) for |X||| S 3R + /D log D
with R = O (v dog d log K1 + log 81
Rea and the network parameters is defined in Appendix Finally, choose e² 1/ 11//Taor Mia. then have that
T 8 / || v log 2 ())
so
(1+ 3) log (1)
(1)++
d2
Bo((na)
81 )
T-8
(2))))
8 8T - 5) V
8,
nead
+
Fr
log
1-a(n.]
As require in Lemma B.1 we need 72.-
1/m
1/4 I ns
which indicates
[[1-o(na]]
C
The Proof of the Optimization Problem
C.1
The Pre-trained Diffusion Model Generate Accurate Enough Latent Distribution
Since we need to use the approximated latent distribution in the few-shot phase, we show that the pre-trained diffusion models with solution (V., 0) can generate an accurate enough latent distribution. As shown Section when considering the optimization perspective of diffusion models, we assume the latent distribution is Gaussian distribution 9. = (00 Σ) with Σ = diag (X?...., X3) > 0. Yuan al. (2023) show that in the setting, the approximation error bound (Lemma D.3 for the target dataset is
8 T
(
(d2 + Dd) log (Ddn.) (d2 V D) log 1 of
T T-8
V of -v- (c -
dt 0
8/
(2(
n
To generate latent distribution, we first introduce the reverse process in the latent space. The introduction mainly follows the outline of Appendix C.2 of Chen et al. (202cc). For X,, we can do the following decomposition: X, A.Z, Xt,I, where 7. A, X With Z-- 2T the reverse process in the latent space is
d2,
2t + +
LD log -T (Z di + d (A. B,)
As shown in Theorem of Chen et al. 2023c), the solution (V., of the pre-trained diffusion models A,A, 1| is small instead of [V2 A+l is small. Hence, Theorem ohen et al. (2023c) assume there exists an orthogonal matrix U, Rdxd and do an orthogonal transformation on V. to obtain V,U,, which can guarantee VVUU. A, is small, After such orthog- onal transformation, the reverse process with an approximated score function and approximated reversing beginning distribution Z N (0, is
only guarantee IV.
d7, - =
2
7.
s
U
LD
T T
dt (U.V, ~ (N,
(2)
18

---

where
1 - - -,)
= 2; and U (2,(
=U;
Then, discretize the above process with the exponential integrator (EI) discretization scheme (Zhagg & Chen 2022) to obtain an implementable algorithm:
d2.
Tzmn" T -11)
dt (U.V where e 1411=
U..0
As shown in Appendix C.4 of Chen et al. (023c), if the target ground truth score function has L2 -cccurree approximated score:
T T- l. (s., - Vlog (2(a)) dt S ce,
the latent score function also has an Ly norm bound latenn which is determined by c:
([ ((T log 0) d3) + 7² Ozz
Catent scure = c.
The remaining term to determined e. Since assume Gaussian latent variable instead of sub- Gaussian one. Hence, we do not use in Chen et al. (202cc). We use in Yuan et al. (2023) (Theorem 4.5 of Yuan et al. (2023), which also considers Gaussian latent variable, achieve the final results. Finally, we have that with probability 81: -
T 8 { V (P)())
1
(.)
U. LD (s " df
d B2 (d +miaa) (d3 + Dd) log (Ddn,) (d& v D) log
2
fatent-score
Amino
Let pt LDD be the distribution of the algorithm (the above discretization process). In the following lemma, adopt Theorem 5 of of(Chen al. (2023a) and show that the pre-trained diffusion model can obtain an accurate enough latent distribution with the above Tız-accurate latent score function.
Lemma C.1. With Giatent score defined in Equation (4), T = log Canoraacor
and K
d' ((TT+ooCAaaa)))) by using the exponentially decreasing (then the constant) stepsize hyg
}11...... _e= 1 the results pl." of sampling algorithm (Equation
emin mmxx tk Xim
K
has the following guarantee with probability 8, (hiding the logarithmic factor):
KL (4oll s (c
d² (d X2 max)
(d2 + Dd) log (Ddn.) (df D) log bn
= δ
Amino
2023a) show that V log oD is L--Lipschitz, diffusion models which close to go in KL divergence. Since qu (0, E). is easy to verify L = Amax Then, we complete the proof.
Proof. The Theorem 5 of Chen et al. with Lacaccurate can generate prr
C.2 The Closed-form Minimizer of Few-shot Diffusion Models
When the latent distribution is Gaussian distribution 9= (O, with Σ diag (Ai,... A3) x 0, the ground truth score function for the target dataset
1 (ID
V log at (X) A
A
Ala X.
of
19

---

where Σg diag (..,m,}?, + of...... Since the matrix Ata is independent of time t, we fix t E [6, T] and minimize the few-shot objective function at this time. With an approximated Σ. which learned by the pre-trained diffusion models, fz(2, = -Ia of 1) where 5, m 5 + of la-
Hence, the expected objective function for the few-shot diffusion models at fixed time is:
min ( Ex [400 (Xeai (xaaivv..00)
Van @ (0)
where
NN) (sv (X
(X,
Vfo (V X,t)
7 :: V RDxd with rank(V) d.
o²
of
of
Note that the constraint rank(V) = does not involve length information.
is a weaker constraint than V Ia since rank(V)
Lemma C.2. Assume Assumption
3.1 and q: -0 (T)). Let C Exenngia [XtaX be the expected data covariance matrix. Then, Via has closed form:
Via VT = m²x² + of (C + (C+ c.
X2
Proof. Let G. = la -07 1 then we have that
da (Xiost8vin.)) = -Ex, Xo X
1
o7 X, - V log o (Xi
oi Bx,IXX=Xia.
[IViaG.Vio X, - Xol1
where the second equality follows lo (X, = -Xi-mix Let C = Ex [ta X to
of be the expected covariance matrix of the target dataset. With the fact Ex,,,, XX,,,, of and Ex,, x. Xx m, Xo the optimization problem can be simplified to the following form (without misunderstandinn, we ignore the subscript ta):
X
C(V) 1l ofID)+
2m(
min VERPX D raank(V)=d
where (mfC + of T à is meaningful since (m,C + TD) is positive-defintte matrix. Let V be the solution of the above minimization problem. We first ignore the constraint rank(V) = d and calculate @L(V)VV = (since V also satisfied (L(V)/OVlv= -v= O), we know that V satisfies the following equality:
(m²C + VV = =
which indicate
(mm) + V.I - m,C)(VVG)) ODXd.
The above equality means rank((mfC ofID)VG.VT m²C) rank(V s d. Since rank(V)= d and rank(G.) d, then have that rank(VVā,) = d and
(męc .... = m²C
(5)
In Section assume the latent distribution is isotropic Gaussian distribution q= = (0,0 In this setting, Σ is equal to X2 la and G. is
mi{²
which indicate the closed form solution of V m m²X²+0?
m x + of (C (C+o2ID)
VV
X2
The last step to prove rank (VV) d. Note that rank(C) rank (Exis (XtaXTT])
rank(AA rank(AEAT) (AEAT) = d, which indicates
(in(rank(VVV) rank rank o(ln)) 2 d.
Combined with E Dxd, complete the proof.
20

---

C.3 The Error Bound for the empirical Closed-form Solution
In this part, we prove the accuracy bound of the empirical version closed form solution VV w.I.t. n, and na The empirical solution has the following form (without misunderstandinng we ignore the subscript la):
T VV
m2 + of
+
X²
where C= EE Xta; ta Bia
Theorem 5.2. Assume Assumption
3.1 and Oz A2( Let âz be the latent distribution
generated by the pre-trained models with (Via; Σ. and M = (d VD DDdlog (Ddn,) (d2 V D).
Then, with probability 51, we have that for any E [6.
2 Via LaVio ta
ATAATo =ō
())
M
d 0mm 0033)
+
mz of
dō/n.
nho
Proof. The empirical solution indicates that
m
of C.
(mfC + +o
(6)
X2
To analyze this equality, we first show that X2 is accurate enough. We know that KL (gollihP) =
(x²/X² Io0--2//g//////> /X²) - -1) Let M1 d² (d+>max))
ViP + Dd) log (Ddn,) (dP v D). Then,
Amind
Lemma C.I show that with probability 1 81, we have
log(1/51)
KT. (10 (lp = d (X2/2² 1og(29/89-1)) -1) =o Miy
Combined with the above inequality, we know that with probability - 81, 122 - 1 <
Mi / [og(1/61)
by using the Taylor Expansion. By using Lemma we know that with probability
dyn,
81:
m. x2 + o
m, + 2207
)
(I (1 2 d + log (1/61)
AAT
X²
X²
Vnea
Mi Mivlo((1/61)
2 + log (1/81)
mZ?² + o7 +
o3
(1+
TAT
d
/Tta
For the left hand of Equation
6). we know that
(mfC Cm(C+o(lD)VV
m2XPAAT m
log (1/81)
+ oLDD
=sT VV
(8)
Tiaa
Combined with Equation 7 and Equation
we have that
(m?A>AA AAT
IDD VV
AAT))
Mi (og///(1/8)) dy
2 2dd= log (1/8,) Vnta
Mi /log(1/8))
07 )) AAT
m2x² + of +
dent
+ 2m
d + log (1/01)
VV
=
/ //r
21

---

Let M2(n,, nta)
Mr Vlog(1/61) dyn,
2 /
m2 + of +
Ms (loloo11/6)) dyna
oi
Accord-
Vieaa
ing to symmetry, we know that
(mp +oīTo) (VV' - -MT))
- 50 20(00(aannn
mf + log (1/61))
==
11
IV
11's
nta
H (D2(M2nn,,,
The last inequality follows that each element of VV bounded by some constant due to the form of the empirical closed form solution and
- tr(AAT -TTAAAAAAAA>> - (T(AAT)
(r(Ta) d.
For the right hand of the above incquality, we know that
(vV - AA )1;
(miXAAT +o[ID)
=Tr ((vV) - AAA
(VV -AAT)
(miX +di1D) (m²>AA +o]lo)
+ +o?) || (VV - -AA )
2 (m (mp
Then, we complete the proof
D Auxiliary Lemmas
The following concentration lemma comes from Lemma 15 of Chen et al. (023c)).
Lemma D.1 (Lemma 15, (Chen et al. 2023c)). Let G be a bounded function class, iee, there exists constant such that any g E 9 R2 B). Let 21, ........ Zn RRd be ii.d random variables. For any å E (0, 1), &. 1, and > 0, we have
(1 (1+3/a)B
N (T.
(T. 11x0) (1. + +(2 +(2+a)7) Sb and
Σ 9 (z.) - (1 aa)B((())]
sup DEG TI
log
3n
81
1=1
1
(1 [1+66 6/a)B
(T. 9. (loo)
sup [z)( DEG
9 (z.)
log
+ (2 + a)r S
3n
51
In the following lemma, show the concentration of the data covariance matrix. Note that the proof sketch of the following lemma mainly follows Lemma 6 of (Du et al. 2020). We prove concentration bound that depends on instead of constant bound with large enough n.
Lemma D.2 (The Modified Lemma A.6, (Du et al. 2020)). Let ai,..., an be i.i.d. d-dimensional random vectors such that E [a:] IE
a.a; 1, and a, is ps-subgaussian. Then with probability
at least - òn we have
1 2p² v d + log (1/61)
a T
1+ |++20*
2p² 2 + log (1/ō1)
Vi
Vn
a.a. Similar et 2020), we use an e-net argument for the unit sphere 8d-1 = E Rdd
Proof. Let
n n
|||||| 1}. For any 8d-1, know that (v T a.)² 1 is zero-mean
and 16p2-subgaussian. By using the Bernstein inequality, we have for any e
Pr [ Aul
< 2exp
(-2 min { (16p2) 16p² )
22

---

Next, we take _ nett N C gá-1 of s with size W s el14) By using the union bound, we know that
e²
Pr
max wEN'
v Aul
S 2| exp
(-2 min
(16p2)2'
16p²
e²
I exp
O(d)
n
min
2
(16p2)2* 16p²
Let the right hand of the above inequality equals to 81. We know that with probability 81:
p² /d log (1/61)
max 1
Au
DEN
vn
Note that for any u E S there exists u' E N such that ||u u' S Then, we have that
|u T Au S (u') T Au' + (u u') T Au'l + +l(u u') A (u u')l
p² /d + log (1/61)
+ 2 ||
vn
where ||A||2 is the operator norm of matrix A. Taking a supreme over u Sd-1, we have that
|||l2 - 1² Vd + log (1/61)
+ 2 ||411/2
vn
Then, we complete the proof.
In Section 5 we assume the latent distribution a Gaussian distribution instead of a subgaussian one. Yuan al. 2023) show that in this setting, the approximation error bound has better dependence on Y1.s
Lemma D.3 (Lemma C.I, Yuan et al. 2023 ). Assume the latent distribution is a Gaussian distribu- tion (0, Σ) with Σ = diag (
X > O. Then, the solution (V., 0) of Equation has
the following approximation error bound with probability 81:
( 8/
T - 8 1'4
(d2 + Dd) log (Ddn,) (d2 V D) log of n
Vlog (1) -8, (.tt
dt 0
L2(
E Additional Experiments
In this section, we do experiments on real-world datasets to show that the new model obtained by only fine-tuning appropriate encoder and decoder layers on target datasets with only 10 images can generate novel images with the target dataset feature. On the contrary, if all parameters can be fine-tuned, the model will suffer from memory phenomenon and only generate the ten images in the target dataset. This phenomenon indicates that only finettunigg the appropriate encoder and decoder will result in model with generalization property.
Setting. In this experiment, downblocks, 2 middlcblocks, and 15 upblocks. When only fine-tuning the encoder and decoder layers, fine-tune the first downblock layers (encoder) and 4 upblock layers (decoder) instead of only using linear layers as the encoder and decoder (discuss the later discussion paragraph).
U-net network with attention layers, which contains 11
The above experiments are conduct on GeForce RTX 4090. We train the neural network using AdamW optimizer with learning rate 0.0001. For the pre-trained phase, we train the models for 200 epochs with batch size 20. It takes 5 hours to obtain pre-trained diffusion models. For the fine-tuning phase, we fine-tune the pre-trained models for 400 epochs with batch size 2. take minutes to fine-tune the pre-trat pre-trained models.
23

---

Target dataset
(b) The results of fine-tuning all parameters
(c) The results of fine-tuning encoder and decoder
Figure The experiments on cat face dataset
Dataset.
Our experiments use 2 real-world datasets: the CelebA64 dataset and the cat face dataset.
CelehA64 (size 3 64 64).
(a) Source dataset: 6400 images of faces with different hairstyles (without the bald feature).
(b) Target dataset: 10 images with the bald feature in CelebA64.
Cat face images (size 3 64 64).
(a) Source dataset: 4200 cat images with different colors (without black color cat).
(b) Target dataset: 10 black color cat images (The color black constitutes more than 70% of the image's composition.).
Discussion on results.
The experiment results of CelebA64 have been discussed in Section 6 The
experiment phenomenon similar for the cat face images, which means the models obtained by only fine-tuning the encoder and decoder can generate novel images with the target feature (Figure 2 We note that when choosing the target cat face dataset if the color black constitutes more than 70% of
the image's s composition, we view this cat image as the black cat. Hence, different colors exist for
cats, such as white and grey, due the target dataset containing a small number of these colors (such as images 1, 3, 4, 6, 8). As result, our fine-tuning results also contain these colors. However, our results do not contain colors other than those the target dataset and can produce novel samples, which also proves the effectiveness of our fine-tuning method.
Discussion linear encoder and decoder. Assumption 3.1 assumes the linear subspace, which indicates linear encoder and decoder. However, we fine-tune the first 4 downblock layers (encoder) and upblock layers (decoder) instead of only using linear layers as the encoder and decoder. We note that this operation does not conflict with our Assumption. Recall that in Stable Diffusion (Rombach) ict al. 2022). the diffusion models run in the VAE embedding space Hence, we can view the first 3 downblock layers and the last 3 upblock layers the VAE encoder and VAE decoder. Then, can obtain X this paper by running the VAE encoder. The remaining downblock and upblock layer can be viewed as linear encoder and decoder A. As mentioned in Section 4.2 of StyleGAN (Karras et al., 2019), the feature of X obtained by running a good-enough VAE encoder has linear separability, which also supports Assumption 3.1
"To distinguish the latent space in this paper. embedding space here.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This work aims to explain the reason why few-shot diffusion models can achieve great performance with limited dataset from the theoretical perspective. We achieve this goal by analyzing the approximation and optimization perspective (Theorem 4.3 and Theorem (5.2). We also do real-world experiments support our theoretical results (Section 6).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss future work and limitation Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have shown all assumptions, theorem and proof sketch in the main paper. The detailed proof appears in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We has shown all experiments detail in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: As theoretical work, we simply train and fine-tune diffusion models on the
datasets in Appendix B All detail is shown in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips crijps.cc/ublic//uidde.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We has shown all experiments detail including dataset and training detail in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since this work is theoretical work, the experiments are qualitative experi- ments on the dataset we constructed and are used support our theoretical results. Since we only use 10 images to fine-tune the models to generate images with specific target features, it hard to calculate quantitative metrics such as FID. However, our experiment results still show that our methods can generate novel images with the target feature compared to the benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have shown the compute works and computation time in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://eeuris.. htt.e/errppp...ppppptceecchcccce
Answer: [Yes]
Justification: We have checked the code of ethics and make sure that our work satisfies the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impacts of our work at the end of main paper.
28

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "bndxK3YWNY", "reviewer": "Reviewer_XEHa", "summary": "This paper studies the few-shot transfer in diffusion models. Specifically, it focuses on bounding the score-matching loss for the target distribution, which can be considered the estimation error of the corresponding score function. By assuming the source and target distribution share the same linear structure distribution on latent space of dimension $d$,  it gives a $\\tilde{\\mathcal{O}}(n_s^{-2/d} + n_{ta}^{-1/2})$ bound for training the few-shot diffusion model with $n_s$ source domain data and $n_{ta}$ target domain data. As claimed by the authors, this breaks the curse of dimensionality for the target domain with an improvement from $n_{ta}^{-2/d}$ (no few-shot adaptation) to $n_{ta}^{-1/2}$. In addition, they also proved a bound $\\tilde{\\mathcal{O}}(2/n_{ta} + 1/\\sqrt{n_s})$ for the optimization error of score matching loss for a latent Gaussian special case.", "questions": "1. Why $\\tilde{V}_{ta}$ in Lemma 5.1 depends on time $t$ ? Shouldn't it be the minimizer of the integral over $t$? \n\n\nGrammar: Line 271, 278, 280, 289, etc ...\nI suggest the authors check the grammar and revise the paper accordingly.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This paper provides theoretical guarantees for few-shot diffusion, illustrating sample complexity improvement w.r.t non-transfer diffusion process.\n* The structure is clear and easy to understand, even though the writing quality could be further improved.", "weaknesses": "* Approximation error has a specific definition in learning theory, which characterizes the inductive bias caused by the selection of the hypothesis class [1]. When using the terminology as another definition, e.g., the score matching loss of an estimated score function with early stopping in Theorem 4.3, it should be obviously defined to avoid confusion.   \n* I have the following concerns about presenting and interpreting the bounds:\n\t* In Theorem 4.3, the bound depends on $F$, which is related to $d^3$. So, the order should be $\\tilde{\\mathcal{O}}(d^3(n_s^{-2/d} + n_{ta}^{-1/2}))$, I am not sure if $d^3$ could be hidden directly. \n\t* According to this, when $d$ becomes large, the error should grow, which means the requirement of $n_{ta}$ should grow. However, in Table 1, with the increase of latent dimension, $n_{ta}$ decreases for ImageNet and MS-COCO. This is contradictory to your results. \n\t* When $d<4$, isn't few-shot fine-tuning worse than directly training given the theorem? \n\n* I am not quite convinced by the explanation of the experiments.  \n\t* How is the difference between fine-tuning all the parameters and fine-tuning a smaller fraction of encoder-decoders reflected on your bounds? \n\t* I think there exist many ways to adjust the hyper-parameter for fine-tuning all the parameters to achieve similar results as fine-tuning a small part.\n\n[1] Shalev-Shwartz, Shai, and Shai Ben-David. _Understanding machine learning: From theory to algorithms_. Cambridge University Press, 2014.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions. We provide our response to each question below.\n\n**W1 & Presentation.** \n\nThanks again for the valuable comments. Before using the approximation error to represent the score matching loss with finite datasets, we will make a clear definition in the notation paragraph to avoid confusion. We will also polish our presentation according to your suggestions.\n\n**W2 (a) A clearer discussion of the results of Thm. 4.3.**\n\nWe note that the goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models, which means that we consider the relative relationship between $n_{ta}$ and $n_s$. Hence, if the coefficient of $n_{ta}$ and $n_s$ has the same order, we can only consider $1/\\sqrt{n_{ta}}$ and $n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}$. To support the above augmentation, we first recall the results and calculate the coefficients:\n$$\n\\begin{align}\n\\left(\\frac{(1+\\beta)^2 Dd^3}{\\delta\\left(T-\\delta\\right)\\sqrt{n_{ta}}}+\\frac{\\left(d+C_Z\\right) d^2 \\beta^2}{\\delta^2 c_0}n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}\\right)\\log\\left(\\frac{1}{\\delta_1}\\right).\n\\end{align}\n$$\nThe dominated term of coefficient for $n_{ta}$ and $n_s$ are $Dd^3/\\delta$ and $\\frac{d^3}{\\delta^2c_0}$, respectively. The classic choice for the early stopping parameter $\\delta$ and forward time $T$ are $10^{-3}$ and $10$, respectively [1]. Then, with $D=256\\*256\\*3$ as an example (Since smaller $D$ is more friendly to $n_{ta}$, our discussion holds for all datasets in Table 1.),  $Dd^3/\\delta = d^3\\*20\\*10^6$ and $\\frac{d^3}{\\delta^2c_0} = d^3\\*10^6/c_0$, which has the same order. Hence, we consider the relative relationship between $1/\\sqrt{n_{ta}}$ and $n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}$. We will add the above discussion and provide a clearer explanation.\n\n**W2 (b) The discussion of Table 1.** \n\nAs shown in W2 (a), the fine-tuning phase aims to achieve the same error bound compared with source data. To achieve this goal, the first step is to determine the error bound of the pre-trained model. We note that this error bound is heavily influenced by the latent dimension $d$. More specifically, when $d$ is large (e.g. ImageNet), the error bound of pre-trained models has a large error even with large-size source data. We only need a few target data to achieve the same error. When $d$ is small (e.g. CIFAR-10), pre-trained models have a small error, and we need a slightly large target data size to achieve a small error.\n\n**W2 (c): Directly training for small $d$.**\n\nFor $d\\leq 4$, the models achieve $1/\\sqrt{n_{ta}}$ without the prior information from the source data. However, $d$ of common image datasets is larger than $20$ (Table 1). Hence, it is meaningful to consider our few-shot fine-tuning process, which fully uses prior information.\n\n**W3 (b): The discussion on fully fine-tuned methods.**\n\nIn earlier times, fully fine-tuned methods, such as DreamBooth [2], provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters (Fig. 6 of their paper). To deal with this problem, they design a prior preservation loss as a regularizer, which needs to be carefully designed. Furthermore, a fully fine-tuned method is both memory and time inefficient [6].\n\nTo avoid the above problem, many works fine-tune some key parameters and achieve success in many areas, such as text2image and medical image generation [3] [4]. These works not only preserve the prior information but also have a significantly smaller model size, which is more practical for applications. Hence, we focus on these models in our work. We will add the above discussion to our introduction.\n\n**W3 (a): The bound for fully fine-tuned methods.**\n\nAs discussed above, the fully fine-tuned method tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use $\\hat{\\theta}$ learned by the pre-trained model and achieves a $n_{ta}^{-2/d}$ error bound, which suffers from the curse of dimensionality. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term.\n\nWe also note that it is possible to avoid this phenomenon by using a specific loss [2] or carefully choosing the optimization epochs [5]. We leave them as interesting future works.\n\n**Q1: The optimization problem with a fixed $t$.**\n\nWhen considering a linear subspace, the diffusion process happens at the latent space, and $A_{ta}$ is independent of $t$. Hence, we fix a $t$ to solve the optimization problem. As discussed in Sec. 5.1, the objective is more flexible than traditional PCA since it can choose suitable $t$ to avoid the influence of large $\\lambda^2$.\n\n[1] Karras, T., Aittala, M., Aila, T., & Laine, S. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.\n\n[2] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.\n\n[3] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. ICCV 2023.\n\n[4] Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. *arXiv preprint arXiv:2305.08252*.\n\n[5] Li, P., Li, Z., Zhang, H., & Bian, J. On the generalization properties of diffusion models. NeurIPS 2023.\n\n[6] Xiang, C., Bao, F., Li, C., Su, H., & Zhu, J. (2023). A closer look at parameter-efficient tuning in diffusion models. *arXiv preprint arXiv:2303.18181*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPop0xYmJG", "reviewer": "Reviewer_kXcP", "summary": "This paper provides new bounds for the score function approximation and optimization in fine-tuning pretrained diffusion models (under some simplifying assumptions including linear structure of data distributions and equal latent distributions). The new approximation error bound depends only on the square root of the finetuning target sample size, which in turn suggests that fine-tuning diffusion models does not suffer from the curse of dimensionality (unlike their pretraining error bound which has an exponential dependence on the latent dimension). Furthermore, the accuracy bound and closed-form minimizer provides interesting insights into the fine-tuning behavior of diffusion models. The paper also provides limited empirical support for its theoretical claims on CelebA.", "questions": "Please see my questions in the Weaknesses section.", "limitations": "The paper sufficiently discusses its weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides novel theoretical bounds that can explain an important property of diffusion models (namely their fine-tuning convergence). Aside from the inherent theoretical value, I also think the provided insights can be very valuable both for the current practice of fine-tuning diffusion models, and for their future advancement. The paper is well organized and mostly easy to read (although the paper can benefit from simplifying its notations a bit more).", "weaknesses": "I do not see any major weakness in this work, but I do have two minor concerns:\n\n1- The paper mentions that data-efficient fine-tuning of diffusion models is very successful and tries to explain this phenomenon. However, the paper does not clarify the extent of this success. I think the paper would benefit from discussing the many ways in which existing fine-tuning methods are suboptimal (diversity, out of distribution, etc), so as to provide a more nuanced context to the reader.\n\n2- The paper does not sufficiently discuss the importance of its assumption, that is, what happens if each one breaks? Which ones are more crucial than others? Which ones are less realistic than others?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions. We provide our response to each question below.\n\n**W1: The discussion on the existing fine-tuning method.**\n\nIn this part, we discuss the fully fine-tuned method and explain why we need data-efficient fine-tuning methods.\n\nIn earlier times, fully fine-tuned methods, such as DreamBooth [1], provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters (Fig. 6 of their paper). To deal with this problem, DreamBooth designs a prior preservation loss as a regularizer, which is needed carefully designed. Furthermore, a fully fine-tuned method is both memory and time inefficient [2].\n\nTo avoid the above problem, many works fine-tune some key parameters and achieve success in many areas such as text2image and medical image generation [3] [4]. These works not only preserve the prior information but also have a significantly smaller model size, which is more practical for real-world applications. Hence, we focus on these models in our work.\n\nWe will add the above discussion to our introduction part.\n\n**W2: The discussion on each assumption.**\n\nWe note that Assumption 3.1 is the most important assumption for our analysis, and we discuss this assumption in detail below. The other assumptions are standard, and we discuss them one by one.\n\n(a) The linear subspace and shared latent assumption. (Assumption 3.1).\n\nSince the diffusion models can find and adapt to the low-dimensional manifold [5], many theoretical works assume the linear low-dimensional manifold as the first step to understand this phenomenon [6] [7] and we do exactly the same assumption compared with them. We note that since our analysis depends on the formula of score function under the linear subspace assumption, this assumption is important for our paper. An interesting future works is to extend our analysis to the nonlinear low-dimensional manifold and we will discuss it in our future work part.\n\nFor the analysis of few-shot learning, it is a standard, natural and necessary assumption to assume the shared latent representation [8] [9].\n\n(b) The subgaussian latent variable and $\\beta$-Lipschitz score function (Assumption 4.1 and 4.2).\n\nSince a bounded variable is a subgaussian one, this assumption is naturally satisfied by the real-world image datasets and is widely used in many theoretical works [6] [7].  For the Lipschitz score, this assumption is common for the theoretical works of diffusion models [6] [10]. \n\n(c) The isotropic Gaussian latent variable assumption (Sec. 5).\n\nWhen considering the few-shot optimization problem, we assume the latent variable is isotropic Gaussian, which is stronger than subgaussian one in Assumption 4.1. We note that this assumption is necessary for the closed form solution in Lem. 5.1. However, as discussed in future work paragraph, it is not necessary if we use some optimization algorithm instead of directly obtaining a closed-form.\n\nThanks for the concerns on our assumption. We will discuss our assumption in detail in the next version.\n\n\n\n[1] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.\n\n[2] Xiang, C., Bao, F., Li, C., Su, H., & Zhu, J. (2023). A closer look at parameter-efficient tuning in diffusion models. *arXiv preprint arXiv:2303.18181*.\n\n[3] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. ICCV 2023.\n\n[4] Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. *arXiv preprint arXiv:2305.08252*.\n\n[5] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In *International Conference on Artificial Intelligence and Statistics* (pp. 1648-1656). PMLR.\n\n[6] Chen, M., Huang, K., Zhao, T., & Wang, M. (2023, July). Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In *International Conference on Machine Learning* (pp. 4672-4712). PMLR.\n\n[7] Yuan, H., Huang, K., Ni, C., Chen, M., & Wang, M. (2024). Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. *Advances in Neural Information Processing Systems*, *36*.\n\n[8] Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., & Lei, Q. (2020). Few-shot learning via learning the representation, provably. *arXiv preprint arXiv:2002.09434*.\n\n[9] Chua, K., Lei, Q., & Lee, J. D. (2021). How fine-tuning allows for effective meta-learning. *Advances in Neural Information Processing Systems*, *34*, 8871-8884.\n\n[10] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. *arXiv preprint arXiv:2209.11215*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fanuvNPIvE", "reviewer": "Reviewer_7GpV", "summary": "The paper provides a theoretical analysis showing that fine-tuning pretrained diffusion models on a few target samples achieves a small approximation error, particularly in that it requires much fewer target data than source data. The authors also show that the solution has a closed form in a special case, demonstrating the ease of optimization for few-shot models. They provide bounds for this closed-form solution, which also shows better dependency on the target sample size than on the source sample size. Experiments on CelebA are conducted to further support the theoretical conclusions.", "questions": "I wonder if there is any intuitive explanation for assumptions like 4.1 and 4.2, particularly what they could mean in practice. Or are they just technical assumptions without much intuition?", "limitations": "The authors have discussed certain limitations in Section 7.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work offers a theoretical understanding of why few-shot diffusion models can achieve strong performance despite the curse of dimensionality, which is a significant and intriguing topic.\n2. The conclusions are well-supported by theoretical results and are clearly explained. Section 4 seems solid and well-structured, although I have not examined the proof in the appendix. Additionally, in Section 5, they demonstrate the closed-form expression of the minimizer under an isotropic Gaussian latent distribution and its ability to recover the true subspace, which further adds comprehensiveness and aids illustration.", "weaknesses": "I don't have any particular criticisms; however, I am not an expert in this topic, particularly regarding the theoretical analysis of diffusion models. I would like to hear other reviewers' opinions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions on our assumptions. We discuss each assumption in detail below and will make it clearer in our next version.\n\n**Q1: The discussion on each assumption.**\n\n(a) The linear subspace and shared latent assumption. (Assumption 3.1).\n\nSince the diffusion models can find and adapt to the low-dimensional manifold [1], many theoretical works assume the linear low-dimensional manifold as the first step to understanding this phenomenon [2] [3], and we make exactly the same assumption compared with them. \n\nFor the analysis of few-shot learning, it is a standard, natural, and necessary assumption to assume the shared latent representation [4].\n\n(a) The subgaussian latent variable (Assumption 4.1).\n\nIn this assumption, we assume the latent variable is a subgaussian variable. Since a bounded variable is a subgaussian one, this assumption is naturally satisfied by the real-world image datasets. We note that this assumption is more realistic than the Gaussian assumption and is widely used in many theoretical works of diffusion models [1] [2].\n\n(b) The $\\beta$-Lipschitz assumption on the score function (Assumption 4.2).\n\nWe note that the $\\beta$-Lipschitz assumption is a standard assumption on the score function for the theoretical works [2] [5]. As shown in Sec. 3.2 of [5], for a subgaussian variable (Assumption 4.1), we can replace $\\beta$ with  $C_Z/\\delta^2$ and remove this assumption. We will add the above discussion to our paper.\n\n\n\n[1] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In *International Conference on Artificial Intelligence and Statistics* (pp. 1648-1656). PMLR.\n\n[2] Chen, M., Huang, K., Zhao, T., & Wang, M. (2023, July). Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In *International Conference on Machine Learning* (pp. 4672-4712). PMLR.\n\n[3] Yuan, H., Huang, K., Ni, C., Chen, M., & Wang, M. (2024). Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. *Advances in Neural Information Processing Systems*, *36*.\n\n[4] Chua, K., Lei, Q., & Lee, J. D. (2021). How fine-tuning allows for effective meta-learning. *Advances in Neural Information Processing Systems*, *34*, 8871-8884.\n\n[5] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. *arXiv preprint arXiv:2209.11215*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "59M6Rim9dL", "reviewer": "Reviewer_fbjU", "summary": "The paper provides a theoretical analysis of the few-shot fine-tuning problem for diffusion models. It makes the following key assumptions: (1) the pretraining and fine-tuning data distributions share a common latent distribution, and (2) a specific network architecture. Under these assumptions, the paper proves that during the fine-tuning phase, the diffusion model avoids the curse of latent dimensionality and achieves an approximation bound of $\\widetilde{\\mathcal{O}}(n_s^{-2/d}+n_{ta}^{-1/2})$. Furthermore, when considering a low-rank Gaussian distribution, the model attains an accuracy bound of $\\widetilde{\\mathcal{O}}(1/n_{ta}+1/\\sqrt{n_{s}})$. The paper also includes real-world experiments to support the theoretical findings.", "questions": "In section 4.1, Table 1, the requirement derived from $n_{ta}^{\\frac{d+5}{4(1 - \\alpha(n_s))}} \\geq n_s$ is counterintuitive. For datasets like ImageNet with higher latent dimensions, the requirement of $n_{ta}$ is less. Could the author provide more discussions or give some real-world experiments to support this?", "limitations": "The paper has covered the limitations and societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The theoretical part of the paper is solid. As an extension of [1], the paper studies the diffusion model in the few-shot fine-tuning problem. Both the assumption of the data distribution and the definition of fine-tuning are reasonable to me. The conclusion aligns with the practice, that diffusion model fine-tuning escapes the curse of dimensionality and requires much fewer data samples to converge to the underlying distribution. I haven't gone through the details of the proof, if there is no further problem for them, I believe this is a theoretical solid paper.\n\n[1] Chen, M., Huang, K., Zhao, T., and Wang, M. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.07194, 2023c.", "weaknesses": "There is a gap between the real-world experiments and the theoretical results. The experiments demonstrate two main points: (1) fine-tuning the entire diffusion model leads to memorization, and (2) fine-tuning only the encoder and decoder parts of the diffusion model will generalize. However, these findings only support the validity of the fine-tuning definition used in the paper.\nThe gap lies in two areas:\n1. The paper lacks proof that fine-tuning the full model results in memorization.\n2. The real-world experiments do not show that few-shot fine-tuning can escape the curse of dimensionality. Specifically, the diffusion model still requires a similar number of samples for fine-tuning datasets with different intrinsic dimensions.\n\nI think the paper lacks sufficient real-world experiments to fully support its title, \"Few-Shot Diffusion Models Escape the Curse of Dimensionality.\" With such an experiment, I believe the paper will be a strong submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions. We provide our response to each question below.\n\n**W1: The theoretical guarantee of the fully fine-tuned method.**\n\nAs shown in our real experiment and [1], when fine-tuning all parameters with a small target dataset, models tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use $\\hat{\\theta}$ learned by the pre-trained model and achieves a $n_{ta}^{-2/d}$ error bound, which **suffers from the curse of dimensionality**. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term. We will add this proof and discussion in the next version.\n\nWe also note that it is possible to avoid this phenomenon by using a specific loss [1] or carefully choosing the optimization epochs [2]. We leave them as interesting future works.\n\n**W2: The discussion on the curse of the dimensionality.**\n\nWith a very limited target dataset (such as 5-10 images), if a method can fine-tune the pre-trained model (trained with a large source dataset) and generate novel images with the target feature, we say that this models escape the curse of dimensionality. As shown in W1, the fully fine-tuned method has a large approximation error (theoretical guarantee) and suffers the memory phenomenon (experiments). On the contrary, our method can escape the curse of dimensionality from the theoretical (Thm. 4.3) and empirical perspective.\n\nFrom the empirical perspective, as shown in Appendix E (dataset part), our experiments use **$6400$** source data (CelebA64) to train a pre-trained model. Then, we only use **$10$** target images to fine-tune this model and generate novel images with the target feature, which indicates our few-shot models escape the curse of dimensionality. For the cat dataset, we have a similar augmentation (4200 source data and $10$ target data). We note that $10$ target images are very limited compared to the source dataset (including CelebA64 and Cat dataset). Hence, our experiments support the theoretical results under each dataset.\n\nWe will add the above discussion and description of the source and target dataset in Sec. 6 to avoid confusion. Thanks again for the concerns and valuable comments on our real-world experiments. \n\n**Q1: A clearer discussion of the results of Thm. 4.3 and Table 1.**\n\nWe note that **the goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models**, which means that we consider the relative relationship between $n_{ta}$ and $n_s$. Hence, if the coefficient of $n_{ta}$ and $n_s$ has the same order, we can only consider $1/\\sqrt{n_{ta}}$ and $n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}$. To support the above augmentation, we first recall the results and calculate the coefficients:\n$$\n\\begin{align}\n\\left(\\frac{(1+\\beta)^2 Dd^3}{\\delta\\left(T-\\delta\\right)\\sqrt{n_{ta}}}+\\frac{\\left(d+C_Z\\right) d^2 \\beta^2}{\\delta^2 c_0}n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}\\right)\\log\\left(\\frac{1}{\\delta_1}\\right).\n\\end{align}\n$$\nThe dominated term of coefficient for $n_{ta}$ and $n_s$ are $Dd^3/\\delta$ and $\\frac{d^3}{\\delta^2c_0}$, respectively. The classic choice for the early stopping parameter $\\delta$ and forward time $T$ are $10^{-3}$ and $10$, respectively [3]. Then, with $D=256\\*256\\*3$ as an example (Since smaller $D$ is more friendly to $n_{ta}$, our discussion holds for all datasets in Table 1.),  $Dd^3/\\delta = d^3\\*20\\*10^6$ and $\\frac{d^3}{\\delta^2c_0} = d^3\\*10^6/c_0$, which has the same order. Hence, we consider the relative relationship between $1/\\sqrt{n_{ta}}$ and $n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}}$ and require\n$$\n\\frac{1}{\\sqrt{n_{ta}}}\\leq n_s^{-\\frac{2-2\\alpha\\left(n_s\\right)}{d+5}},\n$$\nwhich indicates $n_{ta}^{\\frac{d+5}{4(1-\\alpha(n_s))}}\\ge n_s$.\n\nWe now discuss the reason why the requirement on $n_{ta}$ is smaller when $d$ is larger. We note that the error bound of the pre-trained model is heavily influenced by the latent dimension $d$. More specifically, when $d$ is large (e.g. ImageNet), the error bound of pre-trained models has a large error even with large-size source data. We only need a few target data to achieve the same error. When $d$ is small (e.g. CIFAR-10), pre-trained models have a small error, and we need a slightly large target data size to achieve a small error.\n\nWe will add the above discussion in our next version to make it clearer.\n\n[1] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.\n\n[2] Li, P., Li, Z., Zhang, H., & Bian, J. (2024). On the generalization properties of diffusion models. *Advances in Neural Information Processing Systems*, *36*.\n\n[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. *Advances in neural information processing systems*, *35*, 26565-26577.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper presents a strong and well-founded theoretical framework for understanding few-shot diffusion models. As highlighted by Reviewer kXcP, the theory is both novel and interesting, offering meaningful insights into the training of few-shot diffusion models. Furthermore, the theoretical contributions are robustly supported by comprehensive experimental results, enhancing the overall impact and validity of the work.
Decision: Accept (poster)