Few-Shot Diffusion Models Escape the Curse of Dimensionality

Authors: ruofeng  yang
bo  jiang
cheng  chen
ruinan  jin
baoxiang  wang
shuai  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:31:42

Abstract: While diffusion models have demonstrated impressive performance, there is a growing need for generating samples tailored to specific user-defined concepts. The customized requirements promote the development of few-shot diffusion models, which use limited $n_{ta}$ target samples to fine-tune a pre-trained diffusion model trained on $n_s$ source samples. Despite the empirical success, no theoretical work specifically analyzes few-shot diffusion models. Moreover, the existing results for diffusion models without a fine-tuning phase can not explain why few-shot models generate great samples due to the curse of dimensionality. In this work, we analyze few-shot diffusion models under a linear structure distribution with a latent dimension $d$. From the approximation perspective, we prove that few-shot models have a $\widetilde{O}(n_s^{-2/d}+n_{ta}^{-1/2})$ bound to approximate the target score function, which is better than $n_{ta}^{-2/d}$ results. From the optimization perspective, we consider a latent Gaussian special case and prove that the optimization problem has a closed-form minimizer. This means few-shot models can directly obtain an approximated minimizer without a complex optimization process. Furthermore, we also provide the accuracy bound $\widetilde{O}(1/n_{ta}+1/\sqrt{n_s})$ for the empirical solution, which still has better dependence on $n_{ta}$ compared to $n_s$. The results of the real-world experiments also show that the models obtained by only fine-tuning the encoder and decoder specific to the target distribution can produce novel images with the target feature, which supports our theoretical results.

Main paper: 
 introduction: 
 In recent years, diffusion models have shown an excellent ability to generate diverse, high-quality images and show state-of-the-art performance in the large-scale, standard dataset (Rombach et al., 2022; Ho et al., 2022; Li et al., 2024) . However, users often desire to generate samples that resemble the ones they provide, such as images related to their families, daily lives, or specific items. These user-provided samples are typically limited in number and do not appear frequently in large-scale datasets. Consequently, training a diffusion model from scratch using such limited, personalized samples often results in poor performance. To cater the customized requirements of users, few-shot diffusion models attract much attention. Few-shot diffusion models aim to fine-tune a pre-trained diffusion model using a limited amount of data (5 ∼ 10 samples), and they have recently delivered impressive results in various domains, including image generation (Ruiz et al., 2023; Han et al., 2023; Zhu et al., 2023) , video generation (Chen et al., 2023b) , and the medical domain (Dutt et al., 2023) .Before the fine-tuning phase, we first need to train a diffusion model on the large source dataset {X s,i } ns i=1 as the pre-trained model. A diffusion model consists of a forward process and a reverse process (Song et al., 2020) . The forward process gradually converts the data distribution into Gaussian noise. The reverse process sequentially removes the noise in the data to generate samples, which relies on the gradient of logarithmic forward process density (a.k.a. score function). To run the reverse process, diffusion models use a neural network to approximate the unknown score function.With a pre-trained diffusion model, the paradigm to obtain a few-shot diffusion model is to fine-tune the model using a limited target dataset {X ta,i } nta i=1 . In earlier times, fully fine-tuned methods, such as DreamBooth (Ruiz et al., 2023) , provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters. Furthermore, a fully fine-tuned method is both memory and time inefficient (Xiang et al., 2023) . To avoid the above problems, many works freeze most parameters and fine-tune some key parameters, such as cross-attention layers (Kumari et al., 2023; Moon et al., 2022) , some concept neurons (Liu et al., 2023) or text-embedding (Gal et al., 2022) , to approximate the ground-truth target score function. These works not only preserve the prior information but also have a lower requirement for the target dataset size, which is more practical for applications. Hence, we aim to explain the great performance of these models in this work.Despite the empirical success, no existing theoretical work specifically analyzes the approximation bound for few-shot diffusion models, and the following question remains open: Do few-shot diffusion models with a fine-tuning phase enjoy a small approximation error with a limited target dataset?For the approximation error bound, some works currently analyze diffusion models without a finetuning phase (Oko et al., 2023; Chen et al., 2023c; Yuan et al., 2023; Li et al., 2023b) . Importantly, when analyzing general, bounded data, these works suffer from the curse of dimensionality. More specifically, Oko et al. (2023) analyze bounded distribution and show the n -s ′ /D s approximation bound, where D is the data dimension of X s . Chen et al. (2023c) analyze linear structure distribution X s = A s Z with subgaussian latent variable Z ∈ R d and achieve n -2/d s results. Since the source dataset size is large enough, the influence of dimension is tolerable. However, for the limited target dataset, if trivially using the above technique, the bound is n -1/D ta or n -2/d ta , which is large and can not explain why few-shot diffusion models efficiently approximate the target score function.In this work, for the first time, we propose the approximation bound specifically to few-shot diffusion models with a fine-tuning phase and prove that the few-shot diffusion model can escape the curse of dimensionality. More specifically, we show that when assuming (1) linear structure data and (2) the source and the target data share latent distribution, the few-shot diffusion models with a fine-tuning phase achieve O(n-2/d s + n -1/2 ta) approximation error bound, which makes the first step to explain why few-shot diffusion models have great performance in the application. Generally speaking, due to the component n -1/2 ta , the few-shot diffusion only needs a few target samples to achieve the same bound compared to n -2/d s . To support our augmentation, we calculate the requirement of n ta to obtain an accurate enough approximated target score function in popular datasets. Table 1 shows that the requirement of n ta is about 5 ∼ 10 samples, matching the customized diffusion model requirement. We also do experiments on the real-world datasets and show that 10 target images are enough for few-shot models to generate novel images with the target feature (Section 6).After directly using the property of the minimizer to obtain an approximation bound, we analyze how to optimize the few-shot diffusion models to obtain a minimizer. Since the score-matching objective function is highly non-convex, only a few works analyze the optimization problem of diffusion models (Shah et al., 2023; Bruno et al., 2023; Cui et al., 2023; Li et al., 2023b) . Furthermore, these works either require (1) an exponential size neural network (Li et al., 2023b) or (2) a distribution determined by one variable (Shah et al., 2023; Bruno et al., 2023; Cui et al., 2023) to simplify the optimization problem. This work proves that few-shot diffusion models can simplify the optimization problem without these requirements. When analyzing the optimization problem, we focus on a Gaussian latent variable special case 2 . Then, we prove that the expected few-shot objective function has a closed-form minimizer, which means the empirical solution can be directly obtained without a complex optimization process. We also prove the accuracy bound O(1/n ta + 1/ √ n s ) of empirical 2 Though it is a special case, the previous analysis can not be used since it is determined by two components.closed-form solution, which still has better dependence on the target dataset. In conclusion, we accomplish the following results for few-shot diffusion models under linear structure distribution:• For the approximation bound, we consider a subgaussian latent variable and prove O(n-2/d s + n -1/2 ta) bound for few-shot models, which is better than n -2/d ta result without fine-tuning.• For the optimization problem, we consider a latent Gaussian special case and prove that the expected few-shot objective function has a closed-form minimizer. Furthermore, we prove the accuracy bound O(1/n ta + 1/ √ n s ) for the empirical closed-form solution.• To support our theoretical results, we do real-world experiments and show that the models obtained by only fine-tuning specific encoder and decoder can use only 10 target images to generate novel images with the target feature.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: This work aims to explain the reason why few-shot diffusion models can achieve great performance with a limited dataset from the theoretical perspective. We achieve this goal by analyzing the approximation and optimization perspective (Theorem 4.3 and Theorem 5.2). We also do real-world experiments to support our theoretical results (Section 6).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 The approximation error bound. Recently, some works analyze the approximation error bound of diffusion models without a fine-tuning phase. The approximation error bound of the above works suffers the curse of (latent) dimensionality. To avoid this phenomenon, some works analyze special data distributions. Shah et al. (2023) and Cui et al. (2023) analyze the mixture of Gaussian (MOG) with known variance and achieve a 1/n s approximation bound. Yuan et al. (2023) analyze linear structure distribution with Gaussian latent variable and achieve 1/ √ n s result. Mei & Wu (2023) analyze Ising models and prove that the term corresponds to n s is 1/ √ n s . However, the remaining terms do not converge to 0 when n s goes to +∞. For general bounded data distribution, Li et al. (2023b) provide a n -2/5 s approximation error bound. However, they use a 2-layer random feature network and only allow the second linear layer to be trainable. Hence, the network size is exp (n s ) compared to Poly(n s ) size of all previous works.The optimization of diffusion models. Since the score matching objective function is highly non-convex, only a few works analyze how to optimize it to obtain a minimizer (Shah et al., 2023; Cui et al., 2023; Bruno et al., 2023; Li et al., 2023b) . These works either make assumptions about data distribution or network size to guarantee only one optimization variable, leading to a simpler optimization problem. For special data distributions, Bruno et al. (2023) and Cui et al. (2023) analyze a Gaussian with fixed variance and a 2-mode mixture of Gaussian (MOG) with equal, trainable mean and fixed variance, respectively. Shah et al. (2023) analyze a multi-mode MOG with a fixed variance and prove a local convergence guarantee. Since they assume the distance between any two modes is large enough and a good enough initialization, the optimization problem is similar to optimizing a Gaussian distribution. For the large neural network size, Li et al. (2023b) analyze a general, bounded distribution with a 2-layer NN. Note that they require exp (n s ) hidden neurons and only allow the linear layer to be trainable, which also leads the optimization problem to a convex optimization.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss future work and limitation at Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: We have checked the code of ethics and make sure that our work satisfies the code of ethics. 
the introduction of few-shot diffusion models: 
 With pre-trained models, the paradigm to obtain a few-shot diffusion model is to freeze most parameters and fine-tune some key parameters corresponding to the target data distribution. Since the analysis of few-shot diffusion models relies heavily on the pre-trained model, this section first provides a concise overview of the fundamental concepts and notations associated with diffusion models. Then, we introduce the paradigm of few-shot diffusion models in Section 3.2.Let q 0 be the data distribution. Given X 0 ∼ q 0 ∈ R D , non-decreasing function f (X t , t) and g(t), the forward process is defined by: dXt = f (X t , t)dt + g(t)dB t ,where {B t } t∈[0,T ] is a D-dimensional Brownian motion. In this work, we choose f (X t , t) = -1/2X t and g(t) = 1, which corresponds to variance preserving (VP) forward process and is widely used in practice3 (Shah et al., 2023; Song et al., 2020) . Let q t be the density function of X t . Once a forward process is chosen, the conditional distribution ofX t |X 0 is q t (X t |X 0 ) = N (m t X 0 , σ 2 t I D ), where m t = e -t/2 , σ 2 t = 1 -e -t . Note that when t goes to +∞, q t converges to N (0, I D ), which is helpful in choosing the initial distribution for the sampling process.To generate samples, diffusion models first reverse the forward SDE to obtain the reverse process. Then, we approximate the ground-truth score function ∇ log q t (•) by using a neural network s(•, t) due to the requirement of the reverse process (see Section 3.2). Finally, diffusion models choose a discretization scheme to discretize the continuous reverse process to obtain an implementable algorithm. Let t 0 ≤ t 1 ≤ • • • ≤ t K = T be the discretization points in the forward time and h k = t k -t k-1 be the k-th stepsize. When considering the reverse time, we define t ′ k = T -t K-k . In this work, we choose the exponential integrator (EI) discretization scheme, which has great performance (Zhang & Chen, 2022) . The EI discretization freezes the approximated score at t ′ k and runs the following process in the reverse time:d Y t = f ( Y t , T -t) + g(T -t) 2 s( Y t ′ k , T -t ′ k ) dt + g(T -t)dB t , t ∈ [t ′ k , t ′ k+1 ] ,where Y 0 ∼ N (0, I D ) due to the stationary distribution of the forward process.While the discretization complexity K has been well-studied with an accurate enough score function (Benton et al., 2023; Li et al., 2023a) , there is a lack of analysis for the score-matching process.Therefore, this work focuses on the score approximation and the optimization problem of the few-shot score-matching objective function.In this work, we specifically analyze few-shot diffusion models, which involve two datasets: (1) the source dataset {X s,i } ns i=1 ;(2) the target dataset {X ta,i } nta i=1 . The approach involves first training a pre-trained diffusion model on the source dataset and then freezing the backbone network to fine-tune the diffusion models on the target dataset.For data distributions, we assume that the source distribution q s and the target distribution q ta are both supported on a low-dimensional linear subspace. The low-dimensional structures have been discovered on many popular image datasets (Pope et al., 2021; Gong et al., 2019; Tenenbaum et al., 2000) due to the locally connected and symmetrical property, and it is crucial for diffusion models. For image generation, current popular diffusion models, such as Stable Diffusion (Rombach et al., 2022) , transform images to a latent space and run diffusion models in the latent space. Except for the image generation, Chen et al. (2024) recently show the latent dimension plays an important role in diffusion models to work well in self-supervised learning, and linear subspace is enough.We further assume that the source and target data share the same latent distribution. Note that this is a common assumption in few-shot learning. In particular, previous theoretical works in the context of supervised few-shot learning often assume that the source and target distributions have a common latent representation (Du et al., 2020; Chua et al., 2021; Meunier et al., 2023) . Assumption 3.1. The source datapoints X s and target datapoint X ta admit a low dimensional linear structure and share the same latent distribution X s = A s Z and X ta = A ta Z where A s , A ta ∈ R D×d with orthonormal columns and Z ∼ q z ∈ R d .As mentioned in Chen et al. (2023c) , when assuming linear distribution, the ground-truth score function is decomposed into the latent score function ∇ log q LD t (Z ′ ) and linear encoder and decoder:∇ log q s t (X) = A s ∇ log q LD t A ⊤ s X - 1 σ 2 t I D -A s A ⊤ s X ,where q LD t (Z ′ ) = q t (Z ′ |Z) q z (Z)dZ and q t (•|Z) = N (m t Z, σ 2 t I d ). This form indicates that the diffusion process happens in the latent subspace. A conceptual way to approximate the score function is to minimize the following loss on a function class S N N :min s∈S N N T 0 w(t)E Xt∼q s t ∥∇ log q s t (X t ) -s (X t , t)∥ 2 2 dt ,where w(t) is a weight function. However, the above objective function is intractable since ∇ log q t (•) is unknown. Vincent (2011) propose the following implementable loss:L s (s) = T 0 w(t)E X0 E Xt|X0 ∥∇ log q s t (X t |X 0 ) -s (X t , t)∥ 2 2 dt .Due to the forward process, ∇ log q s t (X t |X 0 ) has an analytical form and is equal to -(X tm t X 0 )/σ 2 t . Vincent (2011) also prove that this objective function only has a constant difference compared to the above one. The empirical loss with the source datasets {X s,i } ns i=1 is defined by:EQUATIONwhereℓ s t (X s,i ; s) = E Xt|X0=Xs,i ∥∇ log q s t (X t |X 0 ) -s (X t , t)∥ 2 2 ,andS NN = {s V,θ (X, t) = 1 σ 2 t V f θ V ⊤ X, t - 1 σ 2 t X :V ∈ R D×d with orthonormal columns, f θ : R d × [δ, T ] → R d a ReLU network } .Note that we take w(t) = 1/(T -δ) for simplicity, where δ is the early stopping parameter to avoid the blow-up phenomenon of score functions at the end of reverse process. Furthermore, we take the integral over the forward time instead of discretizing the timeline since X t is easy to generate.The linear encoder and decoder structure and the shortcut connection in S N N is due to the form of the ground-truth score function. The specific parameters for f θ , such as its length and width, are identical to those used in Chen et al. (2023c) . Generally, with a given network accuracy parameters ϵ, the network size is Poly(1/ϵ). We show the parameter of neural network in Appendix A.The diffusion models minimize the empirical loss to obtain a pre-trained approximated score function.Let the minimizer of Equation ( 1) be ( V s , θ). Chen et al. (2023c) show that ( V s , θ) leads a n -2/d s approximation error bound. If trivially replacing n s with n ta , we obtain a n -2/d ta bound for the target dataset without the fine-tuning phase. Note that this bound suffers from the influence of the latent dimension d, which is still large in popular datasets (Table 1 ). Hence, this results in a large approximation error bound. In the next paragraph, we introduce the few-shot diffusion models with a fine-tuning phase and show that the dependence on n ta is n -1/2 ta in the error bound (Theorem 4.3).The Few shot Diffusion Models with a Fine-tuning Phase. Since the source and target distribution share the same latent data distribution, we freeze θ and only fine-tune the low-rank linear encoder and decoder layer V ta . This method can significantly reduce the fine-tuning parameters and is similar to LoRA (Hu et al., 2021) , which also fine-tunes two low-rank matrices and is widely used in fine-tuning the stable diffusion (Rombach et al., 2022) .Let ℓ ta t be the loss function of the target dataset at time t, which has similar definition compared to ℓ s t . The optimization problem for the target dataset ismin s V ta , θ ∈Q N N ( θ) L ta (s Vta, θ ) = 1 n ta (T -δ) nta i=1 T δ ℓ ta t X ta,i ; s Vta, θ dt , where Q NN (θ) = {s V,θ (X, t) = 1 σ 2 t V f θ V ⊤ X, t - 1 σ 2 t X : V ∈ R D×d with orthonormal columns.} ,Similarly, we define the minimizer of the few-shot objective function as ( V ta , θ).Notations. We denote by I D the D-dimensional identity matrix. For X ∈ R D and A ∈ R D×d , we denote by ∥X∥ 2 the Euclidean norm for vector and ∥A∥ F the Frobenius norm for matrix. We denote by ∥X∥ 2 L 2 (q) the expectation of In this section, we show that few-shot diffusion models with a fine-tuning phase escape the curse of latent dimensionality and have a O(nX in L 2 norm E X∼q [∥X∥ 2 2 ].-2/d s + n -1/2 ta) approximation bound4 . This result makes the first step to explain why few-shot models have great performance with a limited target dataset.Before showing our results, we first introduce standard assumptions on the latent distribution and the on-support ground-truth score function. We first assume that Z has a subgaussian tail and the minimum eigenvalue of Z is lower bound by c 0 , also used in Chen et al. (2023c) . Assumption 4.1. q z > 0 is twice continuously differentiable, λ min (E ZZ ⊤ ) ≥ c 0 and E∥Z∥ 2 2 ≤ C Z . Moreover, there exist positive constants B, C 1 , C 2 such that when∥Z∥ 2 ≥ B, q z (Z) ≤ (2π) -d/2 C 1 exp -C 2 ∥Z∥ 2 2 /2 . Assumption 4.2. The on-support ground truth score A s ∇ log q LD t (Z) and A ta ∇ log q LD t (Z) is β-Lipschitz in Z ∈ R d for any t ∈ [0, T ].Note that different from previous works directly assume ∇ log q t (•) is Lipschitz (Chen et al., 2022 (Chen et al., , 2023d)) , the β-Lipschitz on-support score function assumption does not conflict with the blow-up phenomenon when t goes to 0 due to the existence of (I D -AA ⊤ )X/σ 2 t . With these assumptions, we prove the approximation bound for few-shot models with a fine-tuning phase.Theorem 4.3. Let α(n) = d log log n log n , F = (d+C Z )d 2 β 2 δ 2 c0and network parameter ϵ = n ≥ n s . Then, with probability 1 -δ 1 , the following inequality holds (hiding logarithmic factors)1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (q ta t ) dt ≤ Õ (1 + β) 2 Dd 3 δ (T -δ) √ n ta + F n - 2-2α(ns) d+5 s log 1 δ 1 .The dependence of δ is due to the blow-up property of the score function. Note that when n s is sufficiently large, α(n s ) is negligible. Then, the approximation error bound for few-shot diffusion models is Õ(n-2/d s + n -1/2 ta). Compared to the approximation error bound n -2/d ta without a few-shot phase, it is clear that few-shot diffusion models escape the curse of the (latent) dimensionality. Remark 4.4 (The discussion on the coefficient in Theorem 4.3). The goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models, which means that we consider the relative relationship between n ta and n s . Hence, if the coefficient of n ta and n s has the same order, we can only consider 1/√ n ta and n -2/d s . To support the above augmentation, we calculate the coefficient of n s and n ta in detail. The dominated term of coefficient for n ta and n s are Dd 3 /δ and d 3 /(δ 2 c 0 ), respectively. The classic choice for the early stopping parameter δ and forward time T are 10 -3 and 10, respectively (Karras et al., 2022) . Then, with D = 256 × 256 × 3 as an example5 , Dd 3 /δ = d 3 × 20 × 10 6 and d 3 /(δ 2 c 0 ) = d 3 × 10 6 /c 0 , which has the same order. Hence, we consider the relative relationship between 1/√ n ta and n -2/d s .Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
discussion on the approximation bound: 
 The relationship to empirical phenomenon. In applications, current few-shot diffusion models only require 5 ∼ 10 target images to achieve great performance. Theorem 4.3 makes the first step to explain why the few-shot diffusion models have great performance with a limited target n ta . More specifically, with known source dataset size n s and the corresponding latent dimension d, we can calculate the inequality n d+5 4(1-α(ns)) ta ≥ n s6 to obtain the requirement of n ta to achieve the same accuracy compared to the pre-trained diffusion models. Combined with the latent dimension of popular datasets (Pope et al., 2021) , Table 1 shows the requirement of n ta . It is clear that we only need less than 10 target images to obtain an accurate enough few-shot diffusion model that matches the performance in reality. The real-world experiments also support our discussion (Section 6).Table 1 shows that the requirement of n ta is heavily influenced by the latent dimension d. When d is large (e.g. ImageNet), the approximation bound of pre-trained models is influenced by latent dimension and has a large approximation error even with large-size source data. We only need a few target data to achieve the same error in this setting. When d is small (e.g. CIFAR-10), pre-trained models have a small approximation error. We need a slightly larger target data size.The approximation error of the fully fine-tuned method. As shown in our real experiment Section 6 and DreamBooth (Ruiz et al., 2023) , when fine-tuning all parameters with a small target dataset, models tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use θ learned by the pre-trained model and achieves a n -2/d ta approximation error bound, which suffers from the curse of dimensionality. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term. We also note that it is possible to avoid this phenomenon by using a specific loss (Ruiz et al., 2023) or carefully choosing the optimization epochs (Li et al., 2023b) . We leave them as interesting future works.Proof sketch. The first step is to prove that in Q N N ( θ), there exists a solution ( Vta , θ) has the following inequality (only focusing on n s and n ta )1 T -δ T δ s Vta, θ (X, t) -∇ log q ta t (X) 2 L 2 (q ta t ) dt ≤ O ϵ 2 + n - 2-2α(ns) d+5 s log 1 δ 1 .To prove the above inequality, we first do the following decomposition:s Vta, θ (•, t) -∇ log q ta t (•) 2 2 + s Vta, θ (•, t) -s Vta, θ (•, t) 2 2, where ( Vta , θ) ∈ S N N is a constructed solution. The first term is due to the accuracy of the constructive neural network with network accuracy parameter ϵ. For the second term, since the latent score function is shared and few-shot diffusion models directly use θ, it is control by the approximation bound of the pre-trained diffusion models. Then, by using the inequalityinf s V ta , θ ∈Q( θ) L ta s Vta, θ ≤ L ta s Vta, θ ,we build the bridge between s Vta, θ and s Vta, θ .The second step is using the concentration to control the error between empirical L ta and expected L ta . Roughly speaking, we have thatL ta s Vta, θ -L ta s Vta, θ ≤ 1 n ta ϵ 2 log N 1/n ta , Q NN ( θ), ∥ • ∥ 2 /δ 1 , where N (1/n ta , Q N N ( θ), ∥ • ∥ 2 ) is the covering number of Q N N ( θ) in L 2 norm. Since only V ∈ R D×d can be optimized and θ is fixed in Q N N ( θ), log N 1/n ta , Q NN ( θ), ∥ • ∥ 2 = O(Dd log(1/n ta )) .Then, we balance different terms and achieve the final bound by choosing ϵ 2 = 1/ √ n ta . 
the few-shot diffusion model have a closed-form minimizer: 
 This section focuses on how to optimize the few-shot diffusion model. When considering the optimization problem, we assume the shared latent distribution admits an isotropic Gaussian distribution q z = N (0, λ 2 I d ) with λ 2 > 0, which indicates the score function has the following formulation:∇ log q ta t (X) = - 1 λ 2 A ta A ⊤ ta X - 1 σ 2 t I D -A ta A ⊤ ta X .Note that though q z = N (0, λ 2 I d ) is a special case of Assumption 4.1, we still need to know λ 2 and A ta to generate samples, which indicates the previous optimization analysis for diffusion models without a fine-tuning phase can not be used.We fix a t ∈ [δ, T ] for the few-shot objective function since the matrix A ta is independent of time t. More specifically, with an approximated latent distribution q z = N (0, Σ), where Σ = λ 2 I d , the expected few-shot objective function at a fixed time t ismin s V ta , Σ ∈ QNN ( Σ) L ta,t (s Vta, Σ ) = E Xta∼q ta ℓ ta t X ta ; s Vta, Σ .whereQNN (Σ) = {s V,θ (X, t) = 1 σ 2 t V f Σ V ⊤ X, t - 1 σ 2 t X : V ∈ R D×d with rank(V ) = d.} , In this case, f Σ (Z, t) = (I d -σ 2 t Σ -1 t )Z, where Σ t = m 2 t Σ + σ 2 t I d .The constraint rank(V ) = d is used to guarantee that the few-shot diffusion models learn meaningful subspace instead of 0 D×d . Note that rank(V ) = d is a weaker constraint than V ⊤ V = I d since the pre-trained diffusion model has already learned the length information. This weaker constraint means we need less prior knowledge compared to Q(θ), which is more user-friendly. Let V ta be a minimizer of the above expected few-shot objective function. We show that V ta has a closed form and good property. Lemma 5.1. Assume Assumption 3.1 andq z = N (0, λ 2 I d ). Let C = E Xta∼q ta X ta X ⊤ta be the expected data covariance matrix. Then, V ta has a closed form:V ta V ⊤ ta = m 2 t λ 2 + σ 2 t λ 2 C + σ 2 t I D -1 C .Lemma 5.1 indicates that few-shot diffusion models can directly obtain an approximation of the minimizer without a complex optimization process. Furthermore, this minimizer has good properties and exactly recovers the subspace spanned by A ta . More specifically, the expected minimizer indicates ∥ V ta V ⊤ ta -A ta A ⊤ ta ∥ 2 F = 0 when n s and n ta are infinite. However, the source datasets n s and target datasets n ta are finite, we analyze the empirical closed-form solution¯ V ta ¯ V ⊤ ta = m 2 t λ 2 + σ 2 t λ 2 (m 2 t C + σ 2 t I D ) -1 C , where C = 1 nta nta i=1 X ta,i X ⊤ ta,iis the empirical covariance matrix. Theorem 5.2. Assume Assumption 3.1 and q z = N (0, λ 2 I d ). Let q z be the latent distribution generated by the pre-trained models with( V ta , Σ) and M = d 2 β 2 (d+λ 2 ) λ Dd log (Ddn s ) (d 2 ∨ D). Then, with probability 1 -δ 1 , we have that for any t ∈ [δ, T ] ¯ V ta ¯ V ⊤ ta -A ta A ⊤ ta 2 F ≤ O d log( 1 δ1 ) m 2 t λ 2 + σ 2 t M dδ √ n s + d n ta (m 2 t λ 2 + σ 2 t ) 2 .The above result indicates that the few-shot diffusion models can still recover the true subspace with finite n s and n ta . Note that when the latent distribution is Gaussian distribution, the approximation error bound for the source dataset is n Yuan et al., 2023) . Hence, n s in Theorem 5.2 do not depend on latent dimension d. Remark 5.3. The bound of ∥V V T -AA T ∥ 2 F only guarantees the subspace spanned by V and A is close, which still holds after an orthogonal transformation on V . Hence, this bound does not indicate ∥V -A∥ 2 F is small. Since all previous works (Chen et al., 2023c; Yuan et al., 2023) consider-1/2 s instead of n -2/d s (∥V V T -AA T ∥ 2F , we also use this metric to measure the subspace recovery. However, our results are stronger due to the closed-form solution, where previous works do not consider how to obtain V V T . √ n s dependence on the source dataset. Furthermore, the coefficient of n s term is dependent on the early stopping parameter δ and D. This is due to the δ and D dependence of the approximation bound, which is used in generating q z . However, the n ta term only has d dependence. Hence, even in the latent Gaussian setting, we still need a larger source dataset than the target dataset to obtain a sufficiently accurate closed-form solution.The relationship with principal component analysis (PCA). The expected few-shot score matching objective can be simplified tomin Vta∈ QNN ( Σ) 1/σ 4 t E Xt|X0=Xta,i ∥V ta G t V ⊤ ta X t -m t X 0 ∥ 2 2 ,whereG t = I d -σ 2 t Σ -1 t .Note that when ignoring 1/σ 4 t and choosing t = 0, the above minimization problem is similar to PCA. This suggests that few-shot models implicitly optimize an objective function akin to PCA. However, few-shot models extend beyond traditional PCA. More specifically, when λ 2 is large, classical PCA suffers from the influence of λ 2 . In contrast, due to (m 2 t λ 2 + σ 2 t )/n ta term, few-shot models can select a large t to mitigate the impact of λ 2 and achieve a 1/n ta . 
experiments: 
 To corroborate our theoretical findings, we conducted experiments utilizing real-world datasets. These experiments show that the new model obtained by only fine-tuning appropriate encoder and decoder layers on target datasets can produce novel images with the target feature, which shows the effectiveness of the methods and supports our theoretical results.Datasets and benchmark. Note that human face images tend to exhibit similarity in their latent space, primarily due to shared facial features, while differing in specific features. Hence, we initially pre-train a model using the CelebA64 dataset, focusing on distinct hairstyle features as the goal for the fine-tuning phase. For the source data, we construct a large dataset (6400 images) with different hairstyles (without the bald feature). For the target data, we choose the bald feature as the target feature and select 10 images with this feature to constitute the target dataset, which are much smaller than the size of target dataset (Figure 1 (a) ). To show the effectiveness of our methods, we also fine-tune all parameters of the pre-trained models as the benchmark.Discussion. As shown in Figure 1 , the results obtained by only fine-tuning the encoder and decoder layers can generate novel face images with the bald feature. Conversely, when fine-tuning all parameters, the models suffer from memory phenomenon and can only generate images that slightly modify the brightness and angle of the target dataset. This phenomenon indicates that only fine-tuning the appropriate encoder and decoder will result in a model with a generalization property.We note that these experiments aim to verify the effectiveness of the methods instead of achieving state-of-the-art performance since previous works carefully select specific parameters, such as specific cross-attention layers (Kumari et al., 2023) or special neurons (Liu et al., 2023) , to fine-tune pretrained models. However, we simply fine-tune all encoder and decoder layers simultaneously. There are more experiments on cat faces and more discussion on why Assumption 3.1 is satisfied in our experiments. We refer to Appendix E for more details. 
conclusion: 
 This work aims to provide a deeper understanding of few-shot diffusion models from a theoretical perspective. Our analysis is conducted from two key perspectives: the approximation and optimization aspects, all under linear structure distribution and shared latent space assumptions.From the approximation error bound, we consider general subgaussian latent variable and prove that few-shot models have a O n-2/d s + n -1/2 taapproximation bound, which is better than n -2/d ta results of diffusion models without a fine-tuning phase and escape the curse of dimensionality. This result also makes the first step to explain why few-shot diffusion models only require 5 ∼ 10 images to generate great samples. The experiments on the real-world dataset also show that the fine-tuning phase only requires 10 images to generate novel images with the target feature.When analyzing the optimization process, we consider a more special, shared Gaussian latent variable and prove that the expected score matching has a closed-form minimizer, which indicates that the few-shot diffusion models can simplify the optimization problem. Furthermore, we prove that the empirical closed-form solution has a O 1/n ta + 1/ δ √ n s accuracy bound, which still has better 1/n ta target data dependence compared to 1/ δ √ n s dependence on the source data.Future work and limitation. When considering the approximation bound, we assume a distribution with a linear structure. Though it has been supported by much empirical evidence, it is not as general as bounded distribution. After that, we plan to consider a general, bounded distribution and show the advantage of few-shot diffusion models. One possible way is to analyze the mixture of low-rank Gaussian (Wang et al., 2024) , which is more general than the linear subspace assumption.We focus on a special Gaussian latent distribution when considering the optimization problem. As a next step, we plan to consider a more general latent distribution, such as a log-concave distribution.In this setting, we can not directly obtain the closed-form solution. However, due to the shared information and simplified landscape, it is still possible to use some optimization algorithms, such as gradient descent, to optimize the few-shot objective function to achieve the convergence guarantee.Broader Impact. This paper presents work whose goal is to understand few-shot diffusion models from the theoretical perspective. A noteworthy societal impact is that few-shot diffusion models may be used to imitate the style of artists and generate fake images, thereby infringing on the rights of artists (Mirsky & Lee, 2021) . We recommend adding watermarks to images to determine whether the image was generated by a generative model (Fernandez et al., 2023) . The other societal impact is the same as general generative models (Mishkin et al., 2022) . 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have discussed the broader impacts of our work at the end of main paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: This paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we introduce the multi-layer ReLU network f θ ∈ NN (L, M ′ , J, K 1 , κ, γ, γ t ) in S NN . We note that the following setting is exactly the same as the one in Chen et al. (2023c) , and we show the structure for completeness. We denote by NN (L, M ′ , J, K 1 , κ, γ, γ t ) the following neural network:NN (L, M ′ , J, K 1 , κ, γ, γ t ) = {f (Z, t) = W L σ . . . σ W 1 Z ⊤ , t ⊤ + b 1 . . . + b L : network width bounded by M ′ , sup z,t ∥f (z, t)∥ 2 ≤ K 1 , max {∥b i ∥ ∞ , ∥W i ∥ ∞ } ≤ κ for i = 1, . . . , L, L i=1 (∥W i ∥ 0 + ∥b i ∥ 0 ) ≤ J, ∥f (Z 1 , t) -f (Z 2 , t)∥ 2 ≤ γ ∥Z 1 -Z 2 ∥ 2 for any t ∈ [0, T ], ∥f (Z, t 1 ) -f (Z, t 2 )∥ 2 ≤ γ t |t 1 -t 2 | for any Z} ,where σ is the ReLU activation function. Given an network accuracy ϵ > 0, the parameters is defined byL = O log 1 ϵ + d , K 1 = O 2d 2 log d δϵ , M ′ = O (1 + β) d T τ d d/2+1 ϵ -(d+1) log d/2 d δϵ , J = O (1 + β) d T τ d d/2+1 ϵ -(d+1) log d/2 d δϵ log 1 ϵ + d , κ = O max 2(1 + β) d log d δϵ , T τ , γ = 10d(1 + β), γ t = 10τ , where τ = sup A∈{As,Ata} sup t∈[δ,T ] sup ∥z∥∞≤ √ d log d δϵ ∂ ∂t σ t A∇ log q LD t (z) 2 .Let ( V s , θ) be the minimizer of the pre-trained objective function. The few-shot diffusion model freezes the bottleneck network and fine-tunes V ta ∈ R D×d to obtain the minimizer ( V ta , θ) of the few-shot objective function. As the first step, we show that with the bottleneck parameterized by θ, there also exists a solution( Vta , θ) ∈ Q N N ( θ) achieve the ϵ 2 + n -2/d s error bound. Lemma B.1. If ϵ ≤ n - 1-α(ns) d+5 s, where α(n) = d log log n log n , then with probability 1 -δ 1 , there exists a solution ( Vta , θ)∈ Q N N ( θ) such that T δ E X∼q ta t s Vta, θ (X, t) -∇ log q ta t (X) 2 2 dt ≤ O d δ ϵ 2 + (T -δ)(d + C Z )d 2 β 2 δ 2 c 0 n - 2-2α(ns) d+5 s log( 1 δ 1 ) .Proof. As shown in Theorem 1 of Chen et al. (2023c) , there exists a solution ( Vs , θ) in S N N such thats Vs, θ (•, t) -∇ log q s t (•) 2 ≤ √ d + 1 σ 2 t ϵ , ∀t ∈ [δ, T ] .Hence, we do the following decomposition:s Vta, θ (•, t) -∇ log q ta t (•) 2 2 ≲ s Vta, θ (•, t) -∇ log q ta t (•) 2 2 + s Vta, θ (•, t) -s Vta, θ (•, t) 2 2For the encoder and decoder layer, we choose Vta = A ta . The first term is bounded due to the construction of the neural network. We first show that s Vta, θ is ϵ-close to the true score function ∇ log q ta t . Since the encoder and decoder have been chosen, we only need to focus on the latent bottleneck. For the latent bottleneck, we need to use f θ (Z, t) to approximate ground-truth function h(Z, t) = σ 2 t ∇ log q LD t (Z) + Z for Z ∈ R d . Chen et al. (2023c) show that for any latent variable Z ′ ∈ R d with subgaussian tail, we have that∥h (Z ′ , t) -fθ (Z ′ , t)∥ L 2 (q LD t ) ≤ ( √ d + 1)ϵ .Then, we have thats Vta, θ (•, t) -∇ log q ta t (•) 2 2 ≤ d σ 4 t ϵ 2 .For the second term, we know that with probability 1 -δ 1 :T δ E X∼qt s Vta, θ (•, t) -s Vta, θ (•, t) 2 2 ≤ T δ 1 σ 4 t E Z∼q LD t ∥f θ (Z) -fθ(Z)∥ 2 2 dt ≤ O T -δ δ 2 ( δ c 0 (T -log δ) d • γ 2 + dβ + γ 2 • C Z c 0 )n - 2-2α(ns) d+5 s log 1 δ 1 ,where α(n) = d log log n log n . The first inequality follows A ta is a matrix with orthonormal columns. Since we assume ϵ ≤ n To prove Theorem 4.3, we need to do the following decomposition for the population loss of the target datasetsEQUATIONEQUATIONEQUATION, where a ∈ (0, 1) and L trunc ta is defined asL trunc ta s Vta, θ = E x∼q0 ℓ trunc ta x; s Vta, θ = E x∼q0 ℓ ta x; s Vta, θ 1 {∥x∥ 2 ≤ R} dt .In this section, we take R = O d log d + log K 1 + log nta δ1 to guarantee P Xta,i∼q ta ∥X ta,i ∥ 2 ≤ R for all i = 1, . . . , n ta ) ≥ 1 -δ 1 , where K 1 is defined in Appendix A.•; s V, θ : s V, θ ∈ Q NN ( θ) , which is induced by Q( θ).For the upper bound of G( θ), we directly use the augmentation of Chen et al. (2023c) to obtain thatℓ trunc ta X; s V, θ ≤ O K 2 1 + R 2 δ (T -δ) , for any s V, θ ∈ Q NN ( θ) .Then, by using Lemma D.1, we know that with probability 1 -δ 1 , term (a) is bounded byO   (1 + 3/a) (1 + β) 2 d 2 log d δϵ + log nta δ n ta δ (T -δ) log N τ 1 , G( θ), ∥ • ∥ ∞ δ 1 + τ 1   .To bound the above term, we need to calculate the covering number of G( θ), which is related to a ι-covering of Q N N ( θ). Suppose that given s V1, θ and s V2, θ withsup ∥x∥2≤3R+ √ D log D,t∈[δ,T ] s V1, θ (x, t) -s V2, θ (x, t) 2 ≤ ι, we need to bound ℓ trunc ta •; s V1, θ -ℓ trunc ta •; s V2, θ ∞ .By using the same calculation compared to Term (A) of Chen et al. (2023c) , we know thatℓ trunc ta •; s V1, θ -ℓ trunc ta •; s V2, θ ∞ ≤ O ι T -δ (K 1 + R) log T δ + 4K 1 (K 1 + R) δ (T -δ) (R/D) D-2 exp - 1 σ 2 t R 2 .The above inequality indicates that a ι-covering ofQ N N ( θ) in L 2 norm leads a ι T -δ (K 1 +R) log T δ + 4K1(K1+R) δ(T -δ) (R/D) D-2 exp -1 σ 2 t R 2 -covering of of G( θ) in L ∞ norm. By taking R = O d log d + log K 1 + log nta δ1 , K 1 = O 2d 2 log d δϵ , ι = 2 ntaδ(T -δ) , we know that τ 1 ≤ d 2 n ta δ log( T δ ) log( d δϵ ) log( n ta δ ) ,which indicates with probability 1 -δ 1 , term (a) is bounded byO   (1 + 3/a) (1 + β) 2 d 2 log d δϵ + log nta δ n ta δ (T -δ) log N 1 ntaδ(T -δ) , Q NN ( θ), ∥ • ∥ 2 δ 1 + d 2 n ta δ log( T δ ) log( d δϵ ) log( n ta δ )   .After that, we need to determine the covering number of Q N N ( θ) with a truncated X to bound term (a). Lemma B.2. The logarithmic covering number ofQ N N (θ) for ∥X∥ 2 ≤ 3R + √ D log D, t ∈ [δ, T ] is log N (ι, Q N N (θ), ∥ • ∥ 2 ) = O 2Dd • log 1 + 6Kγ √ d(3R + √ D log D) δι .Proof. Suppose that there exists two orthonormal column matrix Term (c). For the term (c), we know that it is bounded by the constructed solution ( Vta , θ):EQUATIONEQUATION.For the term (C.1), since s Vta, θ is a fixed function, we directly use the results of (Chen et al., 2023c) :Term(C 1 ) ≤ O (1 + 6/a) (1 + β) 2 d 2 log d δϵ + log n δ n ta δ (T -δ) log 1 δ 1 ,with probability 1 -δ 1 . For the term (C.2), we know thatL trunc ta s Vta, θ ≤ L s Vta, θ = 1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (qt) dt + L s Vta, θ - 1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (qt) dt (E).As we show in Section 3.2, the two terms in E are both the score matching objective function and have a constant different E, which is independent of the trainable parameters (V, θ). We denote by this difference E andF = (d+C Z )d 2 β 2 δ 2 c0. With probability 1 -δ 1 , Lemma B.1 shows that term (C.2) is bounded byO d δ(T -δ) ϵ 2 + F n - 2-2α(ns) d+5 s log 1 δ 1 + E .After bounding these three terms, we prove Theorem 4.3.Theorem 4.3. Let α(n) = d log log n log n , F = (d+C Z )d 2 β 2 δ 2 c0and network parameter ϵ = n -1/2 ta . Assume Assumption 3.1, 4.1, 4.2 and n d+5 4(1-α(ns)) ta ≥ n s . Then, with probability 1 -δ 1 , the following inequality holds (hiding logarithmic factors)1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (q ta t ) dt ≤ Õ (1 + β) 2 Dd 3 δ (T -δ) √ n ta + F n - 2-2α(ns) d+5 s log 1 δ 1 .Proof. Equipped with the bound of the term (a), (b), and (c) and hiding the logarithmic term (except for the covering number term), with probability 1 -δ 1 , we have thatL ta s Vta, θ ≤ (1 + a) 2 E + Õ (1 + β) 2 d 2 log d δϵ + log nta δ aδ (T -δ) n ta log N 1 ntaδ(T -δ) , Q NN ( θ), ∥ • ∥ 2 δ 1 + d 2 n ta δ + 1 n ta δ(T -δ) + d δ(T -δ) ϵ 2 + F n - 2-2α(ns) d+5 s log 1 δ 1 . Since L ta s Vta, θ -E = 1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (qt), we have that the following inequality when choosing a = ϵ 2 :1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (qt) ≤ Õ   (1 + β) 2 d 2 ϵ 2 δ (T -δ) n ta log N 1 ntaδ(T -δ) , Q NN ( θ), ∥ • ∥ 2 δ 1 + d δ(T -δ) ϵ 2 + d 2 n ta δ + F n - 2-2α(ns ) d+5 s log 1 δ 1   ≤ Õ (1 + β) 2 Dd 3 δ (T -δ) n ta ϵ 2 log 1 + 6Kγ √ d(3R + √ D log D)n ta δ(T -δ) δ 1 + d δ(T -δ) ϵ 2 + d 2 n ta δ + F n - 2-2α(ns) d+5 s log 1 δ 1 ,where the second inequality follows the convering number ofQ N N ( θ) for ∥X∥ ≤ 3R + √ D log D with R = O d log d + log K 1 + log nta δ1and the network parameters is defined in Appendix A.Finally, we choose ϵ 2 = 1/ √ n ta , then we have that1 T -δ T δ s Vta, θ (•, t) -∇ log q ta t (•) 2 L 2 (qt) ≤ Õ (1 + β) 2 Dd 3 δ (T -δ) √ n ta log 1 δ 1 + d 2 n ta δ + F n - 2-2α(ns) d+5 s log 1 δ 1 .As we require in Lemma B.1, we need ϵ = 1/n 1/4 ta ≤ n Since we need to use the approximated latent distribution in the few-shot phase, we show that the pre-trained diffusion models with solution ( V s , θ) can generate an accurate enough latent distribution. As shown in Section 5, when considering the optimization perspective of diffusion models, we assume the latent distribution is a Gaussian distribution q z = N (0, Σ) with Σ = diag λ 2 1 , . . . , λ 2 d ≻ 0. Yuan et al. (2023) show that in the setting, the approximation error bound (Lemma D.3) for the target dataset is1 T -δ T δ ∇ log q s t (•) -s Vs, θ (•, t) 2 L 2 (q s t ) dt ≤ O   1 δ (d 2 + Dd) log (Ddn s ) (d 2 ∨ D) log 1 δ1 n s   .To generate latent distribution, we first introduce the reverse process in the latent space. The introduction mainly follows the outline of Appendix C.2 of Chen et al. (2023c) . For X t , we can do the following decomposition:X t = A s Z t + X t,⊥ , where Z t = A ⊤ s X t . With Z ← t = Z T -t, the reverse process in the latent space isdZ ← t = 1 2 Z ← t + ∇ log q LD T -t (Z ← t ) dt + d A ⊤ s B tAs shown in Theorem 3 of Chen et al. (2023c) , the solution ( V s , θ) of the pre-trained diffusion models only guarantee∥ V s V ⊤ s -A s A ⊤ s ∥ 2 F is small instead of ∥ V s -A s ∥ 2F is small. Hence, Theorem 3 of Chen et al. (2023c) assume there exists an orthogonal matrix U s ∈ R d×d and do an orthogonal transformation on V s to obtain V s U s , which can guarantee ∥ V s U s -A s ∥ 2 F is small. After such orthogonal transformation, the reverse process with an approximated score function and an approximated reversing beginning distribution Z ←,rEQUATIONwhereZ ←,r t = U ⊤ s Z ← t and s LD Us, θ (Z, t) = 1 σ 2 t -Z + U ⊤ s f θ (U s Z, t) .Then, we discretize the above process with the exponential integrator (EI) discretization scheme (Zhang & Chen, 2022) to obtain an implementable algorithm:d Z ⇐,r t = 1 2 Z ⇐,r t + s LD Us, θ Z ⇐,r kη , T -t ′ k dt + d U ⊤ s V ⊤ s B t , where t ∈ [t ′ k , t ′ k+1 ] . (3)As shown in Appendix C.4 of Chen et al. (2023c) , if the target ground truth score function has a L 2 -accurate approximated score:1 T -δ T δ s Vs, θ (•, t) -∇ log q s t (•) 2 L 2 (q s t ) dt ≤ ϵ 2 ,the latent score function also has an L 2 norm bound ϵ 2 latent-score , which is determined by ϵ:ϵ latent-score = ϵ • O δ c 0 (T -log δ) d • γ 2 + dβ + γ 2 • C Z c 0 .The remaining term is to determined ϵ. Since we assume Gaussian latent variable instead of sub-Gaussian one. Hence, we do not use ϵ in Chen et al. (2023c) . We use ϵ in Yuan et al. ( 2023) (Theorem 4.5 of Yuan et al. ( 2023)), which also considers Gaussian latent variable, to achieve the final results.Finally, we have that with probability 1 -δ 1 :EQUATIONLet p LD t be the distribution of the algorithm (the above discretization process). In the following lemma, we adopt Theorem 5 of Chen et al. (2023a) and show that the pre-trained diffusion model can obtain an accurate enough latent distribution with the above L 2 -accurate latent score function.Lemma C.1. With ϵ 2 latent-score defined in Equation (4), T = logλ 2 max +d ϵ 2 latent-score and K = Θ d 2 (T +log(λ 2 max )) 2 ϵ 2 latent-score, by using the exponentially decreasing (then the constant) stepsizeh k = c min max t k , 1 λ 2 max , 1 , c = log(λ 2 max )+T K, the results p LD T of sampling algorithm (Equation (3)) has the following guarantee with probability 1 -δ 1 (hiding the logarithmic factor):KL q 0 ∥p LD T ≤ O(ϵ 2 latent-score ) = O   d 2 β 2 d + λ 2 max λ min δ (d 2 + Dd) log (Ddn s ) (d 2 ∨ D) log 1 δ1 n s   Proof.The Theorem 5 of Chen et al. (2023a) show that if ∇ log q LD 0 is L-Lipschitz, diffusion models with a L 2 -accurate can generate p LD T , which is close to q 0 in KL divergence. Since q z = N (0, Σ), it is easy to verify L = λ 2 max . Then, we complete the proof. ■When the latent distribution is a Gaussian distribution q z = N (0, Σ) with Σ = diag λ 2 1 , . . . , λ 2 d ≻ 0, the ground truth score function for the target dataset is∇ log q ta t (X) = -A ta Σ -1 t A ⊤ ta X - 1 σ 2 t I D -A ta A ⊤ ta X ,where Σ t = diag . . . , m 2 t λ 2 k + σ 2 t , . . . . Since the matrix A ta is independent of time t, we fix a t ∈ [δ, T ] and minimize the few-shot objective function at this time. With an approximated Σ, which is learned by the pre-trained diffusion models, f θ (Z, t) = (I d -σ 2 t Σ -1 t )Z, where Σ t = m 2 t Σ + σ 2 t I d . Hence, the expected objective function for the few-shot diffusion models at a fixed time t is:min s V ta , θ ∈ QNN ( θ) L ta,t (s Vta, θ ) = E Xta∼qta ℓ ta t X ta ; s Vta, θ ,whereQNN (θ) = {s V,θ (X, t) = 1 σ 2 t V f θ V ⊤ X, t - 1 σ 2 t X : V ∈ R D×d with rank(V ) = d.} , Note that the constraint rank(V ) = d is a weaker constraint than V ⊤ V = I d since rank(V ) = d does not involve length information.Lemma C.2. Assume Assumption 3.1 and q z = N (0, λ 2 I d ). Let C = E Xta∼q ta X ta X ⊤ ta be the expected data covariance matrix. Then, V ta has a closed form:V ta V ⊤ ta = m 2 t λ 2 + σ 2 t λ 2 C + σ 2 t I D -1 C . Proof. Let G t = I d -σ 2 t Σ -1t , then we have thatℓ ta t X ta,i ; s Vta, θ = E Xt|X0=Xta,i ∥ 1 σ 2 t V ta G t V ⊤ ta X t - 1 σ 2 t X t -∇ log q ta t (X t |X 0 )∥ 2 2 = 1 σ 4 t E Xt|X0=Xta,i ∥V ta G t V ⊤ ta X t -m t X 0 ∥ 2 2where the second equality follows ∇ logq ta t (X t |X 0 ) = -Xt-mtX0 σ 2 t . Let C = E Xta∼qta X ta X ⊤ ta be the expected covariance matrix of the target dataset. With the fact E Xt|X0 [X t X ⊤ t ] = m 2 t X 0 X ⊤ 0 + σ 2 t I D and E Xt|X0 [X 0 X ⊤ t ] = m t X 0 X ⊤ 0 ,the optimization problem can be simplified to the following form (without misunderstanding, we ignore the subscript ta):min V ∈R D×d ,rank(V )=d L(V ) = ∥(m 2 t C + σ 2 t I D ) 1 2 V G t V ⊤ ∥ 2 F -2m 2 t tr(V G t V ⊤ C) ,where(m 2 t C + σ 2 t I D ) 1 2 is meaningful since (m 2 t C + σ 2 t I D) is positive-definite matrix. Let V be the solution of the above minimization problem. We first ignore the constraint rank(V ) = d and calculate ∂L(V )/∂V = 0 (since V also satisfied ∂L(V )/∂V | V = V = 0), we know that V satisfies the following equality:EQUATIONIn Section 5, we assume the latent distribution is a isotropic Gaussian distribution q z = N (0, λ 2 I d ).In this setting, Σ is equal toλ 2 I d and G t = m 2 t λ 2 m 2 t λ 2 +σ 2 t , which indicate the closed form solution of V is V V ⊤ = m 2 t λ 2 + σ 2 t λ 2 (C + σ 2 t I D ) -1 C . The last step is to prove rank( V V ⊤ ) = d. Note that rank(C) = rank E Xta∼qta X ta X ⊤ ta = rank(AΣA ⊤ ) = d, which indicates min{rank( V V ⊤ ), rank(m 2 t C + σ 2 t I D )} ≥ d . Combined with V ∈ R D×d , we complete the proof. ■ C.3The Error Bound for the empirical Closed-form SolutionIn this part, we prove the accuracy bound of the empirical version closed form solution ¯ V ¯ V ⊤ w.r.t. n s and n ta . The empirical solution has the following form (without misunderstanding, we ignore the subscript ta):¯ V ¯ V ⊤ = m 2 t λ 2 + σ 2 t λ 2 (m 2 t C + σ 2 t I D ) -1 C ,where C = 1 nta nta i=1 X ta,i X ⊤ ta,i . Theorem 5.2. Assume Assumption 3.1 and q z = N (0, λ 2 I d ). Let q z be the latent distribution generated by the pre-trained models with ( V ta , Σ) andM = d 2 β 2 (d+λ 2 ) λ Dd log (Ddn s ) (d 2 ∨ D). Then, with probability 1 -δ 1 , we have that for any t ∈ [δ, T ] ¯ V ta ¯ V ⊤ ta -A ta A ⊤ ta 2 F ≤ O d log( 1 δ1 ) m 2 t λ 2 + σ 2 t M dδ √ n s + d n ta (m 2 t λ 2 + σ 2 t ) 2 .Proof. The empirical solution indicates thatEQUATIONTo analyze this equality, we first show that λ 2 is accurate enough. We know that KL q 0 ∥p LDT = d λ 2 / λ 2 -log(λ 2 / λ 2 ) -1 . Let M 1 = d 2 β 2 (d+λ 2 max ) λminδ (d 2 + Dd) log (Ddn s ) (d 2 ∨ D). Then, Lemma C.1 show that with probability 1 -δ 1 , we have KL q 0 ∥p LD T = d λ 2 / λ 2 -log(λ 2 / λ 2 ) -1 ≤ O   M 1 log(1/δ 1 ) n s   .Combined with the above inequality, we know that with probability1 -δ 1 , |λ 2 / λ 2 -1| ≤ M1 √ log(1/δ1) d √ nsby using the Taylor Expansion. By using Lemma D.2, we know that with probability 1 -δ 1 :EQUATIONFor the left hand of Equation ( 6), we know thatEQUATIONCombined with Equation (7) and Equation ( 8), we have thatm 2 t λ 2 AA ⊤ + σ 2 t I D ¯ V ¯ V ⊤ -AA ⊤ ≤ O   M 1 log(1/δ 1 ) d √ n s + 2 d + log (1/δ 1 ) √ n ta   m 2 t λ 2 + σ 2 t + M 1 log(1/δ 1 ) d √ n s σ 2 t     AA ⊤ + 2m 2 t λ 2 d + log (1/δ 1 ) √ n ta ¯ V ¯ V ⊤ . Let M 2 (n s , n ta ) = M1 √ log(1/δ1) d √ ns + 2 √ d+log(1/δ1) √ nta m 2 t λ 2 + σ 2 t + M1 √ log(1/δ1) d √ ns σ 2 t . Accord-ing to symmetry, we know thatm 2 t λ 2 AA ⊤ + σ 2 t I D ¯ V ¯ V ⊤ -AA ⊤ 2 F ≤ O M 2 (n s , n ta ) 2 ∥AA ⊤ ∥ 2 F + m 4 t λ 4 (d + log (1/δ 1 )) n ta ∥ ¯ V ¯ V ⊤ ∥ 2 F ≤ O D 2 (M 2 (n s , n ta ) 2 .The last inequality follows that each element of ¯ V ¯ V ⊤ is bounded by some constant due to the form of the empirical closed form solution and∥AA ⊤ ∥ 2 F = tr(AA ⊤ AA ⊤ ) = tr(AA ⊤ ) = tr(I d ) = d .For the right hand of the above inequality, we know thatm 2 t λ 2 AA ⊤ + σ 2 t I D ¯ V ¯ V ⊤ -AA ⊤ 2 F = Tr ¯ V ¯ V ⊤ -AA ⊤ ¯ V ¯ V ⊤ -AA ⊤ m 2 t λ 2 AA ⊤ + σ 2 t I D m 2 t λ 2 AA ⊤ + σ 2 t I D ≥ m 2 t λ 2 + σ 2 t ¯ V ¯ V ⊤ -AA ⊤ 2 F .Then, we complete the proof ■The following concentration lemma comes from Lemma 15 of Chen et al. (2023c) . Lemma D.1 (Lemma 15, (Chen et al., 2023c) ). Let G be a bounded function class, i.e., there exists a constant B such that any g ∈ G : R d → [0, B]. Let z 1 , . . . , z n ∈ R d be i.i.d. random variables. For any δ ∈ (0, 1), a ≤ 1, and τ > 0, we haveP sup g∈G 1 n n i=1 g (z i ) -(1 + a)E[g(z)] > (1 + 3/a)B 3n log N (τ, G, ∥ • ∥ ∞ ) δ 1 + (2 + a)τ ≤ δ 1 and P sup g∈G E[g(z)] - 1 + a n n i=1 g (z i ) > (1 + 6/a)B 3n log N (τ, G, ∥ • ∥ ∞ ) δ 1 + (2 + a)τ ≤ δ 1 .In the following lemma, we show the concentration of the data covariance matrix. Note that the proof sketch of the following lemma mainly follows Lemma 6 of (Du et al., 2020) . We prove a concentration bound that depends on n instead of a constant bound with a large enough n. Lemma D.2 (The Modified Lemma A.6, (Du et al., 2020) ). Let a 1 , . . . , a n be i.i.d. d-dimensional random vectors such that E [a i ] = 0, E a i a ⊤ i = I, and a i is ρ 2 -subgaussian. Then with probability at least 1 -δ 1 we have Du et al. (2020) , we use an ϵ-net argument for the unit sphere S d-1 = v ∈ R d : ∥v∥ = 1 . For any v ∈ S d-1 , we know that v ⊤ a i 2 -1 is zero-mean and 16ρ 2 -subgaussian. By using the Bernstein inequality, we have for any ϵ > 01 - 2ρ 2 d + log (1/δ 1 ) √ n I ⪯ 1 n n i=1 a i a ⊤ i ⪯ 1 + 2ρ 2 d + log (1/δ 1 ) √ n I Proof. Let A = 1 n n i=1 a i a ⊤ i -I. Similar toPr v ⊤ Av > ϵ ≤ 2 exp - n 2 min ϵ 2 (16ρ 2 ) 2 , ϵ 16ρ 2 .Next, we take a d) . By using the union bound, we know that1 5 -net N ⊂ S d-1 of S d-1 with size |N | ≤ e O(Pr max v∈N v ⊤ Av > ϵ ≤ 2|N | exp - n 2 min ϵ 2 (16ρ 2 ) 2 , ϵ 16ρ 2 ≤ exp O(d) - n 2 min ϵ 2 (16ρ 2 ) 2 , ϵ 16ρ 2Let the right hand of the above inequality equals to δ 1 . We know that with probability 1 -δ 1 :max v∈N v ⊤ Av ≤ ρ 2 d + log (1/δ 1 ) √ n .Note that for any u ∈ S d-1 , there exists u ′ ∈ N such that ∥u -u ′ ∥ ≤ 1 5 . Then, we have that|u ⊤ Au| ≤ | (u ′ ) ⊤ Au ′ | + 2| (u -u ′ ) ⊤ Au ′ | + | (u -u ′ ) ⊤ A (u -u ′ ) | ≤ ρ 2 d + log (1/δ 1 ) √ n + 1 2 ∥A∥ 2 ,where ∥A∥ 2 is the operator norm of matrix A. Taking a supreme over u ∈ S d-1 , we have that∥A∥ 2 ≤ ρ 2 d + log (1/δ 1 ) √ n + 1 2 ∥A∥ 2 .Then, we complete the proof. ■In Section 5, we assume the latent distribution is a Gaussian distribution instead of a subgaussian one. Yuan et al. (2023) show that in this setting, the approximation error bound has better dependence on n s .Lemma D.3 (Lemma C.1, (Yuan et al., 2023) ). Assume the latent distribution is a Gaussian distribution q z = N (0, Σ) with Σ = diag λ 2 1 , . . . , λ 2 d ≻ 0. Then, the solution ( V s , θ) of Equation (1) has the following approximation error bound with probability 1 -δ 1 :1 T -δ T δ ∇ log q s t (•) -s Vs, θ (•, t) 2 L 2 (q s t ) dt ≤ O   1 δ (d 2 + Dd) log (Ddn s ) (d 2 ∨ D) log 1 δ1 n s   .In this section, we do experiments on real-world datasets to show that the new model obtained by only fine-tuning appropriate encoder and decoder layers on target datasets with only 10 images can generate novel images with the target dataset feature. On the contrary, if all parameters can be fine-tuned, the model will suffer from memory phenomenon and only generate the ten images in the target dataset. This phenomenon indicates that only fine-tuning the appropriate encoder and decoder will result in a model with a generalization property.Setting. In this experiment, we use a U-net network with attention layers, which contains 11 downblocks, 2 middleblocks, and 15 upblocks. When only fine-tuning the encoder and decoder layers, we fine-tune the first 4 downblock layers (encoder) and 4 upblock layers (decoder) instead of only using linear layers as the encoder and decoder (discuss in the later discussion paragraph).The above experiments are conduct on a GeForce RTX 4090. We train the neural network using AdamW optimizer with learning rate 0.0001. For the pre-trained phase, we train the models for 200 epochs with batch size 20. It takes 5 hours to obtain a pre-trained diffusion models. For the fine-tuning phase, we fine-tune the pre-trained models for 400 epochs with batch size 2. It take 3 minutes to fine-tune the pre-trained models. Discussion on results. The experiment results of CelebA64 have been discussed in Section 6. The experiment phenomenon is similar for the cat face images, which means the models obtained by only fine-tuning the encoder and decoder can generate novel images with the target feature (Figure 2 ). We note that when choosing the target cat face dataset if the color black constitutes more than 70% of the image's composition, we view this cat image as the black cat. Hence, different colors exist for cats, such as white and grey, due to the target dataset containing a small number of these colors (such as images 1, 3, 4, 6, 8). As a result, our fine-tuning results also contain these colors. However, our results do not contain colors other than those in the target dataset and can produce novel samples, which also proves the effectiveness of our fine-tuning method.Discussion on linear encoder and decoder. Assumption 3.1 assumes the linear subspace, which indicates linear encoder and decoder. However, we fine-tune the first 4 downblock layers (encoder) and 4 upblock layers (decoder) instead of only using linear layers as the encoder and decoder. We note that this operation does not conflict with our Assumption. Recall that in Stable Diffusion (Rombach et al., 2022) , the diffusion models run in the VAE embedding space7 . Hence, we can view the first 3 downblock layers and the last 3 upblock layers as the VAE encoder and VAE decoder. Then, we can obtain X in this paper by running the VAE encoder. The remaining 1 downblock and 1 upblock layer can be viewed as linear encoder and decoder A. As mentioned in Section 4.2 of StyleGAN (Karras et al., 2019) , the feature of X obtained by running a good-enough VAE encoder has linear separability, which also supports our Assumption 3.1.Our analysis can be extended to f (Xt, t) = -1/2βtXt and g(t) = √ βt, where {βt} t≥0 is non-decreasing and bounded sequence.Here, the approximation error means the score matching error with finite source and target datasets.Since smaller D is more friendly to nta, our discussion holds for all datasets in Table1.This also indicates the requirement of nta in Theorem 4.3 is easy to satisfy.To distinguish the latent space in this paper, we use embedding space here. 


Decision: The reviewers agree that the paper presents a strong and well-founded theoretical framework for understanding few-shot diffusion models. As highlighted by Reviewer kXcP, the theory is both novel and interesting, offering meaningful insights into the training of few-shot diffusion models. Furthermore, the theoretical contributions are robustly supported by comprehensive experimental results, enhancing the overall impact and validity of the work.

Reviews: review: paper_summary: paper_summary: This paper studies the few-shot transfer in diffusion models. Specifically, it focuses on bounding the score-matching loss for the target distribution, which can be considered the estimation error of the corresponding score function. By assuming the source and target distribution share the same linear structure distribution on latent space of dimension $d$,  it gives a $\tilde{\mathcal{O}}(n_s^{-2/d} + n_{ta}^{-1/2})$ bound for training the few-shot diffusion model with $n_s$ source domain data and $n_{ta}$ target domain data. As claimed by the authors, this breaks the curse of dimensionality for the target domain with an improvement from $n_{ta}^{-2/d}$ (no few-shot adaptation) to $n_{ta}^{-1/2}$. In addition, they also proved a bound $\tilde{\mathcal{O}}(2/n_{ta} + 1/\sqrt{n_s})$ for the optimization error of score matching loss for a latent Gaussian special case.
strength_weakness: strength_weakness: * Approximation error has a specific definition in learning theory, which characterizes the inductive bias caused by the selection of the hypothesis class [1]. When using the terminology as another definition, e.g., the score matching loss of an estimated score function with early stopping in Theorem 4.3, it should be obviously defined to avoid confusion.   
* I have the following concerns about presenting and interpreting the bounds:
	* In Theorem 4.3, the bound depends on $F$, which is related to $d^3$. So, the order should be $\tilde{\mathcal{O}}(d^3(n_s^{-2/d} + n_{ta}^{-1/2}))$, I am not sure if $d^3$ could be hidden directly. 
	* According to this, when $d$ becomes large, the error should grow, which means the requirement of $n_{ta}$ should grow. However, in Table 1, with the increase of latent dimension, $n_{ta}$ decreases for ImageNet and MS-COCO. This is contradictory to your results. 
	* When $d<4$, isn't few-shot fine-tuning worse than directly training given the theorem? 

* I am not quite convinced by the explanation of the experiments.  
	* How is the difference between fine-tuning all the parameters and fine-tuning a smaller fraction of encoder-decoders reflected on your bounds? 
	* I think there exist many ways to adjust the hyper-parameter for fine-tuning all the parameters to achieve similar results as fine-tuning a small part.

[1] Shalev-Shwartz, Shai, and Shai Ben-David. _Understanding machine learning: From theory to algorithms_. Cambridge University Press, 2014.
questions: questions: 1. Why $\tilde{V}_{ta}$ in Lemma 5.1 depends on time $t$ ? Shouldn't it be the minimizer of the integral over $t$? 


Grammar: Line 271, 278, 280, 289, etc ...
I suggest the authors check the grammar and revise the paper accordingly.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper provides new bounds for the score function approximation and optimization in fine-tuning pretrained diffusion models (under some simplifying assumptions including linear structure of data distributions and equal latent distributions). The new approximation error bound depends only on the square root of the finetuning target sample size, which in turn suggests that fine-tuning diffusion models does not suffer from the curse of dimensionality (unlike their pretraining error bound which has an exponential dependence on the latent dimension). Furthermore, the accuracy bound and closed-form minimizer provides interesting insights into the fine-tuning behavior of diffusion models. The paper also provides limited empirical support for its theoretical claims on CelebA.
strength_weakness: strength_weakness: I do not see any major weakness in this work, but I do have two minor concerns:

1- The paper mentions that data-efficient fine-tuning of diffusion models is very successful and tries to explain this phenomenon. However, the paper does not clarify the extent of this success. I think the paper would benefit from discussing the many ways in which existing fine-tuning methods are suboptimal (diversity, out of distribution, etc), so as to provide a more nuanced context to the reader.

2- The paper does not sufficiently discuss the importance of its assumption, that is, what happens if each one breaks? Which ones are more crucial than others? Which ones are less realistic than others?
questions: questions: Please see my questions in the Weaknesses section.
limitations: limitations: The paper sufficiently discusses its weaknesses.
paper_summary: paper_summary: The paper provides a theoretical analysis showing that fine-tuning pretrained diffusion models on a few target samples achieves a small approximation error, particularly in that it requires much fewer target data than source data. The authors also show that the solution has a closed form in a special case, demonstrating the ease of optimization for few-shot models. They provide bounds for this closed-form solution, which also shows better dependency on the target sample size than on the source sample size. Experiments on CelebA are conducted to further support the theoretical conclusions.
strength_weakness: strength_weakness: I don't have any particular criticisms; however, I am not an expert in this topic, particularly regarding the theoretical analysis of diffusion models. I would like to hear other reviewers' opinions.
questions: questions: I wonder if there is any intuitive explanation for assumptions like 4.1 and 4.2, particularly what they could mean in practice. Or are they just technical assumptions without much intuition?
limitations: limitations: The authors have discussed certain limitations in Section 7.
paper_summary: paper_summary: The paper provides a theoretical analysis of the few-shot fine-tuning problem for diffusion models. It makes the following key assumptions: (1) the pretraining and fine-tuning data distributions share a common latent distribution, and (2) a specific network architecture. Under these assumptions, the paper proves that during the fine-tuning phase, the diffusion model avoids the curse of latent dimensionality and achieves an approximation bound of $\widetilde{\mathcal{O}}(n_s^{-2/d}+n_{ta}^{-1/2})$. Furthermore, when considering a low-rank Gaussian distribution, the model attains an accuracy bound of $\widetilde{\mathcal{O}}(1/n_{ta}+1/\sqrt{n_{s}})$. The paper also includes real-world experiments to support the theoretical findings.
strength_weakness: strength_weakness: There is a gap between the real-world experiments and the theoretical results. The experiments demonstrate two main points: (1) fine-tuning the entire diffusion model leads to memorization, and (2) fine-tuning only the encoder and decoder parts of the diffusion model will generalize. However, these findings only support the validity of the fine-tuning definition used in the paper.
The gap lies in two areas:
1. The paper lacks proof that fine-tuning the full model results in memorization.
2. The real-world experiments do not show that few-shot fine-tuning can escape the curse of dimensionality. Specifically, the diffusion model still requires a similar number of samples for fine-tuning datasets with different intrinsic dimensions.

I think the paper lacks sufficient real-world experiments to fully support its title, "Few-Shot Diffusion Models Escape the Curse of Dimensionality." With such an experiment, I believe the paper will be a strong submission.
questions: questions: In section 4.1, Table 1, the requirement derived from $n_{ta}^{\frac{d+5}{4(1 - \alpha(n_s))}} \geq n_s$ is counterintuitive. For datasets like ImageNet with higher latent dimensions, the requirement of $n_{ta}$ is less. Could the author provide more discussions or give some real-world experiments to support this?
limitations: limitations: The paper has covered the limitations and societal impact.

Comments: title: Reviewer-Author discussion period
comment: We would like to thank the reviewer again for the time and efforts! We are more than happy to discuss our work in the rebuttal phase. In case you have any other questions and concerns, please don't hesitate to let us know.

Best regards

Author
Thanks for your response. I am not fully convinced by your reply. So I will not change my score at current stage. I need more time to recall the paper's details and verify your response. I will adjust my score based on your response and discussion with other reviewers.
Thank you for your positive feedback and support! We will add the theoretical guarantee for the fully fine-tuned method in the main content. For the explanation of the memorization phenomenon, we will add a detailed discussion in the future work paragraph according to your comments.
Thanks for your careful response. For W1, I agree the theory does show with full model fine-tuning, there will be a large error bound. However, a large error bound is not equivalent to the memorization phenomenon. Therefore, I think to support this part of the experiment, the author still needs some theoretical analysis on memorization. I am satisfied with the other answers. Overall, I think the work is valuable, I will increase the score to 6.
Dear Reviewers,

The deadline for the reviewer-author discussion period is approaching. If you haven't done so already, please review the rebuttal and provide your response at your earliest convenience.

Best wishes, AC
Thank you for your positive feedback and support!  We will add the discussion on the existing fine-tuning methods and our assumption according to your comments. In case you have any other questions, please don't hesitate to let us know.
Thank you for your response, and the additional discussion of your assumptions. I keep my rating as I think this is a very valuable work.