Title: Few-Shot Diffusion Models Escape the Curse of Dimensionality
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion models

Few-shot learning
Abstract: While diffusion models have demonstrated impressive performance, there is a growing need for generating samples tailored to specific user-defined concepts. The customized requirements promote the development of few-shot diffusion models, which use limited $n_{ta}$ target samples to fine-tune a pre-trained diffusion model trained on $n_s$ source samples. Despite the empirical success, no theoretical work specifically analyzes few-shot diffusion models. Moreover, the existing results for diffusion models without a fine-tuning phase can not explain why few-shot models generate great samples due to the curse of dimensionality. In this work, we analyze few-shot diffusion models under a linear structure distribution with a latent dimension $d$. From the approximation perspective, we prove that few-shot models have a $\widetilde{O}(n_s^{-2/d}+n_{ta}^{-1/2})$ bound to approximate the target score function, which is better than $n_{ta}^{-2/d}$ results. From the optimization perspective, we consider a latent Gaussian special case and prove that the optimization problem has a closed-form minimizer. This means few-shot models can directly obtain an approximated minimizer without a complex optimization process. Furthermore, we also provide the accuracy bound $\widetilde{O}(1/n_{ta}+1/\sqrt{n_s})$ for the empirical solution, which still has better dependence on $n_{ta}$ compared to $n_s$. The results of the real-world experiments also show that the models obtained by only fine-tuning the encoder and decoder specific to the target distribution can produce novel images with the target feature, which supports our theoretical results.

Few-Shot Diffusion Models Escape the Curse of Dimensionality

Abstract

While diffusion models have demonstrated impressive performance, there is a growing need for generating samples tailored to specific user-defined concepts. The customized requirements promote the development of few-shot diffusion models, which use limited n ta target samples to fine-tune a pre-trained diffusion model trained on n s source samples. Despite the empirical success, no theoretical work specifically analyzes few-shot diffusion models. Moreover, the existing results for diffusion models without a fine-tuning phase can not explain why fewshot models generate great samples due to the curse of dimensionality. In this work, we analyze few-shot diffusion models under a linear structure distribution with a latent dimension d . From the approximation perspective, we prove that few-shot models have a ˜ O ( n -2 /d s + n -1 / 2 ta ) bound to approximate the target score function, which is better than n -2 /d ta results. From the optimization perspective, we consider a latent Gaussian special case and prove that the optimization problem has a closed-form minimizer. This means few-shot models can directly obtain an approximated minimizer without a complex optimization process. Furthermore, we also provide the accuracy bound ˜ O (1 /n ta +1 / √ n s ) for the empirical solution, which still has better dependence on n ta compared to n s . The results of the realworld experiments also show that the models obtained by only fine-tuning the encoder and decoder specific to the target distribution can produce novel images with the target feature, which supports our theoretical results.

1 Introduction

In recent years, diffusion models have shown an excellent ability to generate diverse, high-quality samples and show state-of-the-art performance in many areas with large-scale, standard datasets (Rombach et al., 2022; Ho et al., 2022; Li et al., 2024a; Blattmann et al., 2023; Li et al., 2023b; Liu et al., 2024; Li et al., 2024b). However, users often desire to generate samples that resemble the ones they provide, such as images related to their families, daily lives, or specific items. These user-provided samples are typically limited in number and do not appear frequently in large-scale datasets. Consequently, training a diffusion model from scratch using such limited, personalized samples often results in poor performance. To cater the customized requirements of users, few-shot diffusion models attract much attention. Few-shot diffusion models aim to fine-tune a pre-trained diffusion model using a limited amount of data ( 5 ∼ 10 samples), and they have recently delivered

impressive results in various domains, including image generation (Ruiz et al., 2023; Han et al., 2023; Zhu et al., 2023), video generation (Chen et al., 2023b), and the medical domain (Dutt et al., 2023).

Before the fine-tuning phase, we first need to train a diffusion model on the large source dataset { X s,i } n s i =1 as the pre-trained model. A diffusion model consists of a forward process and a reverse process (Song et al., 2020). The forward process gradually converts the data distribution into Gaussian noise. The reverse process sequentially removes the noise in the data to generate samples, which relies on the gradient of logarithmic forward process density (a.k.a. score function). To run the reverse process, diffusion models use a neural network to approximate the unknown score function.

With a pre-trained diffusion model, the paradigm to obtain a few-shot diffusion model is to fine-tune the model using a limited target dataset { X ta,i } n ta i =1 . In earlier times, fully fine-tuned methods, such as DreamBooth (Ruiz et al., 2023), provided an important boost for developing few-shot models. However, they also show that the diffusion models suffer from the overfitting and memory phenomenon when fine-tuning all parameters. Furthermore, a fully fine-tuned method is both memory and time inefficient (Xiang et al., 2023). To avoid the above problems, many works freeze most parameters and fine-tune some key parameters, such as cross-attention layers (Kumari et al., 2023; Moon et al., 2022), some concept neurons (Liu et al., 2023) or text-embedding (Gal et al., 2022), to approximate the ground-truth target score function. These works not only preserve the prior information but also have a lower requirement for the target dataset size, which is more practical for applications. Hence, we aim to explain the great performance of these models in this work.

Despite the empirical success, no existing theoretical work specifically analyzes the approximation bound for few-shot diffusion models, and the following question remains open:

Do few-shot diffusion models with a fine-tuning phase enjoy a small approximation error with a limited target dataset?

For the approximation error bound, some works currently analyze diffusion models without a finetuning phase (Oko et al., 2023; Chen et al., 2023c; Yuan et al., 2023; Li et al., 2023c). Importantly, when analyzing general, bounded data, these works suffer from the curse of dimensionality. More specifically, Oko et al. (2023) analyze bounded distribution and show the n -s ′ /D s approximation bound, where D is the data dimension of X s . Chen et al. (2023c) analyze linear structure distribution X s = A s Z with subgaussian latent variable Z ∈ R d and achieve n -2 /d s results. Since the source dataset size is large enough, the influence of dimension is tolerable. However, for the limited target dataset, if trivially using the above technique, the bound is n -1 /D ta or n -2 /d ta , which is large and can not explain why few-shot diffusion models efficiently approximate the target score function.

In this work, for the first time, we propose the approximation bound specifically to few-shot diffusion models with a fine-tuning phase and prove that the few-shot diffusion model can escape the curse of dimensionality. More specifically, we show that when assuming (1) linear structure data and (2) the source and the target data share latent distribution, the few-shot diffusion models with a fine-tuning phase achieve ˜ O ( n -2 /d s + n -1 / 2 ta ) approximation error bound, which makes the first step to explain why few-shot diffusion models have great performance in the application. Generally speaking, due to the component n -1 / 2 ta , the few-shot diffusion only needs a few target samples to achieve the same bound compared to n -2 /d s . To support our augmentation, we calculate the requirement of n ta to obtain an accurate enough approximated target score function in popular datasets. Table 1 shows that the requirement of n ta is about 5 ∼ 10 samples, matching the customized diffusion model requirement. We also do experiments on the real-world datasets and show that 10 target images are enough for few-shot models to generate novel images with the target feature (see Section 6).

After directly using the property of the minimizer to obtain an approximation bound, we analyze how to optimize the few-shot diffusion models to obtain a minimizer. Since the score-matching objective function is highly non-convex, only a few works analyze the optimization problem of diffusion models (Shah et al., 2023; Bruno et al., 2023; Cui et al., 2023; Li et al., 2023c). Furthermore, these works either require (1) an exponential size neural network (Li et al., 2023c) or (2) a distribution determined by one variable (Shah et al., 2023; Bruno et al., 2023; Cui et al., 2023) to simplify the optimization problem. This work proves that few-shot diffusion models can simplify the optimization problem without these requirements. When analyzing the optimization problem, we focus on a

Gaussian latent variable special case 2 . Then, we prove that the expected few-shot objective function has a closed-form minimizer, which means the empirical solution can be directly obtained without a complex optimization process. We also prove the accuracy bound ˜ O (1 /n ta +1 / √ n s ) of empirical closed-form solution, which still has better dependence on the target dataset. In conclusion, we accomplish the following results for few-shot diffusion models under linear structure distribution:

For the approximation bound, we consider a subgaussian latent variable and prove ˜ O ( n -2 /d s + n -1 / 2 ta ) bound for few-shot models, which is better than n -2 /d ta result without fine-tuning.

For the optimization problem, we consider a latent Gaussian special case and prove that the expected few-shot objective function has a closed-form minimizer. Furthermore, we prove the accuracy bound ˜ O (1 /n ta +1 / √ n s ) for the empirical closed-form solution.

To support our theoretical results, we do real-world experiments and show that the models obtained by only fine-tuning specific encoder and decoder can use only 10 target images to generate novel images with the target feature.

2 Related Work

The approximation error bound. Recently, some works analyze the approximation error bound of diffusion models without a fine-tuning phase. Oko et al. (2023) analyze s ′ -order bounded derivatives distribution and show the approximation error bound is n -s ′ /D s . Chen et al. (2023c) analyze distribution with linear structure and subgaussian latent variable and show that the n -2 /d s result. The approximation error bound of the above works suffers the curse of (latent) dimensionality. To avoid this phenomenon, some works analyze special data distributions. Shah et al. (2023) and Cui et al. (2023) analyze the mixture of Gaussian (MOG) with known variance and achieve a 1 /n s approximation bound. Yuan et al. (2023) analyze linear structure distribution with Gaussian latent variable and achieve 1 / √ n s result. Mei & Wu (2023) analyze Ising models and prove that the term corresponds to n s is 1 / √ n s . However, the remaining terms do not converge to 0 when n s goes to + ∞ . For general bounded data distribution, Li et al. (2023c) provide a n -2 / 5 s approximation error bound. However, they use a 2-layer random feature network and only allow the second linear layer to be trainable. Hence, the network size is exp( n s ) compared to Poly ( n s ) size of all previous works.

The optimization of diffusion models. Since the score matching objective function is highly non-convex, only a few works analyze how to optimize it to obtain a minimizer (Shah et al., 2023; Cui et al., 2023; Bruno et al., 2023; Li et al., 2023c). These works either make assumptions about data distribution or network size to guarantee only one optimization variable, leading to a simpler optimization problem. For special data distributions, Bruno et al. (2023) and Cui et al. (2023) analyze a Gaussian with fixed variance and a 2-mode mixture of Gaussian (MOG) with equal, trainable mean and fixed variance, respectively. Shah et al. (2023) analyze a multi-mode MOG with a fixed variance and prove a local convergence guarantee. Since they assume the distance between any two modes is large enough and a good enough initialization, the optimization problem is similar to optimizing a Gaussian distribution. For the large neural network size, Li et al. (2023c) analyze a general, bounded distribution with a 2-layer NN. Note that they require exp( n s ) hidden neurons and only allow the linear layer to be trainable, which also leads the optimization problem to a convex optimization.

3 The Introduction of Few-shot Diffusion Models

With pre-trained models, the paradigm to obtain a few-shot diffusion model is to freeze most parameters and fine-tune some key parameters corresponding to the target data distribution. Since the analysis of few-shot diffusion models relies heavily on the pre-trained model, this section first provides a concise overview of the fundamental concepts and notations associated with diffusion models. Then, we introduce the paradigm of few-shot diffusion models in Section 3.2.

3.1 The Forward and Reverse Process

Let q 0 be the data distribution. Given X 0 ∼ q 0 ∈ R D , non-decreasing function f ( X t , t ) and g ( t ) , the forward process is defined by:

where { B t } t ∈ [0 ,T ] is a D -dimensional Brownian motion. In this work, we choose f ( X t , t ) = -1 / 2 X t and g ( t ) = 1 , which corresponds to variance preserving (VP) forward process and is widely used in practice 3 (Shah et al., 2023; Song et al., 2020). Let q t be the density function of X t . Once a forward process is chosen, the conditional distribution of X t | X 0 is q t ( X t | X 0 ) = N ( m t X 0 , σ 2 t I D ) , where m t = e -t/ 2 , σ 2 t = 1 -e -t . Note that when t goes to + ∞ , q t converges to N (0 , I D ) , which is helpful in choosing the initial distribution for the sampling process.

To generate samples, diffusion models reverse the forward SDE and run the reverse process. Since the reverse process contains the gradient of forward logarithmic density ∇ log q t ( · ) (a.k.a. score function), the model approximates it by using a neural network s ( · , t ) and the score matching objective function (see Section 3.2). After that, diffusion models discretize the continuous reverse process to obtain an implementable algorithm. Let t 0 ≤ t 1 ≤ · · · ≤ t K = T be the discretization points in the forward time and h k = t k -t k -1 be the k -th stepsize. When considering the reverse time, we define t ′ k = T -t K -k . In this work, we choose the exponential integrator (EI) discretization scheme, which has great performance (Zhang & Chen, 2022). The EI discretization freezes the approximated score at t ′ k and runs the following process in the reverse time:

where ̂ Y 0 ∼ N (0 , I D ) due to the stationary distribution of the forward process.

While the discretization complexity K has been well-studied with an accurate enough score function (Benton et al., 2023; Li et al., 2023a), there is a lack of analysis for the score-matching process. Therefore, this work focuses on the score approximation and the optimization problem of the few-shot score-matching objective function.

3.2 The Score Matching Objective Function

In this work, we specifically analyze few-shot diffusion models, which involve two datasets: (1) the source dataset { X s,i } n s i =1 ; (2) the target dataset { X ta,i } n ta i =1 . The approach involves first training a pre-trained diffusion model on the source dataset and then freezing the backbone network to fine-tune the diffusion models on the target dataset.

For data distributions, we assume that the source distribution q s and the target distribution q ta are both supported on a low-dimensional linear subspace. The low-dimensional structures have been discovered on many popular image datasets (Pope et al., 2021; Gong et al., 2019; Tenenbaum et al., 2000) due to the locally connected and symmetrical property, and it is crucial for diffusion models. For image generation, current popular diffusion models, such as Stable Diffusion (Rombach et al., 2022), transform images to a latent space and run diffusion models in the latent space. Except for the image generation, Chen et al. (2024) recently show the latent dimension plays an important role in diffusion models to work well in self-supervised learning, and linear subspace is enough.

We further assume that the source and target data share the same latent distribution. Note that this is a common assumption in few-shot learning. In particular, previous theoretical works in the context of supervised few-shot learning often assume that the source and target distributions have a common latent representation (Du et al., 2020; Chua et al., 2021; Meunier et al., 2023).

Assumption 3.1. The source datapoints X s and target datapoint X ta admit a low dimensional linear structure and share the same latent distribution X s = A s Z and X ta = A ta Z where A s , A ta ∈ R D × d with orthonormal columns and Z ∼ q z ∈ R d .

As mentioned in Chen et al. (2023c), when assuming linear distribution, the ground-truth score function is decomposed into the latent score function ∇ log q LD t ( Z ′ ) and linear encoder and decoder:

where q LD t ( Z ′ ) = ∫ q t ( Z ′ | Z ) q z ( Z )d Z and q t ( ·| Z ) = N ( m t Z, σ 2 t I d ) . This form indicates that the diffusion process happens in the latent subspace. A conceptual way to approximate the score function is to minimize the following loss on a function class S NN :

where w ( t ) is a weight function. However, the above objective function is intractable since ∇ log q t ( · ) is unknown. Vincent (2011) propose the following implementable loss:

Due to the forward process, ∇ log q s t ( X t | X 0 ) has an analytical form and is equal to -( X t -m t X 0 ) /σ 2 t . Vincent (2011) also prove that this objective function only has a constant difference compared to the above one. The empirical loss with the source datasets { X s,i } n s i =1 is defined by:

where and

Note that we take w ( t ) = 1 / ( T -δ ) for simplicity, where δ is the early stopping parameter to avoid the blow-up phenomenon of score functions at the end of reverse process. Furthermore, we take the integral over the forward time instead of discretizing the timeline since X t is easy to generate.

The linear encoder and decoder structure and the shortcut connection in S NN is due to the form of the ground-truth score function. The specific parameters for f θ , such as its length and width, are identical to those used in Chen et al. (2023c). Generally, with a given network accuracy parameters ϵ , the network size is Poly (1 /ϵ ) . We show the parameter of neural network in Appendix A.

The diffusion models minimize the empirical loss to obtain a pre-trained approximated score function. Let the minimizer of Equation (1) be ( ̂ V s , ̂ θ ) . Chen et al. (2023c) show that ( ̂ V s , ̂ θ ) leads a n -2 /d s approximation error bound. If trivially replacing n s with n ta , we obtain a n -2 /d ta bound for the target dataset without the fine-tuning phase. Note that this bound suffers from the influence of the latent dimension d , which is still large in popular datasets (Table 1). In the next paragraph, we introduce the few-shot diffusion models with a fine-tuning phase and show that the dependence on n ta is n -1 / 2 ta in the error bound (Theorem 4.3).

The Few shot Diffusion Models with a Fine-tuning Phase. Since the source and target distribution share the same latent data distribution, we freeze ˆ θ and only fine-tune the low-rank linear encoder and decoder layer V ta . This method can significantly reduce the fine-tuning parameters and is similar to LoRA (Hu et al., 2021), which also fine-tunes two low-rank matrices and is widely used in fine-tuning the stable diffusion (Rombach et al., 2022).

Let ℓ ta t be the loss function of the target dataset at time t , which has similar definition compared to ℓ s t . The optimization problem for the target dataset is

where

Similarly, we define the minimizer of the few-shot objective function as ( ̂ V ta , ̂ θ ) .

Notations. We denote by I D the D -dimensional identity matrix. For X ∈ R D and A ∈ R D × d , we denote by ∥ X ∥ 2 the Euclidean norm for vector and ∥ A ∥ F the Frobenius norm for matrix. We denote by ∥ X ∥ 2 L 2 ( q ) the expectation of X in L 2 norm E X ∼ q [ ∥ X ∥ 2 2 ] .

4 Few-shot Diffusion Models Enjoy Better Approximation Error Bound

In this section, we show that few-shot diffusion models with a fine-tuning phase escape the curse of latent dimensionality and have a ˜ O ( n -2 /d s + n -1 / 2 ta ) approximation bound 4 . This result makes the first step to explain why few-shot models have great performance with a limited target dataset.

Before showing our results, we first introduce standard assumptions on the latent distribution and the on-support ground-truth score function. We first assume that Z has a subgaussian tail and the minimum eigenvalue of Z is lower bound by c 0 , also used in Chen et al. (2023c).

Assumption 4.1. q z > 0 is twice continuously differentiable, λ min ( E [ ZZ ⊤ ] ) ≥ c 0 and E ∥ Z ∥ 2 2 ≤ C Z . Moreover, there exist positive constants B,C 1 , C 2 such that when ∥ Z ∥ 2 ≥ B , q z ( Z ) ≤ (2 π ) -d/ 2 C 1 exp ( -C 2 ∥ Z ∥ 2 2 / 2 ) .

Assumption 4.2. The on-support ground truth score A s ∇ log q LD t ( Z ) and A ta ∇ log q LD t ( Z ) is β -Lipschitz in Z ∈ R d for any t ∈ [0 , T ] .

Note that different from previous works directly assume ∇ log q t ( · ) is Lipschitz (Chen et al., 2022, 2023d), the β -Lipschitz on-support score function assumption does not conflict with the blow-up phenomenon when t goes to 0 due to the existence of ( I D -AA ⊤ ) X/σ 2 t . With these assumptions, we prove the approximation bound for few-shot models with a fine-tuning phase.

Theorem 4.3. Let α ( n ) = d log log n log n , F = ( d + C Z ) d 2 β 2 δ 2 c 0 and network parameter ϵ = n -1 / 2 ta . Assume Assumption 3.1, 4.1, 4.2 and n d +5 4(1 -α ( ns )) ta ≥ n s . Then, with probability 1 -δ 1 , the following inequality holds (hiding logarithmic factors)

The dependence of δ is due to the blow-up property of the score function. Note that when n s is sufficiently large, α ( n s ) is negligible. Then, the approximation error bound for few-shot diffusion models is ˜ O ( n -2 /d s + n -1 / 2 ta ) . Compared to the approximation error bound n -2 /d ta without a few-shot phase, it is clear that few-shot diffusion models escape the curse of the (latent) dimensionality.

Remark 4.4 (The discussion on the coefficient in Theorem 4.3) . The goal of the fine-tuning phase is to achieve the same order error bound compared with the pre-trained models, which means that n n n n the same order, we can only consider 1 / n ta and n s . To support the above augmentation, we calculate the coefficient of n s and n ta in detail. The dominated term of coefficient for n ta and n s are Dd 3 /δ and d 3 / ( δ 2 c 0 ) , respectively. The classic choice for the early stopping parameter δ and

we consider the relative relationship between ta and s . Hence, if the coefficient of ta and s has √ -2 /d

forward time T are 10 -3 and 10, respectively (Karras et al., 2022). Then, with D = 256 × 256 × 3 as an example 5 , Dd 3 /δ = d 3 × 20 × 10 6 and d 3 / ( δ 2 c 0 ) = d 3 × 10 6 /c 0 , which has the same order. Hence, we consider the relative relationship between 1 / √ n ta and n -2 /d s .

4.1 Discussion on the Approximation Bound

The relationship to empirical phenomenon. In applications, current few-shot diffusion models only require 5 ∼ 10 target images to achieve great performance. Theorem 4.3 makes the first step to explain why the few-shot diffusion models have great performance with a limited target n ta . More specifically, with known source dataset size n s and the corresponding latent dimension d , we can calculate the inequality n d +5 4(1 -α ( ns )) ta ≥ n s 6 to obtain the requirement of n ta to achieve the same accuracy compared to the pre-trained diffusion models. Combined with the latent dimension of popular datasets (Pope et al., 2021), Table 1 shows the requirement of n ta . It is clear that we only need less than 10 target images to obtain an accurate enough few-shot diffusion model that matches the performance in reality. The real-world experiments also support our discussion (Section 6).

Table 1 shows that the requirement of n ta is heavily influenced by the latent dimension d . When d is large (e.g. ImageNet), the approximation bound of pre-trained models is influenced by latent dimension and has a large approximation error even with large-size source data. We only need a few target data to achieve the same error in this setting. When d is small (e.g. CIFAR-10), pre-trained models have a small approximation error. We need a slightly larger target data size.

The approximation error of the fully fine-tuned method. As shown in our real experiment Section 6 and DreamBooth (Ruiz et al., 2023), when fine-tuning all parameters with a small target dataset, models tend to overfit and lose the prior information from the pre-trained model. In our theorem, this phenomenon means that in the fine-tuning phase, the model does not use ̂ θ learned by the pre-trained model and achieves a n -2 /d ta approximation error bound, which suffers from the curse of dimensionality. From an intuitive perspective, the probability density function (PDF) of a distribution learned by an overfitting model is only positive at the interval around the target dataset, which is far away from the PDF of true distribution and leads to a large error term. We also note that it is possible to avoid this phenomenon by using a specific loss (Ruiz et al., 2023) or carefully choosing the optimization epochs (Li et al., 2023c). We leave them as interesting future works.

Proof sketch. The first step is to prove that in Q NN ( ̂ θ ) , there exists a solution ( ¯ V ta , ̂ θ ) has the following inequality (only focusing on n s and n ta )

To prove the above inequality, we first do the following decomposition:

where ( ¯ V ta , ¯ θ ) ∈ S NN is a constructed solution. The first term is due to the accuracy of the constructive neural network with network accuracy parameter ϵ . For the second term, since the latent score function is shared and few-shot diffusion models directly use ̂ θ , it is control by the approximation bound of the pre-trained diffusion models. Then, by using the inequality

we build the bridge between s ̂ V ta , ̂ θ and s ¯ V ta , ̂ θ .

The second step is using the concentration to control the error between empirical ̂ L ta and expected L ta . Roughly speaking, we have that

where N (1 /n ta , Q NN ( ̂ θ ) , ∥ · ∥ 2 ) is the covering number of Q NN ( ̂ θ ) in L 2 norm. Since only V ∈ R D × d can be optimized and ̂ θ is fixed in Q NN ( ̂ θ ) ,

Then, we balance different terms and achieve the final bound by choosing ϵ 2 = 1 / √ n ta .

5 The Few-shot Diffusion Model Have a Closed-form Minimizer

This section focuses on how to optimize the few-shot diffusion model. When considering the optimization problem, we assume the shared latent distribution admits an isotropic Gaussian distribution q z = N (0 , λ 2 I d ) with λ 2 > 0 , which indicates the score function has the following formulation:

Note that though q z = N (0 , λ 2 I d ) is a special case of Assumption 4.1, we still need to know λ 2 and A ta to generate samples, which indicates the previous optimization analysis for diffusion models without a fine-tuning phase can not be used.

We fix a t ∈ [ δ, T ] for the few-shot objective function since the matrix A ta is independent of time t . More specifically, with an approximated latent distribution ̂ q z = N (0 , ̂ Σ) , where ̂ Σ = ̂ λ 2 I d , the expected few-shot objective function at a fixed time t is

where

In this case, f ̂ Σ ( Z, t ) = ( I d -σ 2 t ̂ Σ -1 t ) Z , where ̂ Σ t = m 2 t ̂ Σ + σ 2 t I d . The constraint rank( V ) = d is used to guarantee that the few-shot diffusion models learn meaningful subspace instead of 0 D × d . Note that rank( V ) = d is a weaker constraint than V ⊤ V = I d since the pre-trained diffusion model has already learned the length information. This weaker constraint means we need less prior knowledge compared to Q ( θ ) , which is more user-friendly. Let ˜ V ta be a minimizer of the above expected few-shot objective function. We show that ˜ V ta has a closed form and good property.

Lemma 5.1. Assume Assumption 3.1 and q z = N (0 , λ 2 I d ) . Let C = E X ta ∼ q ta [ X ta X ⊤ ta ] be the expected data covariance matrix. Then, ˜ V ta has a closed form:

Lemma 5.1 indicates that few-shot diffusion models can directly obtain an approximation of the minimizer without a complex optimization process. Furthermore, this minimizer has good properties and exactly recovers the subspace spanned by A ta . More specifically, the expected minimizer indicates ∥ ˜ V ta ˜ V ⊤ ta -A ta A ⊤ ta ∥ 2 F = 0 when n s and n ta are infinite. However, the source datasets n s and target datasets n ta are finite, we analyze the empirical closed-form solution

where ¯ C = 1 n ta ∑ n ta i =1 X ta,i X ⊤ ta,i is the empirical covariance matrix.

Theorem 5.2. Assume Assumption 3.1 and q z = N (0 , λ 2 I d ) . Let ̂ q z be the latent distribution generated by the pre-trained models with ( ̂ V ta , ̂ Σ) and M = d 2 β 2 ( d + λ 2 ) λ √ Dd log ( Ddn s ) ( d 2 ∨ D ) . Then, with probability 1 -δ 1 , we have that for any t ∈ [ δ, T ]

ina

(a)Target dataset

(b) The results of fine-tuning all parameters

The above result indicates that the few-shot diffusion models can still recover the true subspace with finite n s and n ta . Note that when the latent distribution is Gaussian distribution, the approximation error bound for the source dataset is n -1 / 2 s instead of n -2 /d s (Yuan et al., 2023). Hence, n s in Theorem 5.2 do not depend on latent dimension d .

Remark 5.3 . The bound of ∥ V V T -AA T ∥ 2 F only guarantees the subspace spanned by V and A is close, which still holds after an orthogonal transformation on V . Hence, this bound does not indicate ∥ V -A ∥ 2 F is small. Since all previous works (Chen et al., 2023c; Yuan et al., 2023) consider ∥ V V T -AA T ∥ 2 F , we also use this metric to measure the subspace recovery. However, our results are stronger due to the closed-form solution, where previous works do not consider how to obtain V V T .

5.1 Discussion on the Closed-form Minimizer

Better dependence on n ta , δ and d . Note that Theorem 5.2 has better 1 /n ta dependence on the target dataset compared to 1 / √ n s dependence on the source dataset. Furthermore, the coefficient of n s term is dependent on the early stopping parameter δ and D . This is due to the δ and D dependence of the approximation bound, which is used in generating ̂ q z . However, the n ta term only has d dependence. Hence, even in the latent Gaussian setting, we still need a larger source dataset than the target dataset to obtain a sufficiently accurate closed-form solution.

The relationship with principal component analysis (PCA). The expected few-shot score matching objective can be simplified to

where ̂ G t = I d -σ 2 t ̂ Σ -1 t . Note that when ignoring 1 /σ 4 t and choosing t = 0 , the above minimization problem is similar to PCA. This suggests that few-shot models implicitly optimize an objective function akin to PCA. However, few-shot models extend beyond traditional PCA. More specifically, when λ 2 is large, classical PCA suffers from the influence of λ 2 . In contrast, due to ( m 2 t λ 2 + σ 2 t ) /n ta term, few-shot models can select a large t to mitigate the impact of λ 2 and achieve a 1 /n ta .

6 Experiments

To corroborate our theoretical findings, we conducted experiments utilizing real-world datasets. These experiments show that the new model obtained by only fine-tuning appropriate encoder and decoder layers on target datasets can produce novel images with the target feature, which shows the effectiveness of the methods and supports our theoretical results.

Datasets and benchmark. Note that human face images tend to exhibit similarity in their latent space, primarily due to shared facial features, while differing in specific features. Hence, we initially pre-train a model using the CelebA64 dataset, focusing on distinct hairstyle features as the goal for the fine-tuning phase. For the source data, we construct a large dataset ( 6400 images) with different hairstyles (without the bald feature). For the target data, we choose the bald feature as the target feature and select 10 images with this feature to constitute the target dataset, which are much smaller than the size of target dataset (Figure 1 (a)). To show the effectiveness of our methods, we also fine-tune all parameters of the pre-trained models as the benchmark.

Discussion. As shown in Figure 1, the results obtained by only fine-tuning the encoder and decoder layers can generate novel face images with the bald feature. Conversely, when fine-tuning all parameters, the models suffer from memory phenomenon and can only generate images that slightly

modify the brightness and angle of the target dataset. This phenomenon indicates that only fine-tuning the appropriate encoder and decoder will result in a model with a generalization property.

We note that these experiments aim to verify the effectiveness of the methods instead of achieving state-of-the-art performance since previous works carefully select specific parameters, such as specific cross-attention layers (Kumari et al., 2023) or special neurons (Liu et al., 2023), to fine-tune pretrained models. However, we simply fine-tune all encoder and decoder layers simultaneously. There are more experiments on cat faces and more discussion on why Assumption 3.1 is satisfied in our experiments. We refer to Appendix E for more details.

7 Conclusion

This work aims to provide a deeper understanding of few-shot diffusion models from a theoretical perspective. Our analysis is conducted from two key perspectives: the approximation and optimization aspects, all under linear structure distribution and shared latent space assumptions.

From the approximation error bound, we consider general subgaussian latent variable and prove that few-shot models have a ˜ O ( n -2 /d s + n -1 / 2 ta ) approximation bound, which is better than n -2 /d ta results of diffusion models without a fine-tuning phase and escape the curse of dimensionality. This result also makes the first step to explain why few-shot diffusion models only require 5 ∼ 10 images to generate great samples. The experiments on the real-world dataset also show that the fine-tuning phase only requires 10 images to generate novel images with the target feature.

When analyzing the optimization process, we consider a more special, shared Gaussian latent variable and prove that the expected score matching has a closed-form minimizer, which indicates that the few-shot diffusion models can simplify the optimization problem. Furthermore, we prove that the empirical closed-form solution has a ˜ O ( 1 /n ta +1 / ( δ √ n s )) accuracy bound, which still has better 1 /n ta target data dependence compared to 1 / ( δ √ n s ) dependence on the source data.

Future work and limitation. When considering the approximation bound, we assume a distribution with a linear structure. Though it has been supported by much empirical evidence, it is not as general as bounded distribution. After that, we plan to consider a general, bounded distribution and show the advantage of few-shot diffusion models. One possible way is to analyze the mixture of low-rank Gaussian (Wang et al., 2024), which is more general than the linear subspace assumption.

We focus on a special Gaussian latent distribution when considering the optimization problem. As a next step, we plan to consider a more general latent distribution, such as a log-concave distribution. In this setting, we can not directly obtain the closed-form solution. However, due to the shared information and simplified landscape, it is still possible to use some optimization algorithms, such as gradient descent, to optimize the few-shot objective function to achieve the convergence guarantee.

Broader Impact. This paper presents work whose goal is to understand few-shot diffusion models from the theoretical perspective. A noteworthy societal impact is that few-shot diffusion models may be used to imitate the style of artists and generate fake images, thereby infringing on the rights of artists (Mirsky & Lee, 2021). We recommend adding watermarks to images to determine whether the image was generated by a generative model (Fernandez et al., 2023). The other societal impact is the same as general generative models (Mishkin et al., 2022).

Acknowledgments and Disclosure of Funding

The author Bo Jiang is supported by National Natural Science Foundation of China (62072302). The author Baoxiang Wang is partially supported by the National Natural Science Foundation of China (62106213, 72394361), an extended support project from the Shenzhen Science and Technology Program, and support from Longgang District Key Laboratory of Intelligent Digital Economy Security.

References

Benton, J., De Bortoli, V ., Doucet, A., and Deligiannidis, G. Linear convergence bounds for diffusion models via stochastic localization. arXiv preprint arXiv:2308.03686 , 2023.

Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V ., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Bruno, S., Zhang, Y., Lim, D.-Y., Akyildiz, Ö. D., and Sabanis, S. On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates. arXiv preprint arXiv:2311.13584 , 2023.

Chen, H., Lee, H., and Lu, J. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning , pp. 4735-4763. PMLR, 2023a.

Chen, H., Wang, X., Zeng, G., Zhang, Y., Zhou, Y., Han, F., and Zhu, W. Videodreamer: Customized multi-subject text-to-video generation with disen-mix finetuning. arXiv preprint arXiv:2311.00990 , 2023b.

Chen, M., Huang, K., Zhao, T., and Wang, M. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.07194 , 2023c.

Chen, S., Chewi, S., Li, J., Li, Y ., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 , 2022.

Chen, S., Daras, G., and Dimakis, A. G. Restoration-degradation beyond linear diffusions: A non-asymptotic analysis for ddim-type samplers. arXiv preprint arXiv:2303.03384 , 2023d.

Chen, X., Liu, Z., Xie, S., and He, K. Deconstructing denoising diffusion models for self-supervised learning, 2024.

Chua, K., Lei, Q., and Lee, J. D. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems , 34:8871-8884, 2021.

Cui, H., Krzakala, F., Vanden-Eijnden, E., and Zdeborová, L. Analysis of learning a flow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.03575 , 2023.

Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., and Lei, Q. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434 , 2020.

Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., and Hospedales, T. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity. arXiv preprint arXiv:2305.08252 , 2023.

Fernandez, P., Couairon, G., Jégou, H., Douze, M., and Furon, T. The stable signature: Rooting watermarks in latent diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 22466-22477, 2023.

Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Gong, S., Boddeti, V. N., and Jain, A. K. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3987-3996, 2019.

Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., and Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11305 , 2023.

Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 4401-4410, 2019.

Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1931-1941, 2023.

Li, G., Wei, Y., Chen, Y., and Chi, Y . Towards faster non-asymptotic convergence for diffusion-based generative models. arXiv preprint arXiv:2306.09251 , 2023a.

Li, H., Ma, L., Liao, Y., Cheng, L., Hao, Y., and Zhou, P. 3d-goi: 3d gan omni-inversion for multifaceted and multi-object editing. arXiv preprint arXiv:2311.12050 , 2023b.

Li, H., Shi, H., Zhang, W., Wu, W., Liao, Y., Wang, L., Lee, L.-h., and Zhou, P. Dreamscene: 3d gaussian-based text-to-3d scene generation via formation pattern sampling. arXiv preprint arXiv:2404.03575 , 2024a.

Li, H., Tian, Y ., Liao, Y ., Wang, L., Wang, Y ., and Zhou, P. Y . Enhanced 3d generation by 2d editing. arXiv preprint arXiv:2412.05929 , 2024b.

Li, P., Li, Z., Zhang, H., and Bian, J. On the generalization properties of diffusion models. arXiv preprint arXiv:2311.01797 , 2023c.

Liu, M., Shi, R., Chen, L., Zhang, Z., Xu, C., Wei, X., Chen, H., Zeng, C., Gu, J., and Su, H. One-23-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10072-10083, 2024.

Liu, Z., Feng, R., Zhu, K., Zhang, Y., Zheng, K., Liu, Y., Zhao, D., Zhou, J., and Cao, Y. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125 , 2023.

Mei, S. and Wu, Y. Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models. arXiv preprint arXiv:2309.11420 , 2023.

Meunier, D., Li, Z., Gretton, A., and Kpotufe, S. Nonlinear meta-learning can guarantee faster rates. arXiv preprint arXiv:2307.10870 , 2023.

Mirsky, Y. and Lee, W. The creation and detection of deepfakes: A survey. ACM computing surveys (CSUR) , 54(1):1-41, 2021.

Mishkin, P., Ahmad, L., Brundage, M., Krueger, G., and Sastry, G. Dall· e 2 preview-risks and limitations. Noudettu , 28:2022, 2022.

Moon, T., Choi, M., Lee, G., Ha, J.-W., and Lee, J. Fine-tuning diffusion models with limited data. In NeurIPS 2022 Workshop on Score-Based Methods , 2022.

Oko, K., Akiyama, S., and Suzuki, T. Diffusion models are minimax optimal distribution estimators. arXiv preprint arXiv:2303.01861 , 2023.

Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894 , 2021.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10684-10695, 2022.

Ruiz, N., Li, Y., Jampani, V., Pritch, Y ., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 22500-22510, 2023.

Shah, K., Chen, S., and Klivans, A. Learning mixtures of gaussians using the ddpm objective. arXiv preprint arXiv:2307.01178 , 2023.

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Tenenbaum, J. B., Silva, V. d., and Langford, J. C. A global geometric framework for nonlinear dimensionality reduction. science , 290(5500):2319-2323, 2000.

Vincent, P. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Wang, P., Zhang, H., Zhang, Z., Chen, S., Ma, Y., and Qu, Q. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426 , 2024.

Xiang, C., Bao, F., Li, C., Su, H., and Zhu, J. A closer look at parameter-efficient tuning in diffusion models. arXiv preprint arXiv:2303.18181 , 2023.

Yuan, H., Huang, K., Ni, C., Chen, M., and Wang, M. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. arXiv preprint arXiv:2307.07055 , 2023.

Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 , 2022.

Zhu, J., Ma, H., Chen, J., and Yuan, J. Domainstudio: Fine-tuning diffusion models for domain-driven image generation using limited data. arXiv preprint arXiv:2306.14153 , 2023.

Appendix

A The Neural Network Structure

In this section, we introduce the multi-layer ReLU network f θ ∈ NN( L, M ′ , J, K 1 , κ, γ, γ t ) in S NN . We note that the following setting is exactly the same as the one in Chen et al. (2023c), and we show the structure for completeness. We denote by NN( L, M ′ , J, K 1 , κ, γ, γ t ) the following neural network:

where σ is the ReLU activation function. Given an network accuracy ϵ > 0 , the parameters is defined by

where τ = sup A s ta ∥ z ∥ ∞ ≤ d log δϵ ∂t 2 .

B The Proof of the Approximation Error Bound

Let ( ̂ V s , ̂ θ ) be the minimizer of the pre-trained objective function. The few-shot diffusion model freezes the bottleneck network and fine-tunes V ta ∈ R D × d to obtain the minimizer ( ̂ V ta , ̂ θ ) of the few-shot objective function. As the first step, we show that with the bottleneck parameterized by ̂ θ , there also exists a solution ( ¯ V ta , ̂ θ ) ∈ Q NN ( ̂ θ ) achieve the ϵ 2 + n -2 /d s error bound.

Lemma B.1. If ϵ ≤ n -1 -α ( ns ) d +5 s , where α ( n ) = d log log n log n , then with probability 1 -δ 1 , there exists a solution ( ¯ V ta , ˆ θ ) ∈ Q NN ( ˆ θ ) such that

Proof . As shown in Theorem 1 of Chen et al. (2023c), there exists a solution ( ¯ V s , ¯ θ ) in S NN such that

Hence, we do the following decomposition:

For the encoder and decoder layer, we choose ¯ V ta = A ta . The first term is bounded due to the construction of the neural network. We first show that s ¯ V ta , ¯ θ is ϵ -close to the true score function ∇ log q ta t . Since the encoder and decoder have been chosen, we only need to focus on the latent bottleneck. For the latent bottleneck, we need to use f θ ( Z, t ) to approximate ground-truth function h ( Z, t ) = σ 2 t ∇ log q LD t ( Z ) + Z for Z ∈ R d . Chen et al. (2023c) show that for any latent variable Z ′ ∈ R d with subgaussian tail, we have that

Then, we have that

For the second term, we know that with probability 1 -δ 1 :

d log log n is a matrix with orthonormal columns.

where α ( n ) = log n . The first inequality follows A ta Since we assume ϵ ≤ n -1 -α ( ns ) d +5 s , the network has good enough ability to obtain an accurate enough ˆ θ . Hence, we can use Appendix C.4 of Chen et al. (2023c) to obtain the second inequality. Since we directly use the true matrix ¯ V ta = A ta instead of the approximate ˆ V ta , we do not need orthogonal transformation and can choose U = I d in the Appendix C.4 of Chen et al. (2023c). Then, we complete our proof. ■

To prove Theorem 4.3, we need to do the following decomposition for the population loss of the target datasets

where a ∈ (0 , 1) and L trunc ta is defined as

In this section, we take R = O (√ d log d +log K 1 +log n ta δ 1 ) to guarantee P X ta,i ∼ q ta ( ∥ X ta,i ∥ 2 ≤ R for all i = 1 , . . . , n ta ) ≥ 1 -δ 1 , where K 1 is defined in Appendix A.

Term (a). Similar to Chen et al. (2023c), we define a function class G ( ˆ θ ) = { ℓ trunc ta ( · ; s V, ˆ θ ) : s V, ˆ θ ∈ Q NN ( ˆ θ ) } , which is induced by Q ( ̂ θ ) . For the upper bound of G ( ˆ θ ) , we directly use the augmentation of Chen et al. (2023c) to obtain that

Then, by using Lemma D.1, we know that with probability 1 -δ 1 , term (a) is bounded by

To bound the above term, we need to calculate the covering number of G ( ˆ θ ) , which is related to a ι -covering of Q NN ( ˆ θ ) . Suppose that given s V 1 , ˆ θ and s V 2 , ˆ θ with sup ∥ x ∥ 2 ≤ 3 R + √ D log D,t ∈ [ δ,T ] ∥ ∥ ∥ s V 1 , ˆ θ ( x, t ) -s V 2 , ˆ θ ( x, t ) ∥ ∥ ∥ 2 ≤ ι , we need to bound .

By using the same calculation compared to Term (A) of Chen et al. (2023c), we know that

The above inequality indicates that a ι -covering of Q NN ( ˆ θ ) in L 2 norm leads a ι T -δ ( K 1 + R ) log T δ + 4 K 1 ( K 1 + R ) δ ( T -δ ) ( R/D ) D -2 exp ( -1 σ 2 t R 2 ) -covering of of G ( ˆ θ ) in L ∞ norm.

By taking R = O (√ d log d +log K 1 +log n ta δ 1 ) , K 1 = O ( 2 d 2 log ( d δϵ )) , ι = 2 n ta δ ( T -δ ) , we know that

which indicates with probability 1 -δ 1 , term (a) is bounded by

After that, we need to determine the covering number of Q NN ( ˆ θ ) with a truncated X to bound term (a).

Lemma B.2. The logarithmic covering number of Q NN ( θ ) for ∥ X ∥ 2 ≤ 3 R + √ D log D,t ∈ [ δ, T ] is

Proof . Suppose that there exists two orthonormal column matrix V 1 , V 2 such that ∥ V 1 -V 2 ∥ F ≤ δ 2 , then we have

For set { V ∈ R D × d : ∥ V ∥ 2 ≤ 1 } , the δ 2 -covering number is ( 1 + 2 √ d δ 2 ) Dd . Then we know that

Term (b). For the term (b), the proof of Theorem 2 in Chen et al. (2023c) shows that

Term (c). For the term (c), we know that it is bounded by the constructed solution ( ¯ V ta , ˆ θ ) :

For the term (C.1), since s ¯ V ta , ˆ θ is a fixed function, we directly use the results of (Chen et al., 2023c):

with probability 1 -δ 1 . For the term (C.2), we know that

As we show in Section 3.2, the two terms in E are both the score matching objective function and have a constant different E , which is independent of the trainable parameters ( V, θ ) . We denote by this difference E and F = ( d + C Z ) d 2 β 2 δ 2 c 0 . With probability 1 -δ 1 , Lemma B.1 shows that term (C.2) is bounded by

After bounding these three terms, we prove Theorem 4.3.

Theorem 4.3. Let α ( n ) = d log log n log n , F = ( d + C Z ) d 2 β 2 δ 2 c 0 and network parameter ϵ = n -1 / 2 ta . Assume Assumption 3.1, 4.1, 4.2 and n d +5 4(1 -α ( ns )) ta ≥ n s . Then, with probability 1 -δ 1 , the following inequality holds (hiding logarithmic factors)

Proof . Equipped with the bound of the term (a), (b), and (c) and hiding the logarithmic term (except for the covering number term), with probability 1 -δ 1 , we have that

Since L ta ( s ˆ V ta , ˆ θ ) -E = 1 T -δ ∫ T δ ∥ ∥ ∥ s ̂ V ta , ˆ θ ( · , t ) -∇ log q ta t ( · ) ∥ ∥ ∥ 2 L 2 ( q t ) , we have that the following inequality when choosing a = ϵ 2 :

where the second inequality follows the convering number of Q NN ( ˆ θ ) for ∥ X ∥ ≤ 3 R + √ D log D with R = O (√ d log d +log K 1 +log n ta δ 1 ) and the network parameters is defined in Appendix A. Finally, we choose ϵ 2 = 1 / √ n ta , then we have that

As we require in Lemma B.1, we need ϵ = 1 /n 1 / 4 ta ≤ n -1 -α ( ns ) d +5 s , which indicates n d +5 4(1 -α ( ns )) ta ≥ n s . ■

C The Proof of the Optimization Problem

C.1 The Pre-trained Diffusion Model Generate Accurate Enough Latent Distribution

Since we need to use the approximated latent distribution in the few-shot phase, we show that the pre-trained diffusion models with solution ( ̂ V s , ̂ θ ) can generate an accurate enough latent distribution. As shown in Section 5, when considering the optimization perspective of diffusion models, we assume the latent distribution is a Gaussian distribution q z = N (0 , Σ) with Σ = diag ( λ 2 1 , . . . , λ 2 d ) ≻ 0 . Yuan et al. (2023) show that in the setting, the approximation error bound (Lemma D.3) for the target dataset is

To generate latent distribution, we first introduce the reverse process in the latent space. The introduction mainly follows the outline of Appendix C.2 of Chen et al. (2023c). For X t , we can do the following decomposition: X t = A s Z t + X t, ⊥ , where Z t = A ⊤ s X t . With Z ← t = Z T -t , the reverse process in the latent space is

As shown in Theorem 3 of Chen et al. (2023c), the solution ( ̂ V s , ̂ θ ) of the pre-trained diffusion models only guarantee ∥ ̂ V s ̂ V ⊤ s -A s A ⊤ s ∥ 2 F is small instead of ∥ ̂ V s -A s ∥ 2 F is small. Hence, Theorem 3 of Chen et al. (2023c) assume there exists an orthogonal matrix U s ∈ R d × d and do an orthogonal transformation on ̂ V s to obtain ̂ V s U s , which can guarantee ∥ ̂ V s U s -A s ∥ 2 F is small. After such orthogonal transformation, the reverse process with an approximated score function and an approximated reversing beginning distribution ˜ Z ← ,r 0 ∼ N (0 , I d ) is

where

Then, we discretize the above process with the exponential integrator (EI) discretization scheme (Zhang & Chen, 2022) to obtain an implementable algorithm:

As shown in Appendix C.4 of Chen et al. (2023c), if the target ground truth score function has a L 2 -accurate approximated score:

the latent score function also has an L 2 norm bound ϵ 2 latent-score , which is determined by ϵ :

The remaining term is to determined ϵ . Since we assume Gaussian latent variable instead of subGaussian one. Hence, we do not use ϵ in Chen et al. (2023c). We use ϵ in Yuan et al. (2023) (Theorem 4.5 of Yuan et al. (2023)), which also considers Gaussian latent variable, to achieve the final results.

Finally, we have that with probability 1 -δ 1 :

Let p LD t be the distribution of the algorithm (the above discretization process). In the following lemma, we adopt Theorem 5 of Chen et al. (2023a) and show that the pre-trained diffusion model can obtain an accurate enough latent distribution with the above L 2 -accurate latent score function.

Lemma C.1. With ϵ 2 latent-score defined in Equation (4) , T = log ( λ 2 max + d ϵ 2 latent-score ) and K = Θ ( d 2 ( T +log( λ 2 max )) 2 ϵ 2 latent-score ) , by using the exponentially decreasing (then the constant) stepsize h k = c min { max { t k , 1 λ 2 max } , 1 } , c = log( λ 2 max )+ T K , the results p LD T of sampling algorithm (Equation (3) ) has the following guarantee with probability 1 -δ 1 (hiding the logarithmic factor):

Proof . The Theorem 5 of Chen et al. (2023a) show that if ∇ log q LD 0 is L -Lipschitz, diffusion models with a L 2 -accurate can generate ̂ p LD T , which is close to q 0 in KL divergence. Since q z = N (0 , Σ) , it is easy to verify L = λ 2 max . Then, we complete the proof. ■

C.2 The Closed-form Minimizer of Few-shot Diffusion Models

When the latent distribution is a Gaussian distribution q z = N (0 , Σ) with Σ = diag ( λ 2 1 , . . . , λ 2 d ) ≻ 0 , the ground truth score function for the target dataset is

where Σ t = diag ( . . . , m 2 t λ 2 k + σ 2 t , . . . ) . Since the matrix A ta is independent of time t , we fix a t ∈ [ δ, T ] and minimize the few-shot objective function at this time. With an approximated ̂ Σ , which is learned by the pre-trained diffusion models, f ˆ θ ( Z, t ) = ( I d -σ 2 t ̂ Σ -1 t ) Z , where ̂ Σ t = m 2 t ̂ Σ+ σ 2 t I d . Hence, the expected objective function for the few-shot diffusion models at a fixed time t is:

where

Note that the constraint rank( V ) = d is a weaker constraint than V ⊤ V = I d since rank( V ) = d does not involve length information.

Lemma C.2. Assume Assumption 3.1 and q z = N (0 , λ 2 I d ) . Let C = E X ta ∼ q ta [ X ta X ⊤ ta ] be the expected data covariance matrix. Then, ˜ V ta has a closed form:

Proof . Let ̂ G t = I d -σ 2 t ̂ Σ -1 t , then we have that

where the second equality follows ∇ log q ta t ( X t | X 0 ) = -X t -m t X 0 σ 2 t . Let C = E X ta ∼ q ta [ X ta X ⊤ ta ] be the expected covariance matrix of the target dataset. With the fact E X t | X 0 [ X t X ⊤ t ] = m 2 t X 0 X ⊤ 0 + σ 2 t I D and E X t | X 0 [ X 0 X ⊤ t ] = m t X 0 X ⊤ 0 , the optimization problem can be simplified to the following form (without misunderstanding, we ignore the subscript ta ):

where ( m 2 t C + σ 2 t I D ) 1 2 is meaningful since ( m 2 t C + σ 2 t I D ) is positive-definite matrix. Let ˜ V be the solution of the above minimization problem. We first ignore the constraint rank( V ) = d and calculate ∂ L ( V ) /∂V = 0 (since ˜ V also satisfied ∂ L ( V ) /∂V | V = ˜ V = 0 ), we know that ˜ V satisfies the following equality:

The above equality means rank(( m 2 t C + σ 2 t I D ) V ̂ G t V ⊤ -m 2 t C ) + rank( V ̂ G t ) ≤ d . Since rank( V ) = d and rank( ̂ G t ) = d , then we have that rank( V ̂ G t ) = d and

In Section 5, we assume the latent distribution is a isotropic Gaussian distribution q z = N (0 , λ 2 I d ) . In this setting, ̂ Σ is equal to ̂ λ 2 I d and ̂ G t = m 2 t ̂ λ 2 m 2 t ̂ λ 2 + σ 2 t , which indicate the closed form solution of ˜ V is

The last step is to prove rank( ˜ V ˜ V ⊤ ) = d . Note that rank( C ) = rank ( E X ta ∼ q ta [ X ta X ⊤ ta ]) = rank( A Σ A ⊤ ) = d , which indicates

Combined with ˜ V ∈ R D × d , we complete the proof.

which indicate

■

C.3 The Error Bound for the empirical Closed-form Solution

In this part, we prove the accuracy bound of the empirical version closed form solution ¯ ˜ V ¯ ˜ V ⊤ w.r.t. n s and n ta . The empirical solution has the following form (without misunderstanding, we ignore the subscript ta ):

where ¯ C = 1 n ta ∑ n ta i =1 X ta,i X ⊤ ta,i .

Theorem 5.2. Assume Assumption 3.1 and q z = N (0 , λ 2 I d ) . Let ̂ q z be the latent distribution generated by the pre-trained models with ( ̂ V ta , ̂ Σ) and M = d 2 β 2 ( d + λ 2 ) λ √ Dd log ( Ddn s ) ( d 2 ∨ D ) . Then, with probability 1 -δ 1 , we have that for any t ∈ [ δ, T ]

Proof . The empirical solution indicates that

To analyze this equality, we first show that ̂ λ 2 is accurate enough. We know that KL ( q 0 ∥ ˆ p LD T ) = d ( λ 2 / ̂ λ 2 -log( λ 2 / ̂ λ 2 ) -1 ) . Let M 1 = d 2 β 2 ( d + λ 2 max ) λ min δ √ ( d 2 + Dd ) log ( Ddn s ) ( d 2 ∨ D ) . Then, Lemma C.1 show that with probability 1 -δ 1 , we have

Combined with the above inequality, we know that with probability 1 -δ 1 , | λ 2 / ̂ λ 2 -1 | ≤ √ M 1 √ log(1 /δ 1 ) d √ n s by using the Taylor Expansion. By using Lemma D.2, we know that with probability 1 -δ 1 :

For the left hand of Equation (6), we know that

Combined with Equation (7) and Equation (8), we have that

ing to symmetry, we know that

The last inequality follows that each element of ¯ ˜ V ¯ ˜ V ⊤ is bounded by some constant due to the form of the empirical closed form solution and

For the right hand of the above inequality, we know that

Then, we complete the proof

D Auxiliary Lemmas

The following concentration lemma comes from Lemma 15 of Chen et al. (2023c).

Lemma D.1 (Lemma 15, (Chen et al., 2023c)) . Let G be a bounded function class, i.e., there exists a constant B such that any g ∈ G : R d ↦→ [0 , B ] . Let z 1 , . . . , z n ∈ R d be i.i.d. random variables. For any δ ∈ (0 , 1) , a ≤ 1 , and τ > 0 , we have

In the following lemma, we show the concentration of the data covariance matrix. Note that the proof sketch of the following lemma mainly follows Lemma 6 of (Du et al., 2020). We prove a concentration bound that depends on n instead of a constant bound with a large enough n .

Lemma D.2 (The Modified Lemma A.6, (Du et al., 2020)) . Let a 1 , . . . , a n be i.i.d. d -dimensional random vectors such that E [ a i ] = 0 , E [ a i a ⊤ i ] = I , and a i is ρ 2 -subgaussian. Then with probability at least 1 -δ 1 we have

Proof . Let A = 1 n ∑ n i =1 a i a ⊤ i -I . Similar to Du et al. (2020), we use an ϵ -net argument for the unit sphere S d -1 = { v ∈ R d : ∥ v ∥ = 1 } . For any v ∈ S d -1 , we know that ( v ⊤ a i ) 2 -1 is zero-mean and 16 ρ 2 -subgaussian. By using the Bernstein inequality, we have for any ϵ > 0

■

Next, we take a 1 5 -net N ⊂ S d -1 of S d -1 with size |N| ≤ e O ( d ) . By using the union bound, we know that

Let the right hand of the above inequality equals to δ 1 . We know that with probability 1 -δ 1 :

Note that for any u ∈ S d -1 , there exists u ′ ∈ N such that ∥ u -u ′ ∥ ≤ 1 5 . Then, we have that

where ∥ A ∥ 2 is the operator norm of matrix A . Taking a supreme over u ∈ S d -1 , we have that

Then, we complete the proof.

■

In Section 5, we assume the latent distribution is a Gaussian distribution instead of a subgaussian one. Yuan et al. (2023) show that in this setting, the approximation error bound has better dependence on n s .

Lemma D.3 (Lemma C.1, (Yuan et al., 2023)) . Assume the latent distribution is a Gaussian distribution q z = N (0 , Σ) with Σ = diag ( λ 2 1 , . . . , λ 2 d ) ≻ 0 . Then, the solution ( ̂ V s , ̂ θ ) of Equation (1) has the following approximation error bound with probability 1 -δ 1 :

E Additional Experiments

In this section, we do experiments on real-world datasets to show that the new model obtained by only fine-tuning appropriate encoder and decoder layers on target datasets with only 10 images can generate novel images with the target dataset feature. On the contrary, if all parameters can be fine-tuned, the model will suffer from memory phenomenon and only generate the ten images in the target dataset. This phenomenon indicates that only fine-tuning the appropriate encoder and decoder will result in a model with a generalization property.

Setting. In this experiment, we use a U-net network with attention layers, which contains 11 downblocks, 2 middleblocks, and 15 upblocks. When only fine-tuning the encoder and decoder layers, we fine-tune the first 4 downblock layers (encoder) and 4 upblock layers (decoder) instead of only using linear layers as the encoder and decoder (discuss in the later discussion paragraph).

The above experiments are conduct on a GeForce RTX 4090. We train the neural network using AdamW optimizer with learning rate 0 . 0001 . For the pre-trained phase, we train the models for 200 epochs with batch size 20 . It takes 5 hours to obtain a pre-trained diffusion models. For the fine-tuning phase, we fine-tune the pre-trained models for 400 epochs with batch size 2 . It take 3 minutes to fine-tune the pre-trained models.

.

(a)Target dataset

(b) The results of fine-tuning all parameters

(c) The results of fine-tuning encoder and decoder

Dataset. Our experiments use 2 real-world datasets: the CelebA64 dataset and the cat face dataset.

CelebA64 (size 3 ∗ 64 ∗ 64 ).

(a) Source dataset: 6400 images of faces with different hairstyles (without the bald feature).

(b) Target dataset: 10 images with the bald feature in CelebA64.

Cat face images (size 3 ∗ 64 ∗ 64 ).

(a) Source dataset: 4200 cat images with different colors (without black color cat).

(b) Target dataset: 10 black color cat images (The color black constitutes more than 70% of the image's composition.).

Discussion on results. The experiment results of CelebA64 have been discussed in Section 6. The experiment phenomenon is similar for the cat face images, which means the models obtained by only fine-tuning the encoder and decoder can generate novel images with the target feature (Figure 2). We note that when choosing the target cat face dataset if the color black constitutes more than 70% of the image's composition, we view this cat image as the black cat. Hence, different colors exist for cats, such as white and grey, due to the target dataset containing a small number of these colors (such as images 1, 3, 4, 6, 8). As a result, our fine-tuning results also contain these colors. However, our results do not contain colors other than those in the target dataset and can produce novel samples, which also proves the effectiveness of our fine-tuning method.

Discussion on linear encoder and decoder. Assumption 3.1 assumes the linear subspace, which indicates linear encoder and decoder. However, we fine-tune the first 4 downblock layers (encoder) and 4 upblock layers (decoder) instead of only using linear layers as the encoder and decoder. We note that this operation does not conflict with our Assumption. Recall that in Stable Diffusion (Rombach et al., 2022), the diffusion models run in the V AE embedding space 7 . Hence, we can view the first 3 downblock layers and the last 3 upblock layers as the VAE encoder and VAE decoder. Then, we can obtain X in this paper by running the V AE encoder. The remaining 1 downblock and 1 upblock layer can be viewed as linear encoder and decoder A . As mentioned in Section 4.2 of StyleGAN (Karras et al., 2019), the feature of X obtained by running a good-enough VAE encoder has linear separability, which also supports our Assumption 3.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This work aims to explain the reason why few-shot diffusion models can achieve great performance with a limited dataset from the theoretical perspective. We achieve this goal by analyzing the approximation and optimization perspective (Theorem 4.3 and Theorem 5.2). We also do real-world experiments to support our theoretical results (Section 6).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss future work and limitation at Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have shown all assumptions, theorem and proof sketch in the main paper. The detailed proof appears in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We has shown all experiments detail in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: As a theoretical work, we simply train and fine-tune a diffusion models on the datasets in Appendix E. All detail is shown in Appendix E.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We has shown all experiments detail including dataset and training detail in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Since this work is a theoretical work, the experiments are qualitative experiments on the dataset we constructed and are used to support our theoretical results. Since we only use 10 images to fine-tune the models to generate images with specific target features, it is hard to calculate quantitative metrics such as FID. However, our experiment results still show that our methods can generate novel images with the target feature compared to the benchmark.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have shown the compute works and computation time in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the code of ethics and make sure that our work satisfies the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our work at the end of main paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a theoretical analysis of the few-shot fine-tuning problem for diffusion models. It makes the following key assumptions: (1) the pretraining and fine-tuning data distributions share a common latent distribution, and (2) a specific network architecture. Under these assumptions, the paper proves that during the fine-tuning phase, the diffusion model avoids the curse of latent dimensionality and achieves an approximation bound of $\\widetilde{\\mathcal{O}}(n_s^{-2/d}+n_{ta}^{-1/2})$. Furthermore, when considering a low-rank Gaussian distribution, the model attains an accuracy bound of $\\widetilde{\\mathcal{O}}(1/n_{ta}+1/\\sqrt{n_{s}})$. The paper also includes real-world experiments to support the theoretical findings.\n\nSTRENGTHS:\nThe theoretical part of the paper is solid. As an extension of [1], the paper studies the diffusion model in the few-shot fine-tuning problem. Both the assumption of the data distribution and the definition of fine-tuning are reasonable to me. The conclusion aligns with the practice, that diffusion model fine-tuning escapes the curse of dimensionality and requires much fewer data samples to converge to the underlying distribution. I haven't gone through the details of the proof, if there is no further problem for them, I believe this is a theoretical solid paper.\n\n[1] Chen, M., Huang, K., Zhao, T., and Wang, M. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.07194, 2023c.\n\nWEAKNESSES:\nThere is a gap between the real-world experiments and the theoretical results. The experiments demonstrate two main points: (1) fine-tuning the entire diffusion model leads to memorization, and (2) fine-tuning only the encoder and decoder parts of the diffusion model will generalize. However, these findings only support the validity of the fine-tuning definition used in the paper.\nThe gap lies in two areas:\n1. The paper lacks proof that fine-tuning the full model results in memorization.\n2. The real-world experiments do not show that few-shot fine-tuning can escape the curse of dimensionality. Specifically, the diffusion model still requires a similar number of samples for fine-tuning datasets with different intrinsic dimensions.\n\nI think the paper lacks sufficient real-world experiments to fully support its title, \"Few-Shot Diffusion Models Escape the Curse of Dimensionality.\" With such an experiment, I believe the paper will be a strong submission.\n\nQUESTIONS:\nIn section 4.1, Table 1, the requirement derived from $n_{ta}^{\\frac{d+5}{4(1 - \\alpha(n_s))}} \\geq n_s$ is counterintuitive. For datasets like ImageNet with higher latent dimensions, the requirement of $n_{ta}$ is less. Could the author provide more discussions or give some real-world experiments to support this?\n\nLIMITATIONS:\nThe paper has covered the limitations and societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper provides a theoretical analysis showing that fine-tuning pretrained diffusion models on a few target samples achieves a small approximation error, particularly in that it requires much fewer target data than source data. The authors also show that the solution has a closed form in a special case, demonstrating the ease of optimization for few-shot models. They provide bounds for this closed-form solution, which also shows better dependency on the target sample size than on the source sample size. Experiments on CelebA are conducted to further support the theoretical conclusions.

STRENGTHS:
1. This work offers a theoretical understanding of why few-shot diffusion models can achieve strong performance despite the curse of dimensionality, which is a significant and intriguing topic.
2. The conclusions are well-supported by theoretical results and are clearly explained. Section 4 seems solid and well-structured, although I have not examined the proof in the appendix. Additionally, in Section 5, they demonstrate the closed-form expression of the minimizer under an isotropic Gaussian latent distribution and its ability to recover the true subspace, which further adds comprehensiveness and aids illustration.

WEAKNESSES:
I don't have any particular criticisms; however, I am not an expert in this topic, particularly regarding the theoretical analysis of diffusion models. I would like to hear other reviewers' opinions.

QUESTIONS:
I wonder if there is any intuitive explanation for assumptions like 4.1 and 4.2, particularly what they could mean in practice. Or are they just technical assumptions without much intuition?

LIMITATIONS:
The authors have discussed certain limitations in Section 7.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides new bounds for the score function approximation and optimization in fine-tuning pretrained diffusion models (under some simplifying assumptions including linear structure of data distributions and equal latent distributions). The new approximation error bound depends only on the square root of the finetuning target sample size, which in turn suggests that fine-tuning diffusion models does not suffer from the curse of dimensionality (unlike their pretraining error bound which has an exponential dependence on the latent dimension). Furthermore, the accuracy bound and closed-form minimizer provides interesting insights into the fine-tuning behavior of diffusion models. The paper also provides limited empirical support for its theoretical claims on CelebA.\n\nSTRENGTHS:\nThe paper provides novel theoretical bounds that can explain an important property of diffusion models (namely their fine-tuning convergence). Aside from the inherent theoretical value, I also think the provided insights can be very valuable both for the current practice of fine-tuning diffusion models, and for their future advancement. The paper is well organized and mostly easy to read (although the paper can benefit from simplifying its notations a bit more).\n\nWEAKNESSES:\nI do not see any major weakness in this work, but I do have two minor concerns:\n\n1- The paper mentions that data-efficient fine-tuning of diffusion models is very successful and tries to explain this phenomenon. However, the paper does not clarify the extent of this success. I think the paper would benefit from discussing the many ways in which existing fine-tuning methods are suboptimal (diversity, out of distribution, etc), so as to provide a more nuanced context to the reader.\n\n2- The paper does not sufficiently discuss the importance of its assumption, that is, what happens if each one breaks? Which ones are more crucial than others? Which ones are less realistic than others?\n\nQUESTIONS:\nPlease see my questions in the Weaknesses section.\n\nLIMITATIONS:\nThe paper sufficiently discusses its weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the few-shot transfer in diffusion models. Specifically, it focuses on bounding the score-matching loss for the target distribution, which can be considered the estimation error of the corresponding score function. By assuming the source and target distribution share the same linear structure distribution on latent space of dimension $d$,  it gives a $\\tilde{\\mathcal{O}}(n_s^{-2/d} + n_{ta}^{-1/2})$ bound for training the few-shot diffusion model with $n_s$ source domain data and $n_{ta}$ target domain data. As claimed by the authors, this breaks the curse of dimensionality for the target domain with an improvement from $n_{ta}^{-2/d}$ (no few-shot adaptation) to $n_{ta}^{-1/2}$. In addition, they also proved a bound $\\tilde{\\mathcal{O}}(2/n_{ta} + 1/\\sqrt{n_s})$ for the optimization error of score matching loss for a latent Gaussian special case.\n\nSTRENGTHS:\n* This paper provides theoretical guarantees for few-shot diffusion, illustrating sample complexity improvement w.r.t non-transfer diffusion process.\n* The structure is clear and easy to understand, even though the writing quality could be further improved.\n\nWEAKNESSES:\n* Approximation error has a specific definition in learning theory, which characterizes the inductive bias caused by the selection of the hypothesis class [1]. When using the terminology as another definition, e.g., the score matching loss of an estimated score function with early stopping in Theorem 4.3, it should be obviously defined to avoid confusion.   \n* I have the following concerns about presenting and interpreting the bounds:\n\t* In Theorem 4.3, the bound depends on $F$, which is related to $d^3$. So, the order should be $\\tilde{\\mathcal{O}}(d^3(n_s^{-2/d} + n_{ta}^{-1/2}))$, I am not sure if $d^3$ could be hidden directly. \n\t* According to this, when $d$ becomes large, the error should grow, which means the requirement of $n_{ta}$ should grow. However, in Table 1, with the increase of latent dimension, $n_{ta}$ decreases for ImageNet and MS-COCO. This is contradictory to your results. \n\t* When $d<4$, isn't few-shot fine-tuning worse than directly training given the theorem? \n\n* I am not quite convinced by the explanation of the experiments.  \n\t* How is the difference between fine-tuning all the parameters and fine-tuning a smaller fraction of encoder-decoders reflected on your bounds? \n\t* I think there exist many ways to adjust the hyper-parameter for fine-tuning all the parameters to achieve similar results as fine-tuning a small part.\n\n[1] Shalev-Shwartz, Shai, and Shai Ben-David. _Understanding machine learning: From theory to algorithms_. Cambridge University Press, 2014.\n\nQUESTIONS:\n1. Why $\\tilde{V}_{ta}$ in Lemma 5.1 depends on time $t$ ? Shouldn't it be the minimizer of the integral over $t$? \n\n\nGrammar: Line 271, 278, 280, 289, etc ...\nI suggest the authors check the grammar and revise the paper accordingly.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}