Training Diffusion-based Generative Models with Limited Data
Authors: Zhaoyu Zhang
Yang Hua
Guanxiong Sun
Hui Wang
Seán McLoone
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models; Limited Data; LD-Diffusion
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We present a novel theoretical insight for diffusion models and propose a novel diffusion model called Limited Data Diffusion (LD-Diffusion) which can effectively improve the training of diffusion models under limited data settings.

Abstract: Diffusion-based generative models (diffusion models) often require a large amount of data to train a score-based model that learns the score function of the data distribution through denoising score matching. However, collecting and cleaning such data can be expensive, time-consuming, and even infeasible. In this paper, we present a novel theoretical insight for diffusion models that two factors, i.e., the denoiser function hypothesis space and the number of training samples, can affect the denoising score matching error of all training samples. Based on this theoretical insight, it is evident that minimizing the total denoising score matching error is challenging within the denoiser function hypothesis space in existing methods, when training diffusion models with limited data. To address this, we propose a new diffusion model called Limited Data Diffusion (LD-Diffusion), which consists of two main components: a compressing model and a novel mixed augmentation with fixed probability (MAFP) strategy. Specifically, the compressing model can constrain the complexity of the denoiser function hypothesis space and MAFP can effectively increase the training samples by providing more informative guidance than existing data augmentation methods in the compressed hypothesis space. Extensive experiments on several datasets demonstrate that LD-Diffusion can achieve better performance compared to other diffusion models. Codes are available at https://github.com/zzhang05/LD-Diffusion.

Training Diffusion-based Generative Models with Limited Data
Zhaoyu Zhang'
Yang Hual
Guanxiong Sun? Hui Wang' Seán McLoone'
Abstract
ing superior image quality, greater diversity, and more sta- ble training on image generation tasks (Dhariwal & Nichol, 2021; Ho et al., 2020; Karras et al., 2022). In theory, diffu- sion models aim to train a score- score-based model (Song et al., 2021) that matches the score function of the data distribu- tion via denoising score matching (Song et al., 2021; Karras et al., 2022; Wang al.. 2023a). The current success of diffusion models unconditional image synthesis (Vahdat et al., 2021; Song al., 2023; 2020; Nichol & Dhariwal, 2021), text-to-image generation (Podell al., 2023; Rom- bach et al., 2022a; Nichol et al., 2021; Saharia et al., 2022; Wang et al., 2023b) and audio generation (Kong et al., 2020; Popov et al., 2021) tasks is fueled by the almost unlimited supply of samples. Nevertheless, collecting and cleaning such data can be expensive, time-consuming, and sometimes infeasible, which significantly constrains the applications of diffusion models in real-world scenarios.
Diffusion- based generative models (diffusion models) often require a large amount of data to train a score-based model that learns the score function of the data distribution through denoising score matching. However, collecting and clean- ing such data can be expensive, time-consuming, and even infeasible. In this paper, we present a novel theoretical insight for diffusion models that two factors, i.c. the denoiser function hypoth- esis space and the number of training samples, can affect the denoising score matching error of all training samples. Based on this theoretical insight, it is evident that minimizing the total de- noising score matching error is challenging within the denoiser function hypothesis space in exist- ing methods, when training diffusion models with limited data. To address this, we propose a new diffusion model called Limited Data Diffusion (LD-Diffusion), which consists of two main com- ponents: compressing model and novel mixed augmentation with fixed probability (MAFP) strat- egy. Specifically, the compressing model can constrain the complexity of the denoiser func- tion hypothesis space and MAFP can effectively increase the training samples by providing more informative guidance than existing data augmen- tation methods in the compressed hypothesis space. Extensive experiments on several datasets demonstrate that LD-Diffusion can achieve better performance compared to other diffusion mod- cls. Codes are available at https:/ / / gith.b. com/ z Z5mn///han00/// LD-Di fusion.
Focusing on training diffusion models with limited data, we present a novel theoretical insight for diffusion mod- els that the denoising score matching error of all training samples, i.e., the total denoising score matching error, determined by two key factors: the denoiser function hy- pothesis space and the number of training samples ($3.2). To further validate this theoretical insight, we conduct sev- eral experiments to illustrate that both factors significantly influence the training of diffusion models ($3.3). According to the unveiled insight, challenges arise when directly adapt- ing existing methods (Karras et al., 2022; Song & Ermon, 2019; Ho al., 2020; Dhariwal & Nichol, 2021; Karras et al., 2024) to limited data settings because they struggle to minimize the total denoising score matching error within their denoiser function hypothesis space.
To improve the training of diffusion models with limited data, we propose a new diffusion model called Limited Data Diffusion (LD-Diffusion). Motivated by the theoretical in- sight, LD-Diffusion consists of two main components. The first main component is compressing model with the en- coder and decoder obtained from different pre-trained VAE models, which aims to reduce the complexity of denoiser function hypothesis space. Although image compression models have previously been applied to several diffusion models (Esser et al., 2021; Vahdat et al., 2021; Rombach et al., 2022a), they all seek accelcrate the training speed and reduce the computational cost. In contrast, the com-
1. Introduction
Diffusion-basse generative models (diffusion models) have decisively surpassed GANs in generative modeling, achiev-
'Queen's University Belfast Huaawii UKRD. Correspondence to: Yang Hua <y.hua@ qub. ac.uk>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Training Diffusion-baase Generative Models with Limited Data
denoising score matching error is challenging to be min- imized within the denoiser function hypothesis space of existing methods; (2) We propose novel diffusion model called Limited Data Diffusion (LD-Diffusiée which con- sists of two main components and two improved training techniques. These components and techniques can effec- tively improve the training of diffusion models with limited data; (3) Experiments on FFHQ and low-shot (Zhao et al.. 2020) datasets demonstrate that LD-Diffusion can achieve better performance compared to other diffusion models.
Figure 1. The comparison of EDM + (Karras et al., 2022) and LD-Diffusion the FFHQ (Karras al., 2020b) dataset (256 x 256) under limited data settings. Left: Compared Fréchet Inception Distance (FID) (Heusel et al., 2017) scores of LD- Diffusion and EDM+ + DA on the FFHQ dataset. The FID score of LD-Diffusion on FFHQ-100 is even lower than the FTD score of EDM+ DA on FFHQ-IK, which demonstrates that LD-Diffusion can achieve huge data efficiency compared with EDM DA. Right: Comparison of images generated by EDM +DA and LD-Diffusion on the FFHQ dataset, highlighting the superior quality of images produced by the latter.
2. Preliminary
2.1. Diffusion-based Generative Models
In recent years, diffusion models (Karras et al., 2022; Ho et al., 2020; Dhariwal & Nichol, 2021) have exhibited their effecttiveness in imago image-generation tasks. Suppose we are given dataset {ar, } where each sample in the dataset is independently drawn from the data distribution Pdato (ar). The goal of diffusion models is to construct a diffusion process (xt(t)]² indexed by a continuous time variable E [o, According to Song et al. (2021), this diffusion process can be modeled as the solution to a stochastic dif- ferential equation (SDE): da f(xr + g(t)dw,
pressing model in LDD-Diffusion aims to constrain the com- plexity of the denoiser function hypothesis space to obtain low-dimensional hypothesis space that can better repre- sent the diffusion-based high-dimensional hypothesis space ( (Wang al., 2024; Boffi al., 2024; Li et al.). Having such a compressed low-dimensional hypothesis space can make minimizing the total denoising score matching error casier with limited data (64.1). The second main component in LD-Diffusion a novel data augmentation called Mixed Augmentation with Fixed Probability (MAFP), which is designed for the compressed low-dimensional hypothesis space. Concretely, MAFP is more informative than existing data augmentation (DA) methods (Karras et al., 2022) in the compressed low-dimensional hypothesis space, and can effectively increase the number of training samples while avoiding leaking issues (Karras et al., 2020a; Zhang et al., 2020; Zhao et al., 2021; 7hang et al., 2024a). addition, we introduce improved training techniques, i.e., patch training (Wang et al., 2023c) and larization, to further boost performance under limited data settings. With these components and techniques, we select the most representative diffusion model, i.e., EDM (Karras et al., 2022), to implement -i-fiffusiin. As illustrated in Figure 1, LD-Diffusion shows significant improvements compared with EDM + DA (Karras et al., 2022) under limited data settings.
(1)
where w denotes standard Brownian motion, f(.; () Rd + Rd is a vector-valued function called the drift coeffi- cient of x=)), g(4) E IR is a real-valued function called the diffusion coefficient, and dt represents a negative infinites- imal time step. Then, the reverse of diffusion process is also a diffusion process, given by the reverse-time SDE, rumning backwards in time:
dæ [f(z,) () - v( log (a:) di + g(()dw, (2) where o, is a schedule that defines the desired noise level at time 4, w is a standard Wiener process when time flows backwards from T to 0, and dw can be viewed as infinites- imal white noise. The corresponding ordinary differential equation (ODE) of the reverse SDE is the probability flow ODE (Song et al., 2021), expressed as:
(OOD) regu-
1 dæ = f(x, t) 2 (t) log (x) dt,
(3)
where Va log Ps, (x) is the score function (Hyvärinen & Dayan, 2005), a vector field that points towards the higher given noise level. The only unknown
In summary, our paper is pioncering study focusing training diffusion models with limited data with three main contributions as follows: (1) We are the first to propose the novel theoretical insight for diffusion models that the total denoising score matching error is affected by two fac- tors, i.c., denoiser function hypothesis space and the number of training samples. According to this theoretical insight, when training diffusion models with limited data, the total
density of data term in (2) and (3) is the score function. Thus, a func- tion e (æ, ox)) parameterized by a learnable neural network, is applied to estimate the score function's 's values. Denoising score matching is currently the most popular way of esti- mating score functions applied in diffusion models. After learning the estimated score function ep (x, ox). ). an estimated reverse SDE or ODE can be obtained in order to collect data samples from the estimated data distribution.
'Tee EDM + DA is the method that EDM applies DA, where the DA (with a conditional input strategy) is proposed in EDM.
2

---

Training Diflusion-based Generative Models with Limited Data
2.2. Diffusion Models with Limited Data
2022), i.e., Da σ.) parameterized by neural network, to minimize the expected I-2 denoising error for samples drawn from the distribution Pdata independently for every St. This can be expressed as:
Recently, diffusion models have significantly outpaced GANs in generative modeling, delivering higher image qual- ity, enhanced diversity, more stable training, and cutting- edge performance across various image generation tasks. However, training diffusion models commonly requires a large amount of data, which is expensive and difficult to clean. Although some studies (Wang et al., 2023c; Moon et al., 2022; Zhu et al., 2022; Hur et al., 2024; Wang al., 2023a; Lu et al., 2023; Ruiz al., 2023; Sinha et al., 202 Giannone et al., 2022; Yang et al., 2024) have attempted to finetune diffusion models with limited data, their approaches are similar to the transfer learning task and rely on the simi- larity of the source and target domain (Hur et al., 2024). In contrast, except for Patch Diffusion (Wang et al., 2023c), there are very few contributions that focus on training dif- fusion models with limited data from seratch. Specifically, Patch Diffusion introduces a diffusion model with the patch training strategy to reduce data and computational resources. However, experimental results show that Patch Diffusion still needs thousands of training samples. In this study, we aim to train robust diffusion models with only hundreds of samples from scratch, aligning with the training GANs with limited data (Zhang et al., 2025; 2024b).
((Do; = Eyr-pane E Bn Do(y + n;
y 1
(4)
where y is training image, is noise and or is schedule that defines the desired noise level. Then, the score-based model can be represented as:
Ea or) = -DD( - w)/o
(5)
Based Appendix B.3 in EDM (Karras et al., 2022), by expanding the expectations in Eq.(4), it can be rewritten as an integral over the noisy sample x::
(() = J Jza Y Σ
N (x; U. oft) (Doo(a; - valls da.
=:((D;;0,0))
(6) (6) demonstrates that we can minimize L(Do; by minimizing [D); æ; ot) independently for each æ. Based on the theory presented in Song et al. (2021), Karras et al. (2022) and Wang et al. (2023c), with sufficient data pro- vided, the optimal (2; dt) can be expressed as:
Dī 04) = arg min [(Do; (a(
2.3. Limited Data Settings in Generative Models
Based on Eq.(7), the optimal score- score-based model că (z, o.) in EDM can match the score function of data distributions
For limited data settings in GAN- approaches (Zhao et al., 2020; Li al., 2022), there are two common scenar- ios: Case 1: A small subset of a large dataset is used to train the model, while the full dataset serves as the reference dis- tribution for calculating the FID. This case is applied to the experiments with the FFHQ dataset; Case 2: The original small dataset is used both for training and as the reference distribution for FID calculations. This case is typically uti- lized in experiments with low-shot datasets. In this study, we consider both cases when designing LD-Diffusion.
(z) at any time t, which can be formulated as
follows:
(2, = (D) 01) - =x//o = Valogro, (x). (8)
3.2. Theoretical Analysis of Denoising Score Matching with Limited Data
Based on Eqs.(4), (6) and (7), we provide the theoretical analysis of denoising score matching with limited data, Ac- cording to Wang et al. (2020), minimization of (6) can be analyzed by empirical risk minimization (Mohri et al., 2018: 8; Vapnik, 1991) (with possible regularizers) in theory. For a better illustration, we define essential mathematical symbols as follows.
3. Theoretical Insight
3.1. Score-based Model and Denoising Score Matching
Without loss of generality, we choose the most representa- tive diffusion model, i..., EDM (Karras et al., 2022), for our analysis. Notably, EDM can be implemented using various diffusion architectures, such as noising conditional score network (NCSN) (Song& & Ermon, 2019), denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) and ablated diffusion model (ADM) (Dhariwal & Nichol, 2021). According to Song et al. (2021) and Karras et al. (2022), diffusson models aim to train a score-based model co(2,, ot) that matches the score function of data distribu- tions Valogpo, (z:). To achieve this goal, EDM builds up the denoiser function with preconditioning (Karras et al.,
Let (1))),)) = arg min min C(Do; x, o1) be the function that minimizes the expected risk, Do(2,, oz) arg minp,ta:or) L(Do; x0, σI) be the function in D that minimizes the expected risk and DĀ (x. = arg minpy (aDD,)eD C(D x, aT) be the function in D that minimizes the empirical risk with N samples provided. DD is the denoiser function hypothesis space in practice and N is the number of training samples.
As Dj(x,, o1) is unknown, the common approach is to ap- proximate it by certain Do(2; E D. Do(a;; ot) is the
3

---

Training Difflusion-aase based Generative Models with Limited Data
FFB10-14K (256 256)
FFHQ-100 (256 256)
EDR FEHOO 1DM0
Thining durriier skoea to (0!l)imm!!
(b)
(c)
(d)
Figure 2. Validation experiments for the two factors highlighted in $3.2. Here, we select the EDM (Karras et al., 2022) with the FFHQ (Karras et al., 2020b) dataset for experiments: (a) The comparison of FID (Heusel et al., 2017) curves EDM with the FFHQ-100 100 (64 x 64) dataset and EDM with the FFHQ-140K (61 x 64) dataset during training; (b) Generated images of EDM with FFHQ-140K (256 256) dataset; (c) Generated images of EDM with the FFHQ-100 (256 256) dataset; (d) Generated images of EDM with the FFHQ-100 (64 x 64) datasct.
best approximation for Da σ+) E D, and D (æ, ot) is the optimal hypothesis in D obtained by empirical risk minimization. For simplicity, we assume that D2 (a: oe), o.) and D (ar, ot) are unique. Then, the total error of C(Do; æ, o.) in (q) during training can be formulated following existing approaches (Bottou & Bousquet, 2007: Bottou et al., 2018) as:
following the studies of GANs with limited data (Karras et al., 2020a; Zhao al., 2020). We select EDM as the baseline with FFHQ-100 and FFHQ-140K datasets for ex- periments. Specifically, FFHQ-100 is subset of 100 im- ages in the FFHQ dataset and TFHQ-140K is the full FFHQ dataset with xflip augmentation. The FFHQ dataset with 70K images is used as the reference distribution for the FID calculation. We measure quality by computing FID between 50K generated images and all available training images, as recommended by Heusel al. (2017), regardless of the subset actually used for training. Following the EDM, the generated images are based on the same ODE solver, i.e., the number of function evaluations (NFE) is 79.
E =|c (D. iz, a.) a a =E [e (Do: (D):
Exp (D)
+ (D ae, o.) - (Do:
Sye (2,N)
(9)
The influence of the N on the training of the EDM is shown in Figure (a). (b) and (c). Without loss of generality, we select the FFHQ dataset with different image resolutions (representing different dimensions of D) for experiments. Figure 2 (a) compares the FID curves of EDM during train- ing on the FFHQ-100 and FFHQ- 140K datasets with 64 x 64 resolution. Figure 2 (b) and (c) compare the generated im- ages of EDM with the FFHQ-140K (256 x 256) and FFHQ- 100 (256 x 256) datasets. These results demonstrate that N can significantly influence the quality of generated images with the same D.
where Eupp (D) is the approximation error which measures how close the functions in D can approximate the op-
timal (z ot), and Eest (D, N) is the estimation error which measures the effect of minimizing the empirical risk L(D); æz oz) instead of the expected risk C(D;; xr, ot) in D.
(9) demonstrates that the total error is affected by both D and N. In general, with sufficient N, L(D x ot) can be a good approximation of L(D ; æ at) with the empirical risk minimizer D (x, ot). Then, the Eert N) can be easily reduced in DD However, with only limited N pro- vided in D, the L(DØ"; x o1) may then be far from being a good approximation of the expected risk C(Do; x, ot). Thus, the empirical risk minimizer DĀY (xx, ox) is no longer reliable. In this case, based on (9), the total error of L.Do; x. oF)) is difficult to minimize, which leads to sub- optimal score-based model that matches the score function of data distributions in practice, resulting in a decrease in the quality of generated images (shown in Figure 2 (c)).
The effects of the D on the training of the EDM are shown in Figure 2 (c) and (d). Specifically, we select different image resolutions with the same N for experiments. It is clear that the quality of generated images with EDM under the 256 x 256 FFHQ-100 dataset (Figure 2 (c)) is much worse than the quality of generated images with EDM under the 64 64 FFHQ-100 dataset (Figure 2 (d)), demonstrating that -ow-dimensional D is easier to optimize, leading to higher quality generated images.
3.3. Validation Experiments
4. Limited Data Diffusion (LD-Diffusion)
To further demonstrate the theoretical insight in $3.2, we approach the experiments by artificially subsetting larger commonly-used datasets, i.e., FFHQ (Karras et al., 2020b),
To address the challenges identified in our theoretical anal- ysis in $3.2, we propose a novel diffusion model called
4

---

Training Diffusion-basee Generative Models with Limited Data
Compreming
x(T
T: Dill Difl-Angment (Trassiitien
Case
71
75
Tranirt
Z
X
Zy=1
Zz
f:
P1
Adding
Unneninng Sunction li-Nct
x
Denshing strp
2
2
2z
Protrained Enceder
Lasited Date
Ginerated imagen
Lev-diavnsizeal Denolerr Funetion lyyptikesl Spare
Pretraied Becoder
ADA
Negatni Dats Augmontation
kos Reoe pasking poiks sport
C
Figure 3. An overview of Limited Data Diffusion (LD-Diffusion). LD-Diffusion consists of two components: A compressing model with pre-trained encoder (e) and decoder (D) models from similar but slightly different VAE models (with the same latent space) in Rombach et al. (2022h) and a novel data augmentation method called Mixed Augmentation with Fixed Probability (MAFP). The compressing model can obtain compressed wwwddimensional denoiser function hypothesis space by constraining its complexity. MAFP can effectively increase the number of training samples within the compressed low-dimensional hypothesis space. Furthermore, we apply patch training (Wang al., 2023c) Case ($2.3) and propose novel but-of-distributtinn (OOD) regularization for Case ($2.3) further improve the training of diffusion models with limited data.
Limited Data Diffusion (LD-Diffusion) to improve the train- ing of diffusion models with limited data, as shown in Figure 3. DD-Diffusion has two components: (1) A compressing model that reduces the complexity of the D, where D is the denoiser function hypothesis space in practice, same as in $3.2; (2) A novel data augmentation method, MAFP, de- signed to increase the number of training samples N within the compressed low-dimensional 'D. Furthermore, we intro- duce improved training techniques for -D-Diffusion with two different limited data cases (shown in $2.3). More the- oretical interpretations of LD-Diffusion can be found in $A.1.
(Karras et al., 2020a; Zhang et al., 2020; Zhao et al., 2021; Zhang et al., 2024a) during training. Furthermore, applying the pre-trained encoder and decoder models from the same pre-trained VAE models can unavoidably lead to a small loss of image details (Rombach et al., 2022a), which can limit the variety of detailed information in generated images, thereby leading to decrease in performance. Therefore, we select the pre-trained encoder and decoder models from similar but slightly different VAE models (with the same latent space) for the compressing model in practice, which can further improve performance (See BB.2).
4.2. Mixed Augmentation with Fixed Probability
4.1. Compressing Model
Although data augmentation (DA) methods have already been applied to EDM (Karras et al., 2022), this DA is de- signed based on ADA (Karras et al., 2020a) in the larger denoiser function hypothesis space. In contrast, following the proposed compressing model in $, the DA for diffu- sion models with limited data should be more informative in the compressed low-dimensional D. With the fact that Diff- Augment (Zhao et al., 2020) is more informative than ADA in the low-dimensional feature space (Sauer et al., 2021), combining Diff-Augment and ADA is the most straightfor- ward approach to achieving this goal. However, directly combining Diff-Augment and ADA can cause significant leaking issue. To address this, we propose novel DA method called Mixed Augmentation with Fixed Probability (MAFP), which applies two fixed probabilities p1 and p2 with conditional input strategy (Karras et al., 2022) to control Diff- Diff-Augment and ADA, respectively. Specifically, Diff- Diff-Augment (Translation) is applied with probability p1
The compressing model in LD-Diffusion is built using un- paired pre-trained encoder (Rombach et al., 2022d) and decoder (Rombach al., 2022c) models to reduce the com- plexity of DD. The purpose of applying pre-trained encoder and decoder models is to constrain the complexity of D, thereby resulting in now-dimensional D that can better represent the diffusion-based high-dimensional hypothe- sis space (Wang et al., 2024; Boffi et al., 2024; Li et al.). According to Wang et al. (2020), the low-dimensional D consists of a small area to be considered for optimization. With the compressed low-dimensional DD the limited N is sufficient to reduce the total denoising score matching error (Germain et al., 2016; Mahadevan & Tadepalli, 1994; Nguyen & Zakynthinou, 2018). Notably, it has already been proven that the encoder and decoder models do not eliminate significant details (Rombach et al., 2022a). Thus, the compressing model can prevent the heavy leaking issue
5

---

Training Diffusion-baxee Generative Models with Limited Data
Full
IP Patch
F
Patch
NDA
NDA
training differrent craining
I Subact
P
NDA
NDA
Parch truining with data aecincetacion
(a)
(b)
(c)
Figure Understanding the improved training techniques, i.c., patch training and OOD regularization, on two scenarios. (a) Patch training leads to the leaking problem in two situations, i.e., different training data with patch training and augmentation data with patch training. Both situations result in different patches in patch-wise denoising score matching. (b) Schematic overview the learning distributions with two techniques on Case 1 ($2.3). (c) Schematic overview of the learning distributions with two techniques Case 2 ($2.3).
and skipped with a probability 1- P1. and ADA is applied with a probability P2 and skipped with a probability - P2 in MAFP. As a result, MAFP can avoid unnecessary DA operations with a fixed probability during training, thereby effectively avoiding the leaking problem in the compressed now-dimensional denoiser function hypothesis space. Addi- tionally, the proposed MAFP also has great generalization ability, allowing it to be applied to GAN-based methods to improve performance under limited data settings. More details of MAFP can be found in $A.2 and $B.3.
different patches at the same image position during patch- wise denoising score matching. Sampling the images from these random different patches can lead to the leaking prob- lem (Karras et al., 2020a; Zhang et al., 2020; Zhao et al., 2021; Zhang et al., 2024a), i.c., producing OOD samples, in the diffusion models.
Diitrf--ittiibttion (OOD) regularization. Motivated by the success of the regularization method in GANs with lim- ited data (Fang et al., 2022: Tseng et al., 2021), we propose novel regularization method called OOD regularization. To the best of our knowledge, OOD regularization is the first regularisation method designed for diffusion models. Specifically, OOD regularization aims to penalize the gradi- ent if the generated samples are close to the OOD samples. To achieve this, the novel denoiser score matching with OOD regularization is shown as follows:
4.3. Improved Training Techniques for Two Cases of Limited Data Settings
Due to the difference in reference distribution for the two limited data cases ($2.3), the common leaking issue (Kar- ras et al., 2020a; Zhang al., 2020; Zhao et al., 2021: Zhang al., 2024a) affects performance differently in each case. Specifically,, for Case 1, leaking samples (Out-of- Distribution (OOD) samples) from the subset may actually be in-distribution for the full dataset, potentially improving training under limited data. Conversely, for Case 2, leak- ing samples are truly OOD samples for the dataset, which hinders the training under limited data. Based on the above analysis, we introduce the patch training (Wang al., 2023c) for Case and propose novel OOD regularization for Case 2 to enhance the training of diffusion models with limited data, respectively.
((De;as) = By-Pictt Bo-Pdate E Nio. f
DDoy + n. n; vI1
Oaggiaal
Do n1)
0113
Our 0OD regulartzation
(10)
where is the OOD samples which are obtained by apply- ing the Negative Data Augmentation (NDA) (Sinha et al., 2021a) the real samples y, and v is constant 100 that aims to enable the loss function to become stable. (1.( demonstrates that the proposed OOD regularization penal- izes the gradient in denoiser score matching if generated samples are close the OOD samples, which can effectively avoid the potential leaking issue.
Patch training. Recently, Patch Diffusson (Wang et al., 2023c) introduces patch training into diffusion models. Patch Diffusion presents the patch- wise denoising score matching improve the training of diffusion models. How- ever, patch training unavoidably lead to leaking issues. Specifically, we observe that patch training in diffusion models can also result in different patches during patch- wise denoising score matching in two situations, as shown in Figure 4 (a). Both situations can lead to the selection of
More explanation of applying patch training and OOD regularization in two cases. To better understand how the leaking problem influences the training of diffusion models with limited data, the schematic overview of the learning distributions for patch training and OOD regularization in two limited data cases is shown in Figure (b) and (c).
6

---

Training Diflusion-based Generative Models with Limited Data
FFHQ IK
2023c) is applied to LD-Diffusion on experiments with the FFHQ dataset. In contrast, the proposed OOD regulariza- tion is applied to LD-Diffusion in experiments with low-shot datasets. Based on EDM (Karras et al., 2022), the denoise sampling Number of Function Evaluations (NFE) is set as 79 and 511 for all diffusion-based methods on FFHQ and low-shot datasets, respectively.
Method
P? 100
2K
EDM (Karras et al., 2022)
No 79.10
EDM DA (Karras al., 2022)
No 50.73 30.75
27.17
Patch Diffusion (Wang et al., 2023c)
No 44.45 28.03
25.32
L.PD--8 (Wang et al., 2023c)
Yes 32.78 19.67 15.47
LD-Diffusion (ours)
Yes 28.51 17.76 14.36
Table 1. FID scores (lower is better) of different methods on the 256 256 FFHQ dataset. Massive Augmentation (MA) (Cui et al., 2022). i... xflipping. is applied in all methods. The results of the EDM are directly from Adaptive IMLE (Aghabozorgi al., 2023) while the results of EDM + DA are obtained based on the official open-source codes (Karras et al., 2022). The results of Patch Diffusion and LPDM-8 are obtained by ourselves based on official codes (Wang ct al., 2023c). The downsampling factors in the pre- trained encoder and decoder are selected as 8 for LPDM-8 and LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively. Here, P represents the Pre-training model to facilitate the training of diffusion models in the latent space.
5.2. Results on the FFHQ Dataset
We compare LD- LD-Diffusion with other diffusion models with limited data by conducting experiments on the 256 x 256 FFHQ dataset. As shown in Table 1, LD-DDffuuiio Difffusion achieves state-of-the-art (SOTA) performance compared to other dif- fusion models. To further demonstrate the effectiveness of LD- Difffusiion we also compare LD- LD-Diffusion with other dif- fusion methods using Precision and Recall (Kynkäänniemi et al., 2019). As shown in Table 2, LD-Diffusion achieves SOTA Precision and Recall performance compared with other methods.
Figure 4 (b) demonstrates that the leaking issue caused by the patch-training can benefit the training of diffusion models when the reference distribution is based the large- scale dataset, i.e., Case 1 ($2.3). Figure 4 (c) indicates that the leaking issue can decrease the training of the diffusion models when the reference distribution is based on small datasets, i.e., Case ($2.3). Therefore, the patch training should be applied to Case 1 and the OOD regularization should be applied to Case 2.
5.3. Results on Low-shot Datasets
We compare LD-Diffusion with all of the other state-of- the-art diffusion models by conducting experiments on the low-shot datasets. The results are shown in Table 3. LD-Diffusion outperforms other diffusion models and achieves SOTA performance. Additional comparisons of LD-Diffusion with other methods using Precision and Recall can be found in $B.6. To further demonstrate the superi- ority of LD-Diffusion, the comparisons of LD-Diffusion with other SOTA generative models with limited data can be found in $B.7. The compared generated images on low-shot datasets by EDM DA and LD-Diffusion are shown in 5.
5. Experiments
5.1. Datasets and Implementation Details
We select FFHQ (Karras et al., 2020b) and low-shot (Zhao et al., 2020) datasets for the experiments. For fair compar- isons, we follow the official open-source codes² for pre- processing and resizing the FFHQ to 256 256, as used in existing studies (Karras et al., 2020a; Zhao et al., 2020; Li et al., 2022). We select the most representative diffu- sion model, i.e., EDM, as the baseline for LD-Diffusion. Specifically, the noising conditional score network (NCSN) (Song& & Ermon, 2019) is selected as the architecture of the EDM baseline on low-shot datasets and the denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) is selected the architecture of the EDM baseline on the FFHQ dataset (refer to $5.4.4). For the compressing model in LD-DDffussio, the pre-trained encoder is selected from SD-MSE (Rombach et al., 2022d) and the pre-trained de- coder is selected from SD-EMA (Rombach et al., 2022c) with a downsample factor of 8 (refer to $B.2). For the MAFP in LD-Diffusion, we set p1 and pa as 0.1 for all experiments (refer to $5.4.2). Furthermore, patch training (Wang al.,
5.4. Ablation Study
5.4.1. EFFECTIVENESS OF COMPONENTS IN LD-DIFFUSION
We conduct the ablation study to show the components of LD-Diffusion are effective in both limited data settings introduced in $2.3. The results are shown in Tables 4 and 5. It is clear that the design of LD-Diffusion (refer to Figure 3) is reasonable and can indeed improve performance.
5.4.2. EFFECTIVENESS OF FIXED PROBABILITY IN MAFP
To show the effectiveness of the fixed probability P1 and Ps in MAFP, we conduct an ablation study by selecting different values of probability
and p2 in LD-Diffusion. To achieve this goal, we first apply Pa with ADA to LD- Diffusion in order to find the best value of p2 (without p1). and the results are shown in Table 6. Then, applying the best
Phtpps: //gi thub. com cow./o/aaa/ NV1 /edm
7

---

Training Diflusion-based Generative Models with Limited Data
FFHQ-100
FFHQ-1K
FFHQ-2K 2-
Method
R
P
P
R
EDM + DA (Karras et al., 2022)
0.724
0.005
0.668
0.222
0.709
0.281
Patch Diffusion (Wang et al., 2023c)
0.746
0.006
0.707
0.224
0.718
0.292
LPDM-8 (Wang et al., 2023c)
0.789
0.011
0.762
0.241
0.759
0.323
LD-Diffusion (ours)
0.791
0.018
0.767
0.249
0.769
0.323
Table A comparison of Precision (P) and Recall (R) (Kynkäänniemi et al., 2019) (higher is better) of LD-Diffusion with other diffusion models on 256 256 FFHQ dataset. Massive Augmentation (MA), i.e., xflipping, applied all methods. The P and are averaged over three runs; all standard deviations are less than 1,, relatively.
100-shot
Animal-
Method
Pre-training?
Obama
Grumpy
Panda
Cat
Dog
EDM (Karras et al., 2022)
No
51.30
36.90
23.70
48.60
100.10
EDM DA (Karras et al., 2022)
No
37.10
29.94
10.81
36.88
57.14
Patch Diffusion (Wang et al., 2023c)
No
41.47
30.89
13.25
43.71
72.17
LPDM-8 (Wang et al., 2023c)
Yes
14.27
14.56
5.13
14.92
15.95
L.D-Dffusion (ours)
Yes
13.00
13.31
4.70
12.77
12.48
Table 3. FID scores (lower is better) of different methods low-shot datasets (256 x 256). We follow the setting used in (Zhao al., 2020). Massive Augmentation (MA), ie., xflipping. is applied all methods. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
Method
FFHQ-100 (Case I)
P1
0.0
0.1
0.2
0.4
0.8
LD-Diffusion
28.51
FID
13.40
13.00
13.24
13.92
14.67
w/ OOD regularization
32.02
LD-Diffusion
28.51
Table 7. FID scores the 100-shot Obama dataset (256 256) by selecting different pa with Diff- auff-Att (Translation) under best value p2 0.1 with ADA. The FIDs averaged three runs; all standard deviations are less than 1 1%, relatively.
w/o Patch Training
31.57
w/o Compressing model
49.29
w/o MAFP
79.10
Table 4. FID scores on the FFHQ-100 dataset (256 256) by
adding or gradually removing the corresponding component in LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
Method
Obama FFHQ- 100
LD-Diffusion (with DA EDM)
13.57
29.87
LD-Diffusion (with MAFP)
13.00
28.51
Method
100-shot Obama (Case2)
Table 8. FID scores on the 100-shot Obama and FFHQ-100 datasets (256 256) for comparisons of MAFP with the exist- ing DA in EDM using LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
LD-Diffusion
13.00
w/ Patch training
13.97
LD-Diffusion
13.00
w/o OOD regularization
13.28
w/o Compressing model
36.31
w/o MAFP
51.30
value p2 = 0.1 with ADA, we add with Diff-Augment (Translation) in LD- Diffussion to obtain the best value of p1. and the results are shown Table 7. Consequently, we set Pl 0.1 and p2 0.1 for all experiments.
Table FID
the 100-shot Obama dataset (256 256) hy adding or gradually removing the corresponding component in LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
5.4.3. COMPARISON OF DIFFERENT DATA AUGMENTATIONS IN LD-DIFFUSION
P2
0.0
0.1
0.2
0.4
0.8
FID
13.98
13.55
13,86
14.51
15.22
We conduct experiments by applying MAFP and the existing DA to LD-Diffusion. The results are shown Table 8, indicating that MAFP more informative than the existing DA method (Karras al., 2022) for LD-Diffusion.
Table FID scores the 100-shot Obama dataset (256 x 256) by selecting different with ADA. The FIDs are averaged over three runs; all standard deviations are less than 1 1%, relatively.
8

---

Training Diflusion-based Generative Models with Limited Data
37.10
FID 10.81
FID
FID 57.14
FID 13.00 (-24.10)
FID 4.70 (-6.11)
FID 13.11 (-16.63)
FID 12.77 (-24.11)
FID 12.48 (-44.66)
(R)
(b)
(c)
(d)
(e)
Figure 5. The comparison of generated images with LD-Diffusion and EDM DA on (a) 100-shot Obama, (b) 100-shot Panda, (c)
100-shot Grumpy-cat, (d) AnimalFace-cat and (e) AnimalFace-dog datasets. Top: Images generated by EDM + DA. Bottom: Images
generated by LD-Diflusion. The decreasing value of FID in red color demonstrates the improvement of LD-Diffusion over EDM + DA.
Method
DDPM
ADM
NCSN
Method
100-shot Ohama FFHQ-100
EDM + DA
50.73
60.17
52.89
EDM (Karras et al., 2022)
51.30 112.26
79.10
LD-Diffusion
28.51
33.39
29.59
EDM2 (Karras et al., 2024)
119.72
Table FID scores (lower is better) the FFHQ-100 dataset with different selected architectures for EDM DA and the proposed LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
Table I1, Compared FID scores (lower is better) of EDM and EDM2 on the 100-shot obama and FFHQ- 100 datasct (256 x 256).
Method EDM + DA 38.09
DDPM
ADM
NCSN
the proposed LD-Diffusion can significantly improve perfor- mance compared with EDM + DA with all three different diffusion architectures, demonstrating the generality of the proposed method. Additionally, one recent approach called EDM2 (Karras et al., 2024) has been proposed to scale up EDM (Implemented with the ADM architecture). To demon- strate that selecting EDM as the baseline in DD-Diffusion is reasonable, we conduct experiments comparing EDM and EDM2 on both limited data cases. The results are shown in Table 11. Given that EDM2 is implemented with the ADM architecture consisting of more parameters, it is rea- sonable that the results of EDM2 are worse than EDM in both limited data cases.
45.89
37.10
LD-Diffusion 13.11
15.87
13.00
Table 10. FID scores (lower is better) on the 100-shot Obama dataset with different selected architectures for EDM DA and the proposed IDD-Diffusio.. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
5.4.4. COMPARISON OF DIFFERENT DIFFUSION ARCHITECTURES IN DI-DIFFUSION
To demonstrate how different diffusion model architectures influence the training of diffusion models with limited data, we conduct experiments by selecting different diffusion model architectures for EDM DA and LD-Diffusion in two limited data cases. For Case 1, study this by conduct- ing experiments the EDM DA and LD-Diffusion with the FFHQ-100 100 dataset, and the results are shown in Table 9. DDPM (Ho et al.. 2020) slightly outperforms NCSN (Song & Ermon, 2019) and achieves better performance compared to ADM (Dhariwal& & Nichol, 2021). For Case 2, we ap- proach this by conducting experiments on the EDM + DA and LD-Diffusion with the 100-shot Obama dataset. The results are shown in Table 10. NCSN (Song& & Ermon, 2019) marginally outperforms DDPM (Ho et al., 2020) and demon- strates superior performance compared to ADM (Dhariwal & Nichol, 2021). The denoise sampling NFE is selected as 511 and 79 for the 100-shot Obama and FFHQ datasets, respectively. Furthermore, both Tables and 10 show that
Additional ablation studies on utilizing various pre-trained encoders and decoders in the compressing model ($B.2), applying different augmentations in MAFP (SB.3), using diverse denoise sampling NFE during the sampling stage ($B.4) and applying different NDA methods in OOD regu- larization ($B.5) can be found in Appendix.
6. Conclusion
In this paper, we theoretically demonstrate that minimiz- ing the total denoising score matching error is difficult to achieve under limited data within the denoiser function hy- pothesis space of existing methods. To address this, we fur- ther propose a new diffusion model called Limited Data Dif- fusion (LD-Diffusion) to improve the training of diffusion models with limited data. Extensive experiments on several datasets demonstrate the superiority of I.D-Difuusio.

---

Training Diflusion-based Generative Models with Limited Data
Acknowledgement
Germain, P., Bach, F,, Lacoste, A., and Lacoste-Julien, S. PAC-Bayesian theory meets Bayesian inference. In NeurIPS, 2016.
This work partially supported by the High Performance Computing Centre of Queen's University Belfast and the Kelvin-2 supercomputer (EPSRC grant EP/T022175/1) The first author is supported by scholarship of the Queen's University Belfast and the Department for the Economy, Northern Ireland.
Giannone, G., Nielsen, D., and Winther, Few-shot diffu- sion models. arXiv preprint arXiv:2205. 15463, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
Impact Statement
This paper is pioneering study focusing on training diffusion-based generative models with limited data from scratch. The technical contributions of this paper do not raise any particular ethical challenges. However, because technology is usually a double-edged sword, our work may also bring potential social risks when applying diffusion- based generative models with limited data. For example, it may ease the high-que qualitya fake media synthesis using only limited data.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, 2020.
Hur, J., Choi, J., Han, G., Lee, D.-.., and Kim, J. Expanding expressiveness of diffusion models with limited data via self-distillation based fine-tuning, In WACV, 2024.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
References
Jiang, L., Dai, B., Wu, W., and Loy, C. C. Deceive d: Adaptive pseudo augmentation for GAN training with limited data. In NeurlPS, 2021.
Aghabozorgi, M., Peng, S., and Li, K. Adaptive imle for few-shot pretraining-fre generative modelling. In ICML, 2023.
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data. In NeurIPS, 2020a.
Boffi, N. Jacot, A., Tu, S., and Ziemann, I. Shallow dif- fusion networks provably learn hidden low-dimensional structure. In ICLR, 2024.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of StyleGAN. CVPR, 2020b.
Bottou, L. and Bousquet, O. The tradcoffs of large scale learning. NeurIPS, 2007.
Bottou, I.,, Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311 2018.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion generative models. In NeurlPS, 2022.
Chen, T., Cheng, Y., Gan, Z., Liu, J., and Wang. Z. Data- efficient GAN training beyond (just) augmentations: A lottery ticket perspective. In NeurlPS, 2021.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In CVPR, 2024.
Cui, K., Huang, J., Luo, Z., Zhang, G., Zhan, F,, and Lu, S. Genco: Generative co- co- traininiing for generative adversarial networks with limited data. AAAI, 2022.
Kong. Z., Ping. W., Huang. J., Zhao, K., and Catanzaro, Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint
2020.
DeVries, T. and Taylor, W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Kumari, N., Zhang, R., Shechtman, E., and Zhu, J.-Y. En- sembling ohflff models for GAN training. In CVPR, 2022.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021.
Kynkäääniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. In NeurlPS, 2019.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Li, X., Zhang, Li, X., Chen, S., Zhu, Z., Wang, P., and Qu, Q. Understanding diffusion-bassen representation learning via low diw-dimensional modeling. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning.
Fang, T., Sun, R., and Schwing, A. DigGAN: Discriminator gradient gap regularization for GAN training with limited data. In NewrIPS, 2022.
10

---

Training Diflusion-based Generative Models with Limited Data
Li, Z., Wang, C., Zheng, H., Zhang, J., and Li, B. Fakecir: Exploring contrastive learning for solving latent disconti- nuity in data-efficient GANs. In ECCV, 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Lu, H., Tunanyan, H.,, Wang, K., Navasardyan, S., Wang, Z., and Shi, H. Specialist diffusion: Buu--nd--pay sample- efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In CVPR, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, Denton, E. I.., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., al. Photorealistic rext-to-image diffu- sion models with deep language understanding. 2022.
Mahadevan, S. and Tadepalli, P. Quantifying prior deter- mination knowledge using the PAC learning model. Ma- chine Learning, 17:69-105, 1994.
Sauer, A., Chitta, K., Müller, J., and Geiger, A. Projected GANs converge faster. In NeurPS, 2021.
Sinha, A., Ayush, K., Song, J., Uzkent, B., Jin, H., and Ermon, S. Negative data augmentation. In ICLR, 2021a.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of machine learning. MIT press, 2018.
Sinha, A., Song, J., Meng, C., and Ermon, S. D2C: Diffusion-decoding models for few-shot conditional gen- eration. In NeurIPS, 2021b.
Moon, T., Choi, M., Lee, G., Ha, J.-W., and Lee, J. Fine- tuning diffusion models with limited data. In NeurlPS 2022 Workshop on Score-Basee Methods, 2022.
Song, J., Meng, C., and Ermon, Denoising diffusion implicit models. arXiv preprint (rXiv:2010002502. 2020.
Nguyen, H. and Zakynthinou, L. Improved algorithms for collaborative pac learning. In NeurIPS, 2018
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P. McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10744 2021.
Song. Y., Sohl-Dickstein. J.. Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. ICML, 2023.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021.
Tseng, H.-Y., Jiang, L., Liu, C., Yang, M.-H., and Yang, W. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J.. and Rombach, R. Sdxl: im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Vahdat, A., Kreis, K., and Kautz, Score-based generative modeling in latent space. In NeurlPS, 2021.
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudi- nov, M. Grad-tts: diffusion probabilistic model for text-to-speech. In ICML, 2021.
Vapnik, V. Principles of risk minimization for learning theory. NeurlPS, 1991.
Vashist, C., Peng. S., and Li, K. Rejection sampling imle: Designing priors for better few-shot image synthesis. In ECCV, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuusinn models. In CVPR, 2022a.
Wang, P., Zhang, H., Zhang, Z., Chen, S.. Ma, Y., and Qu, Q. Diffusion models learn oow-dinesiional distributions via subspace clustering. arXiv preprint wrXiv:2409.02422 2024.
Rombach, R. Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. hitps:/ bttps://huggingffatt http///gggggngnccccecc co/ sttabilityai. 2022b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. stabil watlllltttttssaaaae ftmem. 2022c.
Wang, X., Lin, B., Liu, D., and Xu, C. Efficient transfer learning diffusion models via adversarial noise. arXiv preprint arXiv:2308. //948, 2023a.
httss////gggigggac.eemco co/
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. http:s:///uugingg/e co/ stabilltyai/sd-vveer ft-mse. 2022d.
Wang, Y., Yao, Q. Kwok, T., and Ni, L. M. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys, 53(3):1-34, 2020.

---

Training Diflusion-based Generative Models with Limited Data
Wang, Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z.,, and Zhou, M. In-context learning unlocked for diffusion models. In NeurlPS, 2023b.
Wang, Z., Jiang, Y., Zheng, H., Wang. P., He, P., Wang. Z.,, Chen, W., and Zhou, M. Patch diffusion: Faster and more data-efficient training of diffusion models. In NeurIPS, 2023c.
Wang, 7.,, Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-GAN: Training GANs with diffusion. In ICLR, 2023d.
Yang, R., Jiang, B., Chen, C., Jin, R., Wang, B., and Li, S. Few-shot diffusion models escape the curse dimension- ality. In NeurIPS, 2024.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Zhang, H., Zhang, Z., Odena, A., and Lee, H. Consistency regularization for generative adversarial networks. In ICLR, 2020.
Zhang, Z., Hua, Y., Sun, G., Wang, H., and McLoone, S. Improving the leaking of augmentations in data-efficient GANs via adaptive negative data augmentation. In WACV, 2024a.
Zhang, Z., Hua, Y., Sun, G., Wang, H., and McLoone, S. Improving the training of the GANs with limited data via dual adaptive noise injection. In ACMMM, 2024b.
Zhang, Z., Hua, Y., Sun, G., Wang, H., and McLoone, S. Improving the training of data efficient GANs via qual- ity aware dynamic discriminator rejection sampling. In CVPR, 2025.
Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y., and Han, S. Differen- tiable augmentation for data-efficient GAN training. In NeurIPS, 2020.
Zhao, Z., Singh, S., Lee, H., Zhang, 7., Odena, A., and Zhang, H. Improved consistency regularization for GANs. In AAAI, 2021.
Zhu, J., Ma. H., Chen, J., and Yuan, J. Few-shot im- age generation with diffusion models. arXiv preprint arXiv:2211.03264, 2022.
12

---

Training Diflusion-based Generative Models with Limited Data
A. More Explanation of LD-Diffusion
ing score matching for LD-Diffusion is in the compressed low-dimensional denoiser function hypothesis space, which consists of a smaller area to be considered for optimiza- tion, thus resulting in better performance with limited data provided.
A.1. Theoretical Interpretations for LD-Diffusion
ODE formulation of LD-Diffusion. Based on the exist- ing studies (Wang et al., 2023c; Sinha et al., 2021a), the proposed improved training techniques do not influence the diffusion model learning data distributions, demonstrating that the proposed improved training techniques will not influence the convergence of diffusion models. Then, fol- et al. (2021); Karras et al. (2022) and Vahdat et al. (2021), the probability flow ordinary differential equation (ODE) of LD-Diffusion can be formulated as:
According to Wang et al. (2023c) and Sinha et al. (2021a), the patch training and proposed OOD regularization do not influence the learning data distribution, demonstrating that our proposed improved training techniques will not influence the convergence of the score-based model. Then, following Karras et al. (2022) and Vahdat et al. (2021), there still exists an optimal D (TT)(();:); oe) for (1.( and E1. in theory. In this case, based on Eq.(5) and (8), the optimal score-based model (e ox)) can match the score function lo(r (e (x))) any time t, which can be expressed as:
lowing the ODE formulation in Song
dæ -àio, Ve(T(w)) logun, (e(())))
(11)
where Tíx is proposed MAFP, or a schedule that defines the desired noise level at time i and is the pre-trained encoder as in Rombach et al. (2022d). According to $4, both and T would not result in the leaking issue.
co(e(T(z))) (D)
(14)
Denoising score matching for LD-Diffusion. According to the denoising score matching in Karras et al. (2022). we build up the denoiser function, i.e., Da( de), that mini- mizes the expected L2 denoising error for samples drawn from distribution p(e (ddala)) independently for every o+- Following the two limited data cases in $2.3, the denoising score matching for Case is based on the patch diffusion (Wang et al., 2023c) and can be expressed as:
where or is a schedule that defines the desired noise level. Eq.()4) demonstrates that the score-based model can also reach optimal in the compressed low-dimensional de- noiser function hypothesis space in theory.
A.2. Generalization of Proposed Mixed Augmentation with Fixed Probability (MAFP)
Based on $4.2, to demonstrate the generalization of the proposed MAFP, we apply it to Generative Adversarial Net- works (GANs), following the existing studies (Karras et al., 2020a; Jiang et al., 2021). Specifically, we set the prob- abilities pı and P2 in MAFP to be controlled adaptively by the overfitting degree of D (Karras et al., 2020a; Jiang et al., 2021), denoted as MAFP (Adaptive). The results are presented in Table 12, revealing that MAFP (Adaptive) outperforms Diff- Diff-Augment (Zhao et al., 2020) and ADA (Karras et al., 2020a) on low-shot datasets with StyleGAN2 (Karras et al., 2020b).
((Ds;o.) (y-ppcctr(aata)))) (1) Nnw F(s.aa)
(12)
| + oz) 2
where n is noise, & is the pre-trained encoder model, T' is the proposed MAFP, ji denotes the uniform distribution on the corresponding value range and Di,j, is the randomly crop small patches for any y ~ p(e(T(data)). Specifically, (i,j) is left-upper corner pixel coordinates to locate each image patch, and s denotes the patch size. Then, following (12), the denoising score matching for Case 2 can be expressed as:
B. More Experiment Results
C(Do: ot) = Exy-pe((T(data)) Tg Ty (U.o7))
B.1. Experimental Implementation Details
[1 + n)) v
We conduct all the experiments on single workstation with two A5000 (24G) GPUs, with a total of 10 same worksta- tions for all experiments. We follow the EDM2 to build up our software environment. We set the overall train- ing duration for all diffusion- based generative models on the FFHQ dataset and low-shot datasets as 20000kimgs and 40000kimgs, respectively. To reduce the inference time for the diffusion models, we evaluate the model per
|Doo + 0c)
012
Original loss
OOD regultaization
(13) where y is training image and ÿ is the OOD sample that is obtained by applying the NDA on the real samples y. and 7 is a constant 100 that aims to enable the loss function to become stable. (12 and E13( are the denoising score matching for FFHQ and low-shot datasets, respec-
tively. Both (1)) and (13) demonstrate that the denois-
tittpa://githing com/ NV1 abs/edm
13

---

Training Diflusion-based Generative Models with Limited Data
100-shot
Animal-Fard
Method
MA
Obama
Grumpy Cat
Panda
Cat
Dog
StyleGAN2 + Diff-Augment (Zhao al., 2020)
Yes
46.87
27.08
12.06
42.44
58.85
StyleGAN2 + ADA (Karras et al., 2020a)
Yes
45.69
26.62
12.90
40.77
56.83
StyleGAN2 + MAFP (Adaptive)
Yes 41.13
25.87
10.93
38.69
54.15
Table 12, FII score (lower better) on several low-shot datasets (256 x 256). We follow the setting as in Zhao et al. (2020). Massive Augmentation (Cui et al., 2022) is applied to all of the methods. The FIDs are averaged over three runs; all standard deviations are less than 1 1%, relatively.
Compressing Model
Obama
AF-Dog
Both encoder and decoder from SD-EMA (Rombach et al., 2022c) Encoder from SI>-EMA and decoder from SD-MSE (Rombach et al., 2022d)
13.31
12.85
13,20
12.71
Both encoder and decoder from SD-MSE
13.13
12.63
Encoder from SD-MSE and decoder from SD-EMA
13.00
12.48
Table 13. FID score (lower is better) by selecting different pre-trat trainra encoders and decoders in the compressing model in LD-Diffusion. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
500kimgs for the FFHQ dataset and per 1000kimgs for low-shot datasets. The snapshot with the best FID for each method is reported in the experiments.
aligns with the different denoising sampling settings in EDM (Karras et al., 2022). It is evident from the results that setting NFE=79 for the FFHQ dataset and NFE=511 for low-shot datasets in the main paper is reasonable.
B.2. Impact of Different Pre-trained Encoder and Decoder in Compressing Model
B.5. More Ablation Studies of NDA in OOD Regularization
To demonstrate which pre-trained encoder and decoder are suitable for the compressing model in LD-Diffusion, con- duct experiments by selecting different pre-trained encoders and decoders with downsample factors 8 in the compressing model, and the results are shown in Table 13. It is clear that the pre-trained encoder and decoder in SDE-MSE and SD-EMA, respectively, can lead to the best performance. Additionally, we do not select the pre-trained encoder and decoder in SDXL (Podell et al., 2023) because they can only work on full precision, i.e., FP32, rather than half precision, i.e., FP16, which can significantly increase computational resources for all of the experiments.
We conduct an ablation study by selecting different NDA methods, i.c., Jigsaw, Stitching, Mixup (Zhang et al., 2018), Cutmix (Yun et al., 2019) and Cutout (DeVries & Taylor, 2017) as in NDA-GAN (Sinha et al., 2021a), for the pro- posed OOD regularization in LD-Diffusion. The results of the 100-shot Obama dataset are shown in Table 17, Jig- saw achieves better performance compared with other NDA methods in the proposed OOD regularization.
B.6. More Experimental Results on Low-shot Datasets
According to $5.3, we present the comparative results of LD-Diffusion and other methods using Precision and Recall low-shot datasets, shown in Table 15. LD-Diffusion outperforms other diffusion models and always achieves the best Precision compared with other diffusion models. This strongly demonstrates that the generated image distribution produced by LD-Diffusion has a high likelihood of falling into the real data distribution, indicating the superiority of LD-Diffusion. Moreover, Patch Diffusion achieves the high- est Recall score in most cases. This is attributed to the fact that Patch Diffusion applies the patch training and does not utilize pre-trained models to compress difficult information within image distributions. Consequently, Patch Diffusion endeavors to learn such difficult information with various patches, increasing the likelihood of the real distribution falling into the generated image distribution compared to
B.3. More Experiments on MAFP
According to $4.2, the ablation study of Diff-Augment (Zhao et al., 2020) in MAFP on the 100-shot Obama dataset is shown in Table 14. It is clear that Diff-Augment (Trans- lation) achieves the best performance compared with other settings. Therefore, we apply the Diff-Augment (Transla- tion) MAFP for LD-Diffusion.
B.4. More Ablation Studies of Denoising Sampling Number of Score Function Evaluations (NFE)
We conduct experiments by varying the denoising sampling NFE during the inference stage for IDD-Diffusio. The results are presented in Table 16. The selection of NFE
14

---

Training Diflusion-based Generative Models with Limited Data
Method
FID
No Diff- Afff-Augmett in MAFP for LD-Diffusion
13.32
Diff- Diff-Augment (Translation) in MAFP for LD-Diffusion
13.00
Diff- Afff-Augmett (Color) in MAFP for LD-Diffusion
13.36
Diff- Diff-Augment (Cutout) in MAFP for LD-Diffusion
16.15
Diff- Auff-Augment (Translation + Color) in MAFP for LD-Diffusion
13.11
Diff- Aaff-Augment (Translation Color + Cutout) in MAFP for LD-Diffusion
15.01
Table 14. FID score (lower is better) on the 100-shot Obama (256 x 256) by selecting different Dift-Augment cases in the MAFP module LD-Diffusion. Massive Augmentation (Cui al., 2022) is applied to all of the settings. The FIDs averaged over three runs; all standard deviations are less than 1%, relatively.
Obama
Grumpy Cat
Panda
AnimalFace-
AnimalFace-Dou
Method
P
R
P
R
P
R
EDM
0.965
0.380
0.870
0.330
0.874
0.360
0.969
0.306
0.955
188
Patch Diffusion
0.952
0.410
0.861
0.370
0.866
0.392
0.961
0.324
0.943
0.201
LPDM-8
0.991
0.180
0.990
0.100
0.998
0.120
0.997
0.146
0.997
0.772
LD-Diffusion (ours)
0.996
0.100
0.994
0.080
1.000
0.012
0.999
0.103
1.000
0.773
256 low-shot datasets. Massive Augmentation (MA) applied to all methods. The Precision and Recall are averaged over three runs; all standard deviations are less than 1%, relatively.
Table 15. A comparison of Precision (P) and Recall (R) of L.D-Diffusion with other diffusion models on the 256
Liffusion-based generative models with pre-trained models without patch training, thereby resulting in higher Recall score.
Number of NFE FID (100-shot Obama) FID (FFHQ-100)
NFE=35
13.56
32.61
NFE=79
13.32
28.51
B.7. Comparisons of LD-Diffusion with Other Generative Models
NFE=511
13.00
30.11
To further demonstrate the superiority of our proposed LD- Diffusion, we compare with other state-of-the-art (SOTA) generative models with limited data, namely Diffusion- Projected GAN (Wang al., 2023d) and RS-IMLE (Vashist et al., 2024). The results, presented in Table 18, show that L.D-Diffusion achieves performance comparable to other SOTA generative models. Notably, LD-Diffusion attains the best FID scores on the AnimalFace-Cat and AnimalFace- Dog datasets.
Table 16. FID score (lower is better) on the 100-shot Obama dataset (256 x 256) and FFHQ-100 dataset (256 256) by ap- plying different numbers of denoising sampling NEE during the inference stage LD- D--Diffusin. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
NDA method in OOD regularization
FID
B.8. Computational Cost
Jigsaw
13.00
Stitching
13.24
The comparison of the training and inference time of EDM + DA and DD-fuffusion on the 100-shot Obama dataset have been demonstrated in Table 19. LDD-Dffusion can reduce the huge training and inference time compared with EDM +
Mixup (Zhang et al., 2018)
14.12
Cutmix (Yun et al., 2019)
13.47
Cutout Vrees & Taylor, 2017)
13.29
DA.
Table 17. FID score (lower is better) on the 100-shot Obama dataset (256 x 256) by selecting different NDA methods for OOD regularization in LD- D-Diifussion. Massive Augmentation (Cui et al., 2022) is applied to all the settings. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
15

---

Training Diflusion-based Generative Models with Limited Data
100-shot
Animal-Faan Face
Method
Obama
Grumpy
Panda
Cat
Dog
Diffusion Projected GAN (Wang al., 2023d)
10.54
15.13
3.39
17.86
17.22
RS-IMLE (Vashist et al., 2024)
14.00
11,50
3.50
15.90
23.10
LD-Diffusion (ours)
13.00
13.31
4.70
12.77
12.48
Table 18. Compared FID scores (lower is better) of LD-Diffusion with other state-of-the-art (SOTA) generative models on low-shot datasets (256 x 256). We follow the setting used Zhao (2020). Massive Augmentation (MA), i.e., xflipping, is applied all methods. The FIDs are averaged over three runs; all standard deviations are less than 1%, relatively.
Method
Training by 50K images (min) Generating 5K images during inference (min)
EDM + DA
40
165
LD-DDffusion (ours)
7
28
Table 19. The training and inference time of EDM DA and LD- D--Diffusion on the 100-shot Obama dataset (256 x 256). The half precision. i.e. FP16, is applied to all methods. The results are calculated by averaging over ten times on the two NVIDIA A5000 GPUs. All standard deviations are less than I %, relatively.
C. Discussion
DE-GANs have applied this dataset in their experiments, demonstrating its application is reasonable and does not raise any ethical issues.
C.1. Discussion about Pre-trained Models and Limited Data Settings
This paper proposes a compressing model with the pre- trained encoder and decoder models in Rombach et al. (2022b) to constrain the complexity of the denoiser function hypothesis space, thus improving the training of the diffu- sion model with limited data. Some viewpoints may think that applying pre-trained models could break the limited data rules because pre-trained models are trained on a large dataset. We argue this from two aspects. First, applying pre-trained models to generative models to improve their training under limited data has been widely used in GANs. For example, the well-known approaches Projected GAN (Sauer et al., 2021) and vision-aided GAN (Kumari et al., 2022) have both applied the pre-trained models to improve the training with limited data. Second, our goal aims to train diffusion models with limited data from scratch. The pre-trained models applied to the ---Diffusion are based on the pre-trained VAE models, which are not related to the diffusion models. Therefore, the proposed LD-Diffusion is trained with limited data from scratch and does not break the limited data rules.
C.3. Limitation
According to ŞB.1, reduce the computational cost dur- ing the inference stage of diffusion models, evaluate the model per 500kimgs for the FFHQ dataset and per 1000kimgs for low-shot datasets. The snapshots with the best FID for each method are reported in the experiments. This evaluation strategy is fair and can indeed reduce the computational cost. However, can also make the reported results in the paper slightly worse than the best result.
C.2. Discussion of Ethical Issues
This paper applies the 100-shot Obama dataset, i.e., the dataset consists of Obama faces, in the experiment section. This dataset is widely and commonly used without limi- tations in Data Effcient GANs (DE-GANs) research, and we follow the recent existing DE-GAN GANs research to apply them to diffusion models with limited data. Furthermore, lot of recent studies (Zhao al., 2020; Cui et al., 2022: Chen al., 2021; Li et al., 2022: Zhang et al., 2024a)
16


Reviews:
[{"review_id": "Krg0PUyLdO", "reviewer": "Reviewer_ivbF", "summary": "The authors propose a method of training diffusion models with limited data. The proposed method uses a compressing model to constrain the complexity of the denoiser function hypothesis space and a mixed augmentation with a fixed probability (MAPF) strategy to provide more informative guidance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Figure 4 shows how the leaking issues are different for case 1 and case 2. I think it might be helpful in understanding the proposed LD-Diffusion to explain how these differences in leaking issues lead to different strategies for cases 1 & 2, while I also think Figure 4 is critical for the motivation of the proposed method and needs to be in the main body.** \n\nThank you for your valuable suggestions. We have already analyzed that the leaking issue affects performance differently for Cases 1 and 2 in Lines 278-291. Therefore,  we introduce the patch training for Case 1\nand propose a novel Out-of-distribution (OOD) regularization for Case 2 to enhance the training of diffusion models with limited data, respectively. Furthermore, due to space constraints, Sec A.1 and Figure 4 are currently included in the Appendix. In the revised version, we will reallocate space to incorporate them into the main paper, improving clarity and accessibility.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. I checked equations 4~10 and didn't find any obvious issues.", "experimental_designs_or_analyses": "Yes. I checked the experiments on FFHQ and low-shot datasets.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The authors propose a new method LD-Diffusion for training diffusion models on limited data which achieves better performance compared to other diffusion models.\n2. The manuscript is well-written and easy to follow.\n3. The authors conduct comprehensive experiments and ablation studies on FFHQ and low-shot datasets, demonstrating the effectiveness of the proposed compressive module and mixed augmentation with fixed probability (MAFP) module.\n4. The experiments of the compressing model (Figure 2 c-d and Table 4-5) show the fact that training the diffusion model on limited data would be more informative in the compressed low-dimensional denoiser function hypothesis space.\n\nWeakness:\n1. Figure 4 shows how the leaking issues are different for case 1 and case 2. I think it might be helpful in understanding the proposed LD-Diffusion to explain how these differences in leaking issues lead to different strategies for cases 1 & 2, while I also think Figure 4 is critical for the motivation of the proposed method and needs to be in the main body.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x8aTKrwIco", "reviewer": "Reviewer_eEHg", "summary": "This paper focuses on the challenges of training diffusion-based models with minimal data. The authors introduce a novel model called Limited Data Diffusion (LD-Diffusion), which includes a compressing model to reduce hypothesis space complexity and a new data augmentation strategy called Mixed Augmentation with Fixed Probability (MAFP). This approach significantly improves the training and performance of diffusion models on limited data sets by effectively managing the hypothesis space and enhancing data utility. Extensive experiments across several data sets showcase LD-Diffusion's superior performance compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. How sensitive of hyperparameters related to the compressing model and MAFP?** \n\nFor the hyperparameters in the compressing model, we follow the default settings from the pretrained VAE models in Stable Diffusion. Regarding MAFP, we conduct experiments using fixed probabilities $p_1$ and $p_2$, with the results shown in Tables 6 and 7. It is evident that different choices of $p_1$ and $p_2$ can influence performance, but the impact is not significant. Specifically, increasing the optimal $p_1$ and $p_2$ values 2 $\\times$ only affects 2% relative performance. For comparison, in Table 6 of the APA [1] paper, increasing the threshold hyperparameter\n$t$ by a factor of 1.33 results in a 7% relative change in performance. \n\n[1] Deceive d: Adaptive pseudo augmentation for gan training with limited data. NeurIPS 2021.\n\n**Q2. How were the optimal parameters selected?** \n\nBased on our response in Q1, aside from default hyperparameters, LD-Diffusion only uses $p_1$ and $p_2$ in the MAFP, with optimal values determined through experiments (Tables 6 and 7).\n\n\n**Q3. Scalability.** \n\nOur paper primarily focuses on research in limited data scenarios. In this case, LD-Diffusion employs the compressing model to constrain the hypothesis space, which can significantly enhance performance.\nFor data-rich settings, Stable Diffusion has already shown that constraining the hypothesis space can significantly accelerate the training speed with minor and acceptable performance drops, demonstrating the scalability of compressing the hypothesis space.\n\n**Q4. Complexity and Overhead of MAFP.** \n\nExisting methods have already shown that both Diff-Augment and ADA introduce only negligible complexity and computational overhead in GANs. MAFP combines Diff-Augment and ADA with a fixed probability also incurs only a negligible increase in complexity and computational cost. Furthermore, MAFP is a plug-in module, non-technical can easily follow the default augmentation methods and fixed probability values provided in our paper to enhance performance under limited data settings.\n\n**Q5. Generalization Concerns.** \n\nFollowing our response in Q3, our paper primarily focuses on research in limited data scenarios. In Secs 2.2 and 2.3, we have shown that minimizing the total denoising score matching error in diffusion models is challenging under limited data conditions. In such cases, the high complexity and variability of the limited data further exacerbate the difficulty for diffusion models in learning meaningful information. To address this issue, LD-Diffusion employs a compressing model to constrain the hypothesis space complexity, leading to significant improvements under limited data settings. The substantial performance gains observed in our experiments demonstrate that the benefits of reducing hypothesis space complexity far outweigh its potential limitations. The insight of constraining the hypothesis space complexity can be also used to understand the effectiveness of other generative models under limited data, such as Projected GAN, further demonstrating the generalization ability of our approach.\n\n**Q6. Impact of Data Augmentation on Authenticity.** \n\nThe problem you mentioned is the well-known leaking of augmentation issue, i.e., augmentation samples falling out of distribution, which leads to the generation of less authentic samples. In Lines 87 and 268-269 (right side), we have already stated that the design of MAFP aims to increase training samples while avoiding this issue. Additionally, improved training techniques such as OOD regularization can further mitigate this issue.\n\n**Q7. Computational resources required for LD-Diffusion.** \n\nRegarding the computational resources required for training and deploying LD-Diffusion, a single GPU with 24GB of memory is sufficient. This is because LD-Diffusion is benefiting from the use of the proposed compressing model. \n\n**Q8. How does the model perform in terms of training time and resource utilization compared to traditional diffusion models?** \n\nThe comparison of training and sampling time between LD-Diffusion and EDM + DA can be found in Table 20. In terms of computational resources, benefiting from the use of the proposed compressing model, LD-Diffusion reduces GPU memory usage by nearly 8$\\times$ under the same batch size and decreases GFLOPs by approximately 6$\\times$ compared with EDM + DA.\n\n**Q9. Were any control experiments?** \n\nWe have already presented ablation studies to demonstrate the effectiveness of each component in LD-Diffusion in Tables 4 and 5. By progressively removing each component (i.e., improved training techniques, compressing model and MAFP), the results help illustrate the individual contributions of the proposed components to the overall model performance, enhancing the understanding of their importance.\n\n**Due to the 5,000-character limit, The results of Inception Scores can be found in https://anonymous.4open.science/r/Inception-Scores/README.md.**", "claims_and_evidence": "Yes, the claims made in the submission are supported by detailed experimental evidence.", "methods_and_evaluation_criteria": "The proposed method appears reasonable, with evaluation based on FID scores and Improved Precision and Recall metrics, which are appropriate for the problem. However, it would have been beneficial if the authors had also compared baseline approaches using the Inception score.", "theoretical_claims": "The authors do not make any theoretical claims.", "experimental_designs_or_analyses": "How sensitive is the model performance to the choice of hyperparameters, especially those related to the compressing model and MAFP?\n\nWere any hyperparameter optimization techniques employed, and how were the optimal parameters selected?", "relation_to_broader_scientific_literature": "By addressing the practical challenge of training with limited data, the paper targets a significant barrier in the deployment of generative models, making it relevant for many applications in industry and academia.", "essential_references_not_discussed": "The related work section provides a comprehensive review of relevant literature, offering valuable context for understanding the study.", "other_strengths_and_weaknesses": "**Strengths:**\nRobustness through Design Choices: The model's architecture, integrating elements like patch training and OOD regularization tailored for different scenarios (Cases 1 and 2 as defined in the study), shows a deep technical understanding of how to enhance robustness and reliability in generative modeling (that’s what I felt at least).\n\nFlexible Architecture: The flexibility of LD-Diffusion to be implemented with various underlying architectures (such as DDPM, ADM, and NCSN) illustrates its adaptability. This flexibility is a substantial technical strength as it allows the model to be tailored to the specific needs and constraints of different applications and datasets.\n\nTheoretical Insights: The work provides novel theoretical insights into the impact of hypothesis space and training sample size on the denoising score matching error, contributing to a deeper understanding of the underlying dynamics of diffusion models.\n\n\n**Weaknesses:**\nScalability: The proposed method, while effective for limited data scenarios, might not scale efficiently when the amount of data increases substantially. The techniques that compress the hypothesis space could potentially restrict the model's learning capacity in data-rich environments.\n\nComplexity and Overhead of MAFP: While the MAFP strategy enhances training data utility, it also adds complexity and computational overhead to the training process. The need to manage and tune fixed probabilities for different augmentation techniques might complicate the training process, especially for users without extensive technical expertise.\n\nGeneralization Concerns: The compressing model's approach to reducing the hypothesis space complexity could theoretically limit the expressiveness of the generative model. By constraining the complexity, there's a risk that the model might not capture the full variability of data, particularly for complex or highly varied datasets, potentially affecting the generalizability of the model.\n\nImpact of Data Augmentation on Authenticity: The introduction of synthetic data through MAFP could potentially lead to the generation of less authentic samples. The balance between enhancing data utility and maintaining the authenticity and quality of generated outputs is not extensively discussed, which might be critical for applications where fidelity is paramount.", "other_comments_or_suggestions": "The paper is well-written but a few editorial stuff could be improved for better readability. For example, Equations 1, 2, and 3 are hard to read. Texts under Section 3.1 seem odd, i.e. gap between words is not evenly spaced, etc.", "questions_for_authors": "Could they provide more details on the computational resources required for training and deploying LD-Diffusion?\n\nHow does the model perform in terms of training time and resource utilization compared to traditional diffusion models?\n\nWere any control experiments conducted to compare the performance of LD-Diffusion without the use of pre-trained components or the MAFP technique? This would help in understanding the individual contributions of these components to the overall model performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxXkUDpDzn", "reviewer": "Reviewer_aWCW", "summary": "This paper proposes a method to train diffusion models under settings with limited training data.\n\nThe suggested method includes several modifications to the standard diffusion model, including:\n1. Learning diffusion in an embedding space (that comes from a pre-trained VAE model),\n2. Applying penalty for producing bad samples (OOD regularization), and\n3. Applying a combination of data augmentation techniques in the embedding space.\n\nEach of these components exists already, but the specific application of the combination of components to training diffusion models with limited-data is novel, and the problem is of importance.\n\nThe method is evaluated against a baseline diffusion model and shown to perform better based on various metrics, including FID score.\n\nSeveral ablation studies are also presented to systematically evaluate the different components of the proposed method.\n\n## Update after rebuttal\nKeeping same rating. Partially satisfied with some answers, not satisfied with answer to Q5 which I feel is important.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. One issue of the FFHQ-100 dataset.** \n\nThe purpose of designing OOD regularization is to mitigate the potential leaking issue in diffusion models under limited data conditions, thus OOD regularization is deliberately designed and can only be applied to the scenarios where\n aiming to prevent the leaking issue. As illustrated in Sec 2.3, the evaluation of two limited data cases differs, which causes the leaking of augmentation issue to play a different role in the performance of each case (shown in Lines 278-291 and\nAppendix A.1). Under this situation, OOD regularization should be applied to low-shot datasets (Case 1) to mitigate the leaking problem, whereas it should not be applied to the FFHQ dataset (Case 2). Therefore, no tuning is needed for the OOD regularization when using it.\n\n**Q2. Is there any intuition behind using different VAE encoder and decoders?** \n\nWe have already explained the intuition behind using slightly different encoder and decoder models (with the same latent space) in Lines 264-274. Specifically, using the same pre-trained VAE encoder and decoder can lead to a small loss of image details, which limits the variety of detailed information in generated images thus decreasing performance. By applying slightly different encoder and decoder models (with the same latent space), we intuitively alleviate this issue by providing more varied information without affecting convergence. The experiments presented in Appendix B.2 further validate this intuition.\n\n**Q3. What is novel about the theoretical insight?** \n\nAlthough empirical risk minimization is a well-established theory in machine learning, it has not been fully explored in diffusion models. Our paper extends the empirical risk minimization and provides pioneering theoretical insight into diffusion models with limited data. Notably, the proposed theoretical insight provides a different viewpoint compared with the commonly overfitting viewpoint in other generative models (such as GANs and IMLEs) with limited data. Therefore, we believe the proposed theoretical insight can inform future studies on diffusion models and even other generative models with limited data. \n\n**Q4. How is the performance affected if the VAE model is trained on a very different domain (not natural images, e.g.)?**\n\nThe primary purpose of the pretrained VAE model in LD-Diffusion is to project images from the pixel space into a latent space while ensuring effective reconstruction back to the image domain. To achieve this, VAE models are typically trained on large-scale datasets such as LAION-Aesthetics, which provide diverse and high-quality natural images for the training of VAE models. Due to there is lack of such large-scale datasets from other domains for training the VAE model, we conduct experiments comparing EDM + DA and LD-Diffusion with non-natural image domain limited data, i.e., the BRECAHAD dataset containing 162 real-world medical images, to enable the pretrained VAE data and limited data are in the different domains. We follow the official codes of EDM for preprocessing and resizing the BRECAHAD data into 3240 partially overlapping crops with resolution $256 \\times 256$ for avoiding the huge computational cost in the experiments.\n\nThe results clearly illustrate that the pretrained VAE model applied in LD-Diffusion can still achieve significant improvements on non-natural image domain limited data such as medical image datasets. This demonstrates that the effectiveness of LD-Diffusion does not depend on domain similarity, which is different from transfer learning methods.\n\n **Method**         | **FID (BRECAHAD)** |\n|---------------|---------------|\n| EDM + DA      | 45.49         |\n| LD-Diffusion  | **12.21**         |\n\n**Table 1:** Comparison of LD-Diffusion and EDM + DA using FID scores on the BRECAHAD dataset.\n\n**Q5. Comparing with the transfer learning methods?** \n\nAccording to our response in Q4, we argue that directly comparing the proposed LD-Diffusion (a training-from-scratch method) to transfer learning approaches is not a relevant evaluation. Below, we outline three key reasons supporting this perspective: (1) LD-Diffusion is a diffusion model trained from scratch (the parameters in Pretrained VAE models are frozen during training), differing from diffusion models that rely on transfer learning for fine-tuning with limited data. (2) As stated in Q4, the effectiveness of LD-Diffusion is not dependent on the similarity between the pretrained VAE model data and the applied limited data. In contrast, diffusion models that rely on transfer learning for fine-tuning typically require the similarity between the source data domain and the target limited-data domain to achieve improvements. (3) To the best of our knowledge, there are no open-source pretrained unconditional diffusion models that have been trained on the same dataset as the pretrained VAE, such as LAION-Aesthetics. Thus, it is difficult to build up a fair comparison baseline.", "claims_and_evidence": "Claim 1: \nThe authors claim to be providing a novel theoretical insight into training with limited samples “We are the first to propose the novel theoretical insight for diffusion models that the total denoising score matching error is affected by two factors, i.e., denoiser function hypothesis space and the number of training samples”.\n\nI find this insight to not be novel. This is already a well established insight from the paper referred to by the authors. The authors are simply interpreting the statement in their context - this is also not novel because the insight relates to learning of any function using empirical risk minimization, which is trivially applicable to almost any neural network method including this one.\n\nClaim 2:\nThe proposed method differs from related works that use transfer learning for diffusion in limited-data settings.\n\nI believe this claim needs to be substantiated a lot more. The basic premise of the claim is this: works that use transfer learning to finetune diffusion models assume that there is a diffusion model that works in a domain that is not very different from the target limited-data domain. The presented work trains models directly from the limited data. However, the authors are using a pre-trained VAE model to first convert the data to an embedding space. There may be an implicit assumption about the behaviour of the VAE model - which depends on some similarity between the dataset that the VAE was trained on, and the new dataset to learn. If this claim is to be substantiated, it must be shown that there are no requirements on the VAE - i.e. it is trained on a very different domain.\n\nIf this claim cannot be substantiated, then the method must also be compared with the transfer learning based methods. \n\nClaim 3:\nThe proposed method improves over the selected baseline diffusion model in low-data settings.\n\nThis claim is mostly reasonable and is supported by a variety of experiments demonstrating it.", "methods_and_evaluation_criteria": "Methods:\nThe specific changes suggested over the baseline diffusion model are all reasonable and well motivated.\n\nEvaluation criteria:\nThe benchmark datasets and metrics used are sensible and sufficiently comprehensive.", "theoretical_claims": "The theoretical insight presented by the authors is not novel, it is already established.", "experimental_designs_or_analyses": "The experimental design is very good, except for the exclusion of transfer learning baselines.", "relation_to_broader_scientific_literature": "This paper presents a combination of several existing components in training of diffusion models. All of the components have been studied in semi-related contexts:\n1. Training diffusion models over embedding spaces\n2. The augmentation techniques Diff-Augment and NDA\n3. OOD regularization\n\nHowever, the novelty lies in the following:\n1.Regularization has been attempted with GANs, not with diffusion models. Therefore, the empirical behaviour of regularization is not well understood.\n2. The evaluation of these techniques to train diffusion models in limited data settings has never been done.  \n\nWhile I have questions about the assumptions regarding the VAE and comparisons with transfer learning related methods, the paper does present a recipe that works under the stated assumptions. It is certainly useful.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The experimental section is very comprehensive (for the specific baseline method used). It covers tests on several datasets.\n2. The ablation studies are really nice and help give valuable insights.", "other_comments_or_suggestions": "1. There is one issue regarding the evaluation of the FFHQ-100 dataset. The evaluation labeled as “Case 1” suffers with the OOD regularization. Does that suggest that OOD regularization cannot be applied as is everywhere? Is that something that needs to be tuned?\n2. Is there any intuition behind using different VAE encoder and decoders?", "questions_for_authors": "1. What is novel about the theoretical insight given over the already published insight that is referred to in the paper?\n2. How is the performance affected if the VAE model is trained on a very different domain (not natural images, e.g.)?\n3. How does the performance compare with the transfer learning methods, if the initial diffusion model is trained on the same dataset that the VAE is trained on?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bv2RlLmfWb", "reviewer": "Reviewer_txiC", "summary": "The paper identifies that many existing diffusion models depend on large corpus of training image, and attempt to offer solutions to enable high quality training with only a fraction of the dataset size. The authors reveal the different cases for how a diffusion model can be trained with limited data, and construct their solution around two main components: latent compression and mixed augmentation with fixed probability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The paper's method section could be expanded on expense of the rest of the paper.** \n\nDue to the 8-page limitation of ICML submission, we have included additional details of our proposed LD-Diffusion in the Appendix. We will extend the Sections. 4.1, 4.2, and 4.3 based on the Appendix to provide more details of our proposed LD-Diffusion in the revised version (9 pages).\n\n**Q2. The OOD regularization is presented without sufficient theoretical justification.** \n\nThe OOD regularization is one of the improved training techniques in our paper, which is mainly based on empirical experiments to further avoid the leaking issue during training. Specifically, OOD regularization aims to maintain the core properties of the diffusion models while introducing the necessary adjustments to enhance performance under limited data settings. We have conducted complete experiments to demonstrate that the changes to the loss function caused by OOD regularization do not significantly affect the stability or other important properties of the diffusion models. Additionally, as highlighted in the results and ablation studies, the LD-Diffusion demonstrates further improvement with OOD regularization on low-shot datasets. \n\n**Q3. The selection of VAE should be clearly explained in the main paper.** \n\nWe have already explained the reason for selecting the pre-trained encoder and decoder models from similar but slightly different VAE models (with the same\nlatent space) for the compressing model in Lines 264-274 (left). Furthermore, we conduct an ablation study for this selection in  Appendix B.2. We will add more details from the Appendix B.2 to the main paper in the revised version.\n\n**Q4. I do not see applying latent diffusion model as contribution of this work.** \n\nAlthough the latent diffusion model has been applied in some existing approaches, these studies primarily focus on accelerating training speed and reducing computational cost. In contrast, our work is the first to apply the latent diffusion model based on theoretical analysis (refer to Sec 3.2) to improve the training of diffusion models with limited data. We believe this presents a novel perspective distinct from existing methods and offers valuable insights for future research.\nFurthermore, based on Reviewer aWCW comments: The evaluation of applying these techniques (including latent diffusion model) to train diffusion models in limited data settings has never been done. This reinforces the viewpoint that applying the latent diffusion model to enhance diffusion model training with limited data can be regarded as a meaningful contribution in our paper.\n\n**Q5. The choice of NFE in the LD-Diffusion.** \n\nWe have already provided a detailed ablation study of how to choose the NFEs for two cases in Table 16 in the Appendix. Specifically, the selection of NFE aligns with the denoising sampling NFE settings in EDM. Although the results in Table 16 show that different NFE selections do not significantly impact performance and are therefore not crucial for high-quality generation, it is evident from the results that setting NFE=79 for the FFHQ dataset and NFE=511 for low-shot datasets is reasonable and achieves the best practical. As shown in Table 16, if the NFE is selected as 79 for the low-shot datasets, it can decrease the performance accordingly.", "claims_and_evidence": "- The claims made in the submission are supported.\n - The paper's method section could be expanded on expense of the rest of the paper. Specifically, I found the core of the algorithm in secs. 4.1 and 4.2, and 4.3 are short and hard to follow, and crucial details are deferred to the appendix. The main manuscript should be self-contained.", "methods_and_evaluation_criteria": "Yes, the method and evaluation make sense for the problem.", "theoretical_claims": "- The OOD regularization is presented without sufficient theoretical justification. The diffusion training is connected to the noisy distribution score, chaning the loss in this fashion may break some of these properties.", "experimental_designs_or_analyses": "- The experiments seem sound to me.\n- The results shown in this work are encouraging.", "relation_to_broader_scientific_literature": "- As far as I understood, the selection of vae models out of the ones trained in previous works is done using a small ablation. This should be clearly explained in the main paper. Also, as using latent diffusion is standard practice, I do not see this as contribution of this work.", "essential_references_not_discussed": "I cannot think of any.", "other_strengths_and_weaknesses": "Strengths:\n - The limited fata regime is a key problem with diffusion models, as many distributions contain limited examples. Thus managing to reach exisiting sampling quality with much fewer samples can have high impact on the field.\n - The paper is comprehensive, offering modified solutions depending on the framing of the problem", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "- Could the authora shed light on their choice to use 511 NFEs for generation with the small datasets? Is this crucial for high quality generation? How do the results change if the original 79 steps are used instead?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses an important problem of training diffusion models with limited data. The authors first show in theory and also in practice how the denoising score matching loss is affected by two factors, including the latent space and the number of training samples. With such insights, the authors propose compressing to a low-dimensional space and a mixed augmentation scheme. 

While latent diffusion model is not new, this paper investigates latent space from the perspective of data-efficient diffusion models. Reviewers appreciate that the method can be applied to EDM and other variants of diffusion models. Experiments are convincing with large margin compared to previous methods. This paper also shows how to apply the method to two different settings.

Reviewer aWCW raised concerns about the lack of comparison to transfer learning. The setting of training from scratch is more challenging than transfer learning, i.e., fine-tuning. The authors mentioned the practical difficulty of training an unconditional diffusion model on the data that VAE is trained on.

This paper has advanced the state of the art of diffusion model with limited data. However, the framework seems to be a mix of variants of existing components such as compressing model, Diff-Augment and ADA augmentation, and patch training. The AC recommends to accpe this paper, but the authors should highlight their contributions in the final version.
Decision: Accept (poster)