Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Authors: Zhendong Wang
Yifan Jiang
Huangjie Zheng
Peihao Wang
Pengcheng He
Zhangyang Wang
Weizhu Chen
Mingyuan Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
training efficiency
data efficiency

Abstract: Diffusion models are powerful, but they require a lot of time and data to train. We propose Patch Diffusion, a generic patch-wise training framework, to significantly reduce the training time costs while improving data efficiency, which thus helps democratize diffusion model training to broader users. At the core of our innovations is a new conditional score function at the patch level, where the patch location in the original image is included as additional coordinate channels, while the patch size is randomized and diversified throughout training to encode the cross-region dependency at multiple scales. Sampling with our method is as easy as in the original diffusion model. Through Patch Diffusion, we could achieve $\mathbf{\ge 2\times}$ faster training, while maintaining comparable or better generation quality. Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e.g.$, as few as 5,000 images to train from scratch. We achieve outstanding FID scores in line with state-of-the-art benchmarks: 1.77 on CelebA-64$\times$64, 1.93 on AFHQv2-Wild-64$\times$64, and 2.72 on ImageNet-256$\times$256. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Patch-Diffusion.

Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Zhendong Wang 1.2, Yifan Jiang', Huangjie Zhengl. Peihao Wang', Pengcheng He?, Zhangyang Wang', Weizhu Chen², and Mingyuan Zhou' The University of Texas at Austin, Microsoft Azure Al
Abstract
Diffusion models are powerful, but they require a lot of time and data to train. We propose Patch Diffusion, generic patch-wise training framework, to sig- nificantly reduce the training time costs while improving data efficiency, which thus helps democratize diffusion model training to broader users. At the core of our innovations is new conditional score function at the patch level, where the patch location in the original image included as additional coordinate channels, while the patch size is randomized and diversified throughout train- ing to encode the cross-region dependency at multiple scales. Sampling with method is easy in the original diffusion model. Through Patch Diffu- sion, we could achieve 2 2x faster training, while maintaining comparable or better generation quality. Patch Diffusion meanwhile improves the performance diffusion models trained on relatively small datasets, e.g., as few as 5,000 images to train from scratch. We achieve outstanding FID scores in line with state- of-the-ar art benchmarks: 1.77 on CelebA-64x x46, 1.93 on AFHQv2 Wild-64 64, and 2.72 ImageNet-256x2556 We share code and pre-trained models com Zhendong--dn--nn Wang Slang/Pacch-iiffusiii Diiffusion
Introduction
Training
Gondicar
11.4]
Res Down Blodk
Paltivubbr
Coodinats
Goardrates
Ro.Domm
floce
Sampling
Fanilo- blo3d
Rusio Block2
Figure 1: Illustration Patch Diffusion on training and sampling.
Diffusion models 50 15
have become the new facto generative AI (GenAl) models. Song ct al. 54 unite the diffusion models into the framework of score-based generative models 17
57
52 53
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Diffusion models have found great success in wide range of applications, including unconditional
image synthesis [T] 51 40 22], text-to-image generation [39 47
45
61, audio generation
24
uncertainty quantification
[12], and reinforcement learning
19
59
Although diffusion models are stable to train and powerful in capturing distributions, they notoriously slow in generation due to the need to traverse the reverse diffusion chain, which involves going through the same U-Net-based generator network hundreds or even thousands of times [51 60). As such, there is great interest in improving the inference speed of diffusion models, leading to many
improved samplers such as DDIM (5N, TDPM 677, DPM-Solver 35 36
and EDM DDM--lmpling 22]
However, diffusion models are also notoriously expensive and data-hungry to train. They require large datasets and many iterations to capture the tigh-dimensional and complex data distributions. For instance, training DDPM 15 on eight V100 GPUs for the L.SUN-Bedrond
663 dataset at
resolutions of 64 64 and 256 x 256 takes approximately four days and over two weeks, respectively. saate-of-the-ar diffusion model in Dhariwal & Nichol [10] consumes 55-1000 V100 GPU days to produce high-quality samples. The training costs grow exponentially as the resolution and diversity of the target data increase. Moreover, the best-performing models 44 45 rely on billion-level image datasets such OpenImages 26 and LAION 48
which are not easily accessible or scalable.
While some most exciting GenAl results are arguably accomplished by training diffusion models with enormous computational power and data (often owned or led by large corporations), the prohibitive time and data scale required to train competitive diffusion models have presented critical bottleneck for democratizing this GenAI workhorse technology to the broader research community who generally lack access to such high-end privileged resources. This problem of democratizing diffusion model training is becoming pressing, yet largely overlooked so far.
To democratize diffusion model training. we propose patch-wise diffusion training (Patch Diffusion), a plug-and-play training technique that is agnostic to any choice of UNet [13 22 architecture, sampler, noise schedule, and so on. Instead of learning the score function on the full-size image for each pixel, propose to learn conditional score function image patches, where both patch location in the original image and patch size are the conditions. Training on patches instead of full images significantly reduces the computational burden per iteration. To incorporate the conditions of patch locations, we construct a pixel-level coordinate system and encode the patch location information as additional coordinate channels, which concatenated with the original image channels as the input for diffusion models. We further propose diversifying the patch sizes in a progressive or stochastic schedule throughout training, capture the cross-region dependency multiple scales.
Patch Diffusion aims to significantly reduce the training time costs while improving the data efficiency of diffusion models. Sampling with our method is as easy as it in the original diffusion model: we compute and parameterize the full coordinates with respect to the original image, concatenate them with the sampled noise, and then reverse the diffusion chain to collect samples, as illustrated in Figure Through Patch Diffusion, we could achieve 2 2x faster training, while maintaining com- parable or better generation quality. We also observe that Patch Diffusion improves the performance of diffusion models trained on relatively small datasets, e.g., clearly superior generation results when training with as few as 5,000 images from scratch. We summarize main contributions follows:
The first patch-level training framework, generally applicable to diffusion models, that targets saving training time and data costs.
Novel strategies of patch coordinate conditioning and patch size conditioning/sceedulngg to balance training efficiency and effective global structure encoding.
Competitive results while generally halving the training time, and notable performance gains at small training data regimes.
2
Related Work
Preliminaries for diffusion models. Diffusion probabilistic models [50 15]
first construct a
forward process that injects noise into data distribution, then reverse the forward process to reconstruct it. The forward process iteratively applies noise to the given images, whereas the reverse process iteratively denoises a noisy observation. After the origin of the diffusion model, many efforts have
been made to apply it to various downstream tasks. Ho al. [16] and Vahdat al. (56) propose a hierarchical architecture to stabilize the training process of diffusion models and further mitigate

---

memory cost issues. Dall-E-2 44 first introduces the diffusion model to the text-to- text-to-image generative task and achieves remarkable success. Later, Saharia al. 447 found that increasing the parameter of the language model improves both sample fidelity and image-text alignment much more than increasing the size of the image model.
Data-efticient training in generative models. Prior to the study of diffusion models, many explorations have been conducted on limited data or even the few-shot setting of generative model's training schemes, e.g., Generative Adversarial Networks (GANs). To avoid expensive data collection, Zhao al. [66] propose differentiable augmentation on both the generator and discriminator of GANS and achieve acceptable performance with only 10% data. Karras et al. [2I] apply adaptive augmentation strategy for the setting of limited data to prevent information leakage. Wang al. [60] propose to train GANs with an adaptive forward diffusion process that injects different levels of noise during training, significantly improving GANs' performance in small datasets. Chen et al. [6] discover the lottery ticket mask of a specific GAN architecture that allows further reducing the training data. Tseng et al. [55] propose to add regularization on top of the existing data augmentation strategy and stabilize the brittle training process of adversarial learning. Others explore the domain adaptation of GAN frameworks when only a few data from target domains are available, e.g.. by using elastic weight consolidation 30 or cross-domain correspondence
42]
Meanwhile, recent works also start exploring the few-shot adaptation of diffusion models [37] DreamBooth [46] uses four frontal images of specific subject to finetune a pre-trained diffusion model and additionally set special indicator in the prompt, where the obtained model is able to generate various examples with the same identity. Later, Zhang et al. [55] reduce the number of required images to only a single example when finetuning the diffusion model, but still generate visually impressive photos given various prompt inputs. However, the above works do not fully explore the setting of training a diffusion model from scratch. Although a few single-image diffussion algorithms [58 41 25 only need a single training image, they can only produce variations of the same image and hence focus drastically differently compared to our proposed method.
Rescirce training in generative models. To address the issue of mode collapse given limited training data, Lin al. [32) propose Anycost-GAN that is able to generate examples with different levels of cost during inference. However, their proposed techniques do not help save the training cost, and may even take longer time to optimize the SuperNet 4 Coco-GAN 31 firstly introduces the patch-wise training scheme on GAN frameworks. However, since their discriminator still needs to combine several generated patches together, its effectiveness on memory saving not distinguishable. Lee et al. [27] further apply the patch-wise training scheme to the INR-GAN [49 framework. Although their approach is able to save GPU memory cost, it will hurt the quality of generated examples (e.g., increasing FID on the FFHQ dataset from 8.51 to 24.38). In the meantime, since the low-resolution images are generally more accessible compared to high-resolution ones, Chai al. [5] adopt any-resolution framework which is able to blend data with different resolutions together to feed the training of GANs, and therefore reduce the partition of the high sigh-resolution part. Despite its superiority of data efficiency, it does not particularly discuss the training costs and the effectiveness of its methodology remains unknown and unexplored for diffusion models.
A handful of works try to tackle the issue of huge training and inference costs in diffusion models. Rombach et al. 45 perform diffusion in latent space instead of the pixel space, which largely reduces the training and inference cost. Other works [35] 36 51 2 study the fast sampling strategy at the inference stage, which does not directly help speed up the training process. Our proposed approach is orthogonal to the aforementioned works and can serve as a plug-and-play module.
Simultancously with our research efforts, Ding et. al. [T] concurrently developed another model of butch-based denoising diffusion. Their model excels in memory memory-efficient high high-resolution image synthesis while avoiding the introduction of boundary artifacts. Central to their methodology the introduction of a novel feature collage technique, achieved through systematic window-sliding process, which effectively enforces spatial consistency.
3 Patch Diffusion Training
We first briefly review diffusion models from the perspective of score-based generative modeling and then present our method details. Suppose we are given a dataset [ where each datapoint is independently drawn from underlying data distribution p(æ). Following Song et al.
3

---

[54, we consider a family of perturbed distributions Po (ã x) = (²; æ of) obtained by adding independent, and identically distributed (i.i.d.) Gaussian noise with standard deviation to the data. With sequence of positive noise scales min do < < < ... < OT max- forward diffusion process, Po, (ar) for i = ...... T, is defined given that Omin small enough such that Pomin (a:) = p(z) and d max is large enough such that Pomax (a:) R (0, dmay T). The target of diffusion models is to learn how to reverse the chain and restore the data distribution p(æ).
Further generalizing to an infinite number of noise scales [4 T + 00, the forward diffusion process could be characterized by a stochastic differential equation (SDE) and further converted to an ordinary differential equation (ODE). The closed form of the reverse SDE is given by
dæ = (a, (((a,))-g{(1))) (t) log (a)) dt + g(t)dw,
where f(:,t= R F is valued function called the drift coefficient, g() R is real-valued function called the diffusion coefficient, dt represents a negative infinitesimal time step, w denotes standard Brownian motion, and du be viewed infinitesimal white noise. The corresponding ODE of the reverse SDE is named probability flow ODE (54), given by
da [ 0) - (L) log (a:) dt.
(2)
As shown in Equations T and 2 solving the reverse SDE or ODE requires us know both the terminal distribution Por (x) and score function V Po,
Po. (z). By design, the former Por (x) is close white Gaussian noise distribution. The analytical form of V = log Po, is generally intractable, and hence we learn a function se (x, o²) parameterized by neural network to estimate its values. Denoising score matching [57 51 is currently the most popular way of estimating score functions applied in diffusion models. After learning the estimated score function sa oi), we can obtain an estimated reverse SDE or ODE collect data samples from the estimated data distribution.
Next, we introduce our patch diffusion training in three subsections. First, we propose to conduct conditional score matching on randomly cropped image patches, with the patch location and patch size as conditions, to improve the efficiency of learning scores. Second, we introduce pixel coordinate systems to provide better guidance on patch-level score matching. Then, we show by using our method, for each reverse step, we could do sampling globally as easily as the original diffusion models, without the need to explicitly sample separate local patches and merge them afterwards.
3.1 wisc Score Matching
Following denoising score-matching in Karras et al. [22.. we build a denoiser, Da ot). that mini- mizes the expected L2 denoising error for samples drawn from data distribution p(a:) independently for any oe:
Ee E. Voo + e,))) (3)0))+ =0
Thus we have the score function as
so( (Do
/
Instead of conducting the score matching on the full images, we propose to learn the score function on random-size patches. As shown in Figure for any æ ~ p(æ), we first randomly crop small patches Lij. where use (i,j), the left-upper corner pixel coordinates, to locate each image patch, and s to denote the patch size, e.g., s = 16. We conduct denoising score matching on image patches with the corresponding patch locations and sizes as the conditions, expressed
Do( oe, 1, s) Zi.j. 12
(4)
where æig..- Dijj + E and U/ denotes the uniform distribution the corresponding value range, e.g., i ~ [-1, 1]. Then, our conditional score function derived from Equation (3). se (x or, i, j, s), is defined on each local patch. We learn the scores for pixels within each image patch conditioning on its location and patch size.
With Equation (4), the training speed is significantly boosted due to the use of small local patches. However, the challenge now lies in that the score function sg (x, ot, i, 8) has only seen local patches and may have not captured the global cross-region dependency between local patches, in other words, the learned scores from nearby patches should form coherent score map to induce coherent image sampling. To resolve this issuc, we propose two strategies: 1) random patch sizes and 2) involving

---

a small ratio of full-size images. Our training patch size is sampled from a mixture of small and large patch sizes, and then the cropped large patch could be seen as a sequence of small patches. In this way, the score function so learns how to unite the scores from small patches to form a coherent score map when training on the large patches. To ensure the reverse diffusion converges towards the original data distribution, in some iterations during training, full-size images are required to be seen.
We further provide a theoretical interpretation in Appendix A to help understand our method, and conduct a detailed empirical study in Section 4.1 to show the impact of the ratio of full images.
3.2 Progressive and Stochastic Patch Size Scheduling
To strengthen the awareness of score function in cross-region dependency, we are motivated to propose patch-size scheduling. Denote the ratio of iterations that takes as input the full-size images as p, and the original image resolution as R. We propose use the patch options, follows
p
when R,
~ P3
[1 p)
when s R//2,
(5)
(1 p) when 5 = R//4.
Two patch-size schedulings could be considered. 1) Stochastie:
During training, we randomly sample s ~ Ps for each mini-batch with the probability mass function defined in Equation (5). 2) Progressive: We train our conditional score function from small patches large patches. In the first 5 p) training iterations, we the patch size as 8 = R/ /4, while in the second 5 - p) training iterations, s = R/ 2 is applied. Finally, we train the models on full-size images for p ratio of the total iterations.
Empirically, we find that p 0.5 with stochastic scheduling reaches sweet point in the trade-off between training efficiency and generation quality, as shown in Figure 3
One natural question is why the denoiser Dg could handle image patches of varying sizes. We note the UNet [2] architecture is fully designed with convolutional layers, and the convolutional filters capable of handling any resolution images by moving themselves around the inputs. Hence, our patch diffusion training could be regarded as a plug-and-plas training technique for any UNet-based diffusion models. The flexibility of UNet on resolutions also makes our sampling easy and fast.
3.3 Conditional Coordinates for Patch Location
Motivated by COCO-GAN 31, to further incorporate and simplify the conditions of patch locations in the score function, we build up pixel-level coordinate system. We normalize the pixel coordinate values [-1, 1] with respect to the original image resolution, by setting the upper left corner of the image (-1, 11) while the bottom right corner (1, 1).
As shown in Figure 1] for any image patch Fi.j.s. we extract its and pixel coordinates as two additional channels. For each training batch, we independently randomly crop each data sample with a sampled patch size for the batch and extract its corresponding coordinate channels. We concatenate the two coordinate channels with the original image channels to form the input of our denoiser Dg. When computing the loss defined in Equation ( we ignore the reconstructed coordinate channels and only minimize the loss on the image channels.
The pixel-level coordinate system together with random patch size could be seen kind of data augmentation method. For example, for an image with resolution 64 x 64, with patch size s = 16, we could have (64 16 + 1)2 2401 possible patches with different locations specified. Hence, we believe training diffusion models on patch-wise would help the data efficiency of diffusion models, in other words, with our method, diffusion models could perform better on small datasets. We validate this hypothesis through experiments in Section 4.4
3.4 Sampling
By utilizing our coordinate system and the UNet, we are able to easily accomplish the reverse sampling defined in either Equation ( or Equation (2). As we have shown Figure we compute and parameterize the coordinates for the full image, and concatenate them together with the image sample from last step as coordinate conditions at each reverse iteration. We abandon the reconstruction output of coordinate channels at each reverse iteration.
5

---

We provide theoretical interpretation of Patch Diffusion Training in Appendix A
4 Experiments
We conduct five sets of experiments to validate our patch diffusion training method. In the first subsection, we conduct an ablation study on what impacts the performance of our method. In the second subsection, diffusion model baselines on commonly-used benchmark datasets. Thirdly, we show that our method could also help improve the efficiency of finetuning large-scale pretrained models. Then, we show that patch diffusion models could achieve better generation quality on typical small datasets. Finally, evaluate the out-painting capability of patch diffusion models.
compare method with its backbone model and other state-of-the-a
Datasets. Following previous works 222
23
we select CelebA (~200k images) 34 FFHQ
(70k images) (20). LSUN (~~200k images) 663 and ImageNet (~11.2 million images) [9] as our large datasets, and AFHOv2-Ca/Dog/Winn (~5k images in each of them) 7 as our small datasets.
Evaluation protocol. We measure image generation quality using Fréchet Inception Distance (FID) [13]. Following Karras et al. 220 22], we measure FID) using 50k generated samples, with the full training set used as reference. We use the number of real images shown to the diffusion models to measure our training duration [22]. Unless specified otherwise, all models are trained with a duration of 200 million images to ensure convergence (these trained with longer or shorter durations are specified in table captions). For the sampling time, we use the number of function evaluations (NFE) as a measure.
Implementations.
We implement our Patch Diffusion on top of the current state-of-the-arr
Unet-based diffusion model EDM-DDPM+ [22] and EDM-ADM [22]. EDM-DDPM++ is our default backbone model for training low-resolution (64x64) datasets, while EDM-ADM coupling with Stable Diffusion
45 latent en/decoders is our backbone model for training high-resolution (256 x 256) datasets. We inherit the hyperparameter settings and the UNet architecture from Karras al. 222 We implement random cropping independently on each data point in the same batch, and the corresponding pixel coordinates are concatenated with the image channels to form the input for the UNet denoiser. When computing the diffusion loss at any level, we ignore the reconstructed pixel coordinates. We adopt the EDM-Sane -ampligg strategy with 50 deterministic reverse steps in the inference stage, for both the baseline model and ours.
4.1 Ablation study
We provide ablation studies to investigate what may impact the performance of Patch Diffusion.
Impact of full-size images. We first study the main effect of p, the ratio of full-size images during training, regarding the quality of generated examples and the training cost. We conduct experiments on a P gird, [0.0, 0.1, 0.25, 0.5, 0.75, 1.0). Note here when p = 0.0, only patch images and their pixel coordinates are available during the training process, and when p = 1.0, patch training becomes the same as the standard training procedure of diffusion models. We train all models on 16 Nvidia V100 GPUs with a batch size of 512 for a duration of 200 million images.
We show the results in Figures 2 and 3 In the extreme case p= 0.0, the conditional score function is only trained on local patches, e.g., 16 x 16 and 32 x 32 image patches, without the knowledge of how the full-size images look like. The FID is reasonably unsatisfactory while we still observe some coherently sampled images, such as the third and fourth faces shown in the 1st row of Figure 3 This observation validates our idea that using a mixture of large and small patch sizes could help the conditional score function to capture cross-region dependency. The learned small patch scores are guided by large patch score matching to form a coherent score mapping. Then, when p is greater than O. even if is as small as 0, the generation quality is dramatically improved in terms of FID. We reason this as that the involvement of full-size images provides the global score for patch-wise wise score matching, even though only small ratio of the global score could guide the score function to learn towards it. As the p increases, we could see the FID also becomes better, and we hypothesize this is because the conditional score function converges better towards the local minimum due to more global score guidance. However, the improvement of FID gained from increasing p is not unlimited. We also observe that when p is large enough, such p 0.75, the FID values converge

---

Figure FID results on C41ebA-64x64 with different p values.
Metrie
pC0.
p=0.1
p=0.25
p=0.5
p=0.75
p=1.0
14.51
2,10
1.77
1.66
Training Time
Ablation study on ratio p
FID
Triining Time
Paimin Time (in hours)
-10
FID
6--
Figure 3: Ablation study dataset CelebA-64x66 for the ratio of full-size images p. We generate from the trained models with different p. and show the generated images for a fixed set of seeds in the p increasing order from top bottom.
to the minimum level, which indicates that sparsely using full-size images during training could be sufficient to guide the conditional score-matching to converge.
On the other hand, larger means more training cost and longer training time. Hence, we pick the sweet point shown the line plot, p 0.5, which provides good trade-off between generation quality and training efficiency, for our following experiments.
Patch size scheduling. Further, we investigate the impact of patch size scheduling, a stochastic or progressive way. Note usually the training diffusion models applies learning rate decay. This may hurt the performance of progressive scheduling, since the training in the front stage is totally based on local patches while has a large learning rate. We report FID comparison 1.66 (stochastic) v.s. 2.05 (progressive) on CelebA-64 x64 and 3.11 (stochastic) v.s. 3.85 (progressive) on FFHQ-64 x64. Therefore, unless otherwise specified, we employ stochastic patch size scheduling, as discussed in for the following experiments.
Section 3.2
4.2 Experiments on Large-scale Dataset
In this section, we aim to compare our Patch Diffusion Model (PDM) with other state-of-the-ar diffusion model baselines in terms of both generation quality and training efficiency. As shown in T Patch Diffusion generally works well in terms of FID while having significantly reduced training time. Note, the FID results of model and backbone based on 50 deterministic reverse steps while the other baselines need a much larger number of reverse steps and are much slower in sampling.
We also combine our method with the Latent Diffusion 45 in training time cost for high-resolution image synthesis, such as [63] and ImageNet-256- x 256 [ and we denote it as Latent Patch Diffusion Model (LPDM). We borrow the pretrained image encoder and decoder from Stable Diffusion (45), encode the original images to a smaller latent space and then apply our patch diffusion training. Note the pretrained auto-encoder is not specifically trained for LSUN/ImageNet, which may limit the performance. We use the same latent diffusion model but without patch diffusion training under our codebase as a baseline, and we denote the implementation as LDM-ADM.
framework to achieve dramatic reduction
256
datasets, we train unconditional sampling, while for ImageNet dataset, we train conditional sampling. Following the classifier-free diffusion guidance (CFG) [14], during training, we randomly drop 10% of the given class labels, and during sampling, we use strength 1.3 for applying CFG, c.3=
For SSUN-Bedroo/CCCe

---

Figure 4: Randomly generated images from Patch Diffusion (EDM-DDPM++ backbone) trained on CelebA-64x64 and EFHQ-64x664 and Latent Patch Diffusion (EDM-ADM backbone) trained on ImageNet-256xx256
A THe SEMPA LUV Ohe
LECH Locvlg Braft restie LETEE chancel
Sar Vilaa Autane Lightfirr
beld Ford
chiedian Carteaediiee Harhee
Figure 5: Finetuning Results of ControlNet with Patch Diffusion Training. We finetune ControlNet on the HED map to image generation task from Stable Diffusion checkpoints with 20k steps.
Figure 6: Randomly generated images from Patch Diffusion (EDM-DDPM++ backbone) trained on small datasets, AFHQv2-Cat-64xe AFHQy2-Dog-64x660 and AFHQQ2-Wiild-4xx-
We present the FID and training cost in Table 2 and Table 3 for LSUN and ImageNet datasets, repsectively. Patch Diffusion notably surpasses both the baseline and previous state-of-the-art Unet- based diffusion models in both generation quality and training efficiency. We show a number of uncurated image samples generated by Patch Diffusion Figure and Appendix Qualitatively, the images generated by Patch Diffusion exhibit photo-realism and are rich in variety.

---

Table 1: We report the FID and T-Time (training time in hours) on both CelebA-64x 64 and FFHQ- 64x dataset. For our model and our backbone model EDM 22 with batch size 512.
we train them on 16x V100 GPUs
Algorithm
NFE,
CelebA-64x64
FFHQ-64x64
Train- Train-Time
DDIM (5
1000
3.51
48h
DDPM 15
1000
3.26
48h
NCSN+ 14
1000
3,25
48h
PNDM 3
1000
2.71
48h
+DPM++
1000
1.90
48h
Soft Diffusion
300
1.85
48h
EDM-DDPM++
22
50
1.66
2.60
48h
PDM- DDM-DDPM++ (ours)
50
1.77
3.11
24h
Table 2: We report the FID and Train-Time (training time in hours) on LSUU-BBddroom2266225 and LSUN-Church-256 256 datasets.
Algorithm
NPEL
LSUN-Bedroom
LSUN--hurch
Train-Time
DDIM
10.58
6.62
days
DDPM
1000
7.89
4.90
14 days
LDM-ADM
50
4.32
4.66
8 Days
LPDM-ADM (ours)
50
2.75
2.66
4 days
Table 3:
We report the FID, sFID, IS, Precision and Recall on ImageNet-256x235 datasets. We also report the GFLOPs columns, one without classifier-ff free guidance and one with, marked with and v. respectively. All methods NFE 250 steps for reversing the learned diffusion process.
measure of compute cost for each algorithm. Each metric is presented in two
Evaluation Metrie
FID
SFID
Ist
Prec.1
Rec.
Cost
Clussifier-Fre free guidance
X
X
X
/
GFLOPs
ADM 10
10.94
4.59
6.02
5.25
100.98
186.70
0.69
0.82
0.63
0.52
1120
ADM-Upsamplee
7.49
3.94
5.13
6.14
127.49
215.84
0.72
0.83
0.63
0.53
LDDM-8 45
15.51
7.76
79.03
209.52
0.65
0.71
0.63
0.62
53
LDM-4
10.56
3.60
103.49
247.67
0.35
0.48
LPDM-ADM cfg==13))
7.64
2.72
5.36
4.86
130.23
243.25
0.73
0.84
0.63
0.57
4.3 Experiments on Finetuning
We evaluate Patch Diffusion in finetuning large-scale pretrained diffusion models. We plug it into ControlNet (64] as one example. We use the data proposed by Brooks al. [3] as our base datasets and extract the HED maps by the HED boundary detector [ as the input controls. We then finetune ControlNet on the HED map to image generation task from the Stable Diffusion "v1-5" checkpoint with 20k steps. We show the qualitative generation from HED maps to images in Figure We observe that patch diffusion can be effectively applied to fine-tuning without compromising performance, while also enhancing training efficiency by approximately two times.
4.4 Experiments on Limited-size Dataset
We further investigate whether coordinate- -uiiddddeegddd score matching could improve the data efficiency of diffusion models. Note by doing random cropping with different patch sizes, even one single image could be extended to thousands of patch samples, which helps overcome the overfitting issue.
Specifically, we conduct experiments on three popular small datasets, AFHQv2-C& -Ct, -Dog, and -Wild, each with as few as around 5k images [7]. We train our Patch Diffusion and the baseline approach EDM-DDPM++ from scratch for duration of 75 million images. We compare the training cost and generation quality between different methods. As shown in Table we observe that by using patch score matching, our model consistently outperforms the baseline model across all three datasets in terms of FID, while achieving 2 2x faster training at the same time. We show number of uncurated

---

Table 4: Results reported on the limited-size dataset (AFHQ-v2). For all models, we train them on 16x V 100 GPUs with batch size 512 for duration of 75 million images.
Data( 5k images)
Algorithm
FID |
NFE
Train- Time
EDM-DDPM++ 22
4.60
50
18h
AFHQv2-Cat
Patch Diffusion (ours)
3.11
50
9h
EDM-DDPM++ 22
4.94
50
18h
AFHQv2-Dog
Patch Diffusion (ours)
4.80
EDM-DDPM++
2.59
18h
AFHQv2-Wild
Patch Diffusion (ours)
1.93
50
9h
image samples generated by Patch Diffusion in Figure 6
This experiment demonstrates that patch diffusion training could help improve the data efficiency of diffusion models.
4.5 Experiments on Image Extrapolation
In this section, we evaluate the out out-painting performance of Patch Diffusion on the -uN-Bedroom dataset. We initially enlarge the coordinate system higher resolution, maintaining the range [-1, 1]. The reference image is positioned at the center of the expanded coordinate system and remains static throughout the Patch Diffusion reversal process. In Figure 7 extend the pixel manifold to dimensions of 384 x 384, even though our Patch Diffusion model trained on 256 x 256 image samples. The model effectively generates new content beyond the original boundary. Additionally, we present the extrapolation to 512 x 512 resolution in Appendix D
Figure 7: Extrapolation Results. Patch Diffusion could generate beyond the boundary by extrapolat- ing the learned coordinate manifold. For each pair of images, the left panel is the reference image in resolution 256 x 256 and it is fixed in the center during the reverse process of Patch Diffusion, while the right panel shows the generated sample in resolution 384 x 384, where the out-of-boundary region regenerated. Note our model is trained only on 256 x 256 images.
5 Conclusion and Future Work
We present Patch Diffusson, a novel patch-level training framework that trains diffusion models via coordinate conditioned score matching. We also propose to diversify the patch sizes for score matching in progressive stochastic schedule during training, multiple scales. Sampling with our method is easy as in the original diffusion model. Patch Diffusion could significantly reduce the training time costs, e.g., 2x faster training, while improving the data efficiency of diffusion models, e.g., improving the performance of diffusion models trained on relatively small datasets. Going forward, the current coordinate system could further improved by advanced positional embeddings, such as periodic one (38] to better incorporate the position information. We also leave the theoretical proof of the convergence of patch- wise score matching in general cases our future work.
capture the cross-region dependency at
10

---

Acknowledgments
Wang, H. Zheng, and M. Zhou acknowledge the support of NSF-IIS 2212418, NIH-R37 CA271186. and the NSF Al Institute for Foundations of Machine Learning (IFML).
References
[1] Bao, F., Li, C., Sun, J., Zhu, J., and Zhang, B. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint 2xXiv:2206.07309 2022.
[2] Bao, F., Li, C., Zhu, J., and Zhang, B. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint vrXiv:2201.06500 2022.
[3] Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions, arXiv preprint 12Xiv:2211.09800 2022.
[4] Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: 09791, 2019.
[5] Chai, L., Gharbi, M., Shechtman, E., Isola, P., and Zhang, R. Any-resolution training for high-resolution image synthesis. arXiv preprint arXiv:2204.07156. 2022.
[6] Chen, T., Cheng, Y., Gan, Z., Liu, J., and Wang, Z. Data-efficient gan training beyond (just) augmentations: lottery ticket perspective. Advances in Neural Information Processing Systems, 34:20941-20955. 2021.
[7] Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. Stargan v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[8] Daras, G., Delbracio, M., Talebi, H., Dimakis, A. G., and Milanfar, P. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.0525 2022.
[9] Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hicrar- chical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
[10] Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
(11) Ding, Z., Zhang, M., Wu, J., and Tu, Z. Patched demoising diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2308.013166 2023.
[12] Han, X., Zheng, H., and Zhou, M. CARD: Classification and regression diffusion models.
Advances in Neural Information Processing Systems, 38:18 1011810001811... 2022.
[13] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1255 125988 2022.
[15] Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[16] Ho, J., Saharia, C., Chan, W., Fleet, D. J.. Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022.
[17] Hyvärinen, A. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709, 2005. URI.tttp: //jmlr. lalemmlrrrrrg/// www.eoorgggppppeeeeeeeeee http//appes/v.... hyvarinen05a. html

---

[18]
Hyvärinen, A. and Dayan, P. Estimation of non non-normalizzd statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[19] Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. arXiv preprint AXXiv:2205009991 2022.
[20] Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adver- sarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4411 2019.
[21] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-1211 121044 2020.
[22] Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. arXiv preprint (xXiv:22066003666 2022.
[23] Kim, D., Shin, S., Song, K., Kang, W., and Moon, I.-C. Soft truncation: universal training technique of score- score-based based diffusion model for high precision score estimation. arXiv e-prints, pp. arXiv-2106 2021.
[24] Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro. B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976) 09761, 2020.
[25] Kulikov, V., Yadin, S., Kleiner, M., and Michaeli, Sinddm: A single image denoising diffusion model. arXiv preprint arXiv:2211.16588 2022.
[26] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., and Ferrari, V. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[27] Lee, N., Kim, H., Lee, G., Yoo, S., and Choi, Y. Memory efficient patch-based based training for inr-based gans. arXiv preprint arXiv:2207.01300 2022.
[28] Li, S. Z. Markov random field models in computer vision. In European conference on computer vision, pp. 361-370. Springer, 1994.
[29] Li, S. Z. Markov random field modeling in image analysis. Springer Science & Business Media, 2009.
[30] Li, Y., Zhang, R., Lu, J., and Shechtman, E. Few-shot image generation with clastic weight consolidation. arXiv preprint arXhv:2012.02788. 2020.
[31] Lin, H., Chang, C.-C., Chen, Y.-S., Juan, D.-C., Wei, W., and Chen, H.TT. Coco-gan: Generation by parts via conditional coordinating. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4512-4521, 2019.
[32] Lin, J., Zhang, R., Ganz, F., Han, S., and Zhu, Anycost gans for interactive image synthesis and editing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11986-14998 2021.
[33] Liu. L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09775. 2022.
[34] Liu, 7., P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[35] Lu, C., Zhou, Y., Bao. F., Chen, J.. Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint 22VV:2200600922,, 2022.
[36] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, J. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint
2022.
12

---

[37] Lu, H., Tunanyan, H., Wang, K. Navasardyan, S., Wang, Z., and Shi, H. Specialist diffusion: Plug-and-plas sample-efficient fine-tuning of text-to-image diffusion models learn any unseen style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14267-14276, 2023.
[38] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T.,, Ramamoorthi, R., and Ng. R. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[39] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text text-guided diffusion models, arXiv preprint arXiv:21 2021.
[40] Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
[41] Nikankin, Y., Haim, N., and Irani, M. Sinfusion: Training diffusion models on single image or video. arXiv preprint arXiv:2211.11744 2022.
[42] Ojha, U., Li, Y., Lu, J., Efros, A. A., Lee, J., Shechtman, E., and Zhang, R. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10743-10752, 2021.
[43] Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. Grad-tts: diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599 8608. PMLR, 2021.
[44] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[45] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[46] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[47] Saharia, C., Chan, W., Saxena, S., Li, L.., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S., Lopes, R. G., et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv: /487, 2022.
[48] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T. Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210 08402,, 2022.
[49] Skorokhodov, I., Ignatyev, S.,, and Elhoseiny, M. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10753-10764, 2021.
[50] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynanics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
[51] Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02555 02502, 2020.
[52] Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 1 19118-11 8-18-111 1930, 2019.
[53] Song, Y. and Ermon, S. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438- 12448, 2020.
13

---

[54] Song, Y.. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Emmon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. In International Con- ference Learning Representations, 2021. URI. httpp:://ppereviiiee www. met/forum?id- PXTIG12RRHS
[55] Tseng, H.-Y., Jiang, L., Liu, C., Yang, M.-H., and Yang. W. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7921-7931, 2021.
[56] Vahdat, A., Kreis, K., and Kautz, J. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1287-11302, 2021.
[57] Vincent, P. connection between score matching and denoising autoencoders. Neural compu- tation, (3(7))1661-167)) 2011.
(58) Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, I.., and Li, H. Sindiffusion: Learning diffusion model from single natural image. arXiv preprint arXiv:2211.12445, 2022.
[59] Wang, Z., Hunt, J.J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.0611 2022.
[60] Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-GAN: Training GANs with diffusion. arXiv preprini arXiv:2206.022: 02262,, 2022.
[61] Wang. Z., Jiang, Y., Lu, Y., Shen, Y., He, P., Chen, W., Wang, Z., and Zhou, M. In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01115. 2023.
[62] Xie, S. and Tu, Hetsst edge detection. Proceedings of the IEEE international conference on computer vision, pp. 1995-1403, 2015.
[63] Yu, F., Seff, A., Zhang, Y., Song. S.. Funkhouser, T., and Xiao, J. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03366
2015.
[64] Zhang, L. and Agrawala, M. Adding conditional control to text-to-image diffusion models. arXiv preprini urXiv:2302.05544 2023.
[65] Zhang, Z., Han, L., Ghosh, A., Metaxas, D., and Ren, J. Sine: Single image editing with to--gegg diffusion models. arXiv preprint arXiv:2212.04489, 2022.
[66] Zhao, S., Liu, Z.,, Lin, J., Zhu, J.-Y., and Han, S. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020.
[67] Zheng, H., Hc, P., Chen, W. and Zhou, M. Truncated diffusion probabilistic models and diffusion-basce adversarial auto-cncoders. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https: pecttss:/ooge
14

---

Appendix
A Theoretical Interpretations
In this section, we provide mathematical intuitions for our patch diffusion from two perspectives.
Markov Random Field. Markov Random Field (MRF) has been widely used to represent image distributions 28 29 due its compactness and expressiveness modeling dependence. Usually, images are modeled as an undirected regular graph (pixel grids) g = (V, E) in MRF, where each pixel is considered as a graph vertex, and each vertex will connect to its neighboring pixels on the image. Therefore, we can adopt the clique factorization form to represent the PDF defined over images:
p(x))
= Z
1
II (a.,) VEV
II de( 3EE
where o and de are node and edge potential functions, Z is normalization term, I, and De index the random variables via the subscripts pairwise function.
One see @, is a univariate function and Φe is
Next, we point out that the score function of the MRF parameterization can be even neat:
V Vlog((>)
Σ NEV
Vlogo.
Σ 3EE
V log (xxe)
where Z is eliminated it is irrelevant æc As suggested by Equation 7 the score function can be eventually decomposed into independent pieces. That being said, we can separately learn each score function V log & and V log de first, and then average them up to approximate the entire score function during the inference. Our training procedure can be viewed as: each time we sample patches which contains subsets of V and and we conduct score matching on the corresponding log & and V log Φe. Note that V log Φ. and V log &e is not necessarily ppatial-invarian We condition the network on the coordinates model the location dependence for V log é. and V log de.
Linear Regression. We also provide an alternative interpretation for patch-wist score matching through the lens of least square. We consider a multivariate Gaussian distribution as the demonstration example. Suppose we parameterize our target distribution as Py (x) = N Σ) where is the optimizee. Then its score function written as: log P,, score matching with full-size image is equivalent to the following least square problem:
{( μ)) [8] The original
Ex (( Σp (Ex - - e
On the other hand, the patch distribution can be considered as the marginal distribution on sub- set of randam variables. The transformation between patch distribution and full-size image dis- tribution is a marginalization integral, which as simple as linear operator (more precisely an orthogonal projection). In our Gaussian example, the patch distribution has a closed form: Pp (xs) = (x Pss, PsEP)). where S indices set of pixels within an image patch identified by the location and size, and Ps is a selection matrix associated with S. Then patch-wise score matching can be written as:
Es EsEa-p((r) Be-A((,,; PsΣμ (PsEr - e/o
From the linear regression perspective, the only difference between patch- and full-size score matching is the measurement matrix. first glance, patch diffusion trades the computation cost off the well-posed However, we argue that due to well- knlln redundancy and symmetry in image distributions, recovering the whole image distribution under limited observation can factually feasible. This guarantees our patch diffusion can also converge to the true distribution. We consider general argument beyond Gaussian examples as promising future exploration.
B
More Generated Images.
More generated images from Patch Diffusion are listed below.
15

---

Figure 8: Randomly generated images from Patch Diffusion (EDM-DDPM++ backbone) trained on SLUUN-BBdooom//hhuch-26666---
Figure 9: More generated images from Latent Patch Diffusion (EDM-ADM backbone) trained on ImageNet-256xx556.
14P PTN (owe)
Cn Methinge Wote hoartte..
Figure 10: More Finetuning Results of ControlNet with patch diffusion training.
C Potential Social Implications
Our work might lead common negative social impacts of generation models for computer vision. One concern is the proliferation of fake manipulated images, leading to crisis of trust and
16

---

credibility. As these models become more sophisticated, it turns increasingly difficult to discern between real and generated images. undermining the integrity of visual evidence. This can have significant implications in journalism, forensics, and other fields that heavily rely on accurate visual representation. Image generation models can also misused for harmful purposes such as creating realistic but false identities, deepfake pornography, or even propaganda and disinformation campaigns. These applications can lead to privacy violations, cyberbullying, defamation, and manipulation of public opinion. Therefore, it is crucial to address the potential negative social impacts of image generation models and implement ethical guidelines and safeguards to mitigate harmful effects.
D
More Extrapolation Results
Figure 11: Extrapolation Results. Patch Diffusion could generate beyond the boundary by extrapo- lating the learned coordinate manifold. For each pair of images, the left panel is the reference image in resolution 256 x 256 and it is fixed in the center during the reverse process of Patch Diffusion. while the right panel shows the generated sample in resolution 512 512, where the out-of-boundary region is regenerated. Note our model is trained only on 256 x 256 images.
17

---

Figure 12: Randomly generated images from Latent Patch Diffusion (EDM-ADM backbone) trained
on 1mageNet-256x256 Left images are generated without CFG, while the right images are generated with CFG.
18


Reviews:
[{"review_id": "f2SdhTN5OW", "reviewer": "Reviewer_nnk4", "summary": "The paper proposes a novel training framework for diffusion models, that significantly reduces the training time, while improving data efficiency. For the first time, the proposed method suggests patch-wise diffusion training, which can be deployed to any UNet-based diffusion models. Experimental results show that the patch-wise diffusion training can halve the training time while maintaining comparable or better image quality than the baseline models. On small scale datasets, the proposed method outperformed other baselines, validating the data efficiency of path-wise training. ", "questions": "- While training, when entering the denoiser (UNet), has the small patches resized into the original image size? If not, will there be a difference between using resizing and not?\n", "limitations": "Limitations are addressed by the authors. ", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well organized and easy to follow. The motivation of the paper is very clear, which is to shorten the training time of diffusion models while maintaining image generation quality. \n\n- Proper ablation studies are delivered to fully validate roles of different components of the model, and affect of different parameters.\n\n- 2x speedup of training time is non-negligible, considering long training time of conventional diffusion models.\n\n- The data efficiency followed by patch-wise training framework is considered a significant discovery.\n\n- The proposed method can be applied to any UNet-based diffusion models in a plug and play method.", "weaknesses": "- Albeit the empirical evidences, the theoretical proof of convergence of patch-wise score matching is missing. \n\n- Experiments on high resolution image synthesis, beyond 256x256 resolution is missing. According to Table1 and 2, compared to the baseline method, the proposed patch diffusion showed better performance boost on the larger scale datasets (LSUN-Bedroom&Church), than the smaller scale datasets (CelebaA and FFHQ). Thus, it might imply that the patch diffusion can be more beneficial in high resolution image synthesis scenarios. Therefore, validation on high resolution image datasets, beyond 256x256 resolution could be interesting. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer nnk4 for your positive and valuable feedback. We appreciate the time and effort you've taken to review our work. We have carefully considered your comments and suggestions, and we would like to respond to each of them.\n\n> Albeit the empirical evidence, the theoretical proof of convergence of patch-wise score matching is missing.\n\nThanks for pointing this out. We mentioned in Section 5 that providing theoretical proof of convergence for patch-wise score matching is a potential future work. Currently, we provide a theoretical interpretation for Patch Diffusion in Appendix A. \n\n> Experiments on high resolution image synthesis, beyond 256x256 resolution is missing.\n\nThanks for the suggestion! Training on high-resolution images is still computationally expensive for us due to our limited computational resources. We will consider training on 512x512 images in the future. We also provide image extrapolation results in the reponse to all, which shows that our model could generate larger size images, such as 384x384 and 512x512 images, while only trained on 256x256 images. \n\n> While training, when entering the denoiser (UNet), has the small patches resized into the original image size? If not, will there be a difference between using resizing and not? \n\nNo, the small patches are not resized into the original image sizes when entering the UNet. Using small patches for training could significantly improve the training efficiency. \n"}, {"review_id": "U8EKUBmPt8", "reviewer": "Reviewer_qnyP", "summary": "The authors propose a new formulation of training diffusion models by sampling different-sized patches from the training data. The models trained with this formulation have comparable FID scores to models trained on full images on many datasets.", "questions": "Do you have any results on models trained on LAION or MS COCO?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors proposed a way to train faster diffusion models, which can cut down training time in half with almost similar performance.\n\nWell-written and to the point paper. \n", "weaknesses": "Training with patches of image with same guidance might not work when the training data is more heterogeneous like LAION. In datasets where the subject might not be centered in image, or occupy small portion of the image or existence of multiple object in the scene, I wonder if model can still perform well. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer qnyP for providing positive feedback and valuable suggestions. Currently, the pretraining scope of our experiment is limited to class conditional experiments due to the limit of computational resources. Thanks for pointing out interesting and promising potentials. We will consider training on text-guided image generation datasets such as LAION or MS-COCO in the future. "}, {"review_id": "asK8X2UzOQ", "reviewer": "Reviewer_ExTh", "summary": "The paper introduces a path-wise diffusion algorithm for faster training. The authors propose patch coordinate conditioned diffusion models and present a patch-size conditioning scheduling technique for efficient training. The method has similar motivation with patch based GAN such as COCOGAN, but it is applied to diffusion model.", "questions": "1. Is it possible to extrapolate an images as shown in COCOGAN?\n2. What is FID scores of baseline model with same train-time without patch training? (e.g. 24h FIDs of EMM-DDPM++ for CelebA)\n", "limitations": "The author provides some limitations on conclusion section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The method presented in the paper is simple yet effective, successfully reducing the training time by half. The motivation behind the approach aligns with that of COCOGAN, and it can be seen as a reasonable extension for the diffusion model to reduce computational requirements.", "weaknesses": "To learn global structure, the algorithm still need a portion of full-resolution diffusion which introduces a bottleneck in terms of time and memory costs. Additionally, the patch-size scheduling approach appears to be manually designed, potentially limiting its adaptability and automation in optimizing the training process. More controlled experiments on various positional encoding or different hyperparameters would provide better intuition about the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ExTh for providing positive feedback. We address your questions and provide more clarifications below. \n\n> Weakness\n\nThe patch-size scheduling could be flexibly set as other values. Different scheduling will induce different levels of gain in training efficiency. The setting shown in the paper is not cherry-picked and is just what we used. We agree incorporating more advanced positional embedding is worth further investigation. We have tried Fourier Positional Encoding (used in Neural Radiance Fields [1]) as a simplest way to replace the raw coordinates inputs, but we did not see obvious improvement in the preliminary experiment. We will try more in the future (as we mentioned in the future work).\n\n[1] Mildenhall, Ben, et al. \"Nerf: Representing scenes as neural radiance fields for view synthesis.\" Communications of the ACM 65.1 (2021): 99-106\n\n> Is it possible to extrapolate an images as shown in COCOGAN?\n\nThanks for pointing this out. We have provided the extrapolation results in the pdf of **Response to All**.\n\n> What is FID scores of baseline model with same train-time without patch training? (e.g. 24h FIDs of EMM-DDPM++ for CelebA)\n\nOur training logs show that when the EDM baseline method is trained with ~24h, it produces 1.85 FID score on CelebA and 3.36 FID score on FFHQ, respectively. \n"}, {"review_id": "VdvSD2kFSI", "reviewer": "Reviewer_Vqdw", "summary": "This paper presents a new training technique that improves the training speed of diffusion models. Instead of training the diffusion model on the entire image, the authors propose training on sampled patches of the image. This approach maintains the theoretical foundation of the diffusion model by keeping the training objective function mostly unchanged. By combining this approach with a fully-convolutional U-Net architecture, the computational complexity is reduced, resulting in faster training. To minimize the quality difference between the partial image approach and the conventional whole image approach, a stochastic/progressive patch size scheduling was proposed, and the corresponding ablation study was conducted. This study investigates the optimal probability of using the whole image, considering the trade-off between training time and generation quality. Summarizing, the proposed method enables faster learning while preserving the theoretical foundation and generation quality of traditional diffusion models.\n\n\n\n\n", "questions": "1. Patch-wise generation:\nIs it feasible to generate the entire image by combining generated parts instead of generating it all at once?\n\n2. Patch diffusion for image extrapolation:\nWhat would happen if channels for (i, j) were provided with a range of (-1.2, 1.2)? Can patch diffusion extrapolate the image using this approach?\n\n3. Figure quality:\nTo enhance the readability and visual appeal of Figure 1, I recommend refining its quality. Currently, the overall figure appears hastily created, resembling a PowerPoint slide. I kindly request aligning each object for the camera-ready submission to improve its structure. The objects currently occupy space without significant value. Also, the range of i, j is -1~1, but the crop has a different scale, such as 16x16. Please provide an example of the actual value when a 16x16 crop is performed. Furthermore, the font size in the figure is very small, making it difficult to read. Considering the available white space, increasing the font size throughout would greatly improve legibility.\n\n4. Typo:\nLine 86: \"e.g.,.\" should be corrected to \"e.g.\"", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents two significant advantages of training diffusion models using partial images instead of the entire images. Firstly, it reduces model complexity, leading to faster training, which is especially beneficial for state-of-the-art baseline diffusion models that require extensive GPU hours for training. This approach offers the potential for energy-efficient training by effectively reducing the overall training time.\n\nSecondly, training with partial images proves effective in scenarios with limited datasets, outperforming traditional methods. In cases where the dataset is insufficient, diffusion models struggle to accurately predict the true data distribution due to overfitting (limited data is essentially a sparse sampling of true data distribution). By partitioning the image into patch units, the training process simulates a larger dataset, providing the model with more training samples. This enables the diffusion model to estimate a more accurate data distribution, resulting in improved generation quality.\n\nThe paper conducted a series of experiments encompassing large-scale datasets, limited-size datasets, and fine-tuning scenarios, to demonstrate the effectiveness of training images in patch units. This approach maintains performance while significantly improving training efficiency. ", "weaknesses": "The key idea of this paper is to modify the input data format while maintaining the training process of the diffusion model. This approach aligns with similar strategies proposed in previous works like COCO-GAN. Considering the large overlap in ideas, the authors should present various case studies, such as showcasing the application of this technique to diffusion models, in order to compensate for the limited novelty of the paper.\n\nFirstly, there is a lack of analysis concerning spatial conditions. The authors convert the traditional three-channel format (R, G, B) to a five-channel format (R, G, B, i, j) incorporating location information. Meanwhile, other methods such as positional encoding in modern Transformer structures or Fourier feature methods in Alias-Free GAN have been proposed for spatial conditioning. Multiple experiments in various literature have demonstrated that positional encoding is more effective than raw methods like (i, j). Conducting an ablation study on different spatial conditioning methods and providing an analysis of the most suitable conditioning approach in a diffusion setting would strengthen the paper's credibility.\n\nSecondly, there is insufficient analysis regarding the ability of patch diffusion to generate structural diversity. The experimental results in Table 1 and Table 2 show a notable improvement in the quality of patch diffusion for datasets with weak common structures (e.g., Bedroom, Church) compared to those with strong common structures (e.g., FFHQ, CelebA). Including an analysis of the factors contributing to this quality improvement, such as visualizing attention layers for patch images, would help readers' understanding of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Vqdw for providing the positive feedback and constructive suggestions. We address your questions and provide more clarifications below. \n\n> Firstly, there is a lack of analysis concerning spatial conditions. \n\nThanks for pointing this out. We agree proper positional embedding could potentially improve the current Patch Diffusion Training. We have tried Fourier Positional Encoding (used in Neural Radiance Fields [1]) as an alternative way to replace the raw coordinates inputs, but we did not see obvious improvement in the preliminary experiment. We will try more in the future (as we mentioned in the future work).\n\n[1] Mildenhall, Ben, et al. \"Nerf: Representing scenes as neural radiance fields for view synthesis.\" Communications of the ACM 65.1 (2021): 99-106\n\n> Secondly, there is insufficient analysis regarding the ability of patch diffusion to generate structural diversity. \n\nWe agree with the observation that Patch Diffusion training helps more for datasets with weak common structures, such as Bedroom, Church, and ImageNet (with classifier free guidance, now we reach FID **2.74**). We hypothesize that training on local patches could help to capture more local details in the image and then improve the general generation quality. \n\n> Question 1\n\nIt is feasible to do generation by parts. The results will depend on the generation scheme. For example, for the CelebA dataset, if you generate the image with four non-overlapping parts (left-top, right-top, left-bottom, right-bottom), then the model will generate a face with each part independently generated. If the entire coordinate manifold is the input, then the entire image is generated and is consistent. \n\n> Question 2\n\nThanks for pointing this out. We have provided the extrapolation results in the pdf of **Response to All**.\n\n> Question 3\n\nWe thank you for pointing out suggestions in refining Figure 1. We will follow your suggestions and modify it in our next revision. \n"}, {"review_id": "pok83iE1cc", "reviewer": "Reviewer_NLEo", "summary": "This paper presents PatchDiffusion, a novel framework designed to address the scalability challenges faced by most diffusion models in terms of training and sampling. The proposed framework adopts a patch-wise training approach, where a denoising network is trained on image patches rather than the entire high-resolution images. To generate patches at the target resolution, PatchDiffusion leverages progressive or stochastic scheduling techniques that utilize different patch sizes throughout the training process. Experimental results on a small-scale dataset demonstrate that PatchDiffusion not only enhances the quality of generated samples but also reduces the overall training time. Furthermore, from the experiment results for medium-scale datasets, PatchDiffusion could be a resource-efficient solution for diffusion training methods.", "questions": "#1. The primary objective of this work is to minimize the computational cost of training/inference through patch-wise training. However, it could be worthwhile to consider an alternative solution by employing a more efficient backbone inherited from ViT, which incorporates a patch-fying (i.e., tokenizing) module. Including a comparison of this work with DiT or U-ViT in the paper would further highlight the benefits of this approach. Moreover, presenting training compute vs. FID plots for the comparison would greatly assist in determining the most effective approach.\n\n#2. In my understanding, the true advantages of this work are likely to be demonstrated through experiments on fine-tuning. This approach has the potential to reduce the fine-tuning cost for any pre-trained diffusion backbone. In this context, incorporating LoRA-type methods could complement this approach. Including empirical analysis of this nature in the paper would enhance the understanding of the benefits of this work.\n\n#3. The GAN community has explored patch-wise training in various ways. Recently, Any-resolution GAN (ECCV’22) has been introduced as a promising solution for training generative models with variable-size images. Although this paper primarily aims to reduce training costs rather than utilizing multiple size images in the training procedure, it would be interesting to explore whether the proposed framework can be extended to generate variable-size images. Doing so would further highlight the benefits of this framework.\n\n#4. A minor comment regarding the inclusion of \"Appendix A.\" To provide a concise overview of the theoretical interpretation of the patch-wise training scheme, it would be beneficial to incorporate a brief summary of these observations within the main body of the paper.", "limitations": "The limitations of this work were not explicitly outlined, and I didn’t observe any discussion regarding potential negative societal impacts. However, there is no clear negative societal impact, since all experiments are conducted in controlled benchmark datasets.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper tackles a significant challenge in training diffusion models, which often involve substantial computational costs. To address this issue, the authors propose PatchDiffusion as an optional solution that can be seamlessly integrated into any diffusion model pipeline, regardless of the chosen backbone, sampler, or other modules within the pipeline.\n\nI think that PatchDiffusion operates as a form of data augmentation, thereby enhancing the generation quality of the model. This approach provides additional benefits beyond reduced computational costs.", "weaknesses": "[Limited experiments - important baseline missing and low performance] \n\nThe primary motivation behind patch-wise training is to reduce computation costs during training. Given this motivation, it might be worth considering a more efficient backbone instead of U-Net. Recently, successful approaches have replaced U-Net with ViT, as demonstrated in the papers \"Scalable Diffusion Models with Transformers\" (arXiv'22) and \"All are Worth Words: A ViT Backbone for Diffusion Models\" (CVPR'23).\n\nIn the case of DiT, reducing the latent resolution by patchifying with a 2x2 patch size has shown improvements in training scalability and performance. It is worth exploring whether applying patch-wise training to DiT and U-ViT backbones could yield better results. However, the potential gains from such an approach are uncertain.\n\nAdditionally, it is important to note that in class-conditional image generation on ImageNet-1K, the FID score of PatchDiffusion appears to be significantly worse than that of DiT and other related works. While the current state-of-the-art methods achieve an FID score of less than 4, this work reports a score of around 7.65.\n\n[Writing needs to be improved. ] \n\nThe writing quality of the paper could be improved further, especially in terms of highlighting the comparison between this work and the baselines. It would be beneficial to include a figure illustrating the trade-off between FID (or any other measure of generation quality) and FLOPs. This would help in understanding how PatchDiffusion and other efficient diffusion backbones compare to each other. The authors can refer to similar trade-off figures presented in the DiT and U-ViT papers.\n\nEvaluating the quality of writing can be subjective, and I am open to hearing other reviewers' comments on this matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer NLEo for providing constructive suggestions. Below, we address each concern raised in your comment point by point. Please let us know if you have any further questions or whether this adequately addresses all these issues.\n\n> Q1+Weakness 1: it could be worthwhile to consider an alternative solution by employing a more efficient backbone, such as DiT. Also, plotting training compute vs. FID for the comparison would greatly assist in determining the most effective approach. ...\n\nWe acknowledge that both U-Net based and transformer-based diffusion models have attracted significant attention and exhibited strong performance. As of now, there is no conclusive evidence favoring one over the other. Notably, U-Net based diffusion models continue to dominate text-to-image generation, as evidenced by their usage in Stable Diffusion, Imagen, and DALL-E 2.\n\nIn reference to your statement, \"Given this motivation, it might be worth considering a more efficient backbone inherited from ViT,\" we respectfully present a differing perspective. Quoting the DiT paper, \"Figure 2 (right) demonstrates the computational efficiency of DiT-XL/2 (**118.6 Gflops**) relative to latent space U-Net models like LDM-4 (**103.6 Gflops**) and notably more efficient than pixel space U-Net models like ADM (1120 Gflops) or ADM-U (742 Gflops).\" In the context of being applied to latent space, U-Net proves to be more efficient than Transformer architecture. In our ImageNet-1K experiment, Patch Diffusion employs ADM-U on latent space (1x4x32x32) with a considerably lower computational cost of around **27 Gflops**, demonstrating its efficiency. It is pertinent to note that our model possesses approximately 290M parameters, whereas DiT-XL/2 boasts 675M parameters. As elucidated in Table 4 of the DiT paper, DiT model exhibits notably inferior performance on ImageNet-1K if it takes less than 300M parameters.\n\nMoreover, transformer-based diffusion models typically necessitate a substantially higher number of training steps to converge. Table 4 of the DiT paper exemplifies this, as DiT-XL/2 requires 7M steps to achieve convergence, while the U-Net model utilized in our work reaches convergence within 0.61M steps.\n\n> Weakness 2: the current state-of-the-art methods (DiT) achieve an FID score of less than 4, this work reports a score of around 7.65.\n\nThe performance reported in Table 2 of our manuscript is under the setting of **NOT** adopting classifier-free guidance. Under the same setting, DiT only achieves 9.62 FID score while Patch Diffusion achieves 7.65 FID score, demonstrating the effectiveness of our proposed method. If applying classifier-free guidance to Patch Diffusion, our model could reach 2.74 FID score on ImageNet-1K, which matches the state-of-the-art performance but costs a significantly lower computational cost (27 Gflops v.s. 118.6 Gflops).\n\n> Weakness 3: It is worth exploring whether applying patch-wise training to DiT and U-ViT backbones could yield better results.\n\nOur current Patch Diffusion is built and designed for U-Net based diffusion models. We agree with Reviewer NLEo that applying patch diffusion training onto transformer based diffusion models is also worth further investigation, but that currently is out of the scope of this paper, and we left that for future study. \n\n> Q2: incorporating LoRA for fine-tuning\n\nWe appreciate Reviewer NLEo for recognizing the potential applicability of our method in finetuning scenarios. We do agree that our patch diffusion training approach harmonizes effectively with LoRA-type methods, thereby potentially enhancing finetuning efficiency. While LoRA is not the primary focus of this paper, we are open to the prospect of integrating LoRA in future investigations.\n\n> Q3: variable-size images? \n\nAny-resolution GAN [5] is primarily tailored for mixed-resolution datasets from images in the wild, whereas our work focuses on improving the training efficiency of diffusion models. In terms of generating variable-size images, we provided the extrapolation results of Patch Diffusion in the pdf of Response to All.\n\n> Q4: Moving Appendix to the main manuscript.\n\nWe would love to include theoretical interpretation in our main paper. We will consider adding a summary of the theoretical interpretation based on the page limit. \n\n> Limitations discussion is missing.\n\nWe discussed limitations and potential future work in Section 5, and potential negative social impacts in Appendix B.1. \n"}]

Author rebuttal:
# Response to All

We'd like to thank all five reviewers for their insightful comments and suggestions. We hereby provide the image extrapolation results that Reviewer ExTh, Vqdw and NELo are interested in, and the state-of-the-art ImageNet-1K(256x256) FID **2.74** for Patch Latent Diffusion with the use of classifier-free guidance. We will incorporate these new results and improvements into the camera ready version of the paper. 

Meta-review: This paper proposes a new method for training diffusion models on patches, thereby speeding up the training process as compared to when training on only full images. Experiments show that the proposed method is promising. The reviewers assess this work as an substantial contribution to an important problem. There was substantial discussion between the authors and reviewers, resulting in a unanimous vote to accept from the reviewers. As such I'm happy to recommend accepting this work.
Decision: Accept (poster)