SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhendong Wang $^{1,2}$ , Yifan Jiang $^{1}$ , Huangjie Zheng $^{1,2}$ , Peihao Wang $^{1}$ , Pengcheng He $^{2}$ , Zhangyang Wang $^{1}$ , Weizhu Chen $^{2}$ , and Mingyuan Zhou $^{1}$
Diffusion models are powerful, but they require a lot of time and data to train. We propose Patch Diffusion, a generic patch-wise training framework, to significantly reduce the training time costs while improving data efficiency, which thus helps democratize diffusion model training to broader users. At the core of our innovations is a new conditional score function at the patch level, where the patch location in the original image is included as additional coordinate channels, while the patch size is randomized and diversified throughout training to encode the cross-region dependency at multiple scales. Sampling with our method is as easy as in the original diffusion model. Through Patch Diffusion, we could achieve  $\geq 2 \times$  faster training, while maintaining comparable or better generation quality. Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, e.g., as few as 5,000 images to train from scratch. We achieve outstanding FID scores in line with state-of-the-art benchmarks: 1.77 on CelebA-64×64, 1.93 on AFHQv2-Wild-64×64, and 2.72 on ImageNet-256×256. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Patch-Diffusion.
Figure 1: Illustration of Patch Diffusion on training and sampling.
Diffusion models [50, 15] have become the new de facto generative AI (GenAI) models. Song et al. [54] unite the diffusion models into the framework of score-based generative models [17, 57, 52, 53].
Diffusion models have found great success in a wide range of applications, including unconditional image synthesis [15, 51, 40, 22], text-to-image generation [39, 47, 44, 45, 61], audio generation [24, 43], uncertainty quantification [12], and reinforcement learning [19, 59].
Although diffusion models are stable to train and powerful in capturing distributions, they are notoriously slow in generation due to the need to traverse the reverse diffusion chain, which involves going through the same U-Net-based generator network hundreds or even thousands of times [51, 60]. As such, there is great interest in improving the inference speed of diffusion models, leading to many improved samplers such as DDIM [51], TDPM [67], DPM-Solver [35, 36], and EDM-Sampling [22].
However, diffusion models are also notoriously expensive and data-hungry to train. They require large datasets and many iterations to capture the high-dimensional and complex data distributions. For instance, training DDPM [15] on eight V100 GPUs for the LSUN-Bedroom [63] dataset at resolutions of  $64 \times 64$  and  $256 \times 256$  takes approximately four days and over two weeks, respectively. A state-of-the-art diffusion model in Dhariwal & Nichol [10] consumes 150-1000 V100 GPU days to produce high-quality samples. The training costs grow exponentially as the resolution and diversity of the target data increase. Moreover, the best-performing models [47, 45] rely on billion-level image datasets such as OpenImages [26] and LAION [48], which are not easily accessible or scalable.
While some most exciting GenAI results are arguably accomplished by training diffusion models with enormous computational power and data (often owned or led by large corporations), the prohibitive time and data scale required to train competitive diffusion models have presented a critical bottleneck for democratizing this GenAI workhorse technology to the broader research community who generally lack access to such high-end privileged resources. This problem of democratizing diffusion model training is becoming pressing, yet largely overlooked so far.
To democratize diffusion model training, we propose patch-wise diffusion training (Patch Diffusion), a plug-and-play training technique that is agnostic to any choice of UNet [15, 22] architecture, sampler, noise schedule, and so on. Instead of learning the score function on the full-size image for each pixel, we propose to learn a conditional score function on image patches, where both patch location in the original image and patch size are the conditions. Training on patches instead of full images significantly reduces the computational burden per iteration. To incorporate the conditions of patch locations, we construct a pixel-level coordinate system and encode the patch location information as additional coordinate channels, which are concatenated with the original image channels as the input for diffusion models. We further propose diversifying the patch sizes in a progressive or stochastic schedule throughout training, to capture the cross-region dependency at multiple scales.
Patch Diffusion aims to significantly reduce the training time costs while improving the data efficiency of diffusion models. Sampling with our method is as easy as it in the original diffusion model: we compute and parameterize the full coordinates with respect to the original image, concatenate them with the sampled noise, and then reverse the diffusion chain to collect samples, as illustrated in Figure 1. Through Patch Diffusion, we could achieve  $\geq 2 \times$  faster training, while maintaining comparable or better generation quality. We also observe that Patch Diffusion improves the performance of diffusion models trained on relatively small datasets, e.g., clearly superior generation results when training with as few as 5,000 images from scratch. We summarize our main contributions as follows:
- The first patch-level training framework, generally applicable to diffusion models, that targets saving training time and data costs.
- Novel strategies of patch coordinate conditioning and patch size conditioning/scheduling, to balance training efficiency and effective global structure encoding.
Competitive results while generally halving the training time, and notable performance gains at small training data regimes.
Preliminaries for diffusion models. Diffusion probabilistic models [50, 15] first construct a forward process that injects noise into data distribution, then reverse the forward process to reconstruct it. The forward process iteratively applies noise to the given images, whereas the reverse process iteratively denoises a noisy observation. After the origin of the diffusion model, many efforts have been made to apply it to various downstream tasks. Ho et al. [16] and Vahdat et al. [56] propose a hierarchical architecture to stabilize the training process of diffusion models and further mitigate
memory cost issues. Dall-E-2 [44] first introduces the diffusion model to the text-to-image generative task and achieves remarkable success. Later, Sahara et al. [47] found that increasing the parameter of the language model improves both sample fidelity and image-text alignment much more than increasing the size of the image model.
Data-efficient training in generative models. Prior to the study of diffusion models, many explorations have been conducted on limited data or even the few-shot setting of generative model's training schemes, e.g., Generative Adversarial Networks (GANs). To avoid expensive data collection, Zhao et al. [66] propose differentiable augmentation on both the generator and discriminator of GANs and achieve acceptable performance with only  $10\%$  data. Karras et al. [21] apply adaptive augmentation strategy for the setting of limited data to prevent information leakage. Wang et al. [60] propose to train GANs with an adaptive forward diffusion process that injects different levels of noise during training, significantly improving GANs' performance in small datasets. Chen et al. [6] discover the lottery ticket mask of a specific GAN architecture that allows further reducing the training data. Tseng et al. [55] propose to add regularization on top of the existing data augmentation strategy and stabilize the brittle training process of adversarial learning. Others explore the domain adaptation of GAN frameworks when only a few data from target domains are available, e.g., by using elastic weight consolidation [30] or cross-domain correspondence [42].
Meanwhile, recent works also start exploring the few-shot adaptation of diffusion models [37]. DreamBooth [46] uses four frontal images of a specific subject to finetune a pre-trained diffusion model and additionally set a special indicator in the prompt, where the obtained model is able to generate various examples with the same identity. Later, Zhang et al. [65] reduce the number of required images to only a single example when finetuning the diffusion model, but still generate visually impressive photos given various prompt inputs. However, the above works do not fully explore the setting of training a diffusion model from scratch. Although a few single-image diffusion algorithms [58, 41, 25] only need a single training image, they can only produce variations of the same image and hence focus drastically differently compared to our proposed method.
Resource-efficient training in generative models. To address the issue of mode collapse given limited training data, Lin et al. [32] propose Anycost-GAN that is able to generate examples with different levels of cost during inference. However, their proposed techniques do not help save the training cost, and may even take a longer time to optimize the SuperNet [4]. Coco-GAN [31] firstly introduces the patch-wise training scheme on GAN frameworks. However, since their discriminator still needs to combine several generated patches together, its effectiveness on memory saving is not distinguishable. Lee et al. [27] further apply the patch-wise training scheme to the INR-GAN [49] framework. Although their approach is able to save GPU memory cost, it will hurt the quality of generated examples (e.g., increasing FID on the FFHQ dataset from 8.51 to 24.38). In the meantime, since the low-resolution images are generally more accessible compared to high-resolution ones, Chai et al. [5] adopt any-resolution framework which is able to blend data with different resolutions together to feed the training of GANs, and therefore reduce the partition of the high-resolution part. Despite its superiority of data efficiency, it does not particularly discuss the training costs and the effectiveness of its methodology remains unknown and unexplored for diffusion models.
A handful of works try to tackle the issue of huge training and inference costs in diffusion models. Rombach et al. [45] perform diffusion in a latent space instead of the pixel space, which largely reduces the training and inference cost. Other works [35, 36, 51, 2, 1] study the fast sampling strategy at the inference stage, which does not directly help speed up the training process. Our proposed approach is orthogonal to the aforementioned works and can serve as a plug-and-play module.
Simultaneously with our research efforts, Ding et. al. [11] concurrently developed another model of patch-based denoising diffusion. Their model excels in memory-efficient high-resolution image synthesis while avoiding the introduction of boundary artifacts. Central to their methodology is the introduction of a novel feature collage technique, achieved through a systematic window-sliding process, which effectively enforces spatial consistency.
3 Patch Diffusion Training
We first briefly review diffusion models from the perspective of score-based generative modeling and then present our method in details. Suppose we are given a dataset  $\{\pmb{x}_n\}_{n=1}^N$ , where each datapoint is independently drawn from an underlying data distribution  $p(\pmb{x})$ . Following Song et al.
[54], we consider a family of perturbed distributions  $p_{\sigma}(\tilde{x} \mid x) = \mathcal{N}(\tilde{x}; x, \sigma I)$  obtained by adding independent, and identically distributed (i.i.d.) Gaussian noise with standard deviation  $\sigma$  to the data. With a sequence of positive noise scales  $\sigma_{\min} = \sigma_0 < \dots < \sigma_t < \dots < \sigma_T = \sigma_{\max}$ , a forward diffusion process,  $p_{\sigma_t}(x)$  for  $t = 1, \ldots, T$ , is defined given that  $\sigma_{\min}$  is small enough such that  $p_{\sigma_{\min}}(x) \approx p(x)$  and  $\sigma_{\max}$  is large enough such that  $p_{\sigma_{\max}}(x) \approx \mathcal{N}(\mathbf{0}, \sigma_{\max}^2 I)$ . The target of diffusion models is to learn how to reverse the chain and restore the data distribution  $p(x)$ .
Further generalizing to an infinite number of noise scales [54],  $T \to \infty$ , the forward diffusion process could be characterized by a stochastic differential equation (SDE) and further converted to an ordinary differential equation (ODE). The closed form of the reverse SDE is given by
$$
d \boldsymbol {x} = \left[ \boldsymbol {f} (\boldsymbol {x}, t) - g ^ {2} (t) \nabla_ {\boldsymbol {x}} \log p _ {\sigma_ {t}} (\boldsymbol {x}) \right] d t + g (t) d \boldsymbol {w}, \tag {1}
$$
where  $\pmb{f}(\cdot, t): \mathbb{R}^d \to \mathbb{R}^d$  is a vector-valued function called the drift coefficient,  $g(t) \in \mathbb{R}$  is a real-valued function called the diffusion coefficient,  $dt$  represents a negative infinitesimal time step,  $\pmb{w}$  denotes a standard Brownian motion, and  $d\pmb{w}$  can be viewed as infinitesimal white noise. The corresponding ODE of the reverse SDE is named probability flow ODE [54], given by
$$
d \boldsymbol {x} = \left[ \boldsymbol {f} (\boldsymbol {x}, t) - 0. 5 g ^ {2} (t) \nabla_ {\boldsymbol {x}} \log p _ {\sigma_ {t}} (\boldsymbol {x}) \right] d t. \tag {2}
$$
As shown in Equations (1) and (2), solving the reverse SDE or ODE requires us to know both the terminal distribution  $p_{\sigma_T}(\pmb{x})$  and score function  $\nabla_{\pmb{x}} \log p_{\sigma_t}(\pmb{x})$ . By design, the former  $p_{\sigma_T}(\pmb{x})$  is close to a white Gaussian noise distribution. The analytical form of  $\nabla_{\pmb{x}} \log p_{\sigma_t}(\pmb{x})$  is generally intractable, and hence we learn a function  $s_{\theta}(\pmb{x}, \sigma_t)$  parameterized by a neural network to estimate its values. Denoising score matching [57, 51] is currently the most popular way of estimating score functions applied in diffusion models. After learning the estimated score function  $s_{\theta}(\pmb{x}, \sigma_t)$ , we can obtain an estimated reverse SDE or ODE to collect data samples from the estimated data distribution.
Next, we introduce our patch diffusion training in three subsections. First, we propose to conduct conditional score matching on randomly cropped image patches, with the patch location and patch size as conditions, to improve the efficiency of learning scores. Second, we introduce pixel coordinate systems to provide better guidance on patch-level score matching. Then, we show by using our method, for each reverse step, we could do sampling globally as easily as the original diffusion models, without the need to explicitly sample separate local patches and merge them afterwards.
3.1 Patch-wise Score Matching
Following denoising score-matching in Karras et al. [22], we build a denoiser,  $D_{\theta}(\pmb{x}; \sigma_t)$ , that minimizes the expected  $L_2$  denoising error for samples drawn from data distribution  $p(\pmb{x})$  independently for any  $\sigma_t$ :
$$
\mathbb {E} _ {\boldsymbol {x} \sim p (\boldsymbol {x})} \mathbb {E} _ {\epsilon \sim \mathcal {N} (\boldsymbol {0}, \sigma_ {t} ^ {2} \boldsymbol {I})} | | D _ {\boldsymbol {\theta}} (\boldsymbol {x} + \epsilon ; \sigma_ {t}) - \boldsymbol {x} | | _ {2} ^ {2}.
$$
Thus we have the score function as
$$
s _ {\boldsymbol {\theta}} (\boldsymbol {x}, \sigma_ {t}) = \left(D _ {\boldsymbol {\theta}} (\boldsymbol {x}; \sigma_ {t}) - \boldsymbol {x}\right) / \sigma_ {t} ^ {2}. \tag {3}
$$
Instead of conducting the score matching on the full images, we propose to learn the score function on random-size patches. As shown in Figure 1, for any  $\pmb{x} \sim p(\pmb{x})$ , we first randomly crop small patches  $x_{i,j,s}$ , where we use  $(i,j)$ , the left-upper corner pixel coordinates, to locate each image patch, and  $s$  to denote the patch size, e.g.,  $s = 16$ . We conduct denoising score matching on image patches with the corresponding patch locations and sizes as the conditions, expressed as
$$
\mathbb {E} _ {\boldsymbol {x} \sim p (\boldsymbol {x}), \epsilon \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {t} ^ {2} \boldsymbol {I}\right), (i, j, s) \sim \mathcal {U}} | | D _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i, j, s}; \sigma_ {t}, i, j, s\right) - \boldsymbol {x} _ {i, j, s} | | _ {2} ^ {2}, \tag {4}
$$
where  $\tilde{\pmb{x}}_{i,j,s} = \pmb{x}_{i,j,s} + \pmb{\epsilon}$  and  $\mathcal{U}$  denotes the uniform distribution on the corresponding value range, e.g.,  $i\sim [-1,1]$ . Then, our conditional score function derived from Equation (3),  $s_\theta (\pmb {x},\sigma_t,i,j,s)$ , is defined on each local patch. We learn the scores for pixels within each image patch conditioning on its location and patch size.
With Equation (4), the training speed is significantly boosted due to the use of small local patches. However, the challenge now lies in that the score function  $s_{\theta}(\boldsymbol{x}, \sigma_t, i, j, s)$  has only seen local patches and may have not captured the global cross-region dependency between local patches, in other words, the learned scores from nearby patches should form a coherent score map to induce coherent image sampling. To resolve this issue, we propose two strategies: 1) random patch sizes and 2) involving
a small ratio of full-size images. Our training patch size is sampled from a mixture of small and large patch sizes, and then the cropped large patch could be seen as a sequence of small patches. In this way, the score function  $s_{\theta}$  learns how to unite the scores from small patches to form a coherent score map when training on the large patches. To ensure the reverse diffusion converges towards the original data distribution, in some iterations during training, full-size images are required to be seen.
We further provide a theoretical interpretation in Appendix A to help understand our method, and conduct a detailed empirical study in Section 4.1 to show the impact of the ratio of full images.
3.2 Progressive and Stochastic Patch Size Scheduling
To strengthen the awareness of score function in cross-region dependency, we are motivated to propose patch-size scheduling. Denote the ratio of iterations that takes as input the full-size images as  $p$ , and the original image resolution as  $R$ . We propose to use the patch options, as follows
$$
s \sim p _ {s} := \left\{ \begin{array}{l l} p & \text {w h e n} s = R, \\ \frac {3}{5} (1 - p) & \text {w h e n} s = R / / 2, \\ \frac {2}{5} (1 - p) & \text {w h e n} s = R / / 4. \end{array} \right. \tag {5}
$$
Two patch-size schedulings could be considered. 1) Stochastic: During training, we randomly sample  $s \sim p_s$  for each mini-batch with the probability mass function defined in Equation (5). 2) Progressive: We train our conditional score function from small patches to large patches. In the first  $\frac{2}{5} (1 - p)$  training iterations, we fix the patch size as  $s = R / / 4$ , while in the second  $\frac{3}{5} (1 - p)$  training iterations,  $s = R / / 2$  is applied. Finally, we train the models on full-size images for  $p$  ratio of the total iterations.
Empirically, we find that  $p = 0.5$  with stochastic scheduling reaches a sweet point in the trade-off between training efficiency and generation quality, as shown in Figure 3.
One natural question is why the denoiser  $D_{\theta}$  could handle image patches of varying sizes. We note the UNet [15, 22] architecture is fully designed with convolutional layers, and the convolutional filters are capable of handling any resolution images by moving themselves around the inputs. Hence, our patch diffusion training could be regarded as a plug-and-play training technique for any UNet-based diffusion models. The flexibility of UNet on resolutions also makes our sampling easy and fast.
3.3 Conditional Coordinates for Patch Location
Motivated by COCO-GAN [31], to further incorporate and simplify the conditions of patch locations in the score function, we build up a pixel-level coordinate system. We normalize the pixel coordinate values to  $[-1, 1]$  with respect to the original image resolution, by setting the upper left corner of the image as  $(-1, -1)$  while the bottom right corner as  $(1, 1)$ .
As shown in Figure 1, for any image patch  $\pmb{x}_{i,j,s}$ , we extract its  $i$  and  $j$  pixel coordinates as two additional channels. For each training batch, we independently randomly crop each data sample with a sampled patch size for the batch and extract its corresponding coordinate channels. We concatenate the two coordinate channels with the original image channels to form the input of our denoiser  $D_{\theta}$ . When computing the loss defined in Equation (4), we ignore the reconstructed coordinate channels and only minimize the loss on the image channels.
The pixel-level coordinate system together with random patch size could be seen as a kind of data augmentation method. For example, for an image with resolution  $64 \times 64$ , with patch size  $s = 16$ , we could have  $(64 - 16 + 1)^2 = 2401$  possible patches with different locations specified. Hence, we believe training diffusion models on patch-wise would help the data efficiency of diffusion models, in other words, with our method, diffusion models could perform better on small datasets. We validate this hypothesis through experiments in Section 4.4.
By utilizing our coordinate system and the UNet, we are able to easily accomplish the reverse sampling defined in either Equation (1) or Equation (2). As we have shown in Figure 1, we compute and parameterize the coordinates for the full image, and concatenate them together with the image sample from last step as coordinate conditions at each reverse iteration. We abandon the reconstruction output of coordinate channels at each reverse iteration.
We provide a theoretical interpretation of Patch Diffusion Training in Appendix A.
We conduct five sets of experiments to validate our patch diffusion training method. In the first subsection, we conduct an ablation study on what impacts the performance of our method. In the second subsection, we compare our method with its backbone model and other state-of-the-art diffusion model baselines on commonly-used benchmark datasets. Thirdly, we show that our method could also help improve the efficiency of finetuning large-scale pretrained models. Then, we show that patch diffusion models could achieve better generation quality on typical small datasets. Finally, we evaluate the out-painting capability of patch diffusion models.
Datasets. Following previous works [22, 8, 23], we select CelebA ( $\sim$ 200k images) [34], FFHQ (70k images) [20], LSUN ( $\sim$ 200k images) [63], and ImageNet ( $\sim$ 1.2 million images) [9] as our large datasets, and AFHQv2-Cat/Dog/Wild ( $\sim$ 5k images in each of them) [7] as our small datasets.
Evaluation protocol. We measure image generation quality using Fréchet Inception Distance (FID) [13]. Following Karras et al. [20, 22], we measure FID using  $50\mathrm{k}$  generated samples, with the full training set used as reference. We use the number of real images shown to the diffusion models to measure our training duration [22]. Unless specified otherwise, all models are trained with a duration of 200 million images to ensure convergence (these trained with longer or shorter durations are specified in table captions). For the sampling time, we use the number of function evaluations (NFE) as a measure.