SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arpit Bansal $^{1}$  Eitan Borgnia $^{*1}$  Hong-Min Chu $^{*1}$  Jie S. Li $^{1}$  Hamid Kazemi $^{1}$  Furong Huang $^{1}$  Micah Goldblum $^{2}$  Jonas Geiping $^{1}$  Tom Goldstein $^{1}$ $^{1}$ University of Maryland  $^{2}$ New York University
Standard diffusion models involve an image transform - adding Gaussian noise - and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes.
Figure 1: Demonstration of the forward and backward processes for both hot and cold diffusions. While standard diffusions are built on Gaussian noise (top row), we show that generative models can be built on arbitrary and even noiseless/cold image transforms, including the ImageNet-C snowification operator, and an animorphosis operator that adds a random animal image from AFHQ.
Diffusion models have recently emerged as powerful tools for generative modeling [Ramesh et al., 2022]. Diffusion models come in many flavors, but all are built around the concept of random noise removal; one trains an image restoration/denoising network that accepts an image contaminated with Gaussian noise, and outputs a denoised image. At test time, the denoising network is used to convert pure Gaussian noise into a photo-realistic image using an update rule that alternates between applying the denoiser and adding Gaussian noise. When the right sequence of updates is applied, complex generative behavior is observed.
The origins of diffusion models, and also our theoretical understanding of these models, are strongly based on the role played by Gaussian noise during training and generation. Diffusion has been understood as a random walk around the image density function using Langevin dynamics [Sohl-Dickstein et al., 2015, Song and Ermon, 2019], which requires Gaussian noise in each step. The walk begins in a high temperature (heavy noise) state, and slowly anneals into a "cold" state with little if any noise. Another line of work derives the loss for the denoising network using variational inference with a Gaussian prior [Ho et al., 2020, Song et al., 2021a, Nichol and Dhariwal, 2021].
In this work, we examine the need for Gaussian noise, or any randomness at all, for diffusion models to work in practice. We consider generalized diffusion models that live outside the confines of the theoretical frameworks from which diffusion models arose. Rather than limit ourselves to models built around Gaussian noise, we consider models built around arbitrary image transformations like blurring, downsampling, etc. We train a restoration network to invert these deformations using a simple  $\ell_p$  loss. When we apply a sequence of updates at test time that alternate between the image restoration model and the image degradation operation, generative behavior emerges, and we obtain photo-realistic images.
The existence of cold diffusions that require no Gaussian noise (or any randomness) during training or testing raises questions about the limits of our theoretical understanding of diffusion models. It also unlocks the door for potentially new types of generative models with very different properties than conventional diffusion seen so far.
Both the Langevin dynamics and variational inference interpretations of diffusion models rely on properties of the Gaussian noise used in the training and sampling pipelines. From the score-matching generative networks perspective [Song and Ermon, 2019, Song et al., 2021b], noise in the training process is critically thought to expand the support of the low-dimensional training distribution to a set of full measure in ambient space. The noise is also thought to act as data augmentation to improve score predictions in low density regions, allowing for mode mixing in the stochastic gradient Langevin dynamics (SGLD) sampling. The gradient signal in low-density regions can be further improved during sampling by injecting large magnitudes of noise in the early steps of SGLD and gradually reducing this noise in later stages.
Kingma et al. [2021] propose a method to learn a noise schedule that leads to faster optimization. Using a classic statistical result, Kadkhodaie and Simoncelli [2021] show the connection between removing additive Gaussian noise and the gradient of the log of the noisy signal density in deterministic linear inverse problems. Here, we shed light on the role of noise in diffusion models through theoretical and empirical results in applications to inverse problems and image generation.
Iterative neural models have been used for various inverse problems [Romano et al., 2016, Metzler et al., 2017]. Recently, diffusion models have been applied to them [Song et al., 2021b] for the problems of deblurring, denoising, super-resolution, and compressive sensing [Whang et al., 2021, Kawar et al., 2021, Sahara et al., 2021, Kadkhodaie and Simoncelli, 2021].
Although not their focus, previous works on diffusion models have included experiments with deterministic image generation [Song et al., 2021a, Dhariwal and Nichol, 2021, Karras et al., 2022] and in selected inverse problems [Kawar et al., 2022]. Recently, Rissanen et al. [2022] use a combination of Gaussian noise and blurring as a forward process for diffusion. Though they show the feasibility of a different degradation, here we show definitively that noise is not a necessity in diffusion models, and we observe the effects of removing noise for a number of inverse problems.
Despite prolific work on generative models in recent years, methods to probe the properties of learned distributions and measure how closely they approximate the real training data are by no means closed fields of investigation.
Indirect feature space similarity metrics such as Inception Score [Salimans et al., 2016], Mode Score [Che et al., 2016], Frechet inception distance (FID) [Heusel et al., 2017], and Kernel inception distance (KID) [Binkowski et al., 2018] have been proposed and adopted to some extent, but they have notable limitations [Barratt and Sharma, 2018]. To adopt a popular frame of reference, we will use FID as the feature similarity metric for our experiments.
3 Generalized Diffusion
Standard diffusion models are built around two components. First, there is an image degradation operator that contaminates images with Gaussian noise. Second, a trained restoration operator is created to perform denoising. The image generation process alternates between the application of these two operators. In this work, we consider the construction of generalized diffusions built around arbitrary degradation operations. These degradations can be randomized (as in the case of standard diffusion) or deterministic.
3.1 Model components and training
Given an image  $x_0 \in \mathbb{R}^N$ , consider the degradation of  $x_0$  by operator  $D$  with severity  $t$ , denoted  $x_t = D(x_0, t)$ . The output distribution  $D(x_0, t)$  of the degradation should vary continuously in  $t$ , and the operator should satisfy  $D(x_0, 0) = x_0$ .
In the standard diffusion framework,  $D$  adds Gaussian noise with variance proportional to  $t$ . In our generalized formulation, we choose  $D$  to perform various other transformations such as blurring, masking out pixels, downsampling, and more, with severity that depends on  $t$ . We explore a range of choices for  $D$  in Section 4.
We also require a restoration operator  $R$  that (approximately) inverts  $D$ . This operator has the property that  $R(x_{t}, t) \approx x_{0}$ . In practice, this operator is implemented via a neural network parameterized by  $\theta$ . The restoration network is trained via the minimization problem
$$
\min _ {\theta} \mathbb {E} _ {x \sim \mathcal {X}} \| R _ {\theta} (D (x, t), t) - x \|,
$$
where  $x$  denotes a random image sampled from distribution  $\mathcal{X}$  and  $\| \cdot \|$  denotes a norm, which we take to be  $\ell_1$  in our experiments. We have so far used the subscript  $R_{\theta}$  to emphasize the dependence of  $R$  on  $\theta$  during training, but we will omit this symbol for simplicity in the discussion below.
3.2 Sampling from the model
After choosing a degradation  $D$  and training a model  $R$  to perform the restoration, these operators can be used in tandem to invert severe degradations by using standard methods borrowed from the diffusion literature. For small degradations  $(t\approx 0)$ , a single application of  $R$  can be used to obtain a restored image in one shot. However, because  $R$  is typically trained using a simple convex loss, it yields blurry results when used with large  $t$ . Rather, diffusion models [Song et al., 2021a, Ho et al., 2020] perform generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm 1.
Algorithm 1 Naive Sampling (Eg. DDIM)
Input: A degraded sample  $x_{t}$
for  $s = t, t - 1, \ldots, 1$  do
$$
\hat {x} _ {0} \leftarrow R (x _ {s}, s)
$$
$$
x _ {s - 1} = D (\hat {x} _ {0}, s - 1)
$$
Return:  $x_0$
Algorithm 2 Transformation Agnostic Cold Sampling (TACoS)
$$
x _ {s - 1} = x _ {s} - D (\hat {x} _ {0}, s) + D (\hat {x} _ {0}, s - 1)
$$
When the restoration operator is perfect, i.e. when  $R(D(x_0,t),t) = x_0$  for all  $t$ , one can easily see that Algorithm 1 produces exact iterates of the form  $x_{s} = D(x_{0},s)$ . But what happens for imperfect
restoration operators? In this case, errors can cause the iterates  $x_{s}$  to wander away from  $D(x_0,s)$ , and inaccurate reconstruction may occur.
We find that the standard sampling approach in Algorithm 1 (explained further in A.8) works well for noise-based diffusion, possibly because the restoration operator  $R$  has been trained to correct (random Gaussian) errors in its inputs. However, we find that it yields poor results in the case of cold diffusions with smooth/differentiable degradations as demonstrated for a deblurring model in Figure 2. We propose Transformation Agnostic Cold Sampling (TACoS) in Algorithm 2, which we find to be superior for inverting smooth, cold degradations.
This sampler has important mathematical properties that enable it to recover high quality results. Specifically, for a class of linear degradation operations, it can be shown to produce exact reconstruction (i.e.  $x_{s} = D(x_{0}, s)$ ) even when the restoration operator  $R$  fails to perfectly invert  $D$ . We discuss this in the following section.
3.3 Properties of TACoS
It is clear from inspection that both Algorithms 1 and 2 perfectly reconstruct the iterate  $x_{s} = D(x_{0}, s)$  for all  $s < t$  if the restoration operator is a perfect inverse for the degradation operator. Hence in this section, we will discuss the reconstruction operator that fails to reconstruct the image perfectly i.e. incurs error. We first analyze the stability of these algorithms to errors in the restoration operator and then theoretically show that for a simple blur degradation, the error incurred using algorithm 1 is always greater than algorithm 2.
Figure 2: Comparison of sampling methods for unconditional generation using cold diffusion on the CelebA dataset. Iterations 2, 4, 8, 16, 32, 64, 128, 192, and 256 are presented. Top: Algorithm 1 produces compounding artifacts and fails to generate a new image. Bottom: TACoS succeeds in sampling a high quality image without noise.
For small values of  $x$  and  $s$ , TACoS as described in 2 is tolerant of error in the restoration operator  $R$ . To see why, consider a problem with linear degradation function of the form  $D(x,s)\approx x + s\cdot e$  for a constant vector  $e$ . We choose this ansatz because the Taylor expansion of any smooth degradation  $D(x,s)$  around  $x = x_0$ ,  $s = 0$  has the form  $D(x,s)\approx x + s\cdot e(x) + \mathrm{HOT}$  where HOT denotes higher order terms. Note, however, the analysis below requires  $e$  to be a constant that does not depend on  $x$ . The constant/zeroth-order term in this Taylor expansion is zero because we assumed above that the degradation operator satisfies  $D(x,0) = x$ .
For a degradation  $D(x,s)$  and any restoration operator  $R$ , the term  $x_{s-1}$  in TACoS becomes
$$
\begin{array}{l} x _ {s} - D (R \left(x _ {s}, s\right), s) + D (R \left(x _ {s}, s\right), s - 1) = D \left(x _ {0}, s\right) - D \left(R \left(x _ {s}, s\right), s\right) + D (R \left(x _ {s}, s\right), s - 1) \\ = x _ {0} + s \cdot e - R \left(x _ {s}, s\right) - s \cdot e + R \left(x _ {s}, s\right) + (s - 1) \cdot e = x _ {0} + (s - 1) \cdot e = D \left(x _ {0}, s - 1\right) \\ \end{array}
$$
By induction, we see that the algorithm outputs the value  $x_{s} = D(x_{0}, s)$  for all  $s < t$ , regardless of the choice of  $R$ . In other words, for any choice of  $R$ , the iteration behaves the same as it would when  $R$  is a perfect inverse for the degradation  $D$ .
By contrast, Algorithm 1 does not enjoy this behavior even for small values of  $s$ . In fact, when  $R$  is not a perfect inverse for  $D$ ,  $x_0$  is not a fixed point of the update rule in Algorithm 1 because  $x_0 \neq D(R(x,0),0) = R(x,0)$  and hence errors compound. If  $R$  does not perfectly invert  $D$ , we should expect Algorithm 1 to incur errors, even for small values of  $s$ . Meanwhile, for small values of  $s$ , the behavior of  $D$  approaches its first-order Taylor expansion, and Algorithm 2 becomes immune to errors in  $R$ . Figure 2 demonstrates the stability of TACoS described in Algorithm 2 vs. Algorithm 1 for a deblurring model.
The above analysis is not a complete convergence theory but rather highlights a desirable theoretical property of our method that a naive sampler lacks. However, we can prove that for a toy problem in which the blur operator removes one frequency at a time, the error incurred by sampling using Algorithm 1 is greater than the error incurred from using Algorithm 2. We present the proof of this claim in A.9.
4 Generalized Diffusions with Various Transformations
In this section, we take the first step towards cold diffusion by reversing different degradations and hence performing conditional generation. We will extend our methods to perform unconditional (i.e. from scratch) generation in Section 5. We empirically evaluate generalized diffusion models trained on different degradations with TACoS proposed in Algorithm 2. We perform experiments on the vision tasks of deblurring, inpainting, and super-resolution. We perform our experiments on MNIST [LeCun et al., 1998], CIFAR-10 [Krizhevsky, 2009], and CelebA [Liu et al., 2015]. In each of these tasks, we gradually remove the information from the clean image, creating a sequence of images such that  $D(x_0,t)$  retains less information than  $D(x_0,t - 1)$ . For these different tasks, we present both qualitative and quantitative results on a held-out testing dataset and demonstrate the importance of the sampling technique described in Algorithm 2. For all quantitative results in this section, the Frechet inception distance (FID) scores [Heusel et al., 2017] for degraded and reconstructed images are measured with respect to the testing data. Additional information about the quantitative results, convergence criteria, hyperparameters, and architecture of the models presented below can be found in the appendix.
4.1 Deblurring
We consider a generalized diffusion based on a Gaussian blur operation (as opposed to Gaussian noise) in which an image at step  $t$  has more blur than at  $t - 1$ . The forward process given the Gaussian kernels  $\{G_s\}$  and the image  $x_{t-1}$  at step  $t - 1$  can thus be written as
$$
x _ {t} = M _ {t} \circ x _ {t - 1} = M _ {t} \circ \ldots \circ M _ {1} \circ x _ {0} = \bar {M} _ {t} \circ x _ {0} = D (x _ {0}, t)
$$
where  $*$  denotes the convolution operator, which blurs an image using a kernel.
We train a deblurring model by minimizing the loss (1), and then use TACoS to invert this blurred diffusion process for which we trained a DNN to predict the clean image  $\hat{x}_0$ . Qualitative results are shown in Figure 3 and quantitative results in Table 1. Qualitatively, we can see that images created using the sampling process are sharper and in some cases completely different as compared to the direct reconstruction of the clean image. Quantitatively we can see that the reconstruction metrics such as RMSE and PSNR get worse when we use the sampling process, but on the other hand FID with respect to held-out test data improves. The qualitative improvements and decrease in FID show the benefits of the generalized sampling routine, which brings the learned distribution closer to the true data manifold. Note: we compare the images reconstructed via Algorithm 2, with direct generation as compared to Algorithm 1. This is because the image reconstruction via Algorithm 1 is much worse than both direct generation and Algorithm 2. Nevertheless, to back our claim, we present their results in A.10.
In the case of blur operator, the sampling routine can be thought of adding frequencies at each step. This is because the sampling routine involves the term  $D(\hat{x}_0,t) - D(\hat{x}_0,t - 1)$  which in the case of blur becomes  $\bar{G}_t * x_0 - \bar{G}_{t-1} * x_0$ . This results in a difference of Gaussians, which is a band pass filter and contains frequencies that were removed at step  $t$ . Thus, in the sampling process, we sequentially add the frequencies that were removed during the degradation process.
Figure 3: Deblurring models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
Table 1: Quantitative metrics for quality of image reconstruction using deblurring models.
Dataset	Degraded	Sampled	Direct
FID	SSIM	RMSE	FID	SSIM	RMSE	FID	SSIM	RMSE
MNIST	438.59	0.287	0.287	4.69	0.718	0.154	5.10	0.757	0.142
CIFAR-10	298.60	0.315	0.136	80.08	0.773	0.075	83.69	0.775	0.071
CelebA	382.81	0.254	0.193	26.14	0.568	0.093	36.37	0.607	0.083
4.2 Inpainting
We define a schedule of transforms that progressively grays-out pixels from the input image. We remove pixels using a Gaussian mask as follows: For input images of size  $n \times n$  we start with a 2D Gaussian curve of variance  $\beta$ , discretized into an  $n \times n$  array. We normalize so the peak of the curve has value 1, and subtract the result from 1 so the center of the mask as value 0. We randomize the location of the Gaussian mask for MNIST and CIFAR-10, but keep it centered for CelebA. We denote the final mask by  $z_{\beta}$ .
Input images  $x_0$  are iteratively masked for  $T$  steps via multiplication with a sequence of masks  $\{z_{\beta_i}\}$  with increasing  $\beta_i$ . We can control the amount of information removed at each step by tuning the  $\beta_i$  parameter. In the language of Section 3,  $D(x_0,t) = x_0 \cdot \prod_{i=1}^{t} z_{\beta_i}$ , where the operator  $\cdot$  denotes entry-wise multiplication.
Figure 4 presents results on test images and compares the output of the inpainting model to the original image. The reconstructed images display reconstructed features qualitatively consistent with the context provided by the unperturbed regions of the image. We quantitatively assess the effectiveness of the inpainting models on each of the datasets by comparing distributional similarity metrics before and after the reconstruction. Our results are summarized in Table 2. Note, the FID scores here are computed with respect to the held-out validation set.
Figure 4: Inpainting models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: Degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
Table 2: Quantitative metrics for quality of image reconstruction using inpainting models.
Dataset	Degraded	Sampled	Direct
FID	SSIM	RMSE	FID	SSIM	RMSE	FID	SSIM	RMSE
MNIST	108.48	0.490	0.262	1.61	0.941	0.068	2.24	0.948	0.060
CIFAR-10	40.83	0.615	0.143	8.92	0.859	0.068	9.97	0.869	0.063
CelebA	127.85	0.663	0.155	5.73	0.917	0.043	7.74	0.922	0.039
4.3 Super-Resolution
For this task, the degradation operator downsamples the image by a factor of two in each direction. The final resolution of  $x_{T}$  is  $4 \times 4$  for MNIST and CIFAR-10 and  $2 \times 2$  in the case of Celeb-A. After each down-sampling, the lower-resolution image is resized to the original image size, using nearest-neighbor interpolation. More details are available in Appendix A.3
Figure 5 presents example testing data inputs for all datasets and compares the output of the superresolution model to the original image. Though the reconstructed images are not perfect for the
more challenging datasets, the reconstructed features are qualitatively consistent with the context provided by the low resolution image. Table 3 compares the distributional similarity metrics between degraded/reconstructed images and test samples.
Figure 5: Superresolution models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
Table 3: Quantitative metrics for quality of image reconstruction using super-resolution models.
Dataset	Degraded	Sampled	Direct
FID	SSIM	RMSE	FID	SSIM	RMSE	FID	SSIM	RMSE
MNIST	368.56	0.178	0.231	4.33	0.820	0.115	4.05	0.823	0.114
CIFAR-10	358.99	0.279	0.146	152.76	0.411	0.155	169.94	0.420	0.152
CelebA	349.85	0.335	0.225	96.92	0.381	0.201	112.84	0.400	0.196
5 Cold Generation
Diffusion models can successfully learn the underlying distribution of training data, and thus generate diverse, high quality images [Song et al., 2021a, Dhariwal and Nichol, 2021, Jolicoeur-Martineau et al., 2021, Ho et al., 2022]. We will first discuss deterministic generation using Gaussian noise and then discuss in detail unconditional generation using deblurring. Finally, we provide a proof of concept that the TACoS described in Algorithm 2 can be extended to other degradations.
5.1 Generation using deterministic noise degradation
Here we discuss image generation using a noise-based degradation presented in our notation from Section 3, which we will later prove is equivalent to DDIM [Song et al., 2021a]. We use the following degradation operator:  $D(x,t) = \sqrt{\alpha_t} x + \sqrt{1 - \alpha_t} z$ .
$D$  is an interpolation between the data point  $x$  and a sampled noise pattern  $z \in \mathcal{N}(0,1)$ . During training,  $D$  is applied once and thus  $z$  is sampled once for every image in every batch. However, sampling involves iterative applications of the degradation operator  $D$ , which poses the question of how to pick  $z$  for the sequence of degradations  $D$  applied in a single image generation.
There are three possible choices for  $z$ . The first would be to resample  $z$  for each application of  $D$ , but this would make the sampling process nondeterministic for a fixed starting point. Another option is to sample a noise pattern  $z$  once for each separate image generation and reuse it in each application of  $D$ . In Table 4 we refer to this approach as Fixed Noise. Finally, one can calculate the noise vector  $z$  to be used in step  $t$  of reconstruction by using the formula
$$
\hat {z} (x _ {t}, t) = \frac {x _ {t} - \sqrt {\alpha_ {t}} R (x _ {t} , t)}{\sqrt {1 - \alpha_ {t}}}.
$$
This method denoted Estimated Noise in Table 4 turns out to be equivalent to the deterministic sampling proposed in Song et al. [2021a]. We discuss this equivalence in detail in Appendix A.6.