SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5.2 Image generation using blur
The forward diffusion process in noise-based diffusion models has the advantage that the degraded image distribution at the final step  $T$  is simply an isotropic Gaussian. One can therefore perform
(unconditional) generation by first drawing a sample from the isotropic Gaussian, and sequentially denoising it with backward diffusion.
When using blur as a degradation, the fully degraded images do not form a nice closed-form distribution that we can sample from. They do, however, form a simple enough distribution that can be modeled with simple methods. Note that every image  $x_0$  degenerates to an  $x_T$  that is constant (i.e., every pixel is the same color) for large  $T$ . Furthermore, the constant value is exactly the channel-wise mean of the RGB image  $x_0$ , and can be represented with a 3-vector. This 3-dimensional distribution is easily represented using a Gaussian mixture model (GMM). This GMM can be sampled to produce the random pixel values of a severely blurred image, which can be deblurred using cold diffusion to create a new image.
Our generative model uses a blurring schedule where we progressively blur each image with a Gaussian kernel of size  $27 \times 27$  over 300 steps. The standard deviation of the kernel starts at 1 and increases exponentially at the rate of 0.01. We then fit a simple GMM with one component to the distribution of channel-wise means. To generate an image from scratch, we sample the channel-wise mean from the GMM, expand the 3D vector into a  $128 \times 128$  image with three channels, and then apply TACoS.
Empirically, the presented pipeline generates images with high fidelity but low diversity, as reflected quantitatively by comparing the perfect symmetry column with results from hot diffusion in Table 4. We attribute this to the perfect correlation between pixels of  $x_{T}$  sampled from the channel-wise mean Gaussian mixture model. To break the symmetry between pixels, we add a small amount of Gaussian noise (of standard deviation 0.002) to each sampled  $x_{T}$ . As shown in Table 4, the simple trick drastically improves the quality of generated images. We also present the qualitative results for cold diffusion using blur transformation in Figure 6, and further discuss the necessity of TACoS proposed in Algorithm 2 for generation in Appendix A.7.
Table 4: FID scores for CelebA and AFHQ datasets using hot (noise) and cold diffusion (blur transformation). Breaking the symmetry within pixels of the same channel further improves FID.
Dataset	Hot Diffusion	Cold Diffusion
Fixed Noise	Estimated Noise	Perfect symmetry	Broken symmetry
CelebA	59.91	23.11	97.00	49.45
AFHQ	25.62	20.59	93.05	54.68
Figure 6: Examples of generated samples from  $128 \times 128$  CelebA and AFHQ datasets using cold diffusion with blur transformation
5.3 Generation using other transformations
In this section, we provide a proof of concept that generation can be extended to other transformations. Specifically, we show preliminary results on inpainting, super-resolution, and animorphosis. Inspired by the simplicity of the degraded image distribution for the blurring routine presented in the previous section, we use degradation routines with predictable final distributions here as well.
To use the Gaussian mask transformation for generation, we modify the masking routine so the final degraded image is completely devoid of information. One might think a natural option is to send all of the images to a completely black image  $x_{T}$ , but this would not allow for any diversity
in generation. To get around this maximally non-injective property, we instead make the mask turn all pixels to a random, solid color. This still removes all of the information from the image, but it allows us to recover different samples from the learned distribution via Algorithm 2 by starting off with different color images. More formally, a Gaussian mask  $G_{t} = \prod_{i = 1}^{t}z_{\beta_{i}}$  is created in a similar way as discussed in the Section 4.2, but instead of multiplying it directly to the image  $x_0$ , we create  $x_{t}$  as  $G_{t}\cdot x_{0} + (1 - G_{t})\cdot c$ , where  $c$  is an image of a randomly sampled color.
For super-resolution, the routine down-samples to a resolution of  $2 \times 2$ , or 4 values in each channel. These degraded images can be represented as one-dimensional vectors, and their distribution is modeled using one Gaussian distribution. Using the same methods described for generation using blurring described above, we sample from this Gaussian-fitted distribution of the lower-dimensional degraded image space and pass this sampled point through the generation process trained on super-resolution data to create one output.
Additionally to show one can invert nearly any transformation, we include a new transformation deemed animorphosis, where we iteratively transform a human face from CelebA to an animal face from AFHQ. Though we chose CelebA and AFHQ for our experimentation, in principle such interpolation can be done for any two initial data distributions.
More formally, given an image  $x$  and a random image  $z$  sampled from the AFHQ manifold,  $x_{t}$  can be written as  $x_{t} = \sqrt{\alpha_{t}x} +\sqrt{1 - \alpha_{t}z}$ . Note this is essentially the same as the noising procedure, but instead of adding noise we are adding a progressively higher weighted AFHQ image. In order to sample from the learned distribution, we sample a random image of an animal and use TACoS.
We present results for the CelebA dataset, and hence the quantitative results in terms of FID scores for inpainting, super-resolution and animorphosis are 90.14, 92.91 and 48.51 respectively. We further show some qualitative samples in Figure 7, and in Figure 1.
Figure 7: Preliminary demonstration of the generative abilities of other cold diffusins on the  $128 \times 128$  CelebA dataset. The top row is with animorphosis models, the middle row is with inpainting models, and the bottom row exhibits super-resolution models.
Existing diffusion models rely on Gaussian noise for both forward and reverse processes. In this work, we find that the random noise can be removed entirely from the diffusion model framework, and replaced with arbitrary transforms. In doing so, our generalization of diffusion models and their sampling procedures allows us to restore images afflicted by deterministic degradations such as blur, inpainting and downsampling. This framework paves the way for a more diverse landscape of diffusion models beyond the Gaussian noise paradigm. The different properties of these diffusions may prove useful for a range of applications, including image generation and beyond.
This work was made possible by the ONR MURI program, the Office of Naval Research (N000142112557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2212182), and by the NSF TRAILS Institute (2229885).
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. arXiv preprint arXiv:2206.13397, 2022.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). arXiv preprint arXiv:1611.02862, 2016.
Chitwan Sahara, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, volume 37 of JMLR Workshop and Conference Proceedings, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Jay Whang, Maurizio Delbracio, Hossein Talebi, Chitwan Sahara, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint arXiv:2112.02475, 2021.
A.1 Deblurring
For the deblurring experiments, we train the models on different datasets for 700,000 gradient steps. We use the Adam Kingma and Ba [2014] optimizer with learning rate  $2 \times 10^{-5}$ . The training was done on the batch size of 32, and we accumulate the gradients every 2 steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995 which is updated after every 10 gradient steps.
For the MNIST dataset, we blur recursively 40 times, with a discrete Gaussian kernel of size  $11 \times 11$  and a standard deviation 7. In the case of CIFAR-10, we recursively blur with a Gaussian kernel of fixed size  $11 \times 11$ , but at each step  $t$ , the standard deviation of the Gaussian kernel is given by  $0.01 * t + 0.35$ . The blur routine for CelebA dataset involves blurring images with a Gaussian kernel of  $15 \times 15$  and the standard deviation of the Gaussian kernel grows exponentially with time  $t$  at the rate of 0.01.
Figure 8 shows an additional nine images for each of MNIST, CIFAR-10 and CelebA. Figures 19 and 20 show the iterative sampling process using a deblurring model for ten example images from each dataset. We further show 400 random images to demonstrate the qualitative results in the Figure 21.
Figure 8: Additional examples from deblurring models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
A.2 Inpainting
For the inpainting transformation, models were trained on different datasets with 60,000 gradient steps. The models were trained using Adam Kingma and Ba [2014] optimizer with learning rate  $2 \times 10^{-5}$ . We use batch size 64, and the gradients are accumulated after every 2 steps. The final model is an Exponential Moving Average of the trained model with decay rate 0.995. This EMA model is updated after every 10 gradient steps. For all our inpainting experiments we use a randomized Gaussian mask and  $T = 50$  with  $\beta_{1} = 1$  and  $\beta_{i+1} = \beta_{i} + 0.1$ .
To avoid potential leakage of information due to floating point computation of the Gaussian mask, we discretize the masked image before passing it through the inpainting model. This was done by rounding all pixel values to the eight most significant digits.
Figure 10 shows nine additional inpainting examples on each of the MNIST, CIFAR-10, and CelebA datasets. Figure 9 demonstrates an example of the iterative sampling process of an inpainting model for one image in each dataset.
A.3 Super-Resolution
We train the super-resolution model per Section 3.1 for 700,000 iterations. We use the Adam Kingma and Ba [2014] optimizer with learning rate  $2 \times 10^{-5}$ . The batch size is 32, and we accumulate the gradients every 2 steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps.
The number of time-steps depends on the size of the input image and the final image. For MNIST and for CIFAR10, the number of time steps is 3, as it takes three steps of halving the resolution to reduce the initial image down to  $4 \times 4$ . For CelebA, the number of time steps is 6 to reduce the initial image down to  $2 \times 2$ . For CIFAR10, we apply random crop and random horizontal flip for regularization.
Figure 12 shows an additional nine super-resolution examples on each of the MNIST, CIFAR-10, and CelebA datasets. Figure 11 shows one example of the progressive increase in resolution achieved with the sampling process using a super-resolution model for each dataset.
A.4 Colorization
Here we provide results for the additional task of colorization. Starting with the original RGB-image  $x_0$ , we realize colorization by iteratively desaturing for  $T$  steps until the final image  $x_T$  is a fully gray-scale image. We use a series of three-channel  $1 \times 1$  convolution filters  $\mathbf{z}(\alpha) = \{z^1(\alpha), z^2(\alpha), z^3(\alpha)\}$  with the form
$$
z ^ {1} (\alpha) = \alpha \left(\frac {1}{3} \frac {1}{3} \frac {1}{3}\right) + (1 - \alpha) (1 0 0)
$$
$$
z ^ {2} (\alpha) = \alpha \left(\frac {1}{3} \frac {1}{3} \frac {1}{3}\right) + (1 - \alpha) (0 1 0)
$$
$$
z ^ {3} (\alpha) = \alpha \left(\frac {1}{3} \frac {1}{3} \frac {1}{3}\right) + (1 - \alpha) (0 0 1)
$$
and obtain  $D(x,t) = \mathbf{z}(\alpha_t)*x$  via a schedule defined as  $\alpha_{1},\ldots ,\alpha_{t}$  for each respective step. Notice that a gray image is obtained when  $x_{T} = \mathbf{z}(1)*x_{0}$ .
We can tune the ratio  $\alpha_{t}$  to control the amount of information removed in each step. For our experiment, we schedule the ratio such that for every  $t$  we have
$$
x _ {t} = \mathbf {z} (\alpha_ {t}) * \dots * \mathbf {z} (\alpha_ {1}) * x _ {0} = \mathbf {z} (\frac {t}{T}) * x _ {0}.
$$
This schedule ensures that color information lost between steps is smaller in earlier stage of the diffusion and becomes larger as  $t$  increases.
We train the models on different datasets for 700,000 gradient steps. We use Adam Kingma and Ba [2014] optimizer with learning rate  $2 \times 10^{-5}$ . We use batch size 32, and we accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 we use  $T = 50$  and for CelebA we use  $T = 20$ .
Figure 9: Progressive inpainting of selected masked MNIST, CIFAR-10, and CelebA images.
Figure 10: Additional examples from inpainting models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
Figure 11: Progressive upsampling of selected downsampled MNIST, CIFAR-10, and CelebA images. The original image is at the left for each of these progressive upsamplings.
We illustrate our recolorization results in Figure 13. We present testing examples, as well as their grey scale images, from all the datasets, and compare the recolorization results with the original images. The recolored images feature correct color separation between different regions, and feature various and yet semantically correct colorization of objects. Our sampling technique still yields minor differences in comparison to the direct reconstruction, although the change is not visually apparent. We attribute this to the shape restriction of colorization task, as human perception is rather insensitive to minor color change. We also provide quantitative measurement for the effectiveness of our recolorization results in terms of different similarity metrics, and summarize the results in Table 5.
Table 5: Quantitative metrics for quality of image reconstruction using recolorization models for all three channel datasets.
Dataset	Degraded Image	Reconstruction
FID	SSIM	RMSE	FID	SSIM	RMSE
CIFAR-10	97.39	0.937	0.078	45.74	0.942	0.069
CelebA	41.20	0.942	0.089	17.50	0.973	0.042
Figure 12: Additional examples from super-resolution models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
Figure 13: Recolorization models trained on the CIFAR-10 and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
A.5 Image Snow
Here we provide results for the additional task of snowification, which is a direct adaptation of the official implementation of ImageNet-C snowification process Hendrycks and Dietterich [2019]. To determine the snow pattern of a given image  $x_0 \in \mathbb{R}^{C \times H \times W}$ , we first construct a seed matrix  $S_A \in \mathbb{R}^{H \times W}$  where each entry is sampled from a Gaussian distribution  $N(\mu, \sigma)$ . The upper-left corner of  $S_A$  is then zoomed into another matrix  $S_B \in \mathbb{R}^{H \times W}$  with spline interpolation. Next, we create a new matrix  $S_C$  by filtering each value of  $S_B$  with a given threshold  $c_1$  as
$$
S _ {C} [ i ] [ j ] = \left\{ \begin{array}{l l} 0, & S _ {B} [ i ] [ j ] \leq c _ {1} \\ S _ {B} [ i ] [ j ], & S _ {B} [ i ] [ j ] > c _ {1} \end{array} \right.
$$
and clip each entry of  $S_{C}$  into the range [0, 1]. We then convolve  $S_{C}$  using a motion blur kernel with standard deviation  $c_{2}$  to create the snow pattern  $S$  and its up-side-down rotation  $S'$ . The direction of the motional blur kernel is randomly chosen as either vertical or horizontal. The final snow image is created by again clipping each value of  $x_{0} + S + S'$  into the range [0, 1]. For simplicity, we abstract the process as a function  $h(x_{0}, S_{A}, c_{0}, c_{1})$ .
Figure 14: Additional examples from Desnowification models trained on the CIFAR-10 and CelebA datasets. Left to right: degraded inputs  $D(x_0, T)$ , direct reconstruction  $R(D(x_0, T))$ , sampled reconstruction with TACoS described in Algorithm 2, and original image.
To create a series of  $T$  images with increasing snowification, we linearly interpolate  $c_{0}$  and  $c_{1}$  between  $[c_0^{\mathrm{start}}, c_0^{\mathrm{end}}]$  and  $[c_1^{\mathrm{start}}, c_1^{\mathrm{end}}]$  respectively, to create  $c_{0}(t)$  and  $c_{1}(t)$ ,  $t = 1, \ldots, T$ . Then for each  $x_{0}$ , a seed matrix  $S_{x}$  is sampled, the motion blur direction is randomized, and we construct each related  $x_{t}$  by  $x_{t} = h(x_{0}, S_{x}, c_{0}(t), c_{1}(t))$ . Visually,  $c_{0}(t)$  dictates the severity of the snow, while  $c_{1}(t)$  determines how "windy" the snowified image seems.
For both CIFAR-10 and Celeb-A, we use the same Gaussian distribution with parameters  $\mu = 0.55$  and  $\sigma = 0.3$  to generate the seed matrix. For CIFAR-10, we choose  $c_0^{\mathrm{start}} = 1.15$ ,  $c_0^{\mathrm{end}} = 0.7$ ,  $c_1^{\mathrm{start}} = 0.05$  and  $c_1^{\mathrm{end}} = 16$ , which generates a visually lighter snow. For Celeb-A, we choose  $c_0^{\mathrm{start}} = 1.15$ ,  $c_0^{\mathrm{end}} = 0.55$ ,  $c_1^{\mathrm{start}} = 0.05$  and  $c_1^{\mathrm{end}} = 20$ , which generates a visually heavier snow.
We train the models on different datasets for 700,000 gradient steps. We use Adam Kingma and Ba [2014] optimizer with learning rate  $2 \times 10^{-5}$ . We use batch size 32, and we accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 we use  $T = 200$  and for CelebA we use  $T = 200$ . We note that the seed matrix is resampled for each individual training batch, and hence the snow pattern varies across the training stage.
A.6 Generation using noise : Further Details
Here we show the equivalence between the sampling method proposed in Algorithm 2 and the deterministic sampling in DDIM Song et al. [2021a]. Given the image  $x_{t}$  at step  $t$ , we have the restored clean image  $\hat{x_0}$  from the diffusion model. Hence given the estimated  $\hat{x_0}$  and  $x_{t}$ , we can estimate the noise  $z(x_{t},t)$  (or  $\hat{z}$ ) as
$$
z (x _ {t}, t) = \frac {x _ {t} - \sqrt {\alpha_ {t}} \hat {x _ {0}}}{\sqrt {1 - \alpha_ {t}}},
$$
Thus, the  $D(\hat{x_0},t)$  and  $D(\hat{x_0},t - 1)$  can be written as
$$
D (\hat {x} _ {0}, t) = \sqrt {\alpha_ {t}} \hat {x} _ {0} + \sqrt {1 - \alpha_ {t}} \hat {z},
$$
$$
D (\hat {x _ {0}}, t - 1) = \sqrt {\alpha_ {t - 1}} \hat {x _ {0}} + \sqrt {1 - \alpha_ {t - 1}} \hat {z},
$$
using which the sampling process in Algorithm 2 to estimate  $x_{t - 1}$  can be written as,
$$
\begin{array}{l} x _ {t - 1} = x _ {t} - D \left(\hat {x} _ {0}, t\right) + D \left(\hat {x} _ {0}, t - 1\right) \\ = x _ {t} - \left(\sqrt {\alpha_ {t}} \hat {x} _ {0} + \sqrt {1 - \alpha_ {t}} \hat {z}\right) + \left(\sqrt {\alpha_ {t - 1}} \hat {x} _ {0} + \sqrt {1 - \alpha_ {t - 1}} \hat {z}\right) \\ = \sqrt {\alpha_ {t - 1}} \hat {x} _ {0} + \sqrt {1 - \alpha_ {t - 1}} \hat {z} \tag {1} \\ \end{array}
$$
which is same as the sampling method as described in Song et al. [2021a]. The only difference from the original Song et al. [2021a] is the order for estimating  $\hat{x_0}$  and  $\hat{z}$ . The original Song et al. [2021a] paper estimated  $\hat{z}$  first and then used this to predict clean image  $\hat{x_0}$ , while we first predict the clean image  $\hat{x_0}$  and then estimate the noise  $\hat{z}$ .
A.7 Generation using blur transformation: Further Details
Figure 15: Examples of generated samples from  $128 \times 128$  CelebA and AFHQ datasets using Method 2 with perfect symmetry.
Figure 15 shows that generation without breaking any symmetry within each channel is quite promising as well.
Necessity of Algorithm 2: In the case of unconditional generation, we observe a marked superiority in quality of the sampled reconstruction using Algorithm 2 over any other method considered. For example, in the broken symmetry case, the FID of the directly reconstructed images is 257.69 for CelebA and 214.24 for AFHQ, which are far worse than the scores of 49.45 and 54.68 from Table 4. In Figure 17, we also give a qualitative comparison of this difference. We can also clearly see from Figure 18 that Algorithm 1, the method used in Song et al. [2021b] and Ho et al. [2020], completely fails to produce an image close to the target data distribution.
A.8 Algorithm 1 is same as DDIM/DDPM sampling
The sampling method proposed in Song et al. [2021a] in its equation 12 is given as
$$
x _ {t - 1} = \sqrt {\alpha_ {t - 1}} \cdot “ \text {p r e d i c t e d} x _ {0} ” + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \epsilon_ {\theta} (x _ {t}) + \sigma_ {t} \epsilon_ {t}
$$
where  $\epsilon_{\theta}(x_t)$  is the noise predicted by the diffusion model given  $x_{t}$  and  $t$ . The term "predicted  $x_0$  or  $\hat{x}_0$  can be computed directly given  $x_{t}$  and  $\epsilon_{\theta}(x_t)$  as
$$
\hat {x _ {0}} = \frac {x _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} (x _ {t})}{\sqrt {\alpha_ {t}}},
$$
Hence using  $\hat{z}$  instead of  $\epsilon_{\theta}(x_t)$  and  $\hat{x_0}$  to indicate predicted clean image, we have
$$
x _ {t - 1} = \sqrt {\alpha_ {t - 1}} \cdot \hat {x _ {0}} + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \hat {z} + \sigma_ {t} \epsilon_ {t}
$$