Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
Authors: Arpit Bansal
Eitan Borgnia
Hong-Min Chu
Jie S. Li
Hamid Kazemi
Furong Huang
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models
Computer Vision
Diffusion Models

Abstract: Standard diffusion models involve an image transform  -- adding Gaussian noise -- and an image restoration operator that inverts this degradation.  We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact, an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. 
The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference and paves the way for generalized diffusion models that invert arbitrary processes.

Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
Arpit Bansal'
Eitan Borgnia"' 1
Hong-Min Chu
Jie Li' Micah Goldblum²
Hamid Kazemi
Furong Huang'
Jonas Geiping'
Tom Goldstein
University of Maryland
NNe York University
Abstract
Standard diffusion models involve image transform adding Gaussian noise and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent the choice image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models be easily generalized create generative models. The success of these fully deterministic models calls into question the community's s understanding of diffuuson models, which relies noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes.
Original
Forward
Degraded
Reverse
Generated
Snow Pixelate Mask Animorph Blur Noise
Figure I:
Demonstration of the forward and backward processes for both hot and cold diffusions. While standard diffusions are built on Gaussian noise (top row), we show that generative models be built arbitrary and even noiseless/cold image transforms, including the ImageNet-6 C snowification operator, and an animorphosis operator that adds a random animal image from AFHQ.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Diffusion models have recently emerged powerful tools for generative modeling (Ramesh et al. 2022]. Diffusion models come in many flavors, but all are built around the concept of random noise removal; one trains an image restoration/denoisnng network that accepts an image contaminated with Gaussian noise, and outputs a denoised image. At test time, the denoising network is used to convert pure Gaussian noise into a photo-realistic image using an update rule that alternates between applying the denoiser and adding Gaussian noise. When the right sequence of updates is applied, complex generative behavior is observed.
The origins of diffusion models, and also our theoretical understanding of these models, are strongly based on the role played by Gaussian noise during training and generation. Diffusion has been understood as a random walk around the image density function using Langevin dynamics [Sohl- Dickstein et al. 2015 Sonn and Ermon] 2019). which requires Gaussian noise in each step. The walk begins in a high temperature (heavy noise) state, and slowly anneals into a "cold" state with little any noise. Another line of work derives the loss for the denoising network using variational inference with a Gaussian prior [Ho et al., 2020 Song et al. 2021a Nichol and Dhariwal 2021].
In this work, we examine the need for Gaussian noise, or any randomness at all. for diffusion models to work in practice. We consider generalized diffusion models that live outside the confines of the theoretical frameworks from which diffusion models arose. Rather than limit ourselves to models built around Gaussian noise, we consider models built around arbitrary image transformations like blurring, downsampling. etc. We train a restoration network to invert these deformations using simple fp loss. When we apply sequence of updates at test time that alternate between the image restoration model and the image degradation operation, generative behavior emerges, and we obtain photo-realistic images.
The existence of cold diffusiins that require no Gaussian noise (or any randomness) during training or testing raises questions about the limits of our thcoretical understanding of diffusion models. also unlocks the door for potentially new types of generative models with very different properties than conventional diffusion seen so far.
2 Background
Both the Langevin dynamics and variational inference interpretations of diffusion models rely on properties of the Gaussian noise used in the training and sampling pipelines. From the score-matching
generative networks perspective [Song and Ermon| 2019, Song et al. 2021b]. noise in the training process is critically thought to expand the support of the low-dimensional training distribution to a set of full measure in ambient space. The noise is also thought to act as data augmentation to improve score predictions in low density regions, allowing for mode mixing in the stochastic gradient Langevin dynamics (SGLD) sampling. The gradient signal in low-density regions can be further improved during sampling by injecting large magnitudes of noise in the early steps of SGLD and gradually reducing this noise in later stages.
Kingma et al. [2021] propose a method learn noise schedule that leads to faster optimization. Using a classic statistical result, Kadkhodaie and Simoncelli [2021] show the connection between removing additive Gaussian noise and the gradient of the log of the noisy signal density in determin- istic linear inverse problems. Here, we shed light on the role of noise in diffusion models through theoretical and empirical results in applications inverse problems and image generation.
Iterative neural models have been used for various inverse problems [oomano et al., 2016 Metzler ct al. 2017). Recently, diffusion models have been applied to them [Song et al.) 2b2lb for the problems of deblurring, denoising, super-resolution, and compressive sensing [Whang et al. 2021
Kawar et al. | Saharia et al. 202l Kadkhodaie and Simoncelli] 2021].
Although not their focus, previous works on diffusion models have included experiments with deterministic image generation [Song et al. 2021a Dhariwal and Nichol, 2021 Karras et al. 2022) and selected inverse problems [Kawar et al. 2022]. Recently, Rissanen et al. [0222] use a combination of Gaussian noise and blurring as a forward process for diffusion. Though they show the feasibility of a different degradation, here we show definitively that noise is not necessity in diffusion models, and we observe the effects of removing noise for a number of inverse problems.

---

Despite prolific work generative models in recent years, methods to probe the properties of learned distributions and measure how closely they approximate the real training data are by no means closed fields of investigation.
Indirect feature space similarity metrics such Inception Score [Salimans al. 2016], Mode Score [Che et al. 2016]. Frechet inception distance (FID) [Heusel et al., 2017], and Kernel inception distance (KID) [Binkowski et al., 2018] have been proposed and adopted some extent, but they have notable limitations [Barratt and Sharma 2018). To adopt popular frame of reference, we will use FID as the feature similarity metric for our experiments.
3 Generalized Diffusion
Standard diffusion models are built around two components. First, there is an image degradation operator that contaminates images with Gaussian noise. Second, trained restoration operator is created to perform denoising. The image generation process alternates between the application of these two operators. In this work, we consider the construction of generalized diffusions built around arbitrary degradation operations. These degradations can be randomized (as in the case of standard diffusion) or deterministic.
3.1 Model components and training
Given image 20 R.W. consider the degradation of zo by operator D with severity t, denoted T = D(oo, l). The output distribution D( I) of the degradation should vary continuously in 4 and the operator should satisfy D(rn, 0) -i0-
In the standard diffusion framework. D adds Gaussian noise with variance proportional to t. In our generalized formulation, choose D to perform various other transformations such as blurring, masking out pixels, downsampling, and more, with severity that depends on We explore a range of choices for D in Section
We also require a restoration operator that (approximately) inverts This operator has the property that R(x,, t) = Zo. In practice, this operator is implemented via neural network parameterized by The restoration network trained via the minimization problem
min E || Ro (1x, (),4) xl),
where z denotes random image sampled from distribution X' and || denotes norm, which take to be l in our experiments. We have so far used the subscript Rg to emphasize the dependence of R on 0 during training, but we will omit this symbol for simplicity in the discussion below.
3.2 Sampling from the model
After choosing a degradation D and train- ing model R. to perform the restoration, these operators can be used in tandem to in- vert severe degradations by using standard methods borrowed from the diffusion liter- 0),
Algorithm Naive Sampling (Eg, DDIM) Input: A degraded sample I;
for = t, 111-1,... 1,..., do
do + R(x,,5)
F- D(āo, --1
ature. For small degradations single application of f can be used to ob- tain a restored image in one shot. However, because R is typically trained using sim- ple convex loss, it yields blurry results when used with large t. Rather, diffusion models [Song et al. 202 la Ho et al.) 2020] per- form generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the stan- dard update sequence in Algorithm
end for Return: .r0
Algorithm Traansformation Agnostic Cold Sampling (TACoS)
Input: A degraded sample Iz.
for = f, 111-1,... 1,..., do
do +- R(x,,8)
F.-1 = 2s D(Eo, + D(āo, 1)
end for
When the restoration operator is perfect, i.e. when R(D(ro, 1), zo for all 4, one can easily see that Algorithm I produces exact iterates of the form V. D(zo, s). But what happens for imperfect
3

---

restoration operators? In this case, errors can cause the iterates I, to wander away from (( s). and inaccurate reconstruction may occur.
We find that the standard sampling approach in Algorithm [ (explained further in A.8) works well for noise-based diffusion, possibly because the restoration operator R has been trained to correct (random Gaussian) errors in its inputs. However, we find that it yields poor results in the case of cold diffusions with smoooth/Gifferentiaab degradations as demonstrated for deblurring model in Figure We propose Transformation Agnostic Cold Sampling (TACoS) in Algorithm 2 which we find to be superior for inverting smooth, cold degradations.
This sampler has important mathematical properties that enable it to recover high quality results. Specifically, for a class of linear degradation operations, it can be shown to produce exact reconstruc- tion ei.e. Is = ( s)) even when the restoration operator R fails to perfectly invert D.. We discuss this in the following section.
3.3 Properties of TACoS
It is clear from inspection that both Algo- rithms [ and 2 perfectly reconstruct the it- erate Ta D(ro,s) s) for all s t if the restoration operator is a perfect inverse for the degradation operator. Hence in this sec- tion, we will discuss the reconstruction op- erator that fails reconstruct the image per- fectly i.e. incurs eгror. We first analyze the stability of these algorithms errors in the restoration operator and then theoretically show that for a simple blur degradation, the error incurred using algorithm is always greater than algorithm??
Figure 2: Comparison of sampling methods for uncon- ditional generation using cold diffusion on the CelebA dataset. Iterations 2, 4, 8, 16, 32, 64, 128, 192, and 256 are presented. Top: Algorithm I produces com- pounding artifacts and fails to generate a new image. Bottom: TACoS succeeds in sampling high quality image without noise.
For small values of æ and s, TACoS as described in 2 is tolerant of error in the restoration operator R. x + s for
To see why, consider a problem with linear degradation function of the form D(x, s)
a constant vector e. We choose this ansatz because the Taylor expansion of any smooth degradation D(z, s) around = ry. s has the form ((x,s) = x + s e(z) + HOT where HOT denotes higher order terms. Note, however, the analysis below requires to be constant that does not depend on x. The constant/zeroth-orddr term this Taylor expansion zero because we assumed above that the degradation operator satisfies D(x, 0) x.
For degradation D(x, 8) and any restoration operator R, the term - i in TACoS becomes
x. D(D(((x,, s). +
1) D( 8) D(R( s) ((((z,,))
= xo 8 R(x.: 8) - 8.E R(5,, 8) + (8 - 1) 3o + (s 1) D( 8-)
By induction, we that the algorithm outputs the value x, D( s) for all < t, regardless of the choice of R. In other words, for any choice of R, the iteration behaves the same as it would when R is perfect inverse for the degradation D.
By contrast, Algorithm [ does not enjoy this behavior even for small values of 8. In fact, when R is not a perfect inverse for D, zo is not fixed point of the update rule in Algorithm ① because To * D(R(z, 0), 0)- = R(x, 0) and hence errors compound. If R does not perfectly invert D, we should expect Algorithm| to incur errors, even for small values of s. Mcanwhile, for small values of s, the behavior of D approaches its first-order Taylor expansion, and Algorithm 2 becomes immune to errors in R. Figure 2 demonstrates the stability of TACoS described in Algorithm 2 vs. Algorithm for a deblurring model.
The above analysis is not a complete convergence theory but rather highlights a desirable theoretical property of our method that naive sampler lacks. However, we can prove that for a toy problem in which the blur operator removes one frequency at a time, the error incurred by sampling using Algorithm Ti is greater than the error incurred from using Algorithm 2 We present the proof of this claim A.9

---

4 Generalized Diffusions with Various Transformations
In this section, we take the first step towards cold diffusion by reversing different degradations and hence performing conditional generation. We will extend our methods to perform unconditional (i.e. from scratch) generation in Section 5 We emprically evaluate generalized diffusion models trained on different degradations with TACoS proposed Algorithm 2 We perform experiments on the vision tasks of deblurring, inpainting, and super-resolution. We perform
experiments on MNIST LeCun et al. [998]. CIFAR-10 Krizhevsky 2009). and CelebA [Liu et al. 2015]. In each of these tasks, gradually remove the information from the clean image, creating sequence of images such that (xo, t) retains less information than ( t 1). For these different tasks, we present both qualitative and quantitative results held-out testing dataset and demonstrate the importance of the sampling technique described in Algorithm 2 For all quantitative results in this section, the Frechet inception distance (FID) scores [Heusel al. 2017] for degraded and reconstructed images are measured with respect to the testing data. Additionnl information about the quantitative results, convergence criteria, hyperparameters, and architecture of the models presented below can be found in the appendix.
4.1 Deblurring
We consider a generalized diffusion based on a Gaussian blur operation (as opposed to Gaussian noise) in which an image step has more blur than at - 1. The forward process given the Gaussian kernels }Ga} and the image 1 at step / - 1 can thus be written as
It M, o xt-1 M, o... o M1 xo = M, xo = ()
where * denotes the convolution operator, which blurs an image using a kernel.
We train a deblurring model by minimizing the loss ( and then use TACoS to invert this blurred diffusion process for which we trained DNN to predict the clean image sg. Qualitative results are shown in Figure 3 and quantitative results in Table I Qualitatively, we can see that images created using the sampling process are sharper and in some cases completely different compared to the direct reconstruction of the clean image. Quantitatively we can see that the reconstruction metrics such as RMSE and PSNR get worse when we use the sampling process, but on the other hand FID with respect to held-out test data improves. The qualitative improvements and decrease in FID show the benefits of the generalized sampling routine, which brings the learned distribution closer to the true data manifold. Note: we compare the images reconstructed via Algorithm 2 with direct generation as compared to Algorithm []] This is because the image reconstruction via Algorithm Dis much worse than both direct generation and Algorithm Nevertheless, to back our claim, we present their results in A0
In the case of blur operator, the sampling routine can be thought of adding frequencies at each step. This is because the sampling routine involves ves the term D t) D (z 1) which in the case of blur becomes G2 * Zy G.-1 zg. This results in a difference of Gaussians, which is band pass filter and contains frequencies that were removed at step t. Thus, in the sampling process, we sequentially add the frequencies that were removed during the degradation process.
Degraded
Direct
TACoS
Original
9
2 3
4 2
22
Figure 3: Deblurring models trained on the MNIST, CIFAR-10, 10, and CelebA datasets. Left to right: degraded inputs D( T') direct reconstruction RDD(zo, T)), sampled reconstruction with TACoS described in Algorithm 2 and original image.
5

---

Table 1: Quantitative metrics for quality of image reconstruction using deblurring models.
Degraded SSIM
Sampled SSIM
Direct SSIM
Dataset
FID
RMSE
FID
RMSE
FID
RMSE
MNIST
438.59
0.287
0.287
4.69
0.718
0.154
5.10
0.757
0.142
CIFAR-10
298.60
0.315
0.136
80.08
0.773
0.075
83.69
0.775
0.071
CclebA
382.81
0.254
0.193
26.14
0.568
0,093
36.37
0.607
0.083
Inpainting
We define schedule of transforms that progressively grays-out pixels from the input image. We remove pixels using a Gaussian mask as follows: For input images of size n X n we start with a 2D Gaussian curve of variance 3, discretized into x array. We normalize the peak of the curve has value 1, and subtract the result from so the center of the mask as value O. We randomize the location of the Gaussian mask for MNIST and CIFAR- 10, but keep it centered for CelebA. We denote the final mask by 233-
Input images Za are iteratively masked for T steps via multiplication with a sequence of masks {zs, } with increasing 3.. We can control the amount of information removed at each step by tuning the 3 parameter. In the language of Section D (aro,4) () = z0 I 2 where the operator denotes entry-wise multiplication.
Figure 4 presents results on test images and compares the output of the inpainting model to the original image. The reconstructed images display reconstructed features qualitatively consistent with the context provided by the unperturbed regions of the image. We quantitatively assess the effectiveness of the inpainting models on each of the datasets by comparing distributional similarity metrics before and after the reconstruction. Our results are summarized Table 2
Note, the FID
scores here are computed with respect to the held-out validation set.
Degraded
Direct
TACoS
Original
3
6
3 6 O
3
6
Figure 4: Inpainting models trained on the MNIST, CIFAR-10, 10, and CelebA datasets. Left to right: Degraded inputs D(ro, direct reconstruction R(D(w T)), sampled reconstruction with TACoS described in Algorithm 2 and original image.
Table 2: Quantitative metrics for quality of image reconstruction using inpainting models.
Degraded SSIM
Sampled SSIM
Direct SSIM
Dataset
FID
RMSE
FID
RMSE
FID
RMSE
MNIST
108.48
0.490
0.262
1.61
0.941
0.068
2.24
0.948
0.060
CIFAR-10
40.83
0.615
0.143
8.92
0.859
0.068
9.97
0.869
0.063
CelebA
127.85
0.663
0.155
5.73
0.917
0.043
7.74
0.922
0.039
4.3
Super-Resolution
For this task, the degradation operator downsamples the image by factor of two in each direction. The final resolution of 2T is 4x4 for MNIST and CIFAR-10 and 2x2 in the case of Celeb-A. A. After each complinn the lower-resolution image is resized to the original image size, using nearest-neighbor interpolation. More details are available in Appendix A.3
Figure presents example testing data inputs for all datasets and compares the output of the super- resolution model to the original image. Though the reconstructed images are not perfect for the

---

more challenging datasets, the reconstructed features are qualitatively consistent with the context provided by the low resolution image. Table B compares the distributional similarity metrics between degraded/reconstrutted images and test samples.
Degraded
Direct
TACoS
Original
2 8 4
2 8 4
2 8
Figure 5: Superresolution models trained on the MNIST, CIFAR-I0 10, and CelebA datasets. Left to right: degraded inputs (xxo, T") direct reconstruction R(D (zo. T)), sampled reconstruction with and original image.
TACoS described in Algorithm 2
Table 3: Quantitative metrics for quality of image reconstruction using super-resolution models.
Degraded SSIM
Sampled SSIM
Direct SSIM
Dataset
FID
RMSE
FID
RMSE
FID
RMSE
MNIST
368.56
0.178
0.231
4.33
0,820
0.115
4.05
0.823
0.114
CIFAR-10
358.99
0.279
0.146
152.76
0.411
0.155
169.94
0.420
0.152
CelebA
349.85
0.335
0.225
96.92
0.381
0.201
112.84
0.400
0.196
5 Cold Generation
Diffusion models can successfully learn the underlying distribution of training data, and thus generate diverse, high quality images [Song et al.) 2021a] Dhariwal and Nichol [021 Jolicoeur- Marttina et al. [021] Ho et al. 2022]. We will first discuss deterministic generation using Gaussian noise and then discuss detail unconditional generation using deblurring. Finally, we provide proof of concept that the TACoS described in Algorithm 2 can be extended to other degradations.
5.1 Generation using deterministic noise degradation
Here discuss image generation using Section which we will later prove equivalent to DDIM [Song et al. 2021a]. We use the following degradation operator: ((x, t) Va,a vI 02
degradation presented in notation from
D is an interpolation between the data point æ and a sampled noise pattern E WO 1). During training, D is applied once and thus is sampled once for every image in every batch. However, sampling involves iterative applications of the degradation operator D, which poses the question of how to pick for the sequence of degradations D applied in single image generation.
There are three possible choices for z. The first would be resample for each application of D, but this would make the sampling process nondeterministic for fixed starting point. Another option is to sample noise pattern z once for each separate image generation and reuse in each application of D. In Table we refer to this approach as Fixed Noise. Finally, one can calculate the noise vector z to be used in step reconstruction by using the formula
R(z,,)
2(x,, ()
VI- ax
This method denoted Estimated Noise in Table turns out to be equivalent to the deterministic sampling proposed in Song al. [2021a]. We discuss this equivalence in detail in Appendix A.6
5.2 Image generation using blur
The forward diffusion process noise-based diffusion models the advantage that the degraded image distribution the final step I' is simply an isotropic Gaussian. One can therefore perform
7

---

(unconditional) generation by first drawing a sample from the isotropic Gaussian, and sequentially denoising it with backward diffusion.
When using blur as a degradation, the fully degraded images do not form a nice closed-form distribution that be modeled with simple methods. Note that every image zo degenerates to an 27 that is constant (i.e., every pixel is the same color) for large T' Furthermore, the constant value exactly the channel-wise mean of the RGB image 20, and can be represented with a 3-vector. This 3-dimensional distribution easily represented using a Gaussian mixture model (GMM). This GMM can be sampled to produce the random pixel values of a severely blurred image, which can be deblurred using cold diffusion to create new image.
sample from. They do, however, form simple enough distribution that can
Our generative model uses a blurring schedule where we progressively blur each image with a Gaussian kernel of size 27 27 over 300 steps. The standard deviation of the kernel starts 1 and increases exponentially at the rate of 0.01. We then fit a simple GMM with one component to the distribution of channel-wise means. To generate an image from scratch, we sample the channel-wise mean from the GMM, expand the 3D vector into 128 x 128 image with three channels, and then apply TACOS.
Empirically, the presented pipeline generates images with high fidelity but low diversity, as reflected quantitatively by comparing the perfect symmetry column with results from hot diffusion in Table 4 We attribute this to the perfect correlation between pixels of øT sampled from the channel- -wise mean Gaussian mixture model. To break the symmetry between pixels, add small amount of Gaussian noise (of standard deviation 0.002) to each sampled ør. As shown in Table the simple trick drastically improves the quality of generated images. We also present the qualitative results for cold diffusion using blur transformation in Figure 6 and further discuss the necessity of TACoS proposed in Algorithm 2 for generation in Appendix AA.7
Table 4: FID scores for CelebA and AFHQ datasets using hot (noise) and cold diffusion (blur transformation). Breaking the symmetry within pixels of the same channel further improves FID.
Hot Diffusion
Cold Diffusion
Dataset
Fixed Noise
Estimated Noise
Perfect symmetry
Broken symmetry
CelebA AFHQ
59.91 25.62
23.11 20.59
97.00
49.45 54.68
93.05
Figure 6: Examples of generated samples from 128 128 CelebA and AFHQ datasets using cold diffusion with blur transformation
5.3 Generation using other transformations
In this section, we provide proof of concept that generation can be extended to other transformations. Specifically, we show preliminary results on inpainting, super-resolution, and animorphosis. Inspired by the simplicity of the degraded image distribution for the blurring routine presented in the previous section, we use degradation routines with predictable final distributions here as well.
To use the Gaussian mask transformation for generation, we modify the masking routine so the final degraded image is completely devoid of information. One might think a natural option is to send all of the images to completely black image æ7, but this would not allow for any diversity
8

---

in generation. To get around this maximally non-injective property, we instead make the mask turn all pixels random. solid color. This still removes all of the information from the image, but allows us to recover different samples from the learned distribution via Algorithm by starting off with different color images. More formally, a Gaussian mask G, = II:= 23, is created in a similar way as discussed in the Section 4.2 but instead of multiplying it directly to the image zo. we create Ty G, 4o + (1- G+) c, where is an image of randomly sampled color.
For super-resolution, the routine down-samples a resolution of x These degraded images can be represented as one-dimensional vectors, and their distribution is modeled using one Gaussian distribution. Using the same methods described for generation using blurring described above, we sample from this Gaussian fitted distribution of the owver-dimensional degraded image space and pass this sampled point through the generation process trained on super- resolution data to create one output.
values in each channel.
Additionally to show one can invert nearly any transformation, we include a new transformation deemed animorphosis, where we iteratively transform human face from CelebA to an animal face from AFHQ. Though we chose CelebA and AFHQ for our experimentation, in principle such interpolation can be done for any two initial data distributions.
More formally, given image and random image z sampled from the AFHQ manifold, be written as T Vazz + vI a²2. Note this is essentially the same as the noising procedure, but instead of adding noise sample from the learned distribution, we sample a random image of an animal and use TACoS.
adding a progressively higher weighted AFHQ image. In order to
We present results for the CelebA dataset, and hence the quantitative results in terms of FID scores for inpainting, super-resolution and animorphosis are 90. 14, 92.91 and 48.51 respectively. We further show some qualitative samples in Figure 7 and in Figure [
Figure Preliminary demonstration of the generative abilities of other cold diffusins on the 128 128 CelebA dataset. The top row is with animorphosis models, the middle row is with inpainting models, and the bottom row exhibits super-resolution models.
6 Conclusion
Existing diffusion models rely on Gaussian noise for both forward and reverse processes. In this work, we find that the random noise can be removed entirely from the diffusion model framework, and replaced with arbitrary transforms. In doing so, our generalization of diffusion models and their sampling procedures allows us to restore images afflicted by deterministic degradations such as blur, inpainting and downsampling. This framework paves the way for a more diverse landscape of diffusion models beyond the Gaussian noise paradigm. The different properties of these diffusions may prove useful for a range of applications, including image generation and beyond.
7 Acknowledgements
This work was made possible by the ONR MURI program, the Office of Naval Research (N0001421 12557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2212182, and by the NSF TRAILS Institute (2229885).

---

References
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint wrXiv:1801.0197 2018.
Mikotaj Binkkoski,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 118801001401 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint arXiv:1612.021166 2016.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. volume 34, 2021.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations, ICLR 20/9. OpenReview.net, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 32, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23, 2022.
Alexia
Rémi Piché- Taillefer, loannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. International Conference on Learning Representations, 2021.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.003:: 003664 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. International Conference on Computer Vision Workshops, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXhv:2201.11799. 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. Advances Neural Information Processing Systems, 34, 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278 222222222222 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Christopher A. Metzler, Ali Mousavi, and Richard G. Baraniuk. Learned D-AMP: principled neural network based compressive image recovery. Advances in Neural Information Processing Systems, 30, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171, 2021.
10

---

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipa- tion. arXiv preprint arXiv:2206. /3397, 2022.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). arXiv preprint wrXiv//61.02266/ 2016.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.m 27X6::2104.07636 2021.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Jascha Sohl-Dickstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, volume JMLR Workshop and Conference Proceedings, 2015.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement, arXiv preprint 12.0422 2021.

---

A Appendix
A.1 Deblurring
For the deblurring experiments, we train the models on different datasets for 700,000 gradient steps. We use the Adam Kingma and Ba 2014) optimizer with learning rate 2 x 10-5. The training was done on the batch size of 32, and we accumulate the gradients every 2 steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995 which is updated after every 10 gradient steps.
For the MNIST dataset, we blur recursively 40 times, with discrete Gaussian kernel of size 11x11 and a standard deviation 7. In the case of CIFAR-10, we recursively blur with Gaussian kernel of fixed size 1 1x but at each step f, the standard deviation of the Gaussian kernel is given by 0.01 t + 0.35. The blur routine for CelebA dataset involves blurring images with a Gaussian kernel of 15x15 and the standard deviation of the Gaussian kernel grows exponentially with time t at the rate 0.01.
Figure 8 shows an additional nine images for each of MNIST, CIFAR-10 10 and CelebA. Figures 19 and 20 show the iterative sampling process using a deblurring model for ten example images from each dataset. We further show 400 random images to demonstrate the qualitative results in the Figure 21
Degraded
Direct
TACoS
Original
5
4
4
2
4
4
2
9
4
8
5
9
3
5
7
7
8
7
9
8
9
9 5
8
7 8
9
Figure 8: Additional examples from deblurring models trained on the MNIST, CIFAR- 10, and CelebA datasets. Left to right: degraded inputs D(zo, T') direct reconstruction R(D(zo, T)),, sampled reconstruction with TACoS described in Algorithm and original image.
A.2 Inpainting
For the inpainting transformation, models were trained on different datasets with 60,000 gradient steps. The models were trained using Adam Kingma and Ba (2014] optimizer with learning rate 2 x 10-5, We use batch size 64, and the gradients are accumulated after every 2 steps. The final model is an Exponential Moving Average of the trained model with decay rate 0.995. This EMA model is updated after every 10 gradient steps. For all our inpainting experiments Gaussian mask and T = 50 with 3, = and 3 = 3r + 0.1.
randomized
To avoid potential leakage of information due to floating point computation of the Gaussian mask we discretize the masked image before passing through the inpainting model. This was done by rounding all pixel values to the eight most significant digits.
12

---

Figure 10 shows nine additional inpainting examples on each of the MNIST, CIFAR- 10, and CelebA datasets. Figure demonsttattes an example of the iterative sampling process of an inpainting model for one image in each dataset.
A.3
Super-Resolution
We train the super-resolution model per Section 3.1 for 700,000 iterations. We use the Adam Kingma and Ba (2014) optimizer with learning rate 2 TO-5, The batch size is 32, and we accumulate the gradients every 2 steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps.
The number of time-steps depends on the size of the input image and the final image. For MNIST and for CIFARIO, the number of time steps is 3, takes three steps of halving the resolution to reduce the initial image down to X 4. For CelebA, the number time steps is 6 to reduce the initial image down x 2. For CIFARIO,
apply random crop and random horizontal flip for regularization.
Figure shows an additional nine super-resolution examples on each of the MNIST, CIFAR- 10, and CelebA datasets. Figure shows one example of the progressive increase in resolution achieved with the sampling process using a super-resolution model for each dataset.
A.4 Colorization
Here we provide results for the additional task of colorization. Starting with the original RGB- image To. we realize colorization by iteratively desaturating for T steps until the final image xr is fully gray-scale image. We series of three-channel 1 x 1 convolution filters z(a)) { {='(a), (a), (a)} with the form
{()) (63 + a1) (100)
2²) (# + a) (0
2 (a) = a (3 + a)
and obtain (xx,t) z(a:)= via schedule defined a1,.., 0t for each respective step. Notice that a gray image obtained when FT = 2(1 Zg-
We can tune the ratio to control the amount of information removed in each step. For our experiment, we schedule the ratio such that for every t we have
Z4 z(aa))
z(a1) xo z( (( Z0.
This schedule ensures that color information lost between steps is smaller in earlier stage of the diffusion and becomes larger as increases.
We train the models different datasets for 700,000 gradient steps. We Adam Kingma and Ba (014)) optimizer with learning rate x 10-5 We use batch size 32, and we accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 we use = 50 and for CelebA we use T 20.
8
8
8
8
8
8
Figure 9: Progressive inpainting of selected masked MNIST, CIFAR- 10, and CelebA images.
13

---

Degraded
Direct
TACoS
Original
0 1 2
D
1 2
D
1 2
3 4 5
3
4 5
3
4 5 8
6
7
8
6
7
8
6
7
12
2
Figure 10: Additional examples from inpainting models trained on the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs Dx T) direct reconstruction R(D(so, T)), sampled reconstruction with TACoS described in Algorithm and original image.
3 1333 3
Figure 11: Progressive upsampling of selected downsampled MNIST, CIFAR-10, and CelebA images. The original image is at the left for each of these progressive upsamplings.
We illustrate recolorization results in Figure 13 We present testing examples, as well as their grey scale images, from all the datasets, and compare the recolorization results with the original images. The recolored images feature correct color separation between different regions, and feature various and yet semantically correct colorization objects. Our sampling technique still yields minor differences in comparison to the direct reconstruction. although the change is not visually apparent. We attribute this to the shape restriction of colorization task, as human perception is rather insensitive minor color change. We also provide quantitative measurement for the effectiveness of our recolorization results in terms of different similarity metrics, and summarize the results in Table
Table Quantitative metries for quality of image reconstruction using recolorization models for all
three channel datasets.
Degraded Image SSIM
Reconstruction SSIM
Dataset
FID
RMSE
FID
RMSE
CIFAR-10
97.39 41.20
0.937
0.078
45.74
0.942
0.069
CelebA
0.942
0.089
17.50
0.973
0.042
14

---

Degraded
Direct
TACoS
Original
L 9 9
7
9 9
L
7 9
7 4 2
745
5
7 4 2
5
7 4 0
3
4
3
4
3 3
Figure 12: Additional examples from super-resolution models trained on the MNIST, CIFAR-10. 10, and CelebA datasets. Left to right: degraded inputs D ( T) dirrect reconstruction ((D T)), sampled reconstruction with TACoS described Algorithm and original image.
Degraded
Direct
TACoS
Original
Figure 13: Recolorization models trained on the CIFAR-10 10 and CelebA datasets. Left right: degraded inputs D( T)) direct reconstruction R(D(zo, T)), sampled reconstruction with TACoS and original image.
described in Algorithm 2
A.5
Image Snow
Here we provide results for the additional task of snowification, which is direct adaptation of the offical implementation of ImageNet-C snowification process Heendrycks and Dietterich (2019). To determine the snow pattern of given image Zo R CX we first construct seed matrix SA E R H where each entry is sampled from a Gaussian distribution N(H, The upper-left corner of SA is then zoomed into another matrix SB TR HxW with spline interpolation. Next, create a new matrix Se by filtering each value of SB with given threshold e₁ as
f0,
S S c1 S {
Sc
>
15

---

and clip each entry of Sc into the range [0, 1]. We then convolve Se using a motion blur kernel with standard deviation C2 create the snow pattern S and its up-side-down rotation S'. The direction of the motional blur kernel is randomly chosen as either vertical or horizontal. The final snow image is created by again clipping each value of xo + S + S" into the range [0, 1]. For simplicity, we abstract the process as a function o( S, Co, c1)-
Degraded
Direct
TACoS
Original
Figure 14: Additional examples from Desnowification models trained on the CIFAR-10 and CelebA datasets. Left to right: degraded inputs (I T) direct reconstruction Rz), T)), sampled reconstruction with TACoS described in Algorithm and original image.
To create series of T images with increasing snowification, we linearly interpolate co and ei between [egur <6nul and [ctann ccrdl respectively, to create co(() and
Then for each Ty. a seed matrix Sy is sampled, the motion blur direction is randomized, and we construct each related Ie by 2t h(20. Sz. ct),), e1 (t)). Visually, co(t) dictates the severity of the snow, while ci (t) determines how "windy" the snowified image seems.
For both CIFAR-10 and Celeb-A, we use the same Gaussian distribution with parameters = 0.55 and a 0.3 to generate the seed matrix. For CIFAR-10, we choose CO 1.15, afnd 0.7,
start
esturt = 0.05 and oind = 16, which generates a visually lighter snow. For Celeb-A, we choose c 1.15, and 0.55, autl 0.05 and "fTd 20, which generates a visually heavier snow.
We train the models on different datasets for 700,000 gradient steps. We use Adam Kingma and Ba 2014] optimizer with learning rate x 10-5 We use batch size 32, and accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 10 we use T = 200 and for CelebA we use T = 200. We note that the seed matrix is resampled for each individual training batch, and hence the snow pattern varies across the training stage.
A.6 Generation using noise Further Details
Here we show the equivalence between the sampling method proposed in Algorithm 2 and the deterministic sampling in DDIM [Song et al. (2021a]. Given the image 2; at step t, we have the restored clean image z from the diffusion model. Hence given the estimated do and It. we can estimate the noise =xx,,t) f) (or 2) as
Ft Varsi
2(1,,
/1 06
Thus, the D(ao: 1) and D( 1 - 1) can be written as
DiEo, t) varzo + vI 2
D 1 - 1) Voi-1zo +
using which the sampling process in Algorithm 2 to estimate Iz-1 can be written as,
16

---

Tt-1 = - t) + D(o0, t- -) (Va v1 a12) (Vat-1 V1 o(-12)
-aa--100 vI 0i--2
which is same as the sampling method as described in Song et al. [2021a] The only difference from
the original Song al. (2021a] is the order for estimating zig and The original [Sng al. [2021a]
paper estimated first and then used this to predict clean image To, while we first predict the clean image 2ig and then estimate the noise å.
A.7 Generation using blur transformation: Further Details
L
Figure 15: Examples of generated samples from 128 x 128 CelebA and AFHQ datasets using Method 2 with perfect symmetry.
Figure 15 shows that generation without breaking any symmetry within each channel quite promising well.
Necessity of Algorithm 2: In the case of unconditional generation, we observe a marked superiority in quality of the sampled reconstruction using Algorithm over any other method considered. For example, the broken symmetry case, the FID of the directly reconstructed images is 257.69 for CelebA and 214.24 for AFHQ, which are far worse than the scores of 49.45 and 54.68 from Table g In Figure 17 we also give qualitative comparison of this difference. We can also clearly see from Figure that Algorithm [ the method used in Song et al. [2021b] and Ho et al. 2020 completely fails to produce an image close to the target data distribution.
A.8 Algorithm i is same as DDIM/DDPM sampling
The sampling method proposed in (ong et al. 2021a in it's equation 12 is given
-t- /or-1 "predicted xo" + a:-1 - 0f50(0)) + afht
where (x;)) is the noise predicted by the diffusion model given xy and t. The term "predicted xo" or Lo can be computed directly given 2, and ea (2x)
Tt
20
Vai
17

---

Hence using 2 instead of eo (z)) and 2ig to indicate predicted clean image, we have
It-1 = V 0 40 + 0+ - of + O,6r
Thus, the sampling step can interpreted as follows: At each step 4, we start with a noisy image 2; and use the diffusion model to estimate the clean image di and the noise 2 that was added to this clean image di to get the noisy image zy- In order to move to lesser noisy image -1 one "adds back" lesser noise to the the "predicted clean image" fo. Now one can add back noise in ways, either the noise which was added to the clean image zo which is & or sample new uncorrelated noise Infact both of these noise can be added using 0₁ the hyperparameter that weighs the amount of each noise added. This is placed in the equation such that for any choice of og the standard deviation of noise added back is /I Q--1- For o. = O, we only add back the estimated noise & and no uncorrelated noise e, which is infact the DDIM sampling. While for o; = v(1 - we get the sampling method described in DDPM.
at) VV - 0z/ /
Nevertheless, for any choice of o₁, the sampling method involves a denoising operation which is I and adding back noise shown - D s - 1) in Algorithm I The only difference between different sampling methods explained in DDPM or DDIM is how one degrades the image back.
shown as R( s) in Algorithm
A.9 Discussion on Algorithm produces lesser error than Algorithm I
Consider the case where the degradation a simple blur operator that removes one frequency from an image every time f increases by one. We can write X 20 + I1 ... 2T + 2T, where each D; is the Fourier mode of X representing frequency T The degradation operator for this blur the signal has been blurred into a
D(X,1) 1) Je ... II- + 2T E Ty and when
constant vector.
Suppose we are performing the reverse process, and we begin at step with X; 3z + .T. Now in order to go from X; to Xt-1, we first use the diffusion model to predict X RR, t). This X has an analogous Fourier expansion X = Σrzo Ln. where èo is the highest frequency mode and Dr is the DC mode. At step I, the error E, can be defined as || X
For the Algorithm 1 in which X,- is given by 8,-1 = D(Ā, t - 1), we can expand the error as follows
| 1-1
11111
T
= ||
Σ
T3
Σ
1111
-=|-
i=t-1
T
||
Σ
(a: 2.)11
#=t-1
T
Σ
||,,, - 2;11
=
1=111
For the Algorithm 2 in which X,-1 is given by X,-1 = X, D(X, + D(S, 1), we can expand the eтror as follows
E
|| - (X, D( 11) + D(X,1 ( 1
= [1((X8 D(X, t- - 1)) - (X, - X(-1)112
T
T
T
T
24
Σ
24) [ x Σ x2)1
1=t
i=t-1
1=t
|
11-1 2
Hence, we can see that the error incurred at step using algorithm is ET - 2.112 which is
clearly greater than or equal to error incurred by using the algorithm 2 which is | x1-1l 1|. We
18

---

now further break down the analysis based on different scenarios depending on the quality of the reconstruction operator R:
1. R is perfect.
In this scenario, both Algorithm and Algorithm indistinguishable. and the error is for both.
2. R. is imperfect only for Xt-1 and reconstructs other signals perfectly. In this scenario, both Algorithm I and Algorithm 2 incur the same error, which is (#r-1 1z-1)²
3. is imperfect for more than one frequency This is the realistic scenario. In this case, the error incurred by Algorithm is strictly greater than Algorithm 2.
Thus we prove that for the realistic scenario when not a perfect reconstruction operator, emrrr incurred using Algorithm T always greater than the Algorithm 2
A.10 Empirical comparisons between Algorithm 2 and Algorithm 1
For all the degradations in Section we compare to "Direct Sampling" which is in fact the "one step" reconstruction of the input to Algorithm [
One of the main contributions of our work is Algorithm which outperforms Algorithm across all degradations. Infact the Algorithm [ worse than the one-step generation as well. To demonstrate this concretely we present the FID results in [6] where we can clearly see that scores for Algorithm are worse than both the one-step and Algorithm We also show qualitatively how bad the Algorithm iis in Figures 16 and T8 In fact, for the case of the celebA dataset, Algorithm fails drastically. while for CIFAR- 10 we can see high-freet high-frequency signals present in the image generation. Hence, we chose our bascline to be direct generation instead of Algorithm
Table 6: FIDs for blur degradation for Algorithm Algorithm and Direct Reconstruction. This table demonstrates that Algorithm is worse than both the one-step generation and Algorithm 2
Dataset
Direct Generation
Algorithm
Algorithm
MNIST
5.10
8.24
4.69
CIFAR-10
83.69
97.89
80.08
CelebA
36.37
299.61
26.14
19

---

Algorithm
Algorithm
6
4
1920
0
1
3
7
Figure 16: Comparison of Algorithm T and Algorithm 2 We demonstrate that Algorithm performs much worse than Algorithm and falls completely for CelebA dataset.
Figure 17: Comparison of direct reconstruction with sampling using TACoS described in Algorithm 2 for generation with blur transformation and broken symmetry. Left-hand column is the initial cold images generated using the simple Gaussian model. Middle column has images generated in one step (i.e. direct reconstruction). Right-hand column are the images sampled with TACoS described in Algorithm 2 We present results for both CelebA (top) and AFHQ (bottom) with resolution 128 128.
20

---

Figure 18: Comparison of Algorithm (top row) and Algorithm 2 (bottom row) for generation with Method 2 and broken symmetry 128 128 CelebA dataset. We demonstrate that A Algorithm T fails completely to generate a new image.
21

---

9 16110
5 3 ل0
Figure 19: Progressive deblurring of selected blurred MNIST and CIFAR-10 images.
22

---

Figure 20: Progressive deblurring of selected blurred CelebA images.
23

---

Figure 21: Deblurred Cifar images
24


Reviews:
[{"review_id": "dgXqWHbBh4", "reviewer": "Reviewer_SFSB", "summary": "This paper introduces a method for image generation based on generic degradation and reconstruction operators. The approach generalizes diffusion models, which correspond to degradation by additive Gaussian noise, and reconstruction by denoising. In TACoS, the sampling scheme is agnostic to the choice of image degradation, and the corresponding restoration operator is learned via least squares regression over the data. \n\nThe authors additionally introduce a sampling iteration with a correction term (Algorithm 2) that induces first order cancellation of errors induced by improper learning of the reconstruction operator. The correction term is shown to greatly improve performance over a naive approach, since it prevents blow-up of fitting error over multiple iterations. The authors also prove that in a toy problem (degradation via frequency filtering) that Algorithm 2 has smaller reconstruction error than the naive approach. \n\nFinally, the authors demonstrate that TACoS can be used for sampling and reconstruction with a variety of degradation operators, such as deblurring, inpainting, and superresolution. First, they show that in these cases, solving the reconstruction problem associated with each degradation is feasible for large-scale image datasets such as CIFAR-10 and CelebA. Then, they show that the blur transformation can be used to sample CelebA and AFHQ images, albeit with significantly reduced sample quality and diversity. They finally show as a proof of concept that other transformations such as inpainting, super-resolution, and animorphosis, can also be used to generate samples. ", "questions": "What are some potential applications (beyond sampling) for TACoS with non-standard degradation operators like animorphosis? \n\nNote: it's a bit confusing to report RMSE in Tables 1-3 but to then discuss the PSNR in the text. It would be helpful stick to one throughout the paper. ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impacts of their work. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clarity: the paper is well written and very clear. To the best of my knowledge the derivations are correct.\n- Novelty: the idea behind this paper is interesting and novel to the best of my knowledge. However, as I will discuss in below, the practical value of this idea is unclear. \n- Low computational cost: the proposed method is simple and computationally chip, given that it only requires fitting one regression model over the dataset. \n- Experimental methodology: the experiments in this paper clearly demonstrate that TACoS can be used for sample generation under a variety of datasets and image transforms. It is interesting that the method can be used to generate samples with arbitrary degradation and reconstruction operators, as opposed to Gaussian noising and denoising via score-matching. ", "weaknesses": "- Unclear practical value: in my opinion, the practical value of this paper is unclear because the proposed algorithm appears to have low sample quality and diversity. There are many existing methods for deterministic iterative sampling algorithms, notably flow based methods like Continuous Normalizing Flows [1] and Probability Flow ODE [2], which can both eliminate the need for sampling noise and attain high sample quality. \n- Unclear applications: it is interesting that the proposed method can use arbitrary image transformations, which may pave the way for other applications beyond sample generation. However, it is currently unclear what these applications may be, which further limits the value of this approach. \n- Lack of baselines: the sample generation experiments in Table 4 should also include baseline values, representing the FID that can be attained by existing methods such as vanilla DDPM or GAN based methods.\n\n[1] Building Normalizing Flows with Stochastic Interpolants (Albergo and Vanden-Eijnden, 2023)\n\n[2] Score-Based Generative Modeling through Stochastic Differential Equations (Song et al., 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We address each of your points below.\n\n> Unclear practical value and unclear applications\n\nWe agree with the reviewer that cold diffusion does not outcompete the much more highly engineered and compute intensive state-of-the-art Gaussian diffusion models. We also agree that multiple works like [1], [2], [3], [4] remove the need to use random gaussian noise during sampling (i.e. at inference time). Moreover, works like [5] and [6] show that degradations other than gaussian noise are useful as diffusion processes.\n\nThis work is not meant to be a SOTA method paper, and we have updated our working draft to clarify this. Instead, this work challenges both theory frameworks where noise is a **critical** component for learning the training distribution and the engineering status-quo that has not been explored beyond Gaussian noise or other stochastic degradations. We believe our experiments effectively demonstrate the surprising fact that entirely noise-free schemes (blur, inpainting, super-resolution) can still work. \n\nThis observation is not only of theoretical interest, but it may move the community to explore other kinds of diffusion for common tasks like upsampling, and with further research investments such methods may someday become standard tools. Furthermore, by exploring deterministic degradations beyond Gaussian noise, we open up the possibility of finding the best degradation that will work most effectively for image generation. This represents a significant departure from traditional diffusion models and offers a new pathway for research and experimentation in the field. By expanding the horizons of what's considered in terms of degradation, we may unlock new avenues and insights that can contribute to advancing the state of generative models.\n\n\n\n[1] Building Normalizing Flows with Stochastic Interpolants \\\n[2] Score-Based Generative Modeling through Stochastic Differential Equations \\\n[3] Denoising Diffusion Implicit Models \\\n[4] Elucidating the Design Space of Diffusion-Based Generative Models \\\n[5] Structured Denoising Diffusion Models in Discrete State-Spaces \\\n[6] DiffusionDet: Diffusion Model for Object Detection\n\n\n\n\n\n\n> Lack of baselines: the sample generation experiments in Table 4 should also include baseline values, representing the FID that can be attained by existing methods such as vanilla DDPM or GAN based methods.\n\nIn Table 4 of our paper, we compare our \"cold diffusion\" which uses a deterministic blur degradation to a noise-based degradation, which we call \"hot diffusion\", this really is a vanilla diffusion model. The noise-based degradation uses Gaussian noise and the sampling provided in Algorithm 2. Though it may appear different from existing methods like DDPM  or DDIM, as discussed in section 5.1, the sampling method underlying the proposed TACoS is equivalent to DDIM. We present the fact that DDIM is a special case of our Algorithm 2 for the case of Gaussian noise in Appendix A.6. Our revised manuscript now clarifies this.\n\n\n\n\n>What are some potential applications (beyond sampling) for TACoS with non-standard degradation operators like animorphosis?\n\nWe present results for various non-standard degradations like animorphosis or snow to show that our proposed sampling algorithm is agnostic to any degradation and is not designed for one specific degradation. Another use of animorphosis can be in building flows between any two arbitrary distributions.\n\n>Note: it's a bit confusing to report RMSE in Tables 1-3 but to then discuss the PSNR in the text. It would be helpful stick to one throughout the paper.\n\nWe thank the reviewer for bringing this confusion to our notice. We have now revised our text and use RMSE everywhere, aligning the metrics throughout the paper for consistency.  We'll include these edits in our camera ready version.\n\nThank you again for your thoughtful review. We made an effort to address your feedback including paper edits and would appreciate it if you would consider raising your score in light of our response.  Do you have any additional questions we can address?\n\n"}, {"review_id": "DtATGlgKe1", "reviewer": "Reviewer_VoTU", "summary": "This work introduces a novel approach called cold diffusion, in which both the forward and backward processes are deterministic. The authors propose a scheme called Tacos, which predicts x_{s-1} from x_s by leveraging the estimated increment D(\\hat{x}_0, s) - D(\\hat{x}_0, s-1).", "questions": "The output quality of other cold generations is questionable. For instance, in the case of super-resolution, starting from a 2x2 image for the backward process may lead to similar issues of low diversity in the generated samples.", "limitations": "This reviewer appreciates the authors' introduction of a novel deterministic generative diffusion framework. However, it is important for the proposed framework to demonstrate comparability to reasonable GAN models in terms of diversity and output quality. The inclusion of Gaussian noise (as seen in predictor-corrector models or Langevin dynamics) or a similar randomization process is typically considered necessary and serves as a key component in diffusion processes. To justify the diffusion without noise, it is crucial for the authors to provide results that are at least comparable to decent GAN models.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The autors propsed nontrivial generalization of diffusion generative model to simple and straightforward deterministic process.\n", "weaknesses": "1. The justification of TACos (Section 3.3) appears weak.\n- Higher-order terms may have a significant impact that is not adequately addressed.\n- TACoS is likely to fail in standard Gaussian diffusion scenarios.\n\n2. As mentioned in Section 5.2, the generated samples exhibit low diversity.\nThis indicates a failure to accurately recover the sample distribution, which is a primary objective of diffusion generative models.\n\n3. Beyond the issue of diversity, the quality of the generated samples is also questionable.\nIn the appendix, the 128x128 generated samples demonstrate significantly poorer quality compared to regular generative models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing the nontrivial generalization of our work. We address each of your points below:\n\n> Higher-order terms may have a significant impact that is not adequately addressed.\n\nIn lines 156-160, we mention that the analysis in Section 3.3 is not a complete convergence theory but rather is to highlight a desirable theoretical property of TACoS that a naive sampler lacks. In the analysis presented in lines 138-148, we assume t to be approximately 0. This implies that the difference between $D(x, t=0)$ and  $D(x, t=\\epsilon)$ is sufficiently small to ignore higher-order terms, compared to the first order terms. \\\nThis analysis is limited to small $t$ and raises the question if this advantage of Algorithm 2 (TACoS) over the naive Algorithm 1 extends to region when t is not close to 0. To this end, we prove in Appendix A.9 that for a toy problem in which the blur operator removes one frequency at a time, the error incurred by Algorithm 1 is higher than Algorithm 2 for all $t$.\n\n\n> TACoS is likely to fail in standard Gaussian diffusion scenarios.\n\nWe appreciate your concern, but as we discuss in detail in Section 5.1 and Appendix A.6, DDIM is a special case of the proposed TACoS algorithm when applied to Gaussian noise. This means that both Algorithm 1 and Algorithm 2 are equivalent to DDIM in the case of Gaussian diffusion. We explicitly verify this in Table 4, where we train a hot diffusion model, i.e. we use Gaussian noise as degradation. As standard diffusion is recovered as a special case, this indeed works well and TACoS does not fail for standard Gaussian diffusion scenarios.\n\n\n> The output quality of other cold generations is questionable. For instance, in the case of super-resolution, starting from a 2x2 image for the backward process may lead to similar issues of low diversity in the generated samples.\n\nIn our experiments, we observed that the *starting point*, whether a 2x2 image is used for down-sampling based degradation in Table 3 or a 1x1 image in the case of cold diffusion in Table 5, has a distinct impact on the final results because of the nature of degradation involved. Your observation about the potential issues of low diversity in the generated samples starting from a 2x2 image is insightful. However, we found that the underlying cause of sub-par performance is related to the number of steps, not necessarily the diversity in the 2x2 region.\n\nSpecifically:\n\n* In the case of blur-based cold generations (Table 4), perfect symmetry at the starting point leads to FID scores similar to those obtained when starting from a 2x2 image using downsampling as degradation (Table 3).\n* The key difference between these results lies in the sampling steps. For cold generation, we utilized 300 steps to generate 128x128 CelebA images, while in the case of downsampling-based degradation, we use just 6 steps.\n* Consequently, we believe that the limitation in performance is attributable to the fewer number of steps rather than less diversity in the 2x2 region.\n* Moreover, for downsampling-based degradation, our results demonstrate that starting from 2x2 images produces more diverse samples compared to 1x1 images for the CelebA dataset.\n\n\n> This reviewer appreciates the authors' introduction of a novel deterministic generative diffusion framework. However, it is important for the proposed framework to demonstrate comparability to reasonable GAN models in terms of diversity and output quality. The inclusion of Gaussian noise (as seen in predictor-corrector models or Langevin dynamics) or a similar randomization process is typically considered necessary and serves as a key component in diffusion processes. To justify the diffusion without noise, it is crucial for the authors to provide results that are at least comparable to decent GAN models.\n\nWe thank the reviewer for their appreciation of our work on introducing a novel deterministic generative diffusion framework. In Table 4, where we compare cold diffusion to \"hot diffusion\" (which uses Gaussian noise as degradation), the use of estimated noise in TACoS sampling makes it equivalent to DDIM [1], a deterministic sampling method, which is in fact better than sampling methods that use noise like DDPM [2]. Hence on our end, we compare our cold diffusion with a standard and widely accepted generative model. \\\nMoreover, this is not meant to be a SOTA method paper (we have updated our local draft to clarify). Instead, this work challenges both theory frameworks where noise is a **critical** component for learning the training distribution and the engineering status-quo that has explored little beyond Gaussian noise. We believe our experiments effectively demonstrate that entirely noise-free schemes (blur, inpainting, super-resolution) can still work. This will not only be of theoretical interest, but it may move the community to explore other kinds of diffusion for common tasks like upsampling, and with further research investments such methods may someday become standard tools. \n\n[1] Denoising Diffusion Implicit Models \\\n[2] Denoising Diffusion Probabilistic Models\n\nThank you again for your thoughtful review. We made an effort to address your feedback including paper edits and would appreciate it if you would consider raising your score in light of our response.  Do you have any additional questions we can address?\n"}, {"review_id": "W7wToXREvY", "reviewer": "Reviewer_pgXS", "summary": "This paper extends the Gaussian diffusion model toward arbitrary image-to-image translations, named Cold Diffusion. Specifically, the authors define a generalized forward diffusion process and its training process, then propose a novel Transformation Agnostic Cold Sampling (TACoS) process for generations.  Experiments show that Cold Diffusion can effectively achieve image generation by learning image-to-image translation. ", "questions": "Is it possible to apply the diffusion process to any domain translation, i.e., dimension agnostic and modality agnostic?", "limitations": "1. It seems that the image-to-image translation needs to have invariant dimensions.\n2. It will be interesting to have a mathematical analysis of Cold Diffusion.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper proposed a novel idea that the diffusion process can be applied to arbitrary image-to-image translations, not limited to Gaussian noise. The ideas proposed in this paper have achieved a certain impact in the field of diffusion models and inspired a lot of work. In light of this, I recommend that the paper be accepted.", "weaknesses": "The author only provides an empirical formulation, without rigorous theoretical analysis. Nonetheless, this cannot overshadow the novelty of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your careful review and positive assessment of our work. Your recognition of the novel approach and potential impact in the field of diffusion models is highly encouraging. Below, we address the points you raised:\n\n> Is it possible to apply the diffusion process to any domain translation, i.e., dimension agnostic and modality agnostic?\n\nYes, we believe our proposed algorithm is modality agnostic and can be adapted to different modalities like speech, text, etc. though diffusion modeling as a method is difficult to apply in variable-dimension settings. For example, text can occur in various lengths and diffusion models whose corruption process can result in text of different lengths present difficulties.  Solving this problem with Cold Diffusion may or may not be more difficult.\n"}, {"review_id": "dxWjRKX2UO", "reviewer": "Reviewer_pjEM", "summary": "The paper presents a general method for learning to de-corrupt datapoints in order to perform generative modelling. The paper proposes a sampling method that applies iterative updates to the current state that significantly outperforms a naive sampling algorithm which reconstructs and denoises alternatively. The authors present a wide variety of corruption processes and outperform a single step model and the naive algorithm.", "questions": "The inpainting results seem so much better than other results in terms of sample quality. When you say Figure 4/Figure 10 is showing 'test images' do you actually mean that these were held out during training or has the network seen those during training. This should be made very clear since it seems that the network has just memorized those images (at least on celebA).", "limitations": "Discussed in weaknesses section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Extending the idea of creating a generative model from corrupting data and learning to reconstruct it to a more general framework is a problem I believe many in this field are interested in. Not least for me personally, I have been interested in this problem and their sampling algorithm would have been useful for me to know when investigating this topic. This paper takes a very general approach making very few assumptions on the corruption process and makes a non-trivial contribution by coming up with a sampling algorithm that can perform iterative updates to generate new data which is found to be crucial for this type of method to work well. \n\nThe paper and methodology have flaws as I discuss next but I do believe that researchers will build on this work and find this contribution useful. The line of work is more engineering focussed and less based on theory which is not necessarily wrong and I think the novelty and likely interest in the topic slightly outweighs the negatives in this case.", "weaknesses": "Starting out, the performance of the method is just not that good in terms of sample quality. I appreciate that getting methods to work well takes development over the course of multiple papers but I worry that this is a limitation of the method itself due to most corruptions tested not working well compared to standard diffusion models.\n\nWhen using this framework as a generative model there appear to be issues when the deterministic corruption transformation is not 1-1 due to the corrupted space being much smaller e.g. the space of images with constant colour or completely blurred images. The authors report problems with reduced diversity because of this and need to add a little bit of noise to increase diversity. I think this is quite a major flaw in the design because unlike standard diffusion models that can rely on maximum likelihood arguments to enable coverage of the data distribution, there are no such guarantees here and it is not clear how much the noise trick alleviates the issue. This flaw seems quite fundamental to some corruption process and a proper investigation into the diversity of samples would be good. Its a little strange how in effect all of the data distribution is being squeezed into this small subspace and then perturbing around the subspace a little bit induces diversity, this seems to be quite ill-conditioned in how small changes in input make large changes in generated output.\n\nThe claims about new questions being raised as to the necessity of noise in generation should be reigned in because of the popularity and performance of flow matching type methods that are deterministic during training (given a randomly sampled pair) and sampling. The links to these methods should also be discussed especially in the case of corruptions such as animorphosis which is quite similar to a flow between two arbitrary data distributions and building a interpolating bridge between two random samples from them.\n\nThe sections on deblurring and inpainting should be better introduced as the main paper talks about generative models and so this section comes as a bit of a surprise when I would have expected pure unconditional generation as the initial experiment.\n\nTypos:\n equation after line 145, need .e on final line\n proof A.9, need i = t-1 on third line of E_t^2. How do you move the sum out of the norm on the final line ?\n\nEdit after rebuttal: I have read the author's rebuttal and as I mention in my reply,  my points regarding the ill-conditioned nature of the generative process and poor performance are still concerning for me and I intend to leave my score as it is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> the performance of the method is just not that good in terms of sample quality.\n\nWe acknowledge that the quality of images generated in Section 5 is not comparable to that of Gaussian noise as a degradation. This work is not meant to be a SOTA method paper, instead we challenge both theory frameworks where noise is a critical component for learning the training distribution and the engineering status-quo that has not made explorations beyond Gaussian noise. We believe our experiments effectively demonstrate that entirely noise-free schemes (blur, inpainting, super-resolution) can still work.\n\n> The sections on deblurring and inpainting should be better introduced as the main paper talks about generative models and so this section comes as a bit of a surprise when I would have expected pure unconditional generation as the initial experiment.\n\nWe thank the reviewer for their valuable suggestion, on our end we introduced conditional generation in Section 4 before unconditional generation in Section 5 because we wanted to present the conditional generation as a first step as it occurred in our own project. If one fails to perform conditional generation, then one will fail on unconditional generation as well which is a more difficult problem.\n\n> Typos: equation after line 145, need .e on final line proof A.9, need i = t-1 on third line of E_t^2. How do you move the sum out of the norm on the final line ?\n\nWe thank the reviewer for pointing us to these typos which we have now fixed in the current revision. \nIn the toy problem discussed in the Appendix A.9, we consider a blur operator that removes one frequency from image every time $t$ increase by one. Hence, given a random sample X, the vevtors $x_j$ and $x_k$ for $j \\neq k$ in the expansion $X = \\sum_{i=0}^{T} x_i$ are orthogonal to each other. \\\nThus we have \\\n$E_t ^ 2 = ||\\sum_{i=t-1}^{T} (x_i - \\hat x_i) || ^ 2$ \\\n$E_t ^ 2 = \\sum_{i=t-1}^{T} || (x_i - \\hat x_i) || ^ 2 +  \\mathop{\\sum\\sum}_{j\\neq k} (x_j - \\hat x_j)^T(x_k - \\hat x_k)$\n\nSince both $X$ and $\\hat X$ have the same orthogonal basis, i.e. the fourier directions, we have terms $x_j^Tx_k$, $x_j^T\\hat x_k$, $x_k^Tx_j$ and $x_k^T\\hat x_j$ to be equal to zero. This results in \\\n$E_t ^ 2 = \\sum_{i=t-1}^{T} || (x_i - \\hat x_i) || ^ 2$\n\n\n\n> When you say Figure 4/Figure 10 is showing 'test images' do you actually mean that these were held out during training or has the network seen those during training.\n\nIn all of our experiments, we evaluate our methods on the held-out testing dataset. We specify the evaluation details in lines 169-173.\n\nThank you again for your thoughtful review. We made an effort to address your feedback including paper edits and would appreciate it if you would consider raising your score in light of our response.  Do you have any additional questions we can address?"}]

Author rebuttal:
We thank all of our reviewers for their thoughtful comments. Based on all the suggestions, we have updated our draft and we would like to highlight a few central contributions of our work.
1.  In this paper, we aim to challenge the common belief that Gaussian noise in any form, either during training or sampling is **necessary** for diffusion models to work. We believe this will not only be of theoretical interest, but it will move the community to investigate other components that could improve diffusion model such as model architecture or training setup. 
2. We demonstrate the above point by using different types of **deterministic** degradations. We are enthusiastic readers of other papers that use alternative noise-based degradations like gamma or salt-and-pepper noise, as mentioned, but whether alternative noise-based degradations are also possible is not a question we are investigating in this work.
3. We noticed that where a few questions regarding baselines. We want to highlight that the presented hot diffusion is in fact DDIM. We present the equivalence of our proposed algorithm TACoS in section 5.1 and Appendix A.6. We have clarified this in our revision.

Overall, we believe our findings that cold diffusions can perform high-quality generation is surprising and provides a key insight in the burgeoning study of diffusion models as a whole. Hence, we believe it would be valuable for the community for this work to appear at NeurIPS.



Meta-review: The reviewers and AC appreciated the novelty of the paper and the generality of the proposed deterministic generative diffusion framework. It is clear from the reviewer comments and the discussion that the proposed method is not state-of-the-art in generation in terms of sample quality. The practical uses of the proposed perturbations is not clear as reviewers raised. It is interesting that the proposed method can use arbitrary image transformations, which may open directions for future applications beyond sample generation or solving inverse problems. Animorphosis and snow perturbations are cool examples of what this method can achieve. (A google search suggests also 'Therianthropy' as a suitable term). The paper is very well written and clarifies also related recent work in a very clear framework that will benefit readers.  Therefore,  I think this paper deserves to be accepted for publication in Neurips due to its novelty, clarity and potential for future applications.
Decision: Accept (poster)