Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
Authors: Arpit Bansal
Eitan Borgnia
Hong-Min Chu
Jie S. Li
Hamid Kazemi
Furong Huang
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)

Abstract: Standard diffusion models involve an image transform  -- adding Gaussian noise -- and an image restoration operator that inverts this degradation.  We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes.

Under review as a conference paper at ICLR 2023
COLD DIFFUSION: INVERTING ARBITRARY IMAGE TRANSFORMS WITHOUT NOISE
Anonymous authors Paper under double-blind review
ABSTRACT
Standard diffusion models involve image transform adding Gaussian noise and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact entire family of generative models can be constructed by varying this choice. Even when using completely determinis- tic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create genera- tive models. The success of these fully deterministic models calls into question the community's 's understanding of diffusion models, which relies on noise in ei- ther gradient Langevin dynamics variational inference, and paves the way for generalized diffusion models that invert arbitrary processes.
Original
Forward
Degraded
Reverse
Generated
Mask Animorph Blur Noise
Snow Pixelate
Demonstration of the forward and backward processes for both hot and cold diffusions. While standard diffusions are built Gaussian noise (top row), show that generative models can be built on arbitrary and even noiseless/cold image transforms, including the ImageNet-CC snow- ification operator, and an animorphosis operator that adds random animal image from AFHQ.
Figure 1:
INTRODUCTION
Diffusion models have recently emerged as powerful tools for generative modeling (Ramesh al.) 2022 Diffusion models come in many flavors, but all are built around the concept of random noise removal; trains image restoration/denoising network that accepts image contaminated with Gaussian noise, and outputs a denoised image. At test time, the denoising network is used to convert pure Gaussian noise into photo-realistic image using an update rule that alternates between

---

Under review as a conference paper at ICLR 2023
applying the denoiser and adding Gaussian noise. When the right sequence of updates is applied, complex generative behavior is observed.
The origins of diffusion models, and also theoretical understanding of these models, are strongly based on the role played by Gaussian noise during training and generation. Diffusion has been understood as random walk around the image density function using Langevin dynamics (Sohl- Dickstein et a 2015; Song & Ermon 2019), which requires Gaussian noise in each step. The The walk begins in high temperature (heavy noise) state, and slowly anneals into "cold" state with little if any noise. Another line of work derives the loss for the denoising network using variational
inference with Gaussian prior (Ho al. 2020, Song et al. 2021a Nichol & Dhariwal 2021).
In this work, we examine the need for Gaussian noise, or any randomness at all, for diffusion models to work in practice. We consider generalized diffision models that live outside the confines of the theoretical frameworks from which diffusion models arose. Rather than limit ourselves to models built around Gaussian noise, we consider models built around arbitrary image transformations like blurring, downsampling, etc. We train a restoration network to invert these deformations using simple Ep loss. When we apply a sequence of updates at test time that alternate between the image restoration model and the image degradation operation, generative behavior emerges, and we obtain photo-realistic images.
The existence of cold diffusions that require no Gaussian noise (or any randomness) during training or testing raises questions about the limits of our theoretical understanding of diffusion models. also unlocks the door for potentially new types of generative models with very different properties than conventional diffusion seen so far.
2 BACKGROUND
Both the Langevin dynamics and variational inference interpretations of diffusion models rely on properties of the Gaussian noise used in the training and sampling pipelines. From the score- matching generative networks perspective (Song & Hrmon 2019 Song et al., 2021b), noise in the training process is critically thought to expand the support of the low-dimensional training distribu- tion to set of full measure in ambient space. The noise is also thought to act data augmentation to improve score predictions in low density regions, allowing for mode mixing in the stochastic gradient Langevin dynamics (SGLD) sampling. The gradient signal in low-densit regions can be further improved during sampling by injecting large magnitudes of noise in the early steps of SGLD and gradually reducing this noise in later stages.
Kingma et al. ) propose a method to learn a noise schedule that leads to faster optimization. Using a classic statistical result, Kadkhodaae & Simoncelli (2021) show the connection between removing additive Gaussian noise and the gradient of the log of the noisy signal density in determin- istic linear inverse problems. Here, we shed light on the role of noise in diffusion models through theoretical and empirical results in applications to inverse problems and image generation.
Iterative neural models have been used for various inverse problems (Romano al. 2016 Metzler et al. 2017). Recently, diffusion models have been applied to them (Song et al., 2021b) for the
problems of deblurring, denoising, super-resolution, and compressive sensing (Whang
2021
Kawar et al. 2021 Saharia et al. 2021 Kadkhodaie & Simoncelli] 2021).
Although not their focus, previous works on diffusion models have included experiments with deter- ministic image generation (Song et al. [20 Dhariwal & Nichol/2021] Karras et al.] (022) and in
selected inverse problems (Kawar et al. 2022). ) Recently, Rissanen et al. (2022) use a combination
of Gaussian noise and blurring as a forward process for diffusion. Though they show the feasibility of a different degradation, here we show definitively that noise is not a necessity in diffusion models, and we observe the effects of removing noise for number of inverse problems.
Despite prolific work on generative models in recent years, methods to probe the properties of learned distributions and measure how closely they approximate the real training data are by means closed fields of investigation.
Indirect feature space similarity metrics such as Inception Score (Salimans et al., 2016), Mode Score (Che et al. 2016), Frechet inception distance (FID) (Heusel et al., (017]) and Kernel inception distance (KID) (Binkowski et al. 2018) have been proposed and adopted to some extent, but they
2

---

Under review as a conference paper at ICLR 2023
have notable limitations (Barratt & Sharma 2018). To adopt popular frame of reference, will use FID as the feature similarity metric for our experiments.
3 GENERALIZED DIFFUSION
Standard diffusion models are built around two components. First, there is an image degradation operator that contaminates images with Gaussian noise. Second, trained restoration operator is created to perform denoising. The image generation process alternates between the application of these two operators. In this work, we consider the construction of generalized diffusions built around arbitrary degradation operations. These degradations can be randomized (as in the case of standard diffusion) or deterministic.
3.1 MODEL COMPONENTS AND TRAINING
Given an image Zy E R.W. consider the degradation of Ty by operator D with severity t, denoted It (zu, t). The output distribution D(ro, t) of the degradation should vary continuously in t, and the operator should satisfy D(7g, 0) 20-
In the standard diffusion framework, D adds Gaussian noise with variance proportional to t. In our generalized formulation, we choose / to perform various other transformations such as blurring, masking out pixels, downsampling, and more, with severity that depends t. We explore range of choices for D in Section
We also require a restoration operator R that (approximately) inverts D. This operator has the prop- erty that R(x,, 1) x To. In practice, this operator is implemented via a neural network parameterized by 0. The restoration network trained via the minimization problem
min Ex~x Ro
Ro (z, t),t) æ=|,
where x denotes a random image sampled from distribution X' and || denotes a norm, which we take to be & in experiments. We have so far used the subscript Rg to emphasize the dependence of R on 0 during training, but we will omit this symbol for simplicity the discussion below.
3.2 SAMPLING FROM THE MODEL.
After choosing degradation D and train- ing model R to perform the restoration, these operators can be used in tandem to in- vert severe degradations by using standard methods borrowed from the diffusion liter- ature. For small degradations (f zs 0), a single application of R can be used to ob- tain a restored image in one shot. How- ever, because R is typically trained ing simple convex loss. it yields blurry results when used with large t. Rather, diffusion models (Song et al.) 2021a Ho al., 2020) perform generation by itera- tively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm
Algorithm Naive Sampling
Input: A degraded sample I; for = t, -,,..., do
to - ())
2-- D 1)
end for Return: 350
Algorithm Transformation Agnostic Cold Sampling for = t, 1,..., do
Input: A degraded sample
20 + R(x..8) 8)
x x--1 = Vg D(2u, 8) + D(io.s
end for
When the restoration operator is perfect, i.e. when R(D 1), () = xop for all I, one can easily see that Algorithm I produces exact iterates of the form I, D(ro, s). But what happens for imperfect restoration operators? In this case, errors can cause the iterates w. to wander away from D(zo, s), and inaccurate reconstruction may occur.
We find that the standard sampling approach in Algorithm | (explained further in A.8) works well for noise-based diffusion, possibly because the restoration operator R has been trained to correct (random Gaussian) errors in its inputs. However, we find that it yields poor results in the case of cold diffusions with smoothidifferentiable degradations demonstrated for deblurring model in
3

---

Under review as a conference paper at ICLR 2023
Figure We propose Transformation Agnostic Cold Sampling (TACoS) in Algorithm which find to be superior for inverting smooth, cold degradations.
This sampler has important mathematical properties that enable it to recover high quality results. Specifically, for a class of linear degradation operations, can be shown to produce exact recon- struction (i.e. Fg Dx s)) even when the restoration operator R fails to perfectly invert D. We discuss this in the following section.
3.3
PROPERTIES OF TACOS
is clear from inspection that both Algo- rithms and 2 perfectly reconstruct the it- erate Ig ((r,,)) for all 8 < L if the restoration operator is perfect inverse for the degradation operator. In this section, we analyze the stability of these algorithms to errors in the restoration operator.
Figure 2: Comparison of sampling methods for un- conditional generation using cold diffusion on the CelebA dataset. Iterations 2, 4, 8, 16, 32, 64, 128, 192, and 256 are presented. Top: Algorithm | pro- duces compounding artifacts and fails to generate new image. Bottom: TACoS succeeds in sampling a high quality image without noise.
For small values of x and s, TACoS as described in 2 is tolerant of error in the restoration operator RoTo see why, consider model problem with a linear degradation function of the form (xx, 8) Rs x + e for constant vector We chose this ansatz because the Taylor expansion of any smooth degradation D(x, s) around r = zg, = 0 has the form D(x 3) R x-+s- (e)sse(()++OO0)) +OT where HOT denotes higher order terms. Note, however, the analysis below requires to be constant that does not depend on x. The constant/zeroth-ordd term in this Taylor expansion is zero because we assumed above that the degradation operator satisfies D(x, 0) x.
For a degradation D(x, 8) and any restoration operator R, the term F.-1 in TACoS becomes
F, DD(R(x, ((,((x,))) s), s) + D D(R(z,,))
(x g. s) 8) + ((((2,))),) 1) ( D
= x0 + e R(x,, 5) - s e R(x,, 8) + (s 1) = 2o + (s 1) = D(xo,
By induction, we see that the algorithm produces the value v. = Da s) for all < t, regardless of the choice of In other words, for any choice of R. the iteration behaves the same as it would when R is perfect inverse for the degradation D.
By contrast, Algorithm D does not enjoy this behavior even for small values of s. In fact, when R is not a perfect inverse for D, xi is not a fixed point of the update rule in Algorithm D because To * D(R(x, 0), 0) = R(x, 0) and hence compounds errors. If R does not perfectly invert D we should expect Algorithm I to incur errors, even for small values of Meanwhile, for small values of 8, the behavior of approaches its first-order Taylor expansion and Algorithm 2 becomes immune to errors in R. (euune]]dmoonntraaal
the stability of TACoS described in Algorithm 2 vs Algorithm T for a deblurring model. Note that our analysis is not meant to be a complete convergence theory, rather to highlight a desirable theoretical property of our method that naive sampler lacks.
4 GENERALIZED DIFFUSIONS WITH VARIOUS TRANSFORMATIONS
In this section, take the first step towards cold diffusion by reversing different degradations and hence performing conditional generation. We will extend our methods to perform unconditional (i.e. from scratch) generation in Section 5 We emprically evaluate generalized diffusion models trained on different degradations with TACoS proposed in Algorithm? 2 We perform experiments on the vi- sion tasks of deblurring, inpainting, super-resolution, and the unconventional task of synthetic snow removal. We perform our experiments on MNIST (LeCun et al. 1998), CIFAR-10 (Krizhevsky) 2009) and CelebA (Liu et al. 2015). each of these tasks, we gradually remove the information from the clean image, creating a sequence of images such that D(ro, t) retains less information than 1 1). For these different tasks, we present both qualitative and quantitative results on a held-out testing dataset and demonstrate the importance of the sampling technique described in Algorithm 2 For all quantitative results in this section, the Frechet inception distance (FID) scores (Heusel et al. 2017) for degraded and reconstructed images measured with respect to the testing data. Additional information about the quantitative results, convergence criteria, hyperparameters, and architecture of the models presented below can be found in the appendix.
4

---

Under review as a conference paper at ICLR 2023
4.1 DEBLURRING
We consider generalized diffusion based on Gaussian blur operation (as opposed to Gaussian noise) in which an image at step has more blur than - The forward process given the Gaussian kernels {G,} and the image 14-1 at step / 1 can thus be written as
It C. It-1 G. 11 3r G. xn D(xo ()
where * denotes the convolution operator, which blurs an image using a kernel.
We train deblurring model by minimizing the loss equation 1 and then use TACoS to invert this blurred diffusion process for which we trained DNN to predict the clean image &0. Qualitative results shown in Figure quantitative results in Table Qualitatively, created using the sampling process are sharper and in some cases completely different as compared to the direct reconstruction of the clean image. Quantitatively metrics such as RMSE and PSNR get worse when we use the sampling process, but on the other hand FID with respect to held-out test data improves. The qualitative improvements and decrease in FID show the benefits of the generalized sampling routine, which brings the learned distribution closer to the true data manifold.
that images
see that the reconstruction
In the case of blur operator, the sampling routine can be thought of adding frequencies at each step- This is because the sampling routine involves the term D t) ( 1) which in the case of blur becomes G3 * Zg - G * This results in difference of Gaussians, which is band pass filter and contains frequencies that were removed at step t. Thus, the sampling process, we sequentially add the frequencies that were removed during the degradation process.
Degraded
Direct
TACoS
Original
9
2 3
4 3 2
922
Figure 3: Deblurring models trained on the MNIST, CIFAR-10 10, and CelebA datasets. Left right: degraded inputs Drro, T'), direct reconstruction R(D( T)), sampled reconstruction with TACoS and original image.
described in Algorithm 2
Table 1: Quantitative metrics for quality of image reconstruction using deblurring models.
Degraded SSIM
Sampled SSIM
Direct SSIM
Dataset
FID
RMSE
FID
RMSE
FID
RMSE
MNIST
438.59
0.287
0.287
4.69
0.718
0.154
5.10
0.757
0.142
CIFAR-10
298.60
0.315
0.136
80.08
0.773
0.075
83.69
0.775
0.071
CelebA
382.81
0.254
0.193
26.14
0.568
0.093
36.37
0.607
0.083
4.2 INPAINTING
We define schedule of transforms that progressively grays-out pixels from the input image. We remove pixels using Gaussian mask as follows: For input images of size n X n we start with a 2D Gaussian curve of variance 8, discretized into an array. We normalize so the peak of the curve has value 1, and subtract the result from so the center of the mask as value We randomize the location of the Gaussian mask for MNIST and CIFAR- 10, but keep it centered for CelebA. We denote the final mask by Zz.
Input images Zo are iteratively masked for T' steps via multiplication with sequence of masks {2,,) with increasing B. We can control the amount of information removed at each step by tuning the 3; parameter. In the language of Section ( t) = Zo I[ Zg,, where the operator denotes entry-wise multiplication.
5

---

Under review as a conference paper at ICLR 2023
Figure 4 presents results test images and compares the output of the inpainting model to the original image. The reconstructed images display reconstructed features qualitatively consistent with the context provided by the unperturbed regions of the image. We quantitatively assess the effectiveness of the inpainting models on each of the datasets by comparing distributional similarity metries before and after the reconstruction. Our results summarized in Table 2 scores here are computed with respect to the held-out validation set.
Note, the FII)
Degraded
Direct
TACoS
Original
3
6
3 6 O
3
6
Figure 4: Inpainting models trained the MNIST, CIFAR-10 10, and CelebA datasets. Left to right:
Degraded inputs D T) direct reconstruction R(D T)). sampled reconstruction with TACoS and original image.
described in Algorithm
2
Table Quantitative metrics for quality image reconstruction using inpainting models.
Degraded SSIM
Sampled SSIM
Direct SSIM
Dataset
FID
RMSE
FID
RMSE
FID
RMSE
MNIST
108.48
0.490
0.262
1.61
0.941
0.068
2.24
0.948
0,060
CIFAR-10
40.83
0.615
0.143
8.92
0.859
0.068
9.97
0.869
0.063
CelebA
127.85
0.663
0.155
5.73
0.917
0.043
7.74
0.922
0.039
4.3
SUPER-RESOLUTEER
For this task, the degradation operator downsamples the image by a factor of two in each direction. The final resolution of æт 4x4 for MNIST and CIFAR-10 and 2x2 in the case of Celeb-A. After each down-sampling, the lower-resolution image is resized to the original image size, using nearest- neighbor interpolation. More details are available in Appendix
Figure presents example testing data inputs for all datasets and compares the output of the super- resolution model to the original image. Though the reconstructed images are not perfect for the more challenging datasets, the reconstructed features are qualitatively consistent with the context provided by the low resolution image.
the distributional similarity metrics between degradeddrecoontruuted images and test samples.
Degraded
Direct
TACoS
Original
2
8
2
8
2 281 1
Figure 5: Superresolution models trained on the MNIST, CIFAR- 10, and CelebA datasets. Left to right: degraded inputs ( T) direct reconstruction (xD, T)), sampled reconstruction with TACoS described in Algorithm and original image.
Table 3: Quantitative metrics for quality of image reconstruction using super-resolution models.
Degraded SSIM
Sampled SSTM
Direct
Dataset
RMSE
FID
RMSE
FID
SSIM
RMSE
MNIST
368.56
0.178
0.231
4.33
0.820
0.115
4.05
0.823
0.114
CIFAR-10
358.99
0.279
0.146
152.76
0.411
0.155
169.94
0.420
0.152
CelebA
349.85
0.335
0,225
96.92
0.381
0.201
112.84
0.400
0.196

---

Under review as a conference paper at ICLR 2023
4.4 SNOWIFICATION
Apart from traditional degradations, we additionally provide results for the task of synthetic snow re- moval using the offical implementation of the snowification transform from ImageNet-C (Hendrycks) & Dietterich] 2019). The purpose of this experiment is to demonstrate that generalized diffusion can succeed even with exotic transforms that lack the scale-space and compositional properties of blur operators. Similar to other tasks, we degrade the images by adding snow, such that the level of snow increases with step We provide more implementation details in Appendix.
We illustrate our desnowification results in Figure 6 We present testing examples, as well as their snowified images, from all the datasets, and compare the desnowified results with the original im- ages. The desnowified images feature near-perfect reconstruction results for CIFAR-10 examples with lighter snow, and exhibit visually distinctive restoration for Celeb-A examples with heavy snow. We provide quantitative results in Table 4
Degraded
Direct
TACoS
Original
Figure 6: Desnovification models trained on the CIFAR-10, 10, and CelebA datasets. Left to right: degraded inputs D T). direct reconstruction (xo, T)), sampled reconstruction with TACoS and original image.
described in Algorithm 2
Table 4: Quantitative metrics for quality of image reconstruction using desno models... models.
Degraded Image SSIM
Reconstruction SSIM
Dataset
FID
RMSE
FID
RMSE
CIFAR- 10 CelebA
125.63 398.31
0.419 0.338
0.327
31.10
0.074
0.838
0,283
27.09
0.033
0.907
COLD GENERATION
Diffusion models can successfully learn the underlying distribution of training data, and thus gener- ate diverse, high quality images (Song et al. 2021a| Dhariwal & Nichol 2021; Jolicoeur- Martttnaaa al. 202T Ho et al. 2022). We will first discuss deterministic generation using Gaussian noise and then discuss in detail unconditional generation using deblurring. Finally, we provide a proof of concept that the TACoS described in Algorithm 2 can be extended to other degradations.
5.1 GENERATION USING DETERMINISTIC NOISE DEGRADATION
Here we discuss image generation using a noise-based degradation presented in our notation from Section 33 which will later prove is equivalent to DDIM (Song et al. 2021a). We use the follow- ing degradation operator: D(x,1 !) = Varzz + VI 02222
D is an interpolation between the data point and a sampled noise pattern = E (0 1). During training, D is applied once and thus z is sampled once for every image in every batch. However, sampling involves iterative applications of the degradation operator D, which poses the question of how to pick z for the sequence of degradations D applied in a single image generation.
There are three possible choices for z. The first would be resample for each application of D, but this would make the sampling process nondeterministic for a fixed starting point. Another option is to sample a noise pattern & once for each separate image generation and reuse it in each application of D. In Table we refer to this approach as Fixed Noise. Finally, one can calculate the noise vector z to be used step of reconstruction by using the formula
/a, R(x,,))
2 t)
O
This method denoted Estimated Noise in Table 5 turns out to be equivalent to the deterministic sampling proposed in /Song al. (021a). We discuss this equivalence in detail in Appendix A.6
7

---

Under review as a conference paper at ICLR 2023
5.2 IMAGE GENERATION USING BLUR
The forward diffusion process in moise-based diffusion models has the advantage that the degraded image distribution at the final step is simply an isotropic Gaussian. One can therefore perform (unconditional) generation by first drawing a sample from the isotropic Gaussian, and sequentially denoising it with backward diffusion.
When using blur as degradation, the fully degraded images do not form nice closed-form distri- bution that we can sample from. They do, however, form a simple enough distribution that can be modeled with simple methods. Note that every image To degenerates to an Ty that is constant (i.e., every pixel is the same color) for large T. Furthermore, the constant value is exactly the channel- wise mean of the RGB image Zp, and can be represented with 3-vector. This 3-dimensional distribution is easily represented using a Gaussian mixture model (GMM). This GMM can be sam- pled to produce the random pixel values of severely blurred image, which can be deblurred using cold diffusion to create new image.
Our generative model uses a blurring schedule where we progressively blur each image with Gaus- sian kernel of size 27 x 27 over 300 steps. The standard deviation of the kernel starts at I and increases exponentially at the rate of 0.01. We then fit simple GMM with one component to the distribution of channel-wise means. To generate an image from scratch, we sample the channel-wise mean from the GMM, expand the 3D vector into 128 x 128 image with three channels, and then apply TACoS.
Empirically, the presented pipeline generates images with high fidelity but low diversity, as reflected quantitatively by comparing the perfect symmetry column with results from hot diffusion in Table We attribute this to the perfect correlation between pixels of aT sampled from the channel-wise mean Gaussian mixture model. To break the symmetry between pixels, we add small amount of Gaussian noise (of standard deviation 0.002) to each sampled æT. As shown in Table the simple trick drastically improves the quality of generated images. We also present the qualitative results for cold diffusion using blur transformation in Figure 7 and further discuss the necessity of TACoS proposed in Algorithm 2 for generation in Appendix A.7
Table 5: FID scores for CelebA and AFHQ datasets using hot (noise) and cold diffusion (blur transformation). Breaking the symmetry within pixels of the same channel further improves FID.
Hot Diffusion
Cold Diffusion
Dataset
Fixed Noise
Estimated Noise
Perfect symmetry
Broken symmetry
CelebA
59.91 25.62
23.11
97.00
49.45
AFHQ
20.59
93.05
54.68
Figure 7: Examples of generated samples from 128 x 128 CelebA and AFHQ datasets using cold diffusion with blur transformation
5.3 GENERATION USING OTHER TRANSFORMATIONS
In this section, we provide proof of concept that generation can be extended to other transforma- tions. Specifically, show preliminary results inpainting, super-resolution, and animorphosis.
8

---

Under review as a conference paper at ICLR 2023
Inspired by the simplicity of the degraded image distribution for the blurring routine presented in the previous section, we use degradation routines with predictable final distributions here as well.
To use the Gaussian mask transformation for generation, modify the masking routine so the final degraded image is completely devoid of information. One might think a natural option is to send all of the images to completely black image 27, but this would not allow for any diversity in generation. To get around this maximally non-injective property, we instead make the mask turn all pixels random, solid color. This still removes all of the information from the image, but allows us to recover different samples from the learned distribution via Algorithm by starting off with different color images. More formally, a Gaussian mask G2 = II 29, is created in a similar way as discussed in the Section 4.2 but instead of multiplying it directly to the image zo, we create Te as G2 Zo + (1 G+) c, where is an image of randomly sampled color.
For super-resolution the routine down-sample to a resolution These degraded images can be represented as one-dimensional vectors, and their distribution is modeled using one Gaussian distribution. Using the same methods described for generation using blurring described above, we sample from this Gaussian-fin fiitted distribution of the lower-dimensional degraded image space and pass this sampled point through the generation process trained on super- resolution data create one output.
x 2, or 4 values in each channel.
Additionally to show one can invert nearly any transformation, we include a new transformation deemed animorphosis, where iteratively transform human face from CelebA to an animal face from AFHQ. Though we chose CelebA and AFHQ for our experimentation, in principle such interpolation can be done for any two initial data distributions.
More formally, given an image x and a random image 2 sampled from the AFHQ manifold, z, can be written but instead of adding noise we are adding a progressively higher weighted AFHQ image. In order sample from the learned distribution, we sample random image of an animal and use TACoS.
Vayzt + vI 0g2. Note this is essentially the same as the noising procedure,
We present results for the CelebA dataset, and hence the quantitative results in terms of FID scores for inpainting, super-resolution and animorphosis 90.14, 92.91 and 48.51 respectively. We fur- ther show some qualitative samples in Figure 8 and in Figure !
Figure 8: Preliminary demonstration of the generative abilities other cold diffusins on the 128 x 128 CelebA dataset. The top row is with animorphosis models, the middle row is with inpainting models, and the bottom row exhibits super-resolution models.
CONCLUSION
Existing diffusion models rely Gaussian noise for both forward and reverse processes. In this work, we find that the random noise can be removed entirely from the diffusion model framework, and replaced with arbitrary transforms. In doing so, our generalization of diffusion models and their sampling procedures allows us to restore images afflicted by deterministic degradations such as blur, inpainting and downsampling. This framework paves the way for a more diverse landscape of diffusion models beyond the Gaussian noise paradigm. The different properties of these diffusions may prove useful for range of applications, including image generation and beyond.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provided our full code base supplementary material, which is a modified version of the tra- ditional diffusion database found at facilitate the reproducibility of our results, have included detailed hyperparameters for training each of our cold diffusion models in Appendices A.1-A.5. Due to space constraints in the main body, we opted to present relatively small number of qualitative results. Many more examples of both conditionally and unconditionally generated images can be found in the Appendix.
To
REFERENCES
Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.0111 2018.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: (801.00000 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint arXiv:16/2.022166, 2016.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. volume 34, 2021.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 32, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23, 2022.
Alexia Jolicoeu--Martneenr Martineau, Rémi Piché-Taillefe Ioannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. International Conference Learning Representations, 2021.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the
prior implicit in denoiser. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint w:Xiv2206.003664 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. International Conference on Computer Vision Workshops, 2021.
Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201.1 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffu- sion models. Advances Neural Information Processing Systems, 34, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEFE, 86(1 1):2278-2333 1998.
10

---

Under review as a conference paper at ICLR 2023
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Christopher A. Metzler, Ali Mousavi, and Richard G. Baraniuk. Learned D-AMP: principled neural network based compressive image recovery. Advances in Neural Information Processing Systems, 30, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Severi Rissanen, Markus Heinonen, and Amo Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint 2ttT::22001139997 2022.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). arXiv preprint arXiv: 1611.0286,, 2016.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.0760 2021.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, volume 37 of JMLR Workshop and Conference Proceedings, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Interna- tional Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32. 2019.
Yang Song, Jascha Dohh-Diccstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representutions, 2021b.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint 2rXiv:21 2021.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DEBLURRING
For the deblurring experiments. we train the models on different datasets for 700,000 gradient steps. We use the Adam (Kingma & Ba, 2014) optimizer with learning rate x 10-5, The training was done on the batch size 32, and we accumulate the gradients every steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995 which is updated after every 10 gradient steps.
For the MNIST dataset, we blur recursively 40 times, with discrete Gaussian kernel of size 11x11 and a standard deviation 7. In the case of CIFAR-10, 10, we recursively blur with a Gaussian kernel of fixed size 11x11, but at each step the standard deviation of the Gaussian kernel is given by 0.01 + + 0.35. The blur routine for CelebA dataset involves blurring images with a Gaussian kernel of 15x15 and the standard deviation of the Gaussian kernel grows exponentially with time the rate of 0.01.
Figure shows an additional nine images for each of MNIST, CIFAR-10 10 and CelebA. Figures 19 and 20 show the iterative sampling process using a deblurring model for ten example images from each dataset. We further show 400 random images demonstrate the qualitative results in the Figure
21
Degraded
Direct
TACoS
Original
5
4
9
2
4
4
2 9
4
8
5
9
3
5
7
7
8
7
9
8
9
9 5
8
7 8
9
Figure 9: Additional examples from deblurring models trained the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs ( T") direct reconstruction R(D(zo, T)), sampled reconstruction with TACoS described in Algorithm 2 and original image.
A.2 INPAINTING
For the inpainting transformation, models were trained on different datasets with 60,000 gradient steps. The models were trained using Adam (Kingma Bal (014) optimizer with learning rate x 10-5, We use batch size 64, and the gradients are accumulated after every 2 steps. The final model is an Exponential Moving Average of the trained model with decay rate 0.995. This EMA model updated after every 10 gradient steps. For all inpainting experiments Gaussian mask and T = 50 with 31 = and 3 = 3 + 0.1.
randomized
To avoid potential leakage of information due to floating point computation of the Gaussian mask, we discretize the masked image before passing through the inpainting model. This was done by rounding all pixel values to the eight most significant digits.
12

---

Under review as a conference paper at ICLR 2023
Figure 11 shows nine additional inpainting examples on each of the MNIST, CIFAR-10, and CelebA datasets. Figure[[[demonnttrate an example of the iterative sampling process of an inpainting model for one image in each dataset.
A.3 RESERRROSOOTIONN
We train the super-resolution model per Section 3.1 for 700,000 iterations, We use the Adam (Kingma & Ba, 2014) optimizer with learning rate 2 x 10 10- 5. The batch size is 32, and ac- cumulate the gradients every 2 steps. Our final model is an Exponential Moving Average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps.
The number of time-steps depends on the size of the input image and the final image. For MNIST and for CIFARIO, the number of time steps is 3, as it takes three steps halving the resolution to reduce the initial image down to 4 x 4. For CelebA, the number of time steps is 6 to reduce the initial image down to 2 x 2. For CIFARIO, we apply random crop and random horizontal flip for regularization.
Figure shows an additional nine super-resolution examples on each of the MNIST, CIFAR-10 and CelebA datasets. Figure 12 shows one example of the progressive increase in resolution achieved with the sampling process using a super-resolution model for each dataset.
A.4 COLORIZATION
Here provide results for the additional task of colorization. Starting with the original RGB- image we realize colorization by iteratively desaturating for T steps until the final image fully gray-scale image. We use a series of three-channel 1 1 convolution filters z(a) {=)(a), = (a), (0))) (a)) with the form
z' (a) = a (1 (1-aa))100)))
2²) = a (211) () (0
2 (a) a (113) +1 -1--a) (1-a) (001)
and obtain D(x, t) that gray image obtained when øт
z(az) # via schedule defined as ai,..., for each respective step. Notice
z(1) z0-
We can tune the ratio o, to control the amount of information removed in each step. For our experi- ment, schedule the ratio such that for every we have
Zt z((a) ..... z(ai) xo z( 30.
This schedule ensures that color information lost between steps is smaller in earlier stage of the diffusion and becomes larger as increases.
We train the models on different datasets for 700,000 gradient steps. We use Adam Kingmma & Ba, 2014] optimizer with learning rate 2 10-5, We use batch size 32, and we accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 we use T 50 and for CelebA we use 7 20.
8
8
8
8
8
8
Figure 10: Progressive inpainting of selected masked MNIST, CIFAR-1O, 10, and CelebA images.
13

---

Under review as a conference paper at ICLR 2023
Degraded
Direct
TACoS
Original
0
1
2
D
1 2
D
1 2
3
4
5
3
4
5
3 4 5 8
6
7
8
6
7
8
6
7
S
S
Figure 11: Additional examples from inpainting models trained the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs (xx,, T) direct reconstruction R(D(zo,))
sampled reconstruction with TACoS described in Algorithm and original image.
3 333
Figure 12: Progressive upsampling of selected downsampled MNIST, CIFAR-10, and CelebA im- ages. The original image is the left for each of these progressive upsamplings.
We illustrate our recolorization results Figure 14 We present testing examples, as well as their grey scale images, from all the datasets, and compare the recolorization results with the original images. The recolored images feature correct color separation between different regions, and feature various and yet semantically correct colorization of objects. Our sampling technique still yields minor differences in comparison to the direct reconstruction, although the change is not visually apparent. We attribute this to the shape restriction of colorization task, as human perception is rather insensitive to minor color change. We also provide quantitative measurement for the effectiveness of our recolorization results in terms of different similarity metries, and summarize the results in Table 6
Table Quantitative metrics for quality of image reconstruction using recolorization models for all
three channel datasets.
Degraded Image SSIM
Reconstruction SSIM
Dataset
FID
RMSE
RMSE
CIFAR-10
97.39
0.937
0.078 0.089
45.74
0.942
0.069
CelebA
41.20
0.942
17.50
0.973
0.042
14

---

Under review as a conference paper at ICLR 2023
Degraded
Direct
TACoS
Original
L
9 9
L 9 199 9
L 7 9
7
4
5
7 748
5
7 4
6
3 4
2
3 2 4
3 4
Figure 13: Additional examples from super-resolution models trained the MNIST, CIFAR-10, and CelebA datasets. Left to right: degraded inputs D( T). direct reconstruction R(D(zo, T)), sampled reconstruction with TACoS described in Algorithm and original image.
Degraded
Direct
TACoS
Original
Figure 14: Recolorization models trained on the CIFAR-10 and CelebA datasets. Left to right: degraded inputs D( T direct reconstruction R(D(xo, T)), sampled reconstruction with TACoS
described in Algorithm 2
and original image.
A.5
IMAGE SNOW
Here we provide results for the additional task of snowification, which is a direct adaptation of the offical implementation of ImageNet-C snowification process (Hendrycks & Dietterich 2019). To determine the snow pattern of given image Zo E RC Rxx we first construct seed matrix SA E TR. NW where each entry sampled from a Gaussian distribution N(4, The upper-left corner of SA is then zoomed into another matrix SB E RH* W with spline interpolation. Next, we create new matrix Se by filtering each value of SB with given threshold e as
[0,
S S ei SB M > c1
Sc [bb
(S
and clip each entry of Se into the range [0, 1]. We then convolve Se using a motion blur kernel with standard deviation co create the snow pattern S and its up-side-down rotation S'. The direction of
15

---

Under review as a conference paper at ICLR 2023
the motional blur kernel is randomly chosen as either vertical or horizontal. The final snow image is created by again clipping each value of zu + S + into the range [0, 1]. For simplicity, we abstract the process as function h(r,, Sa, co, C1).
Degraded
Direct
TACoS
Original
Figure 15: Additional examples from Desnowification models trained on the CIFAR- 10 and CelebA datasets. Left to right: degraded inputs D ((zo, T) direct reconstruction R(D(zo. T)), sampled reconstruction with TACoS described in Algorithm and original image.
To create a series of T images with increasing snowification, we linearly interpolate co and ci between |s] cati and [ctl.n cerl) respectively, to create co and (t), = ,. Then for each æo. seed matrix Sz is sampled, the motion blur direction is randomized, and we construct each related x, by 2t h( Sz. (t), (, (t)). Visually, (t dictates the severity of the snow, while (t) determines how "windy" the snowified image seems.
T.
For both CIFAR-I0 and Celeb-A, we use the same Gaussian distribution with parameters μ 0.55 and a = 0.3 to generate the seed matrix. For CIFAR-10, we choose eStutt 1.15, end 0.7, ettant = 0.05 and ct = 16, which generates a visually lighter snow. For Celeb-A, we choose c 1.15, c0 0.55, caart 0.05 and cT 20, which generates visually heavier snow.
We train the models on different datasets for 700,000 gradient steps. We use Adam (Kingma & Ba 2014) optimizer with learning rate 2 x 10-5, We use batch size 32, and we accumulate the gradients every 2 steps. Our final model is an exponential moving average of the trained model with decay rate 0.995. We update the EMA model every 10 gradient steps. For CIFAR-10 we use T = 200 and for CelebA we use T = 200. We note that the seed matrix is resampled for each individual training batch, and hence the snow pattern varies across the training stage.
A.6 GENERATION USING NOISE: FURTHER DETAILS
Here we show the equivalence between the sampling method proposed in Algorithm Sanithmmman and the deter- ministic sampling DDIM (Song et al. 2021a). Given the image x, at step t, we have the restored clean image Zo from the diffusion model. Hence given the estimated zi and we can estimate the noise 2 f) (or ě) as
Ft Vayi0
2(x,, ()
VI at
Thus, the D( 1) and D(ño: 1 1) can be written as
D(Fo, Vouto + vI a1²,
D Vac-ido + vI - 11-12
using which the sampling process in Algorithm 2 to estimate T;-1 can be written as,
16

---

Under review as a conference paper at ICLR 2023
-t-1 = 24 - D(E0, 1) + D( 1)
(Var + -I--(t) + (Vac-1d0 + v1 ac-12)
=Vo--100 v1- VI-a-11
(1)
which same as the sampling method as described in (Song et al. 2021a). The only difference from the original (Song et al., 2021a) is the order for estimating 4h and ž. The original (Song al. 202 paper estimated 2 first and then used this to predict clean image vh, while we first predict the clean image zig and then estimate the noise 2.
A.7 GENERATION USING BLUR TRANSFORMATION: FURTHER DETAILS
Figure 16: Examples of generated samples from 128x 128 CelebA and AFHQ datasets using Method 2 with perfect symmetry.
The Figure 16 shows the generation without breaking any symmetry within each channel are quite promising as well.
Necessity of Algorithm 2: In the case of unconditional generation, observe marked superiority in quality of the sampled reconstruction using Algorithm 2 over any other method considered. For example, in the broken symmetry case, the FID of the directly reconstructed images is 257.69 for CelebA and 214.24 for AFHQ, which are far worse than the scores of 49.45 and 54.68 from Table In Figure 17 we also give qualitative comparison of this difference. We can also clearly see from Figure 18 that Algorithm|] [ the method used in [Son et al. (2021b) and Ho et al. (2020). completely fails to produce an image close the target data distribution.
A.8 ALGORITHM D IS SAME AS DDIM/DDPM SAMPLING
The sampling method proposed in Song et al. 2222aa in it's equation 12 is given
-t-1 a "predicted mo" + v1 0:-1 - cxr) + azf
where co(x;) is the noise predicted by the diffusion model given and f. The term "predicted an" or do can be computed directly given o; and eø (x,) as
It
(cca(
0r
17

---

Under review as a conference paper at ICLR 2023
Hence using 2 instead of co (24) and dy to indicate predicted clean image, have
It-1 V 10-1- 40 + V1 04-1 - of? + dcct
Thus, the sampling step interpreted follows: At each step t, we start with noisy image It and use the diffusion model to estimate the clean image di and the noise that was added to this clean image do get the noisy image Σς- In order move to lesser noisy image 24-1, one "adds back" lesser noise to the the "predicted clean image" zo. Now one can add back noise in 2 ways, either the noise which uncorrelated noise et. Infact both of these noise can be added using as the hyperparameter that weighs the amount of each noise added. This dy placed in the equation such that for any choice of or the standard deviation of noise added back is vI- 0t-1- For the estimated noise and no uncorrelated noise & which is infact the DDIM sampling. While for o, = ///1a//))//11 aa)aaa)/// aa/a:-1 we get the sampling method described in DDPM. Nevertheless, for any choice of the sampling method involves denoising operation which is shown as R(x;,;) 8) in Algorithm and adding back noise shown as Ix-1 D 8 8- 1) in Algorithm
added the clean image zip which is sample new
0, we only add back
[ The only difference between different sampling methods explained in DDPM DDIM is how one degrades the image back.
Figure 17: Comparison of direct reconstruction with sampling using TACoS described in Algorithm for generation with blur transformation and broken symmetry. Left-hand column is the initial cold images generated using the simple Gaussian model. Middle column has images generated in one step (i.e. direct reconstruction). Right-hand column are the images sampled with TACoS described in Algorithm 2
We present results for both CelebA (top) and AFHQ (bottom) with resolution
128 x 128.
18

---

Under review as a conference paper at ICLR 2023
Figure 18: Comparison of Algorithm T (top row) and Algorithm 2 (bottom row) for generation with Method 2 and broken symmetry on 128 x 128 CelebA dataset. We demonstrate that Algorithm fails completely to generate a new image.
19

---

Under review as a conference paper at ICLR 2023
9
10100
5 3 6
1
Figure 19: Progressive deblurring of selected blurred MNIST and CIFAR-10 images.
20

---

Under review as a conference paper at ICLR 2023
Figure 20: Progressive deblurring of selected blurred CelebA images.
21

---

Under review as a conference paper at ICLR 2023
Figure 21: Deblurred Ci images
22


Reviews:
[{"review_id": "gNJ4i4Tn60c", "reviewer": "Reviewer_8WRE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new variant of two related classes of diffusion models, e.g., denoising diffusion and denoising score-matching. Unlike traditional diffusion models that generate data through denoising reverse processes, this work uses deterministic degradations, such as blurring, and trains the corresponding restoration neural network for the restoration reverse processes. Numerical validations are conducted on several image degradation models, such as blurring, inpainting, and super-resolution.", "summary_of_the_review": "Consequently, given the pros and cons on balance, I feel this is a borderline paper, and I vote for borderline accept tentatively.", "strength_and_weaknesses": "Strengths:\n\n1, The idea of using treating the reverse process of a diffusion model as a series of deterministic image restoration rather than denoising is very interesting. \n\n2, The analysis and experiments are extensive and can support the claims.\n\n3,The paper is well written and easy to follow.\n\nWeakness: \n\n1, Numerical comparison with existing denoising-based diffusion models is missing.  The fact that this paper edges out the existing result is okay, as this paper provides a totally different scheme for diffusion models. However, the quantitative results of existing diffusion models should be adequately reported in the table, and additional qualitative comparisons are necessary. For example, the authors can compare their models’ generation quality with DDIM[Song.etal2021] where  the sampling step is also deterministic.\n\n2, Similarly, the comparisons with other baseline methods ,e.g., generative models , are not presented in this paper.\n \n3, Only low-resolution image datasets are considered in this work (e.g., 32, 64,128).  As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed cold diffusion compares to DDPM/DDIM on higher-resolution dataset, such as 256x256.\n\n4, In addition to the insufficient comparisons to the baseline, a clear motivation to use different degradations rather than using AWGN seems to be missing in its current state.  Although this works aims to proof of concept, it is unclear the benefits to do so. Will this change result in better performance, or a more efficient reverse process ? The authors don’t show clear cut about this.\n\n5, There is no unified routines to design the latent space. Unlike traditional diffusion models sampled from a simple distribution, the approach loses its appeal when considering sophisticated design of the latent space. For example, the number of steps should depend on the image dimensions when using SR as degradation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aaX4K6SZ1u", "reviewer": "Reviewer_BSRf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method for deterministic generation of images called TACoS. It allows to invert arbitrary image transformations, with a few investigated experimentally: deterministic gaussian noise, blurs, animorphosis, masking, pixellating and snowing. The paper first evaluates it on the task of reconstruction based on a latent produced from a real image, and then on the task of unconditional generation by heuristically suggesting a prior for generating a latent.", "summary_of_the_review": "This is an interesting empirical paper contributing to our understanding that per-step sampling might not be such a crucial component of diffusion generative models that was previously thought. I am learning towards acceptance.", "strength_and_weaknesses": "\\+ The topic is certainly interesting and advances our understanding of generative models.\n\n\\+ Paper is clearly written and easy to read.\n\n\\- Empirical results are not great in a sense that we can't generate better images with this method.\n\n\\- One downside of the method is that it leaves the question of learning the prior to ad-hoc approximations, or severely limits the number of generated images (as in case of animorphosis, when we can't generate more images with a deterministic procedure than the number of animal images in the training set).\n\n\\- There are some questions about the motivation for the specific functional form of TACoS which need clarification (see below).\n\nQuestions/minor comments:\n* worth citing https://arxiv.org/abs/2206.00364, they also investigate deterministic sampling\n* \"Diffusion has been understood as a random walk around the image density function using Langevin dynamics (Sohl- Dickstein et al., 2015\" - not sure Sohl-Dickstein is relevant, it's variational\n* \"denotes a norm, which we take to be l1 in our experiments\" - does it work with l2, which is more commonly used?\n* [motivation] \"While this ansatz may seem rather restrictive, note that the Taylor expansion of any smooth degradation D(x,s) around x = x0,s = 0 has the form D(x,s) ≈ x + s · e + HOT\" - the crucial detail though is that e = e(x) because it is the gradient of D and D depends on x. Would the following derivation still hold in that case? If not, you should state it clearly, because the way the current text is written suggests that it holds for all differentiable corruptions.\n* \"x0 +(s−1)\" - e is missing?\n* Figure 2 - could you give more details on this experiment? For example, is it a reconstruction (starting with a latent generated based on a real image) or generation task (starting from a latent sampled from a prior)? How many steps were taken between the shown images? Currently, the second image in the top row looks totally blank, did the predictor already produce a blank image?\n* \"TACoS proposed in Algorithm 2 can be applied in this case by fixing the noise z used in the degradation operator D(x, s)\" - I assume z is a constant used for generating all images, not resampled for every image?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KurFIle77o", "reviewer": "Reviewer_h4aq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a general framework for diffusion models with different types of deterministic degradation process. The framework minimizes a training objective similar to the original diffusion models, which is about minimizing l1 norm between the original clean sample and the predicted clean sample by a restoration network. A new sampler is proposed and some analysis conducted to show that this sampler is preferred than the naive noise-denoise sampler if the sampling process starts from an infinitesimal noise level. Experimental results show that the proposed framework can produce reasonable samples for generation and several image restoration tasks, with several deterministic degradation processes. ", "summary_of_the_review": "Overall speaking, I think this paper serves as a good proof of concept that it is indeed possible to generalize normal Gaussian diffusion to highly non-trivial degradation processes by designing the reversal process carefully. However, the results unfortunately are not quite convincing. The theoretical analysis also doesn't well support the claim that the new sampler is better. ", "strength_and_weaknesses": "### Strengths:\n- It is interesting to see that by some simple tweaks of the training objective and the sampler can work for several very different degradation  processes. The results is suggestive that more principled stepwise reversal of arbitrary lossy transformations could work really well. \n- The paper is well-written and easy to follow. \n- The paper conducts exhaustive experiments to valid the proposed approach across multiple tasks and multiple degradation processes.\n\n### Weaknesses:\n- I'm confused by the analysis of TACoS: The assumption that the degradation operator is linear only holds with an infinitesimal $s$. Therefore, the conclusion that TSCoS is robust to error only holds if sampling starts from an infinitesimal $t$. However, the empirical comparisons of the two samplers are shown with very large $s$ which cannot be explained by the analysis. In practice, the sampling around high-nois levels also likely contributes a lot to the final sample quality.\n- In Appendix, it is shown that the proposed sampler is the same as DDIM with frozen $z$ in training and $\\hat{z}$ in inference. First, I feel it is kind of weird way of choosing the noise by using $\\hat{z}$ in sampling. Second, the FID is much worse than DDIM from a normal Gaussian diffusion. Does it indicate that switching from stochastic to deterministic degradation process can be a very important factor for degradation in performance? \n- Why using $\\ell1$ norm for the training objective instead of $\\ell2$?\n- The results are not very convincing: By checking FID scores, there's a huge gap between the proposed deterministic processes and the normal Gaussian diffusion. The results are also worse than other recent works on deterministic diffusion processes, e.g. [1]. Also [1] is a missing reference. \n- Compared with Direct restoration baseline, TACoS seems to introduce more artifacts in some scenarios (e.g., Fig. 6 for celebA). \n\n1. Rissanen, Severi, Markus Heinonen, and Arno Solin. \"Generative modelling with inverse heat dissipation.\" (2022).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "20s5FIf6br", "reviewer": "Reviewer_wT7w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to use forward processes other than the Gaussian noise. This work learns using the x0 prediction formulation. This author proposes TACoS to build the reverse process. Experiments provide various types of forward processes.", "summary_of_the_review": "See \"Strength And Weaknesses\"", "strength_and_weaknesses": "Strength:\n\nThis paper demonstrates that it is possible to use forward process other than the Gaussian noise.\n\n\n\nWeakness:\n\n1. This work lacks a comparison to the diffusion model that adds Gaussian noise. This work provides some generated samples, e.g., Figure 7 and Figure 21, and their sample quality looks worse than the diffusion model that adds Gaussian noise. For example, the diffusion model that adds Gaussian noise can generate sharper images than this work.\n\n2. Since the sample quality is worse than the diffusion model that adds Gaussian noise, the argument \" The success of these fully deterministic models calls into question the community’s understanding of diffusion models..\" is overclaimed. \n\n3. In Section 3.2, the author states \"Rather, diffusion models (Song et al., 2021a; Ho et al., 2020) perform generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm 1.\" However, Algorithm 1 does not correspond to the denoising process of diffusion models (Song et al., 2021a; Ho et al., 2020). Indeed, these methods use a linear combination of $x_0$ and $x_s$ to generate $x_{s-1}$, instead of only using $x_0$ as in Algorithm 1.\n\n4. In Section 3.2, the author states \"We find that the standard sampling approach in Algorithm 1 works well for noise-based diffusion\". Is their any quantative result?\n\n5. Missing experimental details. What is the number of generated samples to calculate FID? Which subset is used as the reference dataset to calculate FID?\n\n6. It seems that TACoS sometimes performs worse than direct reconstruction. For example, in Figure 6, the samples of TACoS have more artifacts than the direct reconstruction. In Table 4, the TACoS quantative result is missing.\n\n7. In Section 3.3, the author uses the first order Taylar approximation for arbitary $s$. However, this approximation only holds for small $s$.\n\n8. In Appendix A.6,  it author set $D(\\hat{x}_0, t) = \\sqrt{\\alpha_t} \\hat{x}_0 + \\sqrt{1-\\alpha_t} \\hat{z}$, which is inconsistent with that in Section 5.1. It would be better to choose another notation for $D$ in Appendix A.6.\n\n9. In Table 5, the author shows that using DDIM sampling method would improve the sample quality compared to TACoS. Would the sample quality be further improved if we use the traditional diffusion model training objective? i.e., using a random $z$ instead of a fixed $z$ during training.\n\n\nSummary:\n\nOverall, the sample quality of the proposed method is not competitive to diffusion model with Gaussian noise. This is perhaps because TACoS is not able to exactly reverse the forward process. This contrasts to the reverse SDE or the ODE formulation in Gaussian diffusion model, which are able to exactly reverse the forward process. Besides, the idea of using different forward process is not new. For example, [1,2] have formally studied diffusion models with other forward processes. So in my opinion this work does not provide a sufficient contribution for a ICLR publication.\n\n[1] Denoising Diffusion Gamma Models\n\n[2] Structured Denoising Diffusion Models in Discrete State-Spaces\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject