Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a general framework for diffusion models with different types of deterministic degradation process. The framework minimizes a training objective similar to the original diffusion models, which is about minimizing l1 norm between the original clean sample and the predicted clean sample by a restoration network. A new sampler is proposed and some analysis conducted to show that this sampler is preferred than the naive noise-denoise sampler if the sampling process starts from an infinitesimal noise level. Experimental results show that the proposed framework can produce reasonable samples for generation and several image restoration tasks, with several deterministic degradation processes. 
strength_and_weaknesses: ### Strengths:
- It is interesting to see that by some simple tweaks of the training objective and the sampler can work for several very different degradation  processes. The results is suggestive that more principled stepwise reversal of arbitrary lossy transformations could work really well. 
- The paper is well-written and easy to follow. 
- The paper conducts exhaustive experiments to valid the proposed approach across multiple tasks and multiple degradation processes.

### Weaknesses:
- I'm confused by the analysis of TACoS: The assumption that the degradation operator is linear only holds with an infinitesimal $s$. Therefore, the conclusion that TSCoS is robust to error only holds if sampling starts from an infinitesimal $t$. However, the empirical comparisons of the two samplers are shown with very large $s$ which cannot be explained by the analysis. In practice, the sampling around high-nois levels also likely contributes a lot to the final sample quality.
- In Appendix, it is shown that the proposed sampler is the same as DDIM with frozen $z$ in training and $\hat{z}$ in inference. First, I feel it is kind of weird way of choosing the noise by using $\hat{z}$ in sampling. Second, the FID is much worse than DDIM from a normal Gaussian diffusion. Does it indicate that switching from stochastic to deterministic degradation process can be a very important factor for degradation in performance? 
- Why using $\ell1$ norm for the training objective instead of $\ell2$?
- The results are not very convincing: By checking FID scores, there's a huge gap between the proposed deterministic processes and the normal Gaussian diffusion. The results are also worse than other recent works on deterministic diffusion processes, e.g. [1]. Also [1] is a missing reference. 
- Compared with Direct restoration baseline, TACoS seems to introduce more artifacts in some scenarios (e.g., Fig. 6 for celebA). 

1. Rissanen, Severi, Markus Heinonen, and Arno Solin. "Generative modelling with inverse heat dissipation." (2022).


clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is clear in written.
- Quality: the theoretical analysis and experimental results can be further improved.
- Novelty: the framework is general and prosed sampler is kind of novel. 
- Reproducibility: they provide enough details to reproduce the main results. 
summary_of_the_review: Overall speaking, I think this paper serves as a good proof of concept that it is indeed possible to generalize normal Gaussian diffusion to highly non-trivial degradation processes by designing the reversal process carefully. However, the results unfortunately are not quite convincing. The theoretical analysis also doesn't well support the claim that the new sampler is better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new variant of two related classes of diffusion models, e.g., denoising diffusion and denoising score-matching. Unlike traditional diffusion models that generate data through denoising reverse processes, this work uses deterministic degradations, such as blurring, and trains the corresponding restoration neural network for the restoration reverse processes. Numerical validations are conducted on several image degradation models, such as blurring, inpainting, and super-resolution.
strength_and_weaknesses: Strengths:

1, The idea of using treating the reverse process of a diffusion model as a series of deterministic image restoration rather than denoising is very interesting. 

2, The analysis and experiments are extensive and can support the claims.

3,The paper is well written and easy to follow.

Weakness: 

1, Numerical comparison with existing denoising-based diffusion models is missing.  The fact that this paper edges out the existing result is okay, as this paper provides a totally different scheme for diffusion models. However, the quantitative results of existing diffusion models should be adequately reported in the table, and additional qualitative comparisons are necessary. For example, the authors can compare their models’ generation quality with DDIM[Song.etal2021] where  the sampling step is also deterministic.

2, Similarly, the comparisons with other baseline methods ,e.g., generative models , are not presented in this paper.
 
3, Only low-resolution image datasets are considered in this work (e.g., 32, 64,128).  As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed cold diffusion compares to DDPM/DDIM on higher-resolution dataset, such as 256x256.

4, In addition to the insufficient comparisons to the baseline, a clear motivation to use different degradations rather than using AWGN seems to be missing in its current state.  Although this works aims to proof of concept, it is unclear the benefits to do so. Will this change result in better performance, or a more efficient reverse process ? The authors don’t show clear cut about this.

5, There is no unified routines to design the latent space. Unlike traditional diffusion models sampled from a simple distribution, the approach loses its appeal when considering sophisticated design of the latent space. For example, the number of steps should depend on the image dimensions when using SR as degradation.

clarity,_quality,_novelty_and_reproducibility: Clarity/Quality : The description of the method is clearly stated, and the paper is easy to follow.

Novelty: The studied problem is interesting and worth devoting to. To this best of the reviewer’s knowledge, this is the first work to show that multiple degradations can be used to design diffusion models, but there are still several remaining issues as above.

Reproducibility: Code is provided in the supplementary, so I think the reproducibility is not the concern here.
summary_of_the_review: Consequently, given the pros and cons on balance, I feel this is a borderline paper, and I vote for borderline accept tentatively.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method for deterministic generation of images called TACoS. It allows to invert arbitrary image transformations, with a few investigated experimentally: deterministic gaussian noise, blurs, animorphosis, masking, pixellating and snowing. The paper first evaluates it on the task of reconstruction based on a latent produced from a real image, and then on the task of unconditional generation by heuristically suggesting a prior for generating a latent.
strength_and_weaknesses: \+ The topic is certainly interesting and advances our understanding of generative models.

\+ Paper is clearly written and easy to read.

\- Empirical results are not great in a sense that we can't generate better images with this method.

\- One downside of the method is that it leaves the question of learning the prior to ad-hoc approximations, or severely limits the number of generated images (as in case of animorphosis, when we can't generate more images with a deterministic procedure than the number of animal images in the training set).

\- There are some questions about the motivation for the specific functional form of TACoS which need clarification (see below).

Questions/minor comments:
* worth citing https://arxiv.org/abs/2206.00364, they also investigate deterministic sampling
* "Diffusion has been understood as a random walk around the image density function using Langevin dynamics (Sohl- Dickstein et al., 2015" - not sure Sohl-Dickstein is relevant, it's variational
* "denotes a norm, which we take to be l1 in our experiments" - does it work with l2, which is more commonly used?
* [motivation] "While this ansatz may seem rather restrictive, note that the Taylor expansion of any smooth degradation D(x,s) around x = x0,s = 0 has the form D(x,s) ≈ x + s · e + HOT" - the crucial detail though is that e = e(x) because it is the gradient of D and D depends on x. Would the following derivation still hold in that case? If not, you should state it clearly, because the way the current text is written suggests that it holds for all differentiable corruptions.
* "x0 +(s−1)" - e is missing?
* Figure 2 - could you give more details on this experiment? For example, is it a reconstruction (starting with a latent generated based on a real image) or generation task (starting from a latent sampled from a prior)? How many steps were taken between the shown images? Currently, the second image in the top row looks totally blank, did the predictor already produce a blank image?
* "TACoS proposed in Algorithm 2 can be applied in this case by fixing the noise z used in the degradation operator D(x, s)" - I assume z is a constant used for generating all images, not resampled for every image?
clarity,_quality,_novelty_and_reproducibility: * Quality is high - the experiments are conducted properly.
* Clarity is also high, except the motivation for the functional form.
* Originality is moderate - prior work did investigate deterministic sampling, and I can't say the quality of samples in this work stands out in any way.
* Reproducibility is high - the authors provided source code in a supplementary material.
summary_of_the_review: This is an interesting empirical paper contributing to our understanding that per-step sampling might not be such a crucial component of diffusion generative models that was previously thought. I am learning towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes to use forward processes other than the Gaussian noise. This work learns using the x0 prediction formulation. This author proposes TACoS to build the reverse process. Experiments provide various types of forward processes.
strength_and_weaknesses: Strength:

This paper demonstrates that it is possible to use forward process other than the Gaussian noise.



Weakness:

1. This work lacks a comparison to the diffusion model that adds Gaussian noise. This work provides some generated samples, e.g., Figure 7 and Figure 21, and their sample quality looks worse than the diffusion model that adds Gaussian noise. For example, the diffusion model that adds Gaussian noise can generate sharper images than this work.

2. Since the sample quality is worse than the diffusion model that adds Gaussian noise, the argument " The success of these fully deterministic models calls into question the community’s understanding of diffusion models.." is overclaimed. 

3. In Section 3.2, the author states "Rather, diffusion models (Song et al., 2021a; Ho et al., 2020) perform generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm 1." However, Algorithm 1 does not correspond to the denoising process of diffusion models (Song et al., 2021a; Ho et al., 2020). Indeed, these methods use a linear combination of $x_0$ and $x_s$ to generate $x_{s-1}$, instead of only using $x_0$ as in Algorithm 1.

4. In Section 3.2, the author states "We find that the standard sampling approach in Algorithm 1 works well for noise-based diffusion". Is their any quantative result?

5. Missing experimental details. What is the number of generated samples to calculate FID? Which subset is used as the reference dataset to calculate FID?

6. It seems that TACoS sometimes performs worse than direct reconstruction. For example, in Figure 6, the samples of TACoS have more artifacts than the direct reconstruction. In Table 4, the TACoS quantative result is missing.

7. In Section 3.3, the author uses the first order Taylar approximation for arbitary $s$. However, this approximation only holds for small $s$.

8. In Appendix A.6,  it author set $D(\hat{x}_0, t) = \sqrt{\alpha_t} \hat{x}_0 + \sqrt{1-\alpha_t} \hat{z}$, which is inconsistent with that in Section 5.1. It would be better to choose another notation for $D$ in Appendix A.6.

9. In Table 5, the author shows that using DDIM sampling method would improve the sample quality compared to TACoS. Would the sample quality be further improved if we use the traditional diffusion model training objective? i.e., using a random $z$ instead of a fixed $z$ during training.


Summary:

Overall, the sample quality of the proposed method is not competitive to diffusion model with Gaussian noise. This is perhaps because TACoS is not able to exactly reverse the forward process. This contrasts to the reverse SDE or the ODE formulation in Gaussian diffusion model, which are able to exactly reverse the forward process. Besides, the idea of using different forward process is not new. For example, [1,2] have formally studied diffusion models with other forward processes. So in my opinion this work does not provide a sufficient contribution for a ICLR publication.

[1] Denoising Diffusion Gamma Models

[2] Structured Denoising Diffusion Models in Discrete State-Spaces

clarity,_quality,_novelty_and_reproducibility: This work is written generally clearly, although there is some notation inconsistency. This work is relatively less novel, since using other forward processes is not new.
summary_of_the_review: See "Strength And Weaknesses"
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper challenges the current mainstream direction that for diffusion-based generative learning we need noise injection to corrupt the data. Instead, it proposes using a series of deterministic processes that corrupt the input data via operations such as blurring, downsampling, masking, etc. For such processes, the paper also presents a novel sampling mechanism that improves results by a large margin. Experimentally, proof of concept experiments are presented but in most cases, the obtained results are much worse than mainstream diffusion models even on small datasets.

On the positive side, this paper goes out of the current school of thought that we need noise injection for learning diffusion models and provides very preliminary results that one can train these models on image datasets. However, it falls short in two aspects: It does not provide a theoretical guarantee that one can obtain a well-defined generative model using deterministic processes (similar to how diffusion models such as DDPMs are derived).  In the absence of such theoretical discussion, one would expect strong empirical results showing that deterministic process can in fact obtain generative results as good as current diffusion models with noise-injected corruption. Unfortunately, in the current form, the paper does not successfully create a compelling case that noise injection is not needed in diffusion models. Given this, the paper does not appear to be ready for publication at ICLR.

On a minor note, as the authors observed in sec 5.2, it is not straightforward to define a generative model with a forward blurring process with no noise injection. In fact, the blurring process will map all the images with the same average pixel value to the same delta distribution. Reversing this process with no noise injection will simply generate the pixel-wise average image of all those images that resulted in the same delta distribution. As you observed, noise injection is in fact necessary for breaking symmetries in the de-blurring process, however, if the generative model is not trained with any noise injection, it may not know how to use that in the reverse process. 
justification_for_why_not_higher_score: All the reviewers agreed that the paper is not ready for acceptance at ICLR.
Decision: Reject