Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of the diffusion-based model with "inverse heat dissipation" (equivalent to continuous de-blurring Gaussian blur). Compared to the standard diffusion model, inverting the heat equation -- a PDE locally erases fine-scale information -- makes the generation consider multi-scale structures naturally. The paper also shows empirical evidence on emergent properties when applying IHDM, such as disentanglement of color and shape.
strength_and_weaknesses: Strength:
-- An interesting study is one of the first several works applying blurring transformations in the diffusion process. Unlike existing works, the paper directly derives the formulation from the heat equation SDE and analyzes the inductive bias compared to the standard diffusion models.

Weakness:
-- The empirical results are relatively much worse than standard diffusion models. The paper seems to hide the quantitative comparisons (only small captions in the figure). However, the visual quality is also over-smoothed compared to the baselines. Although it is brought by the change of diffusion process, it is hard to justify "why we want to apply heat equation in generative models in the first place," considering there is less merit in doing so.
-- Moreover, there are also existing works that try to incorporate Gaussian blurring in the diffusion process. For example, 

Lee, Sangyun, Hyungjin Chung, Jaehyeon Kim, and Jong Chul Ye. "Progressive deblurring of diffusion models for coarse-to-fine image synthesis." arXiv preprint arXiv:2207.11192 (2022).

Similar to this submission, their method applies a stochastic blurring process in diffusion models. While having differences, their quantitative and qualitative results are much better and match the same performance of standard diffusion models. I think comparing more carefully than citing them solely and indicating solutions to improve the current method is necessary. 

-- Noise scheduling is less clear than the standard diffusion models, especially since this paper involves two hyperparameters, "sigma" (forward) and "delta" (backward), which, however, are unclear how to choose both of them and how to adapt with the blurring schedule (Gaussian kernel)
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality:  the paper is clearly written and easy to follow. The mathematical formulation is technically meaningful.
Novelty: the paper proposes a novel method by inverting a stochastic heat equation, which only extends standard diffusion models with Gaussian noise.
Reproducibility: the paper includes a detailed description of implementation and training details.


summary_of_the_review: Overall, based on the review above, I think the methodology and the proposed method are technically reasonable and novel, which can be a good contribution toward a better understanding of diffusion-based models beyond Gaussian noises. However, as the concerns raised in the weakness, the poor visual performance and lack of proper comparison/justification with baselines decrease the actual impact.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new generative model by using inverse heat dissipation. The proposed model is motivated by diffusion models and the empirical success of coarse-to-fine modelling. Specifically, the authors propose to synthesize images by iteratively inverting the heat equation, which is basically a PDE that locally erases fine-scale information. Furthermore, they theoretically interpret the noise-relaxed solution of the forward heat equation as a variational approximation in a diffusion-like latent variable model. Extensive theoretical and experimental analyses have been conducted to demonstrate the effectiveness of the proposed model. 
strength_and_weaknesses: Strengths:
This paper proposes a new type of generative model based on inverse heat dissipation, which possesses a different mechanism compared to existing diffusion models. Moreover, a solid and extensive mathematical derivation of method details has been provided in the Appendix. The proposed method shows emergent qualitative properties not seen in existing diffusion models, which has also been proved by experimental results.

Weaknesses:
From the qualitative results shown in this paper, it seems to me that existing SOTA generative models (e.g., StyleGAN, IMAGEN, DALLE) clearly outperform the proposed method. Yet, since the paper opens a new way for designing generative models, it is acceptable for me that the quality of synthesis results obtained by the proposed one is not as good as SOTA ones. However, there is no qualitative and quantitative performance comparison between the existing methods and the proposed one. Please explain the reasons. In fact, no matter whether the performance is good or not, it is still necessary to compare the proposed method against existing approaches. 

clarity,_quality,_novelty_and_reproducibility: Good. 
summary_of_the_review: This paper proposes a new generative model which is both theoretically and technically sound. Extensive mathematical deviations of method details have also been provided making it a solid paper. Despite the above-mentioned weaknesses, I believe that this paper is still deserved to be published in ICLR after some necessary revisions. The proposed method is interesting and shows up some valuable potential for future research. 

However, the quality of synthesized images of the proposed method is not as good as the state of the art, and there is no comparison between existing methods and the proposed one. Otherwise, I could give it a higher score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to combine heat dissipation forward process with gaussian diffusion.
strength_and_weaknesses: \+ This is an interesting topic and it improves our understanding of what is possible in generative modeling.

\+ The experiments are well-analyzed.

\- The main text of the paper contains excessive details which make it harder to read.

\- Experimental results do not show that the proposed method is a promising direction, at least in terms of the sample quality.

Questions/minor comments:
* "Their generative process does not explicitly consider the inductive biases of natural images, such as their multi-scale nature" - Imagen (https://arxiv.org/abs/2205.11487) does use upsamplers, some of its predecessors also do.
* This work is not just heat dissipation, but rather a combination of heat dissipation with gaussian noising - this needs to be highlighted early in the paper. "noise-relaxed solution of the forward heat equation" in the abstract sounds rather obscure, just saying that it's a combination with gaussian diffusion would be more clear.
* " We obtain samples by taking a training example, blurring it with F(tK), and adding noise with variance δ2" - does it mean that the prior is non-parametric, so we would need the training set during the generation time? That sounds like a limitation which should be mentioned early on?
clarity,_quality,_novelty_and_reproducibility: * Quality is high, experiments are well-conducted and analyzed.
* Clarity is moderate, with too many details cluttering the main text.
* Originality is high.
* Reproducibility is high - the authors included the source code in a supplementary material.
summary_of_the_review: This is an interesting work proposing a novel approach to generative modeling of images based on heat dissipation. I am learning towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use heat dissipation to define a forward process (blurring) in the diffusion model. It uses eigendecomposition to decompose the Laplace operation and uses a variational model, similar to DDPM to learn the generative process. Gaussian noise is added to ensure the diversity of the generated images.

Overall using different blurring ops is an interesting exploration direction.
strength_and_weaknesses: Strength:
1. Overall the idea is valid and interesting;
2. This paper provides several interesting analysis such as PSD analysis.
3. Lots of experiments are conducted to show interesting properties of the model.

Weakness:
1. There is not much supportive evidence to demonstrate the unique advantage of using heat dissipation. This greatly reduces the novelty of the paper. I believe there should be some important undiscovered advantages to using this approach. As the effective resolution is changing --- can it save the inference time?

2. There are several numerical approach to deal with the Laplace ops: 1. the eigendecomposition used in this paper; 2. directly using Gaussian blurring; 3. grid-based finite difference; To me there seems no significant difference between these three approaches. Although author mentioned numerical stability and some gradient analysis, I would say it is worth really running those models to exam the outcomes.


clarity,_quality,_novelty_and_reproducibility: Questions:

1. Can you give a definition of the power spectral density (PSD) of natural images?
2. Seems the prior is estimated using KDE ---- that means we need extra space complexity for that model?

Overall, the clarity is good. The novelty is less due to my comments in weakness.
summary_of_the_review: Please see above. I will adjust my score based on the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission builds on top of a classical connection between the heat equation and the blurring process and introduces a new generative model by inverting the blurring process. Spectral analysis reveals nice connections to the implicit inductive biases in diffusion models, showing that the proposed process has potentially better corruption mechanism for natural images. Overall, the reviewers have recognized the novelty of this work and the importance of the spectral analysis. However, their major criticism is around the empirical results that show significantly inferior performance compared to existing diffusion models. After careful consideration, the AC believes that in the long run, the research community can benefit from this work and therefore recommends accept. 

Minor comment: The spectral inductive biases of diffusion models are also concurrently discussed in the following tutorial [here](https://youtu.be/cS6JQpEY9cs?t=2571). Please include a reference to:
Kreis et al. CVPR 2022 Tutorial on Denoising Diffusion-based Generative Modeling: Foundations and Applications. 
justification_for_why_not_higher_score: Unfortunately, the empirical results are not strong enough and the community may not immediately switch from diffusion models to models proposed in this work. 
justification_for_why_not_lower_score: The spectral analysis and connection between heat equation and blurring can benefit the community in long run.
Decision: Accept: poster