Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes to train diffusion models via multi-stage progressive signal transformations. Compared to traditional diffusion models, the signal transformation here is more general and can be any transformation from coarse to fine. The authors focus on three main transformations (down-sample, Gaussian blur and transformations learned from a neural network) in the experiments, and demonstrate the effectiveness of the approach.
strength_and_weaknesses: Strength:
1. The paper is well-written with thorough experimental results.
2. This paper considers a very interesting direction of diffusion models by incorporating progressive signal transformation.

Weakness:
1. Limited novelty: Similar ideas (e.g., incorporating progressive signal transformation into generative modeling) have be extensively explored for other generative models including GANs [1], normalizing flows [2,3], and VAEs. For the three major transformations considered in the paper (down-sample, gaussian blur and VQ-VAE/VQ-GAN), there are already existing work on diffusion models using each of the transformation as mentioned in the paper. Specifically, for transformations learned by neural networks, diffusion models trained on a latent space fit into this category: there have been various works in this direction and it is not a new idea. For instance, besides LDM, [4,5,6] also perform diffusion on data transformed with an encoder. In some sense, the contribution of this paper seems to unify these approaches into one framework by calling them "diffusion models learned with progressive signal transformation".

2.  Limited theoretical insights: the contribution of this paper is mainly on the empirical side. There are not enough theoretical insights or guarantee.

3. It seems that the performance (e.g., sample quality/speed) depends heavily on the transformation $f$ and the stage scheduling. More discussion should be provided on the selection of the optimal tranformations. Although the transformation are claimed to be any transformation, it remains unclear how effective/useful the other transformations are besides the ones used in the paper  (down-sample, Gaussian blur and transformations learned from a neural network). However, there are already existing work using these transformations. 
 
4. In the experiment, the reimplemented Cascaded DM (figure 4) seems to have much worse performance than the ones reported in the original paper. It is unclear how trustworthy the results are. 

5. For comparison with DDPM with fewer steps (DDPM 1/2), the authors should also consider comparing with DDIM 1/2 [7] in Table 1.

6. How expensive it is to find the stage scheduling used in the experiments? Is there any theoretical connection to ODE/SDE? 

[1] Progressive Growing of GANs for Improved Quality, Stability, and Variation: https://arxiv.org/abs/1710.10196

[2] Wavelet Flow: Fast Training of High Resolution Normalizing Flows: https://arxiv.org/abs/2010.13821

[3] Improving Continuous Normalizing Flows
using a Multi-Resolution Framework: https://arxiv.org/abs/2106.08462

[4] D2C: Diffusion-Denoising Models for Few-shot Conditional Generation: https://arxiv.org/abs/2106.06819

[5] Diffusion-LM Improves Controllable Text Generation: https://arxiv.org/abs/2205.14217

[6] Symbolic Music Generation with Diffusion Models: https://arxiv.org/abs/2103.16091

[7] Denoising Diffusion Implicit Models: https://arxiv.org/abs/2010.02502

clarity,_quality,_novelty_and_reproducibility: The paper is well written. However, it has limited technical novelty (see weakness). 
summary_of_the_review: This work considers learning diffusion models on data processed with a sequence of signal transformations. Similar ideas have been explored in other generative models, including normalizing flows, GANs and VAEs. It is almost straightforward to apply similar ideas to diffusion models. Although the proposed approach is claimed to work for any transformation, it remains unclear how effective it would be for random transformations besides the ones the authors performed experiments on. At the same time, there are already works that learn diffusion models on down-sampled images or latent spaces. Given these related works, the novelty of this work is questionable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a diffusion model which incorporates functions other than the addition of Gaussian noise in the forward process. Concretely, they show results with downsampling functions, blurring, and a learned neural encoding. 
strength_and_weaknesses: Strengths:
- At a high level, I think this is an elegant method to integrate transformations into a diffusion process. This could be of considerable interest to the ICLR community.
- Well thougt-out design choices including interpolating between $\hat{x}_k$ and $\hat{x}_k$ at each stage between transformations (as described in Equation 4), parameterisation of the network to predict both $\epsilon$ and $\delta$, and definition of $q(z_t|x)$.
- Good experimental results.


Weaknesses:
 - Slightly confusing presentation - reading Section 3.1 was reasonably difficult. I think I understand it after spending considerable time on it, but suspect that it could be made clearer with more explanation (or perhaps a diagram/expanded version of Fig 2b) of the relationships between each $x$, $\hat{x}$, $z_t$, etc. 
- In Equation 7, $\Omega$ is said to be the "minimal interested patch", with no further explanation. And the "SIGNAL" and "NOISE" functions in Eq. 7 are not explained at all. The conclusions reached about setting $\alpha_{\tau}$, $\alpha_{\tau^-}$, $\sigma_{\tau}$, and $\sigma_{\tau^-}$ seem reasonable, but their relationship to Eq. 7 needs more explanation.
clarity,_quality,_novelty_and_reproducibility: Clarity: Could be improved - see weaknesses above

Quality: See strengths/weaknesses.

Novelty: Novel AFAIK

Reproducibility: Contains all required details as far as I can tell. The authors also commit to releasing code.
summary_of_the_review: The proposed method seems to be well-thought out and of interest to the ICLR community. I recommend acceptance, although the clarity/presentation could still be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission deals with designing diffusion models by using progressive signal transformation. It proposes a generalized formulation of diffusion models, termed f-DM, with a modified sampling that is applied to image generation tasks with a range of signal transformations such as down-sampling, blurring, and learn compression based on pretrained VAEs. The reported experiments show better efficiency, quality and interpretation for image generation based on a few standard image datasets. 

strength_and_weaknesses: Strength
- It is a solid idea that generalized denoising diffusions to other signal transformation tasks
- Experiments are performed a diverse set of generation tasks and datasets

Weakness
- The comparison with LDM, based on the experiments, is not conclusive. LDM (GAN) seems to be achieving the best FID, while Fig. 4 shows images with artifacts. Also, for more realistic and diverse datasets such as LSUN and ImageNet no comparison is performed with LDMs, while LDMs tend to achieve much better speed. 

Questions and comments
- About f-DMs versus LDMs, it is not theoretically convincing if progressive transformation (such as down-sampling) can be more efficient. - At least from the experiments one can see that LDMs are much more efficient, and one needs to see the LDM trained with strong encoder and decoders on diverse datasets such as LSUN and ImageNet to make a conclusion.  

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The idea of using progressive signal transformation is also meaningful and novel. 
summary_of_the_review: This paper deals with an important and timely problem. The proposed solution is also a solid generalization of denoising diffusions to general signal transformations and distortions. The reported experimental results also demonstrate significant FID improvement over some benchmarks. However, comparison with LDMs is not quite convincing and clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes different types of learned and fixed encoders for DDPMs and tests the proposed schemes against other models for a range of benchmarks.

All reviewers appreciate the contribution and vote for acceptance.
justification_for_why_not_higher_score: The contribution could be clearer formulated.
justification_for_why_not_lower_score: Good solid contribution within a red-hot area.
Decision: Accept: poster