Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a diffusion-based generative model trained on reversing a noisy and corruption process. Using the idea is that denoising score matching also works with general corruptions, score matching for corruption is proposed. To use it in a generative model, a momentum sampler is proposed on top of the naive sampler (which seems to have inferior performance). Experiments results on CIFAR-10 and CelebA show promising results in generative modeling, specifically on CelebA 64x64.
strength_and_weaknesses: *Strengths*

- Proposed a diffusion process combined with corruptions, such as blurring. 
- A leap-frog like procedure is proposed for sampling, which addresses the issue of changing corruptions in the naive sampler.
- Competitive FID results on generative modeling.
- The authors are rather honest about the limitations of the method, as detailed in Appendix E.

*Weaknesses*

**Reverse SDE might not recover desired distribution** 

The reverse SDE is based on an ad-hoc modification to the Gaussian diffusion SDE (specialized to the case where data distribution is delta), which makes it hard to justify why this would recover the desired data distribution (e.g., with Fokker-Planck). Wouldn't annealed Langevin dynamics / predictor-corrector be more principled?

**Experiments are limited**

All the experiments are done in CIFAR-10 and CelebA-64. While that is not anything to blame alone, it is difficult to generalize the conclusions to higher resolutions or even another dataset. One of the reasons is in Figure 3, it shows that the claimed SOTA FID is obtained by heavily tuning the number of NFEs, which is quite different from general findings that sample quality improves with more sample steps in standard diffusion models. Is this finding true for CIFAR-10 as well? 

Also, some results seem to be missing. 
- The Probability Flow ODE sampling with momentum is only listed for CIFAR-10 and not CelebA. 
- CIFAR10 results should also have a table, with results on naive sampler as well.
- It does not seem unreasonable to train a model on VP-SDE / Elucidated DM given their superiority than VE.

The standard Gaussian diffusion models is a high bar to overcome, and it is unclear whether the experimental results would convince practitioners to switch to methods based on blurring. There does not seem to be advantages regarding architecture design, or sampling speed, or training speed. The most promising direction is to use these corruption models in inverse problems, as the authors suggested in Section 6, but there is no empirical evidence on this either. 

**Novelty of Theorem 3.1**

Anything new about the proof of Theorem 3.1 that is not from Vincent (2011)? It appears all the tricks are the same, and $C_t$ does not even show up. The only change is $q_\sigma(\tilde{x} | x)$ (in Vincent's work that is a noise model) to $q(x_t | x)$ (that is the corrupted noise model), but the set of equations to prove the statements is basically the same.

**Inflated claims**

It is said that "Our method is superior in sampling time, for both CIFAR-10 and CelebA.". First, it is not mentioned the superior is against which method (likely VE-SDE, but that is probably the worst baseline for comparing sampling time). Second, the claim is not backed up for CIFAR-10 in the main paper (even in Figure 8, it is only shown for NFE within 100 and 170); only CelebA results are shown. 


------

These are not weaknesses, but simple out of curiosity:

- I would recommend an ablation study on the effect on blurring degrees to the generative model, since you can interpolate between the blur kernel that is being used and doing no corruption at all. This could substantially improve the arguments on section 3.3, and indeed show its promise in generative modeling. 

- The DDIM paper describes a series of solvers for the probability flow ODE and various SDEs (with different stochasticity in the sampling process). Is it possible to adopt that into this framework?
clarity,_quality,_novelty_and_reproducibility: Clarity (high): The clarify is generally clear, apart from some cases there are some inconsistencies with math notations (missing bold fonts for vector r.v.s, notation for Wiener process, $s_t$ only appears once).

Quality (high): there are some typos in the equations here and there, but generally it is okay.

Novelty (medium): technically this is not very surprising given existence of similar ideas, although there are some differences with prior work. The proposed SDE / ODE / samplers are not precisely tied to the original Fokker-Planck view of Gaussian diffusion SDE sampling, and can be viewed as an approximation of the original technique at best.

Originality (high): despite burgeoning similar ideas, the use of  is technically not explored before. It is a bit similar to the work on " Generative modelling with inverse heat dissipation", despite differences in training (denoising target is clean image instead of blurry image) and sampling (introduced momentum sampler).
summary_of_the_review: While learning diffusion models for blurring corruptions is interesting, the theorem is essentially rewriting the one in Vincent (2011), and the experiments are not comprehensive enough to demonstrate why one would favor this method than regular Gaussian diffusion. Even for a systematic study of soft diffusion alone, the work does not have a lot of CIFAR-10 results reported for CelebA, does not report NFE systematically (for many FID numbers except for the best ones, it is unclear how many NFE is used), does not have VP-SDE results, and does not explore ablation studies over the corruption process. Therefore, at the current stage, I am leaning towards rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work explores general linear corruption processes for diffusion models and practical training and sampling to improve performance for the new diffusion models. However, there are several technical questions unclear.
strength_and_weaknesses: Strength:

1. Authors show denoising score matching~(DSM) can be used to learn score for noised diffusion with general linear corruption processes theoretically. 

2. With the proposed several training and sampling improvements, diffusion models with general linear corruption can achieve competitive sampling quality on CIFAR10 and CelebA. 

Weaknesses:

**general corruption**

1. Theoretically, extending diffusion models from the isotropic corruption process to general non-isotropic corruption is quite straightforward based on original DSM work or recent.

2. Regarding performance, the claimed SOTA FID on CelebA 64 is not accurate,  where [3,4]  have lower FID. 

3. The performance of the proposed diffusion model in popular CIFAR-10 is worse than VPSDE or its variants. Authors compare with a weak baseline VESDE with reverse diffusion and ODE solver, whose performance is much better than the suggested Predictor-Corrector sampler. Authors claim "it is known[5] that diffusion models with VP SDEs usually do better in CIFAR-10". However, [5] suggests their VESDE can beat previous VPSDE and VESDE, and different diffusion models with isotropic corruption can be transformed into VESDE. The performance difference is due to various pre-condition and noise schedules during training. 

4. To the best of the reviewer's knowledge, the SDE of the general corruption, which has a general linear shift term, can also be transformed into VESDE with a similar technique in [5] or exponential transformation introduced in [6].

**Momentum Sampler**

1. The idea is quite interesting. It will be nice to compare with SOTA sampler in diffusion models, such as ODE solvers introduced in [5,6]. 

**Scheduling**

1. The idea that choosing scheduling based on minimal distance from D0 to D1 is new. However, there is no theoretical evidence to justify why minimizing distance with the selected w2 distance is better. It is hard to predict the benefits of the choice or show the choice can minimize discretization error when solving sampling SDE or ODE. 

2. Besides, estimating Wasserstein distances accurately between two datasets is quite expensive, even in CIFAR10 32x32 datasets. It would be nice to include more details, such as whether authors use partial datasets or some approximations in solving optimal transportation problems in the image dataset scale.



[3] Kim et al. Maximum Likelihood Training of Implicit Nonlinear Diffusion Model
[4] Wang et al. Diffusion-GAN: Training GANs with Diffusion
[5] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models
[6] Zhang et al. Fast Sampling of Diffusion Models with Exponential Integrator
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
novelty: marginally
Reproducibility: looks good
summary_of_the_review: In summary I think the proposed method is interesting and technically sound. But the benefits over previous methods are not clear both in theory and practice. However, the benefits over previous methods are not clear both in theory and practice. I would like to increase my score if the authors can address my questions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper extends the original diffusion models to linear corruptions, which include Gaussian blur and masking. Motivated by the SDE form of the backward process, the paper proposes the so-called "momentum sampler".
The noise/corruption scheduling are selected based on some Wasserstein matric. Empirically, they improve the FID score from previous SOTA (1.90) to 1.85 on CelebA 64x64.
strength_and_weaknesses: 
Strength:

(1) Build a framework for diffusion with general linear corruptions.

(2) Corruption scheduling using the Wasserstein metric seems interesting.

(3) Momentum sampler seems interesting.



Weakness:

(1) The authors should provide specific forms of $C_t$ for Gaussian blur and masking.

(2) The scheduling of the diffusion is unclear. e.g., how do you estimate the Wasserstein distance? How is the noise level $s_t$ scheduled? (Figure 6 alone is not enough for replicating the result.) What is the tunable parameter in $C_t$? 

(3) Experiments: you should faithfully report your results on CIFAR10, and summarize them as a Table.  

(4) To argue about the superiority of the sampling speed, the authors should early-stop the sampling process, e.g. at the 200 steps, and provide a visualization of examples. 

(5) About Figure 3: Why only compare your method with NCSN++? what about DDPM++, which is the previous SOTA as reported in Table 1?
clarity,_quality,_novelty_and_reproducibility: -
summary_of_the_review: The delivery of the paper should be improved. The experimental sections should be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a diffusion process that can include various corruption types, such as blur. It uses a leap-frog-like sampling procedure and achieves competitive FID scores on benchmark datasets. 

The paper receives three reviews. All the reviewers rate the paper below the bar. Several weakness includes potential flaws in the proposed framework, limited experiment results, and over-claiming. The authors, based on the reviewers' suggestions, include a large set of new results. The submission is substantially different from the original one. However, none of the reviewers update their ratings despite the authors' consistent push. The AC consults with several reviewers separately, confirming that they have read the rebuttal and updated the paper. However, it appears they still have doubts about the acceptance of the paper.

After reading the paper, reviews, and rebuttal, the AC feels the reviewers' concerns are justified. The paper isn't convincing enough in its current form and requires additional work to make the cut.
justification_for_why_not_higher_score: The proposed framework contains potential flaws. None of the reviewers feel comfortable recommending acceptance.
Decision: Reject