Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a generalized framework based on DDIM that explains the acceleration led by DDIM and can be extended to non-isotropic diffusion models like Blurring diffusion model (BDM) and Critically-damped Langevin diffusion model (CLD). The authors build up their insights using a toy dataset consisting of only one data point, in which the parameterization of DDIM can produce smooth network output along trajectories and thus enable large step size to achieve sampling acceleration. Then this insight is generalized to real dataset by considering the manifold hypothesis and a mixture of Dirac distributions. As an application of this framework, authors first rewrite non-isotropic BDM and CLD into gDDIMs and show the acceleration achieved by the smoothed estimation along the trajectory.
strength_and_weaknesses: Strengths:

1. The analysis toward understanding why DDIM accelerates the sampling process is well organized and inspiring. Following that, because complexity can be decomposed into simple cases, extending and interpreting the results in real datasets is natural. (even though in a less rigorous way) The research methodology is of merit, starting from a clear analysis of the simple and synthetic case and finally carefully verifying this on realistic datasets.
2. Given the fact that authors train an adjusted CLD, (which is not training free to achieve the best results), the final results are pretty good.

Weaknesses/Concerns:

1. To apply the acceleration techniques, one first needs to formulate irregular diffusion models into a regular form provided by this paper and make a correspondence between less structured generalized diffusion model designs with parameters and formulations in this framework, which incurs an invisible bar for many practitioners. This is somehow not an off-the-shelf acceleration method.
2. Authors spend a notable portion explaining the hyperparameters in CLD and the correspondence between the original CLD and newly designed in this paper, like the $L_t$ and $R_t$. In this case, the gDDIM is not a training free method. To achieve the best acceleration, training from scratch is necessary, which actually changes the model design for CLD.
3. The writing of the introduction can be polished much further. Without good logical guidance, it is a bit confusing for readers to see many less structured references to the method section. The authors should compress the related works in the introduction, move this part to the related works, and use the additional space here to organize and present the *logic* of this paper.

Minor comments:

1. Related works should be presented in the main paper.
2. The definition of $L_t$ should be introduced earlier in the main paper, as this is discussed and compared in the experiments but not clearly stated.
3. It should be $N$ score evaluation in Table 8, not $n$ score evaluation.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is with good clarity (except the introduction). This paper is based on DDIM, BDM, and CLD, so has limited novelty. However, the authors showed interesting results.
summary_of_the_review: This paper is overall interesting but not appealing enough. Its merits and weaknesses are both clear. Authors make great efforts to provide a comprehensive analysis. However, there are numerous weaknesses that limit the scope of this paper. Based on the authors' response, I am happy to update my score accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This submission deals with DDIMs. It proposes a generalization of DDIMs to general diffusion models beyond the isotropic denoising by modifying the parameterization of score. It links DDIMs to SDEs by a score approximation, and provides an interpretation for the success of DDIMs. It validates the results for BDM and CLD settings, where it achieves strong FID scores for CIFAR-10 sample generation under a small number of function evaluations compared with the original samplers. 
strength_and_weaknesses: Strength
- This is the first work to generalize DDIMs (gDDIM) using a theoretically grounded framework
- It provides insightful interpretations for the DDIMs that explains why ODE samplers perform well
- The reported experiments show strong improvements over the original samplers
- The paper is well written and contains sufficient details 

Weakness
- The results could be more solid if tested with more realistic datasets beyond the toy CIFAR-10 such as for LSUN and ImageNet

clarity,_quality,_novelty_and_reproducibility: This is a well written paper with a clearly explained idea. The proposed gDDIM is novel and original, and the explanations for the success of DDIM are also novel. 

summary_of_the_review: This paper addresses an important and timely problem. It proposes a theoretically grounded gDDIM framework that generalized DDIM beyond isotropic denoising, and provides insightful interpretations for the good performance of DDIMs. The experiments also show significant improvement over original samplers.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explains that the exact solution can be numerically found in DDIM with fast sampling.
Extending this explanation, this paper proposes "generalized DDIM" that modifies parameterization of the score networks of existing diffusion models to achieve 20x acceleration and comparable or much better FID even on non-isotropic diffusion models.
strength_and_weaknesses: 
Strengths

(+) The proposed generalized DDIM is versatile to different diffusion models.

(+) Step-by-step logical flow: Dirac distribution to Gaussian distribution to real distribution, deterministic case to stochastic case, etc.

(+) Straightforward illustration for understanding ideas.

Weaknesses

(-) Connection from Dirac distribution to the real distribution appears at the bottom of page 6. Bringing it to the top of Section 3 and describing "empirical distribution with weights vs Dirac = real distribution vs one sample" would provide the whole picture to the readers.
(-) Missing link: how do we move from Gaussian and to real distributions? I do not find assumptions or proofs.

Misc.
(-) Please leave vspace for readability, e.g., Figure 1.
(-) Please place figures close to its connected paragraphs, e.g., Figure 1.
(-) In Table 4, please fix the typo "# of arameters".

clarity,_quality,_novelty_and_reproducibility: (+) This paper provides sound mathematical backups for gDDIM.

(-) The terms are excessively omitted. E.g., time-varying noise schedule $R_t$ -> time-varying $R_t$

(+) The theoretic results and applications are original.
summary_of_the_review: This paper shows non-trivial advance of diffusion models in identifying principles of DDIM and enabling fast sampling. Though its content is valuable, the presentation can be nicer to be easier to follow.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a theoretical framework that extends DDIMs, which were derived only for DDPMs, to a more general form of diffusion SDEs.
strength_and_weaknesses: Strengths:
- The work proposes a generalized framework for deriving accelerated samplers from standard diffusion models.
- The proposed framework performs well empirically, and allows one to make informed choices on how the learned score function should be reparameterized (e.g. CLD appears to be much improved in the low NFE scheme by choosing $K_t = L_t$).

Weaknesses:
- While the derivation produces the original DDIM, when applied to DDPM models, it involves several steps that make questionable assumptions (e.g., Section 3 assumes unperturbed data is a Dirac distribution on a single point, Section 4 assumes data is normally distributed.)
clarity,_quality,_novelty_and_reproducibility: Clarity:

Section 3: The authors state that the Euler and Runge Kutta solvers on the ODE cannot obtain a single-step recovery of a data point, while DDIM can. However, the original work on DDIMs [1] appear to contradict this claim: [1] states that DDIMs are probability flow ODEs of the variance exploding diffusion SDE as described in [2]. 

Proposition 3 states that Eq. 13 holds when the data is a Dirac distribution for a single data point. However, Thm 1 assumes that Eq. 13 is approximately true for general data. Why is this a valid assumption?

Dirac $p_0(u)$ and the manifold hypothesis: I found the reasoning in this section unclear. Can the authors elaborate on why the manifold hypothesis explains the effectiveness of DDIMs?

Does the integral in Eq. 18 have a closed form solution? If not, how is it computed during sampling?

[1]  Song, J., Meng, C. and Ermon, S., 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.

[2] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S. and Poole, B., 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
summary_of_the_review: Overall, the contribution of this work is simple yet significant. While the proposed framework for building generalized DDIMs relies on several somewhat unrealistic assumptions (see first point in Weaknesses), these choices are at least partially justified by the promising empirical performance of the resulting samplers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work generalizes the DDIM framework and proposes acceleration ideas that are demonstrated to be effective for three examples. 

At this point, it is still not clear if the training-free property holds in general cases, but the improvements in the current results suggests that the proposed ideas hold their ground in interesting subcases. 

The writing of the paper is quite clear and the authors do a great job in explaining and interpreting their results.  
justification_for_why_not_higher_score: Mainly the generality of the proposed accelerations is not clear. Otherwise, overall a solid paper. 
justification_for_why_not_lower_score: The demonstrations are quite compelling.
Decision: Accept: notable-top-25%