Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach to improving the sample efficiency of diffusion models by incorporating high-order derivatives. As high-order derivatives are often expensive to compute, the authors propose an approximation. Empirically, the proposed approach is able to generate images using small number of sampling steps. 
strength_and_weaknesses: **Strength:**
1. This work considers an important direction of diffusion models: as diffusion models are known for their inefficiency in sampling, improving the sampling speed is of great interest to the community.
2. Using high-order derivates to improve the sampling speed is an interesting and promising direction.


**Weakness:**
1. The writing needs to be improved. There are many equations with parameters/notations that are not properly defined in Section 3.4. The main algorithms (Algorithms 1, 2) are not properly justified or explained with rigorous proof. The method formulation is hard to follow.
2. Proposition 1 is not rigorously stated. For instance, what does "in many cases" mean? What does an "arbitrary vector" mean? What would be the dimension of the vector? What would be the domain?
3. Although in the left figure in figure 4, the proposed approach (Taylor 2nd, 3rd) is able to achieve better performance than the baselines, the performance degrades after 20 steps. Why is that the reason? At the same time, the DDIM performance reported in figure 2, 4 is much worse than the one reported in the original DDIM paper: in the original paper, DDIM is able to achieve an FID of 13.36 on CIFAR-10, and 17.33 on CelebA 64 using 10 steps---a performance better than all of the proposed methods using 10 steps. However, in both figure 2 and figure 4, the reported performance for DDIM is much worse. My understanding is that this can be caused by using different noise scheduling. If that is the case, the comparison in figure 2 and 4 might not be fair for the baselines.
5. It seems that noise scheduling will affect the performance. How do you select the noise scheduling parameters?
6. It is hard to tell the difference between the samples from DDIM and the ones from the proposed method in Figure 1. At the same time, the samples seem to have shifted color. If it is because of not having enough sampling steps, then it would be better to visualize samples with higher quality but using more steps.
clarity,_quality,_novelty_and_reproducibility: Although this paper considers a very interesting direction that could potentially have novelty, the clarity of the writing and the quality of the experiments need to be improved.
summary_of_the_review: Given that the clarity of the paper is limited, the experiments are not very impressive or convincing, the entire paper needs to be improved for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a novel solver of the Probability Flow ODE for diffusion models introduced in [1]. By taking a Taylor expansion of the ODE and including higher-order terms, the solver can take larger steps. This speeds up sampling, which is a well-known computational bottleneck in diffusion models. However, the higher order terms in the Taylor expansion are themselves expensive to compute. Thus the authors propose substituting these terms with "ideal derivatives" which involve 

[1] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. 2021

strength_and_weaknesses: Strengths:
* The work tackles a relevant problem in current score-based diffusion models: slow data generation.
Weaknesses:
* The empirical results in the main Figure 2 are not very compelling, compared to, e.g., [2, 3, 4].
* There is limited discussion on why the "ideal derivatives" are a suitable replacement for the true derivatives. Arguments (in the appendix) are mainly intuitive, with no proofs.
* The extra theory is dense, and it is unclear whether it is worth the empirical improvements.

[2] Salimans, T. and Ho, J., 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.
[3] Song, J., Meng, C. and Ermon, S., 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
[4] Kong, Z. and Ping, W., 2021. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132.
clarity,_quality,_novelty_and_reproducibility: Writing is unclear: For example,

"[DDIM] is not necessarily derived directly from PF-ODEs, and its relationship to PF-ODE was revealed through a little argumentation"

"Nevertheless, in diffusion models, the derivatives are expected to have good structure, and are effectively evaluated."

"The Jacobian matrix is diagonal assuming that each dimension is independent of each other." <--- Is this really a valid assumption?

"To date, this approximation has often been understood as a "tractable surrogate". <--- What does this mean? Citations?

"50,000 images were generated for each condition to compute the FID scores." Why 50,000? The standard is 10,000 [5].

Correctness and Novelty:
The proposed solver hinges on the use of "ideal derivatives", which replace the true higher order terms in the Taylor expansion of the ODE. While the idea of applying an idealized derivative substitution to a Taylor expansion of the diffusion ODE is novel, its correctness is unclear.
Moreover, the general derivation of the solver requires assuming that the diffusion in each data dimension is independent. This seems like a very strong (and unrealistic) assumption to me.

[5] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. and Hochreiter, S., 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
summary_of_the_review: Overall, I find the motivation of the paper relevant. However, I find key steps of the derivation of the method problematic (i.e., the "idealized derivative" assumption, the "independent dimensions" assumption). Moreover, empirical evaluations are lacking. From Figures 2 and 4, it appears that image quality in terms of FID is not competitive compared to other faster diffusion models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors deal with the acceleration of denoising diffusion models. In particular, they propose improved samplers with higher order in order to reduce the number of steps requires at sampling times. The acceleration proposed in that paper is described for both Ordinary Differential Equation (ODE) and Stochastic Differential Equation (SDE) flows. It is based on the higher-order integrators [1]. In order to efficiently compute the derivatives the authors identify the score $\nabla \log p(x_t)$ with the conditional score $\nabla \log p(x_t|x_0)$ which has a tractable expression. Doing so they are able to compute efficient approximations of the derivatives. The theoretical and methodological study is complemented with experiments on CelebA 64x64 and CIFAR10.

[1] Kloeden, Platen, Schurz - Numerical Solution of SDE through computer experiments
strength_and_weaknesses: STRENGTHS:
* One of the strenght of this paper is that it introduces a novel way to sample from diffusion models using the theoretical of higher order integrator. To the best of my knowledge this approach is novel.
* I also found the use of the approximation of the score by the conditional score to be quite interesting.
* The paper is well-written and the ODE, SDE and reverse ODE, SDE are clearly introduced.

WEAKNESSES:
* I think the authors make misleading claim regarding the Backward Kolmogorov Equation (BKE) (by the way, maybe I missed it in the main text but FPE and BKE are never defined, I just assumed that FPE was Fokker-Planck Equation and BKE was Backward Kolmogorov Equation). I could not find in [1] any reference to the backward Kolmgorov Equation but maybe I missed something here. However I disagree that the reverse-time dynamics is somehow associated with BKE. In fact FPE and BKE are dual of each other but there is no connection here with the time-reversal. The time-reversal dynamics also satisfies FPE and BKE evolutions but these are not related to the BKE of the forward process (even though one can use the BKE to establish the time-reversal SDE, see [1]).
* More important maybe, I don't really understand why the authors say that the derivatives of the learned score functions cannot be computed? It is easy to perform autodifferenciation w.r.t. the (one-dimensional) time variable and one can use vector-Jacobian-product to compute the  term $a . \nabla s_\theta$. I might be missing something but it seems that these issues can be leveraged by the careful use of automatic differenciation.
* Proposition 1 is very poorly worded. What are these "many cases"? I understand that here the authors are trying to justify their method but this is too hand-wavey. As of now the statement is too imprecise. What are the edge cases? How could we extend this proposition to hold in a more general setting?
* The deterioriation of the method noted by the author "the proposed Quasi-Taylor methods tend to give good results around $N=16,20$ and the FID deteriorates from there" is quite worrisome. This deterioriation which to me is a key limitation of the work should have been investigated in greater depth.
* The FID results provided by the authors are quite surprising. For CelebA a FID of 20 something is very large. In the original DDIM paper [2], the authors reported way better results with FID around 17 even for 10 steps and FID around 6 for 100 steps. Can the authors explain this stricking discrepancy between the announced numbers and the reported numbers in [2]? 
* I think that the authors do not discuss important part of the literature dealing with acceleration of diffusion models. I understand that the authors are not going to compare themselves with knowledge distillation approaches like the one of [3] because these approaches are quite different from the one considered in that paper which is focused in better samplers but improved samplers have already been proposed in the literature, like [4,5] for instance. Comparisons with these methods are important and omitted here.

[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole - Score-Based Generative Modeling through Stochastic Differential Equations

[2] Song, Meng, Ermon - Denoising Diffusion Implicit Models 

[3] Luhman, Luhman - Knowledge distillation in iterative generative models for improved sampling speed

[4] Liu, Ren, Lin, Zhao - Pseudo Numerical Methods for Diffusion Models on Manifolds

[5] Zhang, Chen - Fast Sampling of Diffusion Models with Exponential Integrator
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is quite clear (with the exception of the discussion on BKE which I found misleading as emphasized before).

The methodological contribution of the paper is quite interesting but I did not find the theory and experiments of the paper to be compelling. 

The work is quite novel and I think that the use of the approximation  $\nabla \log p(x_t)$ with the conditional score $\nabla \log p(x_t|x_0)$  could be useful.

Experimental details to reproduce the introduced method are provided.
summary_of_the_review: While I think that the issue of accelerating diffusion models is interesting, I think that the paper has several important flaws from a methodological and experimental point of view.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes high order methods for solving both SDE and ODE, by approximating high order gradients in Taylar expansion. It seems that this method can be extended to arbitrary order by computing some constants. Experiments are conducted on CIFAR10 and CelebA 64.

strength_and_weaknesses: Strength:

This work proposes high order methods for solving both SDE and ODE. It seems that this method can be extended to arbitrary order without much extra cost.

Weakness:

1. The approximation in Eq.(19) and Eq.(21) looks rough. Is there any bound of the approximation error?

2. It seems this method needs to use a specific noise schedule function in Eq.(28), which makes it less flexible. Would this method work on other noise schedule functions?

3. There are only experiments on CIFAR10 and CelebA 64. Besides, the used diffusion model is also weak, which has a FID around 10 on CIFAR10.

4. Missing related works on speeding up diffusion models, such as [1, 2, 3].

[1] Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

[2] Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models

[3] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

clarity,_quality,_novelty_and_reproducibility: The paper is written clear. Since the idea of approximating high order gradients exists, such as [3], this work is relatively less novel.
summary_of_the_review:
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes high order methods for solving both SDE and ODE, by approximating high order gradients in Taylor expansion. The idea is applied to diffusion models, with experiments conducted on comparing different integrators of the backward SDE.

Reviewers agreed that the paper aims to address an important research question for diffusion models -- speeding up the generation process -- and they thought the proposed approach is novel. 

On the downside, however, reviewers found the presentation of their main approach to be confusing, and they have concerns on the reported experimental results as compared with results in previous paper (e.g., the overall FID results seem to be too bad). Even beyond that, after revision with the updated results, the proposed approach are not significantly better than prior arts.

Although the paper is not good enough in its current form for acceptance at ICLR, I would encourage the authors to revise and resubmit this paper to other ML venues. Perhaps better presentations for the insights behind the ideal derivative approach, plus a check up on the experimental implementations, e.g., the FID evaluation framework, could help for the next submission.
justification_for_why_not_higher_score: While the method could potentially be useful, reviewers found that in the paper's current form, the method is not presented in good clarity, and the experimental results are not significant.
Decision: Reject