Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 10: Award quality: Technically flawless paper with groundbreaking impact, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel higher-order denoising diffusion solver, called GENIE, which is applicable particularly for denoising diffusion models with the variational preserving forward noising process. The GENIE solver is established from two novel approaches proposed in the paper.

First, the paper identifies that p-th order Taylor approximation (truncated Taylor Method, TTM) to DDIM ODE results in much simpler terms than the TTM of the equivalent probability flow ODE.
- Note that an equivalent probability flow ODE of a denoising SDE, aka denoising diffusions, is an ordinary differential equation whose marginals equal (in distribution) to the denoising SDE's marginals at every time step. Moreover, DDIM ODE is a scaled stochastic process (time-inhomogeneous manner) of the equivalent probability flow ODE.  

More specifically, the authors show that the 2nd-order term in the DDIM ODE's Taylor approximation reduces to two Jacobian-vector products (JVP) plus one time-derivative; One JVP is a product of the Jacobian of the score function (wrt input) and the score function, and the Jacobian and input.

Second, instead of using the finite difference approximation of the Jacobian (of the score wrt input) during inference, it proposes to distill the JVPs (or a sum of two) by another network. The training of the JVP network is via minimizing the mean squared error to the finite difference approximation.
- Note that JVP will have the same dimensionality of the data. This is much lower than some previous approaches to learning higher order scores, which is $d^2$ for $d$-dimensional data.
To reduce the complexity of learning additional networks, the paper proposes a model both predicts score and JVP by sharing feature extractors.

In addition, the paper further introduces higher-order (>2) GENIE solvers.

For the experiments, the paper demonstrates the effectiveness of GENIE on multiple image generation benchmarks.
strengths_and_weaknesses: **Strength**

In my understanding, the paper's contributions are clear, and I also consider that the results are essential for several reasons:

1. The paper proposes a novel higher-order denoising diffusion solver applicable to pre-trained score-based generative models.
2. The paper motivates each part of the proposed method well so that readers can understand how each step contributes to the merits of GENIE.
3. I found that the paper has a well-organized structure that makes it clear to understand the proposed method. Significantly, the paper shares sufficient information to understand the new solver's behaviors, which will help readers understand relevant backgrounds and the proposed method.
    - For example, the first-order method to DDIM ODE is equivalent to the second-order with the "ideal derivative trick". Therefore, the first-order method to DDIM ODE performs better than the first-order method to the original probability flow ODE. 

In general, the paper's contributions wrt the novelty are clear, and the proposed methods are well-defined. In addition, I found that the paper has a well-organized structure that makes it clear to understand the proposed methods. Thus, I'm inclined to accept the paper.
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a higher-order denoising diffusion solver called GENIE to speed up the sampling process for DPM. 

In particular, the idea of the paper is inspired by the fact that DDIM is a first-order discretization of the probability ODE. Naturally, the paper attempts to use higher-order methods to solve the ODE with smaller errors in the same step or equivalently, obtain a good result with much fewer steps.

This paper considers the second-order method in particular. The resulting sampling process requires the second-order score functions under the hood, which is also natural. 

The paper tackles the problem by distilling the terms related to the second-order score functions (approximated by the jacobian of a score network). Namely, learning neural networks to predict the jacobian vector product terms appeared in the sampling process.

The proposed method is compared to training-free methods under FID on several standard benchmarks.
strengths_and_weaknesses: Strengths:

1. At a very high level, I agree that introducing higher-order methods into DPMs can be helpful. 

2. The presentation is clear and the related work is properly discussed.

Weakness:

1. The way to deal with the second-order score functions is not that principled.

2. It would be much better if the authors could provide some theoretical benefits of the higher-order methods.

3. Likelihood results are missing. It is well known that the likelihood and FID are complementary metrics for generative modeling. It is necessary to evaluate all models under both metrics.

4. The baselines in Table 1 are mainly training-free methods. Why not include [69] in Table 1? [69] is more comparable to GENIE since it also trains additional models for distillation.

5. Why GENIE is not that good with NFE=5 in Table 1? I did not see an analysis in depth.
questions: See weakness.
limitations: I did not see a direct negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a high order method to solve the DDM ODE. To overcome the expansive computation of the JVP appeared in the high order term, this paper uses NN to distill the high order term. This paper also uses parameter sharing to avoid to much computation of NN. Experiments show GEINE has a good speed up performance.
strengths_and_weaknesses: Strengths:

Although the idea of high order solver already exists in diffusion models, this work further improves the performance of the high order solver. Therefore, the significance is sufficient in my opinion. Besides, this paper is also clearly written, and sufficient experiments are provided.


Weakness \& Questions:

1. How are the coefficients in Eq.(14) constructed? It does not directly match the coefficients in Eq.(12).

2. The $x_t$ in $d_{\gamma_t} \epsilon_\theta(x_t, t)$ follows the ODE. However, $x_t$ from $d_{\gamma_t} \epsilon_\theta(x_t, t)$ in Eq.(15) comes from $x_t = \alpha_t x_0 + \sigma_t \epsilon$. Does $d_{\gamma_t} \epsilon_\theta(x_t, t)$ in Eq.(15) actually denotes the function in the r.h.s. of Eq.(12), where $x_t$ can be arbitrary. If does, perhaps it would be better to use a new notation for the function in the r.h.s. of Eq.(12).

3. The author has mentioned prior high order solvers are suboptimal. The key difference that makes GENIE perform better can be discussed more. For example, both aiming to solve Eq.(8), why the NN approximation is better than other approximations in prior solvers?

4. I suggest to cite and discuss [1], which also learns an extra head at the top of the UNet to speed up sampling of diffusion models.


[1] Bao et al., Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models

questions:  
limitations:
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors proposed Higher-Order Denoising Diffusion Solvers (GENIE) using truncated Taylor methods,  which significantly accelerates image synthesis. The solver relies on higher-order score functions. To obtain the Jacobian-vector products (JVPs) for the higher-order score function, the authors proposed to extract them from the first-order score network via automatic differentiation and using a small additional head on top of the first-order score network.  The authors provide empirical  results to verify the advantages of the GENIE.
strengths_and_weaknesses: Strengths: The derivation of the higher-order score function using automatic differentiation and small additional head is novel. The empirical results are persuading.

Weakness:  The training scheme for the higher-order score function appears heuristic, which directly minimizes the differences. Given that the score training comes from more mathematical reasoning from the variational lower bound, the suggested heuristic derivation may need more theoretical justification from that aspect.
questions: It would be better to quantify the discretization error of the GENIE compared to the existing higher order methods.
limitations: Still, the slow convergence is considered a limitation.  Aside from the higher-order score function, it would be better to consider other type of acceleration method for the reverse sampling procedure. I
review_title: null

Meta-review: metareview: There is overall consensus that the paper has significant contributions, good experimental validation and a clear presentation. The authors have addressed the majority of concerns raised by the reviewers during the author-reviewer discussions, leading to several scores being raised. I would like to thank the authors and reviewers for actively engaging in discussions. The recommendation is to accept this paper.
confidence: Certain
Decision: Accept