Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors present efficient integrator for diffusion models based on polynomial interpolation of the noise during trajectory estimation. 
strength_and_weaknesses: Strengths:
1. Idea somewhat interesting from theoretical stand point
2. Detailed motivation for ODE case why this should work

Weakness:
1. It's not clear why you mention application to SDE case as you mention that your method do not work for integrating SDE equation
2. Notation is somewhat overcomplicated for no reason, e.g. \Phi(s, t) -- why you need to mention hard matrix ODE as in your case it is always scalar multiplied by identity matrix? 
clarity,_quality,_novelty_and_reproducibility: The paper seems notationally and mathematically overcomplicated, idea is novel. I did not see link to the code whilst the method seems highly hard to replicate just from description as requires accurate invocation of ODE solvers.
summary_of_the_review: The paper should be simplified and made more clear, at this point it is too hard to read due to unnecessary mathematical overload and the reproducibility is under question.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a fast sampling method, termed as diffusion exponential integrator (DEIS), for diffusion models with a new discretization of the reverse process. In particular, this work first investigates the existing ODE solvers for diffusion models and found that reducing the discretization error is crucial for fast sampling. This work then proposes a new discretization that is mainly based on the exponential integrator (EI) and polynomial extrapolation of the noise prediction function. Besides, this work shows that DDIM is a special case of DEIS. Experiments show that the proposed method can achieve 4.17 FID with 10 function evaluations on CIFAR-10, outperforming other ODE discretization based methods. 

strength_and_weaknesses: Strengths:
- The proposed idea from the exponential integrator perspective is interesting and novel to me. The motivation of reducing the discretization error using polynomial extrapolation is sound and intuitive. 
- This work provides a good insight into the understanding of DDIM from the numerical discretization perspective. That is, applying EI into the noise prediction reparameterization in VPSDE without the polynomial extrapolation becomes the DDIM sampling.
- Experimental results on CIFAR-10 show the effectiveness of the proposed method: the best variant of DEIS largely outperforms previous ODE-based training-free methods (DDIM, A-DDIM, PNDM) when the NFE is less than 20.

Weaknesses:
- The presentation of the proposed method is not clear. I understand that this work tries to make the main idea well-motivated and general, but I think many design choices that do not work well can be moved to the appendix, such that we understand the core idea more easily. Below are some details: 1) if EI works better than Euler with the noise prediction reparameterization (while it works worse than Euler with score prediction reparameterization), we can present the idea with the noise prediction reparameterization, which has been commonly used in most diffusion models for images. The analysis for different network reparameterizations can be deferred to the appendix. 2) I don’t fully understand why we need Proposition 3. In other words, why do we need to transform Eq. (10) into “a simple non-stiff ODE”, if the resulting $\rho$AB-DEIS is no better than $t$AB-DEIS (Table 2)? Also, I don’t fully understand why it is a good thing to apply generic ODE solvers without “worrying about the semi-linear structure”. 3) If DEIS does not work well for SDE, we can defer most details in Section 5 to the appendix. 4) I’m not exactly sure what “+optimizing timesteps” in Figure 5 means. It seems that the main text doesn’t directly explain its meaning.
- The experiment settings and results are also confusing. Below are some details: 1) It mentions that the proposed method compares with other training-free baselines (DDIM, A-DDIM, PNDM) on several datasets. However, in the main text, we don’t see the results of A-DDIM and PNDM at all, and there is no quantitative result on other datasets except CIFAR-10. I suggest presenting these main experimental results in a more compact and coherent way instead of deferring most of them to the appendix. 2) In the experimental setting, I think several more recent training-free baselines (e.g. [1,2]) are missing. I noticed that the authors discussed their relationship with the proposed methods in the appendix. I would prefer to discuss these relevant baselines and show quantitative comparison results in the main text. 3) I appreciate that the authors considered ImageNet as one of the datasets to evaluate the proposed method. But I think ImageNet 32x32 is a bit low resolution and difficult to visually see the generation quality. I suggest similar to many previous works (e.g., [2,3]) that consider ImageNet with a resolution of at least 64x64 for obtaining both quantitative and qualitative results.
- Minor things: In section 2, when describing the conditional marginal distribution $p_{0t}(x_t|x_0)$, what does $\mu_t$ mean? I suggest replacing the term “Diffusion model” with “diffusion model”.


[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.
[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.
[3] Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022.

clarity,_quality,_novelty_and_reproducibility: I liked the idea of using EI and polynomial extrapolation for the fast sampling in the ODE version of diffusion models, which is novel and original to me. However, I think the presentations of the idea and experiment results should be largely improved. Lastly, I think the work provided sufficient details for reproducing the main results.
summary_of_the_review: Overall, I think the idea is interesting and useful, but the idea and experimental results are poorly presented, which makes me not sure how it compares with previous relevant methods. My initial suggestion is “leaning to reject” but I’m open to adjusting my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a sampler for diffusion models based on the exponential integrator and the semi-linear structure of the probability flow ODE. The authors demonstrate that this sampler maintains strong performance for low numbers of function evaluations.   
strength_and_weaknesses: **Strengths** 
The formulation of the solver is relatively well-structured and motivated. The three ingredients proposed in section 3.2 and the related ablation of each on the toy problem is neat. The explanation of DDIM in terms of a first order method for the semi-linear ODE is nice, although a similar argument was already made in DPM-Solver (Lu et al) (although I acknowledge it was concurrent work, more discussion below). The reparameterization in section 4 removing the linear term in section 4 is also neat, but again the discussion overlaps with Karras et al (also discussed more below). The experimental results, primarily Table 2, are convincing, especially showing the worth of the higher order methods. 

**Weaknesses** 
The experiments section is quite concise. There is quite a lot of additional content in appendices G and H. It would be good to compare to more methods in the main paper (as well as the concurrent work discussed below) and not relegate this to the appendix. It would also be good to expand on the use of the proposed sampler for pretrained models like Stable Diffusion beyond qualitative samples -- showing improved performance here over competitive approaches would be a strong result. 

Related to iPNDM: 'We further propose Improved PNDM (iPNDM) that uses lower order multistep methods for the first few steps in sampling to avoid expensive warming start.' Isn't it the case that for linear multistep methods the expensive warm start is necessary? i.e. if you don't warm start an order K methods with steps of at least order (K - 1), the resulting method is bottlenecked by the lowest order used for the warm start? 

**Nits**
- Intro: 'DM is not hyperparameter sensitive' - maybe less sensitive than GANs, otherwise this is too strong a statement.
- Intro: 'extremely slow sampling' - again, 'extreme' is probably too strong a statement, compared with e.g. AR sampling 
- Figure 2: the vertical-axis isn't labeled 
- 7: 'DEIS also works amazingly in practice' - again too strong 
clarity,_quality,_novelty_and_reproducibility: **Clarity**
I find the use of matrix notation throughout makes the paper more difficult to understand. Moreover, the matrix notation is never actually needed, since in Table 1, all of these coefficients are multiples of the identity.  

It's unclear to me why you need the general form in eq 4. with lambda nonzero. In Section 5, you mention discretizing this with lambda nonzero 'does not work well in practice', and 'We do not pursue the discretization of the SDE Eq. (4) further in this paper and leave it for future'. Why include it at all then? You also include motivation as to why we should only consider the lambda = 0 case ('From this perspective, the ODE with λ = 0 is the best option since it minimizes the weight of nonlinear term.'), but why not make this case earlier and just consider the simplified version of eq. 4? 

Overall there is some difficulty parsing the paper. This includes some awkward phrasing (e.g. abstract: 'a fast sampling method for DMs with a much less number of steps' ->  'a fast sampling method for DMs with fewer steps') and some sentences which don't make sense (e.g. 4: 'Absorbing useful structures such as semi-linear structures through a transformation so that the transformed ODE does not possess any useful structure that one can utilize to improve generic ODE methods.'). It would be good to revise some of this. 

The polynomial extrapolation and associated coefficient computation in Ingredient 3 (eqs 14 and 15) might benefit from an explicit algorithm description i.e. to make clear that the coefficients are computed once in advance when a particular discretization is chosen (if I understand correctly).  

**Originality**
The paper has a lot in common with DPM-Solver https://arxiv.org/abs/2206.00927 (Lu et al), as you've noted in Appendix B. I appreciate that this is concurrent work, but I feel this comparison should be made in the main body of the paper, rather than in the appendix, especially since you claim these methods are special cases of your approach. It would be useful to tease out the exact correspondence to this paper, and explain the choices in https://arxiv.org/abs/2206.00364 (Karras et al) in terms of your formulation, as well as compare to both empirically. 
summary_of_the_review: Overall I think this is a well-motivated paper with convincing empirical results. A more thorough discussion and comparison of competing methods, as well as some restructuring to move results from the appendix would improve the paper further. 

Edit post-rebuttal: I'd like to thank the authors for their response. Some of the main criticism of the paper concerns concurrent work, which I feel the authors have attempted to address. Both the concurrent work and this paper constitute useful contributions, and I think this paper also deserves to appear in proceedings, even if the overall presentation is perhaps less clear. I've raised my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper targets at improving the sampling efficiency of diffusion model via a semi-linear ODE methods, termed as Diffusion Exponential Integrator Sampler (DEIS). The paper is well motivated with experiment observations that the changes of noise predicted by the neural network is small for most of the sampling steps as shown in Fig. 4a in the paper. Thus the values can be reused. The author also explores a variety of functions by measuring the  discretization error. The author finally proposes a principled guideline on achieving fast sampling of DM models with High order polynomial and . The experiments on CIFAR-10 and ImageNet with image size downsampled to 32x32 support the main claims of the paper.
strength_and_weaknesses: Strengths:
1. The paper is well written. The flow is easy to follow and the contributions of the paper is clearly presented.
2. The background knowledge on diffusion model is well presented and the readers can easily get the main claims of the paper, which is the sampling efficiency.
3. I like the analysis conducted by the authors. The claims are supported by the experiments results.

Weakness:
1. The paper only conducted experiment on easy & object centric datasets: CIFAR and ImageNet with 32x32 size. However, the main advantages of DM are on complex scenes where the generated images contain multiple objects. Besides, one of the well known disadvantages of DM is the high resolution images. Could the proposed algorithm achieve promising results on applied to the above mentioned cases?

2. In Appendix B, authors claim that DPM-Solver is a concurrent work. The authors claim that DPM-solver is a special case of DEIS. So, how about the comparison with DPM-Solver? Does the more general exploration of DEIS outperform DPM-Solver?

3. Recent popular LDM models use PLMS solver and achieves even faster sampling speed. How the proposed DEIS compared to PLMS sampler?
clarity,_quality,_novelty_and_reproducibility: The work is well presented. The quality is good in terms of analysis and writing. The paper is clean and detailed enough for reproducibility. However, there are a few of concurrent works such as DPM-Solver and PLMS sampler are not compared in the current version.
summary_of_the_review: Overall, I like the paper's style and I think the authors have put in enough efforts on this work. However, the advantages of the proposed methods are not clean enough. The experiments are relatively weak in terms of both dataset and selected baselines. Please refer to the `strength & weakness' section for the details. Nevertheless, the paper have put in enough analysis that reveals some of the properties of DM. I would vote for board-line accept for this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Ratings: 5/6/6/6.
Confidences: 2/3/4/3.
Recommendation: Accept.

The article discusses a new method for speeding up the slow sampling procedure in Diffusion Models (DMs), called the Diffusion Exponential Integrator Sampler (DEIS). The DEIS method uses the Exponential Integrator for discretizing ordinary differential equations and leverages the semilinear structure of the learned diffusion process to reduce the discretization error. The proposed method can generate high-fidelity samples in as few as 10 steps and outperforms previous methods in terms of sample quality and speed.

Strengths: (1) the proposed idea is novel and well-motivated. (2) The work provides good insight into the understanding of DDIM from a numerical discretization perspective. (3) Experimental results show the effectiveness of the proposed method.

Weaknesses: (1) The presentation of the proposed method is not very clear (2) The experiment settings and results are confusing and do not include recent training-free baselines. (3) Minor things, such as a lack of clarity in certain descriptions and terminology.

The authors wrote a comprehensive rebuttal, and after some discussion, one of the reviewers raised his score from 5 to 6.

Overall, I do recommend to accept this paper.
justification_for_why_not_higher_score: See Weaknesses.
justification_for_why_not_lower_score: The authors wrote a strong rebuttal and did a substantial amount of work improving the paper based on the reviewer comments, adding larger-scale experiments and improving their writing. The appendix of this paper is also very detailed. It's obvious the authors put in a lot of time.
Decision: Accept: poster