Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new numerical solver for stochastic differential equations and demonstrated significant improvement over existing ones such as the Euler-Maruyama solver in terms of the quality/computation trade-off for score-based generative modeling with SDEs.
main_review: ## Strengths

1. This paper proposes the first adaptive step-size numerical SDE solver for score-based generative modeling. Experimental results demonstrate clear improvement over previous methods like Euler-Maruyama methods and probability flow ODEs. It also demonstrates improvement over DDIM for a moderate budget of iteration numbers.

2. The method itself is simple and clear, and only requires tuning one hyperparameter. 

## Weaknesses

1. No theoretical understanding of the proposed numerical SDE solver. It would be better to include an analyze on the convergence order of the proposed approach.

2. Results of predictor-corrector for VP SDEs in Table 1 are significantly worse than results reported in the original score sde paper. What could be the reason for this? Did you use the correct signal-to-noise ratio for the Langevin corrector?

3. Some minor writing issues. The first page has a footnote "equal contribution", while all author names should be anonymized. In section 3.1.3, $E_q$ is defined to be a scalar (the $L_q$ norm), but it is referred as a vector in the expression for computing $\| x\|_2 $.

4. In section 3.1.5, it is mentioned that a different step size is applied to each data sample. How is this implemented in deep learning frameworks? Will it cause noticeable slowdown on GPUs?
summary_of_the_review: This paper proposes a simple and effective numerical SDE integrator for score-based generative modeling based on SDEs. Paper can be stronger with a deeper theoretical understanding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a number of treatments to speed up and improve the generation quality in the process of the reverse diffusion process, diffusion-based generative models. This paper is mostly empirical by nature, and authors suggested five techniques in improving the SDE solver performance.
main_review: 1)
I understand the paper's nature lies in the empirical side. However, some claims can be further elaborated through formal analyses. For example, Norm calculation in 3.1.3 can be analyzed by following algorithmic complexity, i.e. big-oh notations. I suggest that authors find such analysis opportunity throughout the paper. Moreover, it would be great if you can come up with a table to organize such complexity comparisons.

2)
I cannot follow why Eq 5 could be better in both quality and speed at the same time. Eq 5 has more constraints to setup the tolerance, so the speed-up seems to be an obvious benefit. However, the quality could be damaged while I also note that the quality is not being too much hurt from your result report. Why is that?

This question goes same to utilizing L2 norm, instead of L-$\infty$ norm

3)
Is there any theoretic argument on the integration method? Which can be proved or contests as a proposition or a theorem?
I could not fully comprehend the argument in the current text, and it just looks like swapping SDE solvers to find the best match.

4)
These improvements can be further investigated through ablation studies, which seems to be must-do, in my opinion. Without the ablation study, I cannot argue which technique contributed to more or less.
summary_of_the_review: Good paper with interesting and essential ideas in improving the usability of diffusion models. However, the current experimental result and the method justification are weak.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Score-based/diffusion-based generative models can achieve high sample quality. However, their sampling speed is slow due to the large number of evaluations required by numerical SDE solvers. This works aims to accelerate the sampling process by using a more efficient SDE solver. The proposed approach generates data 2 to 10 times faster than the baselines while achieving reasonably well sample qualities.
main_review: 1. The major contribution of the paper seems to be an application of existing numerical solvers (Roberts, 2012) to sampling from SDE models. The technical novelty could be limited as the solver already exists despite the tricks used to improve the empirical results. 
2. There is no theoretical analysis of the method.
3. It seems that the major contribution is on the empirical side. However, the empirical results are not very impressive especially when NFE is small. For instance, in Table 1, DDIM outperforms the proposed method when NFE=49. As the goal of the paper is to improve the sampling speed of score-based models, having a good result when NFE is small enough is important. An FID of 72.29 using 49 NFE might not be impressive since NFE=49 is already a large budget. Wondering when the proposed method can outperform DDIM when NFE=100.
4. There are some formatting issues (e.g., too much space) on the first page. In the tables, some best values are in bold while some are not. It would be good to be consistent.
5. Algorithm 1 should be better explained in section 3.2. The required parameters and their selections should also be explained more clearly.
6. The paper writing can be improved. For instance, in section 3.1.1, "the stochastic Improved Euler’s method" should be explained more clearly.  "Dynamic step size algorithm" is an important contribution of the paper, the setting should be better explained. If possible, can move some details from the appendix to the main paper.
7. It would also be interesting to perform experiments on even higher-dimensional images if model checkpoints are available.
summary_of_the_review: The major contribution of the paper seems to be an application of existing numerical solvers to sampling from SDE models. The novelty could be limited as the solver already exists (despite the tricks proposed to improve empirical results). Given that the theoretical contribution is limited, the major contribution would be on the empirical side. However, the empirical results are not impressive enough: when using a small number of sampling steps, the proposed method has much worse results compared to DDIM. As the goal of the paper is to efficiently sample from SDE models, having strong performance when the sampling steps are small is important.

The paper writing can be improved. There are some formatting issues.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new SDE solver for the reverse process in score-based models. The algorithm is fast and offers high quality, and avoids some hyperparameter tuning. There is theoretical analysis on the stability and bias of the algorithm. The paper also has experiments comparing the proposed algorithm to several baseline methods. 
main_review: The research problem of this paper is significant and interesting to the community of deep generative models, especially score-based models and denoising diffusion probabilistic models. The writing of this paper is clear. 

Strengths:
- A novel SDE solver with a stability and bias analysis.
- The empirical success of the algorithm, i.e., high generation quality with less steps (function evaluations). 
- The algorithm can readily be adapted to other score-based models with SDE. 

I am not an expert in numerical solvers for differential equations, so I will leave evaluation of this part to other reviewers.

Weaknesses (and questions):
- One major concern is about the title. The title is too exaggerated and does not reflect the content of the paper. Instead, I would like to see a title that is precise, for example, "An efficient SDE solver for score-based models". 
- Fig 1: What about the original Improved Euler baseline? 
- Section 3: The several components of the algorithm seem to be parameter tuning and engineering trick. There is no theoretical or even intuitive justification for why they offer better performance (Appendix F does not answer this question). In addition, I do not see any ablation study on the contribution of each component. 
- Section 3.1.3: The last equation ($\ell_2$ norm) in this subsection cannot be understood.
- The algorithm: Although the algorithm does not need stepsize or scheduler tuning, there are other hyperparameters involved so some tuning is still needed.
- Experiments: How often do you reject under different NFE? There should be experiments on the relationship between NFE and percentage of rejection. Next, there should be a figure similar to Fig 1 but the x-axis is the time spent rather than NFE. 
- Appendix A: Are those methods $\ell_2$ or $\ell_{\infty}$? 
- Appendix E: What can we read from this table?
summary_of_the_review: The algorithm seems to be empirically successful. However, the paper lacks justification for why their algorithm can be better. There is also much space for improvements in the experiments (especially, on the overall generation speed measured by seconds). In addition, the title is exaggerated. 

---------
Updates after the rebuttal

I appreciate that the authors agreed to use a more precise title. Some of my concerns are also addressed. Therefore, I will increase my score to 5. However, I still think the experiments lack some key components such as time spent and number of rejections, although they might strongly correlate to NFE. The empirical results are good but not impressive enough. I encourage the authors to further improve the method and make additional theoretical/empirical justifications.
review_title: null

Meta-review: The paper proposes numerical method for solving SDEs that empirically are faster than previous approaches. Two reviewers felt the paper was above threshold, while two felt it was below threshold for acceptance. While the paper is borderline in this sense, all four reviewers noted that the paper lacked a theoretical justification and rested on empirical evidence for the usefulness of the approach. Several reviewers also pointed out that these empirical results are on the weak side. While the paper may add a potentially useful learning trick to the optimization literature, these two significant concerns put it on the side of a borderline reject.
Decision: Reject