Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on accelerating guided diffusion sampling, where the backward ODE to be solved consists of two parts: the first one from the diffusion contribution and the second one from the classifier contribution. It is empirically found that non-splitting high-order ODE solvers does not work well when the number of time steps is small. To address the above issue, the authors consider applying splitting methods (LTSP 1959 and Strange splitting 1968) for solving the backward ODE, where the two parts in the ODE are treated separately and sequentially.  
strength_and_weaknesses: Strength: 
1. The paper, for the first time, applies existing splitting methods (LTSP 1959 and Strange splitting 1968) for solving the backward ODE in guided diffusion sampling and obtain promising sampling results, which I think is interesting and novel.
2. The literature over existing high order methods seems up-to-date.   

Weaknesses: 
It is unclear from a theoretical point of view why non-splitting high-order methods do not work well. There may exist other high-order methods that work which yet to be discovered. 



 

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is easy to follow.  The newly designed sampling methods LTSP and STSP sound reasonable, where the ODE term from the classifier contribution is solved by a first order method and the ODE term from the diffusion contribution is solved by a high-order method.

There are a few inconsistent statements:
1. On page 4 of experiments, it says DDIM at 1000 steps are taken as reference solutions  while in Figure 2 and 3 later on, DDIM of 250 steps are considered.      
2. On page 6, it says condition subproblem (Equation 10)... diffusion subproblem (Equation 11), which I believe is a mistake. 

I think it is more convincing to replace LPIPS in Figure 2 and 3 with FID, which most readers are more familiar with. 
summary_of_the_review: The paper tried out splitting methods for solving the backward ODE in guided diffusion sampling and obtained better performance. No theoretical justification is provided to motivate the splitting methods over non-splitting methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes classifier-guided sampling of diffusion models and observes that applying higher-order methods for accelerated sampling from the model does not work well in the guidance scenario. The root cause for that is that the additional guidance term defined by the classifier makes the generative ODE harder to solve. Explicit higher-order integrators do not seem to be suitable for the resulting stiff classifier-guided ODE. Based on that observation, the work proposes to split the ODE into two separate terms, the diffusion model's score term and the classifier term. These are then solved separately with different integrators and afterwards combined, using numerical splitting methods. The paper shows improved performance of classifier-guided diffusion model sampling over some selected baselines when using a limited number of synthesis steps.
strength_and_weaknesses: **Strengths:**

- The idea to split the diffusion model's main score and classifier term in the generative ODE into separate parts, and solving them separately, is novel and certainly seems like a good and sensible idea.

- The analyses that are performed in Figure 2 are interesting and insightful. They nicely show that the additional classifier term is the problem that makes higher-order methods break down in the classifier-guidance setting.

- The experimental results on classifier-guided diffusion model sampling support the value of the proposed method.

**Weaknesses:**

- Most state-of-the-art conditional diffusion models these days rely on classifier-free guidance for conditional sampling. Unfortunately, the proposed methodology only applies to classifier guidance. This reduces the significance of the proposed method.

- The baseline comparisons are insufficient. Comparisons to the recent state-of-the-art solvers DEIS [1] and DPM solver [2] are missing. Furthermore, for the experiments on inpainting, colorization and super-resolution, only qualitative comparisons are presented. Also, only DDIM is considered there for comparison.

- The paper lacks a proper theoretical analyses of the proposed methods. Convergence, order and stability of the Lie-Trotter Splitting and Strang Splitting techniques are nowhere discussed. It is insufficient to just write sentences like "This method can be proved to have better accuracy". Moreover, I believe the deep learning community is not deeply familiar with these splitting schemes, so a much more thorough discussion and analysis is needed.

- The statement that "no prior diffusion work uses splitting numerical methods" is incorrect. Critically-damped Langevin Diffusion [3] uses Strang Splitting to develop a sampler for their setting (see their Section 3.3). This work is not cited or discussed at all.


[1] Zhang and Chen, Fast Sampling of Diffusion Models with Exponential Integrator, 2022

[2] Lu et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, 2022

[3] Dockhorn et al., Score-Based Generative Modeling with Critically-Damped Langevin Diffusion, 2021
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is easy to read and follow.

**Quality:** The paper is of mediocre quality. As discussed above, baseline comparisons are insufficient, some relevant work is not cited or discussed, and the mathematical analyses of the proposed integration scheme are lacking.

**Novelty:** The method is novel. It is a sensible approach for classifier-guided sampling, but the overall significance is somewhat limited, because the other important setting of classifier-free guidance is not covered.

**Reproducibility:** There are no concerns with respect to reproducibility. The submission also includes code.

(Small note: Eq. (4) has a typo. I believe it should be $e_4$ on the far right)
summary_of_the_review: In summary, this paper proposes some interesting and sensible ideas to improve classifier-guided diffusion model sampling. However, the paper is lacking on multiple fronts (see weaknesses above). (a) Baseline comparisons are insufficient. (b) The proposed splitting methods lack theoretical discussion and are only very superficially discussed in the paper. (c) The significance is somewhat limited because only classifier guidance, but not classifier-free guidance is covered.

Therefore, overall I do not think that this paper is ready for publication in its current form. Most importantly, I would suggest the authors to include more thorough experiments and a more in-depth analysis and discussion of the proposed splitting integration methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an acceleration method of guided diffusion sampling based on splitting numerical methods. Based on the finding that the high-order numerical methods are unsuitable for the conditional function, it develops a method based on Strang splitting and a combination of fourth and first-order numerical methods. Experimental results show that the proposed can accelerate the guided diffusion sampling.
strength_and_weaknesses: Strength:

The paper proposes a simple method that accelerates the guided diffusion sampling. The proposed method is efficient and can be applied to other problems, e.g., super-resolution, colorization, for acceleration. Experimental analysis shows the effectiveness of the proposed method.

Weaknesses:

I mainly concern the limited theoretical analysis about the proposed method.

The paper claims that the high-order numerical methods are unsuitable for the conditional function. However, it does not explain this finding clearly. It would be better to provide a theoretical analysis of this finding.

The paper claims that the Strange splitting algorithm can be proved to have better accuracy. However, there is no theoretical analysis.
clarity,_quality,_novelty_and_reproducibility: The paper is well written.
summary_of_the_review: Although the experimental results are good, I expect the authors could provide more theoretical analysis about the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of accelerating the sampling process of the classifier-guided diffusion models. The classifier-guided diffusion consists of two components: the conditional function (gradient of classifier) and the standard diffusion term. The authors observe that high-order numerical solvers can lead to degraded performance when sampling with the conditional function. To alleviate the issue while speed up the sampling, they opt to use higher-order method for standard diffusion term and forward Euler method for conditional function. They combine the two interleaved procedures through operator splitting methods. Experimentally, the proposed sampling method outperforms DDIM in speed and sample quality.
strength_and_weaknesses: ### Strong points

- The paper applies the operator splitting method to the classifier-guided generation. They study various combinations of solvers of the individual parts, and the best combination outperforms the baseline DDIM.

- The proposed method consistently improved over the baselines across image resolutions and tasks.



### Weak points

- The current dominating method is classifier-free generation, such as stable diffusion. The problem of classifier-guided diffusion models seems to have lower practical value. As the authors mention in Section 5, classifier-free involves more expensive function evaluation per step. However, they allow higher-order solvers for all components. Could the authors compare the FID/sampling time of the two methods, in order to demonstrate the utility of classifier-guided generation. (I understand it could be hard to complete the experiments in the short rebuttal time, but it really boosts the utility of classifier-guided generation if it has certain advantages.)

- The "stiffness" reasoning of the condition function is vague and insufficient. The phenomenon is the major point in the paper, and it would be very helpful to dive deeper into this problem: why does the condition function perform better when paired with a simpler ODE solver in practice? It's also a bit counter-intuitive when the function is stiff for higher-order solvers but not for simpler ones. Could the authors give some illustrative examples? The review has one plausible hypothesis: the gradient of classifier (condition function) has large variance or changes rapidly across different $\sigma$. Hence, it's harmful to combine the evaluations along the ODE trajectory.

- The evaluation protocol in Section 4.1 is a bit problematic. The generated samples are compared against DDIM w/ 1000 steps. It assumes that the samples generated by DDIM w/ 1000 steps are true samples. As we see in hindsight, there are many samplers better than DDIM, and they could generate better samples while incurring high LPIPS. Also, in this setup, the goal is to purely decrease the ODE simulation error (there is nothing to do with network estimation error). It's counter-intuitive that higher-order methods do not work out under such a metric.


### Writing suggestions

Below are some points that could potentially make the paper more readable and consistent:

- Section 3.1 & 3.2: the $\sigma$_s is increasing over time. But normally, they should decrease over time during sampling. 
- Section 4, page 5, "with 1000 total steps" - do you mean "with 1000 total sampling steps"? 
- Section 4.1: It would be helpful to change PLMS1 to Euler's method.




clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written. The demonstrated phenomenon of the paper remains mysterious. It would be helpful to elaborate on the behavior of the condition function. Intuitively, since we want to minimize the discretization error of an ODE, a higher order method could lead to better performance. 
summary_of_the_review: The reviewer leans toward rejection due to the lack of justification of the proposed method, and the seemingly low practical value of the considered (classifier-guidance) problem.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper reports that high-order ODE solvers for diffusion models do not work well when classifier guidance is used. It argues that this is due to the gradients from the score models and those from the classifiers having very different perspectives. To address the issue, the paper proposes to use the split numerical methods in the ODE literature to solve the issue. Experiment results on limited scenarios verify the effectiveness of the proposed method.

The paper receives 4 reviews. Initially, three reviewers considered the paper slightly above the bar, and one reviewer considered it slightly below the bar. Generally, the reviewers feel the proposed approach makes sense, but several were not convinced that classifier guidance is important as classifier-free guidance is now the dominant approach. The authors rebut the idea with evidence that classifier guidance remains relevant for the diffusion model community. In the end, the authors successfully convince the negative reviewer and reach a consensus above the bar evaluation. 

After reading through the paper, reviews, and rebuttal, the area chair agrees with the reviewer's assessment. The paper does have merits to be presented at the ICLR conference despite the fact that the presentation was a bit below the bar. The authors are encouraged to incorporate the review feedback and strength the paper for the final version.
justification_for_why_not_higher_score: The presentation of the paper could be further improved. The experimental validation can be strengthened.
justification_for_why_not_lower_score: The split numerical method is new to the diffusion model community and seems to be useful. We would like to make sure researchers in the field know such tools to help them conduct better reserch.
Decision: Accept: poster