Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on a practical aspect of diffusion-based generative models, i.e., training-free fast guided sampling of DDPM. It is a direct extension of the recently proposed fast unconditional sampler namely DPM-Solver.  The authors identify the unstable sampling trajectory of DPM-Solver for guided sampling, and present several tricks to resolve this issue. These include 1) reformulating the diffusion ODE with data prediction parameterization in lieu of noise prediction; 2) deriving the corresponding ODE higher-order solver leveraging the semi-linear structure of diffusion ODE and exponential integrators; 3) combining the solver with the thresholding trick. The numerical results show these techniques successfully resolve the convergence issue of the DPM-Solver for guided sampling, in both pixel space and latent space settings. 
strength_and_weaknesses: [Pros.]
- Good practical extension of DPM-Solver for guided sampling
- The presentation is clear and easy to follow

[Cons.]
- The technical novelty behind this work is rather incremental. The main techniques used in this paper are directly borrowed from its predecessor [A] (e.g, semi-linear diffusion ODE solver, exponential integrators) without significant deltas. The innovations introduced are the data prediction formulation of the diffusion ODE and the multistep solver, which is good but not surprising. It seems the key to stabilizing sampling is owing to the dynamic thresholding method, which also comes from another paper [B]. As a result, it's hard to evaluate the technical novelty of this paper, the proposed method looks sound but is more of an "engineering" trick
- Could you please explain why the data prediction parameterization yields better results then noise prediction one even without using thresholding?

[A] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022
[B] Photorealistic text-to-image diffusion models with deep language understanding, arxiv 2022



clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: IMO, the ideas presented in this manuscript serve a nice contribution as a journal extension of its predecessor but are not appropriate to be published as another top-tier conference paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work proposes a fast ODE solver for the guided sampling of diffusion probabilistic models with parameterization and thresholding. However, experiments and comparisons are not fair.
strength_and_weaknesses: Strength:

* Authors propose data prediction model from noise prediction model based on parameterization and its corresponding ODEs.

Weakness:

* The algorithm is quite similar to previous works, DEIS and DPM-Solver. The improvement is incremental.

* It would be nice if the author could add a comparison between Algorithm1 and DPM-Solver-2 in the language of the same model and highlight the difference. Because the data prediction model is induced by $x_\theta = (x_t - \sigma_t \epsilon) / \alpha_t$, Algorithm1 should be able to be rewritten as noise prediction model too. 

* Table 1 is biased. First, both DEIS and DPM-Solver can be combined with thresholding easily. Notice that $x_\theta = (x_t - \sigma_t \epsilon) / \alpha_t$, we can infer corresponding "thresholded" $\epsilon$ from "thresholded" $x_\theta$ based on $\epsilon = (x_t - \alpha_t x_\theta) / \sigma_t $.
Second, DEIS can be shown as a high-order method and enjoy high-order convergence. If the reviewer understands it correctly, the analytical form can be obtained by exponential transformation introduced in DEIS~(see sec 4 in DEIS). 

* Therefore, most experiments are not comprehensive and need to show the performance of PNDM/DPM/DEIS with thresholding.
clarity,_quality,_novelty_and_reproducibility: Clarity: ok

Quality: questionable

Novelty: questionable

Reproducibility: ok
summary_of_the_review: The work proposes a fast ODE solver for the guided sampling of diffusion probabilistic models with parameterization and thresholding. However, experiments and comparisons are not fair. I would like to increase the score if authors can address the questions I raised.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: this submission deals with accelerating guided sampling of DPMs. The challenge however is that high-order samplers based on noise prediction become unstable for large guidance steps, and they suffer from test-train mismatch. To address these challenges, this work proposes an ODE solver based on the data prediction model that approximates the integrator of the score with Taylor approximation and calculates the first-order and second-order derivatives. In addition, a clipping is used to keep the solution within the range. Experiments with both pixel and latent space DPMs indicate that it can generate high fidelity images in 15-20 steps. 

strength_and_weaknesses: Strength
- experimental results show significant improvement over DDIM for ImageNet in 15-20 steps

Weakness
- The idea is not principled and this reviewer is confused to find a principled way to think about the generality of the idea for handling instability for distributions beyond the reported experiments. It is not quite clear why switching to a data prediction model avoids the instability. Clipping is also a common method for such scenarios and not surprising.


Questions and comments
- About latent diffusions and experiments with stable diffusions, is there instability observed as in the pixel space? The latent space seems to be smoother and as a result the higher order derivatives of the score could be more bounded. Can the instability still be a serious issue there?

clarity,_quality,_novelty_and_reproducibility: The idea is clearly explained and well supported by the experiments. The combination of applying data prediction model and thresholding also seems to be novel in this context. 
summary_of_the_review: This paper addresses an interesting and timely topic. The contribution however seems limited. It simply switches to the model prediction method and uses the common clipping to avoid instability. None of this seems to be a surprise, and thus the significance and generality of the idea is not clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles fast sampling from diffusion models when using guidance. Guided sampling has become a crucial ingredient in conditional diffusion models to achieve high synthesis quality. However, most existing solvers for accelerated synthesis are not developed specifically with guidance in mind. In fact, the paper shows that some previous higher-order methods for fast sampling perform poorly in the guidance setting. The paper then proposes the DPM-Solver++, a higher-order solver that is based on an exponential integrator framework, similar to previous work. The approach is essentially a direct follow-up of DPM-Solver, tailored to the guidance setting. Compared to previous works, there are two crucial differences: (a) the new solver works with the x-prediction diffusion model parametrization, in contrast to previous works using epsilon-prediction. (b) This enables an easy application of thresholding algorithms to prevent samples from going out of bounds when performing the iterative synthesis process with large guidance weights. Moreover, the paper develops its novel solver both with a single-step framework and a multi-stepping framework.

Experimentally, the novel DPM-Solver++ compares favourably to a variety of baselines on various fast synthesis tasks. The solver is applied on conditional pixel space diffusion models as well as on Stable Diffusion, a latent text-to-image diffusion model.
strength_and_weaknesses: **Strengths:**

- Investigating fast higher-order solvers for diffusion model sampling specifically for the guidance scenario is a good idea and practically very important, due to the wide usage of guidance in the diffusion model literature.
- The shown experimental results are strong and the new sampling approach works well in almost all settings. It usually outperforms all existing methods.

**Weaknesses:**

- The technical novelty of the solver is small. It is a direct follow-up of DPM-Solver, using the same framework. Changing from epsilon- to x-prediction is simple and applying thresholding is also trivial. The multistepping approach has also been used previously in the diffusion modeling literature.

- Why exactly does x-prediction work better than epsilon-prediction (in isolation, without thresholding)? This should be properly discussed.

- The paper points out that other higher-order baselines such as DEIS and DPM-Solver cannot use thresholding in their higher-order forms. I am not sure this is entirely correct. Unless I am misunderstanding something, I would think that also DEIS and DPM-Solver can always take their epsilon prediction, convert into x-prediction via $x_\theta = (x_t - \sigma_t \epsilon_\theta)/\alpha_t$, apply the threshold, and convert back to epsilon prediction (and then just use the corresponding frameworks). I think the paper should analyze these details much more thoroughly and discuss in more detail. I would recommend the authors to lay out in detail how exactly thresholding is applied in DPM-Solver++ in the higher-order scheme and why this shouldn't be possible in the baselines --- because it looks to me like it was possible. Moreover, if the other methods can use thresholding, after all, the experimental baseline comparisons would have to be adapted accordingly.

- The paper writes that it searches over step size schedules and then uses the same schedule in all experiments for all solvers. Maybe I am confusing something, but I am not sure this is an ideal approach. I think using the same step size schedule in all experiments could potentially be unfair to methods that may have been designed with other schedules in mind. I think for every baseline the best possible step size schedule should be chosen. It sounds as if only DEIS and DPM-Solver++ are considered when finding step size schedules. DPM-Solver and PNDM could potentially have other optimal choices.

Minor question: In Figure 1, does DPM-Solver++ use thresholding or not? It would be interesting to add the synthesized image for DPM-Solver++ both with and without thresholding.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is overall well-written and easy to follow. However, I am missing some details about how exactly the thresholding is implemented in DPM-Solver++ and also about why other methods cannot use it (see discussion above). This is currently difficult to follow. Also more discussions on x- vs. epsilon-prediction would make the paper stronger (also see above).

**Quality:** Overall, the paper is of good quality for the most part. It's written well, has thorough experiments, and the work is put nicely into context.

**Novelty:** As discussed in *Weaknesses, point 1* above, the methodological novelty of the paper is rather incremental (see discussion there).

**Reproducibility:** Overall, the paper seems sufficiently easily reproducible. All solver details and hyperparameters are provided.
summary_of_the_review: In summary, the paper studies an important problem, fast sampling from conditional diffusion models using guidance with higher-order solvers. However, I think the claims made around other higher-order methods not being able to apply thresholding are either incorrect, or should be discussed in more detail to avoid confusion. Furthermore, the conceptual novelty of the paper is rather small. Experimentally, the results appear strong, but arguably guidance and thresholding should be applied for all baselines, if possible (see discussion on this above).

In conclusion, I believe the paper is not quite ready yet for publication in its current form. I am willing to increase my paper rating, if my concerns can be addressed and potential confusions resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper extends the DPM-Solver for sampling with classifier-free guidance. Two major changes were made. First, it switches from the epsilon prediction in DPM-Solver to the x0 prediction. Second, it applies the dynamic thresholding trick proposed in the Imagen paper. Experiment results show it outperforms DPM-Solver.

The paper receives four reviews, with three reviewers rating the paper above the bar and one reviewer rating the paper below the bar. The negative reviewers are unhappy with the presented novelty and presentation. The positive reviewer thinks the presented algorithm is useful, and the presented results are convincing.

As there is no consensus, the AC must decide. After reading the paper, reviews, and rebuttal, the AC does not consider the paper a clear rejection. The AC agrees that the presented algorithm is useful and could be widely adopted. However, the AC does not feel the paper presents any surprising results. Both switching to the x0 prediction and using thresholding have been used in prior works. There is no surprise that they should help DPM-Solver improve the quality when applied in the classifier-free guidance setting.
justification_for_why_not_higher_score: Using existing techniques known to work for a problem to a problem with a different solver. It does not contain enough technical novelty to make the cut in my opinion.
Decision: Reject