Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes to speed up diffusion models by progressively distilling a score network with deterministic dynamics. Authors report significant improvement of sample quality using very few iterations. Authors also provide a proof that DDIM is a special numerical integrator for the probability flow ODE, connecting two existing approaches. In addition, empirical studies on the weighting function and parameterization of score functions are also discussed.
main_review: ## Strengths

1. Speeding up diffusion models is an important research direction, and achieving competitive performance with as few as 4 iterations is a remarkable achievement, which I expect to engender major impacts in the community. 
2. Design choices like different parameterization of the score model, and different weighting functions in the score matching loss are clearly discussed. Their impact on model performance is also investigated rigorously in Table 1.
3. I especially like the discussion of DDIM as a numerical integrator for the probability flow ODE, and the experimental study on the performance of different numerical ODE solvers in the appendix.
4. Experiments demonstrate clear advantage over competing methods. It is surprising that even one iteration of the method is already sufficient for generating realistic images.

## Weaknesses

This paper is mostly focused on the variance preserving diffusion process. It is unclear how the distillation approached introduced in this paper can be applied to other SDEs such as VE SDEs and subVP SDEs, as well as how to choose the weighting function and score function parameterization in those cases.
summary_of_the_review: Well written paper with impressive experimental results and clear description of design choices.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an approach to improve sampling speed of diffusion models. The idea is to iteratively learn integrators for the probability flow ODE which, given a diffusion model, maps noise to data samples. In each iteration, the integrator is trained to integrate steps of the ODE over twice the length than the previous model. Thus, the approach resembles an iterative distillation process, where a faster student integrator is learned from a slower teacher integrator. After the student has finished learning, it becomes the teacher for the next distillation iteration.

Enabling this distillation requires changes to the parameterization and loss weighting, for which different, intuitive options are presented. Experiments evaluate the different design choices and compare the resulting models to other recent works on the topic, which demonstrates significant gains of the proposed approach in the regime where a small number of evaluations are used.
main_review: Strengths
- An important topic given that diffusion models achieve very good results but that their sampling time is a barrier to wider adoption in practice.
- The idea of progressively distilling the model seems to be original and useful to avoid the need to evaluate the teacher model for many steps.
- Distillation to very few number of steps requires adaptions on the parameterization and loss weighting. Different plausible choices are presented and evaluated against each other. I particularly liked that this was evaluated on the original diffusion models without distillation to avoid confounding factors and to also provide insight on what works for these models themselves. Knowing that x-prediction works just as well is good to know since it can simplify training quite a bit.
- Overall, there are quite a few design choices for diffusion models which haven't been evaluated exhaustively and many of the results in this paper will help towards a better understanding on these choices (e.g. parameterization, weighting function, using signal-noise-ratio of 0 at t=1, different integrators evaluated in the supplementary).
- It also shows a nice interpretation of DDIM as an integrator of the probability flow ODE.
- Impressive results on retaining sample quality with very few steps, especially compared to previous approaches.
- The overall presentation is very clear, including the background on diffusion models, the proposed approach and difficulties, the algorithm, the results and the discussion of related works which also include very recent works.

Weaknesses
- The effect of distilling iteratively/progressively is not fully explored. Results in Tab. 2 suggest that when distilling to a single step model, the "one-step" distillation approach of Luhman&Luhman, 2021, performs similar to the presented approach. While potential advantages of not having to create a training datasets using the original model are mentioned in the Related Work Section, it would be nice to see a more detailed analysis, which, e.g. compares training costs between the two approaches. It would also be interesting to see the effect of choosing a different schedule for the progressive distillation, for example reducing the number of integration steps by a factor of four instead of two for each new student. This could have effects on both the time required for the distillation process as well as the quality achieved by the distilled models. In general, a bit more details on the time required for the distillation process and comparing that to other approaches might be useful.

Minor
- A bit contrary to what is claimed in the second paragraph of the introduction, [Dhariwal and Nichol, 2021b] also report good results with 25 steps in a class conditional setting (even though I agree with the general sentiment that tasks with less conditioning are more challenging).
summary_of_the_review: Given the impressive results obtained with diffusion models, the paper addresses an important and timely topic. The proposed approach significantly outperforms other methods in terms of quality achieved with distilled diffusion models at small number of evaluation steps, which, ultimately, is one of the main goals on this topic (even though the distillation process might be relatively expensive). The presentation is very clear and easy to follow and different possible design choices are well motivated and intuitive to understand. The different choices are evaluated in well designed experiments and comparisons are very convincing of the benefits of the presented approach and also include very recent works on the topic. The supplementary material also contains interesting material on different formulations and connections between different approaches. Thus, I recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studies the problem of improving the sampling speed of diffusion models, which is currently a timely and challenging topic. To tackle this problem, this work first proposes several new parameterizations of diffusion models for better stability along with a fast sampling, and then proposes a progressive distillation method that progressively distills two DDIM steps of teacher diffusion models to one DDIM step of student diffusion models. In the end, the method can reduce the number of model evaluations in sampling to as small as 4 or 8 steps while retaining high image quality. Experiments on CIFAR10, ImageNet (64x64), LSUN-bedrooms (128x128) and LSUN-Church (128x128) were conducted to show the effectiveness of the progrssive distillation in reducing the sampling time.

main_review: Strengths:

(1) Overall, the presentation is clear and easy to understand, though there exist some minor issues with typos and confusing sentences (see comments below).

(2) The proposed progressive distillation is novel and interesting. Though the distillation idea to reduce the sampling time of diffusion models is not new, doing so in a progressive way seems to be more effective and efficient (which has been clearly discussed in the paper).

(3) The new parameterizations and training losses of diffusion models have been well motivated and explained to accommodate the proposed progressive distillation.  

(4) Experiments have well supported the main claims and the effectiveness of the proposed method in reducing sampling time. In particular, the comparison with previous fast sampling methods on CIFAR-10 demonstrates that it largely outperforms these strong methods regarding FID scores with a few model evaluations. 

Weaknesses:

(1) The progressive distillation process seems to need a much larger computational cost than many previous fast sampling methods, such as DDIM and DDPM respacing. As stated in the paper, its training budget is almost the same as training a diffusion model from scratch. I wonder how this concern can be addressed in practice?

(2) The main claim is that the method can reduce the number of model evaluations in sampling to as small as 4 or 8 steps while retaining high image quality. Currently, the highest resolution of considered datasets is 128x128. I wonder if the claim still holds for datasets with higher resolution? Does the resolution or complexity of the dataset impact the final steps of model evaluations? 

(3) Minor issues in the writing. 1) Typos. For example, in the abstract, “as little as 4 steps” => “as few as 4 steps”. 2) Repeated references. For example, 	“Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis.”, and “Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. ”. 3) Confusing sentences. For example, what do you mean by saying “... unlike the original data point x, since multiple different data points x could conceivably have led to observing noisy data $z_t$”? Also, when saying “we found this to work slightly better than starting from a non-zero signal-to-noise ratio as used by e.g. Ho et al. (2020)”, does it refer to the undistilled sampler or the distilled sampler?
summary_of_the_review: The proposed method is novel and effective in reducing the sample time of diffusion models, and experiments on several datasets well support its effectiveness and better performance than prior fast sampling methods. Still, I have some concerns about the method regarding computational cost and generalization to larger datasets. Thus, my initial rating is a weak accept. I’m willing to increase the score if my concerns can be addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Diffusion models have recently emerged as a promising class of generative models. Their main drawback is the slow, iterative sampling, which often requires hundreds or thousands of neural network calls. The paper proposes a method to progressively distill this slow iterative generation process into much faster synthesis relying on fewer denoising steps, while almost preserving generation quality as measured by metrics quantifying perceptual image quality. The method progressively reduces the number of sampling steps required by a factor of 2 in multiple rounds of distillation and relies on a teacher-student framework. The student model learns to model 2 steps of the teacher in a single step. The method leverages the deterministic synthesis scheme from Denoising Diffusion Implicit Models for that purpose. To get the method to work successfully, new score model parametrizations and score matching loss weightings are proposed. The quantitative results are promising, for example achieving FID scores $\leq3.0$ on CIFAR-10 in as few as 4 steps.
main_review: **Strengths**:
- The paper successfully addresses a crucial downside of generative diffusion models, i.e., their slow sampling speed. This is a problem of very high relevance.
- The idea to use the deterministic DDIM [1] sampling scheme and distill 2 teacher steps into a single student step in each round of distillation is interesting and novel. I think it makes the distillation process easier compared to the situation where we would have to deal with a fully stochastic sampler during distillations. The overall algorithm is relatively straight-forward, which is good.
- The paper demonstrates strong empirical results, outperforming all previous works that aimed specifically to accelerate sampling from diffusion models. In particular, reaching FID scores $\leq3.0$ in as few as 4 steps on CIFAR-10 is a very strong result.

**Weaknesses**:
- The work tackles image generation and aims purely at reaching high FID in as few as possible sampling steps. If our only goal is to reach high perceptual quality as measured by FID with quick sampling, then one could also simply use state-of-the-art GANs [2], which reach FID scores $\leq3.0$ in a single shot (on CIFAR-10). I think that one of the crucial advantages of generative diffusion models is that they reach particularly strong diversity, do not easily drop modes, and better cover the full data distribution in their standard formulation (i.e. no distillation), unlike GANs, which are known to suffer from mode dropping and related challenges. It is unclear whether these advantages of diffusion models are preserved or lost in the proposed distillation process. This aspect is not discussed in the paper and no experiments to investigate this aspect are presented. This suggests the following: (a) Is it possible to somehow measure the likelihood of held-out validation data under the final distilled model? (b) As I assume that this may be difficult, could we calculate standard recall metrics [3] and show that distribution coverage is not significantly decreased in the distillation (and maybe also compare to GANs)? (c) To provide some further evidence towards diversity preservation and mode coverage, we could also run 2D toy experiments on a mixture of Gaussians and show that the distilled model still faithfully covers all modes (standard GANs often tend to miss modes in such settings).
- The explanation why $\epsilon$-prediction is not well-suited for the distillation approach makes sense. However, the proposed solutions, this is, the different proposed model parametrizations and loss weightings, seem to come primarily from trial-and-error and aren't overly well motivated. Is there anything we can say about which parametrizations and loss weightings should be optimal with respect to certain criteria? Can we provide more insights here?
- I sort of agree with the intuition that using the DDIM framework for the distillation protocol makes sense. However, what would happen if we did a similar distillation while using the standard stochastic DDPM sampling? Would this still be possible? A more thorough explanation or an experiment would be very interesting to clearly show that it would not work and provide a more solid motivation for choosing DDIM-based distillation.
- The method is evaluated on image generation only. Will the distillation approach also work for diffusion models in other domains? What could be additional challenges that may potentially occur? In fact, I think it would be very promising to also apply the method on other data types, because in non-image domains there is less "competition" from the GAN literature.

**Additional Questions and Suggestions**:
- In the interest of a clean presentation, I would suggest to properly introduce the $\epsilon$ in equation (2).
- In cection 3: "By inverting a single step of DDIM, we then calculate the value the student model would need to predict in order to step from $z_t$ to $z_{t-1/N}$ in a single step." seems to correspond to the operation at "Teacher $\hat{x}$ target" in algorithm 2. I would consider explaining this in a bit more detail with more background and explicitly derive the expression for $\tilde{x}$ in algorithm 2. I think it could be made clearer and easier to follow and this is one of the most central operations in the proposed algorithm.
- If we distill the slow iterative sampling from diffusion models into these few-step samplers, one may intuitively expect that we pay a price in terms of requiring bigger network capacity to make up for the fewer synthesis steps. This does not seem to be the case, however. Why not? Note that this question is also related to the first point under *Weaknesses* above (i.e., maybe we don't use a bigger model, but have to pay a price with regards to diversity and mode coverage?).


[1] Song et al. "Denoising Diffusion Implicit Models", 2020.

[2] Karras et al. "Training Generative Adversarial Networks with Limited Data", 2020.

[3] Kynkäänniemi et al. "Improved Precision and Recall Metric for Assessing Generative Models", 2019.
summary_of_the_review: The distillation idea in itself isn't entirely novel, as it was proposed by [4], too. However, I certainly acknowledge that this work shows how to do it in an elegant, scalable and high-performance manner using a progressive protocol. Overall, the paper addresses an important problem and shows promising quantitative results. Hence, I am leaning towards recommending acceptance. That said, some weaknesses remain. I would be willing to raise my score, if the weak points were addressed in a satisfactory manner.

[4] Luhmann and Luhmann, "Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed", 2021.
review_title: null

Meta-review: This paper presents a faster sampling method for diffusion based generative models which are usually slow in practice. The key idea is based a progressive distillation approach (e.g., how to distill a 4 step sampler into a 1 step sampler). The paper studies the various design choices for diffusion models which existing work hasn't looked at that deeply and sheds light on the effects of these choices. The paper also shows that DDIM can be seen as a numerical integrator for probability flow ODE. The experimental results are impressive. 

There were some concerns such as the effect of progressive distillation and the overhead of distilling the diffusion model but the authors provided a satisfactory response and backed it up with additional results.

Overall, this is a nice paper on making diffusion based generative models generate faster samples and also provides novel insights into the behavior of these models under various design choices. Given the significant recent interest in these models which are pretty impressive in terms of generation quality but slow, the paper indeed makes a timely contribution which will fuel further interest in these models.  All the reviewers have voted for acceptance. Based on my own reading, the reviewers' assessments, the discussions, and the authors' response, I would vote for acceptance.
Decision: Accept (Spotlight)