Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims at reducing the number of discretization steps for the generation process of diffusion-based generative models. To do that, the paper proposes a new probabilistic generative model, which exploits pre-trained denoising diffusion probabilistic models. In particular, similarly to Denoising Diffusion Implicit Models (DDIM, Song et al. (2020)), the paper proposes a non-markovian hierarchical generative model, whose densities are defined as;
- $p(x_0, ..., x_T) = \prod_{t =0}^{T} p(x_{t} | x_{>t})$, where $p(x_T)$ is a prior distribution such as a standard Normal distribution.
- $p(x_{t} | x_{>t}) = q_{\mu, \sigma}(x_{t} |x_{>t}, x_0 = \hat{x}(x_{t}, t) )$
- $q_{\mu, \sigma}(x_{t} |x_{>t}, x_0) = N(x_t | \Sigma_{u = t+1}^{T} \mu_{t, u} x_u, \sigma_t^2 I_d)$
- $\hat{x}(x_t, t)$ is a reconstruction network similar to the one in the DDIM.
- Note that the proposed method employs $q_{\mu, \sigma}(x_{t} |x_{>t}, x_0))$, while DDIM uses conditionally Markov $q_{\mu, \sigma}(x_{t} |x_{s}, x_0)$ for any $t < s$.

Noting that the discrete-time diffusion-based generative models are fully differentiable, the paper proposes to train the models while the pre-trained reconstruction network is fixed; here, the authors name this training as Differentiable Diffusion Sampler Search (DDSS). To train the models, the paper first emphasizes that maximizing the ELBO of the proposed method deteriorates the performance wrt FID. Then the authors instead propose to minimize perception scores between models' sample and training data. Here, the perception score is a squared distance between empirical means of two sets of samples on the feature space of a pre-trained classifier.

Moreover, observing that training the entire discrete-time diffusion-based generative models requires huge memory costs, the paper proposes to use gradient rematerialization (Kumar et al., 2019b) methods.

In the experiments, the paper tests the effectiveness of the proposed method in image generation quality benchmarks, such as CIFAR-10. Specifically, the analysis of the experiments focuses on two perspectives. First, the authors compare the generation qualities of the proposed method and the baseline methods, including denoising diffusion probabilistic models (DDPM), in the few-step regime. Second, the paper compares the generation qualities of the DDIM and the proposed model when both are trained via DDSS. The paper claims that the proposed generative models trained via DDSM outperform the previous methods in the few-step regime.

===== POST-REBUTTAL COMMENTS ========  
In the updated version, Theorem 1 shows that under certain choices of model parameterizations, the proposed models' ELBOs will be the same as the DDPMs'. This clarifies that it is valid to plug-in pre-trained DDPM models (or other similar types), potentially without fine-tuning; therefore, it seems valid to treat the proposed method as improving sampling process. 
main_review: **The strengths of the paper**  
In general, I found that the paper provides interesting values to the ML community:

1. It tackles an important problem in diffusion-based generative models, i.e., preserving the qualities of the generated samples with fewer discretization steps.
2. The paper proposes a new method using non-Markovian hierarchical generative models with the pre-trained diffusion-based generative models.
3. The paper discusses practical techniques to support the scalability of the proposed method.


**The weaknesses of the paper**  
While the novelty of the proposed method, I found that the following aspects can be improved.

First of all, the motivation of the proposed model needs to be clarified, meaning the authors need to discuss the optimality of the non-Markovian but simple Gaussian hierarchical generative models and the proposed posteriors. I also consider this issue connects to the insufficient justification of minimizing the conceptual loss instead of maximizing the ELBO.
    
- For example, suppose the proposed model performs significantly worse when it is trained via maximizing ELBOs. In that case, it implies that the proposed posteriors are not well suited for the proposed generative models. This means the ELBO can not be tightened regardless of the number of discretization steps. Consequently, the pair of the proposed posterior and the generative model are inevitably suboptimal, similar to simple variational autoencoders. This raises the question of what family of densities the proposed model can model or not.  
    
- On the contrary, DDIM is designed under the premise that the DDIM models will follow the same marginal $p(x_t)$ of the DDPM models. This implies that DDPMs' expressivity and tightness of the ELBO indirectly justify DDIMs'. Thus, DDIMs' approach makes much more sense, as we can expect that DDIM will model arbitrary densities by maximizing their ELBOs (or weighted ones). Thus, the tuning weights for DDIMs and DDPMs are valid engineering trade-offs depending on the application scenarios.  
    
    - A side note: Sohl-Dickstein et al., 2015 and DDPMs are inspired by the fact that the first-order discretization of continuous-time It\^o diffusion is a Markov chain, each of whose transition probability follows a Normal distribution. Moreover, it has been proven that their reverse-time It\^ diffusion process exists (under mild conditions), and thus their first-order discretization again is Gaussian Markov chains.  
    
- To resolve this, I consider that it is necessary to either provide (1) theoretical proofs that the proposed generative models are arbitrary expressive and the ELBO with the posteriors can be tightened, or (2) empirical analyses that the proposed method can achieve comparable performance when it is trained via maximizing ELBOs. Unless either of them cannot be demonstrated, the significance of the proposed method is doubtful.
    
Second, the introduction of the proposed method can be improved. For instance, the authors briefly mention that the proposed method will use pre-trained reconstruction networks in the abstract and Sec 2. However, I found that clarifying how Equation 7 is used to define generative models will be helpful for readers.

Thirds, the descriptions of the proposed method in the context of the previous works seem confusing. For example, the usage of the term "diffusion process" sounds confusing. "Diffusion process" have commonly referred to as It\^o diffusion process, which is a solution of specific types of stochastic differential equations; thus, continuous-time models. I found that it is a little problematic to use the "diffusion process" for the proposed method.
summary_of_the_review: In general, the paper's contributions are unclear, while the paper tackles a very interesting problem in the context of diffusion-based generative models and provides empirical improvements. Unfortunately, the proposed method is not well-motivated, and the discussion about it is insufficient. Thus, I opt to reject the paper. However, I'm also inclined to improve my evaluation if the aforementioned weak points are well addressed.

===== POST-REBUTTAL COMMENTS ========  
The rebuttal had addressed some of my concerns. Consequently, I raised my score from 3 to 6, and I also raised the "Correctness" & "Technical Novelty And Significance" scores from 2 to 3.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work examines the problem of the large number of iterations necessary to get high quality samples from denoising diffusion generative models. 
The paper mostly consists of two new components:

1. A generalized family of probabilistic samplers called Generalized Gaussian Diffusion Processes (GGDP). This construction is similar to the denoising implicit processes found in [1], in the sense that it discards the Markov assumption, and $p(x_{t=i})$ may rely on all $p(x_{t < i})$. The difference is that the Gaussian marginals of the GGPD may not match those of the original process. 

2. An optimization procedure for finding the sampler variances. This procedure is general and may be applied to any family of samplers. The idea is to minimize the distance in feature space between a classifier's representation given the real data, and the final generated iterate in the process, optimizing over the parameters of the sampler family. A computationally amenable scheme is proposed to allow for backpropagating through the diffusion process making use of gradient rematerialization (recomputation of intermediate gradients). 

The method is compared against DDPMs [2] and DDIM [1] in CIFAR10 and Imagenet 64x64. 

[1] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

[2] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015

main_review: I find this paper to be very interesting and its results are quite good. I will address the main points of this paper as outlined above. 

1. It appears that the introduction of a more flexible family of samplers is introduced mainly to admit the optimization process for finding the variances of a more efficient sampler (DDSS). As optimizing for $v_1 \dots v_k, \sigma$ would be difficult if the corresponding marginals were constrained to be equal. Indeed this is shown in 5.1 where optimizing $\sigma$ for DDIM does not perform well. This is to say that I am not sure there is any benefit or purpose to GGDP beyond admitting the use of DDSS. This is not necessarily a bad thing, but it's worth pointing out that GGPD doesn't look to stand on its own as a better sampler without DDSS. If this is not the case I am happy to be wrong here. 

2. I'm not sure about the practicality of DDSS. Given that gradient rematerialization (only feasible in JAX as far as I know) is leveraged to compute the gradient of the objective, I can only assume that the computational cost of a naive implementation is quite high even for CIFAR10. That being said, this work is about efficient (in number of steps) samplers, not scalable ones. I found the results of GGDP+DDSS to be quite impressive. The samples shown in figure 2 are in my opinion strikingly better than the baseline approach. 

I found the evaluation to be reasonable if a bit light on the number of datasets. Regardless, the results are compelling, and given that the DDSS objective is quite simple, I am happy with the approach. 

All things considered, I think this optimization approach is a natural direction when considering how to construct more efficient DDPMs, and the results support it. 

-------------------------------------------------------------
### Post Rebuttal Comments. 

I appreciate the updates to the submission made by the authors. In particular, the improved motivation of using a perceptual loss instead of maximizing an ELBO. I will retain my score, though I hope that the authors follow through on their claim that they will provide additional results in the camera ready submission. 

summary_of_the_review: This work proposed a relatively simple, if computationally difficult optimization procedure for finding the parameters of a sampler able to generate compelling images in fewer iterations. I found the approach reasonable given that a more flexible family of samplers is proposed to be amenable to the optimization process. The results are compelling, achieving better sample quality in fewer timesteps. In the regime of fewer than 10 steps, this work achieves better sample quality in half the steps.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples, but they are not efficient at inference. This paper proposes Differentiable Diffusion Sampler Search (DDSS) to generate high-quality samples while using fewer inference steps than DDPM. The proposed approach uses reparametrization trick and gradient rematerialization to optimize over a class of parametric samplers that use fewer steps than DDPMs. By optimizing perceptual losses, the proposed approach can generate high-quality samples using a smaller number of inference steps compared to existing approaches for sampling from DDPMs.
main_review: 1. It seems that optimizing the perceptual loss might not give you the samples from the true underlying DDPM model. Although this approach might produce reasonable samples empirically, it lacks theoretical insights: it is unclear what distribution these samples are optimized to match with. It seems that the training objective is hand-crafted, and might lack theoretical guarantees. 
2. Although samples from the proposed method have good FID/IS scores, optimizing the model via the perceptual loss could be directly/indirectly overfitting the evaluation metrics. So the improvements in FID/IS scores might be expected. 
3. Are you still able to get likelihoods (lower bound) on the generated samples from GGDP? 
4. As mentioned in the paper, GGDP can use pre-trained DDPM models. It would be more convincing to also show results on more challenging high-resolution datasets (e.g. LSUN, ImageNet, CelebA) using pre-trained models since the inference time for high-resolution datasets are often significantly longer, and improving the inference time would be a more crucial task for larger models. Currently, only small-scale datasets are considered.
5. For theorem 1, there is only "proof sketch". Is there a formal proof?
6. In section 4.1, it would be better to mention clearly what "the collection of all the introduced trainable variables" is.
7. It seems that the samples of the proposed method in figure 2 look reasonable when using 5 steps on CIFAR-10. However, in the appendix, the samples using 5 steps on ImageNet 64 do not look that realistic. Does it imply that the performance of the proposed approach is dependent on the dataset? If that is the case, more datasets are needed to show the strength of the method.
summary_of_the_review: 1. Although the proposed approach can generate high-quality samples, the objective used is hand-crafted and might lack theoretical guarantees. As mentioned in the paper "GGDP family does not guarantee that the marginals of the new forward process match that of the original DDPM." Although the sample quality can be reasonable, it is unclear what the model is doing theoretically (e.g., is it optimizing likelihood, or doing score matching).  
2. The samples, after optimizing perceptual loss, might not correspond to the samples of the underlying DDPM model. Optimizing via perceptual distance is expected to have better performance under FID/IS scores, but it would also change the statistical property of DDPM. In some sense, the samples are not from DDPM, and it might be useful to discuss in more detail the statistical properties of GGDP.
3. As mentioned in the paper, GGDP can use pre-trained DDPM models. It would be more convincing to also show results on more challenging high-resolution datasets (e.g. LSUN, ImageNet, CelebA) using pre-trained models, since the inference time of high-resolution models are ofter longer, and improving the inference time of larger models is more crucial.
4. The samples on ImageNet 64 in the appendix are not very good compared to the samples on CIFAR-10 in figure 2, which also raises questions on how general and robust the proposed approach is.
5. Given the existing work (e.g., DDIM), conditioned on all the previous noisier images in the sampling chain might not be very novel.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses how to accelerate a diffusion generative model via Differentiable Diffusion Sampler Search (DDSS). First, a family of non-Markovian samplers is introduced, allowing the models to use multiple previous steps for generation. Next, a sampler search method is proposed to optimize the perceptual loss, with certain engineering issues addressed such as bounded and monotonically increasing steps and gradient parametrization. Empirically, this is able to improve sample quality in a few steps compared to other heuristics such as linear and quadratic schedules on CIFAR10 and ImageNet 64x64, especially when there is a very limited amount of steps that can be taken. The proposed DDSS + GGDP method outperforms DDIM and DDSS + DDIM, showing the advantage of DDSS.
main_review: **Strengths**

The idea in the paper is relatively simple and straightforward, and has good empirical performance. The paper discusses how to address the engineering challenges in general DDSS search problems (discontinuity of timesteps, memory constraints). Empirically, optimizing perceptual losses benefits sample quality and is better than evaluation likelihoods (especially when most methods need many steps to have marginally better likelihoods).

**Weaknesses**

The clarity of the paper is a bit on the weaker side, and I think the paper can be significantly improved by having better clarity.
- It was quite unclear at first glance what the double subscripts mean in Equation (2) and beyond. It should be made explicit that subscript $ts$ means starting from $t$ and ending at $s$. These double subscripts also make it difficult to identify the main idea behind GGDP. Perhaps having a figure illustrating the GGDP idea helps with clarity.
- It is unclear whether, with the changed GGDP variational distribution, it is still possible to recover the same "diffusion / score-matching" solution with a new variational objective (not defined in the paper). I am inclined to believe this is the case, but I think a proof (like the one in DDIM paper) would be warranted.
- Equation (9) should be written explicitly as the batch version, since the actual algorithm is not an unbiased estimator of (9). Also is there a reason behind only matching the first-order moment, rather than higher-order moments? Is there a possibility that the DDSS sampler collapses to a single point that has the correct mean? This might need a bit of explanation.
- Minor, it would be helpful to have more details on how the parametrization in DDSS is performed in the appendix. Part of the confusion may be from the fact that we have a lot of notations, such as $\mu_{tu}$ and $\sigma_t$, $a$, $v$ etc. It would be great to explicitly explain what these mean (or correspond to in DDPM / DDIM).
- I have a hard time understanding the code `t = cumsum(softmax([*v, 1.]))[:-1]`. What are the range of v and t? Maybe an example in the appendix can help?
- I think the proof sketch can be moved into appendix as a more formal proof. At a high-level everything is Gaussian, so the claims themselves are not too hard to understand.

**Comments**

- The GGDP methods seem to be quite related to multistep methods in numerical integration, [here](https://en.wikipedia.org/wiki/Linear_multistep_method). I wonder what would be the connections to established multistep methods, such as Adam-Bashforth? It would be interesting to see what parameters does DDSS learn to further understand what it does. 
- Is there a reason why datasets with larger images such as LSUN (256x256) are not tested?


==== Post rebuttal review ===
I think the rebuttal addressed most of my concerns. I am a bit disappointed that no discussion about larger image datasets are given, but the other points are adequately addressed. So I will keep my score as is.
summary_of_the_review: I think the problem of accelerating diffusion models is important and believe that the paper makes a novel contribution towards it. That being said, I believe that the paper has serious issues regarding clarity and minor issues on experiments over larger datasets. 

I am willing to increase the score if the authors address clarity issues well enough in the rebuttal and especially in the revisions that are allowed by the ICLR format. If the revision does not change by much, then my score will be around 5 and 6 (good idea, relatively poorly polished paper).
review_title: null

Meta-review: The paper tackles a very interesting problem in the context of diffusion-based generative models and provides empirical improvements. Pre-rebuttal, reviewers' main concerns lie in the motivation and clarification of the method, while after rebuttal, all reviewers satisfied the response and gave positive scores. The authors should include the additional results to well address the reviewers' concerns in the final version.
Decision: Accept (Poster)