Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a modification of DDPM for conditional generation, in which they explicitly model the conditional mean ("cluster center") in the forward and reverse diffusion processes. This modification is motivated by the empirical observation that for the large portion of the diffusion samples form several distinct cluster. The resulting model, ST-DDPM, shows competitive results on conditional image generation and inpainting; and enjoys interpretability by means of cluster center visualisation.
main_review: **Strengths**
* The method is well-motivated through empirical observations that are clearly set forth in the paper.
* The paper is written in a way that is accessible to a wide audience, yet contains detailed derivations in the appendix.
* I expect the paper to be interesting to a wide audience due to the wide range of tasks it considers, and separately due to the proposed method being more (easily) interpretable than existing score-based generative model.

**Weaknesses**
* The baseline conditional generation method considered in the paper (cond. DDPM) appears to be worse than it's unconditional counterpart (DDPM) in terms of the FID / IS metrics (Table 1). One would expect the conditional generative model would outperform the unconditional one by a large margin. It seems that the baseline may not have been properly tuned / optimised, and absent a well-tuned baseline it's difficult to judge the performance of ST-DDPM.
* Faster sampling experiments are not particularly strong, and miss important baselines (e.g., "Learning to Efficiently Sample from Diffusion Probabilistic Models" by Watson et al.). The proposed technique still requires 70%-95% of the denoising steps of the full generation chain - far more than competing methods or hand-crafted beta schedules.

**Minor comments**
* Was not clear what "ST" in ST-DDPM stands for. Is that "shift"?
* How were the 2D projects in Figures 2 and 3 made? If t-SNE or a similar dimensionality reduction method, please include hyper-parameters in the Appendix.
* The word "Appendix" was not consistently capitalized (noticed this for "appendix B", but there may be more).
* What is a "fire embedding" (Section 6.1)?
* Left plot of Figure 10 is somewhat trivial: skipping x% of the training steps gives you a 1 / (1 - x / 100) speed up. E.g. 1.43 = 1 / 0.7, 1.12 ~= 1 / 0.9. This comes from the fact that all denoising steps have the same runtime complexity.
summary_of_the_review: The paper provides an interesting and well-motivated modification of DDPMs for conditional generation, but needs stronger baselines to help assess the performance of the proposed modification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Score-based generative models show great performance on image generation tasks.
Score-based generative models are composed of the two-phase, forward process, and reverse process.
On the forward process, the score-based model gradually corrupts the data with noise, and the reverse process restores the image by denoising.
This paper measures the ratio of intra-class and inter-class, and they find that the class separation decreases sharply and the data distribution is totally covered by the noise.
From a new finding, this paper proposes a class center on both forward and reverse processes for a controllable generation.
main_review: < Strength >
1. This paper provides a new variant of a score-based generative model, and it can be attractive for many machine learning researchers.

2. This paper suggests a new interesting finding, the changes of the ratio of intra-class and inter-class.

3. From a new finding, this paper proposes a new class center-based algorithm for conditional image generation. The proposed model, ST-DDPM, shows a significant performance improvement on the FID score and Inception Score.

< Weakness and Questions >
1. The quantitative results are reported for the CIFAR-10 dataset only. There are qualitative results for other datasets such as FFHQ and LSUN, but there are no quantitative results for that kind of dataset.

2. This paper only reports the FID and inception score. I am curious about the results of negative log-likelihood.

3. There are many new score-based model research [1,2,3], and it will be interesting if further discussion will be added to the paper.

[1] Tae, Jaesung, Hyeongju Kim, and Taesu Kim. "EdiTTS: Score-based Editing for Controllable Text-to-Speech." arXiv preprint arXiv:2110.02584 (2021).

[2] Kim, Dongjun, et al. "Score Matching Model for Unbounded Data Score." arXiv preprint arXiv:2106.05527 (2021).

[3]  Tashiro, Yusuke, et al. "CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation." arXiv preprint arXiv:2107.03502 (2021).
summary_of_the_review: Please see the weakness and questions section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper observes the ratio of intra-class variance to the inter-class variance during different time steps of the forward process of diffusion models and notes that the ratio initially remains stable for the early steps of the process and then greatly increases when the data distribution finally gets converted to noise distribution. Taking inspiration from this class clustering process, the paper proposes to explicitly model the class center in the forward and the reverse process, whereby during the forward process the samples are pulled towards the center, before ultimately becoming noise. A separate conditioning network is used to model the centre. By using class as conditions for the conditioning network, the paper is able to perform class condition generation from a diffusion model and shows good image generation performance in terms of FID and Inceptions scores. The paper also shows good performance for other controllable image generation tasks such as image inpainting, attribute conditioned generation, and text to image generation by efficiently conditioning the diffusion model.
main_review: Strength - 
1) The idea of doing controllable generation from diffusion models across different tasks by efficiently conditioning the model is interesting. 
2) The paper shows strong conditional image generation results. Qualitative results for the image inpainting tasks also look good.
3) Extensive experiments across different tasks and datasets.

Weakness - 
1) I am not convinced by the motivation of the paper. As the noise strength added in the initial stages of forward process is small and it grows over time, it is natural to expect the intra-class to inter-class ratio to remain stable in the initial stages of the forward process.
2) Adding on to the above point, the motivation around modeling class centers is not clear. The experiments do not reveal the intra-class variance to decrease during the forward process, but rather they remain stable. Why should then the forward process pull the data points towards the center?
3) The results of faster diffusion denoising do not convince me. By starting from earlier starting points around the center, I expected the FID for instance when using 700 steps to not drop from 3 to 7. As a baseline, what's the performance difference when we naively use 700 steps for the DDPM model instead of using 1000 steps?
4) I found the experiments around image inpainting hard to understand. Can the authors make it clearer as to how is the training done for it? Do you randomly mask out regions of the image during training at each iteration? How mc-uch portion of the image is masked out at max?
5) I couldn't find a comparison of the text to image generation against other works. Can the authors include the FID numbers for other related works?
summary_of_the_review: The paper achieves good empirical performance but needs more polishing around the motivation and some experimental details. I will be happy to increase my rating if my queries are clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new forward conditional process. The transition kernel is controlled by a condition embedding. It also derives a reverse process. The $\epsilon_\theta$ module is decomposed into two parts. The paper conducts experiments on different conditional generation tasks.
main_review: Strengths:
The conditional framework proposed in this paper is general and is applicable to different tasks. The learned condition embedding contains some semantic information.

Questions:
1. The earlier starting (ES) is only slightly mentioned in Sec.6.5. Although the author claims that details can be found in Appendix B, I didn't find how it is derived and what is the principle. Besides, it seems that only using strided sampling (SS) can work even better than combing ES and SS, e.g., Table 4 in Appendix shows that SS with 100 steps can achieve a FID of 4.29, which is better than 4.45 of ES+SS.

2. Is the decoupling of $\epsilon_\theta$ reasonable? According to Eq.(11) and Eq.(12), the optimal $\epsilon_\theta^*(x_t, u, t)=E[\epsilon|x_t]=E[\frac{x_t - \sqrt{\overline{\alpha}_t} x_0 }{\sqrt{1-\overline{\alpha}_t}}|x_t] - \frac{n_t}{\sqrt{1-\overline{\alpha}_t}}$. The first term is still related to $u$, since the joint distribution of $x_0, x_t$ is $q(x_0|c) q(x_t|x_0,c)$, which is related to $u$. However, the author models the first term with $\epsilon_\theta(x_t, t)$, which is independent of $u$.

3. Algorithm 1 and Algorithm 2 don't show how to get the label $c$. Besides, in Algorithm1, shouldn't $u=u_\phi(x_0)$ be $u=u_\phi(c)$?

4. At the beginning of Section 5, the author denotes $u_\phi(c)$ as the class center. However, $u_\phi(c)$ is actually learned, which is not rigorously the class center. It is likely to cause misunderstanding.

5. As mentioned earlier, the optimal $\epsilon_\theta^*(x_t, u, t)=E[\epsilon|x_t]$. What will the optimal condition embedding $u_\phi(c)$ be?

6. I didn't find a strong relationship between the clustering phenomenon and the formulation of ST-DDPM. How the phenomenon motivates ST-DDPM should be discussed in detail and formally.

7. To distinguish this paper from prior conditional methods, the strengths than the traditional conditional DDPM (i.e., the cond. DDPM in Table1) should be discussed more.

8. This paper is about conditional diffusion models, thereby it would be better if experimental comparisons with prior conditional diffusion models (e.g., [1*, 2* ,3*]) are added. The author should at least discuss these works in the related work section.

9. Some missing experimental details. For example, what is the batch size?

[1*] Improved Denoising Diffusion Probabilistic Models

[2*] Diffusion Models Beat GANs on Image Synthesis

[3*] ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

summary_of_the_review: This work has some questions, which should be addressed to reach the acceptance threshold.
review_title: null

Meta-review: The paper presents an interesting approach for defining conditional diffusion models. The core idea of this work is based on a new analysis of how class centers evolve in the forward diffusion process. On this positive side, this work builds on top of this analysis and introduces conditional diffusion processes that are guided towards class centers. This paper shows marginal improvements in small image datasets (MNIST and CIFAR-10) and auxiliary applications such as image inpainting and attribute-based image synthesis (demonstrated through only qualitative experiments). On the negative side, the proposed approach has the fundamentally limiting assumption that a class can be represented by a cluster center in the RGB space. Unfortunately, this assumption does not hold for practical datasets such as ImageNet where samples in each class have high diversity, and the class centers in the pixel space are not very distinct for different categories. The reviewers have rated this paper slightly above the borderline. They have acknowledged the novelty of the proposed guided diffusion. But they have criticized the submission for the lack of experiments on more common and challenging benchmarks. They also have criticized this work for not providing quantitative results on the auxiliary tasks. I agree with the reviewers that these experiments would shed light on whether the class center idea would hold for more challenging scenarios.

In the rebuttal, the authors provided additional quantitative results for the text-to-image generation task. However, these results show that the proposed method is outperformed by prior works. Most other auxiliary tasks including image inpainting and attribute-to-face generation are still demonstrated through qualitative experiments without detailed quantitative results. 

In summary, given the limitation of the proposed approach, this submission currently lacks an in-depth analysis of the proposed work on challenging benchmarks and a detailed quantitative comparison to relevant baselines for the auxiliary tasks. Because of these concerns, we believe that the paper in its current form is not ready for publication at ICLR at this point.
Decision: Reject