Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors suggest how to apply the idea of score matching to the problem of conditional generation. They introduce a new loss, which is essentially a score matching loss for the conditional distribution p(y| x). They demonstrate how to effectively calculate it without calculating p(y|x) explicitly, which seems to be the main contribution of the paper. Then the authors demonstrate how to train the model for the conditional generation in practice: it is necessary to add to the proposed loss the usual cross-entropy loss, because the loss that they came up with is noisy (this can be seen in Figure 5). They got better results compared to baselines for cifar-10 and cifar-100.
main_review: Some disadvantages

1) The loss is quite noisy (see Figure 5), and the authors do not explain why. 
I personally think that the reason is because they approximate p(y|x) with a mixture of Gaussians with a small variance and there is the same effect with smoothing as in case of the standard score matching. 
Probably we should start training using larger variance values and gradually decrease the variance during the training.

2) I think that the motivation of the paper has too many somewhat irrelevant details. The authors compare different models, trained under different conditions, and conlude that the problem is in the loss. It seems that it was enough to say that it is difficult to approximate the gradients, so it is necessary to explicitly add them to the loss.

3) Models (a) and (c) are different (in the first case it is a single model, in the second case the authors use different models for different classes). The first model minimizes cross-entropy loss, the second model optimizes the score matching function. 
It is not entirely clear why the authors believe that the model (a) works worse than the model (c)  because of the loss? The models are also different. Maybe this is the main reason?

4) In the same section with the motivation I propose to add the cross-entropy loss between p(y|x) and p(x|y), and not just the loss between their gradients (maybe the model (a) also approximates them poorly). The question immediately arises, why a good cross-entropy loss does not guarantee a good approximation of the gradients?

5) Why do the authors train the classifier p(y | x, \theta) and the score model (x, \phi) independently, not end-to-end?

6) Why is their loss noisy in practice? How much does the value of \tau (noise variance) affect the stability of the loss and the results? Why don't the authors do the same trick with \tau as with \sigma (that is, they don't train several different models with different values of \tau)?

7) Experiments are done only on cifar-10/cifar-100, which is not very convincing, of course.
summary_of_the_review: In general, the paper is well written, it has a clear and logical structure, it is easy to read. 
In the introduction, only clearly highlighted contributions are missing.

The proposed criteria can be useful when training models for conditional generation.
The authors did rather detailed discussion of different variants of the score matching procedure.

============

I thank the authors for their responses. The updated version of the paper, with additional information here and there, improves the readability and the overall narrative. Given the authors responses now I am convinced in the concept proposed by the authors, so I can raise my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper points up a previously underappreciated problem in training classifiers in the context of conditional generations of diffusion-based generative models. The authors propose a novel objective for training classifiers to tackle the problems. Informally, the generation process of the diffusion-based generative models can be described by repeatedly applying an update rule with initial values: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \sigma \epsilon$ where $\epsilon \sim N(0,I)$ and the initial points are sampled from a prior distribution such as standard Normal distributions. Similarly to unconditional generations, one can perform conditional generations, e.g., $p(\tilde{x} | y)$ by using the following update rule: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x}) + \sigma \epsilon$. Note that $\nabla_{\tilde{x}} \log p(\tilde{x} | y) = \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$. Thus, for a given pre-trained generative model, one needs to learn $\log p_{\textrm{model}}(y | \tilde{x})$ (or its gradient wrt $\tilde{x}$. Maximum likelihood training (MLE), i.e., minimizing cross-entropy loss, is commonly used. Inevitably, the qualities of the gradient $\nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$ will determine the qualities of the conditional generations.


First, the paper emphasizes that MLE-training of classifiers results in non-smooth gradient landscape wrt input; thus, the resulting gradients negatively affect the generation qualities. More precisely, with MLE, the learned $\log p_{\textrm{model}} (y|\tilde{x})$ is high, while its gradient wrt $\tilde{x}$ isn't necessarily close to the ground truth. In the paper, the authors refer to this phenomenon as *a score mismatch issue.* The authors analyze the mismatch issue with toy experiments and demonstrate its negative effects on generation qualities compared to the ground truth.


Second, to resolve the *score mismatch issue*, the paper proposes a new objective function, called Explicit Likelihood Score-Matching loss (ELSM), in which the mean squared errors between $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ and $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$ is minimized. Here, due to the inaccessibility $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$, the authors reduces the ELSM loss to another objective function, named *Denoising Likelihood Score-Matching* (DLSM), similarly to denoising score matching derivation. In DLSM, $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ is trained to match to $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x}) - \nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x} | x)$. Acknowledging that $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is still unattainable, the authors propose to minimize approximate DLSM loss, where $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is substituted by the pre-trained scores $\nabla_{\tilde{x}} \log p_{\textrm{model}} (\tilde{x})$.


Then, the authors demonstrate that by toy experiments, training classifiers with the approximate DLSM improves the conditional generation qualities compared to MLE-based training. Moreover, the authors mention that approximate DLSM-based training can be unstable in high-dimensional datasets and show that training classifiers by minimizing combined loss of approximate DLSM and cross-entropy can further improve the generation qualities.


Lastly, the paper demonstrates the effectiveness of the proposed methods by evaluating conditional generation qualities on CIFAR-10 and CIFAR-100 datasets.
main_review: **Strengths of the paper**  
In general, the paper's contributions are clear, and I also consider that the results are essential for several reasons:  


First, the paper highlights a previously less-recognized problem in the context of conditional generations of diffusion-based generative models. Moreover, the authors provide sufficient analysis to help readers understand the problem and potential drawbacks of commonly accepted methods. In particular, the analysis to visualize the difference between the MLE-based training and the proposed method is very interesting to read.


Second, the paper introduces a novel training objective to tackle the problem above, called Denoising Likelihood Score Matching (DLSM). Recently, the interest in diffusion-based generative models has increased rapidly. Consequently, the interest in controlling the generation process of such models has also increased. Regarding this, I consider that the proposed method has huge applicability and will contribute to the ML communities.


Third, the authors have devoted themselves to discussing practical techniques for applications of the proposed method. For example, the authors discuss that the effect of combining two losses, i.e., MLE and approximate DLSM, for training classifiers. In particular, the paper discusses the potential roles of two losses on the performance of the conditional generations throughout ablation studies. Furthermore, the authors provide additional analysis about the scaling method, including how they contribute to high-precision samples.

**Weaknesses of the paper**  
However, the following aspects of the paper can be improved.


First, clarifying three objectives seems necessary, i.e., ELSM, DLSM, and approximate DLSM. In my opinion, when the pre-trained score is plugged into the DLSM objective, it is no longer DLSM loss. However, the current version hasn't sufficiently emphasized the difference, so I found that the current version can be potentially misleading. Moreover, the provided proofs and relevant theoretical discussions are about ELSM and DLSM, not approximate. As a result, readers infer the behaviors of the approximate DLSM only by empirical results. Thus, it is important to distinguish between DLSM and approximate DLSM.


Second, while the experiments are well-thought and easy to follow in general, some experimental results are unclear. Moreover, additional experiments seem required. For instance, it is unclear how FID and IS are evaluated. At first glance, I understood that the FID/IS are evaluated per class (and their averaged value are reported). However, the paper describes that they are evaluated unconditionally. If the same model was used, the unconditional generators should have been the same for the baselines and the proposed method. If it wasn't, the description of the experiments needed to be updated. Another concern is that it is unclear when the combined loss is used and when it isn't during the experiments. In particular, it is unclear if "ours" in Table 1 corresponds to the approximate DLSM or the combined loss since the rest of the experiments have used the combined loss. I consider it important to clarify Table 1 and potentially include the results of models trained with the combined loss. Similarly, it would be interesting to see the results of CIFAR-10/100 experiments without MLE-losses.


Third, the presentation of some backgrounds can be improved. In particular, it seems like that the current submission introduces the discrete-time Langevin dynamics, which is time-homogeneous, as a building block of the diffusion-based generative models. However, the paper follows the recent works, specifically time-inhomogeneous diffusions, which have made significant distinctions of the recent works against the time-homogeneous ones in several perspectives; for example, practicality in the context of generative models.

**Minor comments**
- I found that matching the y-axis scale in Figs 4 and A.1. will help interpret the results.
- Optionally, I found it beneficial to run 1-dimensional experiments as done in Sec 3 but present results similar to Fig A.1. In my understanding, the classifiers trained with the approximate DLSM will underestimate the gradient (thus smoother) in comparison to the DLSM (I may be wrong). On the other hand, the MLE-based training results in unnecessarily sharper. Plotting the learned gradients and the ground truth will be helpful to explain why combining two losses makes sense.
- In my understanding, $\tilde{y}$ may not be necessary to derive DLSM. Could you explain why noisy $\tilde{y}$ is necessary?
summary_of_the_review: In general, the paper's contributions are clear; the proposed methods are well-motivated and well-discussed. In addition, I also found that the paper has a well-organized structure so that it is clear to understand the proposed method and other practical techniques to improve training. Thus, I'm inclined to accept the paper. However, I found that several aspects of the submission can be improved. I hope that the aforementioned weak points are well addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a conditional diffusion(score-matching) model to tackle the score mismatch issue in the conditional generation scenario. This paper tests multiple alternatives in creating a conditional distribution through diffusion models, and this paper introduces a classifier assisted structure for the conditioning.
main_review: 1.
I have a question on the derivation of Eq (A1). Particularly, authors suddenly introduce $Z(\tilde{x})$. It will be great if authors can explain how to main the equality, after the introduction of $Z$.

2.
It seems that the scores are mismatched across conditions. For example, the score on condition A cannot be matched to the score on condition B, which is quite natural. Given that it is good to see that the authors performed the posterior score-matching by separating the score-matching by conditions. At lease, Figure 1 shows not much difference between posterior SM and suggested models. Also, Table 1 suggests not much difference (or rather favorable to (c) posterior SM than the suggested models) between the posterior SM and the suggested model.

Then, what would be the gain by not just using posterior SM and by following the suggested models?

3.
I cannot find the clear definition of $L_{CE}$ which would be basically classifier, but it would be better to give a clear form of classifier with inputs and outputs.

4.
The experiment does not report negative loglikelihood, or bpd. Why don't you report NLL?

5.
There is no evaluation through the comparisons with the baseline models. As authors suggested that there are recent rush on the conditional modeling with diffusion approaches, why don't you find some and compare the performance with them?

summary_of_the_review: Niche paper to introduce a conditional generative model by the diffusion process
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a new objective, denoising likelihood score-matching (DLSM), and training mechanism for conditional score-based data generation. The new method is motivated by poor conditional score estimates observed in low-dimensional examples for the standard, Bayes-theorem-based, conditional data generation procedure of score-based models. The new objective is equivalent to directly learning to parameterise the score of p(y|x) demonstrates is high-dimensional benchmark datasets also. FID and IS improve due to this method, with the largest improvements being class-wise, and there is an overall tendency to trade off generated data recall for precision.
main_review: **Strengths**
- Sections 1 and 2 are very clear and exhibit strong understanding of the field and the most recent literature. Upon examination, the proof of (2) in A.2 appears to be correct.
- The motivating example, in Section 3 and Figure 1, is clearly presented, describes a relevant synthetic case, and the results demonstrate enough improvement to mean the paper's flow and narrative works well. The metrics $D_P$ and $D_L$ are also valuable in this more tractable setting.
- In Section 4, Equations (9) and (10) follow intuitively and are well explained. I have been through the proof of Theorem 1 in detail and I believe it is correct (there wasn't even a typo!). Figure 2 is also a useful/pertinent depiction of the training process.
- I believe all of the experiments presented in Section 5 are useful, clear, and well laid out.

**Weaknesses**
The principle weakness I can see in the paper is the lack of a breakdown of "inaccurate likelihood scores".  Although empirical evidence suggests the classifier is at fault, the paper only uses references for this motivation. There is no explanation as to how one might stratify these classifier score inaccuracies. As one example, does a stronger classifier provide better gradients for conditional generation? This line of research would potentially bolster the training procedure in Figure 2 by further guiding the involvement of the classifier.

- The use of "some extra information" at the start of Section 2.4 is casual and belies the importance of the class.
- In Section A.3, I believe "was $p_{\alpha}(\mathbf{\tilde{x}},\mathbf{\tilde{y}})$ not" should be "$p_{\alpha}(\mathbf{\tilde{x}},\mathbf{\tilde{y}})$ was not".
- At the end of page 5: "numerical results in Table 1 are" rather than  "numerical results in Table 1 is"
- The introduction of $p_{0,\tau}(\mathbf{x}|\mathbf{\tilde{y}})$ in the 5th line of the $G(\theta)$ derivation is slightly jarring given that the zero in the subscript is not used elsewhere or explicitly described.

Stylistic choices:
- Use of \left[ and \right] and latex would be nicer, but this is subjective.
- Same goes for the use of \lVert and \rVert for norms instead of \parallel (more for divergences).
summary_of_the_review: Overall, I think this is a strong paper that would make a positive contribution to the conference and the current discussion surrounding SBMs.  The paper is very well presented and explained throughout, with appropriate figures and tables, as well as accurate derivations. Class conditional generation does appear to improve based on this method, and the complementary performances trade-offs (distribution precision) are broken down and discussed. Besides one line of research that could be included in a future paper, weaknesses are minor and/or aesthetic.
review_title: null

Meta-review: This paper discusses an issue with decomposing a conditional generative model into an unconditional model and a separate classifier using Bayes' theorem, which is an approach that has recently received increased attention in the context of score-based generative models. It explores several alternatives for mitigating this issue, including a novel one, which is to use a different loss function to train the classifier.

Reviewers praised the writing and the way this work draws attention to an issue that is underappreciated in the community. Although several weaknesses (clarity, scale of experiments, appropriateness of baselines, missing experiments) were also highlighted in the original reviewers, all reviewers agree after discussion that the authors have adequately addressed these for the paper to be considered for acceptance. I will follow their recommendation and recommend acceptance as well.
Decision: Accept (Poster)