Denoising Likelihood Score Matching for Conditional Score-based Data Generation
Authors: Chen-Hao Chao
Wei-Fang Sun
Bo-Wun Cheng
Yi-Chen Lo
Chia-Che Chang
Yu-Lun Liu
Yu-Lin Chang
Chia-Ping Chen
Chun-Yi Lee
Venue: ICLR 2022 Poster (2022)
Keywords: score-based generative model
conditional sampling

Abstract: Many existing conditional score-based data generation methods utilize Bayes' theorem to decompose the gradients of a log posterior density into a mixture of scores. These methods facilitate the training procedure of conditional score models, as a mixture of scores can be separately estimated using a score model and a classifier. However, our analysis indicates that the training objectives for the classifier in these methods may lead to a serious score mismatch issue, which corresponds to the situation that the estimated scores deviate from the true ones. Such an issue causes the samples to be misled by the deviated scores during the diffusion process, resulting in a degraded sampling quality. To resolve it, we theoretically formulate a novel training objective, called Denoising Likelihood Score Matching (DLSM) loss, for the classifier to match the gradients of the true log likelihood density. Our experimental evidences show that the proposed method outperforms the previous methods on both Cifar-10 and Cifar-100 benchmarks noticeably in terms of several key evaluation metrics. We thus conclude that, by adopting DLSM, the conditional scores can be accurately modeled, and the effect of the score mismatch issue is alleviated.

Published as a conference paper at ICLR 2022
DENOISING LIKELIHOOD SCORE MATCHING FOR CONDITIONAL SCORE-BASED DATA GENERATION
Chen-Hao Chao
Wei-Fang Sun
National Tsing Hua University
National Tsing Hua University
Bo-Wun Cheng National Tsing Hua University
Yi-Chen Lo Mediatek Inc.
Chia-Che Chang Mediatek Inc.
Yu-Lun Liu Mediatek Inc.
Yu-Lin Chang Mediatek Inc.
Chia-Ping Chen Mediatek Inc.
Chun-Yi Yi Lee National Tsing Hua University
ABSTRACT
Many existing conditional score-based data generation methods utilize Bayes' theorem to decompose the gradients of a log posterior density into a mixture of scores. These methods facilitate the training procedure of conditional score models, as a mixture of scores can be separately estimated using a score model and classifier. However, the classifier in these methods may lead to serious score mismatch issue, which corresponds to the situation that the estimated scores deviate from the true ones. Such an issue causes the samples to be misled by the deviated scores during the diffusion process, resulting in a degraded sampling quality. To resolve it, we formulate a novel training objective, called Denoising Likelihood Score Matching (DLSM) loss, for the classifier match the gradients of the true log likelihood density. Our experimental evidences show that the proposed method outperforms the previous methods on both Cifar- 10 and Cifar- 100 benchmarks noticeably in terms of several key evaluation metrics. We thus conclude that, by adopting DLSM, the conditional scores can be accurately modeled, and the effect of the score mismatch issue is alleviated.
analysis indicates that the training objectives for
INTRODUCTION
Score-based generative models are probabilistic generative models that estimate score functions, i.e.. the gradients of the log density for some given data distribution. As described in the pioneering work (Hyvärinen) 2005),, the process of training score-based generative models is called Score Matching (SM). in which score-based generative model is iteratively updated to approximate the true score function. Such a process often incurs heavy computational burdens, since the calculation of the wvore-macching objective involves the explicit computation of the partial derivatives of the score model during training. Therefore, a branch of study in this research domain (Vincent 2011 Song al.) 2019) resorts reformulating the score-matching objective reduce the training cost. Among these works, the author in /Vincent [201]) introduced the Denoising Score-Matching (DSM) method. This method facilitates the training process of score-based generative models, and thus lays the foundation for a number of subsequent researches. Recently, the authors in (Song & Ermon 2019) proposed an unified framework based on DSM, and achieved remarkable performance on serval world datasets. Their success inspired several succeeding works (Song & Ermon 2020 Ho et al. 2020 Song et al. 2021a b) [Dhariwal Nichol| 202T), which together contribute to making score-based generative models an attractive choice for contemporary image generation tasks.
favorable aspect of score-based generative models is their flexibility be easily extended to conditional variants. This characteristic comes from a research direction that utilizes Bayes" theorem
Work done during an internship at Mediatek Inc. E-mail: lance_ www.@gapp.nthu...oo Corresponding author. F-mail: cy eeégapp.

---

Published as a conference paper at ICLR 2022
to decompose a conditional score into mixture of scores (gguyen al. 2017). Recent endeavors followed this approach and further extended the concept of conditional score-based models to number of application domains, including colorization (Song al.] 2021b), inpainting (Song ltt al. 2021b), and source separation (Jayaram & Thickstun 2020). In particular, some recent researchers (Song al. [202bb; Dhariwal & ficholl/2221] applied this method to the field of class- conditional image generation tasks, and proposed the classifier-
method. Different from method adopted by (Ho al. (02T), they utilized a score model and a classifier to generate the posterior scores (i.e., the gradients of the log posterior density), with which the data samples of certain classes can be generated through the diffusion process. The authors in (Dhariwal & Nichol (222I) showed that the classifier guidance method is able to achieve improved performance on large image generation benchmarks. In spite of their success, our analysis indicates that the conditional generation methods utilizing a score model and classifier may suffer from score mismatch issue, which is the situation that the estimated posterior scores deviate from the true ones. This issue causes the samples to be guided by inaccurate scores during the diffusion process, and may result in a degraded sampling quality consequently.
the flassifier-uuidance
To resolve this problem, we first analyze the potential causes for the score mismatch issue through motivational -iw-eimesioonl example. Then, we formulate a new loss function called Denoising Likelihood Score-Matching (DLSM) loss, and explain how it can be integrated into the current training method. Finally, we evaluate the proposed method under various con anfigurations, and demonstrate its advantages improving the sampling quality over the previous methods in terms of several evaluation metrics.
2 BACKGROUND
In this section, we introduce the essential background material for understanding the contents of this paper. We first introduce Langevin diffusion (Roberts & Tweedie 1996 Roberts & Rosenthal 1998) for generating data samples z E R& of a certain dimension d from an unknown probability density function (pdf) p(): using the score function V log p()). Next, we describe Parzen density estimation (Parzen 1962) and denoising score matching (DSM) (Vincent 2011) for approximating F log p(ū) with limited data samples. Finally, we elaborate on the conditional variant of the score function, i.e., Va log p(x)j)) @)), and explains how it can be decomposed into Va log p(œ) and Ve log p( i) for some conditional variable ý e Re dimension c.
2.1 LANGEVIN DIFFUSION
I Langevin diffusion RRoberts& & Tweedie 1996 Roberts & Rosenthal 1998) be used generate data samples from an unknown data distribution p(æ) using only the score function Va log p()). which is said be well-defined if p()) ) is everywhere non-zero and differentiable. Under the condition that V a log p(ž) is well-defined, Langevin diffusion enables p(क) to be approximated iteratively based on the following equation:
2 + = -2.1 V log (();))) + EZ
(1)
where To is sampled from an arbitrary distribution, e a fixed positive step size, and 2, is a noise vector sampled from a normal distribution (0 Laxa) for simulating d-dimensional standard Brownian motion. Under suitable regularity conditions, when e - and T + 00, 3y is generated as if it is directly sampled from p(æ) (Roberts & Tweedie 1996 Welling & Teh 2011). In practice, however, the data samples are generated with e > O and < co, which violates the convergence guarantee. Although it is possible to use Metropolized algorithms (Roberts & Tweedie, 1996 Roberts Rosenthal 1998) to recover the convergence guarantee, we follow the assumption of the prior work (Song & Ermon 2019) and presume that the errors are sufficiently small to be negligible when e is small and T' is large.
The sampling process introduced in Eq. ( can be extended to a time-inhomogeneout variant by making p(zz)) and e dependent on (i.e., Pr()t) and ee). Such a time-inhomogeneous variant is commonly adopted by recent works score-based generative models (Song & Ermon 2019 Song et al. 20221), as it provides flexibility in controlling pt (ăt) and &, The experimental results in (Song
& Ermon 2019: Song al. 2021b) demonstrated that such time-inhomogeneous sampling process
2

---

Published as a conference paper at ICLR 2022
improve the sampling quality real-world datasets. In Appendix A.6.1 on the detailed implementation such a mim-innhomoeeneuu sampling process in this work.
offer discussion
2.2 PARZEN DENSITY ESTIMATION
Given true data distribution Pdata the empirical data distribution (o( is constructed by sampling independent and identically distributed data points {x(* Ji=1 and be represented sum of Dirac functions -i= 8[[]
x01) []). Such discrete data distribution po(x) constructed from the dataset often violates the previous assumptions that everywhere non-zero and is differentiable.
Therefore, it is necessary to somehow adjust the empirical data distribution (o( before applying Langevin diffusion in such cases.
To deal with the above issue, a previous literature (Vincent 2011)
utilized Parzen density es- timation to replace the Dirac functions with isotropic Gaussian smoothing kernels pa (i)az) = -- with variance o². Specifically, Parzen density estimation enables the calcula- tion of Po (z) m E Po (o (+). When 0, the score function becomes well-defined and
(2m) 4/2d
can thus be represented as the following:
E Tz (x ")
Va Po
(2)
E Po (o x())
The proof for Eq. (2) is provided in Appendix A.2 This equation can be directly applied to Eq. ( to generate samples with Langevin diffusion. Unfortunately, this requires summation over all data points during every iteration, preventing it from scaling to large datasets due to the rapid growth in computational complexity.
2.3 DENOISING SCORE MATCHING
Score matching (SM) Hyvärinen 2005) was proposed to estimate the score function with a model s(ā;; σ). parameterized by Φ. Given trained score model s( Φ), the scores can be generated by single forward pass, which reduces the computational complexity of Eq. (2) by factor of m. To train such a score model, a straightforward approach is to use the Explicit Score-Matching (ESM) loss LESM, represented as:
LESM (Ø) = Ep (a)
[|( Φ) - V log Po (+)]l²
(3)
This objective requires evaluating Eq. for each training step, which also fails scale well to large datasets. Based on Parzen density estimation, an efficient alternative, called Denoising Score- Matching (DSM) loss (Vincent 2011), is proposed efficiently calculate the equivalent loss LDSM. expressed as:
(o) Ep. (#,
||s(@;; Φ) Va log pa (i ax)
where V log Po (ã(x) is simply (x .). Since the computational cost of denoising score
matching is relatively lower comparison to other reformulation techniques (Hyvärinen, 20055 Song
al. 2019, it extensively adopted in recent score-based generative models (Song & Ermon 2019
2020
Song et al.) 20216).
2.4
CONDITIONAL SCORE DECOMPOSITION VIA BAYES' THEOREM
Score models be extended to conditional models when conditioned certain label Similar to c, the smoothing kernels with variance 72 can be applied on to meet the requirement that the pdf is everywhere non-zero. Typically, is assumed be sufficiently small that pr
p(y). A popular approach adopted by researchers utilizes Bayes' theorem Pe.r (Ø ()) Pa.r y a Pa /pr (7) to
decompose the conditional score a log Po,r (zlū) into mixture of scores (Nguyen al., 2017). which enables conditional data generation. Following the assumptions in the previous study (Song al. 202Tb). the decomposition can be achieved by taking the gog-ggrdient on both sides of the equation, expressed as follows:
V a log Po,r (zlğ) Va log Po.r (ū) V (w)
Va log Soggprle
(5)
=0
3

---

Published as a conference paper at ICLR 2022
(a) Base
(b) Scaling
(c) Posterior SM
(d) Ours
(e) Oracle
(class (t)
Pusterfor Posteriox (class
Sampled poins
Figure 1: The visualized results on the inter-twining moon dataset. The plots presented the first two rows correspond to the visualized vector fields for the posterior scores of the class Ci (upper crescent) and 02 (lower crescent), respectively. The plots in the third row are the sampled points. Different columns correspond to different experimental settings in Section The detailed configurations are presented in Appendix A.6.2
where Va lo,
(āll) is the posterior score, V Por (g]z) is the likelihood score, and Va log (ā) is the prior score. Base on Eq. (5) conditional score model V log pl (0lū: jr: ø, 0) can be represented as the combination of the log-gradient of differentiable classifier p(ù à 0) and prior score model s( c):
V po = V log ((pl(; s(ē;
This formulation enables conditional data generation using a classifier p( (z; 0) trained with cross- entropy loss (O) Ep.,, (a. -o p( ē;; 0).. and a score model s( Φ) trained with denoising score matching loss LDSM(0) Unfortunately, few previous studies (Nguyen et al. 2017 Dhariwal Nichol 2021) have noticed that the approximation V log p( y: Φ) empinially inaccurate (referred to as the score mismatch issue in this work), and leveraged a scaling factor >> 0 to adjust the likelihood score V log p(p) a 0). Such a scaling factor is hyperparameter that empirically controls the amount of conditional information incorporated during the sampling process (see Appendix A.3). However, this usually causes the diversity of generated data samples to degrade noticeably (Dhariwal
& Nichol 202I, which is later discussed in Section 5.2
In this work, we examine and investigate the score mismatch issue from different perspective, We first offer motivational example to show that the issue remains even with the scaling technique. Then, we theoretically formulate a new loss term that enables a classifier p(ū) žo; 0) to produce better likelihood score estimation.
ANALYSIS ON THE SCORE MISMATCH ISSUE
To further investigate the score mismatch issue, we first leverage motivational experiment the inter-twining moon dataset to examine the extent of the discrepancy between the estimated and true posterior scores. In this experiment, we consider five different methods to calculate the posterior scores denoted as (a)~(e);
(a) Base method. The posterior scores are estimated using a score model s( o) trained with LDSM and classifier p(ğ jē; 0) trained with LCF, described in Eq. (6).
(b) Scaling method. The posterior scores estimated in similar fashion as method (a) except that the scaling technique (Dhariwal & Nichol (02T) is adopted to scale the likelihood scores, i.e., V p( y: 0, x) a n ply az; 0) s(Ø;; Φ), where
0.
(c) Posterior SM method. The posterior scores for different class conditions, in this case, Ci and c2 are separately estimated using different score models s(x; Φi) and s(ē; trained with IDSM- Note that, since this method requires network architectures different from
4

---

Published as a conference paper at ICLR 2022
Table The experimental results on the inter-twining moon dataset. The quality of the sampled data for different methods are measured in terms of the precision and recall metrics. The errors of the score functions for different methods are measured using Eu(a) [Dy O)] and Eu(a; (.) ū)], where U (()) is uniform distribution over a two-dimensional subspace (i.e., the space of each subplot in Fig. D. represents the classes specified in the second row of the table, and Dp and Dr are defined in Eq. 77. The configurations and the hyperparameters for different experimental settings of this experiment are presented in Appendix A.6.2
(a) Base
(b) Scaling
(c) Posterior SM
(d) Ours
(c) Oracle
y
Cr
F
Cy
Sp(a, 9)
Va (og=());;;0 + s(e
V paj 9)
s(2; c)
s(a; 01)
Va log pbyl( (1) stan 6)
Va log,,, (1)
Sula.
Vglogpoaaa 2666
aV, (a)
V 2;
Epral Dn(tt, g]l
0.198
0.197
2.734
2.627
0,063
0.060
0.066
0.064
0.000
0,000
Berca (0,(r s0)
0.190
0.181
2,730
2.618
0.052
0.052
0.000
0,000
Precision
0.94
0.93
0.95
0.94
0.97
0.97
0.96
0.95
1,00
1.00
Recall
1.00
1.00
0.97
0.97
1.00
1.00
1.00
1,00
1.00
1.00
those used in methods (a), (b), and (d), it is only adopted for analytical purposes. For more detailed discussions about this method, please refer to Appendix [A.4
(d) Ours. The posterior scores are estimated similar fashion as method (a) except that
the classifier is trained with the proposed loss function ITotal, which is described in detail in Section
(e) Oracle. The oracle posterior scores are directly computed using the conditional variant of Eq. 2 which is detailed in Appendix A.6.2
Fig. [ visualizees the posterior scores and the sampled points based on the five methods. It is observed that the posterior scores estimated using methods (a) and (b) significantly different from the true posterior scores measured by method (e). This causes the sampled points in methods (a) and (b) to deviate from those sampled based on method (e). On the other hand, the estimated posterior scores and the sampled points in method (c) are relatively similar to those in method (e). The above results therefore suggest that the score mismatch issue is severe under the cases of methods (a) and (b), but is alleviated when method (c) is used.
In order to inspect the potential causes for the differences between the results produced by methods (a), (b), and (c), we incorporate metrics for evaluating the sampling quality and the errors between the scores in an quantitative manner. The sampling quality is evaluated using the precision and recall (Kynkäänniemi et al.] 2019) metrics. On the other hand, the estimation errors of the score functions are measured by the expected values of Dp and Dr, which are formulated according to the Euclidean distances between the estimated scores and the oracle scores. The expressions of Dp and Dr are represented as the following:
Dp 9) [|Sp(ā,)) V Po.
D2 9) = [ā 9) - Va log (ū) +|
where the subscripts / and L denote 'Posterior' and 'Likelihood', while the terms Sp(ă, (α; 9) and Sz (w, 9) correspond to the estimated posterior score and the likelihood score for certain pair (æz, 9). respectively.
Table D presents Sp( 9) and Sz ÿ). the expectations of Dp and Dr. and the precision and recall for different methods. It can be seen that the numerical results in Table (Iree consistent with the observations revealed in Fig. [ since the expectations of Dr are greater in methods (a) and (b) as compared to method (c), and the evaluated recall values in methods (a) and (b) are lower than those method (c). Furthermore, the results Table [ suggest that the possible reasons for the disparity of the posterior scores between methods (a)~~() are twofold. First, adding the scaling factor a to the likelihood scores increases the expected distance between the estimated score and oracle score, which may exacerbate the score mismatch issue. Second, since the main difference between methods (a)~(c) lies in their score- score-matchine matching objectives, i.c., the parameters in methods (a) and (b) optimized using IAE only while the parameters di and d2 of the score models in method (c)
5

---

Published as a conference paper at ICLR 2022
optimized through LDSM, the adoption of the score-matching objective may potentially be the key factor to the success method (c).
The above experimental clues therefore shed light two essential issues to be further explored and dealt with. First, although employing a classifier trained with LCE to assist estimating the oracle posterior score is theoretically feasible, this method may potentially lead to considerable discrepancies in practice. This implies that the score mismatch issue stated in Section 2.4 may be the result of the inaccurate likelihood scores produced by classifier. Second, the comparisons between methods (a), (b), and (c) suggest that score matching may potentially be the solution to the score mismatch issue. Based on these hypotheses, this paper explores an alternative approach to the previous works, called denoising likelihood score matching, which incorporates the mateehn technique when training the classifier to enhance its capability capture the true likelihood scores. In the next section, we detail the theoretical derivation and the property of our method.
DENOISING LIKELIHOOD SCORE MATCHING
In this section, we introduce the proposed denoising likelihood score-matching (DLSM) loss, new training objective that encourages the classifier to capture the true likelihood score. In Section [.1 we derive this loss function and discuss the intuitions behind it. In Section 4.2 we elaborate on the training procedure of DL.MM and highlight the key property and implications of this training objective.
4.1 THE DERIVATION OF THE DENOISING LIKELIHOOD SCORE MATCHING Loss
As discussed in Section 3 a score model trained with the score- matching objective can potentially be beneficial in producing better posterior score estimation. In light of this, classifier may be enhanced if the score-matching process involved during its training procedure. An intuitive way to accomplish this aim is through minimizing the explicit likelihood score- mote-mnt loss LELSM, which defined as the following:
[
LELSM(() = p (ai.j)
V. logp( 0) V Po, Po, (ū (1))
2
This loss term, however, involves the calculation of the true likelihood score V log Po,r 3). whose computational cost grows with respect to the dataset size. In order reduce the computational cost, we follow the derivation of DSM as well as Bayes' theorem, and formulate an alternative objective called DLSM loss (LDLSM):
LDLSM (0) = Ep,, (2.
V logp(ūj + V log V loggp
(9)
Theorem 1. (DLSM(O) LELSM(() + C, where C is constant with respect to 0.
The proof of this theorem is provided in Appendix A.5 Theorem
1 suggests that optimizing 0 with LELSM(() (f) is equivalent to optimizing f with DDLSM (0). In contrast to LELSM- LDLSM can be approximated in computationally fcasible fashion. This is because V log (o (Ø) can be estimated using a score model s (æ;; Φ) trained with IDSM. and Va log Pa (ā(zz) = x² (æc Φ) as described in Section 2.3 As V a log Po (.) and Va log Po (ø ax) can be computed in a tractable manner, the classifier can be updated by minimizing the approximated variant of LDLSM, defined as:
1
LDLSM (0) = E Ep,,. (..))
Va log + s(x: Φ) V log (x)x)(1
(10)
The underlying intuition of LDLSM and LDLSM is to match the likelihood score via matching the posterior score. More specifically, the sum of the first two terms V log p( ã;; 0) and Va log Pa in LDLSM should ideally construct the posterior score (i.e., Eq. 5b) By following the posterior score, the perturbed data sample ø should move towards the clean sample æ, since V log po (#) x)) is equal to the weighted difference + ( æ) (i.c., Eq. 44).. In the next section, we discuss the training procedure with TDLLMM well as the property of

---

Published as a conference paper at ICLR 2022
2))
Stage
Stuge
2
Score modill <(ā; 6)
Classifier pi (.
log
V
Loe
Lasis
Figure 2: The training procedure of the proposed methodology.
4.2 THE APPROXIMATED DLSM OBJECTIVE IN THE CLASSIFIER TRAINING PROCESS
Following the theoretical derivation in Section we next discuss the practical aspects during train- ing, and propose to train the classifier by jointly minimizing the approximated denoising likelihood score-matching loss LDLSM and the cross-entropy loss LCF. In practice, the total training objective of the classifier can be written follows:
(o) LDLSM (0) AL)
(11)
where A > 0 is balancing coefficient. According to Theorem [ employing LDLSM allows the gradients of the classifier estimate the likelihood scores. However, since the computation of the term LDLSM requires the approximation of V log (= from score model s(æ; 6), the training process that utilizes LDLSM' alone is undesirable due its instability on real-world datasets. To reinforce the stability of it, an additional cross-entropy loss LCF is adopted during the training process of the classifier. The advantage ICF is that it leverages the ground truth labels to assist the classifier to learn to match the true likelihood density Pa.r (ú f.), which in turn helps the score estimation to be improved. To validate the above advantage, an ablation analysis are offered in Section [53 to demonstrate the differences between the classifiers trained with Lottal LCE, and LDLSM' The analysis reveals that Lotal does provide the best score- mateematcching results while maintaining the stability.
Fig. 2 depicts a two-stage training procedure adopted in this work. In stage I, a score model s( Φ) is updated using LDSM match V log pa (.) (i.e., Eq. (4). In stage 2, the weights of the trained score model are fixed, and a classifier p(ğ ( 0) is updated using Totaal After these two training stages, s(;; σ) and V log p( ( 0) can then be added together based on Eq. (6) to perform conditional sampling.
EXPERIMENTS
In this section, we present experiments to validate the effectiveness of the proposed method. First, we provide the experimental configurations and describe the baseline methods. Next, evaluate the proposed method quantitatively against the baselines, and demonstrate its improved performance on the Cifar-10 and Cifar-100 datasets. Lastly, we offer an ablation analysis inspect the impacts of LCE and LDLMM in LTtaall
5.1 EXPERIMENTAL SETUPS
In this work, quantitatively compare the base method, scaling method, and our method (i.e., (a), (b), and (d) in Section 3) on the Cifar-10 and Cifar-100 datasets, which contain real-world RGB images with 10 and 100 categories of objects. respectively. For both of the datasets, the image size is 32x3, and the pixel values in an image are first rescaled to [-1, before training. For the sampling algorithm, we adopt the predictor-correcte (PC) sampler described in (Song et al.) 202Th) with the sampling steps set to 7=1,000. The score model architecture is exactly the same as the one used (Song et al., 2021b), while the architecture of the classifier based on ResNet (He et al. 2016) with conditional branch for encoding the information of the standard deviation (Song et al. 2021b). When training the classifier, 7 is assumed to be sufficiently small, thus the unperturbed labels adopted. Furthermore, the balancing coefficient À is set to 1 for the Cifar- 10 and Cifar- datasets, and 0.5 for the motivational example. For more details about our implementation and additional experimental results, please refer to Appendix An6land Appendix A.7
7

---

Published as a conference paper at ICLR 2022
Base
Table 2: The evaluation results on the Cifar-10 and Cifar-100 datasets. The P/ R D/ C metrics with (CW)' in the last four rows represents the average class-wise metrics described in Section 5.2 The arrow symbols represent that higher lower evaluation result correspond to a better performance.
Scaling
Cifar-10 Scaling
Cifar- 100
Base
Ours
Base
Sculing
Ours
FID
4.10
12.48
2.25
4.52
12.58
3.86
IS
9.08
9.37
9.90
11.53
11.59
11.62
Precision
0.67
0.75
0.65
0.62
0.65
0.61
Recall
0.61
0.49
0.62
0.62
0.52
0.63
Density
1.05
1.36
0.96
0.84
0.93
0.82
A comparison of the curated samples generated via the base method and the scaling method. As for the uncurated visu- alized samples, please refer to Ap-
Figure 3:
Coverage
0.80
0.75
0.81
0.71
0.61
0.71
CAS
0.38
0.46
0.58
0.15
0.24
0.28
(CW) Precision
0.51
0.70
0.56
0.33
0.59
0.42
(CW) Recall
0.59
0.42
0.61
0.68
0.41
0.68
(CW) Density
0.63
1,23
0.76
0.30
0.78
0.43
(CW) Coverage
0.60
0.66
0.71
0.38
0.51
0.52
pendix A.7.4
5.2 RESULTS ON CIFAR-10 AND CIFAR-100
In this section, we examine the effectiveness of the base method, the scaling method, and our proposed method on the Cifar-10 and Cifar- benchmarks with several key evaluation metrics. We adopt the Inception Score (IS) (Barratt & Sharma 2018) and the Fréchet Inception Distance (FID) (Heusel et al., 2017) as the metries for evaluating the overall sampling quality by comparing the similarity between the distributions of the generated images and the real images. We also evaluate the methods using the Precision (P), Recall (R) (Kynkäänniemi al. 2019), Density (D), and Coverage (C) (Naeem et al., 2020 metrics to further examine the fidelity and diversity of the generated images. In addition, we report the Classification Accuracy Score (CAS) (Ravuri & Vinyals (209) to measure if the generated samples bear representative class information. Given a dataset containing m, images for each class i, we first conditionally generate the same number of images (i.e., m;) for each class. The FID, IS, P P R /D/ C, and CAS metrics are then evaluated based on all the generated m = Σi= my images, where is the total number of classes. In other words, the above metrics are evaluated in an unconditional manner.
Table T reports the quantitative results of the above methods. is observed that the proposed method outperforms the other two methods with substantial margins in terms of FID and IS, indicating that the generated samples bear closer resemblance to the real data. Meanwhile, for the P D C metrics, the scaling method is superior to the other two methods in terms of the fidelity metrics (i.e., precision and density). However, this method may cause the diversity of the generated images to degrade, as depieted in Fig. resulting in significant performance drops for the diversity metrics (i.e., the recall and the coverage metrics).
Another insight is that the base method achieves relatively better performance on the precision and density metrics in comparison to our method. However, it fails to deliver analogous tendency on the CAS metrie. This behavior indicates that the base method may be susceptible to generating false positive samples, since the evaluation of the P R / D/ C metrics does not involve the class information, and thus may fail consider samples with wrong classes. Such a phenomenon motivates us to further introduce a set of class-wise (CW) metrics, which takes the class information into account by evaluating the P/R/ / / C metrics on per-class basis. Specifically, the class-wise metrics are evaluated separately for each class i with m. images. The evaluation results of these metrics shown in Fig. 4 reveal that our method outperforms the base method for a majority of classes. Moreover, the results of the average class-wise metrics presented in the last four rows of Table 2 also show that our method yields a better performance as compared to the base method.
Base on these evidences, it can be concluded that the proposed method outperforms both baseline methods in terms of FID and IS, implying that our method does possess a better ability to capture the true data distribution. Additionally, the evaluation results on the CAS and the class-wise metrics suggest that our method does offer a superior ability for a classifier to learn accurate class information as compared to the base method.
8

---

Published as a conference paper at ICLR 2022
Cifar-10
Cifar-100
0:25
6:3-
0.27
0.15
0.3
663-
6:5-
0.64
0.10
1.62
0.31
8.30
Dandty
Caverage
0.4
028
0.39
028-
006
4.55
20
40
110
Sentod
Figure 4: The sorted differences between the proposed method and the base method evaluated on the Cifar-10 10 and Cifar-100 100 datasets for the class-wise P R D/ C metrics. Each colored bar in the plots represents the differences hetween our method and the base method evaluated using one of the P R D/ C metrics for certain class. A positive difference represents that our method outperforms the base method for that class.
5.3 ABLATION ANALYSIS
To further investigate the characteristic of LTotal we perform an ablation analysis on the two components LEE and LDLSM in TTotl using the inter-twinning moon dataset, and observe the trends of (a) the score
0335
0230
4375
4150
0135
errors (x) [Dz (it ct)] ( [D (z, o2)]. and
(b) the cross entropy Ep... (ā. (a, - log p(ù) @ 0)] dur- ing the training iterations. Metrics (a) and (b) mea- sure the accuracies of the estimated likelihood scores Va log p(ū i;; 8) and the estimated likelihood density p( (;; 0), respectively. As depicted in Fig 5 an in- creasing trend is observed for metric (a) when the classifier is trained with ICE alone. On the contrary, the opposite tendency is observed for those trained with LTtaa and LDLSM The results thus suggest that matching the likelihood scores implicitly through train- ing a classifier with LeE alone leads to larger approx- imation errors. In contrast, LDLSM and Lotta explic- itly encourage the classifier to capture accurate like- lihood scores, since they involve the acore-matching objective. On the other hand, for metric (b), the classifiers trained with T-Totl and LCEE yield stable decreasing trend in comparison to that trained with LDLSM alone. These results suggest that in addition to minimizing LDLSM' between V log Po, ( and V log pl ( 0), the utilization of LCE an auxiliary objective enhances the stability during training. Based on the above observations, the interplay of LCE and LDLSM synergistically achieves the best results in terms of metric (a) while ensuring the stability of the training process. These clues thus validate the adoption of L during the training of the classifier.
1.0
0.4
5900
10000
15000
20000
25000
30000)
35000
Step
Figure 5: The evaluation curves of (a) the score errors and (b) the cross entropy during the training iterations for LCE, LDLSM, and L The curves depict the mean and 95% confidence interval of five times of training.
CONCLUSION
In this paper, we highlighted the score mismatch issue in the existing conditional score-based data generation methods, and theoretically derived a new denoising likelihood score-matching (DLSM) loss, which is training objective for the classifier to match the true likelihood score. We have demonstrated that, by adopting the proposed DLSM loss, the likelihood scores can be better estimated, and the negative impact of the score mismatch issue be alleviated. Our experimental results validated that the proposed method does offer benefits in producing higher-qualit, conditional sampling results both Cifar-10 10 and Cifar- 100 datasets.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the Ministry of Science and Technology (MOST) in Taiwan under grant number MOST 11 11-2628-EE000-000. The authors acknowledge the financial support from MediaTek Inc., Taiwan. The authors would also like to acknowledge the donation of the GPUs from NVIDIA Corporation and NVIDIA AI Technology Center (NVAITC) used in this research work. The authors thank National Center for High-Peeformance Computing (NCHC) for providing computational and storage resources. Finally, the authors would also like to thank the time and effort of the annoymous reviewers for reviewing this paper.
REFERENCES
S. Barratt and R. Sharma. A note on the inception score. 2018.
P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.088 0553333 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of the Conf. Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. In Proc. of Conf. on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proc. of Conf. on Neural Information Processing Systems (NeurIPS). 2020.
J. Ho, Chitwan S., W. Chan, D. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. ArXiv, abs/2106.15282 2021.
A. Hyvärinen, Estimation of non nor-nooaalize statisicaal models by score matching. Journal of Machine Learning Research (JMLR), 6(24):695-709, 2005. URL PRLL:tt::/// /tmttp://jm...... wwt.:/gpp..//...........
hyvarinen05a. html
V. Jayaram and J. Thickstun. Source separation with deep generative priors. In Proc. of the Int. Conf. on Machine Learning (ICML), 2020.
A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
Kynkäänniemi, T. Karras, S. Laine, Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. In Proc. of Conf. on Neural Information Processing Systems (NeurIPS), 2019.
M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. Proc, of the Int. Conf. on Machine Learning (ICML), 2020.
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 4467-4477, 2017.
E. Parzen. On estimation of probability density function and mode. In The annals of mathematical statistics, volume 33, pp. 1065-1076. JSTOR, 1962.
S. Ravuri and O. Vinyals. Classification accuracy score for conditional generative models. In Proc. of Conf. Neural Information Processing Systems (NeurIPS). 2019.
G. Roberts and Rosenthal. Optimal scaling of discrete approximations langevin diffusions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(1):255-2688 1998.
G. O Roberts and R. L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341 363, 1996. doi: bj/1178291835. URL https 7/doi.org)
10

---

Published as a conference paper at ICLR 2022
J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In Proc. of the Int. Conf. Machine Learning (ICML), 2021a.
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2019.
Y. Song and S. Ermon. Improved techniques for training score-based generative models. In Proc. of Conf. on Neural Information Processing Systems (NeurlPS), 2020.
Y. Song, S. Garg. J. Shi, and S. Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proc. of the Conf. on Uncertainty Artificial Intelligence (UAI), pp. 204, 2019. URL http:://aaaaa org. ua 1201 9/ http///////eeeeeneeeeee ngg pttp://ps/////ppappapaaaaaee
pdf
Y. Song, Jascha S.-D., Diederik K., Abhishek K., Stefano E., and Ben P. Score-based generative modeling through stochastic differential equations. In Int. Conf. on Learning Representations (ICLR), 2021b. /ttLtttt////peereeeees
Vincent. connection between score matching and denoising autoencoders. Neural computation, (317):161111774 2011.
M. Welling and Y. W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 681-688. Citeseer, 2011.

---

Published as a conference paper at ICLR 2022
A APPENDIX
In this Appendix, we first provide the definitions for the symbols used in the main manuscript and the Appendix in Section A.I Next, we derive the theoretical proof for the closed form of V log Pa (#) in Section A.2 Then, we offer a number of discussions on the scaling technique in Section A.33 and different conditional score-based data generation methods in Section Subsequently, we offer the theoretical proof for Theorem iin Section A.3] Finally, in Sections detailed experimental setups and additional experimental results.
A.7 we provide the
A.1 THE LIST OF SYMBOLS USED IN THE PAPER
In this section, we provide the list of symbols used throughout the main manuscript and the Appendix. These symbols are summarized in Tables A1 and A2 containing their notations as well as their detailed descriptions.
Symbol
Description
(u, v) = u unv v= E, 4idn ||ul| V/u,2. u)
dot (inner) product hetween two vectors. Euclidcan norm of vector. identity matrix with dimension d x d. the step size used in Langevin Diffusion. the discrete timestep used in Langevin Diffusion.
Taxd
6 1 E {0, T}
4, 4t, IR"
data sample with dimension d, where ao sampled from (0,, Laxa)-
E Rd
perturbed data sample used Denoising Score Matching. conditional class label with dimension c. perturbed class label used in Denoising Score Matching.
y E IR
V E R
R
the noise vector used in Langevin Diffusion. (sampled from (0, Laxa))
(10) E 12d
the i-th example (input) from dataset ID. the class label associated with (1) for supervised learning (one-hot encoded).
( E (0, 13
=(x y) ()), ... (m) (())))))))))) (m))
Training set: data-label pairs from Pdata- (i.i.d. samples from Pdata)
Pdata (x:)
unknown true probability density function (pdf) data samples. empirical pdf of data samples associated with the training set D. smoothing kemel or noise model: sampled from isotropic Gaussian N(x o Laxa) with mean and variance o². joint pdf of perturbed data and clean data samples.
Po(x)) 1
m - 8(1x (1)
Po( .)) (2
+oي is =l1
Po x) Pe p=(0)))oo())
Po (#) 1 Σ Pa ( )))
Parzen density estimate based on D, obtainable by marginalizing Po (i ())
Pylan(y)
unknown true pdf of data labels.
Po(y)= + Σm 5(11g 11
empirical pdf of data labels associated with the training set D.
(rll)))) (1m)o/2ş0 خيج ツリ
smoothing kernel or noise model: sampled from isotropic Gaussian N V(y, 7 Texe) with mean and variance ??. joint pdf of perturbed labels and clean labels.
Pr (ÿ. y) = pa (oly)oo)))
P- (ū) 1n "m= P- (1)
Parzen density estimate based on D, obtainable by marginalizing
Pr (i. 3).
Po.o(ar. It. y)
defined as 1 EE _01 " ())((ll "I))
m equals to p. (#(x)po.ole (.,y). y)-
Po.o (T, x, y)
Po. (ar, Ø. y)
equals to p- (7) Po y).
Po.- (#, x. Ú. v)
defined as Pe (6) (ol
s(;;))
Score Model: approximated score function with parameters d.
plū [6;
Classifier:
approximated probability function parameterized by 8.
Table A1: The list of symbols used in this paper.
12

---

Published as a conference paper at ICLR 2022
the scaling factor.
Va log p( ( y
posterior score.
Va log p( y [))
likelihood score.
V log p( m
prior score.
LESM (+)
Explicit Score Matching (ESM) loss optimizing s( Φ).
LDSM (x)
Denoising Score Matching loss (DSM) for optimizing s( s).
LELSM
Explicit Likelihond Score Matching (EL.M) loss for optimizing p(yl az; 0).
(DLSM(()
Denoising Likelihood Score Matching (DLSM) loss for optimizing p(y àē; 0).
LDLSM
approximated Denoising Likelihood Score Matching loss for optimizing p ãz; 0).
[C)(O)
Cross Entropy (CF) loss for optimizing p(y ( 0) full loss for optimizing p(y|0; 0).
LToaa (0)
Table A2: The list of symbols used in this paper.
A.2
A PROOF OF THE CLOSED FORM FOR Va Pr())
In Sections 2.2 and we utilized the closed form formula to calculate the true scores Va lo po (t).
To show the equivalence holds for Eq. (2). we provide a formal derivation as shown in the following proposition.
Proposition 1. Given Po (z) = 1/ E Pa (z) (0)x))), Po (x] = (2m)d/aga e Tol 1 [m==] and a > 0,
the closed form of Va logpo (i) can be represented as follows:
Ei zz (xx (z)=(())
V (o
E Po (z)
Proof.
1 m
Va log
V log
Po (0) (1)
m
1
V log
Per (क) (0)
Va log
1m
V Σn
Po (+)(xx))))
- Pe (2) (+))
VaPo (2 (1)
-= Per @ (+)))
11
20 v- (+) 11
i=l
V
(2m (2ad
Σ Po ( (+)))
(1)()) /0²
e t -
D
(2m) 4/2
>
Pe (x (z()))
7² (x((
(i)
i po (z
ii (
A.3 A DISCUSSION ON THE IMPACTS OF THE SCALING TECHNIQUE
In this section, we first provide a proof to demonstrate the reason behind the non-equivalence between the scaling technique and the re-normalization of likelihood density. Then, we present the
13

---

Published as a conference paper at ICLR 2022
derivation of the posterior scores based re-normalized likelihood density. Finally, we adopt low-dimensional example to explain the effect of re-normalization on a posterior density.
The authors in (Dhariwal & Nichol 2021) leveraged a scaling factor > 0 to adjust the likelihood score V~ log p))(z) To demonstrate the effect of such a scaling factor a, the authors claimed that there always exists normalizing constant Z such that a log p(y æ) V log { where p is a re-normalized likelihood density. However, such an equality only holds for specific conditions, and is not applicable to general cases, as explained in Proposition
Proposition 2. Given constamt > O, and a probability density function (pdf) { p(z)(z) that is everywhere non-zero and differentiable, there does not exist a constant Z. such that zPly ()) is pdf for all a in general.
Proof. The proposition can be proved by contradiction. Assume that such constant Z exists and consider the following case:
a = 2,
x:, TR
f(y: f(y:p,o7) (2TT0) -(w-p) 20
(o (m+1) + 1) (11))) 2
plyl)) 2(x²2 + 1)
Note that p(ylæ) a pdf and is everywhere non-zero and differentiable.
Since a pdf sums to 1 (i.e.. I, 1 dl = 1, Vx), by utilizing the Gaussian integral and the above assumption, the following derivation holds:
1
1 | 1.2 dy
7 plyl (-1))"
=
42 = p(u((- p(ul(-1))2dy p(y)
4
[w: 0. 1 )
2.
J, [v; 0, 1)
4 2
(TT)
3 --0²) dy / (2T))
(2m)
7 +²²
dy
1
1
-x²²
42
dy
dy
2x
y
4
#2
1
2y² dy
1
v2 dy
π y
2T
1
-- 1²
$ 2
du
dy
V2T
2T
1
1
$
2
VT
VT
V
2M
2m
1
$ 7
V2T
2/T
The final equation leads to a contradiction for the value of Z, which completes the proof of Proposi- tion 2 Therefore, we concluded that there does not exist constant Z such that p (z) is pdf for all æ in general.
To the best of our knowledge, the scaling technique still lacks theoretical justification. However, if Z is allowed to be a function of x rather than a constant, re-normalization of the likelihood density becomes feasible. In other words, setting Z Z(x) I,y p(yl dy enables S, Z " dy 1.
14

---

Published as a conference paper at ICLR 2022
p M/)
p(a: (4,
p(z)
Pa M.)
P. (a: 14a)
Original
Cc
033
0:5
0.3
645
055
000
668
000
Figure Al: The original and the renormalized densities under different choices of a.
1 plyla) (a (z) is re- re-normalized likelihood density, In this case, the score of the where 2(x)
re-normalized likelihood density can be written as,
V log Pa (3) v log
Z(xx (()))(())))))
Vn (ogg(yla)) log Z (22)
(A1)
Va logp(()x)) v log ((x)
a logp(y
V log Z(x).
Given the joint density (2, y) (y () (<)), the posterior of the re-normalized density can be derived by Bayes' Theorem, i.e., Pa (xr yy) = Pa (y) (p))))((())))))))))))))) Pa (x, y). Based on Eq. (5), the posterior score of the re-normalized density can be written as:
Vn log Pa (mly) vr Po (y|=) + Va logp(æ:) V (g)l) V. log ((x) + Va logp(æ).
(A2)
Through the above equation, conditional sampling based on the re-normalized likelihood density becomes feasible.
To further investigate the effect of re-normalization on P., (xly), we provide an illustrative example in Fig. A1 In this example, the condition y is assumed be sampled from (y1. with equal probability 1/2. The original posterior densities ((xly)) and p(x))2)) are normal distributions with variance 2.25 centered at æ -.5 and æ 1.5, which are plotted in orange and green, respectively, On the other hand, the prior density p(æ) A ( 1 (y2) is plotted in blue, and the likelihood density p(y x)) p( ( (y))))yyy))))))))))))))))))))( pp))) ) is defined according to Bayes' theorem.
Subsequently, the re-normalized posterior densities pa (arlyn) and Pa (ar y2) be constructed based on the equation p ( x= P( ))) and Bayes' theorem. In Fig. Al it is observed that under the case that 5, the re-normalized posterior densities are more concentrated. Conversely, when a = 0.2, the re-normalized posterior densities are much more dispersed. These results therefore suggest that the re-normalized posterior density can be adjusted by controlling the value of a.
A.4 CONDITIONAL SCORE-BASED DATA GENERATION METHODS
As described in Section 3 methods (a), (b), and (d) utilize a classifier p( ãē; () and a score model s(ē; σ) to estimate the posterior scores. In contrast, method (c) separately trains the posterior score models s s(z; Φ1) and sa;; Φ2) for different class conditions. Although method (c) achieves better evaluation results, it requires training score models for each possible which is usually considered impractical in real-world applications. A potential solution to this problem to train a score model s(Ø, Di σ) that can estimate the scores for the joint probability of ø and (denoted as 'Method (c*)' in Fig. A2 This method, however, is inflexible in comparison to methods (a), (b), and (d), since any change the condition could lead to re- training the joint score model s a y: Ø). Therefore. in this work, we only compare the performance methods (a), (b), and (d), and consider method (c) to be another orthogonal research direction.
15

---

Published as a conference paper at ICLR 2022
Methods (a), (b), (d)
Method (c)
Method (c*)
Score model
Score model s <(:; (9)
Score model s( D: 6)
a(2; d))
Classifier p(g 0)
Score model s( %
Figure A2: An illustration for different conditional -core-based data generation methods in Sec- tion
A.4
A.5 A PROOF FOR DENOISING LIKELIHOOD SCORE MATCHING
In Section we stated the equivalence between (ELSM and LDLSM (0) when optimizing O. To demonstrate how such an equivalence stands, we provide the theoretical proof as follows. Theorem 1. (DLSM(() (ELSM(O) + C, where C' is constant with respect to O.
Proof. (0 and (DLSM( are defined as follows:
1
LELSM (0) Ep.. (0.5
Va logp(ylã; V Po. Po,r (p). (1)=)11
2
(DLSM(0) Epe. I log 0) + Va (5) V lo (1)
The proof expands (ELSM(() and LDLSM (0) individually, leading to constant difference.
(f) Ep., (
V a log logp(g)(;;)) V log Po,r (ū x
Ep.. (a.
V a log p( ( 1)
F(0) + C1, where
C p..., (7.0)
V log Pe, Po.- ( (1)
is a constant, and
F(0) pp... ( i) log
V log (g (ū])))
Ep..- ( ;) pog 0), v log (#lg) V log (*))
G(f) Ep,, (5. (V
(V log V (m)), where
G( Ep... ( log Ve log (+(g))]
Pa.r (w. 0) Vo
log (ā
/
pr ( log V VaPo. (i 9) pa.- (x 9) dÿdē
1
/
pr (1) logp(gla; 8), Vapa. (ol))
pr (5)
V
po,r (x:
(xl Po.r (alæ, ÿ (x)d
pr (ū)
V
Po, (o. Po.- (x) x. ÿ. yddddr) dÿjd
p- (+( (gg
po.- (a.r 9) (āran,, ū. dyddd) dÿjdä
16

---

Published as a conference paper at ICLR 2022
Pr (1)po, (y/a, p), v. log 0), Va loggo, dyddddddd
Já 1
Po.r (i, n. logp 0), V logg (0l y dydaddd
[
Po (z, a. y. logp(yla; 0), V logpo, (ä, æ. y, y) dydrdjdē
9 Po, (z. a. y. log 2) 0). Va log(ps (p Po.u y) () dydaddda
Po. (2,
V log Pa (3)
Ep., (... (VV log
lo and now we expand (DLSM(0)
(DLSM(()) = po.. (a, (.3)
V log
V log po V (x|))
12
=B Be. (
V logp m; 0)1
E,, (.) (V log púg Va log
pe (m.g)
(Va # logr 2; 0), V log (+)) + C2, where
C2 Ep (f.m)
Va log
Ep (ش),5t
V l
2
2
p..
( n lo
F log [ is a constant, leading to the conclusion
(DLSM(() = LELSM C1 C22
Note that the domain of the integrals is {(0: x. ÿ. y) Pa,r (t. x, .. + 0} by the definition of expected value.
Remark.
Theorem also holds for the clean label which can be shown as follows:
Ep, (i.v)
2 v logg(yl(;;)) 0) V (z)
=Ep Ep.
logply V log - V log logpo + C.
A.6 THE DETAILED EXPERIMENTAL. CONFIGURATIONS
In this section, we elaborate on the implementation details and the experimental setups. Sec- 1 we describe the noise perturbation method and the sampling algorithm adopted in this work. In Sections and.2]andAA66..
we present the training configurations for the experiments on the inter- twinning moon, Cifar- 10, and Cifar-100 datasets. The code implementation for these experiments is provided in the Github repository poctts://githhe com/ chen hao chao/ dllm).
A.6.1 NOISE PERTURBATION AND SAMPLING ALGORITHM
this work, we adopt the PC sampler (VE-SDE) identical to that in (Song et al. 2b21b), which is a time-inhomogeneous sampling algorithm. The PC sampler (VE-SDE) is described in Algorithm 1. In Algorithm 1, a moise-conditionn score model (;; Φ, oe) is adopted, where W is perturbed by noises sampled from Gaussian distributions using different standard devia- tions ot. The standard deviations of the noises, denoted d; auin Guis
Sampling Algorithm.
(mm ) determined by the
time step t in Algorithm 1. We d min 0.01 for all of our experiments. On the other hand, set maax 50 for the Cifar- 10 and Cifar- 100 datasets, and Tmax = 10 for the inter-twinne twinning moom dataset. As for the total time steps, the value of 7 is set 1,000.
17

---

Published as a conference paper at ICLR 2022
Algorithm 1 The PC Sampler (VE-SDE)
I =V mmax I)
2: for t = to do
3:
1* Predictor */
4:
2 2~ (0.
ãx - ã. + (02+1 of) (1 VoL - -0
5:
6:
1 Corrector #/
7:
I~.1 3t f कt + & ((t - 22Ez
8:
9:
return To
Training with Different Standard Deviations. Since the PC sampler (VB-SDE) described in Al- gorithm 1 requires the score model s(ă; Ø, σ,) conditioned on different values of standard deviation Tr the training objectives described in Eqs. 44 and (TT) modified accordingly. For the DSM loss
described in Eq. (4). the modified loss LDSM (Ø) is represented as:
(() (tD (.,=))
[1): Φ, ox) V log [
(A3)
where (1) is uniform distribution on the interval [0, T). In Eq. A3 The additional coefficient (() multiplied with the original DSM loss for better training results (Song & Ermon 2019).
For the approximated DLSM loss described in Eq. (9. the modified loss IDISMI (0) is represented as:
Eeg. [ADLSM (G..s.. [ a (i + s(ē; 0.) V logPo (o 2 [3]]]
(A4)
where p(g o;; 0, σe) noise-conditioned classifier. ADLSM (f) is the balancing coefficient for different os For the cross-entropy loss, the modified loss LCE (0) is represented as:
(Mo,r((t) (t.. log
o,)]]
(A5)
where (!) is the balancing coefficient for different o, In all of our experiments, ACE(E) (4) is set to The values of ADSM and ÄDLSM (t) are customized according to different settings, and specified in Sections Sections|A.6.2) and A.6.3 Please note that ADsM ADLSM (t), and ACE(t) are adopted to balance the loss functions for different at, and are different from the balancing coefficient À in Eq.
A.6.2 THE EXPERIMENTS ON THE INTER-TWINNING MOON DATASET
Dataset. We perform our motivational experiments the inter-twinning moon dataset provided by the sklearn library in python. In this dataset, the data points are sampled from two interleaving crescents. The data points of the upper crescent are generated based (cos(1), sin(x)), Vo [0, n. and are labeled as On the other hand, the data points of the lower crescent are generated based on cos(x). 0.5 sin(x)), Vz [0, x and are labeled as c2. Before training, the data points are first normalized such that their mean become O and then their coordinates are scaled by factor of 20.
Setups and Evaluation Method.
this experiment, the scores Va log Po (æ) and V log Pa (ālg) 2 The likelihood scores are computed according to Va Po Po (w) Eq. (5). As for the Dp and Dr, metrics
are obtained by calculating Eq. (ū .) = V log Por ( ÿ)
V log (1ū
in [ the errors between two score functions are measured using 1,225 data points uniformly sampled from the two-dimensional subspace [-25, 25] x [-40, 40). The scaling factor a adopted in method (b) is set to 10. The values of (D) and ADLSM (t) are both set to and the value of X set to 025.
or.
Training and Implementation Details. The network architectures for the score model and the classifier are simple three-layer multilayer perceptrons (MLPs) with ReLu as the activation function. These networks are implemented using the pytorch library with nn Linear layers that contain (128, 64, 32) neurons per layer for both of the score model and the classifier. The score model is trained using the Adam optimizer with a learning rate of 6.5 10-4 4 and batch size of 4,000 for 40,000 iterations. The classifier is trained using the Adam optimizer with a learning rate 2.0 x 10-5 and batch size of 4,000 for 40,000 iterations.
18

---

Published as a conference paper at ICLR 2022
Table A3: The experimental results on the inter-twining moon dataset, where the classifiers are trained with LcE. LDLSM', and Irotal, respectively. The enrors of the score functions for different methods are measured using Ev(a) [ā, 8)] and ( (ă) ÿ)], where U (@)) over two-dimensional subspace, represents the classes specified in the second row of the table, and Dp and Dr, are defined in Eq. The arrow symbol + indicates that lower values correspond to better performances.
uniform distribution
LCe
Lmsssw
L Total
Eoraa Du( ūi]
1
0.198
0.197
0.066
0.065
0.066
0.064
Eu Pu(# g)
0.190
0.181
0.053
0.053
0.052
0.052
A.6.3 THE EXPERIMENTS ON THE CIFAR-10 AND CIFAR-100 DATASETS
Dataset. We adopt the Cifar- 10 and Cifar- datasets Krizhevsky et al. 2009) for evaluating our design image generation tasks. Cifar-10 consists of 60,000 32 32 RGB images with 10 classes, where each class contains 6,000 images. Cifar-100 similar to Cifar- except it has 100 classes, where each class contains 600 images. The images first normalized to [-1, 1] before training.
Setups and Evaluation Method. The results the CAS metric evaluated using classifier (ResNet-50 (He et al. 2016)) trained with the generated samples. This classifier is trained to minimize the cross-entropy loss with the Adam optimizer using learning rate of 2.0 10-4 with batch size of 150. The results on the P/ R D C metrics are calculated using the officially released code The hyperparameter & used in the P/R D C metrics is fixed 3. The results on the FID and IS metrics are computed on 50,000 samples using the tensorf f flow gan library in python. The value of (t) set to of while the value of (TLSM(t) is set 1. The value of À is set
Training and Implementation Details. The network architecture for the score model is the same as (Song et al. 2021b). The network architecture for the classifier is a modified ResNet-18 (He) et al.][]016 for Cifar-10 10 and modified ResNet-34 for Cifar-100 The modification only involves an additional conditional branch for encoding the standard deviation o,. The score model is trained with the Adam optimizer using a learning rate of 2.0 10-4 and a batch size of 128. The classifier is trained with the Adam optimizer using a learning rate of 2.0 x 10-4 and batch size of 100.
A.7 ADDITIONAL EXPERIMENTS
In this section, we present a number of additional experimental results and provide discussions on them. We first report the results of our method using different values of on Cifar-10 10 to demonstrate that ToTtaal is less sensitive to the choices of X. Next, we apply the scaling technique on both the base method and our method with different choices of a to demonstrate the influence of the hyperparameter the evaluation results of several metrics. Finally, provide uncurated examples on the Cifar-10 10 and 100 datasets.
A.7.1 ABLATION ANALYSIS ON THE INTER- INTER-TWINNING TWINNING MOON DATASET
Table A3 compares the evaluation results on the (x) DD( 7)] and Ey ( 5)] metrics when the classifiers are trained with LEE, LDLSM' and Lotall respectively. is observed that the score errors ( DD((t, (w, y)] and Erra)) [Dy(ă, ÿ)] are the lowest when L is adopted. The
(Dr((;
experimental results therefore validate the effectiveness of the proposed loss TeTotal-
A.7.2 THE EXPERIMENTS ON THE BALANCING COEFFICIENT
Table A4 demonstrates the evaluation results on FID, IS, and P R / D/ C metrics using Lrotal with different values of X. It is observed that the evaluation results only slightly fluctuate the six metries with different choices of A. The results thus experimentally validate that Loatal is less sensitive to the choice of X.
19

---

Published as a conference paper at ICLR 2022
Table A4: The experimental results of our method using Total with different value of X. The arrow symbols 1 / represent that higher lower evaluation result correspond to better performance.
À 2.0
À
1.0
À = 0.5
À
0,125
FID
2.33
2.25
2.36
2.25
IS
9.90
9.90
9.98
9.79
Precision
0.64
0.65
0.64
0.64
Recall
0.62
0.62
0.62
0.62
Density
0.94
0.96
0.95
0.94
Coverage
0.79
0.81
0.79
0.78
Table A5: A comparison of the evaluation results of the base method, the scaling method with different a, ours method, and ours + scaling method with different a on the Cifar- 10 dataset. The arrow symbols represent that higher lower evaluation result correspond to better performance.
Base
Scaling
Ours
Ours + Scaling
1.0
0.5
3,0
5.0
10.0
20.0
1.0
0.5
3.0
5.0
10.0
20.0
FID
4.10
3.22
6.16
8.06
12:48
2.25
2.57
3.96
6.40
11.06
14.92
9.08
9.50
9.14
9.38
9.37
9.23
9.90
9.79
10.00
9.95
9.73
Precision
0.67
0.65
0.70
0.72
0.75
0.75
0.65
0.62
0.69
0.71
0.73
0.74
Recall
0.61
0.61
0.56
0.53
0,49
0.44
0.62
0.63
0.57
0.53
0.49
0.44
Density
1.05
0.98
1.17
1.25
1.36
1.36
0.96
0.87
1.15
1.23
1.28
1.30
Coverage
0.80
0.78
0.78
0.77
0.75
0.67
0.81
0.77
0.79
0.77
0.71
0.66
(CW) Precision
0.51
0.43
0.63
0.67
0.70
0.73
0.56
0.48
0.65
0.68
0.71
0.72
(CW) Recall
0.59
0.42
0.37
0.61
0.55
0.50
0.39
(CW) Density
0.46
0.97
1.10
0.76
0.58
1.16
1.26
(CW) Coverage
0.60
0.47
0.69
0.69
0.66
0.61
0.71
0.61
0.75
0.73
0.66
0.61
A.7.3 THE EXPERIMENTS ON THE SCALING TECHNIQUE
Table A5 demonstrates the evaluation results of the base method, the scaling method using different values of a, our method, and our method with the scaling technique using different values of a (i.e.. 'ours + scaling') on the Cifar- 10 dataset. It is observed that the performance of the 'scaling method' and 'ours + scaling' method on the fidelity metrics, i.c., precision and density, improves the value of a grows. Conversely, the performance of the 'scaling method' and 'ours scaling' method on the FID, recall, and coverage metrics degrades as the value of a grows. This results thus experimentally demonstrate the effects of the scaling technique the performance of the base method and our method.
A.7.4 UNCURATED EXAMPIEES
Figs. A3 A5 of different methods.
depict a few uncurated examples that qualitatively compare the effectiveness
20

---

Published as a conference paper at ICLR 2022
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure A3: The uncurated examples generated using the base method on the Cifar- dataset.
21

---

Published as a conference paper at ICLR 2022
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure A4: The uncurated examples generated using the scaling method on the Cifar- 10 dataset.
22

---

Published as a conference paper at ICLR 2022
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure A5: The uncurated examples generated using our method on the Cifar- dataset.
23

---

Published as a conference paper at ICLR 2022
Base
Scaling
Ours
Figure A6: The uncurated examples of classes "baby", 'apple", 'aquarium fish', and 'beargenerated' using the base method, the scaling method, and method the Cifar- 100 dataset.
24


Reviews:
[{"review_id": "qbQ9f5aMNwr", "reviewer": "Reviewer_UXBu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\nI have a question on the derivation of Eq (A1). Particularly, authors suddenly introduce $Z(\\tilde{x})$. It will be great if authors can explain how to main the equality, after the introduction of $Z$.\n\n2.\nIt seems that the scores are mismatched across conditions. For example, the score on condition A cannot be matched to the score on condition B, which is quite natural. Given that it is good to see that the authors performed the posterior score-matching by separating the score-matching by conditions. At lease, Figure 1 shows not much difference between posterior SM and suggested models. Also, Table 1 suggests not much difference (or rather favorable to (c) posterior SM than the suggested models) between the posterior SM and the suggested model.\n\nThen, what would be the gain by not just using posterior SM and by following the suggested models?\n\n3.\nI cannot find the clear definition of $L_{CE}$ which would be basically classifier, but it would be better to give a clear form of classifier with inputs and outputs.\n\n4.\nThe experiment does not report negative loglikelihood, or bpd. Why don't you report NLL?\n\n5.\nThere is no evaluation through the comparisons with the baseline models. As authors suggested that there are recent rush on the conditional modeling with diffusion approaches, why don't you find some and compare the performance with them?\n", "summary_of_the_paper": "This paper presents a conditional diffusion(score-matching) model to tackle the score mismatch issue in the conditional generation scenario. This paper tests multiple alternatives in creating a conditional distribution through diffusion models, and this paper introduces a classifier assisted structure for the conditioning.", "summary_of_the_review": "Niche paper to introduce a conditional generative model by the diffusion process", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "P-a7-gupzto", "reviewer": "Reviewer_LMQh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some disadvantages\n\n1) The loss is quite noisy (see Figure 5), and the authors do not explain why. \nI personally think that the reason is because they approximate p(y|x) with a mixture of Gaussians with a small variance and there is the same effect with smoothing as in case of the standard score matching. \nProbably we should start training using larger variance values and gradually decrease the variance during the training.\n\n2) I think that the motivation of the paper has too many somewhat irrelevant details. The authors compare different models, trained under different conditions, and conlude that the problem is in the loss. It seems that it was enough to say that it is difficult to approximate the gradients, so it is necessary to explicitly add them to the loss.\n\n3) Models (a) and (c) are different (in the first case it is a single model, in the second case the authors use different models for different classes). The first model minimizes cross-entropy loss, the second model optimizes the score matching function. \nIt is not entirely clear why the authors believe that the model (a) works worse than the model (c)  because of the loss? The models are also different. Maybe this is the main reason?\n\n4) In the same section with the motivation I propose to add the cross-entropy loss between p(y|x) and p(x|y), and not just the loss between their gradients (maybe the model (a) also approximates them poorly). The question immediately arises, why a good cross-entropy loss does not guarantee a good approximation of the gradients?\n\n5) Why do the authors train the classifier p(y | x, \\theta) and the score model (x, \\phi) independently, not end-to-end?\n\n6) Why is their loss noisy in practice? How much does the value of \\tau (noise variance) affect the stability of the loss and the results? Why don't the authors do the same trick with \\tau as with \\sigma (that is, they don't train several different models with different values of \\tau)?\n\n7) Experiments are done only on cifar-10/cifar-100, which is not very convincing, of course.", "summary_of_the_paper": "The authors suggest how to apply the idea of score matching to the problem of conditional generation. They introduce a new loss, which is essentially a score matching loss for the conditional distribution p(y| x). They demonstrate how to effectively calculate it without calculating p(y|x) explicitly, which seems to be the main contribution of the paper. Then the authors demonstrate how to train the model for the conditional generation in practice: it is necessary to add to the proposed loss the usual cross-entropy loss, because the loss that they came up with is noisy (this can be seen in Figure 5). They got better results compared to baselines for cifar-10 and cifar-100.", "summary_of_the_review": "In general, the paper is well written, it has a clear and logical structure, it is easy to read. \nIn the introduction, only clearly highlighted contributions are missing.\n\nThe proposed criteria can be useful when training models for conditional generation.\nThe authors did rather detailed discussion of different variants of the score matching procedure.\n\n============\n\nI thank the authors for their responses. The updated version of the paper, with additional information here and there, improves the readability and the overall narrative. Given the authors responses now I am convinced in the concept proposed by the authors, so I can raise my score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D1VbqWS9Op_", "reviewer": "Reviewer_2i3W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths of the paper**  \nIn general, the paper's contributions are clear, and I also consider that the results are essential for several reasons:  \n\n\nFirst, the paper highlights a previously less-recognized problem in the context of conditional generations of diffusion-based generative models. Moreover, the authors provide sufficient analysis to help readers understand the problem and potential drawbacks of commonly accepted methods. In particular, the analysis to visualize the difference between the MLE-based training and the proposed method is very interesting to read.\n\n\nSecond, the paper introduces a novel training objective to tackle the problem above, called Denoising Likelihood Score Matching (DLSM). Recently, the interest in diffusion-based generative models has increased rapidly. Consequently, the interest in controlling the generation process of such models has also increased. Regarding this, I consider that the proposed method has huge applicability and will contribute to the ML communities.\n\n\nThird, the authors have devoted themselves to discussing practical techniques for applications of the proposed method. For example, the authors discuss that the effect of combining two losses, i.e., MLE and approximate DLSM, for training classifiers. In particular, the paper discusses the potential roles of two losses on the performance of the conditional generations throughout ablation studies. Furthermore, the authors provide additional analysis about the scaling method, including how they contribute to high-precision samples.\n\n**Weaknesses of the paper**  \nHowever, the following aspects of the paper can be improved.\n\n\nFirst, clarifying three objectives seems necessary, i.e., ELSM, DLSM, and approximate DLSM. In my opinion, when the pre-trained score is plugged into the DLSM objective, it is no longer DLSM loss. However, the current version hasn't sufficiently emphasized the difference, so I found that the current version can be potentially misleading. Moreover, the provided proofs and relevant theoretical discussions are about ELSM and DLSM, not approximate. As a result, readers infer the behaviors of the approximate DLSM only by empirical results. Thus, it is important to distinguish between DLSM and approximate DLSM.\n\n\nSecond, while the experiments are well-thought and easy to follow in general, some experimental results are unclear. Moreover, additional experiments seem required. For instance, it is unclear how FID and IS are evaluated. At first glance, I understood that the FID/IS are evaluated per class (and their averaged value are reported). However, the paper describes that they are evaluated unconditionally. If the same model was used, the unconditional generators should have been the same for the baselines and the proposed method. If it wasn't, the description of the experiments needed to be updated. Another concern is that it is unclear when the combined loss is used and when it isn't during the experiments. In particular, it is unclear if \"ours\" in Table 1 corresponds to the approximate DLSM or the combined loss since the rest of the experiments have used the combined loss. I consider it important to clarify Table 1 and potentially include the results of models trained with the combined loss. Similarly, it would be interesting to see the results of CIFAR-10/100 experiments without MLE-losses.\n\n\nThird, the presentation of some backgrounds can be improved. In particular, it seems like that the current submission introduces the discrete-time Langevin dynamics, which is time-homogeneous, as a building block of the diffusion-based generative models. However, the paper follows the recent works, specifically time-inhomogeneous diffusions, which have made significant distinctions of the recent works against the time-homogeneous ones in several perspectives; for example, practicality in the context of generative models.\n\n**Minor comments**\n- I found that matching the y-axis scale in Figs 4 and A.1. will help interpret the results.\n- Optionally, I found it beneficial to run 1-dimensional experiments as done in Sec 3 but present results similar to Fig A.1. In my understanding, the classifiers trained with the approximate DLSM will underestimate the gradient (thus smoother) in comparison to the DLSM (I may be wrong). On the other hand, the MLE-based training results in unnecessarily sharper. Plotting the learned gradients and the ground truth will be helpful to explain why combining two losses makes sense.\n- In my understanding, $\\tilde{y}$ may not be necessary to derive DLSM. Could you explain why noisy $\\tilde{y}$ is necessary?", "summary_of_the_paper": "The paper points up a previously underappreciated problem in training classifiers in the context of conditional generations of diffusion-based generative models. The authors propose a novel objective for training classifiers to tackle the problems. Informally, the generation process of the diffusion-based generative models can be described by repeatedly applying an update rule with initial values: $\\tilde{x} ← \\tilde{x} + \\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(\\tilde{x}) + \\sigma \\epsilon$ where $\\epsilon \\sim N(0,I)$ and the initial points are sampled from a prior distribution such as standard Normal distributions. Similarly to unconditional generations, one can perform conditional generations, e.g., $p(\\tilde{x} | y)$ by using the following update rule: $\\tilde{x} ← \\tilde{x} + \\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(\\tilde{x}) + \\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(y | \\tilde{x}) + \\sigma \\epsilon$. Note that $\\nabla_{\\tilde{x}} \\log p(\\tilde{x} | y) = \\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(\\tilde{x}) + \\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(y | \\tilde{x})$. Thus, for a given pre-trained generative model, one needs to learn $\\log p_{\\textrm{model}}(y | \\tilde{x})$ (or its gradient wrt $\\tilde{x}$. Maximum likelihood training (MLE), i.e., minimizing cross-entropy loss, is commonly used. Inevitably, the qualities of the gradient $\\nabla_{\\tilde{x}} \\log p_{\\textrm{model}}(y | \\tilde{x})$ will determine the qualities of the conditional generations.\n\n\nFirst, the paper emphasizes that MLE-training of classifiers results in non-smooth gradient landscape wrt input; thus, the resulting gradients negatively affect the generation qualities. More precisely, with MLE, the learned $\\log p_{\\textrm{model}} (y|\\tilde{x})$ is high, while its gradient wrt $\\tilde{x}$ isn't necessarily close to the ground truth. In the paper, the authors refer to this phenomenon as *a score mismatch issue.* The authors analyze the mismatch issue with toy experiments and demonstrate its negative effects on generation qualities compared to the ground truth.\n\n\nSecond, to resolve the *score mismatch issue*, the paper proposes a new objective function, called Explicit Likelihood Score-Matching loss (ELSM), in which the mean squared errors between $\\nabla_{\\tilde{x}} \\log p_{\\textrm{model}} (y|\\tilde{x})$ and $\\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (y|\\tilde{x})$ is minimized. Here, due to the inaccessibility $\\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (y|\\tilde{x})$, the authors reduces the ELSM loss to another objective function, named *Denoising Likelihood Score-Matching* (DLSM), similarly to denoising score matching derivation. In DLSM, $\\nabla_{\\tilde{x}} \\log p_{\\textrm{model}} (y|\\tilde{x})$ is trained to match to $\\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (\\tilde{x}) - \\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (\\tilde{x} | x)$. Acknowledging that $\\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (\\tilde{x})$ is still unattainable, the authors propose to minimize approximate DLSM loss, where $\\nabla_{\\tilde{x}} \\log p_{\\textrm{data}} (\\tilde{x})$ is substituted by the pre-trained scores $\\nabla_{\\tilde{x}} \\log p_{\\textrm{model}} (\\tilde{x})$.\n\n\nThen, the authors demonstrate that by toy experiments, training classifiers with the approximate DLSM improves the conditional generation qualities compared to MLE-based training. Moreover, the authors mention that approximate DLSM-based training can be unstable in high-dimensional datasets and show that training classifiers by minimizing combined loss of approximate DLSM and cross-entropy can further improve the generation qualities.\n\n\nLastly, the paper demonstrates the effectiveness of the proposed methods by evaluating conditional generation qualities on CIFAR-10 and CIFAR-100 datasets.", "summary_of_the_review": "In general, the paper's contributions are clear; the proposed methods are well-motivated and well-discussed. In addition, I also found that the paper has a well-organized structure so that it is clear to understand the proposed method and other practical techniques to improve training. Thus, I'm inclined to accept the paper. However, I found that several aspects of the submission can be improved. I hope that the aforementioned weak points are well addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1b3_o7HVmRw", "reviewer": "Reviewer_jmK6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- Sections 1 and 2 are very clear and exhibit strong understanding of the field and the most recent literature. Upon examination, the proof of (2) in A.2 appears to be correct.\n- The motivating example, in Section 3 and Figure 1, is clearly presented, describes a relevant synthetic case, and the results demonstrate enough improvement to mean the paper's flow and narrative works well. The metrics $D_P$ and $D_L$ are also valuable in this more tractable setting.\n- In Section 4, Equations (9) and (10) follow intuitively and are well explained. I have been through the proof of Theorem 1 in detail and I believe it is correct (there wasn't even a typo!). Figure 2 is also a useful/pertinent depiction of the training process.\n- I believe all of the experiments presented in Section 5 are useful, clear, and well laid out.\n\n**Weaknesses**\nThe principle weakness I can see in the paper is the lack of a breakdown of \"inaccurate likelihood scores\".  Although empirical evidence suggests the classifier is at fault, the paper only uses references for this motivation. There is no explanation as to how one might stratify these classifier score inaccuracies. As one example, does a stronger classifier provide better gradients for conditional generation? This line of research would potentially bolster the training procedure in Figure 2 by further guiding the involvement of the classifier.\n\n- The use of \"some extra information\" at the start of Section 2.4 is casual and belies the importance of the class.\n- In Section A.3, I believe \"was $p_{\\alpha}(\\mathbf{\\tilde{x}},\\mathbf{\\tilde{y}})$ not\" should be \"$p_{\\alpha}(\\mathbf{\\tilde{x}},\\mathbf{\\tilde{y}})$ was not\".\n- At the end of page 5: \"numerical results in Table 1 are\" rather than  \"numerical results in Table 1 is\"\n- The introduction of $p_{0,\\tau}(\\mathbf{x}|\\mathbf{\\tilde{y}})$ in the 5th line of the $G(\\theta)$ derivation is slightly jarring given that the zero in the subscript is not used elsewhere or explicitly described.\n\nStylistic choices:\n- Use of \\left[ and \\right] and latex would be nicer, but this is subjective.\n- Same goes for the use of \\lVert and \\rVert for norms instead of \\parallel (more for divergences).", "summary_of_the_paper": "The authors present a new objective, denoising likelihood score-matching (DLSM), and training mechanism for conditional score-based data generation. The new method is motivated by poor conditional score estimates observed in low-dimensional examples for the standard, Bayes-theorem-based, conditional data generation procedure of score-based models. The new objective is equivalent to directly learning to parameterise the score of p(y|x) demonstrates is high-dimensional benchmark datasets also. FID and IS improve due to this method, with the largest improvements being class-wise, and there is an overall tendency to trade off generated data recall for precision.", "summary_of_the_review": "Overall, I think this is a strong paper that would make a positive contribution to the conference and the current discussion surrounding SBMs.  The paper is very well presented and explained throughout, with appropriate figures and tables, as well as accurate derivations. Class conditional generation does appear to improve based on this method, and the complementary performances trade-offs (distribution precision) are broken down and discussed. Besides one line of research that could be included in a future paper, weaknesses are minor and/or aesthetic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses an issue with decomposing a conditional generative model into an unconditional model and a separate classifier using Bayes' theorem, which is an approach that has recently received increased attention in the context of score-based generative models. It explores several alternatives for mitigating this issue, including a novel one, which is to use a different loss function to train the classifier.

Reviewers praised the writing and the way this work draws attention to an issue that is underappreciated in the community. Although several weaknesses (clarity, scale of experiments, appropriateness of baselines, missing experiments) were also highlighted in the original reviewers, all reviewers agree after discussion that the authors have adequately addressed these for the paper to be considered for acceptance. I will follow their recommendation and recommend acceptance as well.
Decision: Accept (Poster)