Year: 2021

Abstract:
Many existing conditional score-based data generation methods utilize Bayes’ theorem to decompose the gradients of a log posterior density into a mixture of scores. These methods facilitate the training procedure of conditional score models, as a mixture of scores can be separately estimated using a score model and a classifier. However, our analysis indicates that the training objectives for the classifier in these methods may lead to a serious score mismatch issue, which corresponds to the situation that the estimated scores deviate from the true ones. Such an issue causes the samples to be misled by the deviated scores during the diffusion process, resulting in a degraded sampling quality. To resolve it, we theoretically formulate a novel training objective, called Denoising Likelihood Score Matching (DLSM) loss, for the classifier to match the gradients of the true log likelihood density. Our experimental evidences show that the proposed method outperforms the previous methods on both Cifar-10 and Cifar-100 benchmarks noticeably in terms of several key evaluation metrics. We thus conclude that, by adopting DLSM, the conditional scores can be accurately modeled, and the effect of the score mismatch issue is alleviated.

heading: 1 INTRODUCTION
text: Score-based generative models are probabilistic generative models that estimate score functions, i.e., the gradients of the log density for some given data distribution. According to the definition of the pioneering work (Hyvärinen, 2005), the process of training score-based generative models is called Score Matching (SM), in which a score-based generative model is iteratively updated to approximate the true score function. Such a process often incurs heavy computational burdens, since it requires scanning through every data sample in a dataset to perform an update in each training iteration. Therefore, a branch of study in this research domain (Vincent, 2011; Martens et al., 2012; Song et al., 2020) resorts to reformulating the score-matching objective to reduce the training cost. Among these works, the author in (Vincent, 2011) introduced the Denoising Score-Matching (DSM) method. This method facilitates the training process of score-based generative models, and thus lays the foundation for a number of subsequent researches. Recently, the authors in (Song & Ermon, 2019) proposed an unified framework based on DSM, and achieved remarkable performance on Cifar-10 (Krizhevsky et al., 2009). Their success inspired several succeeding works (Song & Ermon, 2020; Ho et al., 2020; Song et al., 2021a;b; Dhariwal & Nichol, 2021), which together contribute to making score-based generative models an attractive choice for contemporary image generation tasks.
A favorable aspect of score-based generative models is its flexibility to be easily extended to their conditional variants. This characteristic comes from a research direction that utilizes Bayes’ theorem to decompose a conditional score into a mixture of scores (Nguyen et al., 2017). Recent endeavors followed this approach and further extended the concept of conditional score-based models to a number of application domains, including colorization (Song et al., 2021b), inpainting (Song et al., 2021b), and source separation (Jayaram & Thickstun, 2020). In particular, some recent researchers (Song et al., 2021b; Dhariwal & Nichol, 2021) applied this method to the field of classconditional image generation tasks, and proposed the classifier-guidance method. Different from the classifier-guidance-free method adopted by (Ho et al., 2021), they utilized a score model and a classifier to generate the posterior scores (i.e., the gradients of the log posterior density), with which the data samples of certain classes can be generated through the diffusion process. The authors in (Dhariwal & Nichol, 2021) showed that the classifier guidance method is able to achieve improved
performance on large image generation benchmarks. In spite of their success, our analysis indicates that the conditional generation methods utilizing a score model and a classifier may suffer from a score mismatch issue, which is the situation that the estimated posterior scores deviate from the true ones. This issue causes the samples to be guided by inaccurate scores during the diffusion process, and may result in a degraded sampling quality consequently.
To resolve this problem, we first analyze the potential causes for the score mismatch issue through a motivational low-dimensional example. Then, we theoretically formulate a new loss function called Denoising Likelihood Score-Matching (DLSM) loss, and explain how it can be integrated into the current training method. Finally, we evaluate the proposed method under various configurations, and demonstrate its advantages in improving the sampling quality over the previous methods in terms of several evaluation metrics.

heading: 2 BACKGROUND
text: In this section, we introduce the essential background material for understanding the contents of this paper. We first introduce Langevin diffusion (Roberts & Tweedie, 1996; Roberts & Rosenthal, 1998) for generating data samples x̃ ∈ Rd of a certain dimension d from an unknown probability density function (pdf) p(x̃) using the score function∇x̃ log p(x̃). Next, we describe Parzen density estimation (Parzen, 1962) and denoising score matching (DSM) (Vincent, 2011) for approximating ∇x̃ log p(x̃) with limited data samples. Finally, we elaborate on the conditional variant of the score function, i.e., ∇x̃ log p(x̃|ỹ), and explains how it can be decomposed into ∇x̃ log p(x̃) and ∇x̃ log p(ỹ|x̃) for some conditional variable ỹ ∈ Rc of dimension c.

heading: 2.1 LANGEVIN DIFFUSION
text: Langevin diffusion (Roberts & Tweedie, 1996; Roberts & Rosenthal, 1998) can be used to generate data samples from an unknown data distribution p(x̃) using only the score function ∇x̃ log p(x̃), which is said to be well-defined if p(x̃) is everywhere non-zero and differentiable. Under the condition that ∇x̃ log p(x̃) is well-defined, Langevin diffusion enables p(x̃) to be approximated iteratively based on the following equation:
x̃t = x̃t−1 + 2
2 ∇x̃ log p(x̃t−1) + zt, (1)
where x̃0 is sampled from an arbitrary distribution, is a fixed positive step size, and zt is a noise vector sampled from a normal distribution N (0, Id×d) for simulating a d-dimensional standard Brownian motion. Under suitable regularity conditions, when → 0 and T →∞, x̃T is generated as if it is directly sampled from p(x̃) (Roberts & Tweedie, 1996; Welling & Teh, 2011). In practice, however, the data samples are generated with > 0 and T < ∞, which violates the convergence guarantee. Although it is possible to use Metropolized algorithms (Roberts & Tweedie, 1996; Roberts & Rosenthal, 1998) to recover the convergence guarantee, we follow the assumption of the prior work (Song & Ermon, 2019) and presume that the errors are sufficiently small to be negligible when is small and T is large.
The sampling process introduced in Eq. (1) can be extended to a time-inhomogeneous variant by making p(x̃t) and dependent on t (i.e., pt(x̃t) and t). Such a time-inhomogeneous variant is commonly adopted by recent works on score-based generative models (Song & Ermon, 2019; Song et al., 2021b), as it provides flexibility in controlling pt(x̃t) and t. The experimental results in (Song & Ermon, 2019; Song et al., 2021b) demonstrated that such a time-inhomogeneous sampling process can improve the sampling quality on real-world datasets. In Appendix A.6.1, we offer a discussion on the detailed implementation of such a time-inhomogeneous sampling process in this work.

heading: 2.2 PARZEN DENSITY ESTIMATION
text: Given a true data distribution pdata, the empirical data distribution p0(x) is constructed by sampling m independent and identically distributed data points {x(i)}mi=1, and can be represented as a sum of Dirac functions 1m ∑m i=1 δ(‖x − x(i)‖). Such a discrete data distribution p0(x) constructed from the dataset often violates the previous assumptions that everywhere is non-zero and is differentiable.
Therefore, it is necessary to somehow adjust the empirical data distribution p0(x) before applying Langevin diffusion in such cases.
To deal with the above issue, a previous literature (Vincent, 2011) utilized Parzen density estimation to replace the Dirac functions with isotropic Gaussian smoothing kernels pσ(x̃|x) =
1 (2π)d/2σd
e −1 2σ2 ‖x̃−x‖2 with variance σ2. Specifically, Parzen density estimation enables the calculation of pσ(x̃) = 1m ∑m i=1 pσ(x̃|x(i)). When σ > 0, the score function becomes well-defined and can thus be represented as the following:
∇x̃ log pσ(x̃) = ∑m i=1 1 σ2 (x
(i) − x̃)pσ(x̃|x(i))∑m i=1 pσ(x̃|x(i)) . (2)
The proof for Eq. (2) is provided in Appendix A.2. This equation can be directly applied to Eq. (1) to generate samples with Langevin diffusion. Unfortunately, this requires summation over all m data points during every iteration, preventing it from scaling to large datasets due to the rapid growth in computational complexity.

heading: 2.3 DENOISING SCORE MATCHING
text: Score matching (SM) (Hyvärinen, 2005) was proposed to estimate the score function with a model s(x̃;φ), parameterized by φ. Given a trained score model s(x̃;φ), the scores can be generated by a single forward pass, which reduces the computational complexity of Eq. (2) by a factor of m. To train such a score model, a straightforward approach is to use the Explicit Score-Matching (ESM) loss LESM, represented as:
LESM(φ) = Epσ(x̃) [ 1
2 ‖s(x̃;φ)−∇x̃ log pσ(x̃)‖2
] . (3)
This objective requires evaluating Eq. (2) for each training step, which also fails to scale well to large datasets. Based on Parzen density estimation, an efficient alternative, called Denoising ScoreMatching (DSM) loss (Vincent, 2011), is proposed to efficiently calculate the equivalent loss LDSM, expressed as:
LDSM(φ) = Epσ(x̃,x) [ 1
2 ‖s(x̃;φ)−∇x̃ log pσ(x̃|x)‖2
] . (4)
where ∇x̃ log pσ(x̃|x) is simply 1σ2 (x − x̃). Since the computational cost of denoising score matching is relatively lower in comparison to other reformulation techniques (Hyvärinen, 2005; Song et al., 2020), it is extensively adopted in recent score-based generative models (Song & Ermon, 2019; 2020; Song et al., 2021b).

heading: 2.4 CONDITIONAL SCORE DECOMPOSITION VIA BAYES’ THEOREM
text: Score models can be extended to conditional models when conditioned on a certain label ỹ. Similar to x̃, the smoothing kernels with variance τ2 can be applied on y to meet the requirement that the pdf is everywhere non-zero and differentiable. Typically, τ is assumed to be sufficiently small so that pτ (ỹ) ≈ p(y). A popular approach adopted by researchers utilizes Bayes’ theorem pσ,τ (x̃|ỹ) = pσ,τ (ỹ|x̃)pσ(x̃)/pτ (ỹ) to decompose the conditional score ∇x̃ log pσ,τ (x̃|ỹ) into a mixture of scores (Nguyen et al., 2017), which enables conditional data generation. Following the assumptions in the previous study (Song et al., 2021b), the decomposition can be achieved by taking the loggradient on both sides of the equation, expressed as follows:
∇x̃ log pσ,τ (x̃|ỹ) = ∇x̃ log pσ,τ (ỹ|x̃) +∇x̃ log pσ(x̃)−∇x̃ log pτ (ỹ)︸ ︷︷ ︸ =0 , (5)
where ∇x̃ log pσ,τ (x̃|ỹ) is the posterior score, ∇x̃ log pσ,τ (ỹ|x̃) is the likelihood score, and ∇x̃ log pσ(x̃) is the prior score. Base on Eq. (5), a conditional score model ∇x̃ log p(x̃|ỹ;φ, θ) can be represented as the combination of the log-gradient of a differentiable classifier p(ỹ|x̃; θ) and a prior score model s(x̃;φ):
∇x̃ log p(x̃|ỹ; θ, φ) = ∇x̃ log p(ỹ|x̃; θ) + s(x̃;φ). (6)
This formulation enables conditional data generation using a classifier p(ỹ|x̃; θ) trained with crossentropy loss LCE(θ) , Epσ,τ (x̃,ỹ)[− log p(ỹ|x̃; θ)], and a score model s(x̃;φ) trained with denoising score matching loss LDSM(φ). Unfortunately, a few previous studies (Nguyen et al., 2017; Dhariwal & Nichol, 2021) have noticed that the approximation of∇x̃ log p(x̃|ỹ; θ, φ) is empirically inaccurate (referred to as the score mismatch issue in this work), and leveraged a scaling factor α > 0 to adjust the likelihood score ∇x̃ log p(ỹ|x̃; θ). Such a scaling factor is a hyperparameter that controls the amount of conditional information incorporated during the sampling process (see Appendix A.3). However, this usually causes the diversity of generated data samples to degrade noticeably (Dhariwal & Nichol, 2021), which is later discussed in Section 5.2.
In this work, we examine and investigate the score mismatch issue from a different perspective. We first offer a motivational example to show that the issue remains even with the scaling technique. Then, we theoretically formulate a new loss term that enables a classifier p(ỹ|x̃; θ) to produce better likelihood score estimation.

heading: 3 ANALYSIS ON THE SCORE MISMATCH ISSUE
text: To further investigate the score mismatch issue, we first leverage a motivational experiment on the inter-twining moon dataset to examine the extent of the discrepancy between the estimated and true posterior scores. In this experiment, we consider five different methods to calculate the posterior scores denoted as (a)∼(e):
• (a) Base method. The posterior scores are estimated using a score model s(x̃;φ) trained with LDSM and a classifier p(ỹ|x̃; θ) trained with LCE, as described in Eq. (6). • (b) Scaling method. The posterior scores are estimated in a similar fashion as method
(a) except that the scaling technique (Dhariwal & Nichol, 2021) is adopted to scale the likelihood scores, i.e., ∇x̃ log p(x̃|ỹ; θ, φ) = α∇x̃ log p(ỹ|x̃; θ) + s(x̃;φ), where α > 0. • (c) Posterior SM method. The posterior scores for different class conditions, in this case, c1
and c2, are separately estimated using different score models s(x̃;φ1) and s(x̃;φ2) trained with LDSM. Note that, since this method requires network architectures different from those used in methods (a), (b), and (d), it is only adopted for analytical purposes. For more detailed discussions about this method, please refer to Appendix A.4.
• (d) Ours. The posterior scores are estimated in a similar fashion as method (a) except that the classifier is trained with the proposed loss function LTotal, which is described in detail in Section 4.
Fig. 1 visualizes the posterior scores and the sampled points based on the five methods. It is observed that the posterior scores estimated using methods (a) and (b) are significantly different from the true posterior scores measured by method (e). This causes the sampled points in methods (a) and (b) to deviate from those sampled based on method (e). On the other hand, the estimated posterior scores and the sampled points in method (c) are relatively similar to those in method (e). The above results therefore suggest that the score mismatch issue is severe under the cases of methods (a) and (b), but is alleviated when method (c) is used.
In order to inspect the potential causes for the differences between the results produced by methods (a), (b), and (c), we incorporate metrics for evaluating the sampling quality and the errors between the scores in an quantitative manner. The sampling quality is evaluated using the precision and recall (Kynkäänniemi et al., 2019) metrics. On the other hand, the estimation errors of the score functions are measured by the expected values of DP and DL, which are formulated according to the Euclidean distances between the estimated scores and the oracle scores. The expressions of DP and DL are represented as the following:

heading: DP (x̃, ỹ) = ‖SP (x̃, ỹ)−∇x̃ log pσ,τ (x̃|ỹ)‖,
text: DL(x̃, ỹ) = ‖SL(x̃, ỹ)−∇x̃ log pσ,τ (ỹ|x̃)‖,
(7)
where the subscripts P and L denote ‘Posterior’ and ‘Likelihood’, while the terms SP (x̃, ỹ) and SL(x̃, ỹ) correspond to the estimated posterior score and the likelihood score for a certain pair (x̃, ỹ), respectively.
Table 1 presents SP (x̃, ỹ) and SL(x̃, ỹ), the expectations of DP and DL, and the precision and recall for different methods. It can be seen that the numerical results in Table 1 are consistent with the observations revealed in Fig. 1, since the expectations of DP are greater in methods (a) and (b) as compared to method (c), and the evaluated recall values in methods (a) and (b) are lower than those in method (c). Furthermore, the results in Table 1 suggest that the possible reasons for the disparity of the posterior scores between methods (a)∼(c) are twofold. First, adding the scaling factor α to the likelihood scores increases the expected distance between the estimated score and oracle score, which may exacerbate the score mismatch issue. Second, since the main difference between methods (a)∼(c) lies in their score-matching objectives, i.e., the parameters θ in methods (a) and (b) are optimized using LCE only while the parameters φ1 and φ2 of the score models in method (c) are optimized through LDSM, the adoption of the score-matching objective may potentially be the key factor to the success of method (c).
The above experimental clues therefore shed light on two essential issues to be further explored and dealt with. First, although employing a classifier trained with LCE to assist estimating the oracle posterior score is theoretically feasible, this method may potentially lead to considerable discrepancies
in practice. This implies that the score mismatch issue stated in Section 2.4 may be the result of the inaccurate likelihood scores produced by a classifier. Second, the comparisons between methods (a), (b), and (c) suggest that score matching may potentially be the solution to the score mismatch issue. Based on these hypotheses, this paper explores an alternative approach to the previous works, called denoising likelihood score matching, which incorporates the score-matching technique when training the classifier to enhance its capability to capture the true likelihood scores. In the next section, we detail the theoretical derivation and the property of our method.

heading: 4 DENOISING LIKELIHOOD SCORE MATCHING
text: In this section, we introduce the proposed denoising likelihood score-matching (DLSM) loss, a new training objective that encourages the classifier to capture the true likelihood score. In Section 4.1, we derive this loss function in a theoretical manner and discuss the intuitions behind it. In Section 4.2, we elaborate on the training procedure of DLSM, and highlight the key property and implications of this training objective.

heading: 4.1 THE DERIVATION OF THE DENOISING LIKELIHOOD SCORE MATCHING LOSS
text: As discussed in Section 3, a score model trained with the score-matching objective can potentially be beneficial in producing a better posterior score estimation. In light of this, a classifier may be enhanced if the score-matching process is involved during its training procedure. An intuitive way to accomplish this aim is through minimizing the explicit likelihood score-matching loss LELSM, which is defined as the following:
LELSM(θ) = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ)−∇x̃ log pσ,τ (ỹ|x̃)‖2
] . (8)
This loss term, however, requires the derivation of the true likelihood score ∇x̃ log pσ,τ (ỹ|x̃), which is computationally unaffordable during the classifier training. In order to reduce the computational cost, we follow the derivation of DSM as well as Bayes’ theorem, and formulate an alternative objective LDLSM:
LDLSM(θ) = Epσ,τ (x̃,x,ỹ,y) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ) +∇x̃ log pσ(x̃)−∇x̃ log pσ(x̃|x)‖2
] . (9)
Theorem 1. Optimizing θ with LELSM(θ) is equivalent to optimizing θ with LDLSM(θ).
The proof of this theorem is provided in Appendix A.5. In contrast to LELSM, LDLSM can be approximated in a computationally feasible fashion. This is because∇x̃ log pσ(x̃) can be estimated using a score model s(x̃;φ) trained with LDSM, and ∇x̃ log pσ(x̃|x) = 1σ2 (x − x̃) as described in Section 2.3. As ∇x̃ log pσ(x̃) and ∇x̃ log pσ(x̃|x) can be computed in a tractable manner, the classifier can be updated by minimizing the approximated variant of LDLSM, defined as:
LDLSM′(θ) = Epσ,τ (x̃,x,ỹ,y) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ) + s(x̃;φ)−∇x̃ log pσ(x̃|x)‖2
] . (10)
The underlying intuition of LDLSM and LDLSM′ is to match the likelihood score via matching the posterior score. More specifically, the sum of the first two terms ∇x̃ log p(ỹ|x̃; θ) and ∇x̃ log pσ(x̃) in LDLSM should ideally construct the posterior score (i.e., Eq. (5)). By following the posterior score, the perturbed data sample x̃ should move towards the clean sample x, since∇x̃ log pσ(x̃|x) is equal to the weighted difference 1σ2 (x − x̃) (i.e., Eq. (4)). In the next section, we discuss the training procedure with LDLSM′ as well as the property of it.

heading: 4.2 THE APPROXIMATED DLSM OBJECTIVE IN THE CLASSIFIER TRAINING PROCESS
text: Following the theoretical derivation in Section 4.1, we next discuss the practical aspects during training, and propose to train the classifier by jointly minimizing the approximated denoising likelihood score-matching loss LDLSM′ and the cross-entropy loss LCE. In practice, the total training objective of the classifier can be written as follows:
LTotal(θ) = LDLSM′(θ) + λLCE(θ), (11)
where λ > 0 is a balancing coefficient. According to Theorem 1, employing LDLSM′ allows the gradients of the classifier to estimate the likelihood scores. However, since the computation of the term LDLSM′ requires the approximation of ∇x̃ log pσ(x̃) from a score model s(x̃;φ), the training process that utilizes LDLSM′ alone is undesirable due to its instability on real-world datasets. To reinforce the stability of it, an additional cross-entropy loss LCE is adopted during the training process of the classifier. The advantage of LCE is that it leverages the ground truth labels to assist the classifier to learn to match the true likelihood density pσ,τ (ỹ|x̃), which in turn helps the score estimation to be improved. In addition, the computation of LCE requires no approximation. To validate the above advantage, an ablation analysis are offered in Section 5.3 to demonstrate the differences between the classifiers trained with LTotal, LCE, and LDLSM′ . The analysis reveals that LTotal does provide the best score-matching results while maintaining the stability.
Fig. 2 depicts a two-stage training procedure adopted in this work. In stage 1, a score model s(x̃;φ) is updated using LDSM to match∇x̃ log pσ(x̃) (i.e., Eq. (4)). In stage 2, the weights of the trained score model are fixed, and a classifier p(ỹ|x̃; θ) is updated using LTotal. After these two training stages, s(x̃;φ) and ∇x̃ log p(ỹ|x̃; θ) can then be added together based on Eq. (6) to perform conditional sampling.

heading: 5 EXPERIMENTS
text: In this section, we present experiments to validate the effectiveness of the proposed method. First, we provide the experimental configurations and describe the baseline methods. Next, we evaluate the proposed method quantitatively against the baselines, and demonstrate its improved performance on the Cifar-10 and Cifar-100 datasets. Lastly, we offer an ablation analysis to inspect the impacts of LCE and LDLSM′ in LTotal.

heading: 5.1 EXPERIMENTAL SETUPS
text: In this work, we quantitatively compare the base method, scaling method, and our method on the Cifar-10 and Cifar-100 datasets, which contain real-world RGB images with 10 and 100 categories of objects, respectively. For both of the datasets, the image size is 32×32, and the pixel values in an image are first rescaled to [−1, 1] before training. For the sampling algorithm, we adopt the predictorcorrector (PC) sampler described in (Song et al., 2021b) with the sampling steps set to T=1,000. The score model architecture is exactly the same as the one used in (Song et al., 2021b), while the architecture of the classifier is based on ResNet (He et al., 2016) with a conditional branch for encoding the information of the standard deviation σ (Song et al., 2021b). When training the classifier, τ is assumed to be sufficiently small, thus the unperturbed labels y are adopted. Furthermore, the balancing coefficient λ is fixed to 1 for all settings. For more details about our implementation and additional experimental results, please refer to Appendix A.6 and Appendix A.7.

heading: 5.2 RESULTS ON CIFAR-10 AND CIFAR-100
text: In this section, we examine the effectiveness of the base method, the scaling method, and our proposed method on the Cifar-10 and Cifar-100 benchmarks with several key evaluation metrics. We adopt the Inception Score (IS) (Barratt & Sharma, 2018) and the Fréchet Inception Distance (FID) (Heusel et al., 2017) as the metrics for evaluating the overall sampling quality by comparing the similarity between the distributions of the generated images and the real images. We also evaluate the methods using the precision (P), recall (R) (Kynkäänniemi et al., 2019), density (D), and coverage (C) (Naeem et al., 2020) metrics to further examine the fidelity and diversity of the generated images. In addition, we
report the classification accuracy score (CAS) (Ravuri & Vinyals, 2019) to measure if the generated samples bear representative class information. Specifically, given a dataset containing mi images for each class i, we conditionally generate the same number of images (i.e., mi) for each class. The FID, IS, P / R / D / C, and CAS metrics are evaluated based on all the generated m = ∑c i=1mi images, where c is the total number of classes.
Table 2 reports the quantitative results of the above methods. It is observed that the proposed method outperforms the other two methods with substantial margins in terms of FID and IS, indicating that the generated samples bear closer resemblance to the real data. Meanwhile, for the P / R / D / C metrics, the scaling method is superior to the other two methods in terms of the fidelity metrics (i.e., precision and density). However, this method may cause the diversity of the generated images to degrade, as depicted in Fig. 3, resulting in significant performance drops for the diversity metrics (i.e., the recall and the coverage metrics).
Another insight is that the based method achieves relatively better performance on the precision and density metrics in comparison to our method. However, it fails to deliver analogous tendency on the CAS metric. This behavior indicates that the base method may be susceptible to generating false positive samples, since the evaluation of the P / R / D / C metrics does not involve the class information, and thus may fail to consider samples with wrong classes. Such a phenomenon motivates us to further introduce a set of class-wise (CW) metrics, which takes the class information into account by evaluating the P / R / D / C metrics on a per-class basis. Specifically, the class-wise metrics are evaluated separately for each class i with mi images. The evaluation results of these metrics shown in Fig. 4 reveal that our method outperforms the base method for a majority of classes. Moreover, the results of the average class-wise metrics presented in the last four rows of Table 2 also show that our method yields a better performance as compared to the base method.
Base on these evidences, it can be concluded that the proposed method outperforms both baseline methods in terms of FID and IS, implying that our method does possess a better ability to capture the true data distribution. Additionally, the evaluation results on the CAS and the class-wise metrics suggest that our method does offer a superior ability for a classifier to learn accurate class information as compared to the base method.

heading: 5.3 ABLATION ANALYSIS
text: To further investigate the characteristic of LTotal, we perform an ablation analysis on the two components LCE and LDLSM′ in LTotal using the inter-twinning moon dataset, and observe the trends of (a) the score errors 12EU(x̃)[DL(x̃, c1)] + 1 2EU(x̃)[DL(x̃, c2)], and (b) the cross entropy Epσ,τ (x̃,ỹ)[− log p(ỹ|x̃; θ)] during the training iterations. Metrics (a) and (b) measure the accuracies of the estimated likelihood scores∇x̃ log p(ỹ|x̃; θ) and the estimated likelihood density p(ỹ|x̃; θ), respectively. As depicted in Fig 5, an increasing trend is observed for metric (a) when the classifier
is trained with LCE alone. On the contrary, the opposite tendency is observed for those trained withLTotal and LDLSM′ . The results thus suggest that matching the likelihood scores implicitly through training a classifier with LCE alone leads to larger approximation errors. In contrast, LDLSM′ and LTotal explicitly encourage the classifier to capture accurate likelihood scores, since they involve the score-matching objective. On the other hand, for metric (b), the classifiers trained with LTotal and LCE yield stable decreasing trend in comparison to that trained with LDLSM′ alone. These results suggest that in addition to minimizing LDLSM′ between ∇x̃ log pσ,τ (ỹ|x̃) and ∇x̃ log p(ỹ|x̃; θ), the utilization of LCE as an auxiliary objective enhances the stability during training. Based on the above observations, the interplay
of LCE and LDLSM′ synergistically achieves the best results in terms of metric (a) while ensuring the stability of the training process. These clues thus validate the adoption of LTotal during the training of the classifier.

heading: 6 CONCLUSION
text: In this paper, we highlighted the score mismatch issue in the existing conditional score-based data generation methods, and theoretically derived a new denoising likelihood score-matching (DLSM) loss, which is a training objective for the classifier to match the true likelihood score. We have demonstrated that, by adopting the proposed DLSM loss, the likelihood scores can be better estimated, and the negative impact of the score mismatch issue can be alleviated. Our experimental results validated that the proposed method does offer benefits in producing higher-quality conditional sampling results on both Cifar-10 and Cifar-100 datasets.

heading: A APPENDIX
text: In this Appendix, we first provide the definitions for the symbols used in the main manuscript and the Appendix in Section A.1. Next, we derive the theoretical proof for the closed form of ∇x̃ log pσ(x̃) in Section A.2. Then, we offer a number of discussions on the scaling technique in Section A.3, and different conditional score-based data generation methods in Section A.4. Subsequently, we offer the theoretical proof for Theorem 1 in Section A.5. Finally, in Sections A.6 and A.7, we provide the detailed experimental setups and additional experimental results.

heading: A.1 THE LIST OF SYMBOLS USED IN THE PAPER
text: In this section, we provide the list of symbols used throughout the main manuscript and the Appendix. These symbols are summarized in Tables A1 and A2, containing their notations as well as their detailed descriptions.
Symbol Description 〈u,v〉 = uTv = ∑ i uivi dot (inner) product between two vectors.
‖u‖ = √ 〈u,u〉 Euclidean norm of vector.
Id×d identity matrix with dimension d× d. the step size used in Langevin Diffusion. t ∈ {0, ..., T} the discrete timestep used in Langevin Diffusion. x,xt ∈ Rd data sample with dimension d, where x0 is sampled from N (0, Id×d). x̃ ∈ Rd perturbed data sample used in Denoising Score Matching. y ∈ Rc conditional class label with dimension c. ỹ ∈ Rc perturbed class label used in Denoising Score Matching. z : Rd the noise vector used in Langevin Diffusion. (sampled from N (0, Id×d)) x(i) ∈ Rd the i-th example (input) from a dataset D. y(i) ∈ {0, 1}c the class label associated with x(i) for supervised learning (one-hot encoded). D = {(x(1),y(1)), ..., (x(m),y(m))} Training set: m data-label pairs from pdata. (i.i.d. samples from pdata) pdata(x) unknown true probability density function (pdf) of data samples. p0(x) =
1 m ∑m i=1 δ(‖x− x
(i)‖) empirical pdf of data samples associated with the training set D. pσ(x̃|x) = 1(2π)d/2σd e −1 2σ2 ‖x̃−x‖2 smoothing kernel or noise model: sampled from isotropic Gaussian N (x, σ2Id×d) with mean x and variance σ2. pσ(x̃,x) = pσ(x̃|x)p0(x) joint pdf of perturbed data and clean data samples. pσ(x̃) =
1 m ∑m i=1 pσ(x̃|x
(i)) Parzen density estimate based on D, obtainable by marginalizing pσ(x̃,x).
pdata(y) unknown true pdf of data labels. p0(y) =
1 m ∑m i=1 δ(‖y − y
(i)‖) empirical pdf of data labels associated with the training set D. pτ (ỹ|y) = 1(2π)c/2τc e −1 2τ2 ‖ỹ−y‖2 smoothing kernel or noise model: sampled from isotropic Gaussian N (y, τ2Ic×c) with mean y and variance τ2. pτ (ỹ,y) = pτ (ỹ|y)p0(y) joint pdf of perturbed labels and clean labels. pτ (ỹ) =
1 m ∑m i=1 pτ (ỹ|y
(i)) Parzen density estimate based on D, obtainable by marginalizing pτ (ỹ,y).
p0,0(x,y) defined as 1m ∑m i=1 δ(‖x− x
(i)‖)δ(‖y − y(i)‖). pσ,0(x̃,x,y) equals to pσ(x̃|x)p0,0(x,y). p0,τ (x, ỹ,y) equals to pτ (ỹ|y)p0,0(x,y). pσ,τ (x̃,x, ỹ,y) defined as pσ(x̃|x)pτ (ỹ|y)p0,0(x,y). s(x̃;φ) Score Model: approximated score function with parameters φ. p(ỹ|x̃; θ) Classifier: approximated probability function parameterized by θ.
Table A1: The list of symbols used in this paper.
α the scaling factor. ∇x̃ log p(x̃|y) posterior score. ∇x̃ log p(y|x̃) likelihood score. ∇x̃ log p(x̃) prior score. LESM(φ) Explicit Score Matching (ESM) loss for optimizing s(x̃;φ). LDSM(φ) Denoising Score Matching loss (DSM) for optimizing s(x̃;φ). LELSM(θ) Explicit Likelihood Score Matching (ELSM) loss for optimizing p(y|x̃; θ). LDLSM(θ) Denoising Likelihood Score Matching (DLSM) loss for optimizing p(y|x̃; θ). LDLSM′(θ) approximated Denoising Likelihood Score Matching loss for optimizing p(y|x̃; θ). LCE(θ) Cross Entropy (CE) loss for optimizing p(y|x̃; θ). LTotal(θ) full loss for optimizing p(y|x̃; θ).
Table A2: The list of symbols used in this paper.
A.2 A PROOF OF THE CLOSED FORM FOR ∇x̃ log pσ(x̃)
In Sections 2.2 and 3, we utilized the closed form of ∇x̃ log pσ(x̃) to compute the true posterior scores. To show the equivalence holds for Eq. (2), we provide the formal theoretical proof as the following proposition.
Proposition 1. Given pσ(x̃) = 1m ∑m i=1 pσ(x̃|x(i)), pσ(x̃|x) = 1 (2π)d/2σd e −1 2σ2 ‖x̃−x‖2 , and σ > 0, the closed form of∇x̃ log pσ(x̃) can be represented as follows:
∇x̃ log pσ(x̃) = ∑m i=1 1 σ2 (x
(i) − x̃)pσ(x̃|x(i))∑m i=1 pσ(x̃|x(i)) .
Proof.
∇x̃ log pσ(x̃) = ∇x̃ log 1
m m∑ i=1 pσ(x̃|x(i))
= ∇x̃ log m∑ i=1 pσ(x̃|x(i)) +∇x̃ log 1
m︸ ︷︷ ︸ =0
= ∇x̃ ∑m i=1 pσ(x̃|x(i))∑m
i=1 pσ(x̃|x(i))
= ∑m i=1∇x̃pσ(x̃|x(i))∑m i=1 pσ(x̃|x(i))
=
∑m i=1∇x̃ [ 1 (2π)d/2σd e −1 2σ2 ‖x̃−x‖2 ] ∑m i=1 pσ(x̃|x(i))
=
∑m i=1 [ (x(i)−x̃)/σ2 (2π)d/2σd e −1 2σ2 ‖x̃−x‖2 ] ∑m i=1 pσ(x̃|x(i))
=
∑m i=1 1 σ2 (x
(i) − x̃)pσ(x̃|x(i))∑m i=1 pσ(x̃|x(i))

heading: A.3 A DISCUSSION ON THE IMPACTS OF THE SCALING TECHNIQUE
text: In this section, we first provide a proof to demonstrate the reason behind the non equivalence between the scaling technique and the re-normalization of a likelihood density. Then, we present the
derivation of the posterior scores based on a re-normalized likelihood density. Finally, we adopt a low-dimensional example to explain the effect of re-normalization on a posterior density.
The authors in (Dhariwal & Nichol, 2021) leveraged a scaling factor α > 0 to adjust the likelihood score ∇x log p(y|x). To demonstrate the effect of such a scaling factor α, the authors claimed that there always exists a normalizing constant Z such that α∇x log p(y|x) = ∇x log 1Z p(y|x)
α, where 1 Z p(y|x)
α is a re-normalized likelihood density. However, such an equality only holds for specific conditions, and is not applicable to general cases, as explained in Proposition 2. Proposition 2. Given a constant α > 0, and a probability density function (pdf) p(y|x) that is everywhere non-zero and differentiable, there does not exist a constant Z such that 1Z p(y|x)
α is a pdf for all x in general.
Proof. The proposition can be proved by contradiction. Assume that such a constant Z exists and consider the following case:
α = 2,
x, y ∈ R,
f(y;µ, σ2) = (2πσ2) −1 2 e
−(y−µ)2
2σ2 ,
p(y|x) = 1 2(x2 + 1)
( (x+ 1)2f(y; 0, 1) + (x− 1)2f(y; 0, 1
2 )
) .
Note that p(y|x) is a pdf and is everywhere non-zero and differentiable. Since a pdf sums to 1 (i.e., ∫ y 1 Z p(y|x)
αdy = 1,∀x), by utilizing the Gaussian integral and the above assumption, the following derivation holds:
1 = ∫ y 1 Z p(y|(−1))αdy = ∫ y 1 Z p(y|1)αdy
⇔ Z = ∫ y p(y|(−1))2dy = ∫ y p(y|1)2dy
⇔ Z = ∫ y f(y; 0, 1 2 )2dy = ∫ y f(y; 0, 1)2dy
⇔ Z = ∫ y ( (π) −1 2 e−y 2 )2 dy = ∫ y ( (2π) −1 2 e −y2 2 )2 dy
⇔ Z = 1 π ∫ y ( e−y 2 )2 dy = 1 2π ∫ y ( e −y2 2 )2 dy
⇔ Z = 1 π ∫ y e−2y 2 dy = 1 2π ∫ y e−y 2 dy
⇔ Z = 1√ 2π ∫ u e−u 2 du = 1 2π ∫ y e−y 2 dy
⇔ Z = 1√ 2π √ π = 1 2π √ π
⇔ Z = 1√ 2π = 1 2 √ π
The final equation leads to a contradiction for the value of Z, which completes the proof of Proposition 2. Therefore, we concluded that there does not exist a constant Z such that 1Z p(y|x)
α is a pdf for all x in general.
To the best of our knowledge, the scaling technique still lacks a theoretical justification. However, if Z is allowed to be a function of x rather than a constant, re-normalization of the likelihood density becomes feasible. In other words, setting Z(x) = ∫ y p(y|x)αdy enables ∫ y 1 Z(x)p(y|x) αdy = 1,
= 5 = 0.2Original
Figure A1: The original and the renormalized densities under different choices of α.
where 1Z(x)p(y|x) α , pα(y|x) is a re-normalized likelihood density. In this case, the score of the re-normalized likelihood density can be written as,
∇x log pα(y|x) = ∇x log ( 1
Z(x) p(y|x)α ) = ∇x ( log p(y|x)α − logZ(x)
) = ∇x log p(y|x)α −∇x logZ(x) = α∇x log p(y|x)−∇x logZ(x).
(A1)
Given the joint density pα(x,y) , pα(y|x)p(x), the posterior of the re-normalized density can be derived by Bayes’ Theorem, i.e., pα(x|y) = pα(y|x)p(x)/ ∫ x pα(x,y)dx. Based on Eq. (5), the posterior score of the re-normalized density can be written as:
∇x log pα(x|y) = ∇x log pα(y|x) +∇x log p(x) = α∇x log p(y|x)−∇x logZ(x) +∇x log p(x).
(A2)
Through the above equation, conditional sampling based on the re-normalized likelihood density becomes feasible.
To further investigate the effect of re-normalization on pα(x|y), we provide an illustrative example in Fig. A1. In this example, the condition y is assumed to be sampled from {y1,y2} with equal probability 1/2. The original posterior densities p(x|y1) and p(x|y2) are normal distributions with variance 2.25 centered at x = −1.5 and x = 1.5, which are plotted in orange and green, respectively. On the other hand, the prior density p(x) , 12p(x|y1) + 1 2p(x|y2) is plotted in blue, and the likelihood density p(y|x) = p(x|y)p(y)/p(x) is defined according to Bayes’ theorem. Subsequently, the re-normalized posterior densities pα(x|y1) and pα(x|y2) can be constructed based on the equation 1Z(x)p(y1|x) α, 1Z(x)p(y2|x) α, and Bayes’ theorem. In Fig. A1, it is observed that under the case that α = 5, the re-normalized posterior densities are more concentrated. Conversely, when α = 0.2, the re-normalized posterior densities are much more dispersed. These results therefore suggest that the re-normalized posterior density can be adjusted by controlling the value of α.

heading: A.4 CONDITIONAL SCORE-BASED DATA GENERATION METHODS
text: As described in Section 3, methods (a), (b), and (d) utilize a classifier p(ỹ|x̃; θ) and a score model s(x̃;φ) to estimate the posterior scores. In contrast, method (c) separately trains the posterior score models s(x̃;φ1) and s(x̃;φ2) for different class conditions. Although method (c) achieves better evaluation results, it requires training score models for each possible ỹ, which is usually considered impractical in real-world applications. A potential solution to this problem is to train a score model s(x̃, ỹ;φ) that can estimate the scores for the joint probability of x̃ and ỹ (denoted as ‘Method (c∗)’ in Fig. A2). This method, however, is inflexible in comparison to methods (a), (b), and (d), since any change on the condition ỹ could lead to re-training the joint score model s(x̃, ỹ;φ). Therefore, in this work, we only compare the performance of methods (a), (b), and (d), and consider method (c) to be another orthogonal research direction.
Score model
Classifier
Score model
...
Score model 1
Score model c
Methods (a), (b), (d) Method (c) Method (c*)
Figure A2: An illustration for different conditional score-based data generation methods in Section A.4.

heading: A.5 A PROOF FOR DENOISING LIKELIHOOD SCORE MATCHING
text: In Section 4, we stated the equivalence between LELSM(θ) and LDLSM(θ) when optimizing θ. To demonstrate how such an equivalence stands, we provide the theoretical proof as follows. Theorem 1. Optimizing θ with LELSM(θ) is equivalent to optimizing θ with LDLSM(θ):
LELSM(θ) = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ)−∇x̃ log pσ,τ (ỹ|x̃)‖2
] .
LDLSM(θ) = Epσ,τ (x̃,x,ỹ,y) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ) +∇x̃ log pσ(x̃)−∇x̃ log pσ(x̃|x)‖2
] .
Proof. The proof exapnds LELSM(θ) and LDLSM(θ) individually, leading to constant difference. LELSM(θ) = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ)−∇x̃ log pσ,τ (ỹ|x̃)‖2 ] = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ)‖2
] − F (θ) + C1, where
C1 = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log pσ,τ (ỹ|x̃)‖2
] is a constant, and
F (θ) = Epσ,τ (x̃,ỹ) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (ỹ|x̃)〉] = Epσ,τ (x̃,ỹ) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃|ỹ)−∇x̃ log pσ(x̃)〉] = G(θ)− Epσ,τ (x̃,ỹ) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ(x̃)〉] , where
G(θ) = Epσ,τ (x̃,ỹ) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃|ỹ)〉]
= ∫ x̃ ∫ ỹ pσ,τ (x̃, ỹ)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃|ỹ)〉dỹdx̃
= ∫ x̃ ∫ ỹ pτ (ỹ)pσ,τ (x̃|ỹ)〈∇x̃ log p(ỹ|x̃; θ), ∇x̃pσ,τ (x̃|ỹ) pσ,τ (x̃|ỹ) 〉dỹdx̃
= ∫ x̃ ∫ ỹ pτ (ỹ)〈∇x̃ log p(ỹ|x̃; θ),∇x̃pσ,τ (x̃|ỹ)〉dỹdx̃
= ∫ x̃ ∫ ỹ pτ (ỹ)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ ∫ x p0,τ (x|ỹ)pσ,τ (x̃|x, ỹ)dx〉dỹdx̃
= ∫ x̃ ∫ ỹ pτ (ỹ)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ ∫ x ∫ y p0,τ (x|ỹ)p0,τ (y|x, ỹ)pσ,τ (x̃|x, ỹ,y)dydx〉dỹdx̃
= ∫ x̃ ∫ ỹ pτ (ỹ)〈∇x̃ log p(ỹ|x̃; θ), ∫ x ∫ y p0,τ (x|ỹ)p0,τ (y|x, ỹ)∇x̃pσ,τ (x̃|x, ỹ,y)dydx〉dỹdx̃
= ∫ x̃ ∫ ỹ ∫ x ∫ y pτ (ỹ)p0,τ (x|ỹ)p0,τ (y|x, ỹ)pσ,τ (x̃|x, ỹ,y)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃|x, ỹ,y)〉dydxdỹdx̃
= ∫ x̃ ∫ ỹ ∫ x ∫ y pσ,τ (x̃,x, ỹ,y)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃|x, ỹ,y)〉dydxdỹdx̃
= ∫ x̃ ∫ ỹ ∫ x ∫ y pσ,τ (x̃,x, ỹ,y)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ,τ (x̃,x, ỹ,y)〉dydxdỹdx̃
= ∫ x̃ ∫ ỹ ∫ x ∫ y pσ,τ (x̃,x, ỹ,y)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log(pσ(x̃|x)pτ (ỹ|y)p0,0(x,y))〉dydxdỹdx̃
= ∫ x̃ ∫ ỹ ∫ x ∫ y pσ,τ (x̃,x, ỹ,y)〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ(x̃|x)〉dydxdỹdx̃
= Epσ,τ (x̃,x,ỹ,y) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ(x̃|x)〉] , and now we expand LDLSM(θ) : LDLSM(θ) = Epσ,τ (x̃,x,ỹ,y) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ) +∇x̃ log pσ(x̃)−∇x̃ log pσ(x̃|x)‖2 ] = Epσ,τ (x̃,ỹ) [ 1
2 ‖∇x̃ log p(ỹ|x̃; θ)‖2
] − Epσ(x̃,x,ỹ,y) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ(x̃|x)〉] +
Epσ,τ (x̃,ỹ) [〈∇x̃ log p(ỹ|x̃; θ),∇x̃ log pσ(x̃)〉] + C2, where C2 = Epσ(x̃,x) [ 1
2 ‖∇x̃ log pσ(x̃)‖2
] + Epσ(x,x̃) [ 1
2 ‖∇x̃ log pσ(x̃|x)‖2
] −
Epσ(x̃,x) [〈∇x̃ log pσ(x̃),∇x̃ log pσ(x̃|x)〉] , is a constant, leading to the conclusion LDLSM(θ) = LELSM(θ)− C1 + C2
Note that the domain of the integrals is {(x̃,x, ỹ,y) : pσ,τ (x̃,x, ỹ,y) 6= 0} by the definition of expected value.

heading: A.6 THE DETAILED EXPERIMENTAL CONFIGURATIONS
text: In this section, we elaborate on the implementation details and the experimental setups. In Section A.6.1, we describe the noise perturbation method and the sampling algorithm adopted in this work. In Sections A.6.2 and A.6.3, we present the training configurations for the experiments on the inter-twinning moon, Cifar-10, and Cifar-100 datasets.

heading: A.6.1 NOISE PERTURBATION AND SAMPLING ALGORITHM
text: Sampling Algorithm. In this work, we adopt the PC sampler (VE-SDE) identical to that in (Song et al., 2021b), which is a time-inhomogeneous sampling algorithm. The PC sampler (VE-SDE) is described in Algorithm 1. In Algorithm 1, a noise-conditioned score model s(x̃;φ, σt) is adopted, where x̃ is perturbed by noises sampled from Gaussian distributions using different standard deviations σt. The standard deviations of the noises, denoted as σt = σmin(σmaxσmin ) t T , are determined by the time step t in Algorithm 1. We set σmin = 0.01 for all of our experiments. On the other hand, we set σmax = 50 for the Cifar-10 and Cifar-100 datasets, and σmax = 10 for the inter-twinning moom dataset. As for the total time steps, the value of T is set to 1,000 for the Cifar-10 and Cifar-100 datasets, and 50 for the inter-twinning moon dataset.
Training with Different Standard Deviations. Since the PC sampler (VE-SDE) described in Algorithm 1 requires the score model s(x̃;φ, σt) conditioned on different values of standard deviation σt, the training objectives described in Eqs. (4) and (11) are modified accordingly. For the DSM loss described in Eq. (4), the modified loss is represented as:
LDSM(φ) = 1
σ2t EU[0,T ](t)
[ Epσt (x̃,x) [ ‖s(x̃;φ, σt)−∇x̃ log pσt(x̃|x)‖2 ]] , (A3)
where U[0,T ](t) is a uniform distribution on the interval [0, T ]. In Eq. (A3), The additional coefficient 1 σ2t is multiplied with the original DSM loss for better training results (Song & Ermon, 2019). For the
Algorithm 1 The PC Sampler (VE-SDE)
1: x̃ = N (0, σ2maxI) 2: for t = T − 1 to 0 do 3: x̃ ′
t ← x̃t+1 + (σ2t+1 − σ2t ) s(x̃t+1;φ, σt+1) 4: z ∼ N (0, I) 5: x̃t ← x̃ ′ t + √ σ2t+1 − σ2t z 6: for t = 1 to T do 7: z ∼ N (0, I) 8: x̃t ← x̃t + t s(x̃t;φ, σt)− √ 2 tz 9: return x̃0
approximated DLSM loss described in Eq. (9), the modified loss is represented as: LDLSM′(θ) = EU[0,T ](t) [ Epσt,τ (x̃,x,ỹ,y) [ ‖∇x̃ log p(ỹ|x̃; θ, σt) + s(x̃;φ, σt)−∇x̃ log pσt(x̃|x)‖2 ]] , (A4) where p(ỹ|x̃; θ, σt) is a noise-conditioned classifier. For the cross-entropy loss, the modified loss is represented as:
LCE(θ) = EU[0,T ](t) [ Epσt,τ (x̃,ỹ)[− log p(ỹ|x̃; θ, σt)] ] . (A5)

heading: A.6.2 THE EXPERIMENTS ON THE INTER-TWINNING MOON DATASET
text: Dataset. We perform our motivational experiments on the inter-twinning moon dataset provided by the sklearn library in python. In this dataset, the data points are sampled from two interleaving crescents. The data points of the upper crescent are generated based on (cos(x), sin(x)),∀x ∈ [0, π] and are labeled as c1. On the other hand, the data points of the lower crescent are generated based on (1− cos(x), 0.5− sin(x)),∀x ∈ [0, π] and are labeled as c2. Before training, the data points are first normalized such that their mean become 0 and then their coordinates are scaled by a factor of 20.
Setups and Evaluation Method. In this experiment, the scores∇x̃ log pσ(x̃) and∇x̃ log pσ(x̃|ỹ) are obtained by calculating Eq. (2). The likelihood scores are computed according to ∇x̃ log pσ(ỹ|x̃) = ∇x̃ log pσ(x̃|ỹ) − ∇x̃ log pσ(x̃) in Eq. (5). As for the DP and DL metrics in Table 1, the errors between two score functions are measured using 1,225 data points uniformly sampled from the two-dimensional subspace [−25, 25]× [−40, 40]. The scaling factor α adopted in method (b) is set as 5.
Training and Implementation Details. The network architectures for the score model and the classifier are simple three-layer multilayer perceptrons (MLPs) with ReLu as the activation function. These networks are implemented using the pytorch library with nn.Linear layers that contains (256,128,64) and (128,64,32) neurons per layer for the score model and the classifier, respectively. The score model is trained using the Adam optimizer with a learning rate of 6.5× 10−4 and a batch size of 2,048 for 400,000 iterations. The classifier is trained using the Adam optimizer with a learning rate of 1× 10−4 and a batch size of 4,096 for 200,000 iterations.

heading: A.6.3 THE EXPERIMENTS ON THE CIFAR-10 AND CIFAR-100 DATASETS
text: Dataset. We adopt the Cifar-10 and Cifar-100 datasets (Krizhevsky et al., 2009) for evaluating our design on image generation tasks. Cifar-10 consists of 60,000 32× 32 RGB images with 10 classes, where each class contains 6,000 images. Cifar-100 is similar to Cifar-10, except it has 100 classes, where each class contains 600 images. The images are first normalized to [−1, 1]d before training.
Setups and Evaluation Method. The results on the CAS metric are evaluated using a classifier (ResNet-50 (He et al., 2016)) trained with the generated samples. This classifier is trained to minimize the cross-entropy loss with the Adam optimizer using a learning rate of 2.0 × 10−4 with a batch size of 150 for 400,000 iterations. The results on the P / R / D / C metrics are calculated using the officially released code. The hyperparameter k used in the P / R / D / C metrics is fixed to 3. The
Table A3: The experimental results on the inter-twining moon dataset, where the classifiers are trained with LCE, LDLSM′ , and LTotal, respectively. The errors of the score functions for different methods are measured using EU(x̃)[DP (x̃, ỹ)] and EU(x̃)[DL(x̃, ỹ)], where U(x̃) is a uniform distribution over a two-dimensional subspace, ỹ represents the classes specified in the second row of the table, and DP and DL are defined in Eq. (7). The arrow symbol ↓ indicates that lower values correspond to better performances.
LCE LDLSM′ LTotal
ỹ c1 c2 c1 c2 c1 c2
EU(x̃)[DP (x̃, ỹ)] ↓ 0.216 0.217 0.115 0.126 0.080 0.095 EU(x̃)[DL(x̃, ỹ)] ↓ 0.191 0.198 0.097 0.094 0.053 0.058
Table A4: The experimental results of our method using LTotal with different value of λ. The arrow symbols ↑ / ↓ represent that a higher / lower evaluation result correspond to a better performance.
λ = 2.0 λ = 1.0 λ = 0.5 λ = 0.125 FID ↓ 2.33 2.25 2.36 2.25 IS ↑ 9.90 9.90 9.98 9.79
Precision ↑ 0.64 0.65 0.64 0.64 Recall ↑ 0.62 0.62 0.62 0.62 Density ↑ 0.94 0.96 0.95 0.94 Coverage ↑ 0.79 0.81 0.79 0.78
results on the FID and IS metrics are computed on 50,000 samples using the tensorflow_gan library in python.
Training and Implementation Details. The network architecture for the score model is the same as (Song et al., 2021b). The network architecture for the classifier is a modified ResNet-18 (He et al., 2016) for Cifar-10 and a modified ResNet-34 for Cifar-100. The modification only involves an additional conditional branch for encoding the standard deviation σt. The score model is trained with the Adam optimizer using a learning rate of 2.0× 10−4 and a batch size of 128 for 1.0× 106 iterations. The classifier is trained with the Adam optimizer using a learning rate of 2.0× 10−4 and a batch size of 100 for 1.0× 106 iterations.

heading: A.7 ADDITIONAL EXPERIMENTS
text: In this section, we present a number of additional experimental results and provide discussions on them. We first report the results of our method using different values of λ on Cifar-10 to demonstrate that LTotal is less sensitive to the choices of λ. Next, we apply the scaling technique on both the base method and our method with different choices of α to demonstrate the influence of the hyperparameter α on the evaluation results of several metrics. Next, we provide uncurated examples on the Cifar-10 and Cifar-100 datasets.

heading: A.7.1 ABLATION ANALYSIS ON THE INTER-TWINNING MOON DATASET
text: Table A3 compares the evaluation results on the EU(x̃)[DP (x̃, ỹ)] and EU(x̃)[DL(x̃, ỹ)] metrics when the classifiers are trained with LCE, LDLSM′ , and LTotal, respectively. It is observed that the score errors EU(x̃)[DP (x̃, ỹ)] and EU(x̃)[DL(x̃, ỹ)] are the lowest when LTotal is adopted. The experimental results therefore validate the effectiveness of the proposed loss LTotal.

heading: A.7.2 THE EXPERIMENTS ON THE BALANCING COEFFICIENT
text: Table A4 demonstrates the evaluation results on FID, IS, and P / R / D / C metrics using LTotal with different values of λ. It is observed that the evaluation results only slightly fluctuate on the six metrics with different choices of λ. The results thus experimentally validate that LTotal is less sensitive to the choice of λ.
Table A5: A comparison of the evaluation results of the base method, the scaling method with different α, ours method, and ours + scaling method with different α on the Cifar-10 dataset. The arrow symbols ↑ / ↓ represent that a higher / lower evaluation result correspond to a better performance.
Base Scaling Ours Ours + Scaling α - 1.0 0.5 3.0 5.0 10.0 20.0 1.0 0.5 3.0 5.0 10.0 20.0 FID ↓ 4.10 3.22 6.16 8.06 12.48 16.59 2.25 2.57 3.96 6.40 11.06 14.92 IS ↑ 9.08 9.50 9.14 9.38 9.37 9.23 9.90 9.79 10.00 9.95 9.73 9.53 Precision ↑ 0.67 0.65 0.70 0.72 0.75 0.75 0.65 0.62 0.69 0.71 0.73 0.74 Recall ↑ 0.61 0.61 0.56 0.53 0.49 0.44 0.62 0.63 0.57 0.53 0.49 0.44 Density ↑ 1.05 0.98 1.17 1.25 1.36 1.36 0.96 0.87 1.15 1.23 1.28 1.30 Coverage ↑ 0.80 0.78 0.78 0.77 0.75 0.67 0.81 0.77 0.79 0.77 0.71 0.66 (CW) Precision ↑ 0.51 0.43 0.63 0.67 0.70 0.73 0.56 0.48 0.65 0.68 0.71 0.72 (CW) Recall ↑ 0.59 0.62 0.51 0.46 0.42 0.37 0.61 0.63 0.55 0.50 0.45 0.39 (CW) Density ↑ 0.63 0.46 0.97 1.10 1.23 1.30 0.76 0.58 1.05 1.16 1.23 1.26 (CW) Coverage ↑ 0.60 0.47 0.69 0.69 0.66 0.61 0.71 0.61 0.75 0.73 0.66 0.61

heading: A.7.3 THE EXPERIMENTS ON THE SCALING TECHNIQUE
text: Table A5 demonstrates the evaluation results of the base method, the scaling method using different values of α, our method, and our method with the scaling technique using different values of α (i.e., ‘ours + scaling’) on the Cifar-10 dataset. It is observed that the performance of the ‘scaling method’ and ‘ours + scaling’ method on the fidelity metrics, i.e., precision and density, improves as the value of α grows. Conversely, the performance of the ‘scaling method’ and ‘ours + scaling’ method on the FID, recall, and coverage metrics degrades as the value of α grows. This results thus experimentally demonstrate the effects of the scaling technique on the performance of the base method and our method.

heading: A.7.4 UNCURATED EXAMPLES
text: Figs. A3, A4, A5, and A6 depict a few uncurated examples that qualitatively compare the effectiveness of different methods.
Figure A3: The uncurated examples generated using the base method on the Cifar-10 dataset.
Figure A4: The uncurated examples generated using the scaling method on the Cifar-10 dataset.
Figure A5: The uncurated examples generated using our method on the Cifar-10 dataset.
Figure A6: The uncurated examples of classes ‘baby’, ‘apple’, ‘aquarium fish’, and ‘beargenerated’ using the base method, the scaling method, and our method on the Cifar-100 dataset.

Review summaries:
The authors suggest how to apply the idea of score matching to the problem of conditional generation. They introduce a new loss, which is essentially a score matching loss for the conditional distribution p(y| x). They demonstrate how to effectively calculate it without calculating p(y|x) explicitly, which seems to be the main contribution of the paper. Then the authors demonstrate how to train the model for the conditional generation in practice: it is necessary to add to the proposed loss the usual cross-entropy loss, because the loss that they came up with is noisy (this can be seen in Figure 5). They got better results compared to baselines for cifar-10 and cifar-100.

The paper points up a previously underappreciated problem in training classifiers in the context of conditional generations of diffusion-based generative models. The authors propose a novel objective for training classifiers to tackle the problems. Informally, the generation process of the diffusion-based generative models can be described by repeatedly applying an update rule with initial values: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \sigma \epsilon$ where $\epsilon \sim N(0,I)$ and the initial points are sampled from a prior distribution such as standard Normal distributions. Similarly to unconditional generations, one can perform conditional generations, e.g., $p(\tilde{x} | y)$ by using the following update rule: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x}) + \sigma \epsilon$. Note that $\nabla_{\tilde{x}} \log p(\tilde{x} | y) = \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$. Thus, for a given pre-trained generative model, one needs to learn $\log p_{\textrm{model}}(y | \tilde{x})$ (or its gradient wrt $\tilde{x}$. Maximum likelihood training (MLE), i.e., minimizing cross-entropy loss, is commonly used. Inevitably, the qualities of the gradient $\nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$ will determine the qualities of the conditional generations.   First, the paper emphasizes that MLE-training of classifiers results in non-smooth gradient landscape wrt input; thus, the resulting gradients negatively affect the generation qualities. More precisely, with MLE, the learned $\log p_{\textrm{model}} (y|\tilde{x})$ is high, while its gradient wrt $\tilde{x}$ isn't necessarily close to the ground truth. In the paper, the authors refer to this phenomenon as *a score mismatch issue.* The authors analyze the mismatch issue with toy experiments and demonstrate its negative effects on generation qualities compared to the ground truth.   Second, to resolve the *score mismatch issue*, the paper proposes a new objective function, called Explicit Likelihood Score-Matching loss (ELSM), in which the mean squared errors between $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ and $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$ is minimized. Here, due to the inaccessibility $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$, the authors reduces the ELSM loss to another objective function, named *Denoising Likelihood Score-Matching* (DLSM), similarly to denoising score matching derivation. In DLSM, $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ is trained to match to $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x}) - \nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x} | x)$. Acknowledging that $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is still unattainable, the authors propose to minimize approximate DLSM loss, where $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is substituted by the pre-trained scores $\nabla_{\tilde{x}} \log p_{\textrm{model}} (\tilde{x})$.   Then, the authors demonstrate that by toy experiments, training classifiers with the approximate DLSM improves the conditional generation qualities compared to MLE-based training. Moreover, the authors mention that approximate DLSM-based training can be unstable in high-dimensional datasets and show that training classifiers by minimizing combined loss of approximate DLSM and cross-entropy can further improve the generation qualities.   Lastly, the paper demonstrates the effectiveness of the proposed methods by evaluating conditional generation qualities on CIFAR-10 and CIFAR-100 datasets.

The authors present a new objective, denoising likelihood score-matching (DLSM), and training mechanism for conditional score-based data generation. The new method is motivated by poor conditional score estimates observed in low-dimensional examples for the standard, Bayes-theorem-based, conditional data generation procedure of score-based models. The new objective is equivalent to directly learning to parameterise the score of p(y|x) demonstrates is high-dimensional benchmark datasets also. FID and IS improve due to this method, with the largest improvements being class-wise, and there is an overall tendency to trade off generated data recall for precision.