Denoising Likelihood Score Matching for Conditional Score-based Data Generation

1 INTRODUCTION . Score-based generative models are probabilistic generative models that estimate score functions , i.e. , the gradients of the log density for some given data distribution . According to the definition of the pioneering work ( Hyvärinen , 2005 ) , the process of training score-based generative models is called Score Matching ( SM ) , in which a score-based generative model is iteratively updated to approximate the true score function . Such a process often incurs heavy computational burdens , since it requires scanning through every data sample in a dataset to perform an update in each training iteration . Therefore , a branch of study in this research domain ( Vincent , 2011 ; Martens et al. , 2012 ; Song et al. , 2020 ) resorts to reformulating the score-matching objective to reduce the training cost . Among these works , the author in ( Vincent , 2011 ) introduced the Denoising Score-Matching ( DSM ) method . This method facilitates the training process of score-based generative models , and thus lays the foundation for a number of subsequent researches . Recently , the authors in ( Song & Ermon , 2019 ) proposed an unified framework based on DSM , and achieved remarkable performance on Cifar-10 ( Krizhevsky et al. , 2009 ) . Their success inspired several succeeding works ( Song & Ermon , 2020 ; Ho et al. , 2020 ; Song et al. , 2021a ; b ; Dhariwal & Nichol , 2021 ) , which together contribute to making score-based generative models an attractive choice for contemporary image generation tasks . A favorable aspect of score-based generative models is its flexibility to be easily extended to their conditional variants . This characteristic comes from a research direction that utilizes Bayes ’ theorem to decompose a conditional score into a mixture of scores ( Nguyen et al. , 2017 ) . Recent endeavors followed this approach and further extended the concept of conditional score-based models to a number of application domains , including colorization ( Song et al. , 2021b ) , inpainting ( Song et al. , 2021b ) , and source separation ( Jayaram & Thickstun , 2020 ) . In particular , some recent researchers ( Song et al. , 2021b ; Dhariwal & Nichol , 2021 ) applied this method to the field of classconditional image generation tasks , and proposed the classifier-guidance method . Different from the classifier-guidance-free method adopted by ( Ho et al. , 2021 ) , they utilized a score model and a classifier to generate the posterior scores ( i.e. , the gradients of the log posterior density ) , with which the data samples of certain classes can be generated through the diffusion process . The authors in ( Dhariwal & Nichol , 2021 ) showed that the classifier guidance method is able to achieve improved performance on large image generation benchmarks . In spite of their success , our analysis indicates that the conditional generation methods utilizing a score model and a classifier may suffer from a score mismatch issue , which is the situation that the estimated posterior scores deviate from the true ones . This issue causes the samples to be guided by inaccurate scores during the diffusion process , and may result in a degraded sampling quality consequently . To resolve this problem , we first analyze the potential causes for the score mismatch issue through a motivational low-dimensional example . Then , we theoretically formulate a new loss function called Denoising Likelihood Score-Matching ( DLSM ) loss , and explain how it can be integrated into the current training method . Finally , we evaluate the proposed method under various configurations , and demonstrate its advantages in improving the sampling quality over the previous methods in terms of several evaluation metrics . 2 BACKGROUND . In this section , we introduce the essential background material for understanding the contents of this paper . We first introduce Langevin diffusion ( Roberts & Tweedie , 1996 ; Roberts & Rosenthal , 1998 ) for generating data samples x̃ ∈ Rd of a certain dimension d from an unknown probability density function ( pdf ) p ( x̃ ) using the score function∇x̃ log p ( x̃ ) . Next , we describe Parzen density estimation ( Parzen , 1962 ) and denoising score matching ( DSM ) ( Vincent , 2011 ) for approximating ∇x̃ log p ( x̃ ) with limited data samples . Finally , we elaborate on the conditional variant of the score function , i.e. , ∇x̃ log p ( x̃|ỹ ) , and explains how it can be decomposed into ∇x̃ log p ( x̃ ) and ∇x̃ log p ( ỹ|x̃ ) for some conditional variable ỹ ∈ Rc of dimension c . 2.1 LANGEVIN DIFFUSION . Langevin diffusion ( Roberts & Tweedie , 1996 ; Roberts & Rosenthal , 1998 ) can be used to generate data samples from an unknown data distribution p ( x̃ ) using only the score function ∇x̃ log p ( x̃ ) , which is said to be well-defined if p ( x̃ ) is everywhere non-zero and differentiable . Under the condition that ∇x̃ log p ( x̃ ) is well-defined , Langevin diffusion enables p ( x̃ ) to be approximated iteratively based on the following equation : x̃t = x̃t−1 + 2 2 ∇x̃ log p ( x̃t−1 ) + zt , ( 1 ) where x̃0 is sampled from an arbitrary distribution , is a fixed positive step size , and zt is a noise vector sampled from a normal distribution N ( 0 , Id×d ) for simulating a d-dimensional standard Brownian motion . Under suitable regularity conditions , when → 0 and T →∞ , x̃T is generated as if it is directly sampled from p ( x̃ ) ( Roberts & Tweedie , 1996 ; Welling & Teh , 2011 ) . In practice , however , the data samples are generated with > 0 and T < ∞ , which violates the convergence guarantee . Although it is possible to use Metropolized algorithms ( Roberts & Tweedie , 1996 ; Roberts & Rosenthal , 1998 ) to recover the convergence guarantee , we follow the assumption of the prior work ( Song & Ermon , 2019 ) and presume that the errors are sufficiently small to be negligible when is small and T is large . The sampling process introduced in Eq . ( 1 ) can be extended to a time-inhomogeneous variant by making p ( x̃t ) and dependent on t ( i.e. , pt ( x̃t ) and t ) . Such a time-inhomogeneous variant is commonly adopted by recent works on score-based generative models ( Song & Ermon , 2019 ; Song et al. , 2021b ) , as it provides flexibility in controlling pt ( x̃t ) and t. The experimental results in ( Song & Ermon , 2019 ; Song et al. , 2021b ) demonstrated that such a time-inhomogeneous sampling process can improve the sampling quality on real-world datasets . In Appendix A.6.1 , we offer a discussion on the detailed implementation of such a time-inhomogeneous sampling process in this work . 2.2 PARZEN DENSITY ESTIMATION . Given a true data distribution pdata , the empirical data distribution p0 ( x ) is constructed by sampling m independent and identically distributed data points { x ( i ) } mi=1 , and can be represented as a sum of Dirac functions 1m ∑m i=1 δ ( ‖x − x ( i ) ‖ ) . Such a discrete data distribution p0 ( x ) constructed from the dataset often violates the previous assumptions that everywhere is non-zero and is differentiable . Therefore , it is necessary to somehow adjust the empirical data distribution p0 ( x ) before applying Langevin diffusion in such cases . To deal with the above issue , a previous literature ( Vincent , 2011 ) utilized Parzen density estimation to replace the Dirac functions with isotropic Gaussian smoothing kernels pσ ( x̃|x ) = 1 ( 2π ) d/2σd e −1 2σ2 ‖x̃−x‖2 with variance σ2 . Specifically , Parzen density estimation enables the calculation of pσ ( x̃ ) = 1m ∑m i=1 pσ ( x̃|x ( i ) ) . When σ > 0 , the score function becomes well-defined and can thus be represented as the following : ∇x̃ log pσ ( x̃ ) = ∑m i=1 1 σ2 ( x ( i ) − x̃ ) pσ ( x̃|x ( i ) ) ∑m i=1 pσ ( x̃|x ( i ) ) . ( 2 ) The proof for Eq . ( 2 ) is provided in Appendix A.2 . This equation can be directly applied to Eq . ( 1 ) to generate samples with Langevin diffusion . Unfortunately , this requires summation over all m data points during every iteration , preventing it from scaling to large datasets due to the rapid growth in computational complexity . 2.3 DENOISING SCORE MATCHING . Score matching ( SM ) ( Hyvärinen , 2005 ) was proposed to estimate the score function with a model s ( x̃ ; φ ) , parameterized by φ . Given a trained score model s ( x̃ ; φ ) , the scores can be generated by a single forward pass , which reduces the computational complexity of Eq . ( 2 ) by a factor of m. To train such a score model , a straightforward approach is to use the Explicit Score-Matching ( ESM ) loss LESM , represented as : LESM ( φ ) = Epσ ( x̃ ) [ 1 2 ‖s ( x̃ ; φ ) −∇x̃ log pσ ( x̃ ) ‖2 ] . ( 3 ) This objective requires evaluating Eq . ( 2 ) for each training step , which also fails to scale well to large datasets . Based on Parzen density estimation , an efficient alternative , called Denoising ScoreMatching ( DSM ) loss ( Vincent , 2011 ) , is proposed to efficiently calculate the equivalent loss LDSM , expressed as : LDSM ( φ ) = Epσ ( x̃ , x ) [ 1 2 ‖s ( x̃ ; φ ) −∇x̃ log pσ ( x̃|x ) ‖2 ] . ( 4 ) where ∇x̃ log pσ ( x̃|x ) is simply 1σ2 ( x − x̃ ) . Since the computational cost of denoising score matching is relatively lower in comparison to other reformulation techniques ( Hyvärinen , 2005 ; Song et al. , 2020 ) , it is extensively adopted in recent score-based generative models ( Song & Ermon , 2019 ; 2020 ; Song et al. , 2021b ) . 2.4 CONDITIONAL SCORE DECOMPOSITION VIA BAYES ’ THEOREM . Score models can be extended to conditional models when conditioned on a certain label ỹ . Similar to x̃ , the smoothing kernels with variance τ2 can be applied on y to meet the requirement that the pdf is everywhere non-zero and differentiable . Typically , τ is assumed to be sufficiently small so that pτ ( ỹ ) ≈ p ( y ) . A popular approach adopted by researchers utilizes Bayes ’ theorem pσ , τ ( x̃|ỹ ) = pσ , τ ( ỹ|x̃ ) pσ ( x̃ ) /pτ ( ỹ ) to decompose the conditional score ∇x̃ log pσ , τ ( x̃|ỹ ) into a mixture of scores ( Nguyen et al. , 2017 ) , which enables conditional data generation . Following the assumptions in the previous study ( Song et al. , 2021b ) , the decomposition can be achieved by taking the loggradient on both sides of the equation , expressed as follows : ∇x̃ log pσ , τ ( x̃|ỹ ) = ∇x̃ log pσ , τ ( ỹ|x̃ ) +∇x̃ log pσ ( x̃ ) −∇x̃ log pτ ( ỹ ) ︸ ︷︷ ︸ =0 , ( 5 ) where ∇x̃ log pσ , τ ( x̃|ỹ ) is the posterior score , ∇x̃ log pσ , τ ( ỹ|x̃ ) is the likelihood score , and ∇x̃ log pσ ( x̃ ) is the prior score . Base on Eq . ( 5 ) , a conditional score model ∇x̃ log p ( x̃|ỹ ; φ , θ ) can be represented as the combination of the log-gradient of a differentiable classifier p ( ỹ|x̃ ; θ ) and a prior score model s ( x̃ ; φ ) : ∇x̃ log p ( x̃|ỹ ; θ , φ ) = ∇x̃ log p ( ỹ|x̃ ; θ ) + s ( x̃ ; φ ) . ( 6 ) This formulation enables conditional data generation using a classifier p ( ỹ|x̃ ; θ ) trained with crossentropy loss LCE ( θ ) , Epσ , τ ( x̃ , ỹ ) [ − log p ( ỹ|x̃ ; θ ) ] , and a score model s ( x̃ ; φ ) trained with denoising score matching loss LDSM ( φ ) . Unfortunately , a few previous studies ( Nguyen et al. , 2017 ; Dhariwal & Nichol , 2021 ) have noticed that the approximation of∇x̃ log p ( x̃|ỹ ; θ , φ ) is empirically inaccurate ( referred to as the score mismatch issue in this work ) , and leveraged a scaling factor α > 0 to adjust the likelihood score ∇x̃ log p ( ỹ|x̃ ; θ ) . Such a scaling factor is a hyperparameter that controls the amount of conditional information incorporated during the sampling process ( see Appendix A.3 ) . However , this usually causes the diversity of generated data samples to degrade noticeably ( Dhariwal & Nichol , 2021 ) , which is later discussed in Section 5.2 . In this work , we examine and investigate the score mismatch issue from a different perspective . We first offer a motivational example to show that the issue remains even with the scaling technique . Then , we theoretically formulate a new loss term that enables a classifier p ( ỹ|x̃ ; θ ) to produce better likelihood score estimation .

Summary: The paper points up a previously underappreciated problem in training classifiers in the context of conditional generations of diffusion-based generative models. The authors propose a novel objective for training classifiers to tackle the problems. Informally, the generation process of the diffusion-based generative models can be described by repeatedly applying an update rule with initial values: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \sigma \epsilon$ where $\epsilon \sim N(0,I)$ and the initial points are sampled from a prior distribution such as standard Normal distributions. Similarly to unconditional generations, one can perform conditional generations, e.g., $p(\tilde{x} | y)$ by using the following update rule: $\tilde{x} ← \tilde{x} + \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x}) + \sigma \epsilon$. Note that $\nabla_{\tilde{x}} \log p(\tilde{x} | y) = \nabla_{\tilde{x}} \log p_{\textrm{model}}(\tilde{x}) + \nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$. Thus, for a given pre-trained generative model, one needs to learn $\log p_{\textrm{model}}(y | \tilde{x})$ (or its gradient wrt $\tilde{x}$. Maximum likelihood training (MLE), i.e., minimizing cross-entropy loss, is commonly used. Inevitably, the qualities of the gradient $\nabla_{\tilde{x}} \log p_{\textrm{model}}(y | \tilde{x})$ will determine the qualities of the conditional generations.   First, the paper emphasizes that MLE-training of classifiers results in non-smooth gradient landscape wrt input; thus, the resulting gradients negatively affect the generation qualities. More precisely, with MLE, the learned $\log p_{\textrm{model}} (y|\tilde{x})$ is high, while its gradient wrt $\tilde{x}$ isn't necessarily close to the ground truth. In the paper, the authors refer to this phenomenon as *a score mismatch issue.* The authors analyze the mismatch issue with toy experiments and demonstrate its negative effects on generation qualities compared to the ground truth.   Second, to resolve the *score mismatch issue*, the paper proposes a new objective function, called Explicit Likelihood Score-Matching loss (ELSM), in which the mean squared errors between $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ and $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$ is minimized. Here, due to the inaccessibility $\nabla_{\tilde{x}} \log p_{\textrm{data}} (y|\tilde{x})$, the authors reduces the ELSM loss to another objective function, named *Denoising Likelihood Score-Matching* (DLSM), similarly to denoising score matching derivation. In DLSM, $\nabla_{\tilde{x}} \log p_{\textrm{model}} (y|\tilde{x})$ is trained to match to $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x}) - \nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x} | x)$. Acknowledging that $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is still unattainable, the authors propose to minimize approximate DLSM loss, where $\nabla_{\tilde{x}} \log p_{\textrm{data}} (\tilde{x})$ is substituted by the pre-trained scores $\nabla_{\tilde{x}} \log p_{\textrm{model}} (\tilde{x})$.   Then, the authors demonstrate that by toy experiments, training classifiers with the approximate DLSM improves the conditional generation qualities compared to MLE-based training. Moreover, the authors mention that approximate DLSM-based training can be unstable in high-dimensional datasets and show that training classifiers by minimizing combined loss of approximate DLSM and cross-entropy can further improve the generation qualities.   Lastly, the paper demonstrates the effectiveness of the proposed methods by evaluating conditional generation qualities on CIFAR-10 and CIFAR-100 datasets.