Annealed Denoising score matching: learning Energy based model in high-dimensional spaces

1 INTRODUCTION AND MOTIVATION . Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core for solving a large variety of application problems , such as error correction/denoising ( Vincent et al. , 2010 ) , outlier/novelty detection ( Zhai et al. , 2016 ; Choi and Jang , 2018 ) , sample generation ( Nijkamp et al. , 2019 ; Du and Mordatch , 2019 ) , invariant pattern recognition , Bayesian reasoning ( Welling and Teh , 2011 ) which relies on good data priors , and many others . Energy-Based Models ( EBMs ) ( LeCun et al. , 2006 ; Ngiam et al. , 2011 ) assign an energy E ( x ) to each data point x which implicitly defines a probability by the Boltzmann distribution pm ( x ) = e−E ( x ) /Z . Sampling from this distribution can be used as a generative process that yield plausible samples of x . Compared to other generative models , like GANs ( Goodfellow et al. , 2014 ) , flowbased models ( Dinh et al. , 2015 ; Kingma and Dhariwal , 2018 ) , or auto-regressive models ( van den Oord et al. , 2016 ; Ostrovski et al. , 2018 ) , energy-based models have significant advantages . First , they provide explicit ( unnormalized ) density information , compositionality ( Hinton , 1999 ; Haarnoja et al. , 2017 ) , better mode coverage ( Kumar et al. , 2019 ) and flexibility ( Du and Mordatch , 2019 ) . Further , they do not require special model architecture , unlike auto-regressive and flow-based models . Recently , Energy-based models has been successfully trained with maximum likelihood ( Nijkamp et al. , 2019 ; Du and Mordatch , 2019 ) , but training can be very computationally demanding due to the need of sampling model distribution . Variants with a truncated sampling procedure have been proposed , such as contrastive divergence ( Hinton , 2002 ) . Such models learn much faster with the draw back of not exploring the state space thoroughly ( Tieleman , 2008 ) . 1.1 SCORE MATCHING , DENOISING SCORE MATCHING AND DEEP ENERGY ESTIMATORS . Score matching ( SM ) ( Hyvärinen , 2005 ) circumvents the requirement of sampling the model distribution . In score matching , the score function is defined to be the gradient of log-density or the negative energy function . The expected L2 norm of difference between the model score function and the data score function are minimized . One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator ( Parzen , 1962 ) of the data : pσ0 ( x̃ ) = ∫ qσ0 ( x̃|x ) p ( x ) dx . Though hard to evaluate , the data score is well defined : sd ( x̃ ) = ∇x̃ log ( pσ0 ( x̃ ) ) , and the corresponding objective is : LSM ( θ ) = Epσ0 ( x̃ ) ‖ ∇x̃ log ( pσ0 ( x̃ ) ) +∇x̃E ( x̃ ; θ ) ‖ 2 ( 1 ) Vincent ( 2011 ) studied the connection between denoising auto-encoder and score matching , and proved the remarkable result that the following objective , named Denoising Score Matching ( DSM ) , is equivalent to the objective above : LDSM ( θ ) = Epσ0 ( x̃ , x ) ‖ ∇x̃ log ( qσ0 ( x̃|x ) ) +∇x̃E ( x̃ ; θ ) ‖ 2 ( 2 ) Note that in ( 2 ) the Parzen density score is replaced by the derivative of log density of the single noise kernel ∇x̃ log ( qσ0 ( x̃|x ) ) , which is much easier to evaluate . In the particular case of Gaussian noise , log ( qσ0 ( x̃|x ) ) = − ( x̃−x ) 2 2σ20 + C , and therefore : LDSM ( θ ) = Epσ0 ( x̃ , x ) ‖ x − x̃ + σ0 2∇x̃E ( x̃ ; θ ) ‖2 ( 3 ) The interpretation of objective ( 3 ) is simple , it forces the energy gradient to align with the vector pointing from the noisy sample to the clean data sample . To optimize an objective involving the derivative of a function defined by a neural network , Kingma and LeCun ( 2010 ) proposed the use of double backpropagation ( Drucker and Le Cun , 1991 ) . Deep energy estimator networks ( Saremi et al. , 2018 ) first applied this technique to learn an energy function defined by a deep neural network . In this work and similarly in Saremi and Hyvarinen ( 2019 ) , an energy-based model was trained to match a Parzen density estimator of data with a certain noise magnitude . The previous models were able to perform denoising task , but they were unable to generate high-quality data samples from a random input initialization . Recently , Song and Ermon ( 2019 ) trained an excellent generative model by fitting a series of score estimators coupled together in a single neural network , each matching the score of a Parzen estimator with a different noise magnitude . The questions we address here is why learning energy-based models with single noise level does not permit high-quality sample generation and what can be done to improve energy based models . Our work builds on key ideas from Saremi et al . ( 2018 ) ; Saremi and Hyvarinen ( 2019 ) ; Song and Ermon ( 2019 ) . Section 2 , provides a geometric view of the learning problem in denoising score matching and provides a theoretical explanation why training with one noise level is insufficient if the data dimension is high . Section 3 presents a novel method for training energy based model , Multiscale Denoising Score Matching ( MDSM ) . Section 4 describes empirical results of the MDSM model and comparisons with other models . 2 A GEOMETRIC VIEW OF DENOISING SCORE MATCHING . Song and Ermon ( 2019 ) used denoising score matching with a range of noise levels , achieving great empirical results . The authors explained that large noise perturbation are required to enable the learning of the score in low-data density regions . But it is still unclear why a series of different noise levels are necessary , rather than one single large noise level . Following Saremi and Hyvarinen ( 2019 ) , we analyze the learning process in denoising score matching based on measure concentration properties of high-dimensional random vectors . We adopt the common assumption that the data distribution to be learned is high-dimensional , but only has support around a relatively low-dimensional manifold ( Tenenbaum et al. , 2000 ; Roweis and Saul , 2000 ; Lawrence , 2005 ) . If the assumption holds , it causes a problem for score matching : The density , or the gradient of the density is then undefined outside the manifold , making it difficult to train a valid density model for the data distribution defined on the entire space . Saremi and Hyvarinen ( 2019 ) and Song and Ermon ( 2019 ) discussed this problem and proposed to smooth the data distribution with a Gaussian kernel to alleviate the issue . To further understand the learning in denoising score matching when the data lie on a manifold X and the data dimension is high , two elementary properties of random Gaussian vectors in highdimensional spaces are helpful : First , the length distribution of random vectors becomes concentrated at √ dσ ( Vershynin , 2018 ) , where σ2 is the variance of a single dimension . Second , a random vector is always close to orthogonal to a fixed vector ( Tao , 2012 ) . With these premises one can visualize the configuration of noisy and noiseless data points that enter the learning process : A data point x sampled from X and its noisy version x̃ always lie on a line which is almost perpendicular to the tangent space TxX and intersects X at x . Further , the distance vectors between ( x , x̃ ) pairs all have similar length √ dσ . As a consequence , the set of noisy data points concentrate on a set X̃√dσ , that has a distance with ( √ dσ − , √ dσ + ) from the data manifold X , where √ dσ . Therefore , performing denoising score matching learning with ( x , x̃ ) pairs generated with a fixed noise level σ , which is the approach taken previously except in Song and Ermon ( 2019 ) , will match the score in the set X̃√dσ , and enable denoising of noisy points in the same set . However , the learning provides little information about the density outside this set , farther or closer to the data manifold , as noisy samples outside X̃√dσ , rarely appear in the training process . An illustration is presented in Figure 1A . Let X̃C√ dσ , denote the complement of the set X̃√dσ , . Even if pσ0 ( x̃ ∈ X̃ C√ dσ , ) is very small in high-dimensional space , the score in X̃C√ dσ , still plays a critical role in sampling from random initialization . This analysis may explain why models based on denoising score matching , trained with a single noise level encounter difficulties in generating data samples when initialized at random . For an empirical support of this explanation , see our experiments with models trained with single noise magnitudes ( Appendix B ) . To remedy this problem , one has to apply a learning procedure of the sort proposed in Song and Ermon ( 2019 ) , in which samples with different noise levels are used . Depending on the dimension of the data , the different noise levels have to be spaced narrowly enough to avoid empty regions in the data space . In the following , we will use Gaussian noise and employ a Gaussian scale mixture to produce the noisy data samples for the training ( for details , See Section 3.1 and Appendix A ) . Another interesting property of denoising score matching was suggested in the denoising autoencoder literature ( Vincent et al. , 2010 ; Karklin and Simoncelli , 2011 ) . With increasing noise level , the learned features tend to have larger spatial scale . In our experiment we observe similar phenomenon when training model with denoising score matching with single noise scale . If one compare samples in Figure B.1 , Appendix B , it is evident that noise level of 0.3 produced a model that learned short range correlation that spans only a few pixels , noise level of 0.6 learns longer stroke structure without coherent overall structure , and noise level of 1 learns more coherent long range structure without details such as stroke width variations . This suggests that training with single noise level in denoising score matching is not sufficient for learning a model capable of high-quality sample synthesis , as such a model have to capture data structure of all scales . 3 LEARNING ENERGY-BASED MODEL WITH MULTISCALE DENOISING SCORE MATCHING . 3.1 MULTISCALE DENOISING SCORE MATCHING . Motivated by the analysis in section 2 , we strive to develop an EBM based on denoising score matching that can be trained with noisy samples in which the noise level is not fixed but drawn from a distribution . The model should approximate the Parzen density estimator of the data pσ0 ( x̃ ) =∫ qσ0 ( x̃|x ) p ( x ) dx . Specifically , the learning should minimize the difference between the derivative of the energy and the score of pσ0 under the expectation EpM ( x̃ ) rather than Epσ0 ( x̃ ) , the expectation taken in standard denoising score matching . Here pM ( x̃ ) = ∫ qM ( x̃|x ) p ( x ) dx is chosen to cover the signal space more evenly to avoid the measure concentration issue described above . The resulting Multiscale Score Matching ( MSM ) objective is : LMSM ( θ ) = EpM ( x̃ ) ‖ ∇x̃ log ( pσ0 ( x̃ ) ) +∇x̃E ( x̃ ; θ ) ‖ 2 ( 4 ) Compared to the objective of denoising score matching ( 1 ) , the only change in the new objective ( 4 ) is the expectation . Both objectives are consistent , if pM ( x̃ ) and pσ0 ( x̃ ) have the same support , as shown formally in Proposition 1 of Appendix A . In Proposition 2 , we prove that Equation 4 is equivalent to the following denoising score matching objective : LMDSM∗ = EpM ( x̃ ) qσ0 ( x|x̃ ) ‖ ∇x̃ log ( qσ0 ( x̃|x ) ) +∇x̃E ( x̃ ; θ ) ‖ 2 ( 5 ) The above results hold for any noise kernel qσ0 ( x̃|x ) , but Equation 5 contains the reversed expectation , which is difficult to evaluate in general . To proceed , we choose qσ0 ( x̃|x ) to be Gaussian , and also choose qM ( x̃|x ) to be a Gaussian scale mixture : qM ( x̃|x ) = ∫ qσ ( x̃|x ) p ( σ ) dσ and qσ ( x̃|x ) = N ( x , σ2Id ) . After algebraic manipulation and one approximation ( see the derivation following Proposition 2 in Appendix A ) , we can transform Equation 5 into a more convenient form , which we call Multiscale Denoising Score Matching ( MDSM ) : LMDSM = Ep ( σ ) qσ ( x̃|x ) p ( x ) ‖ ∇x̃ log ( qσ0 ( x̃|x ) ) +∇x̃E ( x̃ ; θ ) ‖ 2 ( 6 ) The square loss term evaluated at noisy points x̃ at larger distances from the true data points x will have larger magnitude . Therefore , in practice it is convenient to add a monotonically decreasing term l ( σ ) for balancing the different noise scales , e.g . l ( σ ) = 1σ2 . Ideally , we want our model to learn the correct gradient everywhere , so we would need to add noise of all levels . However , learning denoising score matching at very large or very small noise levels is useless . At very large noise levels the information of the original sample is completely lost . Conversely , in the limit of small noise , the noisy sample is virtually indistinguishable from real data . In neither case one can learn a gradient which is informative about the data structure . Thus , the noise range needs only to be broad enough to encourage learning of data features over all scales . Particularly , we do not sample σ but instead choose a series of fixed σ values σ1 · · ·σK . Further , substituting log ( qσ0 ( x̃|x ) ) = − ( x̃−x ) 2 2σ20 + C into Equation 4 , we arrive at the final objective : L ( θ ) = ∑ σ∈ { σ1···σK } Eqσ ( x̃|x ) p ( x ) l ( σ ) ‖ x − x̃ + σ 2 0∇x̃E ( x̃ ; θ ) ‖2 ( 7 ) It may seem that σ0 is an important hyperparameter to our model , but after our approximation σ0 become just a scaling factor in front of the energy function , and can be simply set to one as long as the temperature range during sampling is scaled accordingly ( See Section 3.2 ) . Therefore the only hyper-parameter is the rang of noise levels used during training . On the surface , objective ( 7 ) looks similar to the one in Song and Ermon ( 2019 ) . The important difference is that Equation 7 approximates a single distribution , namely pσ0 ( x̃ ) , the data smoothed with one fixed kernel qσ0 ( x̃|x ) . In contrast , Song and Ermon ( 2019 ) approximate the score of multiple distributions , the family of distributions { pσi ( x̃ ) : i = 1 , ... , n } , resulting from the data smoothed by kernels of different widths σi . Because our model learns only a single target distribution , it does not require noise magnitude as input .

Summary: This paper presents a method of learning of energy based models using denoising score matching. This technique has been used before but only with limited success. The authors hypothesize that this is due to the fact that the matching was only performed over a single noise scale. The main idea of this work is to employ a range of scales to learn a single energy function. This trick helps to alleviate the problem of noisy samples concentrating in a low-volume region of the ambient space.