Annealed Denoising score matching: learning Energy based model in high-dimensional spaces
Authors: Zengyi Li
Yubei Chen
Friedrich T. Sommer
Venue:  (2020)
Keywords: Energy based models
score matching
annealing
likelihood
generative model
unsupervised learning

Abstract: Energy  based  models  outputs  unmormalized  log-probability  values  given  datasamples.  Such a estimation is essential in a variety of application problems suchas sample generation, denoising, sample restoration, outlier detection, Bayesianreasoning, and many more.  However, standard maximum likelihood training iscomputationally expensive due to the requirement of sampling model distribution.Score matching potentially alleviates this problem, and denoising score matching(Vincent, 2011) is a particular convenient version.  However,  previous attemptsfailed to produce models capable of high quality sample synthesis.  We believethat  it  is  because  they  only  performed  denoising  score  matching  over  a  singlenoise scale. To overcome this limitation, here we instead learn an energy functionusing all noise scales.   When sampled using Annealed Langevin dynamics andsingle step denoising jump, our model produced high-quality samples comparableto state-of-the-art techniques such as GANs, in addition to assigning likelihood totest data comparable to previous likelihood models.  Our model set a new sam-ple quality baseline in likelihood-based models.  We further demonstrate that our model learns sample distribution and generalize well on an image inpainting tasks.

Under review as a conference paper at ICLR 2020
LEARNING ENERGY-BASED MODELS IN HIGH- DIMENSIONAL SPACES WITH MULTI-SCALE DENOIS- ING SCORE MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
Energy-Based Models (EBMs) assign unnormalized log-probability to data sam- ples. This functionality has variety of applications, such sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood ex- tremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score match- ing (Vincent, 2011) has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good re- sults in data denoising (Saremi and Hyvarinen 2019). However, demonstrations such models high quality sample synthesis of high dimensional data were lacking. Recently, [Song and Ermon (2019) have shown that generative model trained by denoising score matching accomplishes excellent sample synthesis, when trained with data samples corrupted with multiple levels of noise. Here we provide analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose novel EBM trained with multi-scale denoising score matching. Our model exhibits data generation performance comparable to state-of-the teftthh-rtt tech- niques such as (Song and Ermon 2019) and GANs, and sets new baseline for EBMs. The proposed model also provides density information and performs well in an image inpainting task.
INTRODUCTION AND MOTIVATION
Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is the core for solving large variety of application problems, such as error correction/denoising (Vincent et al. 2010), outlier/novelty detection (Zhai et al.] [016]] Choi and Jang 2018), sample generation (Nijkamp al. 2019 Du and Mordatch] 2019), invariant pattern recognition, Bayesian reasoning (Welling and Teh others.
2011 which relies on good data priors, and many
Energy-Based Models (EBMs) (LeCun et al. 2006 Ngiam et al. 2011 assign an energy F(x) to each data point which implicitly detines probability by the Boltzmann distribution Pm (z) e-((x)/2 /Z. Sampling from this distribution can be used generative process that yield plausible samples of x. Compared to other generative models, like GANs (Goodfellow et al., 2014). flow- based models (Dinh al., 2015 Kingma and Dhariwal 2018), or auto-regressive models (van den Oord et al. 2016 Ostrovski et al. 2018), energy-based models have significant advantages. First, they provide explicit (unnormalized) density information, compositionality (Hinton 1999 |999|Haarooo] HHarrojja et al. 2017). better mode coverage (Kumar et al.] 2019) and flexibility (Du and Mordatch] 2019). Further, they do not require special model architecture, unlike auto-regressive and flow-based mod- els. Recently, Energy-based models has been successfully trained with maximum likelihood (Ni- jkamp et al. 2019 Du and Mordatch| 2011)), but training can be very computationally demanding due to the need of sampling model distribution. Variants with truncated sampling procedure have been proposed, such as contrastive divergence (Hinton 2002). Such models learn much faster with the draw back of not exploring the state space thoroughly (Tieleman 2008).

---

Under review as a conference paper at ICLR 2020
1.1 SCORE MATCHING, DENOISING SCORE MATCHING AND DEEP ENERGY ESTIMATORS
Score matching (SM) (Hyvärinen 2005) circumvents the requirement of sampling the model dis- tribution. In score matching. the score function is defined to be the gradient of log-density or the negative energy function. The expected L2 norm of difference between the model score function exppiiizen
and the data score function are minimized. One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator (Parzen 1962) of the data: Poo (it = Joo ( x (l)))((z)d Though hard evaluate, the data score is well defined: sa( Va log(Poo (z)). and the corresponding objective is:
LgM = E (p) V log(poo (5)) V F(; ||
Vincent (201T) studied the connection between denoising auto-encoder and score matching, and proved the remarkable result that the following objective, named Denoising Score Matching (DSM), equivalent to the objective above:
LDSM (0) = pep (.)
V (00()00(z)x)) + V= ||
(2)
Note that in ( the Parzen density score is replaced by the derivative of log density of the single noise kernel Vi log(9og (z)z))) which is mueh easier to evaluate. the particular case of Gaussian
noise, log(2ro (z[z))
(2-x)²
200 C, and therefore:
LDsM = (p)o((.)) || x +0 2 VxE(E;)) 1|2
(3)
The interpretation of objective (3) is simple, forees the energy gradient to align with the vector pointing from the noisy sample to the clean data sample. To optimize an objective involving the derivative of function defined by a neural network, Kingma and LeCun (2010) proposed the use of double backpropagation (Drucker and Le Cun 1991>. Deep energy estimator networks (Saremi 20188) first applied this technique to learn an energy function defined by a deep neural network. In this work and similarly in /arrmi and Hyvarinen (2019), energy-based model was trained to match a Parzen density estimator of data with a certain noise magnitude. The previous models were able to perform denoising task, but they were unable generate high-quality data samples from random input initialization. Recently, (Song and Ermon (2019) trained an excellent generative model by fitting series of score estimators coupled together single neural network, each matching the score of a Parzen estimator with different noise magnitude.
The questions we address here is why learning energy- models with single noise level does not permit high-quality sample generation and what can be done to improve energy based models. Our work builds on key ideas from /aremi et al. 2); [Saremi and Hyvarinen (019). Section
[Snn and Ermon 2 provides a geometric view of the learning problem in denoising score matching and provides a theoretical explanation why training with one noise level is insufficient if the data dimension is high. Section 3 presents novel method for training energy based model, Multiscale Denoising Score Matching (MDSM). Deeciiondeserbbee empirical results of the MDSM model and comparisons with other models.
A GEOMETRIC VIEW OF DENOISING SCORE MATCHING
Song and Ermon 2019) used denoising score matching with range of noise levels, achieving great empirical results. The authors explained that large noise perturbation are required to enable the learning of the score in low-data density regions. But it is still unclear why a series of different Saremii
noise levels are necessary, rather than single large noise level. Following Saremi and Hyvarinen (2019), we analyze the learning process in denoising score matching based on measure concentration properties of high-dimensional random vectors.
We adopt the common assumption that the data distribution to be learned is high- nigh-dimensional, but only has support around relatively low-dimensional manifold (Tenebbaum et al. 2000; Roweis and Saul 2000 Lawrence] 2005). the assumption holds, it causes a problem for score matching: The density, or the gradient of the density is then undefined outside the manifold, making it difficult
2

---

Under review as a conference paper at ICLR 2020
A. Training
B. Sampling
30
Single noise level
nolse levels
Figure 1: Illustration of anneal denoising score matching, A. During training, derivative of log- likelihood forced to point toward data manifold, establishing energy difference between points within manifold and points outside. Note that energy is negative log-likelihood therefore energy is higher for point further away from data manifold. B. During annealed Langevin sampling, sample travel from outside data manifold data manifold. Shown are singled step denoised sample during sampling of an energy function trained with MDSM on Fashion-MNIST (see text for details).
to train a valid density model for the data distribution defined on the entire space. [Saremi and Hyvarinen (2019) and (Sog and Ermon (2019) discussed this problem and proposed to smooth the data distribution with Gaussian kernel to alleviate the issue.
To further understand the learning in denoising score matching when the data lie manifold X and the data dimension high, two elementary properties of random Gaussian vectors high- dimensional spaces are helpful: First, the length distribution of random vectors becomes concen- trated at vda (Vershynin 2018), where o² is the variance of a single dimension. Second, a random vector is always close to orthogonal to a fixed vector (To 2012). With these premises one can visualize the configuration of noisy and noiseless data points that enter the learning process: A data point & sampled from X' and its noisy version & always lie on a line which is almost perpendicular the tangent space Tr and intersects X' at æ. Further, the distance vectors between (x, pairs all have similar length dda. As consequence, the set of noisy data points concentrate on set & Vdo.e that has distance with (vdo e, vo + e) from the data manifold X,, where vol.
Therefore, performing denoising score matching learning with (a, pairs generated with a fixed noise level a, which is the approach taken previously except in [Snn and Ermon (2019), will match the score in the set X Ldd. and enable denoising of noisy points in the same set. However, the learning provides little information about the density outside this set, farther closer to the data manifold, noisy samples outside X Vaa, rarely appear in the training process. An illustration is presented in Figure TA.
Vdo,e denote the complement of the set &Vdo.e Even if Pop ( E XC is very small in Vdo,t high-dimensional space, the score in Ee still plays critical role sampling from random Vdo,e initialization. This analysis may explain why models based on denoising score matching, trained with single noise level encounter difficulties generating data samples when initialized at random. For an empirical support of this explanation, see our experiments with models trained with single noise magnitudes (Appendix B). To remedy this problem, one has apply a learning procedure of the sort proposed in Song and Ermon (2019), in which samples with different noise levels are used. Depending on the dimension of the data, the different noise levels have be spaced narrowly enough to avoid empty regions in the data space. In the following, will use Gaussian noise and employ Gaussian scale mixture to produce the noisy data samples for the training (for details, See Section 3.1 and Appendix A)
Let &CC.
Another interesting property of denoising score matching was suggested in the denoising auto- encoder literature (Vincent et al. 2010 Karklin and Simoncelli 2011). With increasing noise level, the learned features tend to have larger spatial scale. In our experiment we observe similar phe- nomenon when training model with denoising
matching with single noise scale. If one com- pare samples in Figure [. Appendix B is evident that noise level of 0.3 produced a model that learned short range correlation that spans only few pixels, noise level of 0.6 learns longer stroke
3

---

Under review as a conference paper at ICLR 2020
structure without coherent overall structure, and noise level of 1 learns more coherent long range structure without details such as stroke width variations. This suggests that training with single noise level in denoising score matching is not sufficient for learning model capable of high-quality sample synthesis, as such a model have to capture data structure of all scales.
3. LEARNING ENERGY-BASED MODEL WITH MULTISCALE DENOISING SCORE MATCHING
3.1 MULTISCALE DENOISING SCORE MATCHING
Motivated by the analysis in section 2 strive develop an EBM based on denoising score matching that can be trained with noisy samples in which the noise level is not fixed but drawn from distribution. The model should approximate the Parzen density estimator of the data Pao (ã) I 90. (ã x )p(x) )x. Specifically, the learning should minimize the difference between the derivative of the energy and the score of Pro under the expectation Eas (#) rather than Eyog (é), the expectation
taken in standard denoising score matching. Here Pas (0) = ʃ qur ar p((x) drr chosen to cover the signal space more evenly to avoid the measure concentration issue described above. The resulting Multiscale Score Matching (MSM) objective is:
LMSM (0) (x) ||
Va log(Pro (#)) + E(; 11?
Compared to the objective of denoising score matching UI the only change in the new objective 4 is the expectation. Both objectives are consistent, if Pas (i) and Poo (z) have the same support,
as shown formally in Proposition [ of Appendix [ Proposition 2 we prove that Equation equivalent to the following denoising score matching objective:
LMDSM- = -Ep (E) (m)))
log({=o(ã)x)) E+E(2;0) 11?
(5)
The above results hold for any noise kernel go, (i|z), but Equation 5 contains the reversed ex- pectation, which is diffieult to evaluate in general. To proceed, choose you (i (z) to be Gaus- sian, and also choose qas (क) 2) to be a Gaussian scale mixture: qar (x]x)) = J qa (Ā x x)p(ar)do and qo (z/zz) = Nzx o² I2). After algebraic manipulation and one approximation (see the derivation following Proposition in Appendix A), can transform Equation (aniiionniiin into more convenient form, which we call Multiscale Denoising Score Matching (MDSM):
LMDSM = Ep(orra (#(x)p(x) ||
V Log((0o (ž(z)) + VzE(Z;0) ||
(6)
The square loss term evaluated at noisy points è at larger distances from the true data points will have larger magnitude. Therefore, in practice it is convenient add a monotonically decreasing term l(ar for balancing the different noise scales, e.g. I(0) 1 z² Ideally, we want our model to
learn the correct gradient everywhere, so we would need to add noise of all levels. However, learning denoising score matching at very large very small noise levels is useless. At very large noise levels the information the original sample is completely lost. Conversely, the limit of small noise, the noisy sample is virtually indistinguishable from real data. In neither case can learn gradient which is informative about the data structure. Thus, the noise range needs only to be broad enough to encourage learning of data features over all scales. Particularly, do sample but instead
choose a series of fixed a values ai ... O.- Further, substituting log(4ro (z/==)
x)=
C
20
into Equation we arrive at the final objective:
L(0)
Σ
(())))))))) || x ã of 1|
(7)
oE(
may seem that dg is an important hyperparameter to our model, but after our approximation de become just scaling factor in front of the energy function, and can be simply set to as long as
the temperature range during sampling is scaled accordingly (See Section 3.2). Therefore the only hyper-parameter the rang of noise levels used during training.
4

---

Under review as a conference paper at ICLR 2020
On the surface, objective looks similar to the one in Song and Ermon (2019). The important dif- ference is that Equation 7 approximates a single distribution, namely Pao (z), the data smoothed with one fixed kernel 9r, ). In contrast, Song and Ermon (2019) approximate the score of multiple distributions, the family of distributions (Po, (0)
..... J.,, resulting from the data smoothed by kernels of different widths o,. Because our model learns only single target distribution, it does not require noise magnitude as input.
3.2 SAMPLING BY ANNEALED LANGEVIN DYNAMICS
Langevin dynamics has been used to sample from neural network energy functions (Du and Mor- datch] 2019 Nijkamp et al. 2019). However, the studies described difficulties with mode exploration unless very large number of sampling steps is used. To improve mode exploration, we propose in- corporating simulated annealing in the Langevin dynamics. Simulated annealing (Kirkpatrick al.) 1983 Neal 200T) improves mode exploration by sampling first at high temperature and then cooling down gradually. This has been successfully applied to challenging computational problems, such as combinatorial optimization.
To apply simulated annealing to Langevin dynamics. Note that in model of Brownian motion of physical particle, the temperature in the Langevin equation enters as factor vT in front of the noise term, some literature uses /B-1 where 3 1/T (Jordan et al. 1998). Adopting the vT convention, the Langevin sampling process (Bellec et al. 2017) Is given by:
Ft+1 = x 2 V + +e/T.N(0,
(8)
where T. follows some annealing schedule, and denotes step length, which is fixed. During sam- pling, samples behave very much like physical particles under Brownian motion in a potential field. Because the particles have average energies close to the their current thermic energy. they explore the state space at different distances from data manifold depending on temperature. Eventually, they settle somewhere on the data manifold. The behavior of the particle's energy value during a typical annealing process is depicted in Appendix Figure F.I.
If the obtained sample is still slightly noisy, we can apply single step gradient denoising jump (Saremi and Hyvarinen 2019) to improve sample quality:
Felean Enossy of Vi E( 0)
This denoising procedure can be applied to noisy sample with any level of Gaussian noise because in our model the gradient automatically has the right magnitude to denoise the sample. This process is justified by the Empirical Bayes interpretation of this denoising process, as studied in Saremi and
Hyvarinen (219).
Song and Ermon 2019) also call their sample generation process annealed Langevin dynamics, It should be noted that their sampling process does not coincide with Equation g Their sampling procedure is best understood as sequentially sampling a series of distributions corresponding to data distribution corrupted by different levels of noise.
IMAGE MODELING RESULTS
Training and Sampling Details. The proposed energy-based model is trained on standard image datasets, specifically MNIST, Fashion MNIST, CelebA (Liu et al 2015) and CIFAR-10 (Krizhevsky al. (2009). During training we set do = 0.1 and train over a noise range the different noise uniformly spaced on the batch dimension. For MNIST and Fashion MNIST we used geometrically distributed noise in the range 10.1,3). The weighting factor 1(a) is always set to 1/a2 to make the square term roughly independent of a. We fix the batch size at 128 and use the Adam optimizer with a learning rate of x 10 For MNIST and Fashion MNIST, we use a 12-Layer ResNet with 64 filters, for the CelebA and CIFAT- 10 data sets we used a 18-Layer ResNet with 128 filters (He et al. 2016a b). No normalization layer was used in any of the networks, We designed the output layer of all networks to take generalized quadratic form (Fan et al., 2018). Because the energy function is anticipated to be approximately quadratic with respect to the noise level, this modification was able to boost the performance significantly. For more detail on training
E [0.05, 1.2). with
5

---

Under review as a conference paper at ICLR 2020
Figure 2: Samples from our model trained on Fashion MNIST, CelebA and CIFAR-10. See Figure E.3 and Figure E.4 in Appendix for more samples and comparison with training data.
and model architecture, see Appendix D One notable result is that since our training method does not involve sampling, we achieved a speed up of roughly an order of magnitude compared to maximun-likelhhod training using Langevin dynamics Our method thus enables the training of energy-based models even when limited computational resources prohibit maximum likelihood methods.
We found that the choice of the maximum noise level has little effect on learning as long as it is large enough to encourage learning of the longest range features in the data. However, as expected, learning with too small or too large noise levels is not beneficial and can even destabilize the training process. Further, our method appeared be relatively insensitive how the noise levels dis- tributed over a chosen range. Geometrically spaced noise as in (Song and Ermon 2019) and linearly spaced noise both work, although in our case learning with linearly spaced noise somewhat more robust.
For sampling the learned energy function used annealed Langevin dynamics with an empiri- cally optimized annealing schedule,see Figure F.IB for the particular shape of annealing schedule we used. In contrast, annealing schedules with theoretical guaranteed convergence property takes extremely long (Geman and Geman 1984). The range of temperatures to use in the sampling pro- cess depends on choice of do, as the equilibrium distribution is roughly images with Gaussian noise magnitude VTog added on top. To ease traveling between modes far apart and ensure even sampling, the initial temperature needs to be high enough to inject noise of sufficient magnitude. A choice of T 100. which corresponds to added noise of magnitude /100 * 0.1 = 1, seems to be sufficient starting point. For step length e we generally used 0.02, although any value within the range [0.015, 0.05) seemed to work fine. After the annealing process we performed single step denoising to further enhance sample quality.
Table I: Unconditional Inception score, FID scores and Likelihoods for CIFAR-10
Model
IS
FID
Likelihood
NNL (bits/dim)
iResNet [Behrmann al. 2019
65.01
Yes
3.45
PixelCNN (van den Oord ét al. 2016)
4.60
65.93
Yes
3.14
PixelIQN (Ostrvvsk al. 2018)
5.29
49.46
Yes
Residual Flow (Chen et al.
2019
46.37
Yes
3.28
GLOW (Kingma and Dhariwar 2018)
46.90
Yes
3.35
EBM (ensemble) (Du and Mordatch
2019
6.78
38.2
Yes(density)
-M
SNGAN MMyyto et al. 2018)
8.22
21.7
No
MDSM (Ours)
8.31
31.7
Yes(density)
-0.96
NCSN (ong and Ermon 2019
8.91
25.32
No
For example, on a single GPU, training MNIST with a 12-layer Resnet takes 0.3s per batch with our method, while maximum likelihood training with modest 30 Langevin step per weight update takes 3s per batch. Both methods need similar number of weight updates train.

---

Under review as a conference paper at ICLR 2020
Unconditional Image Generation. We demonstrate the generative ability of our model by display- ing samples obtained by annealed Langevin sampling and single step denoising jump. We evaluated 50k sampled images after training on CIFAR-10 with two performance scores, Inception (Salimans et al. 2016) and FID (Heusel et al. 2017). We achieved Inception Score of 8.31 and FIID of 31.7, comparable to modern GAN approaches. Scores for CelebA dataset are not reported here they are not commonly reported and may depend on the specific pre-processing used. More samples and training images are provided in Appendix for visual inspection. We believe that visual assessment is still essential because of the possible issues with the Inception score (Barratt and Sharma 2018). Indeed, we also found that the visually impressive samples were not necessarily the one achieving the highest Inception Score.
Although overfitting is not a common concern for generative models, we still tested our model for overfitting, We found no indication for overfitting by comparing model samples with their nearest neighbors in the data set, see Figure C.1 in Appendix.
Mode Coverage. We repeated with our model the 3 channel MNIST mode coverage experiment similar to the one in Kumar et al. (2019). An energy-based model was trained on 3-channel data where each channel is random MNIST digit. Then 8000 samples were taken from the model and each channel was classified using a small MNIST classifier network. We obtained results of the 966 modes, comparable GAN approaches. Training energy to all the learned modes, but some modes were not accessed during sampling, likely due to the Langevin Dynamics failing to explore these modes. A better sampling technique such as HMC Neal et al. (011] or a Maximum Entropy Generator (Kumar et al. 2019) could improve this result.
successful and our model assigned low
Image Inpainting. Image impainting can be achieved with our model by clamping a part of the image to ground truth and performing the same annealed Langevin and Jump sampling procedure on the missing part of the image. Noise appropriate to the sampling temperature need to be added to the clamped inputs. The quality of inpainting results of our model trained on CelebA and CIFAR-10 -00 can be assessed in Figure 3 For CIFAR- 10 inpainting results we used the test set.
Figure 3: Demonstration of the sampling process (left), and image inpainting (right). The sampling process is shown with Gaussian noise (top left), and denoised by single step gradient jump (lower left). The column next to sampling process shows samples after the last denoising jump at the end of sampling. Inpainting results are shown next to initial image (left column) and the ground truth image (right column).
Log likelihood estimation. For energy- models, the log density can be obtained after estimat- ing the partition function with Annealed Importance Sampling (AIS) (Salakhutdinov and Murray) 2008) or Reverse AIS (Burda et al. 2015). In our experiment CIFAR-1O model, similar reports Du and Mordatch (2019), there is still a substantial gap between AIS and Reverse AIS estimation, even after very substantial computational effort. In Table D we report result from Reverse AIS, as it tends to over-estimate the partition function thus underestimate the density. Note that although
AAuthor reported difficulties evaluating Likelihood Upper Bound obtained by Reverse AIS
7

---

Under review as a conference paper at ICLR 2020
density values and likelihood values are not directly comparable, list them together due to the sheer lack of density model for CIFAR-10.
We also report a density of 1.21 bits/dim on MNIST dataset, and we refer readers to Du and Mor- datch (2019) for comparison to other models on this dataset. More details on this experiment is provided in the Appendix.
Outlier Detection. Choi and Jang 2018) and Nalisnick et al. 2019) have reported intriguing be- havior of high dimensional density models out of distribution samples. Specifically, they showed that a lot of models assign higher likelihood to out of distribution samples than real data samples. We investigated whether our model behaves similarly.
Our energy function is only trained outside the data manifold where samples are noisy, so the energy value at clean data points may not always be well behaved. Therefore, we added noise with magni- tude oo before measuring the energy value. We find that our network behaves similarly to previous likelihood models, it assigns lower energy, thus higher density, to some OOD samples. We show one example of this phenomenon in Appendix Figure F.IA
We also attempted to use the denoising performance, or the objective function to perform outlier detection. Intriguingly, the results are similar as using the energy value. Denoising performance seems to correlate more with the variance of the original image than the content of the image.
DISCUSSION
In this work we provided analyses and empirical results for understanding the limitations of learning the structure of high-dimensional data with denoising score matching. We found that the objective function confines learning to small set due to the measure concentration phenomenon in random vectors. Therefore, sampling the learned distribution outside the set where the gradient is learned does not produce good result. One remedy to learn meaningful gradients in the entire space is to use samples during learning that are corrupted by different amounts of noise. Indeed, Song and Ermon 2019) applied this strategy very successfully.
The central contribution of our paper is to investigate how to use similar learning strategy in EBMs. Specifically, we proposed a novel EBM model, the Multiscale Denoising Score Matching (MDSM) model. The new model is capable of denoising, producing high-quality samples from random noise, and performing image inpainting. While also providing density information, our model learns an order of magnitude faster than models based on maximum likelihood.
Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing Geras and Sutton 2015; Chandra and Sharma 2014 Zhang and Zhang 2018) though this idea was proposed in the context of pre-training neural networks for classification applications. Previous efforts of learning energy based models with score matching (Kingma and LeCun 2010 [ong et al., 2019) were either computationally intensive unable produce high-quality samples comparable to those obtained by other generative models such as GANs. [Saremi et al.] (2018) and (Saremi and Hyvarinen (2019) trained energy-based model with the denoising score matching objective but the resulting models cannot perform sample synthesis from random noise initialization.
Recently, SSng and Ermon 2019) proposed the NCSN model, capable of high-quality sample syn- thesis. This model approximates the score of family of distributions obtained by smoothing the data by kernels of different widths. The sampling in the NCSN model starts with sampling the dis- tribution obtained with the coarsest kernel and successively switches to distributions obtained with finer kernels. Unlike NCSN, our method learns an energy-based model corresponding to Pap (i) for a fixed ag- This method improves score matching in high-dimensional space by matching the gradient of an energy function to the score of Pro (i) in set that avoids measure concentration issue.
All told, we offer novel EBM model that achieves high-quality sample synthesis, which among other EBM approaches provides a new state-of-the art. Compared to the NCSN model, our model is more parsimonious than NCSN and can support single step denoising without prior knowledge of the noise magnitude. But our model performs sightly worse than the NCSN model, which could have several reasons. First, the derivation of Equation requires procedure tractable, which could reduce the performance. Second, the NCSNs output is a vector that, at least during optimization, does not always have to be the derivative of scalar function. In
approximation to keep the training
8

---

Under review as a conference paper at ICLR 2020
contrast, in model the network output is scalar function. Thus it is possible that the NCSN model performs better because it explores a larger set of functions during optimization.
REFERENCES
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint urXiv:1801.0197. 2018.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Herrik Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 573-582 2019. 2019.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv: 2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Accurate and conservative estimates mrf log-likelihood using reverse annealing. In Artificial Intelligence and Statistics, pages 102-110, 2015.
Chandra and Rajesh Kumar Sharma. Adaptive noise schedule for denoising autoencoder. In International conference on neural information processing, pages 535-542. Springer, 2014.
Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. arXiv preprint arXiv: 1906.02330,, 2019.
Hyunsun Choi and Eric Jang. Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392 12310.01399 392, 2018
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components
timation. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
Harris Drucker and Yann Le Cun. Double backpropagation increasing generalization performance. In IJCNN-91-38 -1--tlle International Joint Conference on Neural Networks, volume 2, pages 145- 150. IEEE, 1991.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 193,:99903,8689, 2019.
Fenglei Fan, Wenxiang Cong, and Ge Wang. A new type of neurons for machine learning. Interna- tional journal for numerical methods biomedical engineering, 34(2)=e2920, 2018
Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, (6): 721-741, 1984.
Krzysztof J. Geras and Charles A. Sutton. Scheduled denoising autoencoders. In 3rd International Conference on Leurning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pages 2672-2680, 2014.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 1352-1361. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In European conference computer vision, pages 630-645. Springer, 2016b.

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
Geoffrey E Hinton. Products of experts. 1999.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695 -0000 2005.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Yan Karklin and Eero Simoncelli. Efficient coding of natural images with population of noisy linear-nonlinear neurons. In Advances in neural information processing systems, pages 999-1007, 2011.
Diederik P. Kingma and Yann LeCun. Regularized estimation of image statistics by score matching. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of meeting held 6-9 December 2010, Van- couver, British Columbia, Canada., pages 1126-112 134, 2010.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
Scott Kirkpatrick, Daniel Gelatt, and Mario Vecchi. Optimization by simulated annealing. science, 220(4599):671-668, 1983.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.068 08558, 2019.
Neil Lawrence. Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of machine learning research, 1(N0v):1783-181) 2005,
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 100), 2006.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738, 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Radford M Neal. Annealed importance sampling. Statistics and computing. 1 11(2):125-133 2001.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo. 2(11):2, 2011.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 1105- 1112, 2011.
10

---

Under review as a conference paper at ICLR 2020
Erik Nijkamp, Mitch Hill, Tian Han, Song- Sang-Chu Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy- based models. arXiv preprint FrXiv:1903.22 2019.
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, pages 3933-3942, 2018.
Emanuel Parzen. On estimation of a probability density function and mode. The Annals of Mathe- matical Statistics, 33(3):1063-1076 1962.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2523-2255 2326, 2000.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. Malaysia; Pearson Education Limited,, 2016.
Ruslan Salakhutdinov and lain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference Machine learning, pages 872-879. ACM, 2008.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2244 2016.
Saeed Saremi and Aapo Hyvarinen. Neural empirical bayes. arXiv preprint 2019.
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv: 1805.0800, 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint Xrviv: (007.00900005000 2019.
Yang Song. Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty- Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 204, 2019.
Terence Tao. Topics random matrix theory, volume 132. American Mathematical Soc., 2012.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 29065500):211--2222 2000.
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference Machine learning, pages 1064- 1071. ACM, 2008.
Aäron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1747-1756, 2016.
Roman Vershynin. High-ddiennsonal probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- lation, 23(7):1661-1674 2011.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 11(Deo::331-3408 2010.
Martin Wainwright and Eero P Simoncelli. Scale mixtures of gaussians and the statistics natural images. In Advances in neural information processing systems, pages 855-861, 2000.

---

Under review as a conference paper at ICLR 2020
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681 688. 2011.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In Proceedings of the 33nd International Conference on Machine Learn- ing, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1100-1109, 2016.
Qianjun Zhang and Lei Zhang. Convolutional adaptive denoising autoencoders for hierarchical feature extraction. Frontiers of Computer Science, 12(6): 1440-1188, 2018.
12

---

Under review as a conference paper at ICLR 2020
A MDSM OBJECTIVE
In this section, we provide a formal discussion the MDSM objective and suggest as an improved score matching formulation in high-dimensional space.
Vincent 2011) illustrated the connection between the model score - Ev: with the score of
Parzen window density estimator V log(Pro (#)). Specifically, the objective is Equation | which we restate here:
LSM (0) = (()
Va log(Poo (i)) + V=b(E; I|²
(10)
Our key observation is: in high-dimensional space, due the concentration of measure, the expectation w.r.t Pon(#) over weighs a thin shell at roughly distance dd the empirical distribution p(x). Though in theory this is not problem, in practice this leads to results that the score are only well matched on this shell. Based on this observation, we suggest to replace the expectation w.r.t Poo(2) with a distribution Por (2) that has the same support as Pro(() but can avoid the measure concentration problem. We call this multiscale score matching and the objective the following:
L LSM (0) (p) |I Va log (+)) E=E(2;0) 1|
(11)
Proposition LAsSM (0) 0 = LsM 0
-
Given that pM (#) and Poo( has the same support, it's clear that MMSM 0 would be equivalent to LSM = 0. Due to the proof of the Theorem 2 Hyvärinen 20055. we have LsM(0) + 0".
Thus, LMSas (0) 0 8 d".
Proposition 2. (0) LMDSM- - (p
V log(go((()z)) VzE(E 11a
We follow the same procedure as in Vincent 2011) to prove this result.
(M) = Eps (+)
V log(Pro VzF(t)) I?
-Epatte (#)
V F( +
S(A)
E, V log(Pon
E=;
PM (
log(Pos Pro
( V ( 0)) dz
VEPon
PM
V ( 0)) di
Poo
PM (x)
ViPoo (T), V E (z; A)) dz
Já Poo
PM
V p(x) (ā 2 da.
V Ex; 0)) dā
Jz Pao
PM
2 V (a: 0)) dz
# Pao
PM
p()) (0 ( log 4.. dz V Ez; 0)) dā
J. Pro
PM
((x)
( ( (x))
log 4. x |x) V ((;; 0)) dēdx
8 m Pao
PM
Poo
+ x Log 4o. (1x) E ( 8)) dädæ
E J Da
PM
(T 4o
log go go.
(3)z), V (G; 0)) dā
13

---

Under review as a conference paper at ICLR 2020
Thus have:
(0) = (#) Vzb(2; _| +2S(0) +
par (e) (a)()
F(;;;) 0) 11 22 (a/a) (Va log loo (#|x), V+F(2:0) + C
(
V log(400(#/2)) V=E(2;0) +C
So LMSM (0) LMDSW--
The above analysis upplies any noise distribution, not limited Gaussian. but MDDM has reversed expectation form that is not easy to work with. To proceed further we study the case where (.n((aaa)) (())))) Gaussian and choose qmr (# (:) as Gaussian scale mixture (Wainwright and Simoncelli, we have the tollowing
2000) and Pxr
gas (#) (x dr. By Proposition 1 and Proposition
form to optimize:
LMDSM- (0)
PM 90. (x]²)
Va log(()o())x))) (zE(;;)) I| dēdæ
aro (20
PM @as (x)²)
V log(
V F(;; 1| dēdæ
QM
A7 (2)
PM (z,
Va l V || dädæ
R. QM (a+(²))
40o (2r
QM (
|| Va log(g=((3
+
V E (;; 0) I| dãtd
d QM (x](²)
LMDSM(()
the following importance sampling procedure (Russell and Norvig 2016): we can sample from the empirical distribution p(z), then sample the Gaussian scale mixture QM (#/z) and finally weight the sample by (ar())
To minimize Equation (*),
qu(((()) We expect the ratio be close to I for
the following reasons: Using Bayes rule, Gro (x| p(z))en(E we can see that go, (I|²) only has Pou
support on discret data points æ, same thing holds for qMr T). because in generated by adding Gaussian noise to real data sample, both estimators should give results highly concentrated on the original sample point æ. Therefore, in practice, we ignore the weighting factor and use Equation Improving upon this approximation is left for future work.
B PROBLEM WITH SINGLE NOISE DENOISING SCORE MATCHING
To compare with previous method, we trained energy-based model with denoising score matching using one noise level on MNIST, initialized the sampling with Gaussian noise of the same level, and sampled with Langevin dynamics at T 1 for 1000 steps and perform the model's best estimate of the clean sample, see Figure B. used the same 12-layer ResNet as other MNIST experiments. Models were trained for 100000 steps before sampling.
denoise jump to recover
C OVERFITTING TEST
We demonstrate that the model does not simply memorize training examples by comparing model samples with their nearest neighbors in the training set. We use Fashion MNIST for this demonstra- tion because overfitting can occur there easier than complicated datasets, Figure C.I
D DETAILS ON TRAINING AND SAMPLING
We used a custom designed ResNet architecture for all experiments. For MNIST and Fashion- MNIST we used 12-layer ResNet with 64 filters first layer, while for CelebA and CIFAR dataset
14

---

Under review as a conference paper at ICLR 2020
0.3
0.6
1.0
2
2
9
4
9
4
4
5
7
R
6
8
A
6
S
7
5
3
J
7
3
6
4
1
4
6
1
&
0
7
7
3
0
2
3
0
6
0
9
2
3
3
7
7
4
5
8
5
4
1
q
S
4
/
y
9
8
3
0
5
4
1
4
P
4
7
J
5
8
0
6 2
7
7
9
2 1
5
6
7
6
5
Figure B.I: Denoised samples from energy-based model trained with denoising score matching with single magnitude Gaussian noise on MNIST. Noise magnitude used in training is shown above samples.
Figure C.1: Samples from energy- bbase model trained on Fashion MNIST (Left column) next to 10 (L2) nearest neighbors in the training set.
used 18-layer ResNet with 128 filters the first layer. All network used the ELU activation function. We did not use any normalization in the ResBlocks and the filer number doubled each downsampling block. Details about the structure of our networks used can be found in our code release. All mentioned models can be trained on 2 GPUs within 2 days.
Since the gradient of our energy model scales linearly with the noise, expected our energy func- tion to scale quadratically with noise magnitude. Therefore, we modified the standard energy-based network output layer to take flexible quadratic form (Fan et al. 2018):
Eost Σ a,h; + b1) Σ ci4y + 62) + Σ d, + b3
(12)
where aj, G. d, and by, by, by are learnable parameters, and h; is the (flattened) output of last residual block. We found this modification to significantly improve performance compared to using a simple linear last layer.
For CIFAR and CelebA results we trained for 300k weight updates, saving a checkpoint every 5000 updates. We then took 1000 samples from each saved networks and used the network with the lowest
15

---

Under review as a conference paper at ICLR 2020
FID score. For MNIST and fashion MNIST we simply trained for 100k updates and used the last checkpoint. During training we pad MNIST and Fashion MNIST to 32*32 for convenience and randomly flipped CelebA images. No other modification was performed. We only constrained the gradient of the energy function, the energy value itself could in principle be unbounded. However, observed that they naturally stabilize
did not explicitly regularize them. The annealing sampling schedule is optimized to improve sample quality for CIFAR- 10 dataset, and consist of total of 2700 steps. For other datasets the shape has less effect on sample quality, see Figure F.1 for the shape of annealing schedule used.
For the Log likelihood estimation we initialized reverse chain on images, then sample 10000 intermediate distribution using 10 steps HMC updates each. Temperature schedule is roughly expo- nential shaped and the reference distribution an isotropic Gaussian. The variance of estimation was generally less than 10% on the log scale. Due to the high variance of results, and to avoid getting dominated by single outlier, we report average of the log density instead of log of average density.
E EXTENDED SAMPLES AND INPAINTING RESULTS
We provide more inpainting examples and further demonstrate the mixing during sampling process Figure E.1 We also provide more samples for readers visually judge the quality of our sample generation in Figure E.2 and E.4 All samples are randomly selected.
Figure E.1: Denoised Sampling process and inpainting results. Sampling process is from left to right.
16

---

Under review as a conference paper at ICLR 2020
7
8
4
8
0
4
3
4
7
2
4
3
4
3
4
9
1
8
7
8
2
9
2
9
5
8
2
4
4
6
5
7
3
9
0
7
9
9
9
4
8
8
6
0
5
1
5
S
3
3
8
4
4 2
9 7
0
2
4
6
3
0
5
2
7
4
9 5 2
4
0
9
9
4
2
0 6
2
3
6
9
3
8
4
5
3
8
5
3
9 3
0 6
2
3
7
5
0
5
3
7
5
3
5
3
8
9
a
2
/
1
7
5
0 6
4
8
9
(
6
>
8
Figure E.2: Extended Fashion MNIST and MNIST samples
F SAMPLING PROCESS AND ENERGY VALUE COMPARISONS
Here we show how the average energy of samples behaves vs the sampling temperature. We also show example of our model making of distribution that is common in other likeli- hood based models (Nalisnick et al. 2019) Figure F.I
17

---

Under review as a conference paper at ICLR 2020
Figure E.3: Samples (left panel) from network trained on CclebA, and training examples from the dataset (right panel).
18

---

Under review as a conference paper at ICLR 2020
Figure E.4: Samples (left panel) from energy- energy-based model trained on CIFAR-10 next to training examples (right panel).
A.
B.
Network trained on CIFAR-10
300000
CIFAR-10
340
CIPAB-10 Test
280000
SWHNN
200020
Tempreattue schélule
100
240000
9
200000
342000
3000 2500
2000
1500
1000
300
500
905
1000
1500
2000 7:00
Energy
12555
Figure F.1: A. Energy values for CIFAR-10 train, CIFAR-10 test and SVHN datasets for network trained on CIFAR-10 images. Note that the network does not over fit to the training set, but just like most deep likelihood model, it assigns lower energy to SVHN images than its training data. B. Annealing schedule and a typical energy trace for a sample during Annealed Langevin Sampling. The energy of the sample is proportional to the temperature, indicating sampling is close to quasi-static process.
19


Reviews:
[{"review_id": "rygT2g4otH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to learn an energy based generative model using an ‘annealed’ denoising score matching objective. The main contribution of the paper is to show that denoising score matching can be trained on a range of noise scales concurrently using a small modification to the loss. Compared to approximate likelihood learning of Energy based models the key benefit is to sidestep the need for sampling from the model distribution which has proven to be very challenging in practice. Using a slightly modified Langevin Sampler the paper further demonstrated encouraging sample qualities on CIFAR10 as measured by FID and IS scores. \n\nOverall I think the paper is well motivated and written, experiments are sound with encouraging results that will be useful for further progress in training energy based models. I currently score the paper as a ‘weak accept’, the reason for not giving ‘accept’ is that I think the paper is closely related to Song & Ermin 2019 (see detailed comments below) - However i can be convinced to bump my score depending on the author feedback \n\nQ1) I think you should elaborate more on how exactly your method is different from the NCSN model presented in Song & Ermon 2019? Especially.\nQ1.1) Is your method similar to the NCSN except that you do linear scaling with temperature in the loss and train a joint model across all temperature scales?\n\nQ1.2) In the related works section you claim that ‘[Song & Ermin] … this model learns p(xhat) for each T as a separate model’. Quickly reading through that paper i do not think that statement is accurate - I think they learn a model where the main difference is that it takes T as input instead of scaling the gradient term in the loss?\n\nQ1.3 )Do you have any intuition for why they seem to get slightly better results than the one you obtain in your paper? Is it simply architecture/training details that differ or something more ‘fundamental’?\n\n\nQ2) In relation to the Score matching objective.\nQ2.1) In eq (4) it is not completely clear to me what the motivation for linear scaling in T is. Can you elaborate on what you mean with ‘We borrow intuition from physics and simply set E_T(xhat) = E(xhat)/T ...’?\nIn relation to the above Can you clarify which part of your results holds for Gaussian noise and which holds in general. \n\nQ2.2) For the gaussian case I think linear scaling as done in eq(5) is sensible, however for arbitrary noise distributions linear scaling is akin to a first order approximation (which might be inaccurate across a range of different noise levels)?\nMinor: I think it would ease the reading of the paper if you showed the derivation (in appendix) that Eq (1) and Eq(2) are equivalent.\n\nMinor Comment: Learning generative models using denoising have also been explored in [Soenderby 2016]. Here the difficulties of different noise scales was also found and explored but (importantly) not solved.\n\n[Song & Ermon]. Generative Modeling by Estimating Gradients of the Data Distribution\n[Soenderby 2016]: Amortised map inference for image super-resolution\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkej7Q86KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "########Updated Review ###########\n\nI would like to thank the author(s) for their reply, which I have carefully read and it partly addresses my original concerns. Still, as agreed by all three reviewers, this paper might not be a significant step up compared with [1]. I am raising my point to weak reject to reflect my updated belief. I think this paper needs a bit more highlights to pass the threshold. \n\n###############################\n\n\nThis paper tries to address the problem of non-parametric maximal likelihood estimation via matching the score function wrt data. It is a clear rejection due to its significant overlap with the recent NeurIPS publication [1]. The author(s) have failed to clarify how their proposal differs from [1] in a significant way. From what I can tell after a quick read, both papers tried to training the score function using the denoising auto-encoder, amortized through a neural network, strategically annealed with a sequence of different noise levels, sampled with the Langevin scheme. I put two papers side-by-side and you can visually tell the uncanny resemblance.  Additionally, the proposed model does not outperform that from [1] (see Table 1). I am also not happy about the misleading statement in the abstract that this work \"assign likelihood to test data\", which is actually performed by AIS.  Section 2.2 is particularly problematic. The assumption of \"data approximately uniformly distributed on the manifold\" is outrageous, which basically invalidates the need for density estimation because of the uniformity. The 1/f power law characteristic is irrelevant to the likelihood estimation problem, and the statements are both heuristic & misleading. \n\n[1] Y Song, S Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkesvLyAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method of learning of energy based models using denoising score matching. This technique has been used before but only with limited success. The authors hypothesize that this is due to the fact that the matching was only performed over a single noise scale. The main idea of this work is to employ a range of scales to learn a single energy function. This trick helps to alleviate the problem of noisy samples concentrating in a low-volume region of the ambient space.\n\nIt seems that the paper draws significant inspiration from the work by Song & Ermon, 19. The difference between the two appears to be minor:\n1) The density is represented as a Boltzman distribution and therefore the score function is reduced to the gradient of the energy function (this has been done before)\n2) Instead of conditioning the energy on the noise level the authors propose to use explicit scaling by the inverse temperature\n \nPros:\n+ The paper is mostly well-written.\n+ I think Section 2 does a good job at illustrating challenges in training energy-based models using denoising score matching with a single noise scale. \n+ As compared to (Song & Ermon, 19) using the Boltzman distribution ensures that the learned score is an actual conservative vector field. Arguably, learning an image to scalar network is easier than learning an image to image one.\n+ Samples from the model are of competitive visual quality.\n\nCons:\n- Scaling energy by the inverse temperature seems to be one of the most important aspects of the paper but is only justified by “intuition from physics”. I’m not entirely sure that this is a valid assumption. In contrast, (Song & Ermon, 19) don’t put any hard constraints on the values of the score for different noise levels besides that they are produced by a single conditional network. I would appreciate if the authors discussed that difference in more detail.\n- The authors don’t provide any analysis as to whether the annealed Langevin MC procedure leads to the samples from the right distribution.\n- The quantitative results don’t seem to be better (actually, they are worse) than those from (Song & Ermon, 19).\n\nNotes/questions:\n* Abstract: “unmormalized” -> “unnormalized”\n* Section 2.1, (1): \\tilde{x} -> \\tilde{\\mathbf{x}}\n* Section 2.2, paragraph 2: What does superscript C mean in the noisy manifold? Never defined.\n* Section 2.2, paragraph 4: “some example” -> “some examples” (?)\n* Section 3, paragraph 1: “CIFAT-10” -> “CIFAR-10”\n* Section 4, paragraph 2: “for each T as a separate model”. I don’t think this is a correct statement. (Song & Ermon, 19) use a single conditional model for all the noise levels.\n* Section 4, paragraph 2: “does not rely on explicit receive noise magnitude” -> “does not rely on receiving noise magnitude explicitly” (?) I also don’t quite understand this entire sentence. Does the model really infer the noise magnitude from a given image? It seems like in Equation (7) there is an assumption that the temperature T is equal to 1. I don’t feel like there is a lot of difference between the proposed model and (Song & Ermon, 19) when it comes to supplying noise information. I’d appreciate if the authors could clarify that bit for me.\n\nMy main concern about this paper is that it doesn’t seem like a big step from its starting point (Song & Ermon, 19). The modifications are shown to work empirically but don’t result in a significantly better model. Moreover, I feel like the paper could do a better job at justifying those changes. I’m giving a borderline score but willing to increase it if the authors address my questions.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a variant of the Noise Conditional Score Network (NCSN) which does score matching using a single Gaussian scale mixture noise model. Unlike the NCSN, it learns a single energy-based model, and therefore can be compared directly to other models in terms of compression. I've read the paper, and the methods, exposition, and experiments all seem solid. Numerically, the score is slightly below the cutoff; reviewers generally think the paper is well-executed, but lacking in novelty and quality of results relative to Song & Ermon (2019). 

Decision: Reject