Title: LEARNING GENERATIVE MODELS USING DENOISING DENSITY ESTIMATORS

Abstract: Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density, is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. In addition, we show how to leverage DDEs to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ODE solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.

1. INTRODUCTION:

Learning generative probabilistic models from raw data is one of the fundamental problems in unsupervised machine learning. The defining property of such models is that they provide functionality to sample from the probability density represented by the input data. In other words, such models can generate new content, which has applications in image or video synthesis for example. In addition, generative probabilistic models may include capabilities to perform density estimation or inference of latent variables. Recently, the use of deep neural networks has led to significant advances in this area. For example, generative adversarial networks (Goodfellow et al., 2014) can be trained to sample very high dimensional densities, but they do not provide density estimation or inference. Inference in Boltzman machines (Salakhutdinov & Hinton, 2009) is tractable only under approximations (Welling & Teh, 2003) . Variational autoencoders (Kingma & Welling, 2014) provide functionality for both (approximate) inference and sampling. Finally, normalizing flows (Dinh et al., 2014 ) perform all three operations (sampling, density estimation, inference) efficiently.

In this paper we introduce a novel type of generative model based on what we call denoising density estimators (DDEs), which supports efficient sampling and density estimation. Our approach to construct a sampler is straightforward: assuming we have a density estimator that can be efficiently trained and evaluated, we learn a sampler by forcing its generated density to be the same as the input data density via minimizing their Kullback-Leibler (KL) divergence. A core component of this approach is the density estimator, which we derive from the theory of denoising autoencoders, hence our term denoising density estimator. Compared to normalizing flows, a key advantage of our theory is that it does not require any specific network architecture, except differentiability, and we do not need to solve ODEs like in continuous normalizing flows. In contrast to GANs, we do not require adversarial training. In summary, our contributions are as follows:

• A novel approach to obtain a generative model by explicitly estimating the energy (unnormalized density) of the generated and true data distributions and minimizing the statistical divergence of these densities. 

2. RELATED WORK:

Generative adversarial networks (Goodfellow et al., 2014) are currently the most widely studied type of generative probabilistic models for very high dimensional data such as images or videos. However, they are often difficult to train in practice, they can suffer from mode-collapse, and they only support sampling, but neither inference nor density estimation. Hence, there has been a renewed interest in alternative approaches to learn generative models. A common approach is to formulate these models as mappings between a latent space and the data domain, and one way to categorize these techniques is to consider the constraints on this mapping. For example, in normalizing flows (Dinh et al., 2014; Rezende & Mohamed, 2015) the mapping is invertible and differentiable, such that the data density can be estimated using the determinant of its Jacobian, and inference can be perfomed by applying the inverse mapping. Normalizing flows can be trained simply using maximum likelihood estimation (Dinh et al., 2017) . The challenge for these techniques is to design computational structures so that their inverses and Jacobians, including their determinants, can be computed efficiently (Huang et al., 2018; Kingma & Dhariwal, 2018) . Chen et al. ( 2018) and Grathwohl et al. (2018) derive continuous normalizing flows by parameterizing the dynamics (the time derivative) of an ordinary differential equation (ODE) using a neural network. They show that this implies that the time derivative of the log density can also be expressed as an ODE, which only involves the trace (not the determinant) of the Jacobian of the network. This makes it possible to use arbitrary network architectures to obtain normalizing flows, but it comes at the computation cost of solving ODEs to produce outputs.

In contrast, in variational techniques the relation between the latent variables and data is probabilistic, usually expressed as a Gaussian likelihood function. Hence computing the marginal likelihood requires integration over latent space. To make this tractable, it is common to bound the marginal likelihood using the evidence lower bound (Kingma & Welling, 2014) . As an advantage over normalizing flows, variational methods do not require an invertible mapping between latent and data space. However, Gaussian likelihood functions correspond to an L 2 reconstruction error, which arguably leads to blurriness artifacts. Recently, Li & Malik (2018) have shown that an approximate form of maximum likelihood estimation, which they call implicit maximum likelihood estimation, can also be performed without requiring invertible mappings. A disadvantage of their approach is that it requires nearest neighbor queries in (high dimensional) data space.

Not all generative models include a latent space, including autoregressive models (van den Oord et al., 2016) or denoising autoencoders (DAEs) (Alain & Bengio, 2014) . In particular, Alain & Bengio (2014) and Saremi & Hyvärinen (2019) use the well known relation between DAEs and the score of the corresponding data distributions (Vincent, 2011; Raphan & Simoncelli, 2011) to construct an approximate Markov Chain sampling procedure. Our approach also builds on DAEs, but we formulate an estimator for the un-normalized, scalar density, rather than for the score (a vector field). This is crucial to allow us to train a generator instead of requiring Markov chain sampling, which has the disadvantages of requiring sequential sampling and producing correlated samples.

In concurrent work, Song & Ermon (2019) are formulating a generative model using Langevin dynamics based on estimating gradients of the data distribution via score matching, which also requires an iterative sampling procedure and can sample the data density exactly only asymptotically. Table 1 summarises the similarities and differences of our approach to these techniques.

3. DENOISING DENSITY ESTIMATORS (DDES):

Here we show how to estimate a density using a variant of denoising autoencoders (DAEs). More precisely, our approach allows us to obtain the density smoothed by a Gaussian kernel, which is equivalent to kernel density estimation (Parzen, 1962) , up to a normalizing factor. Originally, the optimal DAE r : R n → R n (Vincent, 2011; Alain & Bengio, 2014) is defined as the function minimizing the following denoising loss,

EQUATION

where the data x is distributed according to a density p over R n , and η ∼ N (0, σ 2 η ) represents ndimensional, isotropic additive Gaussian noise with variance σ 2 η . It has been shown (Robbins, 1956; Raphan & Simoncelli, 2011) that the optimal DAE r * (x) minimizing L DAE can be expressed as follows, which is also known as Tweedie's formula,

EQUATION

where ∇ x is the gradient with respect to the input x, and p(s) = [p * k](x) denotes the convolution between the data and noise distributions p(x) and k = N (0, σ 2 η ), respectively. Inspired by this result, we reformulate the DAE-loss as a noise estimation loss,

EQUATION

where f : R n → R n is a vector field that estimates the noise vector -η/σ 2 η . The following proposition has also been shown by (Vincent, 2011) , and we also provide a proof in Appendix A: Proposition 1. There is a unique minimizer f * (x) = arg min f L NEs (f ; p, σ η ) that satisfies

f * (x) = ∇ x log p(x) = ∇ x log[p * k](x).

(4)

That is, the optimal estimator corresponds to the gradient of the logarithm of the Gaussian smoothed density p(x), that is, its score.

A key observation is that the desired vector field is the gradient of a scalar function and conservative.

Hence we can write the noise estimation loss in terms of a scalar function s : R n → R instead of the vector field f , which we call the denoising density estimation loss,

L DDE (s; p, σ η ) = E x∼p,η∼N (0,σ 2 η ) ∇ x s(x + η) + η/σ 2 η 2 .

(5)

A similar formulation has recently been proposed by Saremi & Hyvärinen (2019) . Our terminology is motivated by the following corollary: Corollary 1. The minimizer s * (x) = arg min s L DDE (s; p) satisfies

EQUATION

with some constant C ∈ R.

Proof. From Proposition 1 and the definition of L DDE (s; p) we know that ∇ x s * (x) = ∇ x log p(x), which leads immediately to the corollary.

In summary, we have shown how modifying the denoising autoencoder loss (Eq. 1) into a noise estimation loss based on the gradients of a scalar function (Eq. 5) allows us to derive a density estimator (Corollary 1), which we call the denoising density estimator (DDE).

In practice, we approximate the DDE using a neural network s(x; θ). Assuming that the network has enough capacity and is everywhere differentiable both with respect to x and its parameters θ, we can find the unique minimum of Eq. 5 using standard stochastic gradient descent techniques. For illustration, Figure 1 shows 2D distribution examples, which we approximate using a DDE implemented as a multi-layer perceptron. We only use Softplus activations in our network since it is differentiable everywhere.

4. LEARNING GENERATIVE MODELS USING DDES:

By leveraging DDEs, our key contribution is to formulate a novel training algorithm to obtain generators for given densities, which can be represented by a set of samples or as a continuous function.

In either case, we denote the smoothed data density p, which is obtained by training a DDE in case the input is given as a set of samples as described in Section 3. We express our samplers using mappings x = g(z), where x ∈ R n , z ∈ R m (usually n > m), and z is typically a latent variable with standard normal distribution. In contrast to normalizing flows, g(z) does not need to be invertible. Let us denote the distribution of x induced by the generator as q, that is q ∼ g(z), and also its Gaussian smoothed version q = q * k.

We obtain the generator by minimizing the KL divergence D KL (q||p) between the density induced by the generator q and the data density p. Our algorithm is based on the following observation: Proposition 2. Given a scalar function ∆ : R n → R that satisfies the following conditions:

D KL (q||p) = q, log q -log p > q + ∆, log q -log p , (7) ∆, 1 = 0, (8)

∆ 2 < , (pointwise exponentiation) (9)

then D KL (q||p) > D KL (q + ∆||p) for small enough .

Proof. We will use the first order approximation log(q + ∆) = log q + ∆/q + o(∆ 2 ), where the division is pointwise. Using •, • to denote the inner product, we can write

D KL (q + ∆||p) = q + ∆, log(q + ∆) -log p (10) = q + ∆, log q + ∆/q + o(∆ 2 ) -log p (11)

= q, log q -log p + ∆, log q -log p + q, ∆/q + ∆, ∆/q + o(∆ 2 ). ( 12)

This means

D KL (q + ∆||p) -D KL (q||p) = ∆, log q -log p + q, ∆/q + ∆, ∆/q + o(∆ 2 ) < 0 (13)

because the first term on the right hand side is negative (first assumption), the second term is zero (second assumption), and the third and fourth terms are quadratic in ∆ and can be ignored for ∆ < when is small enough.

Based on the above observation, Algorithm 1 minimizes D KL (q||p) by iteratively computing updated densities q + ∆ that satisfy the conditions from Proposition 2, hence D KL (q||p) > D KL (q + ∆||p). This iteration is guaranteed to converge to a global minimum, because D KL (q||p) is convex as a function of q.

At the beginning of each iteration in Algorithm 1, by definition q is the density obtained by sampling our generator x = g(z; φ), z ∼ N (0, 1) (n-dimensional standard normal distribution), and the generator is a neural network with parameters φ. In addition, q = q * k is defined as the density obtained by sampling x = g(z; φ) + η, z ∼ N (0, 1), η ∼ N (0, σ 2 η ). Finally, the DDE s q correctly estimates q, that is log q(x) = s q (x) + C.

In each iteration, we update the generator such that its density is changed by a small ∆ that satisfies the conditions from Proposition 2. We achieve this by computing a gradient descent step of E x=g(z;φ)+η s q (x) -log p(x) + C with respect to the generator parameters φ. The constant C can be ignored since we only need the gradient. A small enough learning rate guarantees that condition one in Proposition 2 is satisfied. The second condition is satisfied because we update the distribution by updating its generator, and the third condition is also satisfied under a small enough learning rate (and assuming the generator network is Lipschitz continuous). After updating the generator, we update the DDE to correctly estimate the new density produced by the updated generator.

Note that it is crucial in the first step in the iteration in Algorithm 1 that we sample using g(z; φ) + η and not g(z; φ). This allows us, in the second step, to use the updated g(z; φ) to train a DDE s q that exactly (up to a constant) matches the density generated by g(z; φ) + η. Even though in this approach we only minimize the KL divergence with the "noisy" input density p, the sampler g(z; φ) still converges to a sampler of the underlying density p in theory (Section 4.1).

4.1. EXACT SAMPLING:

Our objective involves reducing the KL divergence between the Gaussian smoothed generated density q and the data density p. This also implies that the density q obtained from sampling the generator g(z; φ) is identical with the data density p, without Gaussian smoothing, which can be expressed as the following corollary:

Algorithm 1: Training steps for the generator. input : Pre-trained optimal DDE on input data log p(x), learning rate δ initialize generator parameters φ initialize DDE s q = arg min s L DDE (s; q, σ η ) with q ∼ g(z; φ), z ∼ N (0, 1) while not converged do φ = φ + δ∇ φ E x=g(z;φ)+η s q (x) -log p(x) , with z ∼ N (0, 1), η ∼ N (0, σ 2 η ) // q ∼ g(z; φ) now indicates the updated density using the updated φ s q = arg min s L DDE (s; q, σ η ) // s q is now the density (up to a constant) of g(z; φ) + η Corollary 2. Let p and q be related to densities p and q, respectively, via convolutions using a Gaussian k, that is p = p * k, q = q * k. Then the smoothed densities p and q are the same if and only if the data density p and the generated density q are the same.

This follows immediately from the convolution theorem and the fact that the Fourier transform of Gaussian functions is non-zero everywhere, that is, Gaussian blur is invertible.

5. EXPERIMENTS:

Visual Comparisons using 2D Toy Datasets. Similar to prior work, we perform experiments for 2D density estimation and visualization over three datasets (Grathwohl et al., 2018) . Additionally, we use these datasets to learn generative models. For our DDE networks, we used multi-layer perceptrons with residual connections. All networks have 25 layers, each with 32 channels and Softplus activation. Trainings have 2048 samples per iteration. As shown in Figure 1 , the DDEs can estimate the density accurately and capture the underlying complexities of each density. Due to inherent KDE estimation, our method induces a small blur to the distribution to the density compared to BNAF. However, our DDE can estimate the density coherently through the data domain, whereas BNAF produces noisy approximation across the data manifold, where the estimated density is sometimes too small or too large. To demonstrate, we show DDEs trained with both small and large noise standard deviations σ η = 0.05 and σ η = 0.2.

Generator training and sampling is demonstrated in Figure 1 . The sharp edges of the checkerboard samples implies that the generator learns to sample from the target density although the DDEs estimate noisy densities. The generator update requires DDE networks to be optimal at each gradient step. For faster convergence, we take 10 DDE gradient descent steps for each generator update. In Figure 2 we illustrate the influence of the noise level σ η on the generated densities. This shows that in practice larger σ η do not lead to accurate sampling, since inverting the Gaussian blur becomes ill-posed. We summarize the training parameters used in these experiments in Appendix E.

MNIST. Figure 3 illustrates our generative training on the MNIST (LeCun, 1998) dataset using Algorithm 1. We use a dense block architecture with fully connected layers here and refer to Appendix B for the network and training details, including additional results for Fashion-MNIST (Xiao et al., 2017) . Figure 3 shows qualitatively that our generator is able to replicate the underlying distributions. In addition, latent-space interpolation demonstrates that the network learns an intuitive and interpretable mapping from noise to samples of the distribution.

CelebA. Figure 4 shows additional experiments on the CelebA dataset (Liu et al., 2015) . The images in the dataset have 32 × 32 × 3 dimensions and we normalize the pixel values to be in range [-0.5, 0.5]. To show the flexibility of our algorithm with respect to neural network architectures, here we use a style-based generator (Karras et al., 2019) architecture for our generator network. Please refer to Appendix C for network and training details. Figure 4 shows that our approach can produce natural-looking images, and the model has learned to replicate the global distribution with a diverse set of images and different characteristics.

Quantitative Evaluation with Stacked-MNIST. We perform a quantitative evaluation of our approach based on the synthetic Stacked-MNIST (Metz et al., 2016) dataset, which was designed to analyse mode-collapse in generative models. The dataset is constructed by stacking three randomly chosen digit images from MNIST to generate samples of size 28 × 28 × 3. This augments the number of classes to 10 3 , which are considered as distinct modes of the dataset. Mode-collapse can be quantified by counting the number of nodes generated by a model. Additionally, the quality of the distribution can be measured by computing the KL-divergence between the generated class distribution and the original dataset, which has a uniform distribution in terms of class labels. Similar to prior work (Metz et al., 2016) , we use an external classifier to measure the number of classes that each generator produces by separately inferring the class of each channel of the images.

Figure 5 reports the quantitative results for this experiment by comparing our method with welltuned GAN models. DCGAN (Radford et al., 2015) implements a basic GAN training strategy using a stable architecture. WGAN uses the Wasserstein distance (Arjovsky et al., 2017) , and WGAN+GP includes a gradient penalty to regularize the discriminator (Gulrajani et al., 2017) . For a fair comparison, all methods use the DCGAN network architecture. Since our method requires two DDE networks, we have used fewer parameters in the DDEs so that in total we preserve the same number of parameters and capacity as the other methods. For each method, we generate batches of 512 samples per training iteration and count the number of classes within each batch (that is, the maximum number of different labels in each batch is 512). We also plot the reverse KL-divergence to the uniform ground truth class distribution. Using the two measurements we can see how well each method replicates the distribution in terms of diversity and balance. Without fine-tuning and changing the capacity of our network models, our approach is comparable to modern GANs such as WGAN and WGAN+GP, which outperform DCGAN by a large margin in this experiment.

We also report results for sampling techniques based on score matching. We trained a Noise Conditional Score Network (NCSN) parametrized with a UNET architecture, which is then followed by a sampling algorithm using annealed Langevin dynamics (ALD) as described by Song and Ermon (Song & Ermon, 2019) . We refer to this method as UNET+ALD. We also implemented a model based on our approach called DDE+ALD, where we used our DDE network. While our training loss is identical to the score matching objective, the DDE network outputs a scalar and explicitly enforces the score to be a conservative vector field by computing it as the gradient of its scalar output. ALD+DDE uses the spatial gradient of the DDE for sampling with ALD (Song & Ermon, 2019) , instead of our proposed direct, one-step generator. We observe that DDE+ALD is more stable compared to the UNET+ALD baseline, even though the UNET achieves a lower loss during training. We believe that this is because DDEs guarantee conservativeness of the distribution gradients (i.e. scores), which leads to more diverse and stable data generation as we see in Figure 5 . Further, our approach with direct sampling outperforms both UNET+ALD and DDE+ALD.

Real Data Density Estimation. We follow the experiments in BNAF (De Cao et al., 2019) for density estimation on real measured data. This includes POWER, GAS, HEPMASS, and MINI-BOON datasets (Asuncion & Newman, 2007) . Since DDEs can estimate densities up to their normalizing constant, we approximate the normalizing constant using Monte Carlo estimation for these experiments. We show average log-likelihoods over test sets and compare to state-of-the-art methods for normalized density estimation in Table 2 . We have omitted the results of the BSDS300 dataset (Martin et al., 2001) , since we could not estimate the normalizing constant reliably (due to high dimensionality of the data).

To train our DDEs, we used Multi-Layer Perceptrons (MLP) with residual connections between each layer. All networks have 25 layers, with 64 channels and Softplus activations, except for GAS and HEPMASS, which employ 128 channels. We trained the models for 400 epochs using learning rate of 2.5e -4 with linear decay with scale of 2 every 100 epochs. Similarly, we started the training by using noise standard deviation σ η = 0.1 and decreased it linearly with the scale of 1.1 up to a dataset specific value, which we set to 5e -2 for POWER, 4e -2 for GAS, 2e -2 for HEPMASS, and 1.5e -1 for MINIBOON. We estimate the normalizing constant via importance sampling using (Asuncion & Newman, 2007) . The top row indicates input size and dimensionality for each dataset. Best performances are in bold.

a Gaussian distribution with the mean and variance of the DDE input distribution. We average 5 estimations using 51200 samples each (we used 10 times more samples for GAS), and we indicate the variance of this average in Table 2 .

5.1. DISCUSSION AND LIMITATIONS:

Our approach relies on a key hyperparameter σ η that determines the training noise for the DDE, which we currently set manually. In the future we will investigate thorough strategies to determine this parameter in a data-dependent manner. An other challenge is to obtain high-quality results using extremely high-dimensional data such as high-resolution images. In practice, one strategy is to combine our approach with latent embedding learning methods (Bojanowski et al., 2018) , in a similar fashion as proposed by Hoshen et al. (2019) . Finally, our framework uses three networks to learn a generator based on input samples (a DDE for the samples, the generator, and a DDE for the generator). Our generator training approach, however, is independent of the type of density estimator, and techniques other than DDEs could also be used in this step.

6. CONCLUSIONS:

In conclusion, we presented a novel approach to learn generative models using a novel density estimator, called the denoising density estimator (DDE). We developed simple training algorithms and our theoretical analysis proves their convergence to a unique optimum. Our technique is derived from a reformulation of denoising autoencoders, and does not require specific neural network architectures, ODE integration, nor adversarial training. We achieve state of the art results on a standard log-likelihood evaluation benchmark compared to recent techniques based on normalizing flows, continuous flows, and autoregressive models.

A PROOF OF SCORE MATCHING VIA NOISE ESTIMATION:

This is a proof for Proposition 1 in the main paper.

Proof. Clearly L NEs is convex in f hence the minimizer is unique. We can rewrite the noise estimation loss from Equation 3as

EQUATION

which we minimize with respect to the vector-valued function f : R n → R n . Substituting x = x+η yields

EQUATION

We can minimize this with respect to f (x) by differentiating and setting the derivative to zero, which leads to

EQUATION

and hence

EQUATION

EQUATION

which follows from basic calculus and has also been used by Raphan & Simoncelli (2011) .

B VISUAL RESULTS ON FASHION-MNIST:

For the experiments on MNIST and Fashion-MNIST, we used the Dense Block architecture (Huang et al., 2017) with 15 fully-connected layers and 256 additional neurons each. The last layer of the network maps all its inputs to one value, which we train to approximate the density of input images.

For the generator network, we used Dense Blocks with 15 fully connected layers and 256 additional neurons each. The last layer maps all outputs to the image size of 28 × 28 = 784. For the input of the generator, we used noise with a 16 dimensional standard normal distribution. In addition, the DDEs were trained with noise standard deviation σ η = 0.5, where pixel values were scaled to range between 0 and 1.

In addition to the MNIST results, here we include visual results on the Fashion-MNIST dataset, where we have used the exact setup as in our experiments on MNIST for training our generator.

Figure 6 shows our generated images and interpolations in the latent space of Fashion-MNIST.

C NETWORK AND TRAINING DETAILS FOR EXPERIMENTS ON CELEBA:

For our experiments on CelebA we use a style-based generator (Karras et al., 2019) architecture. We use Swish activations (Ramachandran et al., 2017) in all hidden layers of our networks except for their last layer, which we set to be linear. Additionally, we normalized each output of the generator to be in the accepted range [-0.5, 0.5]. We used equalized learning rate (Karras et al., 2018) with learning rate 5e -3 for the DDEs, and a slightly lower learning rate for the generator 3e -3. We trained our DDEs using σ η = 0.5 and set the truncation parameter in the style-based generator to φ = 0.7 when feeding the generator with random noise (Karras et al., 2019) at test time.

D NETWORK MODELS AND TRAINING FOR STACKED-MNIST EXPERIMENT:

In our experiments with Stacked-MNIST, our generative networks are trained using a learning rate of 2e -2, the Adam optimizer with β 1 = 0.9, and the generator updates took place after every 10th The NCSN models are trained to remove Gaussian noise at ten different noise standard deviations within the range [1.0, 0.01] (geometric interpolation). The input to the NCSN models include also the noise level. To further improve the quality of the networks, we use separate last-layers for each noise standard deviation for training and test. This way we can increase the capacity of the network significantly, while we keep the same order of parameters as in the other methods. We used the Adam optimizer with original parameters and a learning rate of 1e -4. 

E DETAILS FOR 2D DATASET TRAINING:



:



Figures and Tables:

Figure1: Density estimation on 2D toy data. We show that we can accurately capture these densities with few visual artifacts. We also show samples generated using our generative model training.

Figure 3: MNIST samples using our generator (a) and from the real dataset (b). Latent space interpolation using our generator (c). (a) Generated samples (b) Real samples

Figure 5: Mode-collapse experiment results on Stacked-MNIST as a function of training iterations (for discriminator or DDE). (a) Number of generated modes per batch of size 512. (b) Reverse KLdivergence between the generated and the data distribution in the logarithmic domain. All methods use the DCGAN architecture with the same capacity except for UNET+ALD. Model POWER d = 6, N ≈ 2M

Figure 6: Fashion-MNIST results using our generator training algorithm (a). Samples from the real dataset (b). Interpolated samples using our Generator (c).

Comparison of different deep generative approaches.

Table content: <table><tr><td>Property</td><td colspan="2">GAN Score Matching Normalizing Flows Ours</td></tr><tr><td>Provides density</td><td>-</td><td>-</td></tr><tr><td>Forward sampling model</td><td/><td>iterative</td></tr><tr><td>Exact sampling</td><td/><td>asymptotic</td></tr><tr><td>Free net architecture</td><td/><td>-</td></tr></table>

Average log-likelihood comparison in four datasets

Table content: <table/>

lists the hyper-parameters we used for different experiments on 2D datasets.

Table content: <table><tr><td>Experiment</td><td>σ η</td><td>Dataset</td><td>Learning rate</td><td>Iterations</td></tr><tr><td>Figure 1 (density estimation)</td><td>0.05, 0.2 0.2</td><td colspan="2">Checkerboard Two spirals Eight Gaussians 0.005 0.001</td><td>15000 15000</td></tr><tr><td/><td>0.05</td><td colspan="2">Eight Gaussians 0.0005</td><td>23750</td></tr><tr><td/><td>0.2</td><td>Checkerboard</td><td>0.001</td><td>200000</td></tr><tr><td>Figure 1 (generative), Figure 2</td><td>0.2</td><td>Two spirals</td><td>Decrease by half</td><td>250000</td></tr><tr><td/><td>0.1</td><td>Eight Gaussians</td><td>every 1000 epochs</td><td>250000</td></tr></table>

Training parameters for 2D datasets.

Table content: <table/>

Peer review (iclr2020):
This paper presents an approach for learning density estimates of a data distribution convolved with noise through “denoising density estimators” and shows how to leverage these density estimates to train generative samplers. The methods are evaluated on toy low-d datasets, MNIST, and fashion MNIST where they show reasonable density estimates and OK sample quality. My major concern with this paper is that the “denoising density estimator” proposed here is identical to denoising score matching (which is not cited or discussed). The second contribution of learning a sampler given a density estimate is interesting but likely suffers from all the instabilities of GAN training, and does not compare to related work on distilling energy-based models. Unless both these concerns are addressed, I cannot recommend this paper for acceptance. Major comments: * The idea of learning a generative model whose energy gradient (score) matches the gradient of the data distribution has been explored extensively under the name “score matching”. The proposed approach of “Deep Denoising Density Estimation” is *identical* to denoising score matching (Vincent et al., 2011, http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf). There’s no discussion of any prior work on score matching in this paper, or comparison with recent approaches in this space (e.g. sliced score matching https://arxiv.org/abs/1905.07088, and https://arxiv.org/abs/1907.05600 that uses denoising score matching). * The algorithm presented for learning a generator given an energy function is not compared to other implicit sampling approaches like MCMC. Additionally, the algorithm requires alternating density estimation with updating the generative model, which is quite similar to GAN-training alternating density ratio estimation with updating the generative model. Thus the proposed algorithm likely experiences similar instabilities and challenges in how to partially solve the density estimation step. There are no comparisons to any GAN-based approaches anywhere in the paper. That said, mixing DSMs and explicit generators is a neat area of research. * The experiments are too limited, and do not compute any quantitative metrics on the image datasets. Minor comments: * “Defining property of PGMs is that they provide functionality to sample” -> what about density estimates? Likelihood ratios? etc. * Boltzmann machines don’t allow for efficient inference * Intro could should spend more time discussing relation to energy-based models, score matching, noise-contrastive estimation * Eqn 2 only holds for sigma^2 -> 0, please add and discuss this limitation * Divergenc -> divergence, above eqn 12 * I found the math in section 4 very confusing. What’s the <> notation in this context? Do you need to introduce Delta? Alternating density (ratio) estimation and generative model updates is super common in GAN literature and is not discussed here. * Table 1: are the estimates of log-likelihood upper or lower bounds? Unlike other approaches, you don’t have exact likelihoods so I’m not sure your #s are comparable. * 2048 samples per iteration -> batch size? * “For faster convergence, we take 10 DDE gradient steps…” -> please add an experiment showing how results are impacted by # of gradient descent steps. Is larger # steps always better? * MNIST/Fashion MNIST samples aren’t that good to my eye and there are no quantitative metrics (e.g. KID, FID, IS)