Learning Generative Models using Denoising Density Estimators
Authors: Siavash Bigdeli
Geng Lin
Tiziano Portenier
Andrea Dunbar
Matthias Zwicker
Venue:  (2020)
Keywords: generative probabilistic models
denoising autoencoders
neural density estimation

Abstract: Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. In addition, we show how to leverage DDEs to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ODE solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.


Under review as a conference paper at ICLR 2020
LEARNING GENERATIVE MODELS USING DENOISING DENSITY ESTIMATORS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning generative probabilistic models that can estimate the continuous den- sity given a set of samples, and that can sample from that density, is one of the fundamental challenges in unsupervised machine learning. In this paper we in- troduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by neural network, that is efficiently trained to represent a kernel density estimator of the data. In addition, we show how to leverage DDEs to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not re- quire specific network architectures like in normalizing flows, ODE solvers as continuous normalizing flows, nor do require adversarial training as in gener- ative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.
INTRODUCTION
Learning generative probabilistic models from raw data is one of the fundamental problems in unsu- pervised machine learning. The defining property of such models is that they provide functionality to sample from the probability density represented by the input data. In other words, such mod- els can generate new content, which has applications in image or video synthesis for example. In addition, generative probabilistic models may include capabilities to perform density estimation or inference of latent variables. Recently, the use of deep neural networks has led significant ad- vances in this area. For example, generative adversarial networks (Goodffllow et al. [014) can be trained to sample very high dimensional densities, but they do not provide density estimation or inference. Inference in Boltzman machines (Salakhutdinov & Hinton 2009) is tractable only under approximations (Welling & Tch 2003). Variational autoencoders (Kingma & Welling 2014) pro-
vide functionality for both (approximate) inference and sampling. Finally, normalizing flows (Dinh) eet al. 2014) perform all three operations (sampling, density estimation, inference) efficiently.
In this paper we introduce novel type of generative model based what call denoising density estimators (DDEs), which supports efficient sampling and density estimation. Our approach to construct sampler is straightforward: assuming we have density estimator that be efficiently trained and evaluated, we learn a sampler by forcing its generated density to be the same as the input data density via minimizing their Kullback-eibler (KL) divergence. this approach is the density estimator, which we derive from the theory of denoising autoencoders, hence our term denoising density estimator. Compared to normalizing flows, a key advantage of our theory is that it does not require any specific network architecture, except differentiabilityy and we do not need to solve ODEs like in continuous normalizing flows. In contrast to GANs, we do not require adversarial training. In summary, our contributions are as follows:
core component of
A novel approach to obtain a generative model by explicitly estimating the energy (un- normalized density) of the generated and true data distributions and minimizing the statis- tical divergence these densities.
A density estimator based on denoising autoencoders called denoising density estimator (DDE), and its parameterization using neural networks, which we leverage to train our novel generative model.

---

Under review as a conference paper at ICLR 2020
Property
GAN
Score Matching
Normalizing Flows
Ours
Provides density
Forward sampling model
iterative asymptotic
Exact sampling
Free net architecture
Table 1: Comparison of different deep generative approaches.
RELATED WORK
Generative adversarial networks (Goodfellow 2014) are currently the most widely studied type of generative probabilistic models for very high dimensional data such as images or videos. How- ever, they are often difficult train in practice, they can suffer from mode-collapse, and they only support sampling, but neither inference nor density estimation. Hence, there has been renewed interest in alternative approaches to learn generative models. A common approach is to formulate these models as mappings between a latent space and the data domain, and one way to catego- rize these techniques is to consider the constraints on this mapping. For example, in normalizing flows (Dinh et al., 2014 Rezende & Mohamed 2015) the mapping is invertible and differentiable, such that the data density can be estimated using the determinant of its Jacobian, and inference can be perfomed by applying the inverse mapping. Normalizing flows can be trained simply using maximum likelihood estimation (Dinh et al., 2017). The challenge for these techniques is to de- sign computational structures so that their inverses and Jacobians, including their determinants, can be computed efficiently (Huang et al., 2018 Kingma & Dhariwal 2018). Chen et al. (2018) and Grathwohl et al. (2018) derive continuous normalizing flows by parameterizing the dynamics (the time derivative) of an ordinary differential equation (ODE) using a neural network. They show that this implies that the time derivative of the log density can also be expressed as an ODE, which only involves the trace (not the determinant) of the Jacobian of the network. This makes it possible to use arbitrary network architectures to obtain normalizing flows, but it comes at the computation cost of solving ODEs to produce outputs.
In contrast, in variational techniques the relation between the latent variables and data is probabilis- Gaussian likelihood function. Hence computing the marginal likelihood
tic, usually expressed requires integration over latent space. To make this tractable, it is common to bound the marginal likelihood using the evidence lower bound (Kingma & Welling 2014). As an advantage malizing flows, variational methods do not require an invertible mapping between latent and data space. However, Gaussian likelihood functions correspond guably leads to blurriness artifacts. Recently, Li & Malik (2018) have shown that an approximate form of maximum likelihood estimation, which they call implicit maximum likelihood estimation, can also be performed without requiring invertible mappings. A disadvantage of their approach is that it requires nearest neighbor queries (high dimensional) data space.
nor-
an L2 reconstruction error, which
Not all generative models include a latent space, including autoregressive models (van den Oord al., 2016) or denoising autoencoders (DAEs) (Alain & Bengio (014). In particular, Allain & Bengio (2014) and (Saremi & Hyvärinen (2019) use the well known relation between DAEs and the score of the corresponding data distributions (Vincent [01T]] Raphan & Simoncelli] 201T) to construct approximate Markov Chain sampling procedure. Our approach also builds DAEs, but we formulate an estimator for the un-normalized, scalar density, rather than for the score (a vector field). This is crucial allow us to train generator instead of requiring Markov chain sampling, which has the disadvantages of requiring sequential sampling and producing correlated samples. In concurrent work, [Song & Ermon (2019) are formulating a generative model using Langevin dynamics based on estimating gradients of the data distribution via score matching, which also requires an iterative sampling procedure and can sample the data density exactly only asymptotically. Table I summarises the similarities and differences of our approach to these techniques.
3 DENOISING DENSITY ESTIMATORS (DDES)
Here we show how to estimate a density using a variant of denoising autoencoders (DAEs). More precisely, our approach allows us to obtain the density smoothed by Gaussian kernel, which is
2

---

Under review as a conference paper at ICLR 2020
equivalent to kernel density estimation Parzen 1962). up normalizing factor. Originally, the
optimal DAE r : R" - IR" (Vincent, 2011: Alain & Bengio 2014) is defined as the function
minimizing the following denoising loss,
LDAE(F: p. o.)) = Ru-p.--(0000) [[](( + n) x1
where the data is distributed according to a density p over RD and (N, o represents n- dimensional, isotropic additive Gaussian noise with variance oi) It has been shown Robbins 1956 Raphan & Simoncelli 2011) that the optimal DAE r (z) minimizing LDAE can be expressed as follows, which is also known as Tweedic's ' formula,
r" ('()) x o V logū(z))
where V. is the gradient with respect to the input z, and p(s) (p (z) denotes the convolution between the data and noise distributions p(æ) and k (0, of), respectively. Inspired by this result, we reformulate the DAE-loss as a noise estimation loss
LNEs (J:p,o)) E==-p.--W((..ot)) [l][ + 1) + ]
where f RR" R is vector field that estimates the noise vector - -n/o The following proposi- tion has also been shown by (Vincent [201]), and we also provide proof in Appendix A
Proposition 1. There is a unique minimizer f (ax) = arg min, LNES (f: o,)) that satisfies
((x) = Va p(r) = V. loglp * (x))
(4)
That is, the optimal estimator corresponds to the gradient of the logarithm of the Gaussian smoothed density p(x), that is, its score.
key observation is that the desired vector field the gradient a scalar function and conservative. Hence we can write the noise estimation loss in terms of scalar function R R instead of the vector field f, which we call the denoising density estimation loss,
CDDE(s; p, on) Ex-p..-N(0..r) [| + 17) n
(5)
A similar formulation has recently been proposed by (aremi & Hyvärinen (0119). Our terminology is motivated by the following corollary:
Corollary 1. The minimizer s" (x) arg min, LDDE p) satisfies
(x)) logā(x)) + C,
(6)
with some constant C R.
Proof. From Proposition 1 [add the definition of (DDR(S;p) we know that (z)*(x) V log ((x)
which leads immediately to the corollary.
In summary, we have shown how modifying the denoising autoencoder loss (Eq. T into a noise estimation loss based the gradients of a scalar function (Eq, (5) allows us to derive a density estimator (Corollary D]. which we call the denoising density estimator (DDE).
In practice, we approximate the DDE using neural network (ar; Assuming that the network has enough capacity and is everywhere differentiable both with respect to x and its parameters 0, find the unique minimum of Eq. using standard stochastic gradient descent techniques. For illustration, Figure [ implemented multi-layer perceptron. We only use Softplus activations in our network since it is differentiable everywhere.
shows 2D distribution examples, which we approximate using a DDE
4 LEARNING GENERATIVE MODELS USING DDES
By leveraging DDEs, our key contribution is to formulate a novel training algorithm to obtain gen- erators for given densities, which can be represented by set of samples In either case, we denote the smoothed data density p. which is obtained by training DDE in case the input is given as set of samples described in Section We express samplers using
continuous function.
3

---

Under review as a conference paper at ICLR 2020
mappings g(2), where E R R" (usually > m). and is typically latent variable with standard normal distribution. contrast to normalizing flows, g(z) does not need to be invert- ible. Let us denote the distribution of induced by the generator q, that is q g(2), and also its Gaussian smoothed version q k.
We obtain the generator by minimizing the KL divergence (Kl(?) between the density induced by the generator ④ and the data density p. Our algorithm is based on the following observation: Proposition 2. Given scalar function A TR." - R that satisfies the following conditions:
DKL p)) (ū, log q log f)) (ă log q log p)
(A, 1) 0,
(8)
A < (pointwise exponentiation)
(9)
then (Kl(Āl >> (K + l for small enough c.
Proof. We will use the first order approximation log(ğ + A) log q A/Q o(A²), where the
division is pointwise. Using (+-) > to denote the inner product, we can write
lk + Alp)
(ÿ + A, log(Ā + A) log p)
(10)
(ē A, log q A o( log
(11)
(g. log q l) + |o, log + (// + + + + (12)
This means
(K + Alpp)
DKL.
D ()
(A, log log + (q. A/9 + (A, / + o
(13)
because the first term on the right hand side is negative (first assumption), the second term is zero (second assumption), and the third and fourth terms are quadratic in A and can be ignored for A <
when 6 small enough.
Based on the above observation, Algorithm minimizes DKL (gl [)) by iteratively computing up- dated densities y + A that satisfy the conditions from Proposition 2 hence DRL. (g p) > DKI. +
Allp). This iteration is guaranteed converge to global minimum, because DRL. ( (p) is convex
as function of q.
At the beginning of each iteration in Algorithm I by definition q the density obtained by sampling our generator x = g(2; Ø), (N( 1) (n-dimensional standard normal distribution), and the generator is neural network with parameters c. In addition, q q q k is defined as the density obtained by sampling z g(2; Φ) + n, 1). ~ NO0 o,)) Finally, the DDE sa correctly estimates q, that is ((z) < +C
In each iteration, we update the generator such that its density is changed by a small A that sat- isfies the conditions from Proposition 2 We achieve this by computing gradient descent step of Ez=g((zd))nn (=) log p(x)]= + +C with respect to the generator parameters Φ. The constant C' can be ignored since we only need the gradient. A small enough learning rate guarantees that condition one in Proposition is satisfied. The second condition is satisfied because we update the distribution by updating its generator, and the third condition is also satisfied under a small enough learning rate (and assuming the generator network is Lipschitz continuous). After updating the generator, we update the DDE to correctly estimate the new density produced by the updated generator.
Note that it is crucial in the first step in the iteration in Algorithm Ihht we sample using ( の) + and not ((; : Φ). This allows us, in the second step, to use the updated g(2; Φ) to train a DDE s that exactly (up to a constant) matches the density generated by g(2; Φ) + 17. Even though in this approach we only minimize the KI. divergence with the "noisy" input density p. the sampler g(2; の) still converges to sampler of the underlying density p in theory (Section 4.1
4.1 EXACT SAMPLING
Our objective involves reducing the KL divergence between the Gaussian smoothed generated den- sity ğ and the data density p. This also implies that the density obtained from sampling the generator g(2; Φ) is identical with the data density p. without Gaussian smoothing, which can be expressed as the following corollary:
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1:
Training steps for the generator.
input : Pre-trained optimal DDE on input data log p(x), learning rate
initialize generator parameters é
initialize DDE ~ = arg min, LDDE($; q, Gn) with [(; の), 10, while not converged do
Φ 8V
(xx) log x(x)) with (O, 1,~,)~,1,00)) N/(,
// ~ g(2; o) now indicates the updated density using the updated
84 arg min, LDDE(S; q,),)
I/ s" is now the density (up to a constant) of 6)
Let p and q be related to densities and respectively, via convolutions using Gaussian k, that is p = pp k. q Then the smoothed densities and q are the same if and only if the data density and the generated density are the same.
Corollary 2.
This follows immediately from the convolution theorem and the fact that the Fourier transform of Gaussian functions is non-zero everywhere, that is, Gaussian blur is invertible.
5 EXPERIMENTS
Visual Comparisons using 2D Toy Datasets. Similar to prior work, we perform experiments for 2D density estimation and visualization over three datasets (Grathwohl et al., 2018). Additionally, we use these datasets to learn generative models. For our DDE networks, we used multi-layer per- ceptrons with residual connections. All networks have 25 layers, each with 32 channels and Softplus activation. Trainings have 2048 samples per iteration. As shown in Figure the DDEs can estimate the density accurately and capture the underlying complexities of each density. Due to inherent KDE estimation, our method induces small blur to the distribution the density compared to BNAF. However, our DDE can estimate the density coherently through the data domain, whereas BNAF produces noisy approximation across the data manifold, where the estimated density is sometimes too small or too large. To demonstrate, we show DDEs trained with both small and large noise standard deviations on 0.05 and o 0.2.
Generator training and sampling is demonstrated in Figure I The sharp edges of the checkerboard samples implies that the generator learns to sample from the target density although the DDEs estimate noisy densities. The generator update requires DDE networks to be optimal at each gradient step. For faster convergence, we take 10 DDE gradient descent steps for each generator update. In Figure 2 we illustrate the influence of the noise level o, on the generated densities. This shows that in practice larger og do not lead to accurate sampling, since inverting the Gaussian blur becomes ill-posed. We summarize the training parameters used in these experiments in Appendix
MNIST. Figure 3 illustrates our generative training on the MNIST (LeCun 1998) dataset using Algorithm [ We use a dense block architecture with fully connected layers here and refer to Ap- pendix B for the network and training details, including additional results for Fashion-MNIST (Xiao) et al. 2017). F shows qualitatively that generator is able to replicate the underlying dis- tributions. In addition, latent-space interpolation demonstrates that the network learns an intuitive and interpretable mapping from noise to samples of the distribution.
CelebA. Figure 4 shows additional experiments on the CelebA dataset Liu et al. 2015) The images the dataset have x x dimensions and we normalize the pixel values to be in range [-0.5, 0.5). To show the flexibility of our algorithm with respect to neural network architectures, here we use a style-hased generator (Karras et al.. 2019) architecture for our generator network. Please refer to Appendix C for network and training details. Figure 4 shows that our approach can produce natural-looking images, and the model has learned to replicate the global distribution with diverse set of images and different characteristics.
Quantitative Evaluation with Stacked-MNISST. We perform a quantitative evaluation of our ap- proach based on the synthetic Stacked-MNIST (Metz al. 2016) dataset, which was designed to
5

---

Under review as a conference paper at ICLR 2020
Real samples
Glow
BNAF
Ours (a,, 0.05)
Ours (o,, 0.2)
Ours generated
Eight Gaussians
Two Spirals
Checkerboard
Figure 1: Density estimation on 2D toy data. We show that we can accurately capture these densities with few visual artifacts. We also show samples generated using our generative model training.
GT
+T+No
Gen.
Gen.+Noisc
GT
Go+so
Gen.
Gen.+Noisc
Bg = 0.2
Tg 0.5
Figure 2:
The influence of on sample generation. We observe that the smoothed sampled density close to the training density. However, for large on the sampled density without smoothing can be quite different from the true density because inversion of Gaussian smoothing becomes ill-posed.
analyse mode-collapse in generative models. The dataset is constructed by stacking three randomly chosen digit images from MNIST to generate samples of size 28 x 28 x 3. This augments the num- ber of classes to 103, which are considered as distinct modes of the dataset. Mode-collapse can be quantified by counting the number of nodes generated by model. Additionally, the quality of the distribution can be measured by computing the KL-divergence between the generated class distri- bution and the original dataset, which has uniform distribution in terms of class labels. Similar to prior work (Metz et al., 2016), we use an external classifier to measure the number of classes that each generator produces by separately inferring the class of each channel of the images.
Figure 55 reports the quantitative results for this experiment by comparing our method with well- tuned GAN models. DCGAN (Radford al. 2015) implements basic GAN training strategy using a stable architecture. WGAN uses the Wasserstein distance (Arjovsky et al.) (017). and WGAN+GP includes a gradient penalty regularize the discriminator (Gulrajani et al. 2017). For fair com- parison, all methods use the DCGAN network architecture. Since our method requires two DDE networks, we have used fewer parameters in the DDEs so that total we preserve the same number of parameters and capacity as the other methods. For each method, we generate batches of 512 sam- ples per training iteration and count the number of classes within each batch (that is, the maximum number of different labels in each batch is 512). We also plot the reverse KL-divergence to the uni- form ground truth class distribution. Using the two measurements we can see how well each method replicates the distribution in terms of diversity and balance. Without fine-tuning and changing the capacity of our network models, our approach is comparable to modern GANs such as WGAN and WGAN+GP, which outperform DCGAN by large margin in this experiment.
We also report results for sampling techniques based on score matching. We trained a Noise Con- ditional Score Network (NCSN) parametrized with UNET architecture, which is then followed

---

Under review as a conference paper at ICLR 2020
(a) Generated samples
(b) Real samples
3
8
4
0
6
9
2
6
9
0
3
7
4
/
2
4
2
7
S
9
4
3
7
9
3
4
4
9
(c) Interpolated samples using our model
/
/
/
/
/
/
/
7
7
7
7
7
7
7
3
3
3
3
3
3
3
5
6
6
6
Figure 3: MNIST samples using our generator (a) and from the real dataset (b). Latent space interpolation using our generator (c).
(a) Generated samples
(b) Real samples
Figure 4: Results of our generator training algorithm on x2 images from the celebA dataset (Liu al. 2015)
by sampling algorithm using annealed Langevin dynamics (ALD) described by Song and Er- mon (Song & Ermon 2019). We refer to this method as UNET+ALD. We also implemented a model based on our approach called DDE+ALD, where we used our DDE network. While our training loss is identical to the score matching objective, the DDE network outputs a scalar and explicitly enforces the score to be a conservative vector field by computing it as the gradient of its scalar output. ALD+DDE uses the spatial gradient of the DDE for sampling with ALD (Song & Ermon 2019), instead of our proposed direct, one-step generator. We observe that DDE+ALD more stable compared to the UNET+ALD baseline, even though the UNET achieves lower loss during training. We believe that this because DDEs guarantee conservativeness of the distribution gradients (i.e. scores), which leads to more diverse and stable data generation as see in Figure Further, our approach with direct sampling outperforms both UNET+ALD and DDE+ALD.
Real Data Density Estimation. We follow the experiments in BNAF (De Cao et al. 2019) for density estimation on real measured data. This includes POWER, GAS, HEPMASS, and MINI- BOON datasets (Asuncion & Newman 2007).. Since DDEs can estimate densities up to their nor- malizing constant, approximate the normalizing constant using Monte Carlo estimation for these experiments. We show average log- ligelikeissoods over test sets and compare to state-of of-the-art meth- ods for normalized density estimation in Table 2 We have omitted the results of the BSDS300 dataset (Martin et al. 2001), since we could not estimate the normalizing constant reliably (due to high dimensionality of the data).
To train our DDEs, we used Multi-Layer Perceptrons (MLP) with residual connections between each layer. All networks have 25 layers, with 64 channels and Softplus activations, except for GAS and HEPMASS, which employ 128 channels. We trained the models for 400 epochs using learning rate of 2.5e - 4 with linear decay with scale of every 100 epochs. Similarly, we started the training by using noise standard deviation on 0.1 and decreased it linearly with the scale of 1.1 up to dataset specific value, which we set to 5e 2 for POWER, 4e - for GAS, 2e - for HEPMASS, and 1.5e 1 for MINIBOON. We estimate the normalizing constant via importance sampling using
7

---

Under review as a conference paper at ICLR 2020
(a) Generated modes per batch
(b) KL-divergence
400
WGAN
of modes
200
1.5-
WGAN+GP
UNET+ALD
200
1.0
Ours
0.5
300 0.0
noortions
1000 iterations
Figure 5: cole-col experiment results on Stacked- MINIST as a function training iterations (for discriminator DDE). (a) Number of generated modes per batch of size 512. (b) Reverse KL- divergence between the generated and the data distribution in the logarithmic domain. All methods use the DCGAN architecture with the same capacity except for UNET+ALD.
Model
POWER N 2M
GAS d 8. = 1M
HEPMASS
MINIBOON
d 21, s 500K
d 43, N = 36K
RealNVP
0.17 4.01
8.33 1.14
-18.71 4.02
-3.55 2.49
Glow
0.17 4.01
8.15 1.40
-18.22 1.08
-11.35 1.07
MADE MoG
0.40 #.01
8.47 +.02
-15.15 #.02
-12.27 #.47
MAF-affine
0.24 +.01
10.08 0.02
-17.73 #.02
-12.24 #:45
MAF-affine MoG
0.30 +.01
9.59 4.02
-17.39 #.02
-11.68 #444
FFJORD
0.46 1.01
8.59 1.12
-14.92 2.08
-10.43 2:04
NAF-DDSF
0.62 2.01
11.96 2.33
-15.09 2.40
8.86 1.15
TAN
0,60 +.01
12.06 4.02
-13.78
4.02
11.01 4..48
BNAF
0.61 4.01
12.06
14.71
8,95
Ours
0.97 2.18
9.73 111.1
-11.3 2.16
-6.94 11.81
Table Average log-likelihood comparison in four datasets (Asuncion & Newman, 2007). The top row indicates input size and dimensionality for each dataset. Best performances in bold.
a Gaussian distribution with the mean and variance the DDE input distribution. We average estimations using 51200 samples each (we used 10 times more samples for GAS), and we indicate the variance of this average T
5.1 DISCUSSION AND LIMITATIONS
Our approach relies on key hyperparameter o, that determines the training noise for the DDE, which we currently set manually. In the future will investigate thorough strategies determine this parameter a data-dependent manner. An other challenge is to obtain high-quality results using extremely high-dimensional data such as high-resolution images. In practice, strategy is to combine our approach with latent embedding learning methods (Bojanowski et al., 2018). similar fashion proposed by Hoshen et al.) (019)) Finally, framework uses three networks to learn generator based on input samples (a DDE for the samples, the generator, and a DDE for the generator). Our generator training approach, however, is independent of the type of density estimator, and techniques other than DDEs could also be used in this step.
CONCLUSIONS
In conclusion, we presented a novel approach to learn generative models using novel density esti- mator, called the denoising density estimator (DDE). We developed simple training algorithms and theoretical analysis proves their convergence to a unique optimum. Our technique is derived from a reformulation denoising autoencoders, and does not require specific neural network archi- tectures. ODE integration. adversarial training. We achieve state of the art results standard oog-likelihood evaluation benchmark compared to recent techniques based on normalizing flows, continuous flows, and autoregressive models.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating
distribution. Journal of Machine Learning Research, 15:3743-3773, 2014. URL http://
jm.r.org/ wwt.://ggppppps/l..... aaaaaaaannnl. 4atmml
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan.
arXiv preprint
arXiv:1701.07875. 2017.
Arthur Asuncion and David Newman. UCI machine learning repository, 2007. URL https:
Marchiiveics uci
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 600-609, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedinggs ml press /v00 bo janowsk 8a html
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6571-6583. Curran Associates, Inc., 2018. URL
nips. cc/paper/
paf
Nicola De Cao, Ivan Titov, and Wilker Aziz. Block neural autoregressive flow. arXiv preprint 1rXiv:1904.04676 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components
estimation. 2014. Utt.ht.tp://axxlw.orrg/ abs /110 851 6, 2014
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proc. ICLR, 2017. URL/attps://azxtt www.axiii....rra..o abs 08800.88003033
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL http://ge //tpprrspppr..... nips.cc/ wts.cc/paper//./ 5423 -aee--ggeeratt..e ive- adversaris -natsstts.po
Will Grathwohl, Ricky T. Q Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. CoRR, abs/1810.01367, 2018.
www./iiii.../i.aooooooooo abs///10s 01367
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances neural information processing systems, pp- 5777, 2017.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregres- sive flows. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2087, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL hhtt: //proceedings mlr www.r.prresa.. ppwass////ee...... v hwwngl8d.nn.cc html
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4703 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. htttstttss://ppt
ew net httpee/eeeeeeooommmmmm/03a3.100.

---

Under review as a conference paper at ICLR 2020
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In Proc. ICLR, 2014. URL https:///axivv: wwg.abs////oooooooooo..ooo...srs 1312. 6114v10
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl con- volutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi. and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215- 10224. Curran Associates, Inc., 2018. URL http://papers:
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Ke and Jitendra Malik. Implicit maximum likelihood estimation. CoRR, abs/1809/ 809.09087, 2018.
URL http:://axtiiii org/ http://ass1119..0090
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik, et database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Icev Vancouver. 2001.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickes bhi--Dickttei. Unrolled generative adversarial networks. arXiv preprint arXiv: /6/ 1,02/63,
Emanuel" Parzen. On estimation of probability density function and mode. Amn. Math. Statist., 33 (3) Sept 1962. doi: 10.77447. URL httpps://oo..ogg/////
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint 1rXiv:1511.06434, 2015.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 2017.
Martin Raphan and Fero P. Simoncelli. Least squares estimation without priors or supervision.
Neural Comput., 23(2):374 February 2011. ISSN 0899-7667. doi: 10.1162/HECO..000070 URL / dww.
htt.pp///.........///0.0........ 1tt.//10..112/EECC0 a 00076
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL httt:://sroceedinggt mlr www.sm.m....pr......mm. v02res///37/rezedde15 html
Herbert Robbins. An empirical bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, 157-163, Berkeley, Calif,, 1956. University of California Press. URL https:
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann machines. Artificial intelligence and statistics, pp. 448 448.455,11 2009.
Saeed Saremi and Aapo Hyvärinen. Neural empirical bayes. ArXiv, abs/1903.02334, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimattinggraaiiinn of the data distribution. CoRR, abs/1907 007.05600, 2019. http:////w////////ooooooo///////
10

---

Under review as a conference paper at ICLR 2020
Aaron den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with PixeICNN decoders. D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Sys-
Tems 29, pp. 4790-4798. Curran Associates, Inc., 2016. URL htttt:::/ppaarr.nnng..... cc/
tional >
pdf
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput.,
23(7):66111677, July 2011. ISSN 0899-7667.
doi: 10.1162/NECO a 00142. URL http:
7/dx.doi. org / 0. 1162 61r66//C_a__0000 0_002
Max Welling and Yee Whye Teh. Approximate inference in boltzmann machines. Ar-
tificial Intelligence. 143(1):19 50, 2003. ISSN 0004-3702.
doi: https//doi.org//10 10.
1016300000-3772202000333--222 URL http://www. sci encedi wwm/cc/c//...oo com/science/
article/At//5004472220331
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST. novel image dataset for bench-
marking machine learning algorithms. arXiv preprint arXiv:1708.07747; 2017.

---

Under review as a conference paper at ICLR 2020
A PROOF OF SCORE MATCHING VIA NOISE ESTIMATION
This is proof for Proposition I in the main paper.
Proof. Clearly bNT& convex in f hence the minimizer unique. We can rewrite the noise esti- mation loss from Equation as
[NEs(f:P.o) = / En- [p(x)]1f(> + + n/ n/ dx,
(14)
which we minimize with respect to the vector-valued function f 12" RR. Substituting x++11) yields
(NEs(fip,o.) = Ep--((..3)) [p(i - -))ff())l(+===)))))) + n/o,,| dz.
(15)
We can minimize this with respect to f(T) by differentiating and setting the derivative to zero, which leads to
Bx-~((0. lp(ē -))f(()))) = - Bn- pp(ē - -n)m),
(16)
and hence
f(i) - 1 B |p n))n
(17)
of B [p(E -n)
V loglp (+) V logā(ã).
(18)
which follows from basic caleulus and has also been used by Raphan & Simoncelli (001)
B VISUAL RESULTS ON FASHION-MNIST
For the experiments MNIST and Bushion-MNIST used the Dense Block architecture (Huang et al., (017)) with 15 fully-connected layers and 256 additional neurons each. The last layer of the network maps all its inputs to value, which train to approximate the density of input images. For the generator network, we used Dense Blocks with 15 fully connected layers and 256 additional neurons each. The last layer maps all outputs the image size of 28 28 784. For the input of the generator, we used noise with 16 dimensional standard normal distribution. In addition, the DDEs were trained with noise standard deviation o, 0.5, where pixel values were scaled to range between 0 and 1.
In addition to the MNIST results, here we include visual results on the Fashion-MNIST MNIST dataset, where we have used the exact setup in our experiments on MNIST for training our generator. Figure 6 shows our generated images and interpolations the latent space Fashion-MNISTT.
C NETWORK AND TRAINING DETAILS FOR EXPERIMENTS ON CELEBA
For our experiments on CelebA we use style-based generator (Krrras al. 2019 architecture. We use Swish activations (Ramachandran et al. 2017) in all hidden layers of our networks except for their last layer, which we set be linear. Additionally, we normalized each output of the generator to be in the accepted range [-0.5, 0.5). We used equalized learning rate (Karras et al. 2018) with learning rate 5c 3 for the DDEs, and a slightly lower learning rate for the generator 3c We trained DDEs using o, 0.5 and set the truncation parameter in the bbyeed generator to & 0.7 when feeding the generator with random noise (Karras et al.) (019) at test time.
D NETWORK MODELS AND TRAINING FOR STACKED-MNISS EXPERIMENT
In our experiments with Stacked- our generative networks are trained using learning rate of 2e 2, the Adam optimizer with 31 0.9, and the generator updates took place after every 10th
12

---

Under review as a conference paper at ICLR 2020
(a) Generated samples
(b) Real samples
(c) Interpolated samples using our model
Figure 6: Bashion-MNIST results using our generator training algorithm (a). Samples from the real
dataset (b). Interpolated samples using our Generator (c).
DDE step. We use standard parameters for the other methods (DCGAN, WGAN, WGAN+GP), including a learning rate of 2e - 4, the Adam optimizer with 31 = 0.5, and we trained the generator every 5th iteration of the discriminator training.
The NCSN models are trained to remove Gaussian noise at ten different noise standard deviations within the range [1.0, 0.01] (geometric interpolation). The input to the NCSN models include also the noise level. To further improve the quality of the networks, we use separate last-layers for each noise standard deviation for training and test. This way significantly, while we keep the same order of parameters as in the other methods. We used the Adam optimizer with original parameters and learning rate of le 4.
increase the capacity of the network
E DETAILS FOR 2D DATASET TRAINING
Table Blists the hyper-parameters used for different experiments 2D datasets.
Experiment
Dataset
Learning rate
Iterations
Checkerboard
0.05, 0.2
0.001
15000
Two spirals
Figure (density estimation)
0.2
Hight Gaussians
0.005
15000
0.05
Eight Gaussians
0.0005
23750
0.2
Checkerboard
0.001
200000
Figure (generative), Figure 2
0.2
Two spirals
Decrease by half
250000
0.1
Bight Gaussians
every 1000 epochs
250000
Table Training parameters for 2D datasets.
13


Reviews:
[{"review_id": "ryg_dkuQqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a method for density estimation in high-dimensional domains. The proposed algorithm combines ideas from denoising autoencoders The noise estimator procedure developed in this paper estimates the logarithm of the unnormalized density. The inference procedure is stochastic gradient descent w.r.t. the neural net parameters that minimize the KL divergence between the Gaussian smoothed version of the generator and the Gaussian smoothed version of the data.  Alternatively, seen the procedure tries to imitate a kernel density estimator using a parametrized network. Experimental results are demonstrated on a toy 2d datasets and a few real datasets. I like the ideas of the paper and the exposition in this paper.  Here are a few comments and questions\n\n1. My only concern is how scalable is this procedure to high-dimensional data. The very fact that the estimator is trying to imitate a kernel density estimator implies that the assumptions made on the underlying data are that the data comes from a smooth distribution. Modulo this assumption, non-parametric estimators do not make any further assumptions on the data and therefore such non-parametric estimators are only useful for low-dimensional data. So, it is not clear to me if the estimator proposed in this paper can scale to structured high-dimensional data distributions.  \n\n2.  Can you estimate the log-likelihood for the 2d datasets in Figure 1?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxvuxujtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for learning density estimates of a data distribution convolved with noise through “denoising density estimators” and shows how to leverage these density estimates to train generative samplers. The methods are evaluated on toy low-d datasets, MNIST, and fashion MNIST where they show reasonable density estimates and OK sample quality. \n\nMy major concern with this paper is that the “denoising density estimator” proposed here is identical to denoising score matching (which is not cited or discussed). The second contribution of learning a sampler given a density estimate is interesting but likely suffers from all the instabilities of GAN training, and does not compare to related work on distilling energy-based models. Unless both these concerns are addressed, I cannot recommend this paper for acceptance.\n\nMajor comments:\n* The idea of learning a generative model whose energy gradient (score) matches the gradient of the data distribution has been explored extensively under the name “score matching”. The proposed approach of “Deep Denoising Density Estimation” is *identical* to denoising score matching (Vincent et al., 2011, http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf). There’s no discussion of any prior work on score matching in this paper, or comparison with recent approaches in this space (e.g. sliced score matching https://arxiv.org/abs/1905.07088, and https://arxiv.org/abs/1907.05600 that uses denoising score matching).\n* The algorithm presented for learning a generator given an energy function is not compared to other implicit sampling approaches like MCMC. Additionally, the algorithm requires alternating density estimation with updating the generative model, which is quite similar to GAN-training alternating density ratio estimation with updating the generative model. Thus the proposed algorithm likely experiences similar instabilities and challenges in how to partially solve the density estimation step. There are no comparisons to any GAN-based approaches anywhere in the paper. That said, mixing DSMs and explicit generators is a neat area of research.\n* The experiments are too limited, and do not compute any quantitative metrics on the image datasets.\n\nMinor comments:\n* “Defining property of PGMs is that they provide functionality to sample” ->  what about density estimates? Likelihood ratios? etc.\n* Boltzmann machines don’t allow for efficient inference\n* Intro could should spend more time discussing relation to energy-based models, score matching, noise-contrastive estimation\n* Eqn 2 only holds for sigma^2 -> 0, please add and discuss this limitation\n* Divergenc -> divergence, above eqn 12\n* I found the math in section 4 very confusing. What’s the <> notation in this context? Do you need to introduce \\Delta? Alternating density (ratio) estimation and generative model updates is super common in GAN literature and is not discussed here.\n* Table 1: are the estimates of log-likelihood upper or lower bounds? Unlike other approaches, you don’t have exact likelihoods so I’m not sure your #s are comparable.\n* 2048 samples per iteration -> batch size?\n* “For faster convergence, we take 10 DDE gradient steps…”  -> please add an experiment showing how results are impacted by # of gradient descent steps. Is larger # steps always better?\n* MNIST/Fashion MNIST samples aren’t that good to my eye and there are no quantitative metrics (e.g. KID, FID, IS)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1lgAj6y9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the estimation of the score vector (gradient of probability density with respect to random variable) using a noise-added objective function similar to Alain & Bengio, 2014. Compared with Alain & Bengio, 2014 which used a denoised output, the prediction function in this paper eliminates the denoised information and only gives the noise.\nThe derivation in Section 3 is nice and clear, and the derivation gives a nice alternative of using Gaussian noise for score estimation. However, the advantage of using the proposed method is not obvious, and the explanation of generator in Section 4 is unclear.\n\nIn Section 4, how is the proposition 2 applied to the sampling, and what is the situation that the condition in Eq. (12) is satisfied? To what in Algorithm 1 corresponds Delta in Proposition 2? Any of these relationships are not explained well, and unfortunately it is hard to capture the contribution of this paper though with some interesting properties.\n\nFinally, one concern is the variance of the output. The proposed algorithm is designed to estimate the noise, and it should be inefficient compared with Alain & Bengio, 2014 which uses the denoised input as the function’s output. Consideration of the variance of the output and it’s effect on learning should be discussed along with empirical comparison with Alain & Bengio, 2014 from this perspective.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of reviewers suggest that this paper is not yet ready for publication. The idea presented in the paper is interesting, but there are concerns about what experiments are done, what papers are cited, and how polished the paper is. This all suggests that the paper could benefit from a bit more time to thoughtfully go through some of the criticisms, and make sure that everything reviewers suggest is covered.
Decision: Reject