Learning Generative Models using Denoising Density Estimators

Abstract: Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. In addition, we show how to leverage DDEs to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ODE solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.
      

Reviews and author responses:

Dear reviewers, thank you again for your insightful feedback. We have uploaded a revised version of our paper that includes the following major changes:

 - We added a quantitative comparison with GAN models and Score-Matching using the Stacked-MNIST dataset, which consists of 10^3 classes of triple-digit images. The experiment quantifies mode collapse and the divergence from the ground truth label distribution. We show that our approach preserves most modes and leads to a low KL divergence in this scenario, similar to modern GANs (WGAN, WGAN+GP). We outperform a basic DCGAN by a large margin. We also compare to iterative sampling with annealed Langevin dynamics, which obtains slightly worse performance than our approach. 

 - We added experiments on the CelebA dataset showing qualitatively that we can successfully generate samples based on this dataset.

 - We extended the discussion of related work to include the literature pointed out by the reviewers. 

 - We revised the writing to clarify the technical issues brought up by the reviewers.

We also appreciate the comment by Saeed Saremi about the difference between a direct parameterization of the score, vs. its implicit parameterization via the energy function as in our approach. We believe this is an issue that is worth to be further explored and illuminated, both theoretically and empirically. Our experiment on Stacked-MNIST shows some practical advantage of the implicit vs. explicit parameterization, hence we believe that this can be relevant in practice.

In summary, our approach to obtain a generative model by explicitly estimating the energy (un-normalized density) of the generated and true data distributions and minimizing the divergence of these densities is significantly different from previous techniques (GANs, normalizing flows, iterative sampling via score matching, autoregressive models, Boltzmann machines, VAEs, etc.). We believe our experiments show that this is a viable alternative to existing techniques that has the potential to inspire much further research. 

This comment is in support of the authors' response, regarding their parametrization approach. This issue also came up in the Blind Review #3.

The distinction between parametrizing the score function $\mathbf{directly/explicitly}$ (as done in DAEs and most recently in a denoising score matching model that was mentioned by AnonReviewer1) and parametrizing it $\mathbf{implicitly}$ through the energy function (as done by the authors of the paper under review) may appear somewhat technical and minor, but it is in fact quite significant. This issue has been talked about in the literature (more on that below) but it was also analyzed recently in the paper "on approximating $\nabla f$ with neural networks" (arXiv:1910.12744).

For reference, there is a section in [Alain & Bengio, 2014], titled "limited parametrization", where the authors did raise concerns over the parametrization of DAEs. In the arXiv paper, referenced above, the necessary condition for approximating $\nabla f$ was derived and it was shown that the explicit/direct parametrization with neural networks (of any depth $L>2$) comes with extreme limitations on the parameters, which makes the explicit $\mathbb{R}^d \rightarrow \mathbb{R}^d$ parameterization provably ineffective. 

There are also other subtleties regarding score matching. [Raphan & Simoncelli, 2011] and [Vincent, 2011] came out on the same volume of Neural Computation (Volume 23), but the former is practically unknown in the machine learning community. One can however make a strong case (this is expressed between the lines in Raphan & Simoncelli) that the empirical Bayes is the more fundamental interpretation of the (Parzen/denoising) score matching learning objective. This sentiment is also expressed most recently in the paper "neural empirical Bayes" (arXiv:1903.02334).

You are correct that we predict the noise, whereas DAEs (Alain and Bengio, 2014) predict the clean input. However, the key difference is that we represent the noise as the gradient of a scalar function. We estimate this scalar function in contrast to a vector field, and this scalar function corresponds to the unnormalized (log) density. Having the unnormalized density is crucial to define our generator training approach, and this is our main contribution. In contrast to previous generative models based on score matching, our approach does not require MCMC sampling.

Note also that it has been shown in image denoising that predicting the noise leads to better results in practice compared to predicting the clean input (see e.g., “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising”, https://github.com/cszn/DnCNN).

In Section 4, the delta corresponds to the change in the generated density ($\tilde{q}$ in Proposition 2) that results from a generator update based on the first line in the loop in Algorithm 1. Our argument is that a small enough generator update (which can be controlled through the learning rate) satisfies Equations 12, 13, and 14. The proof of Proposition 2 guarantees that the KL divergence decreases if Equation 12-14 are satisfied. 


As you point out, our formulation of denoising density estimators is closely related to denoising score matching. Indeed, the connection between least squares optimal denoising and score matching has been well documented, for example in the work by Raphan and Simoncelli (2011) and Alain and Bengio (2014), which we cite. We will also include the work by Vincent, which describes the same connection and which we failed to mention. One small but crucial difference between denoising score matching (Equation 10 in Vincent, 2011) and our formulation is that Vincent expresses the objective in terms of the score (a vector field), whereas we express the objective in terms of a scalar function (the un-normalized log density). As a key contribution of our work, the un-normalized density allows us to formulate a loss function to train a generator that allows random sampling. This is in contrast to all previous generative models based on score matching, which require Markov Chain sampling (e.g. Bengio, Yao, Alain, Vincent, 2013), or an iterative procedure such as the concurrent work by Ermon et al., (https://arxiv.org/abs/1907.05600). Note that we do mention generative models based on score matching and Markov Chain sampling in our related work section. Thank you for pointing out more recent work in this area, which we will be glad to include. 

An advantage of our approach is also that the score (i.e., the gradient field) of our density estimate is guaranteed to be a conservative vector field. This is not guaranteed in practice in score matching because of the limited capacity of the neural networks that are used to parameterize the score. Non-conservative gradient fields, which also occur in GANs (“The Numerics of GANs”, Mescheder et al. 2017), can be problematic for gradient-based optimization algorithms.

In summary, we formulate an un-normalized scalar density estimate that is based on a small but crucial modification of the well known denoising auto-encoder and denoising score matching objectives. As you mention, mixing DSMs and explicit generators is a neat area of research. Our key contribution to this area is that, in contrast to all previous DAE/score matching based generative models, our approach directly estimates the scalar density (as opposed to the score), which allows us to train generators for direct random sampling (as opposed to MCMC or iterative sampling via Langevin dynamics).

Thank you for also providing a list of detailed minor comments. We will incorporate clarifications into the paper to address all of your comments. To answer some of your questions:
- Eqn. 2 is correct. It does hold for non-zero $\sigma$, not only in the limit as $\sigma\rightarrow 0$. Note that $\tilde{p}$ is the smoothed density here. Equivalent formulations have appeared elsewhere, for example in Raphan and Simoncelli’s (2011) work (Eqn. 2.4). If $\tilde{p}$ were replaced by the original, non-smoothed data density $p$, then indeed Eq. 2 only holds in the limit. This is the formulation in Alain and Bengio (2014), Eqn. 4.
- In Section 4, $<\cdot,\cdot>$ is the standard notation for the inner product. $\Delta$ is the change in the generated density caused by a generator update (first line in the loop in Algorithm 1), and we introduce it solely to proof training convergence. $\Delta$ is not explicitly computed in practice.
- You are correct that we cannot estimate exact likelihoods because we need to approximate the normalizing constant using Monte Carlo integration. We ran the Monte Carlo integration multiple times and report the resulting empirical variance of our likelihoods in Table 1. This shows that we achieve better log-likelihoods than the other methods with high confidence.

We are working on improving the paper according to your suggestions and will post an update soon.


You point to an intriguing observation that non-parametric KDE is often considered to be useful only for low dimensional data (typically up to 6D), whereas our approach and related techniques based on denoising score matching have been demonstrated to work on data with hundreds of dimensions (such as images). We believe that because our approach parameterizes KDE using neural networks it is more robust in higher dimensions. We will add more examples of high dimensional data to the paper to further support this. Analyzing this effect more formally is a very interesting avenue for future research, and related to the more general question of understanding generalization in deep neural networks.

We could add the log-likelihoods to the 2D examples using Monte Carlo estimation of the normalizing constant. We also do this for several higher dimensional datasets and report log-likelihood results in Table 1, which indicates that our approach can outperform the current state of the art techniques. We believe these higher dimensional datasets are more relevant as a benchmark than the 2D examples.


This paper presents an approach for learning density estimates of a data distribution convolved with noise through “denoising density estimators” and shows how to leverage these density estimates to train generative samplers. The methods are evaluated on toy low-d datasets, MNIST, and fashion MNIST where they show reasonable density estimates and OK sample quality. 

My major concern with this paper is that the “denoising density estimator” proposed here is identical to denoising score matching (which is not cited or discussed). The second contribution of learning a sampler given a density estimate is interesting but likely suffers from all the instabilities of GAN training, and does not compare to related work on distilling energy-based models. Unless both these concerns are addressed, I cannot recommend this paper for acceptance.

Major comments:
* The idea of learning a generative model whose energy gradient (score) matches the gradient of the data distribution has been explored extensively under the name “score matching”. The proposed approach of “Deep Denoising Density Estimation” is *identical* to denoising score matching (Vincent et al., 2011, http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf). There’s no discussion of any prior work on score matching in this paper, or comparison with recent approaches in this space (e.g. sliced score matching https://arxiv.org/abs/1905.07088, and https://arxiv.org/abs/1907.05600 that uses denoising score matching).
* The algorithm presented for learning a generator given an energy function is not compared to other implicit sampling approaches like MCMC. Additionally, the algorithm requires alternating density estimation with updating the generative model, which is quite similar to GAN-training alternating density ratio estimation with updating the generative model. Thus the proposed algorithm likely experiences similar instabilities and challenges in how to partially solve the density estimation step. There are no comparisons to any GAN-based approaches anywhere in the paper. That said, mixing DSMs and explicit generators is a neat area of research.
* The experiments are too limited, and do not compute any quantitative metrics on the image datasets.

Minor comments:
* “Defining property of PGMs is that they provide functionality to sample” ->  what about density estimates? Likelihood ratios? etc.
* Boltzmann machines don’t allow for efficient inference
* Intro could should spend more time discussing relation to energy-based models, score matching, noise-contrastive estimation
* Eqn 2 only holds for sigma^2 -> 0, please add and discuss this limitation
* Divergenc -> divergence, above eqn 12
* I found the math in section 4 very confusing. What’s the <> notation in this context? Do you need to introduce \Delta? Alternating density (ratio) estimation and generative model updates is super common in GAN literature and is not discussed here.
* Table 1: are the estimates of log-likelihood upper or lower bounds? Unlike other approaches, you don’t have exact likelihoods so I’m not sure your #s are comparable.
* 2048 samples per iteration -> batch size?
* “For faster convergence, we take 10 DDE gradient steps…”  -> please add an experiment showing how results are impacted by # of gradient descent steps. Is larger # steps always better?
* MNIST/Fashion MNIST samples aren’t that good to my eye and there are no quantitative metrics (e.g. KID, FID, IS)

The authors consider the estimation of the score vector (gradient of probability density with respect to random variable) using a noise-added objective function similar to Alain & Bengio, 2014. Compared with Alain & Bengio, 2014 which used a denoised output, the prediction function in this paper eliminates the denoised information and only gives the noise.
The derivation in Section 3 is nice and clear, and the derivation gives a nice alternative of using Gaussian noise for score estimation. However, the advantage of using the proposed method is not obvious, and the explanation of generator in Section 4 is unclear.

In Section 4, how is the proposition 2 applied to the sampling, and what is the situation that the condition in Eq. (12) is satisfied? To what in Algorithm 1 corresponds Delta in Proposition 2? Any of these relationships are not explained well, and unfortunately it is hard to capture the contribution of this paper though with some interesting properties.

Finally, one concern is the variance of the output. The proposed algorithm is designed to estimate the noise, and it should be inefficient compared with Alain & Bengio, 2014 which uses the denoised input as the function’s output. Consideration of the variance of the output and it’s effect on learning should be discussed along with empirical comparison with Alain & Bengio, 2014 from this perspective.


This paper provides a method for density estimation in high-dimensional domains. The proposed algorithm combines ideas from denoising autoencoders The noise estimator procedure developed in this paper estimates the logarithm of the unnormalized density. The inference procedure is stochastic gradient descent w.r.t. the neural net parameters that minimize the KL divergence between the Gaussian smoothed version of the generator and the Gaussian smoothed version of the data.  Alternatively, seen the procedure tries to imitate a kernel density estimator using a parametrized network. Experimental results are demonstrated on a toy 2d datasets and a few real datasets. I like the ideas of the paper and the exposition in this paper.  Here are a few comments and questions

1. My only concern is how scalable is this procedure to high-dimensional data. The very fact that the estimator is trying to imitate a kernel density estimator implies that the assumptions made on the underlying data are that the data comes from a smooth distribution. Modulo this assumption, non-parametric estimators do not make any further assumptions on the data and therefore such non-parametric estimators are only useful for low-dimensional data. So, it is not clear to me if the estimator proposed in this paper can scale to structured high-dimensional data distributions.  

2.  Can you estimate the log-likelihood for the 2d datasets in Figure 1?

Thank you for pointing this out. The images in Figure 1 results have rendering issues and require corrections: the resulting images form BNAF and GLOW are flipped horizontally, and our results are rotated 90 degrees (clock-wise).
Figure  2 images have correct orientation.

Thank you for your answer. I understood what you said

I would like to ask one more regarding the toy 2D experiment result.

In figure 2, DDE sampler generated two spirals well but, in Figure 1, the estimated density looks like it has been rotated clockwise by 90 degrees. (sigma value were equivalent each) 

Could you explain why?

Thanks and good luck

Jaeyoon

1. You are right, there is a mistake in the signs in Equations 7 and 8. Thank you for pointing this out. Equation 9 remains correct.

2. This means to say that we estimate the integral of our un-normalized density model using Monte Carlo integration and importance sampling with a Gaussian density. 

3. You are right that we need to compute two derivatives, one derivative w.r.t x and one w.r.t the neural net parameters. The cost of these computations, however, is manageable. Consider this analogy to training an autoencoder with a scalar bottleneck: The DDE corresponds to the encoder part, and the derivatives wrt. x correspond to the decoder. Finally, computing the derivatives wrt. to the neural network parameters corresponds to the backward pass through both the decoder and encoder. Hence our approach is comparable in computational cost to standard techniques in deep learning.

Hi, it was really interesting paper and I have a few simple questions

First, in eq(6) -> eq(7), differentiating (6) seems to lead minus on the r.h.s in eq (7)

Could you explain 'we use importance sampling using a Gaussian distribution as the base density' more in detail?

Finally, the loss in (10) has a derivative so optimizing process should require hessian (one derivative w.r.t x and one w.r.t neural net parameter).  So doesn't it require heavy computation cost than others? (such as one with solving ODE)

Anyway, I think it's interesting paper for me again.

Thanks

This comment is about missing citations.

This paper has an uncanny similarity to two recent works  "deep energy estimator networks": https://arxiv.org/abs/1805.08306 and "neural empirical Bayes": https://arxiv.org/abs/1903.02334

The authors cite (Raphan & Simoncelli, 2011), whom we also took inspiration from. (Raphan & Simoncelli, 2011) is quite unknown in the machine learning community, and it has a total of 28 citations that is easy to trace—the only papers who have cited them in the context of “neural density estimation” are  (Saremi et al, 2018) and (Saremi & Hyvarinen, 2019).

Lastly, I also add (Vincent, 2011) to the very top of the list of missing citations!



Unfortunately, we were not aware of these references at the time of our paper submission. We developed our work independently based on our own previous research published at NIPS 2017 (we are omitting the full citation to preserve anonymity but would be glad to forward it to the conference chairs if necessary). We learned about the work by Raphan and Simoncelli through personal communication with Eero Simoncelli in 2017 and we did not check for new citations of his work in 2019. We will be glad to discuss similarities and differences to the references you pointed out in an updated version of our paper.

Meta-review: The majority of reviewers suggest that this paper is not yet ready for publication. The idea presented in the paper is interesting, but there are concerns about what experiments are done, what papers are cited, and how polished the paper is. This all suggests that the paper could benefit from a bit more time to thoughtfully go through some of the criticisms, and make sure that everything reviewers suggest is covered.

Decision: reject