Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper provides a method for density estimation in high-dimensional domains. The proposed algorithm combines ideas from denoising autoencoders The noise estimator procedure developed in this paper estimates the logarithm of the unnormalized density. The inference procedure is stochastic gradient descent w.r.t. the neural net parameters that minimize the KL divergence between the Gaussian smoothed version of the generator and the Gaussian smoothed version of the data.  Alternatively, seen the procedure tries to imitate a kernel density estimator using a parametrized network. Experimental results are demonstrated on a toy 2d datasets and a few real datasets. I like the ideas of the paper and the exposition in this paper.  Here are a few comments and questions

1. My only concern is how scalable is this procedure to high-dimensional data. The very fact that the estimator is trying to imitate a kernel density estimator implies that the assumptions made on the underlying data are that the data comes from a smooth distribution. Modulo this assumption, non-parametric estimators do not make any further assumptions on the data and therefore such non-parametric estimators are only useful for low-dimensional data. So, it is not clear to me if the estimator proposed in this paper can scale to structured high-dimensional data distributions.  

2.  Can you estimate the log-likelihood for the 2d datasets in Figure 1?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors consider the estimation of the score vector (gradient of probability density with respect to random variable) using a noise-added objective function similar to Alain & Bengio, 2014. Compared with Alain & Bengio, 2014 which used a denoised output, the prediction function in this paper eliminates the denoised information and only gives the noise.
The derivation in Section 3 is nice and clear, and the derivation gives a nice alternative of using Gaussian noise for score estimation. However, the advantage of using the proposed method is not obvious, and the explanation of generator in Section 4 is unclear.

In Section 4, how is the proposition 2 applied to the sampling, and what is the situation that the condition in Eq. (12) is satisfied? To what in Algorithm 1 corresponds Delta in Proposition 2? Any of these relationships are not explained well, and unfortunately it is hard to capture the contribution of this paper though with some interesting properties.

Finally, one concern is the variance of the output. The proposed algorithm is designed to estimate the noise, and it should be inefficient compared with Alain & Bengio, 2014 which uses the denoised input as the function’s output. Consideration of the variance of the output and it’s effect on learning should be discussed along with empirical comparison with Alain & Bengio, 2014 from this perspective.

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper presents an approach for learning density estimates of a data distribution convolved with noise through “denoising density estimators” and shows how to leverage these density estimates to train generative samplers. The methods are evaluated on toy low-d datasets, MNIST, and fashion MNIST where they show reasonable density estimates and OK sample quality. 

My major concern with this paper is that the “denoising density estimator” proposed here is identical to denoising score matching (which is not cited or discussed). The second contribution of learning a sampler given a density estimate is interesting but likely suffers from all the instabilities of GAN training, and does not compare to related work on distilling energy-based models. Unless both these concerns are addressed, I cannot recommend this paper for acceptance.

Major comments:
* The idea of learning a generative model whose energy gradient (score) matches the gradient of the data distribution has been explored extensively under the name “score matching”. The proposed approach of “Deep Denoising Density Estimation” is *identical* to denoising score matching (Vincent et al., 2011, http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf). There’s no discussion of any prior work on score matching in this paper, or comparison with recent approaches in this space (e.g. sliced score matching https://arxiv.org/abs/1905.07088, and https://arxiv.org/abs/1907.05600 that uses denoising score matching).
* The algorithm presented for learning a generator given an energy function is not compared to other implicit sampling approaches like MCMC. Additionally, the algorithm requires alternating density estimation with updating the generative model, which is quite similar to GAN-training alternating density ratio estimation with updating the generative model. Thus the proposed algorithm likely experiences similar instabilities and challenges in how to partially solve the density estimation step. There are no comparisons to any GAN-based approaches anywhere in the paper. That said, mixing DSMs and explicit generators is a neat area of research.
* The experiments are too limited, and do not compute any quantitative metrics on the image datasets.

Minor comments:
* “Defining property of PGMs is that they provide functionality to sample” ->  what about density estimates? Likelihood ratios? etc.
* Boltzmann machines don’t allow for efficient inference
* Intro could should spend more time discussing relation to energy-based models, score matching, noise-contrastive estimation
* Eqn 2 only holds for sigma^2 -> 0, please add and discuss this limitation
* Divergenc -> divergence, above eqn 12
* I found the math in section 4 very confusing. What’s the <> notation in this context? Do you need to introduce \Delta? Alternating density (ratio) estimation and generative model updates is super common in GAN literature and is not discussed here.
* Table 1: are the estimates of log-likelihood upper or lower bounds? Unlike other approaches, you don’t have exact likelihoods so I’m not sure your #s are comparable.
* 2048 samples per iteration -> batch size?
* “For faster convergence, we take 10 DDE gradient steps…”  -> please add an experiment showing how results are impacted by # of gradient descent steps. Is larger # steps always better?
* MNIST/Fashion MNIST samples aren’t that good to my eye and there are no quantitative metrics (e.g. KID, FID, IS)
review_title: Official Blind Review #1

Meta-review: The majority of reviewers suggest that this paper is not yet ready for publication. The idea presented in the paper is interesting, but there are concerns about what experiments are done, what papers are cited, and how polished the paper is. This all suggests that the paper could benefit from a bit more time to thoughtfully go through some of the criticisms, and make sure that everything reviewers suggest is covered.
Decision: Reject