Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced an alternative sampling method, with an application on generative models, by convolving an unknown distribution $p_x$ with a factorial kernel called multi-measurement noise model (MNM). The resulting M-density $p_y$ is smoother (easier to sample from), and is permutation invariant. Two factorial kernels, Poisson and Gaussian MNMs, are introduced for the convolution, and can be connected to Bayes estimator, and the learning of parametric energy and score functions. Two parameterization schemes are proposed for modeling the energy and score functions of the Gaussian M-density respectively. Empirical results on FFHQ-256 dataset are very impressive. The main contribution is the usage of factorial kernels, which is a direct extension of smoothing a density in non-parametric density estimation, and is not used for generative models before.
main_review: 1. The author claim that the smoothed M-density $p_\mathbf{Y}$ is easier to learn from than the original distribution $p_\mathbf{X}$ in high-dimensional settings. However, there are no simple examples or existing literature included in the manuscript to support this statement. The difficulty of sampling from a high-dimensional distribution is a core problem in designing generative models, so the authors should consider making this motivation clear and better supported.
2. In the introduction, the authors stated that the MCMC is considered an art that cannot converges fast for complex distribution, but later in Section 5, the authors adopted Langevin MCMC to sample the M-density. It is encouraged to compare the MCMC with the Langevin MCMC for M-density since the sampling efficiency is one of the selling point of this paper. For example, is the convergence rate of Langevin MCMC faster? The authors should at least show it in the empirical study.
3. The authors did not provide a good motivation for why considering the Poisson and Gaussian MNMs in Section 2. What about other MNMs? Does the choice of the MNM kernel related to the properties of $p_\mathbf{X}$? Is there an optimal MNM kernel to pick? In Section 2 the authors focused on the derivation of the closed-form expressions of the MNMs, but the reasoning on why they chose them is more important in my opinion.
4. The authors proposed several parameterization schemes in Section 4, including MDAE, MEM${}^2$, and MUVB. What are the pros and cons of them over each other? A discussion is needed or at least added in Appendix.
5. The permutation invariance is one the appealing points of using MNMs, and in the Introduction, the authors stated ``In Sec. 6, we present our experiments on MNIST, CIFAR-10, and FFHQ-256 datasets which were focused on permutation invariant M-densities.''. However, in Section~6, the discussion on the permutation invariant M-densities is missing.
6. An empirical comparison between MDAE and other generative models is lacking. How to compare the quality of the generated samples? In the illustrations, the authors focused on facial images, and in some figures, the hair or background in the images are not generated well, why? It is unclear about the advantages of the proposed MDAE over other generative methods, and a discussion is needed.
7. The abbreviation DAE for denoising auto-encoder is used without definition in the abstract. Although DAE is well-known in the field, it is suggested that the authors still defined all the abbreviations before usage in this manuscript. Another example is MCMC for Monte-Carlo Markov chain.
summary_of_the_review: The proposed MNMs and MDAE are very appealing, and novel. I believe this will be a great publication if the authors could please address the issues listed above. In particular, I think the authors should give (1) a better motivation on why this methodology outperforms other generative methods, (2) how to compare different parameterization schemes, (3) a more thorough discussion on the experiments regarding the permutation invariance, efficiency over MCMC or other methods, and (4) companion on the quality of the generated samples with existing methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a multimeasurement noise model to learn and sample from some unknown distribution and demonstrated its effectiveness on datasets like MNIST, CIAR-10 and FFHQ-256. It studies the convolution of the distribution with different levels of noise and uses the Bayes estimator in each noise level to recover the original sample. 
main_review: This paper combined DAE and empirical Bayes to learn and sample from some unknown distribution. This method is a novel method and is very different from previous methods that use convolution to smooth the distributions. While other methods are based on weighted noise or annealing, different noise levels are equal in this method.

The main body of the paper is in general well-written. I have the following questions on the paper.
1. Sec 2.1 derived the formula for poisson MNM, but it seems the other parts of the paper are mainly based on Gaussian NMN. Are there any further results on Poisson MNM? How does it compare with the Gaussian MNM?
2. Sec 2.3 gives a concentration rate of the estimator, but it is not straightforward to see how sigma_eff is derived. 
3. Sec 5 mentioned that underdamped langevin is better than langevin because UL scales as sqrt d theoretically. It is also mentioned in the appendix that in your experiments, UL mixes faster. Are there any experimental results on how much faster it mixes? 

summary_of_the_review: This paper is in general a good paper. The new generative method it proposed can be of interest to the ICLR community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Given $n$ independent samples $x_i$ in a space of dimension $d$ drawn from an unknown distribution $p(x)$, this paper is interested in drawing new samples independent of $x_i$ but coming from the same unknown distribution p(x). 

The classical approach consists of learning $p(x)$ from $x_i$ (i.e., approximated by $\tilde{p}(x)$) and sampling new points according to the approximated version $\tilde{p}(x)$, which remains a difficult task given sometimes the highly non-convex character of $p(x)$. 

For these reasons, the authors propose to noise the data $x$ using $M$ different channels whose noise levels are chosen to obtain data denoted $y$. Since the level noise of each channel is known, this allows using a Bayesian estimator $\hat x(y)$ to find the samples in the original space. The advantage of this approach is that the Bayesian estimator depends on the density $p(y)$, which allows, thanks to a chosen parameterization of $p(y)$ (which therefore leads to a parametrization of $\hat{x}(y)$), to find the optimal parameters through a least-square objective by minimizing $\|x-\hat{x}(y)\|^2$ on the training data. The Bayesian estimator thus allows generating new samples using the optimized density $p^{\star}(y)$ with the corresponding optimal parameters. 

The authors make a connection between the proposed method and some methods of the literature constituting new intuitions, different views of algorithms proposed in the literature. The experiments conducted suggest the efficiency of the algorithm to generate the most diverse samples
main_review: ${\bf \text{Strengths}}$

1-The problem addressed in this paper (generating new independent samples of a given selection of samples drawn from an unknown distribution) is of great interest to the machine learning community.

2-The approach used (going through a variety where the density estimation is easier) is very original. The intuitions and connection with literature are interesting and allow a new point of view of the algorithms already used (DAE as an example). Several experiments have been designed to justify the relevance of the approach.

3-The article is quite well written and well structured.

4-Although the algorithm uses already known sampling methods and energy function optimization methods, the general idea of the Bayesian estimator is an interesting contribution to the machine learning community.

${\bf \text{Weaknesses}}$

1-The motivation of presenting the Poisson MNM is unclear specifically since it is never used afterward and more importantly no motivation to discard it and prefer the Gaussian MNM. More specifically, isn't it to present a simpler version of MNM before tackling the more involved Gaussian MNM? But more importantly what are the disadvantages of Poisson MNM to discard it?

2-The discussion of the higher noise level is also not convincing. Moreover, the advantage/disadvantage of increasing M is not clearly discussed (only the advantages are pointed out). I am quite convinced that there is a great connection between $M$ and $\sigma$. More concretely, there is no discussion about the time complexity of the algorithm which may be related to increasing $M$. It seems clear that with a low level of noise we do not need to have too many channels measurement (independent samples). Therefore for me, low values of $\sigma$ and low value of $M$ are comparable to high values of $\sigma$ and high value of $M$. So in the comparison for MNIST (between two different values of $\sigma$), it is not fair to change the value of $M$. In an ablation study, you normally need to keep $M$ fixed and change $\sigma$ since the two are connected.  Having big values of $\sigma$ is great to smooth well the manifold but in compensation, we need high values of $M$, which may be problematic for time complexity (which is not discussed).

3-In the abstract, I have the impression that the sentence "Samples from $p(x)$ are obtained by walk jump
sampling, using underdamped Langevin MCMC to sample from $p(y)$" is incomplete since we should close the loop and mention that the empirical Bayes estimator is used after this underdamped Langevin.

4-I the abstract, it is mentioned fast mixing. But the term fast is never concretely shown (no running time,...)

5-In the concentration rate of the empirical estimate, it is mentioned that the optimal estimator is expected to give a better rate than the rate of Tao. This is not proved, right? More concretely what is the rate of the optimal estimator? What is the intuition to say that the rate will be better than just a constant?

6-In the definition of the loss $\sum_{m=1}^{M}\mathcal{L}^{m}(\theta)$, why a normalization with $\frac 1M\sum_{m=1}^{M}\mathcal{L}^{m}(\theta)$ is not used since the large $M$ case is included in the framework? More concretely is the limit $\lim_{M\rightarrow\infty}\sum_{m=1}^{M}\mathcal{L}^{m}(\theta)$ correctly defined?

7-In section Denoising score matching it is written 'empircal' instead of empirical

summary_of_the_review: The paper proposes an original work by offering a way out of the problem of approximating very complex densities by passing through a smoother variety. The method can be inspiring since it makes some connection and gives another point of view of the Denoising AutoEncoder proposed in the literature. The experiment shows that the method allows for generating stable and mixed samples and does not get stuck to some mode. Even if some discussions are not well-motivated, I recommend acceptance for the paper.
review_title: null

Meta-review: This paper studies the problem of generative modeling by convolving an unknown complex density with a factorial kernel called multi-measurement noise model (MNM) to obtain a smoother density that is easier to sample from. Poisson and Gaussian MNMs are proposed for the convolution. Experiment regarding image synthesis are conducted to demonstrate the effectiveness of the proposed framework. The paper studies a problem that is of great interest to the machine learning community, and the results are impressive and promising. However, the paper in the current form lack a comparative discussion and a quantitative comparison with some related works. After the rebuttal, all three reviewers tend to accept the paper. After several rounds of internal discussion among AC, reviewers and authors, the AC agrees with the reviewers, and recommends accepting the paper, given the changes the authors promised to make.  

In summary, the AC recommends an acceptance and urges the authors to further revise their paper by adding a comparative discussion with those closely related works regarding generative models using MCMC sampling.
Decision: Accept (Poster)