Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a method of learning of energy based models using denoising score matching. This technique has been used before but only with limited success. The authors hypothesize that this is due to the fact that the matching was only performed over a single noise scale. The main idea of this work is to employ a range of scales to learn a single energy function. This trick helps to alleviate the problem of noisy samples concentrating in a low-volume region of the ambient space.

It seems that the paper draws significant inspiration from the work by Song & Ermon, 19. The difference between the two appears to be minor:
1) The density is represented as a Boltzman distribution and therefore the score function is reduced to the gradient of the energy function (this has been done before)
2) Instead of conditioning the energy on the noise level the authors propose to use explicit scaling by the inverse temperature
 
Pros:
+ The paper is mostly well-written.
+ I think Section 2 does a good job at illustrating challenges in training energy-based models using denoising score matching with a single noise scale. 
+ As compared to (Song & Ermon, 19) using the Boltzman distribution ensures that the learned score is an actual conservative vector field. Arguably, learning an image to scalar network is easier than learning an image to image one.
+ Samples from the model are of competitive visual quality.

Cons:
- Scaling energy by the inverse temperature seems to be one of the most important aspects of the paper but is only justified by “intuition from physics”. I’m not entirely sure that this is a valid assumption. In contrast, (Song & Ermon, 19) don’t put any hard constraints on the values of the score for different noise levels besides that they are produced by a single conditional network. I would appreciate if the authors discussed that difference in more detail.
- The authors don’t provide any analysis as to whether the annealed Langevin MC procedure leads to the samples from the right distribution.
- The quantitative results don’t seem to be better (actually, they are worse) than those from (Song & Ermon, 19).

Notes/questions:
* Abstract: “unmormalized” -> “unnormalized”
* Section 2.1, (1): \tilde{x} -> \tilde{\mathbf{x}}
* Section 2.2, paragraph 2: What does superscript C mean in the noisy manifold? Never defined.
* Section 2.2, paragraph 4: “some example” -> “some examples” (?)
* Section 3, paragraph 1: “CIFAT-10” -> “CIFAR-10”
* Section 4, paragraph 2: “for each T as a separate model”. I don’t think this is a correct statement. (Song & Ermon, 19) use a single conditional model for all the noise levels.
* Section 4, paragraph 2: “does not rely on explicit receive noise magnitude” -> “does not rely on receiving noise magnitude explicitly” (?) I also don’t quite understand this entire sentence. Does the model really infer the noise magnitude from a given image? It seems like in Equation (7) there is an assumption that the temperature T is equal to 1. I don’t feel like there is a lot of difference between the proposed model and (Song & Ermon, 19) when it comes to supplying noise information. I’d appreciate if the authors could clarify that bit for me.

My main concern about this paper is that it doesn’t seem like a big step from its starting point (Song & Ermon, 19). The modifications are shown to work empirically but don’t result in a significantly better model. Moreover, I feel like the paper could do a better job at justifying those changes. I’m giving a borderline score but willing to increase it if the authors address my questions.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ########Updated Review ###########

I would like to thank the author(s) for their reply, which I have carefully read and it partly addresses my original concerns. Still, as agreed by all three reviewers, this paper might not be a significant step up compared with [1]. I am raising my point to weak reject to reflect my updated belief. I think this paper needs a bit more highlights to pass the threshold. 

###############################


This paper tries to address the problem of non-parametric maximal likelihood estimation via matching the score function wrt data. It is a clear rejection due to its significant overlap with the recent NeurIPS publication [1]. The author(s) have failed to clarify how their proposal differs from [1] in a significant way. From what I can tell after a quick read, both papers tried to training the score function using the denoising auto-encoder, amortized through a neural network, strategically annealed with a sequence of different noise levels, sampled with the Langevin scheme. I put two papers side-by-side and you can visually tell the uncanny resemblance.  Additionally, the proposed model does not outperform that from [1] (see Table 1). I am also not happy about the misleading statement in the abstract that this work "assign likelihood to test data", which is actually performed by AIS.  Section 2.2 is particularly problematic. The assumption of "data approximately uniformly distributed on the manifold" is outrageous, which basically invalidates the need for density estimation because of the uniformity. The 1/f power law characteristic is irrelevant to the likelihood estimation problem, and the statements are both heuristic & misleading. 

[1] Y Song, S Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes to learn an energy based generative model using an ‘annealed’ denoising score matching objective. The main contribution of the paper is to show that denoising score matching can be trained on a range of noise scales concurrently using a small modification to the loss. Compared to approximate likelihood learning of Energy based models the key benefit is to sidestep the need for sampling from the model distribution which has proven to be very challenging in practice. Using a slightly modified Langevin Sampler the paper further demonstrated encouraging sample qualities on CIFAR10 as measured by FID and IS scores. 

Overall I think the paper is well motivated and written, experiments are sound with encouraging results that will be useful for further progress in training energy based models. I currently score the paper as a ‘weak accept’, the reason for not giving ‘accept’ is that I think the paper is closely related to Song & Ermin 2019 (see detailed comments below) - However i can be convinced to bump my score depending on the author feedback 

Q1) I think you should elaborate more on how exactly your method is different from the NCSN model presented in Song & Ermon 2019? Especially.
Q1.1) Is your method similar to the NCSN except that you do linear scaling with temperature in the loss and train a joint model across all temperature scales?

Q1.2) In the related works section you claim that ‘[Song & Ermin] … this model learns p(xhat) for each T as a separate model’. Quickly reading through that paper i do not think that statement is accurate - I think they learn a model where the main difference is that it takes T as input instead of scaling the gradient term in the loss?

Q1.3 )Do you have any intuition for why they seem to get slightly better results than the one you obtain in your paper? Is it simply architecture/training details that differ or something more ‘fundamental’?


Q2) In relation to the Score matching objective.
Q2.1) In eq (4) it is not completely clear to me what the motivation for linear scaling in T is. Can you elaborate on what you mean with ‘We borrow intuition from physics and simply set E_T(xhat) = E(xhat)/T ...’?
In relation to the above Can you clarify which part of your results holds for Gaussian noise and which holds in general. 

Q2.2) For the gaussian case I think linear scaling as done in eq(5) is sensible, however for arbitrary noise distributions linear scaling is akin to a first order approximation (which might be inaccurate across a range of different noise levels)?
Minor: I think it would ease the reading of the paper if you showed the derivation (in appendix) that Eq (1) and Eq(2) are equivalent.

Minor Comment: Learning generative models using denoising have also been explored in [Soenderby 2016]. Here the difficulties of different noise scales was also found and explored but (importantly) not solved.

[Song & Ermon]. Generative Modeling by Estimating Gradients of the Data Distribution
[Soenderby 2016]: Amortised map inference for image super-resolution

review_title: Official Blind Review #3

Meta-review: This paper presents a variant of the Noise Conditional Score Network (NCSN) which does score matching using a single Gaussian scale mixture noise model. Unlike the NCSN, it learns a single energy-based model, and therefore can be compared directly to other models in terms of compression. I've read the paper, and the methods, exposition, and experiments all seem solid. Numerically, the score is slightly below the cutoff; reviewers generally think the paper is well-executed, but lacking in novelty and quality of results relative to Song & Ermon (2019). 

Decision: Reject