Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper tackles the problem of generative modeling by using Langevin dynamics to sample from the denoising score function. Recently, this family of approaches (Song and Ermon 2019, Song and Ermon 2020) has shown promising and competitive results being positioned as a potential alternative to GANs. 

The paper introduces different improvements over Song and Ermon (2020). A different sampling dynamic (Consistent Annealed Sampling) that produces a more stable training that the traditional annealing scheme by carefully scaling the injected noise. Second, it is empirically shown that running a denoising step on the generated sample leads to an improvement of the FID score. Based on this observation, the paper proposes to use a denoiser trained in an adversarial fashion to synthesize more realistic images.

The work addresses the very relevant problem of how to synthesize images in a realistic way, introducing some modifications to existing works that lead to an improvement on the quality of the generated image.  The paper is well written, presents a nice introduction to the method, which allows to motivate the different modifications in a natural way. The proposed modifications are analyzed in low-dimensional toy experiments and in small-scale images (CIFAR, LSUN-churchers, Stacked-MNIST).

In what follows I list a few questions: 

1. Would it be possible to analyze the strategy of sampling presented in Kadkhodaei and Simoncelli 2020 (concurrent work), and compare to the one proposed in the paper? Both strategies seem to improve the stabilization of the procedure by scaling the noise.

2. Regarding the step of applying the denoiser to the generated sample. I wonder what happens if the denoiser is re-applied? Also, is this connected to the fact that the denoiser may have a fixed point and this fixed point might lead to a better sample? 

3. Regarding using an adversarial denoising. In the denoising literature, there are a few works connecting score matching and state-of-the-art image denoisers. I would like to see a better discussion of this. For example, see,

Romano, Y., Elad, M. and Milanfar, P., 2017. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences, 10(4), pp.1804-1844.

Reehorst, E.T. and Schniter, P., 2018. Regularization by denoising: Clarifications and new interpretations. IEEE transactions on computational imaging, 5(1), pp.52-67.

---
After Discussion:
I think this is a good paper and I would like to see it presented at ICLR2021. 

review_title: Improvement of Langevin dynamics to sample from score matching functions adversarially trained.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission presents three contributions. First, the authors show the inconsistencies in the existing annealed Langevin sampling used in score-matching generative models and propose to correct it with the newly proposed Consistent Annealed Sampling (CAS) algorithm. The second contribution claimed is in providing evidence of the benefits of Expected Denoised Sample (EDS). Furthermore, the submission introduces a hybrid adversarial score-matching model that demonstrates improvements in terms of FID on simpler architectures.

The proposed CAS algorithm is theoretically well-motivated based on the observation that ALS is inconsistent with the scaling of the noise during sampling process (although the question whether noise should follow none other than geometric progression is still an open question). The paper is well-written, and the ablation study is carried out well.

However, it is a bit confusing as to whether the EDS (although under a different name — denoising jump) is a contribution of this paper or is something proposed prior to this work. I understand that this denoising procedure has already been presented as a necessary technique in score matching models. Nevertheless, I believe the authors contributed by showing that both ALS and CAS move samples towards the EDS (Proposition 3) and show additional empirical evidence of its benefits on synthetic and real datasets.

Taking EDS on the last Langevin step diminishes the impact of CAS (doesn't bring unambiguous improvement in FID scores in the experiments), otherwise very interesting finding both theoretically and algorithmically, and substitute for ALS.

The effect of the hybrid model is also not persistent and depends on the architecture used. For an incremental improvement (a combination of two models), the improvement is not consistent across architectures. The paper does not explain whether there is a good rationale for such a combination; therefore I remain sceptical about the results.

Given all the above, I am still leaning a bit towards accepting the paper as it covers an interesting finding relating to the ALS. Although the CAS effect on performance is limited by the EDS, score-matching models are of broad interest for the ICLR community.
review_title: Interesting finding about ALS, other contributions are less convincing

final_score: rating: 7: Good paper, accept
review_content: The paper presents a novel approach for denoising score matching, where the Annealed Langevin Sampling has been substituted by Consistent Annealed Sampling, which adds more stability to the process.

The paper is in general clear and well-written. The contributions are clearly highlighted and the proposed approach is conveniently compared with other state of the art methods, demonstrating its superiority.

Positive aspects:
- The Consistent Annealed Sampling proposed in this paper is more stable than the Annealed Langevin Sampling
- The combination between GAN and score matching improves the quality/diversity of the generated sample

Negative aspects: 
- The limitation of the method to Gaussian noise
- The presentation of a real scenario for your approach would have been a plus

However, I have some questions:
1. Who is n_sigma parameter in Algorithm 1? 
2. Algorithm 1, line 4: there is no iteration over 't' in the loop?
3. How does your denoising scheme work? Do you create noisy samples from your real data and try to denoise them using the proposed approach? Because taking a sample affected by random noise (in the test phase) I guess it won't work. 
4. The denoising scheme is used in a GAN framework, the denoised samples being perceived as real by the discriminator. Is the system trained end-to-end or first you denoise the image and afterwards you train the GAN?
5. Could you please indicate an application scenario which could benefit from this approach, e.g. image-to-image translation, domain adaptation, etc.?
6. Your method is assuming Gaussian noise. Can it be extended to the case of general noise (a noise model which could be also learnt)?

review_title: Recommendation to Accept

final_score: rating: 7: Good paper, accept
review_content: The article deals with generative models based on “Annealed Langevin Sampling“ rather than a GAN.
Theses models suffer from worse FID than GAN.
Authors proposed to denoise the last Langevin samples to reduce the gap in performance with Adversarial Network.
The paper is really easy to read with good illustrations and supporting experiments.

In order to gain in comprehension, especially for people new to ALS, it would have been great if authors have proposed an illustration (and comparison) of the samples evolution along Alg 1 and Alg 2 .

Authors are honest in their revised results comments but I don’t known if they will be able to include the erratum  in a final version

As I was not aware before this review of “Annealed Langevin Sampling” my rating may not be confident.
review_title: Review round 0 for “ Adversarial score matching and improved sampling for image generation”

Meta-review: This paper introduces an alternative to Langevin sampling and also the idea of adversarial score sampling. 
The reviewers are generally supportive of the paper.

Pros:
- The idea behind improving Langevin sampling is theoretically justified and leads to a simple algorithm. 
- The idea behind adversarial score matching is also shown to be effective 
- Improvement over baseline

Cons: 
- Two ideas packed  into one paper, which is reflected by the title as well. 
-  From the narrative it could be thought that using EDS on the last step of CAS is the contribution of the paper. 
Decision: Accept (Poster)