Adversarial score matching and improved sampling for image generation
Authors: Alexia Jolicoeur-Martineau
Rémi Piché-Taillefer
Ioannis Mitliagkas
Remi Tachet des Combes
Venue: ICLR 2021 Poster (2021)
Keywords: adversarial
score matching
Langevin dynamics
GAN
generative model

Abstract: Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin samples using the score network.
In addition, we propose two improvements to DSM-ALS:  1) Consistent Annealed Sampling as a more stable alternative to Annealed Langevin Sampling, and 2) a hybrid training formulation, composed of both Denoising Score Matching and adversarial objectives. By combining these two techniques and exploring different network architectures, we elevate score matching methods and obtain results competitive with state-of-the-art image generation on CIFAR-10.

ADVERSARIAL SCORE MATCHING AND IMPROVED SAMPLING FOR IMAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin samples using the score network, In addition, we propose two improvements to DSM-ALS: 1) Consistent Annealed Sampling as more stable alternative to Annealed Langevin Sampling, and 2) a hybrid training formulation, composed of both Denoising Score Matching and adversarial objectives. By combining these two techniques and exploring different network architectures, we elevate score matching methods and obtain results competitive with state-of-the-art image generation CIFAR-10.
INTRODUCTION
Song and Ermon (2019) recently proposed novel method of generating samples from a target distribution through combination of Denoising Score Matching (DSM) (Hyvärinen, 2005; Vincent, 2011; Raphan and Simoncelli, 2011) and Annealed Langevin Sampling (ALS) (Welling and Teh, 2011; Roberts et al., 1996). Since convergence to the distribution is guaranteed by the ALS, their approach (DSM-ALS) produces high-quality samples and guarantees high diversity. Though, this comes at the cost of requiring an iterative process during sampling, contrary to other generative methods. These generative methods can notably be used to diverse tasks like colorization, image restoration and image inpainting (Song and Ermon, 2019; Kadkhodaic and Simoncelli, 2020).
Song and Ermon (2020) further improved their approach by increasing the stability of score matching training and proposing theoretically sound choices of hyperparameters. They also scaled their approach to higher-resolution images and showed that DSM-ALS is competitive with other generative models. Song and Ermon (2020) observed that the images produced by their improved model were more visually appealing than the ones from their original work; however, the reported Fréchet Inception Distance (FID) (Heusel et al., 2017) did not correlate with this improvement.
Although DSM-ALS is gaining traction, Generative adversarial networks (GANs) (Goodfellow et al., 2014) remain the leading approach to generative modeling. GANs are a very popular class of generative models; they have been successfully applied to image generation (Brock al., 2018; Karras et al., 2017; 2019; 2020) and have subsequently spawned wealth of variants (Radford al., 2015a; Miyato al., 2018; this method is to train Discriminator (D) to correctly distinguish real samples from fake samples generated by a second agent, known as the Generator (G). GANs excel at generating high-quality samples as the discriminator captures features that make image plausible, while the generator learns to emulate them.
2018; Zhang et al., 2019). The idea behind
Still, GANs often have trouble producing data from all possible modes, which limits the diversity of the generated samples. A wide variety of tricks have been developed to address this issue in GANs (Kodali et al., 2017; Gulrajani et al., 2017; Arjovsky et al., 2017; Miyato et al., 2018; Jolicoeur- Martineau and Mitliagkas, 2019), though it remains an issue to this day. DSM-ALS, on the other hand, does not suffer from that problem since ALS allows for sampling from the full distribution

---

captured by the network. Nevertheless, the perceptual quality of DSM-ALS higher-resolution images has so far been inferior to that of GAN-generatee images. Generative modeling has since seen some incredible work from Ho et al. (2020), who achieved exceptionally low (better) FID on image generation tasks. Their approach showcased a diffusion-basee method (Sohl-Dickstein et al., 2015; Goyal al., 2017) that shares close ties with DSM-ALS, and additionally proposed convincing network architecture derived from Salimans et al. (2017).
In this paper, after introducing the necessary technical background in the next section, we build upon the work of Song and Ermon (2020) and propose improvements based on theoretical analyses both at training and sampling time. Our contributions are follows:
We propose Consistent Annealed Sampling (CAS) as more stable alternative ALS, correcting inconsistencies relating to the scaling of the added noise;
We show how to recover the expected denoised sample (EDS) and demonstrate its unequiv- ocal benefits w.E. the FID. Notably, we show how to resolve the mismatch observed in DSM-AL.S between the visual quality of generated images and its high (worse) FID;
We propose to further exploit the EDS through a hybrid objective function, combining GAN and Denoising Score Matching objectives, thereby encouraging the EDS of the score network be realistic possible.
In addition, show that the network architecture used used by Ho al. (2020) significantly improves sample quality over the RefineNet (Lin et al., 2017a) architecture used by Song and Ermon (2020). In ablation study performed on CIFAR-10 and LSUN-church, we demonstrate how these contributions bring DSM-ALS in range of the state-of-the-arr for image generation tasks w.r.f the FID. The code to replicate experiments is publicly available [Available in supplementary material].
2
BACKGROUND
2.1 DENOISING SCORE MATCHING
Denoising Score Matching (DSM) (Hyvärinen, 2005) consists of training network to approxi- mate the gradient of the log density of a certain distribution (V, log p(z)), referred to as the score function. This is achieved by training the network approximate noisy surrogate of p multiple levels of Gaussian noise corruption (Vincent, 201 1). The score network 8, parametrized by 0 and conditioned on the noise level a, is tasked minimize the following loss:
E 2
(I)
where p(ã, x. (ā (=(()))((())))( We define further qo (ā]()) Na r²I) the corrupted data distribution, p(x) the training data distribution, and p(ar) the uniform distribution over a set {a,} corresponding to different levels of noise. In practice, this set is defined as a geometrie progression between σi and ar (with L chosen according to some computational budget):
(o:) 2'a1 iE (0, .... L 1},
02 =
σ1
Rather than having learn a different score function for every d;, one can train an unconditional score network by defining (z o;) = sp (7)/o,, and then minimizing Eq. 1. While unconditional networks are less heavy computationally, remains an open question whether conditioning helps performance. Li et al. (2019) and Song and Ermon (2020) found that the unconditional network produced better samples, while Ho et al. (2020) obtained better results than both of them using a conditional network. Additionally, the denoising autoencoder described in Lim et al. (2020) gives evidence supporting the benefits of conditioning when the noise becomes small (also see App. D and E for theoretical discussion of the difference). While our experiments are conducted with unconditional networks, we believe our techniques can be straightforwardly applied to conditional networks; leave that extension for future work.
2

---

2.2 ANNEALED LANGEVIN SAMPLING
Given score function, one 2011) to sample from the corresponding probability distribution. In practice, the score function is generally unknown and estimated through score network trained minimize Eq. 1. Song and Ermon (2019) showed that Langevin sampling has trouble exploring the full support of the distribution when the modes are too far apart and proposed Annealed Langevin Sampling (ALS) as a solution. ALS starts sampling with a large noise level and progressively anneals it down to value close to 0, ensuring both proper mode coverage and convergence to the data distribution. Its precise description is shown in Algorithm 1.
Langevin dynamics (or Langevin sampling) (Welling and Teh,
Algorithm 1 Annealed Langevin Sampling
Algorithm 2 Consistent Annealed Sampling
Require: so: (o
Require: sg, (or 7, 0L+1
1: Initialize x:
1: Initialize a
2: 8 + V - (1 - 10/-e//f/ e/07)² /72
2: for forit to L do
3: 0; a
3: for i + to L do
4: for ni steps do
4:
0; 0 a= o0?
5:
Draw z ~ N( I)
5:
Draw 2 ~ (0,
6:
x + a;s0 (x, σi) + 2202z
6:
x x + an50 σ.) + 30:41
return æ
return æ
2.3
EXPECTED DENOISED SAMPLE (EDS)
A little known fact from Bayesian literature is that recover denoised sample from the score function using the Empirical Bayes mean (Robbins, 1955; Miyasawa, 1961; Raphan and Simoncelli, 2011):
s' =ã, H ()
(3)
x²
where H (", o) A (x0 [z] is the expected denoised sample given noisy sample (or Empirical Bayes mean), conditioned on the noise level. A different way reaching the same result through the closed-form of the optimal score function, as presented in Appendix D. The corresponding result for unconditional score function is presented in Appendix E for completeness.
The EDS corresponds to the expected real image given corrupted image; it can be thought of as what the score network believes to be the true image concealed within the noisy input. It has also been suggested that denoising the samples (i.e., taking the EDS) at the end of the Langevin sampling improves their quality (Saremi and Hyvarinen, 2019; Li et al., 2019; Kadkhodaie and Simoncelli, 2020). In Section 4, we provide further evidence that denoising the final Langevin sample brings closer to the assumed data manifold. In particular, we show that the Fréchet Inception Distance (FID) consistently decreases (improves) after denoising. Finally, Section 5, we build hybrid training objective using the properties of the EDS discussed above.
There are interesting links be made between ALS and the RED algorithm (Romano al., 2017; Rechorst and Schniter, 2018). The RED algorithm attempts to find the maximum a posteriori probability (MAP) denoised sample (i.e., the most plausible real data) given noisy sample. It does so by solving an optimization problem to obtain a sample close to the noisy sample for which the EDS is a fixed point (denoising the sample does not change it because is a real sample). Thus, just like ALS, the RED algorithm generates plausible real data given score network. However, this algorithm does not ensure that we sample from the distribution and obtain full mode coverage. Thus, ALS's 's key benefit is ensuring that sample from the full support of the distribution.
3 CONSISTENT SCALING OF THE NOISE
In this section, we present inconsistencies in ALS relating to the noise scaling and introduce Consistent Annealed Sampling (CAS) as alternative.
3

---

3.1 INCONSISTENCIES ALS
One can think of the ALS algorithm as sequential series of Langevin Dynamics (inner loop in Algorithm 1) for decreasing levels noise (outer loop). If allowed an infinite number of steps ng. the sampling process will properly produce samples from the data distribution.
In ALS, the score network conditioned on geometrically decreasing noise (at). the unconditional case, this corresponds to dividing the score network by the noise level (i.e., s0 0;) so (a:)) Thus, in both conditional and unconditional cases, we make the assumption that the noise of the sample at step will be of variance of, assumption upon which the quality of the estimation of the score depends. While choosing a geometric progression of noise levels seems like a reasonable (though arbitrary) schedule follow, we show that ALS does not ensure such schedule.
Assume we have the true score function 8* and begin sampling using a real image with some added zero-centered Gaussian noise of standard deviation do 50. In Figure la, illustrate how the intensity of the noise in the sample evolves through ALS and CAS, our proposed sampling, for a given sampling step size and geometric schedule in this idealized scenario. We note that, although a large nhe approaches the real geometric curve, it will only reach it at the limit (n, + oo and e - 0). Most importantly, Figure 1b highlights how even when the annealing process does converge, the progression of the noise is never truly geometric; we prove this formally in Proposition 1.
50
A.5
Standard Deviation Difference
ALS
Standard Deviation
40
fu-55
40
30
20
10
10
0
0
1/4
1/2
31. /4
L/4
L/2
3L
Sampling step
Sampling step
(a) Standard deviation of the noise in the image
(b) Difference between the standard deviation from ALS and CAS
Figure I: Standard deviation during idealized sampling using perfect score function s The black
curve in (a) corresponds to the true geometric progression, as demonstrated in Proposition 2.
Proposition Let be the optimal score function from Eq. 3. Following the sampling described in Algorithm 1, the variance of the noise component in the sample will remain greater than of at every step t.
The proof is presented Appendix In particular, for n. oc, sampling has not fully converged and the remaining noise is carried over to the next iteration of Langevin Sampling. It also follows that for any 89 different from the optimal s", the actual noise every iteration expected to be even higher than for the best possible score function s"
3.2 ALGORITHM
We propose Consistent Annealed Sampling (CAS) as sampling method that ensures the noise level will follow a prescribed schedule for any sampling step size c and number of steps L. Algorithm illustrates the process for geometric schedule. Note that for different schedule, 3 will instead depend on the step I, as in the general case, 7, is defined as 0/41/01-
Proposition 2. Let s* be the optimal score function from Eq. 3. Following the sampling described in Algorithm 2, the variance of the noise component in the sample will consistently be equal of at every step I.
The proof is presented in Appendix G. Importantly, Proposition 2 holds no matter how many steps I take to decrease the noise geometrically. For ALS, n, corresponds to the number of steps
4

---

per level of noise. It plays similar role in CAS: we simply dilate the geometric series of noise levels used during training by a factor of ne, such that sampling = (Lurining 1)n + 1. Note that the proposition only holds when the initial sample is comrpрted image (i.e.. To I + og However, by defining do as the maximum Fuclidean distance between all pairs of training data points (Song and Ermon, 2020), the noise becomes in practice much greater than the true image; sampling
with pure noise initialization (i.e., Io = oozz) becomes indistinguishable from sampling with data initialization.
BENEFITS OF THE EDS ON SYNTHETIC DATA AND IMAGE GENERATION
As previously mentioned, has been suggested that one can obtain better samples (closer to the assumed data manifold) by taking the EDS of the last Langevin sample. We provide further evidence of this with synthetic data and standard image datasets.
It can first be observed that the sampling steps correspond to interpolation between the previous point and the EDS, followed by the addition of noise.
Proposition Given a coise-conditional score function, the update rules from Algorithm and Algorithm 2 are respectively equivalent to the following update rules:
- (1 1) (ax + 21oo,
for 2 ~ N(O, I) and 6 of
x x +1 1 71 (2, (2) + B0
The demonstration in Appendix H. This result is equally true for an unconditional score network, with the distinction that 1) would longer be independent of a; but rather linearly proportional to it.
Intuitively, this implies that the sampling steps slowly move the current sample towards a moving target (the EDS). If the sampling behaves appropriately, expect the final sample to be very close to the EDS, ie.,, xr Rs (xx, a..)- However, if the sampling step size is inappropriate, or if the EDS does not stabilize far from one another. As we will show, the FIDs from Song and Ermon (2020) suffer from such distance.
fixed point the end of the sampling, these two quantities may be arbitrarily
From Proposition 3, we see that CAS shares some similarities with the algorithm by Kadkhodaie and Simoncelli (2020). While the weight we give to the denoiser (1) decreases geometrically (by its linearity in σ), their schedule appears to be much steeper. They also estimate the residual noise in their samples by the la norm instead of determining through a schedule, as CAS strives to do. note, we had found weak evidence during development that estimating the residual noise worsened the FID.
The equivalence showed in Proposition 3 suggests instead to take the expected denoised sample at the end the Langevin sampling as the final sample; this would be equivalent to the update rule +- (x= σ at the last step. Synthetic 2D examples shown in Figure 2 demonstrate the immediate benefits of this technique.
Final samples
Denoised samples
Final samples
Denoised samples
(a) Swiss Roll dataset
(b) 25 Gaussians dataset
Figure 2: Langevin sampling synthetic 2D experiments. Circles are real data points, crosses generated data points. On both datasets, taking the EDS brings the samples much closer to the real data manifold.
5

---

ALS Sampling Step Size (E)
1.0
2.0
3.0
4.0
5,0
Partial (5/000 samples)
70
ALS non de-dennois sed denoseed
60
50
ALS denotsed CAS denoised
40
30
20
10
0.5
1.0
CAS Sampling Step Size
1e-5
Figure 3: Partial estimate of FID (lower is better) as function of the sampling step size CIFAR-10, with n, = 1. The interactions between consistent sampling and denoising are shown.
We train a score network on CIFAR-10 (Krizhevsky et al., 2009) and report the FID from both ALS and CAS function of the sampling step size and of denoising in Figure 3. The first observation to be made is just how critical denoising is to the FID score for ALS, even as its effect cannot be perceived by the human eye. For CAS, we note that the score remains small for much wider range of sampling step sizes when denoising. Alternatively, the sampling step size must be very carefully tuned obtain results close the optimal.
Figure also shows that, with CAS, the FID of the final sample is approximately equal to the FID of the denoised samples for small sampling step sizes. Furthermore, we see a smaller gap in FID between denoised and non-denoised for larger sampling step sizes than ALS. This suggests that consistent sampling is resulting the final sample being closer to the assumed data manifold (i.e., Ho L.).
Interestingly, when Song and Ermon (2020) improved their score matching method, they could not explain why the FID of their new model did not improve even though the generated images looked better visually. To resolve that matter, they proposed the use a new metric (Zhou et al., 2019) that did not have this issue. As shown in Figure 3, denoising resolves this mismatch.
5 ADVERSARIAL FORMULATION
The score network is trained to recover an uncorrupted image from a noisy input minimizing the l2 distance between the two. However, is well known from the image restoration literature that l2 does not correlate well with human perception of image quality (Zhang et al., 2012: Zhao et al., 2016). One way to take advantage of the EDS would be to encourage the score network to produce an EDS that is more realistic from the perspective of discriminator. Intuitively, this would incentivize the score network to produce more discernible features at inference time.
We propose to do so by training the score network simultaneously minimize the score-matching loss function and maximize the probability of denoised samples being perceived as real by a discriminator. We use alternating gradient descent to sequentially train discriminator for determined number of steps at every score function update.
In our experiments, we selected the Least Squares GAN (LSGAN) (Mao et al., 2017) formulation as it performed best (see Appendix B for details). For an unconditional score network, the objective functions are follows:
max Ep(a) [()(()) - 1)2] + Ep(z,,,) (Do o) 1)
(4)
min B,(ā, (Do(Ho(ã,o)) 1)2 A
x x
0s 5. 0) +
(5)

---

where Hz( "o( E is the EDS derived from the score network. Eq. 4 is the objective function the LSGAN discriminator, while Eq. 5 the adversarial objective function of the score network derived from Eq. and from the LSGAN objective function.
We note the similarities between these objective functions and those of an LSGAN adversarial autoencoder (Makhzani al., 2015; Tolstikhin et al., 2017; Tran et al., 2018), with the distinction of using a denoising autoencoder as opposed to a standard autoencoder. We can highlight this difference by reformulating Eq. 5 as:
X (Do(Ho((, - 1)² + 2a Ho(
min E P.
20
(6)
As GANs favor quality over diversity, there is concern that this hybrid objective function might decrease the diversity of samples produced by the ALS. Section 6.1, we first study image generation improvements brought by this method and then address the diversity concerns with experiments on the 3-StackedMNIST (Metz al., 2016) dataset Section 6.2.
EXPERIMENTS
6.1 ABLATION STUDY
We ran experiments CIFAR-10 (Krizhevsky et al., 2009) and LSUN-churches (Yu al., 2015) with the score network architecture used by Song and Ermon (2020). We also ran similar experiments with an unconditional version of the network architecture by Ho et al. (2020), given that their approach similar to Song and Ermon (2019) and they obtain very small FIDs. For the hybrid adversarial score matching approach, we used an unconditional BigGAN discriminator (Brock et al., 2018). We compared three factors an ablation study: adversarial training, Consistent Annealed Sampling and denoising.
Details on how the experiments were conducted are found in Appendix B. Unsuccessful experiments with large images also discussed in Appendix C. See also Appendix for discussion pertaining to the use of the Inception Score (Heusel et al., 2017), popular metric for generative models.
Results for CIFAR-10 and LSUN-churches with Song and Ermon (2019) score network architecture are respectively shown in Table and 2. Results for CIFAR-10 with Ho et al. (2020) score network architecture shown in Table 3.
Table 1: [Non-denoised Denoised FID) from 10k samples on CIFAR-10 (32x32) with Song and Ermon (2019) score network architecture
Sampling
Non-adversarial
Adversarial
non-consistent (n,
36.3 13.3
30.0/ 11.8 26.4 / 9.5
non-consistent (no = 5)
33.7 10.9
consistent (ng = 1)
14.7 12.3
11.9 10.8
consistent (ng 5)
12.7 11.2
9.9 /9.7
Table 2: [Non-denoised Denoised FID) from 10k samples on LSUN-Churches (64x64) with Song and Ermon (2019) score network architecture
Sampling
Non-adversarial
Adversarial
non-consistent (n, = 43.2 / 40.3
40.9 / 36.7
non-consistent (no 5)
42.0/39.2
40.0 35.8
consistent (ng 1)
41.5/ 40.7
38.2 / 36.7
consistent (ngg 5)
39.5 / 39.1
36.3 35.4
We always observe an improvement in FID from denoising and by increasing n., from to 5. We observe improvement from using the adversarial approach with Song and Ermon (2019) network
7

---

Table 3: [Non-denoised Denoised FID] from 10k samples on CIFAR-IO (32x32) with Ho al.
(2020) unconditional score network architecture
Sampling
Non-adversarial
Adversarial
non-consistent (n, 1)
25.3 7.5
21.6/7.5
non-consistent (N, = 5)
20.0 5.6
17.7 /.1
consistent (ng = 1)
7.8 7.17.11
7.7 /7.1
consistent (No 5)
6.2 /.2/6. 6.1
6.1
architecture, but denoised samples with the Ho al. (2020) network architecture. We hypothesize that this is limitation of the architecture of the discriminator since, as far as we know, no variant of BigGAN achieves an FID smaller than 6. Nevertheless, remains advantageous for more simple architectures, as shown in Table and 2. We observe that consistent sampling outperforms non-consistent sampling on the CIFAR- 10 task at n. 1, the quickest way to sample.
We calculated the FID of the non-consistent denoised models from 50k samples in order to compare our method with the recent work from Ho et al. (2020). adversarial method and 4.02 for the adversarial method on the CIFAR-10 task when sharing their architecture; these scores are close to their reported 3.17. Although not explicit in their approach, Ho al. (2020) denoised their final sample. This suggests that taking the EDS and using architecture akin to theirs were the two main reasons for outperforming Song and Ermon (2020). Of note, our method only trains the score network for 300k iterations, while Ho et al. (2020) trained their networks for more than million iterations to achieve similar results.
obtained score 3.65 for the non-
6.2 NON-ADVERSARIAL AND ADVERSARIAL SCORE NETWORKS HAVE EQUALLY HIGH DIVERSITY
To assess the diversity of generated samples, evaluate models the 3-Stacked MNIST generation task (Metz et al., 2016) (128k images of 28x28), consisting of numbers from the MNIST dataset (LeCun et al., 1998) superimposed on 3 different channels. We trained non-adversarial and adversarial score networks in the same way as the other models. The results are shown in Table 4.
We that each of the 1000 modes is covered, though the KL divergence is still inferior to PACGAN (Lin et al., 2018), 1), meaning that the mode proportions are not perfectly uniform. Blindness to mode proportions is thought to be fundamental limitation of score-based methods (Wenliang, 2020). Nevertheless, these results confirm a full mode coverage on task where most GANs struggle and, most importantly, that using a hybrid objective does not hurt the diversity of the generated samples.
3-Stacked MNIST
Modes (Max 1000)
KL
DCGAN (Radford et al., 2015h)
99.0
3.40
ALI (Dumoulin et al., 2016)
16.0
5.40
Unrolled GAN (Metz et al., 2016)
48.7
4.32
VEEGAN (Srivastava et al., 2017)
150.0
2.95
PacDCGAN2 (Lin et al., 2017b)
1000.0
0.06
WGAN-GP (Kumar et al., 2019; Gulrajani et al., 2017)
959.0
0.73
PresGAN (Dieng et al., 2019)
999.6
0.115
MEG (Kumar et al., 2019)
1000.0
0.03
Non-adversarial DSM (ours)
1000.0
1.36
Adversarial DSM (ours)
1000.0
1.49
Table 4: As in Lin et al. (2018), generated 26k samples and evaluated the mode coverage and KL divergence based on the predicted modes from a pre-trained MNIST classifier.
8

---

7 CONCLUSION
We proposed Consistent Annealed Sampling as an alternative to Annealed Langevin Sampling, which ensures the expected geometric progression of the noise and brings the final samples closer to the data manifold. We showed how to extract the expected denoised sample and how to use it to further improve the final Langevin samples. We proposed hybrid approach between GAN and score matching. With experiments synthetic and standard image datasets; we showed that these approaches generally improved the quality/diversity
the generated samples.
We found equal diversity (coverage of all 1000 modes) for the adversarial and non-adversarial variant of the difficult StackedMNIST problem. Since we also observed better performance (from lower FIDs) in other adversarial models trained images, we conclude that making score matching adversarial increases the quality of the samples without decreasing diversity. These findings imply that score matching performs better than most GANs and on-par with state-of-the-art GANs. Furthermore, our results suggest that hybrid methods, combining multiple generative techniques together, are very promising direction to pursuc.
As future work, these models should be scaled to larger batch sizes on high-resolution images, since GANs have been shown to produce outstanding high-resolution images very large batch sizes (2048 or more). We also plan to further study the theoretical properties of CAS by considering its corresponding stochastic differential equation.
REFERENCES
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pages 11918-11930, 2019.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
Pascal Vincent. A connection between matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674 2011.
Martin Raphan and Eero P Simoncelli. Least squares estimation without priors supervision. Neural computation, 23(2):374 2011.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688, 2011.
Gareth O Roberts, Richard I. Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363 1996.
Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.136600, 2020.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint wrXiv:2006.09//1 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626 6637, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672-26022 2680. Curran Associates, Inc., 2014. URL http: URLhttp:///pppsss papers.. adweeeariae-aaeeeeeeeeee pdf.
nips cc/ happp/apppp//aaa 5423 gere-rgeerrrttteee
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1809 1096, 2018.


Reviews:
[{"review_id": "riTs2f9tvLu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article deals with generative models based on “Annealed Langevin Sampling“ rather than a GAN.\nTheses models suffer from worse FID than GAN.\nAuthors proposed to denoise the last Langevin samples to reduce the gap in performance with Adversarial Network.\nThe paper is really easy to read with good illustrations and supporting experiments.\n\nIn order to gain in comprehension, especially for people new to ALS, it would have been great if authors have proposed an illustration (and comparison) of the samples evolution along Alg 1 and Alg 2 .\n\nAuthors are honest in their revised results comments but I don’t known if they will be able to include the erratum  in a final version\n\nAs I was not aware before this review of “Annealed Langevin Sampling” my rating may not be confident.", "extra_scores": {}, "extra_text": {}}, {"review_id": "nm9rM6L2aQ8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of generative modeling by using Langevin dynamics to sample from the denoising score function. Recently, this family of approaches (Song and Ermon 2019, Song and Ermon 2020) has shown promising and competitive results being positioned as a potential alternative to GANs. \n\nThe paper introduces different improvements over Song and Ermon (2020). A different sampling dynamic (Consistent Annealed Sampling) that produces a more stable training that the traditional annealing scheme by carefully scaling the injected noise. Second, it is empirically shown that running a denoising step on the generated sample leads to an improvement of the FID score. Based on this observation, the paper proposes to use a denoiser trained in an adversarial fashion to synthesize more realistic images.\n\nThe work addresses the very relevant problem of how to synthesize images in a realistic way, introducing some modifications to existing works that lead to an improvement on the quality of the generated image.  The paper is well written, presents a nice introduction to the method, which allows to motivate the different modifications in a natural way. The proposed modifications are analyzed in low-dimensional toy experiments and in small-scale images (CIFAR, LSUN-churchers, Stacked-MNIST).\n\nIn what follows I list a few questions: \n\n1. Would it be possible to analyze the strategy of sampling presented in Kadkhodaei and Simoncelli 2020 (concurrent work), and compare to the one proposed in the paper? Both strategies seem to improve the stabilization of the procedure by scaling the noise.\n\n2. Regarding the step of applying the denoiser to the generated sample. I wonder what happens if the denoiser is re-applied? Also, is this connected to the fact that the denoiser may have a fixed point and this fixed point might lead to a better sample? \n\n3. Regarding using an adversarial denoising. In the denoising literature, there are a few works connecting score matching and state-of-the-art image denoisers. I would like to see a better discussion of this. For example, see,\n\nRomano, Y., Elad, M. and Milanfar, P., 2017. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences, 10(4), pp.1804-1844.\n\nReehorst, E.T. and Schniter, P., 2018. Regularization by denoising: Clarifications and new interpretations. IEEE transactions on computational imaging, 5(1), pp.52-67.\n\n---\nAfter Discussion:\nI think this is a good paper and I would like to see it presented at ICLR2021. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HImEw6MsX-e", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission presents three contributions. First, the authors show the inconsistencies in the existing annealed Langevin sampling used in score-matching generative models and propose to correct it with the newly proposed Consistent Annealed Sampling (CAS) algorithm. The second contribution claimed is in providing evidence of the benefits of Expected Denoised Sample (EDS). Furthermore, the submission introduces a hybrid adversarial score-matching model that demonstrates improvements in terms of FID on simpler architectures.\n\nThe proposed CAS algorithm is theoretically well-motivated based on the observation that ALS is inconsistent with the scaling of the noise during sampling process (although the question whether noise should follow none other than geometric progression is still an open question). The paper is well-written, and the ablation study is carried out well.\n\nHowever, it is a bit confusing as to whether the EDS (although under a different name — denoising jump) is a contribution of this paper or is something proposed prior to this work. I understand that this denoising procedure has already been presented as a necessary technique in score matching models. Nevertheless, I believe the authors contributed by showing that both ALS and CAS move samples towards the EDS (Proposition 3) and show additional empirical evidence of its benefits on synthetic and real datasets.\n\nTaking EDS on the last Langevin step diminishes the impact of CAS (doesn't bring unambiguous improvement in FID scores in the experiments), otherwise very interesting finding both theoretically and algorithmically, and substitute for ALS.\n\nThe effect of the hybrid model is also not persistent and depends on the architecture used. For an incremental improvement (a combination of two models), the improvement is not consistent across architectures. The paper does not explain whether there is a good rationale for such a combination; therefore I remain sceptical about the results.\n\nGiven all the above, I am still leaning a bit towards accepting the paper as it covers an interesting finding relating to the ALS. Although the CAS effect on performance is limited by the EDS, score-matching models are of broad interest for the ICLR community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9aYI_d7BOrN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel approach for denoising score matching, where the Annealed Langevin Sampling has been substituted by Consistent Annealed Sampling, which adds more stability to the process.\n\nThe paper is in general clear and well-written. The contributions are clearly highlighted and the proposed approach is conveniently compared with other state of the art methods, demonstrating its superiority.\n\nPositive aspects:\n- The Consistent Annealed Sampling proposed in this paper is more stable than the Annealed Langevin Sampling\n- The combination between GAN and score matching improves the quality/diversity of the generated sample\n\nNegative aspects: \n- The limitation of the method to Gaussian noise\n- The presentation of a real scenario for your approach would have been a plus\n\nHowever, I have some questions:\n1. Who is n_sigma parameter in Algorithm 1? \n2. Algorithm 1, line 4: there is no iteration over 't' in the loop?\n3. How does your denoising scheme work? Do you create noisy samples from your real data and try to denoise them using the proposed approach? Because taking a sample affected by random noise (in the test phase) I guess it won't work. \n4. The denoising scheme is used in a GAN framework, the denoised samples being perceived as real by the discriminator. Is the system trained end-to-end or first you denoise the image and afterwards you train the GAN?\n5. Could you please indicate an application scenario which could benefit from this approach, e.g. image-to-image translation, domain adaptation, etc.?\n6. Your method is assuming Gaussian noise. Can it be extended to the case of general noise (a noise model which could be also learnt)?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an alternative to Langevin sampling and also the idea of adversarial score sampling. 
The reviewers are generally supportive of the paper.

Pros:
- The idea behind improving Langevin sampling is theoretically justified and leads to a simple algorithm. 
- The idea behind adversarial score matching is also shown to be effective 
- Improvement over baseline

Cons: 
- Two ideas packed  into one paper, which is reflected by the title as well. 
-  From the narrative it could be thought that using EDS on the last step of CAS is the contribution of the paper. 
Decision: Accept (Poster)