Adversarial score matching and improved sampling for image generation

1 INTRODUCTION . Song and Ermon ( 2019 ) recently proposed a novel method of generating samples from a target distribution through a combination of Denoising Score Matching ( DSM ) ( Hyvärinen , 2005 ; Vincent , 2011 ; Raphan and Simoncelli , 2011 ) and Annealed Langevin Sampling ( ALS ) ( Welling and Teh , 2011 ; Roberts et al. , 1996 ) . Since convergence to the distribution is guaranteed by the ALS , their approach ( DSM-ALS ) produces high-quality samples and guarantees high diversity . Though , this comes at the cost of requiring an iterative process during sampling , contrary to other generative methods . These generative methods can notably be used to diverse tasks like colorization , image restoration and image inpainting ( Song and Ermon , 2019 ; Kadkhodaie and Simoncelli , 2020 ) . Song and Ermon ( 2020 ) further improved their approach by increasing the stability of score matching training and proposing theoretically sound choices of hyperparameters . They also scaled their approach to higher-resolution images and showed that DSM-ALS is competitive with other generative models . Song and Ermon ( 2020 ) observed that the images produced by their improved model were more visually appealing than the ones from their original work ; however , the reported Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) did not correlate with this improvement . Although DSM-ALS is gaining traction , Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) remain the leading approach to generative modeling . GANs are a very popular class of generative models ; they have been successfully applied to image generation ( Brock et al. , 2018 ; Karras et al. , 2017 ; 2019 ; 2020 ) and have subsequently spawned a wealth of variants ( Radford et al. , 2015a ; Miyato et al. , 2018 ; Jolicoeur-Martineau , 2018 ; Zhang et al. , 2019 ) . The idea behind this method is to train a Discriminator ( D ) to correctly distinguish real samples from fake samples generated by a second agent , known as the Generator ( G ) . GANs excel at generating high-quality samples as the discriminator captures features that make an image plausible , while the generator learns to emulate them . Still , GANs often have trouble producing data from all possible modes , which limits the diversity of the generated samples . A wide variety of tricks have been developed to address this issue in GANs ( Kodali et al. , 2017 ; Gulrajani et al. , 2017 ; Arjovsky et al. , 2017 ; Miyato et al. , 2018 ; JolicoeurMartineau and Mitliagkas , 2019 ) , though it remains an issue to this day . DSM-ALS , on the other hand , does not suffer from that problem since ALS allows for sampling from the full distribution captured by the score network . Nevertheless , the perceptual quality of DSM-ALS higher-resolution images has so far been inferior to that of GAN-generated images . Generative modeling has since seen some incredible work from Ho et al . ( 2020 ) , who achieved exceptionally low ( better ) FID on image generation tasks . Their approach showcased a diffusion-based method ( Sohl-Dickstein et al. , 2015 ; Goyal et al. , 2017 ) that shares close ties with DSM-ALS , and additionally proposed a convincing network architecture derived from Salimans et al . ( 2017 ) . In this paper , after introducing the necessary technical background in the next section , we build upon the work of Song and Ermon ( 2020 ) and propose improvements based on theoretical analyses both at training and sampling time . Our contributions are as follows : • We propose Consistent Annealed Sampling ( CAS ) as a more stable alternative to ALS , correcting inconsistencies relating to the scaling of the added noise ; • We show how to recover the expected denoised sample ( EDS ) and demonstrate its unequiv- ocal benefits w.r.t the FID . Notably , we show how to resolve the mismatch observed in DSM-ALS between the visual quality of generated images and its high ( worse ) FID ; • We propose to further exploit the EDS through a hybrid objective function , combining GAN and Denoising Score Matching objectives , thereby encouraging the EDS of the score network to be as realistic as possible . In addition , we show that the network architecture used used by Ho et al . ( 2020 ) significantly improves sample quality over the RefineNet ( Lin et al. , 2017a ) architecture used by Song and Ermon ( 2020 ) . In an ablation study performed on CIFAR-10 and LSUN-church , we demonstrate how these contributions bring DSM-ALS in range of the state-of-the-art for image generation tasks w.r.t . the FID . The code to replicate our experiments is publicly available at [ Available in supplementary material ] . 2 BACKGROUND . 2.1 DENOISING SCORE MATCHING . Denoising Score Matching ( DSM ) ( Hyvärinen , 2005 ) consists of training a score network to approximate the gradient of the log density of a certain distribution ( ∇x log p ( x ) ) , referred to as the score function . This is achieved by training the network to approximate a noisy surrogate of p at multiple levels of Gaussian noise corruption ( Vincent , 2011 ) . The score network s , parametrized by θ and conditioned on the noise level σ , is tasked to minimize the following loss : 1 2 Ep ( x̃ , x , σ ) [ ∥∥∥∥σsθ ( x̃ , σ ) + x̃− xσ ∥∥∥∥2 2 ] , ( 1 ) where p ( x̃ , x , σ ) = qσ ( x̃|x ) p ( x ) p ( σ ) . We define further qσ ( x̃|x ) = N ( x̃|x , σ2I ) the corrupted data distribution , p ( x ) the training data distribution , and p ( σ ) the uniform distribution over a set { σi } corresponding to different levels of noise . In practice , this set is defined as a geometric progression between σ1 and σL ( with L chosen according to some computational budget ) : { σi } Li=1 = { γiσ1 ∣∣∣ i ∈ { 0 , . . . , L− 1 } , γ , σ2 σ1 = ... = ( σL σ1 ) 1 L−1 < 1 } . ( 2 ) Rather than having to learn a different score function for every σi , one can train an unconditional score network by defining sθ ( x̃ , σi ) = sθ ( x̃ ) /σi , and then minimizing Eq . 1 . While unconditional networks are less heavy computationally , it remains an open question whether conditioning helps performance . Li et al . ( 2019 ) and Song and Ermon ( 2020 ) found that the unconditional network produced better samples , while Ho et al . ( 2020 ) obtained better results than both of them using a conditional network . Additionally , the denoising autoencoder described in Lim et al . ( 2020 ) gives evidence supporting the benefits of conditioning when the noise becomes small ( also see App . D and E for a theoretical discussion of the difference ) . While our experiments are conducted with unconditional networks , we believe our techniques can be straightforwardly applied to conditional networks ; we leave that extension for future work . 2.2 ANNEALED LANGEVIN SAMPLING . Given a score function , one can use Langevin dynamics ( or Langevin sampling ) ( Welling and Teh , 2011 ) to sample from the corresponding probability distribution . In practice , the score function is generally unknown and estimated through a score network trained to minimize Eq . 1 . Song and Ermon ( 2019 ) showed that Langevin sampling has trouble exploring the full support of the distribution when the modes are too far apart and proposed Annealed Langevin Sampling ( ALS ) as a solution . ALS starts sampling with a large noise level and progressively anneals it down to a value close to 0 , ensuring both proper mode coverage and convergence to the data distribution . Its precise description is shown in Algorithm 1 . Algorithm 1 Annealed Langevin Sampling Require : sθ , { σi } Li=1 , , nσ . 1 : Initialize x 2 : for i← 1 to L do 3 : αi ← σ2i /σ2L 4 : for nσ steps do 5 : Draw z ∼ N ( 0 , I ) 6 : x← x+ αisθ ( x , σi ) + √ 2αiz return x Algorithm 2 Consistent Annealed Sampling Require : sθ , { σi } Li=1 , γ , , σL+1 = 0 1 : Initialize x 2 : β ← √ 1− ( 1− /σ2L ) 2 /γ2 3 : for i← 1 to L do 4 : αi ← σ2i /σ2L 5 : Draw z ∼ N ( 0 , I ) 6 : x← x+ αisθ ( x , σi ) + βσi+1z return x 2.3 EXPECTED DENOISED SAMPLE ( EDS ) . A little known fact from Bayesian literature is that one can recover a denoised sample from the score function using the Empirical Bayes mean ( Robbins , 1955 ; Miyasawa , 1961 ; Raphan and Simoncelli , 2011 ) : s∗ ( x̃ , σ ) = H∗ ( x̃ , σ ) − x̃ σ2 , ( 3 ) whereH∗ ( x̃ , σ ) , Ex∼qσ ( x|x̃ ) [ x ] is the expected denoised sample given a noisy sample ( or Empirical Bayes mean ) , conditioned on the noise level . A different way of reaching the same result is through the closed-form of the optimal score function , as presented in Appendix D. The corresponding result for unconditional score function is presented in Appendix E for completeness . The EDS corresponds to the expected real image given a corrupted image ; it can be thought of as what the score network believes to be the true image concealed within the noisy input . It has also been suggested that denoising the samples ( i.e. , taking the EDS ) at the end of the Langevin sampling improves their quality ( Saremi and Hyvarinen , 2019 ; Li et al. , 2019 ; Kadkhodaie and Simoncelli , 2020 ) . In Section 4 , we provide further evidence that denoising the final Langevin sample brings it closer to the assumed data manifold . In particular , we show that the Fréchet Inception Distance ( FID ) consistently decreases ( improves ) after denoising . Finally , in Section 5 , we build a hybrid training objective using the properties of the EDS discussed above . There are interesting links to be made between ALS and the RED algorithm ( Romano et al. , 2017 ; Reehorst and Schniter , 2018 ) . The RED algorithm attempts to find the maximum a posteriori probability ( MAP ) denoised sample ( i.e. , the most plausible real data ) given a noisy sample . It does so by solving an optimization problem to obtain a sample close to the noisy sample for which the EDS is a fixed point ( denoising the sample does not change it because it is a real sample ) . Thus , just like ALS , the RED algorithm generates plausible real data given a score network . However , this algorithm does not ensure that we sample from the distribution and obtain full mode coverage . Thus , ALS ’ s key benefit is ensuring that we sample from the full support of the distribution .

Summary: The submission presents three contributions. First, the authors show the inconsistencies in the existing annealed Langevin sampling used in score-matching generative models and propose to correct it with the newly proposed Consistent Annealed Sampling (CAS) algorithm. The second contribution claimed is in providing evidence of the benefits of Expected Denoised Sample (EDS). Furthermore, the submission introduces a hybrid adversarial score-matching model that demonstrates improvements in terms of FID on simpler architectures.