Consistency Regularization for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have recently demonstrated impressive results on image-synthesis benchmarks ( Radford et al. , 2016 ; Zhang et al. , 2017 ; Miyato & Koyama , 2018 ; Zhang et al. , 2018 ; Brock et al. , 2018 ; Karras et al. , 2019 ) . In the original setting , GANs are composed of two neural networks trained with competing goals : the generator is trained to synthesize realistic samples to fool the discriminator and the discriminator is trained to distinguish real samples from fake ones produced by the generator . One major problem with GANs is the instability of the training procedure and the general sensitivity of the results to various hyperparameters ( Salimans et al. , 2016 ) . Because GAN training implicitly requires finding the Nash equilibrium of a non-convex game in a continuous and high dimensional parameter space , it is substantially more complicated than standard neural network training . In fact , formally characterizing the convergence properties of the GAN training procedure is mostly an open problem ( Odena , 2019 ) . Previous work ( Arjovsky & Bottou , 2017 ; Miyato et al. , 2018a ; Odena et al. , 2017 ; Chen et al. , 2019 ; Wei et al. , 2018 ) has shown that interventions focused on the discriminator can mitigate stability issues . Most successful interventions fall into two categories , normalization and regularization . Spectral normalization is the most effective normalization method , in which weight matrices in the discriminator are divided by an approximation of their largest singular value . For regularization , Gulrajani et al . ( 2017 ) penalize the gradient norm of straight lines between real data and generated data . Roth et al . ( 2017 ) propose to directly regularize the squared gradient norm for both the training data and the generated data . DRAGAN ( Kodali et al. , 2017 ) introduces another form of gradient penalty where the gradients at Gaussian perturbations of training data are penalized . One may anticipate simultaneous regularization and normalization could improve sample quality . However , most of these gradient based regularization methods either provide marginal gains or fail to introduce any improvement when normalization is used ( Kurach et al. , 2019 ) , which is also observed in our experiments . These regularization methods and spectral normalization are motivated by controlling Lipschitz constant of the discriminator . We suspect this might be the reason that applying both does not lead to overlaid gain . In this paper , we examine a technique called consistency regularization ( Bachman et al. , 2014 ; Sajjadi et al. , 2016 ; Laine & Aila , 2016 ; Zhai et al. , 2019 ; Xie et al. , 2019 ; Hu et al. , 2017 ) in contrast to gradient-based regularizers . Consistency regularization is widely used in semi-supervised learning to ensure that the classifier output remains unaffected for an unlabeled example even it is augmented in semantic-preserving ways . In light of this intuition , we hypothesize a well-trained discriminator should also be regularized to have the consistency property , which enforces the discriminator to be unchanged by arbitrary semantic-preserving perturbations and to focus more on semantic and structural changes between real and fake data . Therefore , we propose a simple regularizer to the discriminator of GAN : we augment images with semantic-preserving augmentations before they are fed into the GAN discriminator and penalize the sensitivity of the discriminator to those augmentations . This technique is simple to use and surprisingly effective . It is as well less computationally expensive than prior techniques . More importantly , in our experiments , consistency regularization can always further improve the model performance when spectral normalization is used , whereas the performance gains of previous regularization methods diminish in such case . In extensive ablation studies , we show that it works across a large range of GAN variants and datasets . We also show that simply applying this technique on top of existing GAN models leads to new state-of-the-art results as measured by Frechet Inception Distance ( Heusel et al. , 2017 ) . In summary , our contributions are summarized as follows : • We propose consistency regularization for GAN discriminators to yield a simple , effective regularizer with lower computational cost than gradient-based regularization methods . • We conduct extensive experiments with different GAN variants to demonstrate that our technique interacts effectively with spectral normalization . Our consistency regularized GAN ( CR-GAN ) achieves the best FID scores for unconditional image generation on both CIFAR-10 and CelebA . • We show that simply applying the proposed technique can further boost the performance of state-of-the-art GAN models . We improve FID scores for conditional image generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012 . 2 METHOD . 2.1 GANS . A GAN consists of a generator network and a discriminator network . The generator G takes a latent variable z ∼ p ( z ) sampled from a prior distribution and maps it to the observation space X . The discriminator D takes an observation x ∈ X and produces a decision output over possible observation sources ( either from G or from the empirical data distribution ) . In the standard GAN training procedure the generator G and the discriminator D are trained by minimizing the following objectives in an alternating fashion : LD = −Ex∼pdata [ logD ( x ) ] − Ez∼p ( z ) [ 1− logD ( G ( z ) ) ] , LG = −Ez∼p ( z ) [ logD ( G ( z ) ) ] , ( 1 ) where p ( z ) is usually a standard normal distribution . This formulation is originally proposed by Goodfellow et al . ( 2014 ) as non-saturating ( NS ) GAN . A significant amount of research has been done on modifying this formulation in order to improve the training process . A notable example is the hinge-loss version of the adversarial loss ( Lim & Ye , 2017 ; Tran et al. , 2017 ) : LD = −Ex∼pdata [ min ( 0 , −1 +D ( x ) ) ] − Ez∼p ( z ) [ min ( 0 , −1−D ( G ( z ) ) ) ] , LG = −Ez∼p ( z ) [ D ( G ( z ) ) ] . ( 2 ) Another commonly adopted GAN formulation is the Wassertein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) , in which the authors propose clipping the weights of the discriminator in an attempt to enforce that the GAN training procedure implicitly optimizes a bound on the Wassertein distance between the target distribution and the distribution given by the generator . The loss function of WGAN can be written as LD = −Ex∼pdata [ D ( x ) ] + Ez∼p ( z ) [ D ( G ( z ) ) ] , LG = −Ez∼p ( z ) [ D ( G ( z ) ) ] . ( 3 ) Subsequent work has refined this technique in several ways ( Gulrajani et al. , 2017 ; Miyato et al. , 2018a ; Zhang et al. , 2019 ) , and the current widely-used practice is to enforce spectral normalization ( Miyato et al. , 2018a ) on both the generator and the discriminator . 2.2 CONSISTENCY REGULARIZATION . Consistency regularization has emerged as a gold-standard technique ( Sajjadi et al. , 2016 ; Laine & Aila , 2016 ; Zhai et al. , 2019 ; Xie et al. , 2019 ; Oliver et al. , 2018 ; Berthelot et al. , 2019 ) for semisupervised learning on image data . The basic idea is simple : an input image is perturbed in some semantics-preserving ways and the sensitivity of the classifier to that perturbation is penalized . The perturbation can take many forms : it can be image flipping , or cropping , or adversarial attacks . The regularization form is either the mean-squared-error ( Sajjadi et al. , 2016 ; Laine & Aila , 2016 ) between the model ’ s output for a perturbed and non-perturbed input or the KL divergence ( Xie et al. , 2019 ; Miyato et al. , 2018b ) between the distribution over classes implied by the output logits . 2.3 CONSISTENCY REGULARIZATION FOR GANS . The goal of the discriminator in GANs is to distinguish real data from fake ones produced by the generator . The decision should be invariant to any valid domain-specific data augmentations . For example , in the image domain , the image being real or not should not change if we flip the image horizontally or translate the image by a few pixels . However , the discriminator in GANs does not guarantee this property explicitly . To resolve this , we propose a consistency regularization on the GAN discriminator during training . In practice , we randomly augment training images as they are passed to the discriminator and penalize the sensitivity of the discriminator to those augmentations . We use Dj ( x ) to denote the output vector before activation of the jth layer of the discriminator given input x. T ( x ) denotes a stochastic data augmentation function . This function can be linear or nonlinear , but aims to preserve the semantics of the input . Our proposed regularization is given by min D Lcr = min D n∑ j=m λj ∥∥Dj ( x ) −Dj ( T ( x ) ) ∥∥2 , ( 4 ) where j indexes the layers , m is the starting layer and n is the ending layer that consistency is enforced . λj is weight coefficient for jth layer and ‖·‖ denotes L2 norm of a given vector . This consistency regularization encourages the discriminator to produce the same output for a data point under various data augmentations . Algorithm 1 Consistency Regularized GAN ( CR-GAN ) . We use λ = 10 by default . Input : generator and discriminator parameters θG , θD , consistency regularization coefficient λ , Adam hyperparameters α , β1 , β2 , batch size M , number of discriminator iterations per generator iteration ND 1 : for number of training iterations do 2 : for t = 1 , ... , ND do 3 : for i = 1 , ... , M do 4 : Sample z ∼ p ( z ) , x ∼ pdata ( x ) 5 : Augment x to get T ( x ) 6 : L ( i ) cr ← ∥∥D ( x ) −D ( T ( x ) ) ∥∥2 7 : L ( i ) D ← D ( G ( z ) ) −D ( x ) 8 : end for 9 : θD ← Adam ( 1M ∑M i=1 ( L ( i ) D + λL ( i ) cr ) , α , β1 , β2 ) 10 : end for 11 : Sample a batch of latent variables { z ( i ) } Mi=1 ∼ p ( z ) 12 : θG ← Adam ( 1M ∑M i=1 ( −D ( G ( z ) ) ) , α , β1 , β2 ) 13 : end for In our experiments , we find that consistency regularization on the last layer of the discriminator before the activation function is sufficient . Lcr can be rewritten as Lcr = ∥∥D ( x ) −D ( T ( x ) ) ∥∥2 , ( 5 ) where from now on we will drop the layer index for brevity . This cost is added to the discriminator loss ( weighted by a hyper-parameter λ ) when updating the discriminator parameters . The generator update remains unchanged . Thus , the overall consistency regularized GAN ( CR-GAN ) objective is written as LcrD = LD + λLcr , L cr G = LG . ( 6 ) Our design of Lcr is general-purpose and thereby can work with any valid adversarial losses LG and LD for GANs ( See Section 2.1 for examples ) . Algorithm 1 illustrates the details of CR-GAN with Wassertein loss as an example . In contrast to previous regularizers , our method does not increase much overhead . The only extra computational cost comes from feeding an additional ( third ) image through the discriminator forward and backward when updating the discriminator parameters .

Summary: The paper presents a new regularization technique termed consistency regularization for training GANs. The idea is the following: the authors propose to penalize the sensitivity of the last layer of the discriminator to augmented images. This idea is simple yet efficient: it is easy to implement, a regularization term is gradient-free, and its computation is up to 1.8 times faster than standard gradient-based regularization techniques. The authors tested different augmentation techniques and concluded that simple ones behave better (e.g., shifting and flipping). The experimental results show an impressive gain in FID measure, renewing the current state-of-the-art score for class conditional image generation on CIFAR-10 dataset.