PA-GAN: Improving GAN Training by Progressive Augmentation

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are a recent development in the field of deep learning , that have attracted a lot of attention in the research community ( Radford et al. , 2016 ; Salimans et al. , 2016 ; Arjovsky et al. , 2017 ; Karras et al. , 2018 ) . GANs fall into the category of generative models , i.e . models that allow sampling of new data points from encoded high-dimensional data distributions , such as images . The GAN framework can be formulated as a competing game between the generator and the discriminator . Mathematically , training GANs requires solving a min-max problem . Since both the generator and the discriminator are typically parameterized as deep convolutional neural networks with millions of parameters , optimization is notoriously difficult in practice ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) . The difficulty lies in maintaining the healthy competition between the generator and discriminator . A commonly occurring problem arises when the discriminator overshoots , leading to escalated gradients and oscillatory GAN behaviour ( Mescheder et al. , 2018 ) . As a result the generator fails to learn the multimodal structure of the true distribution . Moreover , the supports of the data and model distributions typically lie on low dimensional manifolds and are often disjoint ( Arjovsky & Bottou , 2017 ) . Consequently , there exists a nearly trivial discriminator that can perfectly distinguish real data samples from synthetic ones . Once such a discriminator is produced , its loss quickly converges to zero and the gradients used for updating parameters of the generator become useless . In this work we introduce a new technique to overcome this problem - progressive augmentation of GANs ( PA-GAN ) - that helps to control the behaviour of the discriminator and thus improve the overall training stability . The key idea is to progressively augment the input of the discriminator network with auxiliary random variables , enlarging the sample space dimensionality , in order to gradually increase the discrimination task difficulty ( see Figure 1 ) . In doing so , the discriminator can be prevented from becoming over-confident , enabling continuous learning of the generator . As opposed to standard data augmentation techniques ( e.g . rotation , cropping , resizing ) , the proposed progressive augmentation does not directly modify the data samples , but rather is structurally appended to them . In particular , for the single level augmentation along with the data sample x the discriminator takes also as input the binary random variable s ∈ { 0 , 1 } . The class of the augmented sample ( x , s ) is then set based on the combination x with s , resulting in real and synthetic samples contained in both true and fake classes . This presents a more challenging task for the discriminator , as it needs to tell the real and synthetic samples apart and additionally understand the association rule . We can further increase the task difficulty of the discriminator by progressively augmenting its input space and enlarging the dimensionality of s during the course of training . We show that the proposed PA-GAN preserves the original GAN objective and is an outcome of a systematic derivation . In contrast to prior work ( Salimans et al. , 2016 ; Sønderby et al. , 2017 ; Arjovsky & Bottou , 2017 ) , it does not bias the optimality of the discriminator ( see Sec . 4 ) . Structurally augmenting the input sample space and mapping it to higher dimensions not only challenges the discrimination task , but , in addition , encourages the generator to explore various paths towards the data distribution , leading to improved variation of the generated samples ( see Sec . 5 ) . Our technique is orthogonal to existing work , it can be successfully employed with other regularizations strategies ( Miyato et al. , 2018 ; Gulrajani et al. , 2017 ) and different network architectures ( Chen et al. , 2016 ; Radford et al. , 2016 ) , which we demonstrate in Sec . 5 . We experimentally show the effectiveness of the progressive augmentation of GANs for image generation tasks on multiple benchmarks ( MNIST ( LeCun et al. , 1998 ) , Fashion-MNIST ( Xiao et al. , 2017 ) , CIFAR10 ( Krizhevsky , 2009 ) , CELEBA ( Liu et al. , 2015 ) ) across different evaluation metrics ( IS ( Theis et al. , 2016 ) , FID ( Huszár , 2015 ) , KID ( Bińkowski et al. , 2018 ) ) . 2 RELATED WORK . Many recent works have focused on improving the stability of GAN training and the overall visual quality of generated samples ( Radford et al. , 2016 ; Roth et al. , 2017 ; Karras et al. , 2018 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) . As reported by Arjovsky & Bottou ( 2017 ) , the reason for the unstable behaviour of GANs is partly due to a dimensional mismatch or non-overlapping support between the real data and the generative model distributions , resulting in an almost trivial task for the discriminator . Once the performance of the discriminator is maxed out , it provides a non-informative signal to train the generator . To avoid vanishing gradients , the original GAN paper ( Goodfellow et al. , 2014 ) proposed to modify the min-max based GAN objective ( MM GAN ) to a non-saturating loss ( NS GAN ) . However , even with such a re-formulation the generator updates tend to get worse over the course of training and optimization becomes massively unstable ( Arjovsky & Bottou , 2017 ) . Prior approaches tried to mitigate this issue by using heuristics to weaken the discriminator , such as decreasing its learning rate , adding label or input noise . Salimans et al . ( 2016 ) proposed a one-sided label smoothing technique to smoothen the classification boundary of the discriminator , thereby preventing it from being overly confident , but at the same time biasing its optimality . The works of Arjovsky & Bottou ( 2017 ) and Sønderby et al . ( 2017 ) made the job of the discriminator harder by adding Gaussian noise to both generated and real samples . Moving the manifolds of the data and model distributions closer to each other by adding the input noise ensures a meaningful overlap between their supports , which is desired in order for the generator to eventually approach the data distribution . However , adding high-dimensional noise introduces significant variance in the parameter estimation , which slows down the training and requires multiple samples for counteraction ( Roth et al. , 2017 ) . Similarly , Sajjadi et al . ( 2018 ) proposed to blur the input samples and gradually remove the blurring effect during the course of training . Instead of adding noise to the input , Zhang et al . ( 2018b ) created mixup samples by interpolating between synthetic and real ones , which leads to a more stable behaviour of GANs . These techniques , i.e. , additive noise , blurring and sample mixup , perform direct modifications on the data samples . Another line of work resorts to cost function reformulation to improve the stability of GAN training , e.g . by using the Pearson χ2 divergence for least square GANs ( LS GANs ) ( Mao et al. , 2016 ) , kernel maximum mean discrepancy ( MMD ) for MMD-GANs ( Li et al. , 2017 ; Dziugaite et al. , 2015 ) , or f-divergence for f-GANs ( Nowozin et al. , 2016 ) . Arjovsky et al . ( 2017 ) proposed the Wasserstein GAN ( WGAN ) with the training objective derived from the Wasserstein distance , aiming to mitigate the vanishing gradient problem . The drawback of this approach is the weight clipping of the discriminator employed to enforce smoothness , which adversely reduces the capacity of the discriminator . Alternative to weight clipping , Gulrajani et al . ( 2017 ) added a soft penalty on the gradient norm which ensures a 1-Lipschitz discriminator . The gradient norm penalty can be seen as a weight regularization technique for the discriminator and was shown to improve the performance with other losses as well ( Fedus et al. , 2018 ) . Similarly , Roth et al . ( 2017 ) proposed to add a penalty on the weighted gradient-norm of the discriminator in the context of f-divergences , showing its equivalence to adding input noise . On the downside , regularizing the discriminator with the gradient penalty depends on the model distribution which changes during training and results in increased runtime due to additional gradient norm computation . Miyato et al . ( 2018 ) proposed another way to stabilize the discriminator by normalizing its weights and limiting the spectral norm of each layer to constrain the Lipschitz constant . This normalization technique does not require intensive tuning of hyper-parameters and is computationally light . Most recently , Zhang et al . ( 2018a ) showed that spectral normalization is also beneficial for the generator by preventing the escalation of parameter magnitudes and avoiding unusual gradients . Several methods have proposed to modify the training methodology of GANs in order to further improve stability , e.g . by considering multiple discriminators with different roles ( Durugkar et al. , 2017 ) or growing both the generator and discriminator networks progressively ( Karras et al. , 2018 ) . In this work we introduce an orthogonal way to stabilize the GAN training by progressively increasing the discrimination task difficulty . To this end , a novel and structured way of augmenting the discriminator input space is proposed . In contrast to other techniques , our method does not bias the optimality of the discriminator or alter the training samples . Furthermore , the proposed augmentation is complementary to prior work . It can be employed with different GAN architectures , adapted to various divergence measures and combined with other regularization techniques ( see Sec . 5 ) . 3 THEORETICAL BACKGROUND . For generative modeling , one common approach is to adopt divergence measures as loss functions for the generator . Our method belongs to this line of work . In contrast to prior work , our primary focus is not on explicitly minimizing the divergence between the data and model distributions defined on the sample space X . Alternatively , we first structurally augment the training samples ( both real and synthetic ones ) and then minimize the divergence between distributions defined on the augmented sample space . For computing the divergence , we adopt the adversary process introduced by ( Goodfellow et al. , 2014 ) ( Sec . 3.1 ) , while the proposed augmentation is inspired by the information theory view of Jensen-Shannon ( JS ) divergence ( Sec . 3.2 ) . Both of them are briefly reviewed in this section to lay the theoretical groundwork for our method , which we then discuss in Sec . 4 . 3.1 ADVERSARY PROCESS OF GANS . Let X denote a compact metric space such as the image space [ 0 , 1 ] d of dimension d. The data distribution Pd and the model distribution Pg are both probability measures defined on X . In the context of GANs , Pg is commonly induced by a function G that maps a random noise vector z , following a given prior distribution Pz , to a synthetic data sample , i.e . xg = G ( z ) ∈ X . The core idea behind GAN training is to set up a competing game between two players , commonly termed the discriminator D and generator G. Mathematically , their objective can be formulated as min G max D L ( D , G ) ∆ = Ex ∼ Pd { log [ D ( x ) ] } + Ex ∼ Pg { log [ 1−D ( x ) ] } , ( 1 ) with D : X 7→ [ 0 , 1 ] . The optimal D∗ respectively classifies x ∼ Pd and x ∼ Pg as TRUE and FAKE , i.e . binary classification . Its achieved maximum equals the JS divergence between Pd and Pg , which is then used as the loss function by the generator to optimize G ( Goodfellow et al. , 2014 ) .

Summary: This paper proposes a new trick to improve the stability of GANs. In particular the authors try to tackle the vanishing gradient problem in GANs, when the discriminator becomes to strong and is able to perfectly separate the distribution early in training, resulting in almost zero gradient for the generator. The authors propose to increase the difficulty of the task during training to avoid the discriminator to become too strong.