Generative Adversarial Network Training is a Continual Learning Problem

1 INTRODUCTION . Generative Adversarial Networks ( Goodfellow et al. , 2014 ) ( GANs ) are a popular framework for modeling draws from complex distributions , demonstrating success in a wide variety of settings , for example image synthesis ( Radford et al. , 2016 ; Karras et al. , 2018 ) and language modeling ( Li et al. , 2017 ) . In the GAN setup , two agents , the discriminator and the generator ( each usually a neural network ) , are pitted against each other . The generator learns a mapping from an easy-to-sample latent space to a distribution in the data space , which ideally matches the real data ’ s distribution . At the same time , the discriminator aims to distinguish the generator ’ s synthesized samples from the real data samples . When trained successfully , GANs yield impressive results ; in the image domain for example , synthesized images from GAN models are significantly sharper and more realistic than those of other classes of models ( Larsen et al. , 2016 ) . On the other hand , GAN training can be notoriously finicky . One particularly well-known and common failure mode is mode collapse ( Che et al. , 2017 ; Srivastava et al. , 2017 ) : instead of producing samples sufficiently representing the true data distribution , the generator maps the entire latent space to a limited subset of the real data space . When mode collapse occurs , the generator does not “ converge , ” in the conventional sense , to a stationary distribution . Rather , because the discriminator can easily learn to recognize a modecollapsed set of samples and the generator is optimized to avoid the discriminator ’ s detection , the two end up playing a never-ending game of cat and mouse : the generator meanders towards regions in the data space the discriminator thinks are real ( likely near where the real data lie ) while the discriminator chases after it . Interestingly though , if generated samples are plotted through time ( as in Figure 1 ) , it appears that the generator can revisit previously collapsed modes . At first , this may seem odd . The discriminator was ostensibly trained to recognize that mode in a previous iteration and did so well enough to push the generator away from generating those samples . Why has the discriminator seemingly lost this ability ? We conjecture that this oscillation phenomenon is enabled by catastrophic forgetting ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) : neural networks have a well-known tendency to forget how to complete old tasks while learning new ones . In most GAN models , the discriminator is a binary classifier , with the two classes being the real data and the generator ’ s outputs . Implicit to the training of a standard classifier is the assumption that the data are drawn independently and identically distributed ( i.i.d. ) . Importantly , this assumption does not hold true in GANs : the distribution of the generator class ( and thus the discriminator ’ s training data ) evolves over time . Moreover , these changes in the generator ’ s distribution are adversarial , designed specifically to deteriorate discriminator performance on the fake class as much as possible . Thus , the alternating training procedure of GANs in actuality corresponds to the discriminator learning tasks sequentially , where each task corresponds to recognizing samples from the generator at that particular point in time . Without any measures to prevent catastrophic forgetting , the discriminator ’ s ability to recognize fake samples from previous iterations will be clobbered by subsequent gradient updates , allowing a mode-collapsed generator to revisit old modes if training runs long enough . Given this tendency , a collapsed generator can wander indefinitely without ever learning the true distribution . With this perspective in mind , we cast training the GAN discriminator as a continual learning problem , leading to two main contributions . ( i ) While developing systems that learn tasks in a sequential manner without suffering from catastrophic forgetting has become a popular direction of research , current benchmarks have recently come under scrutiny as being unrepresentative to the fundamental challenges of continual learning ( Farquhar & Gal , 2018 ) . We argue that GAN training is a more realistic setting , and one that current methods tend to fail on . ( ii ) Such a reframing of the GAN problem allows us to leverage relevant methods to better match the dynamics of training the min-max objective . In particular , we build upon the recently proposed elastic weight consolidation ( Kirkpatrick et al. , 2017 ) and intelligent synapses ( Zenke et al. , 2017 ) . By preserving the discriminator ’ s ability to identify previous generator samples , this memory prevents the generator from simply revisiting past distributions . Adapting the GAN training procedure to account for catastrophic forgetting provides an improvement in GAN performance for little computational cost and without the need to train additional networks . Experiments on CelebA and CIFAR10 image generation and COCO Captions text generation show discriminator continual learning leads to better generations . 2 BACKGROUND : CATASTROPHIC FORGETTING IN GANS . Consider distribution preal ( x ) , from which we have data samples Dreal . Seeking a mechanism to draw samples from this distribution , we learn a mapping from an easy-to-sample latent distribution p ( z ) to a data distribution pgen ( x ) , which we want to match preal ( x ) . This mapping is parameterized as a neural networkGφ ( z ) with parametersφ , termed the generator . The synthesized data are drawn x = Gφ ( z ) , with z ∼ p ( z ) . The form of pgen ( x ) is not explicitly assumed or learned ; rather , we learn to draw samples from pgen ( x ) . To provide feedback to Gφ ( z ) , we simultaneously learn a binary classifier that aims to distinguish synthesized samples Dgen drawn from pgen ( x ) from the true samples Dreal . We also parameterize this classifier as a neural network Dθ ( x ) ∈ [ 0 , 1 ] with parameters θ , with Dθ ( x ) termed the discriminator . By incentivizing the generator to fool the discriminator into thinking its generations are actually from the true data , we hope to learn Gφ ( z ) such that pgen ( x ) approaches preal ( x ) . These two opposing goals for the generator and discriminator are usually formulated as the following min-max objective : min φ max θ LGAN ( θ , φ ) = Ex∼preal ( x ) [ logDθ ( x ) ] + Ez∼p ( z ) [ log ( 1−Dθ ( Gφ ( z ) ) ) ] ( 1 ) At each iteration t , we sample from pgen ( x ) , yielding generated data Dgent . These generated samples , along with samples from Dreal , are then passed to the discriminator . A gradient descent optimizer nudges θ so that the discriminator takes a step towards maximizing LGAN ( θ , φ ) . Parameters φ are updated similarly , but to minimize LGAN ( θ , φ ) . These updates to θ and φ take place in an alternating fashion . The expectations are approximated using samples from the respective distributions , and therefore learning only requires observed samples Dreal and samples from pgen ( x ) . The updates to Gφ ( z ) mean that pgen ( x ) changes as a function of t , perhaps substantially . Consequently , samples { Dgen1 , ... , D gen t } come from a sequence of different distributions . At iteration t , only samples from Dgent are available , as Gφ ( z ) has changed , and saving previous instantiations of the generator or samples { Dgen1 , ... , D gen t−1 } can be prohibitive . Thus , Dθ ( x ) is typically only provided Dgent , so it only learns the most recent distribution , with complete disregard for previous pgen ( x ) . Because of the catastrophic forgetting effect of neural networks , the ability of Dθ ( x ) to recognize these previous distributions is eventually lost in the pursuit of maximizing LGAN ( θ , φ ) with respect to only Dgent . This opens the possibility that the generator goes back to generating samples the discriminator had previously learned ( and then forgot ) to recognize , leading to unstable mode-collapsed oscillations that hamper GAN training ( as in Figure 1 ) . Recognizing this problem , we propose that the discriminator should be trained with the temporal component of pgen ( x ) in mind . 3 METHOD . 3.1 CLASSIC CONTINUAL LEARNING . Catastrophic forgetting has long been known to be a problem with neural networks trained on a series of tasks ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) . While there are many approaches to addressing catastrophic forgetting , here we primarily focus on elastic weight consolidation ( EWC ) and intelligent synapses ( IS ) . These are meant to illustrate the potential of catastrophic forgetting mitigation to improve GAN learning , with the expectation that this opens up the possibility of other such methods to significantly improve GAN training , at low additional computational cost . 3.1.1 ELASTIC WEIGHT CONSOLIDATION ( EWC ) . To derive the EWC loss , Kirkpatrick et al . ( 2017 ) frames training a model as finding the most probable values of the parameters θ given the data D. For two tasks , the data are assumed partitioned into independent sets according to the task , and the posterior for Task 1 is approximated as a Gaussian with mean centered on the optimal parameters for Task 1 θ∗1 and diagonal precision given by the diagonal of the Fisher information matrix F1 at θ∗1 . This gives the EWC loss the following form : L ( θ ) = L2 ( θ ) + LEWC ( θ ) , with LEWC ( θ ) , λ 2 ∑ i F1 , i ( θi − θ∗1 , i ) 2 , ( 2 ) where L2 ( θ ) = log p ( D2|θ ) is the loss for Task 2 individually , λ is a hyperparameter representing the importance of Task 1 relative to Task 2 , F1 , i = ( ∂L1 ( θ ) ∂θi ∣∣ θ=θ∗1 ) 2 , i is the parameter index , and L ( θ ) is the new loss to optimize while learning Task 2 . Intuitively , the EWC loss prevents the model from straying too far away from the parameters important for Task 1 while leaving less crucial parameters free to model Task 2 . Subsequent tasks result in additional LEWC ( θ ) terms added to the loss for each previous task . By protecting the parameters deemed important for prior tasks , EWC as a regularization term allows a single neural network ( assuming sufficient parameters and capacity ) to learn new tasks in a sequential fashion , without forgetting how to perform previous tasks . 3.1.2 INTELLIGENT SYNAPSES ( IS ) . While EWC makes a point estimate of how essential each parameter is at the conclusion of a task , IS ( Zenke et al. , 2017 ) protects the parameters according to their importance along the task ’ s entire training trajectory . Termed synapses , each parameter θi of the neural network is awarded an importance measure ω1 , i based on how much it reduced the loss while learning Task 1 . Given a loss gradient g ( t ) = ∇θL ( θ ) |θ=θt at time t , the total change in loss during the training of Task 1 then is the sum of differential changes in loss over the training trajectory . With the assumption that parameters θ are independent , we have : ∫ t1 t0 g ( t ) dθ = ∫ t1 t0 g ( t ) θ′dt = ∑ i ∫ t1 t0 gi ( t ) θ ′ idt , − ∑ i ω1 , i , ( 3 ) where θ′ = dθdt and ( t 0 , t1 ) are the start and finish of Task 1 , respectively . Note the added negative sign , as importance is associated with parameters that decrease the loss . The importance measure ω1 , i can now be used to introduce a regularization term that protects parameters important for Task 1 from large parameter updates , just as the Fisher information matrix diagonal terms F1 , i were used in EWC . This results in an IS loss very reminiscent in form1 : L ( θ ) = L2 ( θ ) + LIS ( θ ) , with LIS ( θ ) , λ 2 ∑ i ω1 , i ( θi − θ∗1 , i ) 2 . ( 4 )

Summary: The authors argue that catastrophic forgetting may cause mode collapse and oscillation, and propose a novel plug-and-play  regularizer that can be applied to a variety of GANs' training process to counter catastrophic forgetting of the discriminator. The regularizer is a clever adaption of EWC and IS into the context of GAN training. With the authors' formulation, this regularizer will account for the discriminator's parameter from all previous "tasks" (snapshots taken at certain iterations) with extra memory budget of only one set of parameters, while assigning higher regularization strengths to parameters learned from recent tasks. Experiments demonstrate such regularizer improves GAN models including DCGAN, SN-DCGAN, WGAN-GP on image generation tasks and textGAN on text generation tasks.