Title: GENERATIVE ADVERSARIAL NETWORK TRAINING IS A CONTINUAL LEARNING PROBLEM

Abstract: Generative Adversarial Networks (GANs) have proven to be a powerful framework for learning to draw samples from complex distributions. However, GANs are also notoriously difficult to train, with mode collapse and oscillations a common problem. We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator. Recognizing this, our contributions are twofold. First, we show that GAN training makes for a more interesting and realistic benchmark for continual learning methods evaluation than some of the more canonical datasets. Second, we propose leveraging continual learning techniques to augment the discriminator, preserving its ability to recognize previous generator samples. We show that the resulting methods add only a light amount of computation, involve minimal changes to the model, and result in better overall performance on the examined image and text generation tasks.

1. INTRODUCTION:

Generative Adversarial Networks (Goodfellow et al., 2014) (GANs) are a popular framework for modeling draws from complex distributions, demonstrating success in a wide variety of settings, for example image synthesis (Radford et al., 2016; Karras et al., 2018) and language modeling (Li et al., 2017) . In the GAN setup, two agents, the discriminator and the generator (each usually a neural network), are pitted against each other. The generator learns a mapping from an easy-to-sample latent space to a distribution in the data space, which ideally matches the real data's distribution. At the same time, the discriminator aims to distinguish the generator's synthesized samples from the real data samples. When trained successfully, GANs yield impressive results; in the image domain for example, synthesized images from GAN models are significantly sharper and more realistic than those of other classes of models (Larsen et al., 2016) . On the other hand, GAN training can be notoriously finicky. One particularly well-known and common failure mode is mode collapse (Che et al., 2017; Srivastava et al., 2017) : instead of producing samples sufficiently representing the true data distribution, the generator maps the entire latent space to a limited subset of the real data space.

When mode collapse occurs, the generator does not "converge," in the conventional sense, to a stationary distribution. Rather, because the discriminator can easily learn to recognize a modecollapsed set of samples and the generator is optimized to avoid the discriminator's detection, the two end up playing a never-ending game of cat and mouse: the generator meanders towards regions in the data space the discriminator thinks are real (likely near where the real data lie) while the discriminator chases after it. Interestingly though, if generated samples are plotted through time (as in Figure 1 ), it appears that the generator can revisit previously collapsed modes. At first, this may seem odd. The discriminator was ostensibly trained to recognize that mode in a previous iteration and did so well enough to push the generator away from generating those samples. Why has the discriminator seemingly lost this ability?

We conjecture that this oscillation phenomenon is enabled by catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990) : neural networks have a well-known tendency to forget how to complete old tasks while learning new ones. In most GAN models, the discriminator is a binary classifier, with the two classes being the real data and the generator's outputs. Implicit to the training of a standard classifier is the assumption that the data are drawn independently and identically distributed (i.i.d.) . Importantly, this assumption does not hold true in GANs: the distribution of the generator class (and thus the discriminator's training data) evolves over time. Moreover, these changes in the generator's distribution are adversarial, designed specifically to deteriorate discriminator performance on the fake class as much as possible. Thus, the alternating training procedure of GANs in actuality corresponds to the discriminator learning tasks sequentially, where each task corresponds to recognizing samples from the generator at that particular point in time. Without any measures to prevent catastrophic forgetting, the discriminator's ability to recognize fake samples from previous iterations will be clobbered by subsequent gradient updates, allowing a mode-collapsed generator to revisit old modes if training runs long enough. Given this tendency, a collapsed generator can wander indefinitely without ever learning the true distribution.

With this perspective in mind, we cast training the GAN discriminator as a continual learning problem, leading to two main contributions. (i) While developing systems that learn tasks in a sequential manner without suffering from catastrophic forgetting has become a popular direction of research, current benchmarks have recently come under scrutiny as being unrepresentative to the fundamental challenges of continual learning (Farquhar & Gal, 2018) . We argue that GAN training is a more realistic setting, and one that current methods tend to fail on. (ii) Such a reframing of the GAN problem allows us to leverage relevant methods to better match the dynamics of training the min-max objective. In particular, we build upon the recently proposed elastic weight consolidation (Kirkpatrick et al., 2017) and intelligent synapses (Zenke et al., 2017) . By preserving the discriminator's ability to identify previous generator samples, this memory prevents the generator from simply revisiting past distributions. Adapting the GAN training procedure to account for catastrophic forgetting provides an improvement in GAN performance for little computational cost and without the need to train additional networks. Experiments on CelebA and CIFAR10 image generation and COCO Captions text generation show discriminator continual learning leads to better generations.

2. BACKGROUND: CATASTROPHIC FORGETTING IN GANS:

Consider distribution p real (x), from which we have data samples D real . Seeking a mechanism to draw samples from this distribution, we learn a mapping from an easy-to-sample latent distribution p(z) to a data distribution p gen (x), which we want to match p real (x). This mapping is parameterized as a neural network G φ (z) with parameters φ, termed the generator. The synthesized data are drawn x = G φ (z), with z ∼ p(z). The form of p gen (x) is not explicitly assumed or learned; rather, we learn to draw samples from p gen (x).

To provide feedback to G φ (z), we simultaneously learn a binary classifier that aims to distinguish synthesized samples D gen drawn from p gen (x) from the true samples D real . We also parameterize this classifier as a neural network D θ (x) ∈ [0, 1] with parameters θ, with D θ (x) termed the discriminator. By incentivizing the generator to fool the discriminator into thinking its generations are actually from the true data, we hope to learn G φ (z) such that p gen (x) approaches p real (x).

These two opposing goals for the generator and discriminator are usually formulated as the following min-max objective:

EQUATION

At each iteration t, we sample from p gen (x), yielding generated data D gen t . These generated samples, along with samples from D real , are then passed to the discriminator. A gradient descent optimizer nudges θ so that the discriminator takes a step towards maximizing L GAN (θ, φ). Parameters φ are updated similarly, but to minimize L GAN (θ, φ). These updates to θ and φ take place in an alternating fashion. The expectations are approximated using samples from the respective distributions, and therefore learning only requires observed samples D real and samples from p gen (x).

The updates to G φ (z) mean that p gen (x) changes as a function of t, perhaps substantially. Consequently, samples {D gen 1 , ..., D gen t } come from a sequence of different distributions. At iteration t, only samples from D gen t are available, as G φ (z) has changed, and saving previous instantiations of the generator or samples {D gen 1 , ..., D gen t-1 } can be prohibitive. Thus, D θ (x) is typically only provided D gen t , so it only learns the most recent distribution, with complete disregard for previous p gen (x). Because of the catastrophic forgetting effect of neural networks, the ability of D θ (x) to recognize these previous distributions is eventually lost in the pursuit of maximizing L GAN (θ, φ) with respect to only D gen t . This opens the possibility that the generator goes back to generating samples the discriminator had previously learned (and then forgot) to recognize, leading to unstable mode-collapsed oscillations that hamper GAN training (as in Figure 1 ). Recognizing this problem, we propose that the discriminator should be trained with the temporal component of p gen (x) in mind.

3. METHOD:

3.1 CLASSIC CONTINUAL LEARNING Catastrophic forgetting has long been known to be a problem with neural networks trained on a series of tasks (McCloskey & Cohen, 1989; Ratcliff, 1990) . While there are many approaches to addressing catastrophic forgetting, here we primarily focus on elastic weight consolidation (EWC) and intelligent synapses (IS). These are meant to illustrate the potential of catastrophic forgetting mitigation to improve GAN learning, with the expectation that this opens up the possibility of other such methods to significantly improve GAN training, at low additional computational cost.

3.1.1. ELASTIC WEIGHT CONSOLIDATION (EWC):

To derive the EWC loss, Kirkpatrick et al. (2017) frames training a model as finding the most probable values of the parameters θ given the data D. For two tasks, the data are assumed partitioned into independent sets according to the task, and the posterior for Task 1 is approximated as a Gaussian with mean centered on the optimal parameters for Task 1 θ * 1 and diagonal precision given by the diagonal of the Fisher information matrix F 1 at θ * 1 . This gives the EWC loss the following form:

EQUATION

where L 2 (θ) = log p(D 2 |θ) is the loss for Task 2 individually, λ is a hyperparameter representing the importance of Task

1 relative to Task 2, F 1,i = ∂L1(θ) ∂θi θ=θ * 1

) 2 , i is the parameter index, and L(θ) is the new loss to optimize while learning Task 2. Intuitively, the EWC loss prevents the model from straying too far away from the parameters important for Task 1 while leaving less crucial parameters free to model Task 2. Subsequent tasks result in additional L EWC (θ) terms added to the loss for each previous task. By protecting the parameters deemed important for prior tasks, EWC as a regularization term allows a single neural network (assuming sufficient parameters and capacity) to learn new tasks in a sequential fashion, without forgetting how to perform previous tasks.

3.1.2. INTELLIGENT SYNAPSES (IS):

While EWC makes a point estimate of how essential each parameter is at the conclusion of a task, IS (Zenke et al., 2017) protects the parameters according to their importance along the task's entire training trajectory. Termed synapses, each parameter θ i of the neural network is awarded an importance measure ω 1,i based on how much it reduced the loss while learning Task 1. Given a loss gradient g(t) = ∇ θ L(θ)| θ=θt at time t, the total change in loss during the training of Task 1 then is the sum of differential changes in loss over the training trajectory. With the assumption that parameters θ are independent, we have:

EQUATION

where θ = dθ dt and (t 0 , t 1 ) are the start and finish of Task 1, respectively. Note the added negative sign, as importance is associated with parameters that decrease the loss.

The importance measure ω 1,i can now be used to introduce a regularization term that protects parameters important for Task 1 from large parameter updates, just as the Fisher information matrix diagonal terms F 1,i were used in EWC. This results in an IS loss very reminiscent in form 1 :

L(θ) = L 2 (θ) + L IS (θ), with L IS (θ) λ 2 i ω 1,i (θ i -θ * 1,i ) 2 .

(4)

3.2. GAN CONTINUAL LEARNING:

The traditional continual learning methods are designed for certain canonical benchmarks, commonly consisting of a small number of clearly defined tasks (e.g., classification datasets in sequence).

In GANs, the discriminator is trained on dataset D t = {D real , D gen t } at each iteration t. However, because of the evolution of the generator, the distribution p gen (x) from which D gen t comes changes over time. This violates the i.i.d. assumption of the order in which we present the discriminator data. As such, we argue that different instances in time of the generator should be viewed as separate tasks. Specifically, in the parlance of continual learning, the training data are to be regarded as D = {(D real , D gen 1 ), (D real , D gen 2 ), ...}. Thus motivated, we would like to apply continual learning methods to the discriminator, but doing so is not straightforward for the following reasons:

• Definition of a task: EWC and IS were originally proposed for discrete, well-defined tasks. For example, Kirkpatrick et al. (2017) applied EWC to a DQN (Mnih et al., 2015) learning to play ten Atari games sequentially, with each game being a clear, independent task. For GAN, there is no such precise definition as to what constitutes a "task," and as discriminators are not typically trained to convergence at every iteration, it is also unclear how long a task should be.

• Computational memory: While Equations 2 and 4 are for two tasks, they can be extended to K tasks by adding a term L EWC or L IS for each of the K -1 prior tasks. As each term L EWC or L IS requires saving both a historical reference term θ * k and either F k or ω k (all of which are the same size as the model parameters θ) for each task k, employing these techniques naively quickly becomes impractical for bigger models when K gets large, especially if K is set to the number of training iterations T .

• Continual not learning: Early iterations of the discriminator are likely to be non-optimal, and without a forgetting mechanism, EWC and IS may forever lock the discriminator to a poor initialization. Additionally, the unconstrained addition of a large number of terms L EWC or L IS will cause the continual learning regularization term to grow unbounded, which can disincentivize any further changes in θ.

To address these issues, we build upon the aforementioned continual learning techniques, and propose several changes.

Number of tasks as a rate::

We choose the total number of tasks K as a function of a constant rate α, which denotes the number of iterations before the conclusion of a task, as opposed to arbitrarily dividing the GAN training iterations into some set number of segments. Given T training iterations, this means a rate α yields K = T α tasks. Online Memory: Seeking a way to avoid storing extra θ * k , F k , or ω k , we observe that the sum of two or more quadratic forms is another quadratic, which gives the classifier loss with continual learning the following form for the (k + 1) th task:

EQUATION

)

where θ * k,i = P k,i S k,i , S k,i = k κ=1 Q κ,i , P k,i = k κ=1 Q κ,i θ * κ,i , and Q κ,i is either F κ,i

or ω κ,i , depending on the method. We name models with EWC and IS augmentations EWC-GAN and IS-GAN, respectively.

1 Zenke et al. (2017) 

instead consider Ω1,i = ω 1,i (∆ 1,i ) 2 +ξ

, where ∆1,i = θ1,i -θ0,i and ξ is a small number for numerical stability. We however found that the inclusion of (∆1,i) 2 can lead to the loss exploding and then collapsing as the number of tasks increases and so omit it. We also change the hyperparameter c into λ 2 .

Controlled forgetting: To provide a mechanism for forgetting earlier non-optimal versions of the discriminator and to keep L CL bounded, we add a discount factor γ:

S k,i = k κ=1 γ k-κ Q κ,i and P k,i = k κ=1 γ k-κ Q κ,i θ * κ,i .

Together, α and γ determine how far into the past the discriminator remembers previous generator distributions, and λ controls how important memory is relative to the discriminator loss. Note, the terms S k and P k can be updated every α steps in an online fashion:

EQUATION

This allows the EWC or IS loss to be applied without necessitating storing either Q k or θ * k for every task k, which would quickly become too costly to be practical. Only a single variable to store a running average is required for each of S k and P k , making this method space efficient.

Augmenting the discriminator with the continual learning loss, the GAN objective becomes:

EQUATION

Note that the training of the generator remains the same; full algorithms are in Appendix A. Here we have shown two methods to mitigate catastrophic forgetting for the original GAN; however, the proposed framework is applicable to almost all of the wide range of GAN setups.

4. RELATED WORK:

Continual learning in GANs There has been previous work investigating continual learning within the context of GANs. Improved GAN (Salimans et al., 2016) introduced historical averaging, which regularizes the model with a running average of parameters of the most recent iterations. Sim-ulated+Unsupervised training (Shrivastava et al., 2017) proposed replacing half of each minibatch with previous generator samples during training of the discriminator, as a generated sample at any point in time should always be considered fake. However, such an approach necessitates a historical buffer of samples and halves the number of current samples that can be considered. Continual Learning GAN (Seff et al., 2018) applied EWC to GAN, as we have, but used it in the context of the class-conditioned generator that learns classes sequentially, as opposed to all at once, as we propose.

Thanh-Tung et al. ( 2018) independently reached a similar conclusion on catastrophic forgetting in GANs, but focused on gradient penalties and momentum on toy problems.

Multiple network GANs:

The heart of continual learning is distilling a network's knowledge through time into a single network, a temporal version of the ensemble described in Hinton et al. (2015) . There have been several proposed models utilizing multiple generators (Hoang et al., 2018; Ghosh et al., 2018) or multiple discriminators (Durugkar et al., 2017; Neyshabur et al., 2017) , while Bayesian GAN (Saatchi & Wilson, 2017) considered distributions on the parameters of both networks, but these all do not consider time as the source of the ensemble. Unrolled GAN (Metz et al., 2017) considered multiple discriminators "unrolled" through time, which is similar to our method, as the continual learning losses also utilize historical instances of discriminators. However, both EWC-GAN and IS-GAN preserve the important parameters for prior discriminator performance, as opposed to requiring backpropagation of generator samples through multiple networks, making them easier to implement and train.

GAN convergence While GAN convergence is not the focus of this paper, convergence does similarly avoid mode collapse, and there are a number of works on the topic (Heusel et al., 2017; Unterthiner et al., 2018; Nagarajan & Kolter, 2017; Mescheder et al., 2017) . From the perspective of Heusel et al. (2017) , EWC or IS regularization in GAN can be viewed as achieving convergence by slowing the discriminator, but per parameter, as opposed to a slower global learning rate.

5. EXPERIMENTS:



5.1. DISCRIMINATOR CATASTROPHIC FORGETTING:

While Figure 1 implies catastrophic forgetting in a GAN discriminator, we can show this concretely.

To do so, we first train a DCGAN (Radford et al., 2016) on the MNIST dataset. Since the generator is capable of generating an arbitrary number of samples at any point, we can randomly draw 70000 samples to comprise a new, "fake MNIST" dataset at any time. By doing this at regular intervals, we create datasets {D gen 1 , ..., D gen T } from p gen (x) at times 1, ..., T . Samples are shown in Appendix B. Having previously generated a series of datasets during the training of a DCGAN, we now reinitialize the discriminator and train to convergence on each D gen t in sequence. Importantly, we do not include samples from D gen <t while fine-tuning on D gen t . After fine-tuning on the train split of dataset D gen t , the percentage of generated examples correctly identified as fake by the discriminator is evaluated on the test splits of D gen ≤t , with and without EWC (Figure 2 ). The catastrophic forgetting effect of the discriminator trained with SGD is clear, with a steep drop-off in discriminating ability on D gen t-1 after fine-tuning on D gen t ; this is unsurprising, as p gen (x) has evolved specifically to deteriorate discriminator performance. While there is still a dropoff with EWC, forgetting is less severe.

While the training outlined above is not what is typical for GAN, we choose this set-up as it closely mirrors the continual learning literature. With recent criticisms of some common continual learning benchmarks as either being too easy or missing the point of continual learning (Farquhar & Gal, 2018) , we propose GAN as a new benchmark providing a more realistic setting. From Figure 2 , it is clear that while EWC certainly helps, there is still much room to improve with new continual learning methods. However, the merits of GAN as a continual learning benchmark go beyond difficulty. While it is unclear why one would ever use a single model to classify successive random permutations of MNIST (Goodfellow et al., 2013) , many real-world settings exist where the data distribution is slowly evolving. For such models, we would like to be able to update the deployed model without forgetting previously learned performance, especially when data collection is expensive and thus done in bulk sometime before deployment. For example, autonomous vehicles (Huval et al., 2015) will eventually encounter unseen car models or obstacles, and automated screening systems at airport checkpoints (Liang et al., 2018) will have to deal with evolving bags, passenger belongings, and threats. In both cases, sustained effectiveness requires a way to appropriately and efficiently update the models for new data, or risk obsolescence leading to dangerous blindspots.

Many machine learning datasets represent singe-time snapshots of the data distribution, and current continual learning benchmarks fail to capture the slow drift of the real-world data. The evolution of GAN synthesized samples represents an opportunity to generate an unlimited number of smoothly evolving datasets for such experiments. We note that while the setup used here is for binary real/fake classification, one could also conceivably use a conditional GAN (Mirza & Osindero, 2014) to generate an evolving multi-class classification dataset. We leave this exploration for future work.

5.2. MIXTURE OF EIGHT GAUSSIANS:

We show results on a toy dataset consisting of a mixture of eight Gaussians, as in the example in Figure 1 . Following the setup of (Metz et al., 2017) , the real data are evenly distributed among eight 2-dimensional Gaussian distributions arranged in a circle of radius 2, each with covariance 0.02I (see Figure 4 ). We evaluate our model with Inception Score (ICP) (Salimans et al., 2016) , which gives a rough measure of diversity and quality of samples; higher scores imply better performance, with the true data resulting in a score of around 7.870. For this simple dataset, since we know the true data distribution, we also calculate the symmetric Kullback-Leibler divergence (Sym-KL); lower scores mean the generated samples are closer to the true data. We show computation time, measured in numbers of training iterations per second (Iter/s), averaged over the full training of a model on a single Nvidia Titan X (Pascal) GPU. Each model was run 10 times, with the mean and standard deviation of each performance metric at the end of 25K iterations reported in Table 1 . The performance of EWC-GAN and IS-GAN were evaluated for a number of hyperparameter settings. We compare our results against a vanilla GAN (Goodfellow et al., 2014) , as well as a state-ofthe-art GAN with spectral normalization (SN) (Miyato et al., 2018) applied to the discriminator. As spectral normalization augments the discriminator loss in a way different from continual learning, we can combine the two methods; this variant is also shown.

Note that a discounted version of discriminator historical averaging (Salimans et al., 2016) can be recovered from the EWC and IS losses if the task rate α = 1 and Q k,i = 1 for all i and k, a poor approximation to both the Fisher information matrix diagonal and importance measure. If we also set the historical reference term θ * k and the discount factor γ to zero, then the EWC and IS losses become 2 weight regularization. These two special cases are also included for comparison.

We observe that augmenting GAN models with EWC and IS consistently results in generators that better match the true distribution, both qualitatively and quantitatively, for a wide range of hyperparameter settings. EWC-GAN and IS-GAN result in a better ICP and FID than 2 weight regularization and discounted historical averaging, showing the value of prioritizing protecting important parameters, rather than all parameters equally. EWC-GAN and IS-GAN also outperform a stateof-the-art method in SN-GAN. In terms of training time, updating the EWC loss requires forward propagating a new minibatch through the discriminator and updating S and P , but even if this is done at every step (α = 1), the resulting algorithm is only slightly slower than SN-GAN. Moreover, doing so is unnecessary, as higher values of α also provide strong performance for a much smaller time penalty. Combining EWC with SN-GAN leads to even better results, showing that the two methods can complement each other. IS-GAN can also be successfully combined with SN-GAN, but it is slower than EWC-GAN as it requires tracking the trajectory of parameters at each step. Sample generation evolution over time is shown in Figure 4 of Appendix C.

5.3. IMAGE GENERATION OF CELEBA AND CIFAR-10:

Since EWC-GAN achieves similar performance to IS-GAN but at less computational expense, we focus on the former for experiments on two image datasets, CelebA and CIFAR-10. Our EWC-GAN implementation is straightforward to add to any GAN model, so we augment various popular implementations. Comparisons are made with the TTUR (Heusel et al., 2017 ) variants2 of DCGAN (Radford et al., 2016) and WGAN-GP (Gulrajani et al., 2017) , as well as an implementation3 of a spectral normalized (Miyato et al., 2018 ) DCGAN (SN-DCGAN). Without modifying the learning rate or model architecture, we show results with and without the EWC loss term added to the discriminator for each. Performance is quantified with the Fréchet Inception Distance (FID) (Heusel et al., 2017) for both datasets. Since labels are available for CIFAR-10, we also report ICP for that dataset. Best values are reported in Table 2 , with samples in Appendix C. In each model, we see improvement in both FID and ICP from the addition of EWC to the discriminator. 

5.4. TEXT GENERATION OF COCO CAPTIONS:

We also consider the text generation on the MS COCO Captions dataset (Chen et al., 2015) , with the pre-processing in Guo et al. (2018) . Quality of generated sentences is evaluated by BLEU score (Papineni et al., 2002) . Since BLEU-b measures the overlap of b consecutive words between the generated sentences and ground-truth references, higher BLEU scores indicate better fluency. Self BLEU uses the generated sentences themselves as references; lower values indicate higher diversity.

We apply EWC and IS to textGAN (Zhang et al., 2017) , a recently proposed model for text generation in which the discriminator uses feature matching to stabilize training. This model's results (labeled "EWC" and "IS") are compared to a Maximum Likelihood Estimation (MLE) baseline, as well as several state-of-the-art methods: SeqGAN (Yu et al., 2017) , RankGAN (Lin et al., 2017) , GSGAN (Jang et al., 2016) and LeakGAN (Guo et al., 2018) . Our variants of textGAN outperforms the vanilla textGAN for all BLEU scores (see Table 3 ), indicating the effectiveness of addressing the forgetting issue for GAN training in text generation. EWC/IS + textGAN also demonstrate a significant improvement compared with other methods, especially on BLEU-2 and 3. Though our variants lag slightly behind LeakGAN on BLEU-4 and 5, their self BLEU scores (Table 4 ) indicate it generates more diverse sentences. Sample sentence generations can be found in Appendix C.

6. CONCLUSION:

We observe that the alternating training procedure of GAN models results in a continual learning problem for the discriminator, and training on only the most recent generations leads to consequences unaccounted for by most models. As such, we propose augmenting the GAN training objective with a continual learning regularization term for the discriminator to prevent its parameters from moving too far away from values that were important for recognizing synthesized samples from previous training iterations. Since the original EWC and IS losses were proposed for discrete tasks, we adapt them to the GAN setting. Our implementation is simple to add to almost any variation of GAN learning, and we do so for a number of popular models, showing a gain in ICP and FID for CelebA and CIFAR-10, as well as BLEU scores for COCO Captions. More importantly, we demonstrate that GAN and continual learning, two popular fields studied independently of each other, have the potential to benefit each other, as new continual learning methods stand to benefit GAN training, and GAN generated datasets provide new testing grounds for continual learning.

A ALGORITHM:

We summarize the continual learning GAN implementations in Algorithm 1 and 2. % Calculate current discriminator loss 8:

L θ = 1 m m j=1 [log D θ (xj)] + 1 m m j=1 [log(1 -D θ (G φ (zj)))] 9:

% Update history buffer for previous tasks 10:

if mod (t, α) = 0 then 11:

% At the end of a task 12:

for parameters θi in θ: 

13: Qi = ∂L θ ∂θ i 2 % Q is the

L θ = 1 m m j=1 [log D θ (xj)] + 1 m m j=1 [log(1 -D θ (G φ (zj)))] 9:

% Update discriminator parameter θ, adding IS 10: 

Lθ = L θ -λ 2 i S k,i (θi -θ * k,i ) 11: g = ∂ Lθ

B GENERATED MNIST DATASETS FOR CONTINUAL LEARNING BENCHMARKING:

To produce a smoothly evolving series of datasets for continual learning, we train a DCGAN on MNIST and generate an entire "fake" dataset of 70K samples every 50 training iterations of the DC-GAN generator. We propose learning each of these generated datasets as individual tasks for continual learning. Selected samples are shown in Figure 3 from the datasets D gen t for t ∈ {5, 10, 15, 20}, each generated from the same 100 samples of z for all t. Note that we actually trained a conditional DCGAN, meaning we also have the labels for each generated image. For experiments in Figure 2 , we focused on the real versus fake task to demonstrate catastrophic forgetting in a GAN discriminator and thus ignored the labels, but future experiments can incorporate such information. 4 . While vanilla GAN occasionally recovers the true distribution, more often than not, the generator collapses and then bounces around. Spectral Normalized GAN converges to the true distribution quickly in most runs, but it mode collapses and exhibits the same behavior as GAN in others. EWC-GAN consistently diffuses to all modes, tending to find the true distribution sooner with lower α. We omit IS-GAN, as it performs similarly to EWC-GAN. 1 . Each frame shows 10000 samples drawn from the true eight Gaussians mixture (red) and 10000 generator samples (blue).

We also show the generated image samples for CIFAR 10 and CelebA in Figure 5 , and generated text samples for MS COCO Captions in Table 5 . Table 5 : Sample sentence generations from EWC + textGAN a couple of people are standing by some zebras in the background the view of some benches near a gas station a brown motorcycle standing next to a red fence a bath room with a broken tank on the floor red passenger train parked under a bridge near a river some snow on the beach that is surrounded by a truck a cake that has been perform in the background for takeoff a view of a city street surrounded by trees two giraffes walking around a field during the day crowd of people lined up on motorcycles two yellow sheep with a baby dog in front of other sheep an intersection sits in front of a crowd of people a red double decker bus driving down the street corner an automobile driver stands in the middle of a snowy park five people at a kitchen setting with a woman there are some planes at the takeoff station a passenger airplane flying in the sky over a cloudy sky three aircraft loaded into an airport with a stop light there is an animal walking in the water an older boy with wine glasses in an office two old jets are in the middle of london three motorcycles parked in the shade of a crowd group of yellow school buses parked on an intersection a person laying on a sidewalk next to a sidewalk talking on a cell phone a chef is preparing food with a sink and stainless steel appliances

:

https://github.com/bioinf-jku/TTUR

https://github.com/minhnhat93/tf-SNDCGAN

Figures and Tables:

Figure 1: Real samples from a mixture of eight Gaussians in red; generated samples in blue. (a) The generator is mode-collapsed in the bottom right. (b) The discriminator learns to recognize the generator oversampling this region and pushes the generator away, so the generator gravitates toward a new mode. (c) The discriminator continues to chase the generator, causing the generator to move in a clockwise direction. (d) The generator eventually returns to the same mode as (a). Such oscillations are common while training a vanilla GAN. Best seen as a video: https://youtu.be/91a2gPWngo8.

Figure 2: Each line represents the discriminator's test accuracy on the fake GAN datasets. Note the sharp decrease in the discriminator's ability to recognize previous fake samples upon fine-tuning on the next dataset using SGD (left). Forgetting still occurs with EWC (right), but is less severe.

Continual learning GAN with EWC 1: Input: Training data D real , latent distribution p(z), hyperparameters of continual learning α, γ, λ, step size 2: Output: θ, φ, and generated samples D gen = {xj} N j=1 3: for t = 1, ..., T do 4: % Sample minibatch of size m 5: Noise sample: {zj} m j=1 ∼ p(z) 6: Data sample: {xj} m j=1 ∼ D real 7:

Figure 3: Image samples from generated "fake MNIST" datasets

Figure 4: Each row shows the evolution of generator samples at 5000 training step intervals for GAN, SN-GAN, and EWC-GAN for two α values. The proposed EWC-GAN models have hyperparameters matching the corresponding α in Table1. Each frame shows 10000 samples drawn from the true eight Gaussians mixture (red) and 10000 generator samples (blue).

Figure 5: Generated image samples from random draws of EWC+GANs.

Iterations per second, inception score, and symmetric KL divergence comparison on a mixture of eight Gaussians.

Table content: <table><tr><td>Model</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Method</td><td>α</td><td>λ</td><td>γ</td><td>Iter/s ↑</td><td>ICP ↑</td><td>Sym-KL ↓</td></tr><tr><td>GAN</td><td>-</td><td>-</td><td>-</td><td colspan="3">87.59 ± 1.45 2.835 ± 2.325 19.55 ± 3.07</td></tr><tr><td>GAN + 2 weight</td><td>1</td><td colspan="2">0.01 0</td><td/><td colspan="2">5.968 ± 1.673 15.19 ± 2.67</td></tr><tr><td colspan="2">GAN + historical avg. 1</td><td colspan="2">0.01 0.995</td><td/><td colspan="2">7.305 ± 0.158 13.32 ± 0.88</td></tr><tr><td>GAN + SN</td><td>-</td><td>-</td><td>-</td><td colspan="3">49.70 ± 0.13 6.762 ± 2.024 13.37 ± 3.86</td></tr><tr><td>GAN + IS</td><td colspan="3">1000 100 0.8</td><td colspan="3">42.26 ± 0.35 7.039 ± 0.294 15.10 ± 1.51</td></tr><tr><td>GAN + IS</td><td>100</td><td>10</td><td>0.98</td><td colspan="3">42.29 ± 0.10 7.500 ± 0.147 11.85 ± 0.92</td></tr><tr><td>GAN + IS</td><td>10</td><td colspan="2">100 0.99</td><td colspan="3">41.07 ± 0.07 7.583 ± 0.242 11.88 ± 0.84</td></tr><tr><td>GAN + SN + IS</td><td>10</td><td colspan="2">100 0.99</td><td colspan="3">25.69 ± 0.09 7.699 ± 0.048 11.10 ± 1.18</td></tr><tr><td>GAN + EWC</td><td colspan="3">1000 100 0.8</td><td colspan="3">82.78 ± 1.55 7.480 ± 0.209 13.00 ± 1.55</td></tr><tr><td>GAN + EWC</td><td>100</td><td>10</td><td>0.98</td><td colspan="3">80.63 ± 0.39 7.488 ± 0.222 12.16 ± 1.64</td></tr><tr><td>GAN + EWC</td><td>10</td><td>10</td><td>0.99</td><td colspan="3">73.86 ± 0.16 7.670 ± 0.112 11.90 ± 0.76</td></tr><tr><td>GAN + SN + EWC</td><td>10</td><td>10</td><td>0.99</td><td colspan="3">44.68 ± 0.11 7.708 ± 0.057 11.48 ± 1.12</td></tr></table>

Fréchet Inception Distance and Inception Score on CelebA and CIFAR-10

Table content: <table><tr><td/><td/><td/><td/><td>CelebA</td><td colspan="2">CIFAR-10</td></tr><tr><td/><td/><td>Method</td><td/><td>FID ↓</td><td>FID ↓</td><td>ICP ↑</td></tr><tr><td/><td/><td>DCGAN</td><td/><td>12.52</td><td colspan="2">41.44 6.97 ± 0.05</td></tr><tr><td/><td/><td colspan="2">DCGAN + EWC</td><td>10.92</td><td colspan="2">34.84 7.10 ± 0.05</td></tr><tr><td/><td/><td>WGAN-GP</td><td/><td>-</td><td colspan="2">30.23 7.09 ± 0.06</td></tr><tr><td/><td/><td colspan="2">WGAN-GP + EWC</td><td>-</td><td colspan="2">29.67 7.44 ± 0.08</td></tr><tr><td/><td/><td colspan="2">SN-DCGAN</td><td>-</td><td colspan="2">27.21 7.43 ± 0.10</td></tr><tr><td/><td/><td colspan="2">SN-DCGAN + EWC</td><td>-</td><td colspan="2">25.51 7.58 ± 0.07</td></tr><tr><td/><td/><td colspan="5">Table 3: Test BLEU ↑ results on MS COCO</td></tr><tr><td>Method</td><td colspan="7">MLE SeqGAN RankGAN GSGAN LeakGAN textGAN EWC</td><td>IS</td></tr><tr><td colspan="2">BLEU-2 0.820</td><td>0.820</td><td>0.852</td><td>0.810</td><td>0.922</td><td>0.926</td><td>0.934 0.933</td></tr><tr><td colspan="2">BLEU-3 0.607</td><td>0.604</td><td>0.637</td><td>0.566</td><td>0.797</td><td>0.781</td><td>0.802 0.791</td></tr><tr><td colspan="2">BLEU-4 0.389</td><td>0.361</td><td>0.389</td><td>0.335</td><td>0.602</td><td>0.567</td><td>0.594 0.578</td></tr><tr><td colspan="2">BLEU-5 0.248</td><td>0.211</td><td>0.248</td><td>0.197</td><td>0.416</td><td>0.379</td><td>0.400 0.388</td></tr></table>

Self BLEU ↓ results on MS COCO

Table content: <table><tr><td>Method</td><td colspan="7">MLE SeqGAN RankGAN GSGAN LeakGAN textGAN EWC</td><td>IS</td></tr><tr><td colspan="2">BLEU-2 0.754</td><td>0.807</td><td>0.822</td><td>0.785</td><td>0.912</td><td>0.843</td><td>0.854 0.853</td></tr><tr><td colspan="2">BLEU-3 0.511</td><td>0.577</td><td>0.592</td><td>0.522</td><td>0.825</td><td>0.631</td><td>0.671 0.655</td></tr><tr><td colspan="2">BLEU-4 0.232</td><td>0.278</td><td>0.288</td><td>0.230</td><td>0.689</td><td>0.317</td><td>0.388 0.364</td></tr></table>

Training data D real , latent distribution p(z), hyperparameters of continual learning α, γ, λ, step size 2: Output: θ, φ, and generated samples D gen = {xj} N

Table content: <table><tr><td colspan="2">Under review as a conference paper at ICLR 2019</td></tr><tr><td colspan="2">Algorithm 2 Continual learning GAN with IS</td></tr><tr><td/><td>j=1</td></tr><tr><td colspan="2">3: for t = 1, ..., T do</td></tr><tr><td>4:</td><td>% Sample minibatch of size m</td></tr><tr><td>5:</td><td>Noise sample: {zj} m j=1 ∼ p(z)</td></tr><tr><td>6:</td><td>Data sample: {xj} m j=1 ∼ D real</td></tr><tr><td>7:</td><td>% Calculate current discriminator loss</td></tr><tr><td>8:</td><td/></tr><tr><td/><td>Fisher for EWC</td></tr><tr><td>14:</td><td>Si = γSi + Qi</td></tr><tr><td>15: 16:</td><td>Pi = γPi + Qiθ  *  i θ *  i = P i S i</td></tr><tr><td>17:</td><td>end if</td></tr><tr><td>18: 19: 20:</td><td>% Update discriminator parameter θ, adding EWC Lθ = L θ -λ 2 k,i ) i S k,i (θi -θ *  ∂ Lθ θt+1 ← θt + t ∂θ t</td></tr><tr><td>21: 22: 23:</td><td>% Update generator parameter φ L φ = 1 m i=1 [log(1 -D θ (G φ (zi)))] m ∂L φ φt+1 ← φt -t ∂φ t</td></tr><tr><td colspan="2">24: end for</td></tr></table>

Peer review (iclr2019):
 The authors argue that catastrophic forgetting may cause mode collapse and oscillation, and propose a novel plug-and-play regularizer that can be applied to a variety of GANs* training process to counter catastrophic forgetting of the discriminator. The regularizer is a clever adaption of EWC and IS into the context of GAN training. With the authors* formulation, this regularizer will account for the discriminator*s parameter from all previous "tasks" (snapshots taken at certain iterations) with extra memory budget of only one set of parameters, while assigning higher regularization strengths to parameters learned from recent tasks. Experiments demonstrate such regularizer improves GAN models including DCGAN, SN-DCGAN, WGAN-GP on image generation tasks and textGAN on text generation tasks. Pros: The paper is well-written. The formulation of online memory and controlled forgetting are clever, giving rise to the adaption of EWC and IS as a practical regularizer to overcome the problem of catastrophic forgetting in GANs. The experiments also demonstrate the regularizer is superior than historical averaging and SN on the synthetic dataset, and it is able to improve multiple GAN models in both image and text generation tasks. Cons/Suggestions: 1. Although I can see the method is working, the empirical evidence to support "mode oscillation" is not strong enough for me. I think in order for continual learning to make perfect sense, mode oscillation should be an obvious issue for GANs; otherwise, we probably don*t need remembering the history, as the generator is probably evolving towards the right direction even in the vanilla approach. Still, since there have been several papers showing history is important, it should be helpful in some sense. In Figure 1, I cannot tell whether in (d), the generator returned to the previous space (probably refers to (a)). Even the centers of mass of (a) and (d) look different for me. Figure 2 (left) only shows the distribution of generated data is changing as the training proceeds in vanilla GANs, since few of them (some shallow blue lines) have low peaks in previous datasets. If the mode oscillates and the generator returns to previous state, there should at least be another peak along the line, which is missing in curves on later datasets (darker blue ones). (I guess I have understood this figure correctly, but Figure 2 seems horizontally flipped to me. Since you are testing on previous fake datasets, and the accuracy should drop on previous datasets; however, the accuracy drops on later datasets in the figure.) 2. I doubt the authors may not have tried enough sets of hyper parameters for baseline models. In table 1, the variance of GAN, GAN + l2 weight and GAN + SN are significantly higher than the others. I don*t think with l2 weight regularizer, the model will be much more unstable than the authors* approach. 3. The authors didn*t give the results of their regularizer with LeakGAN on text generation. Currently their model has lower test BLEU than LeakGAN, which indicates lower fluency, but its self BLEU is lower than LeakGAN, which indicates higher diversity. It would be much better if the proposed method can surpass LeakGAN on both metrics. 4. Using inception score on mixture of eight Gaussians may not make much sense, if they are using the ImageNet pre-trained model, since such a model is not trained to fit this distribution. Still, the author has reported symmetric KL. 5. The authors did not specify their inception score on real Celeb-A and CIFAR10 images. Overall, I tend to accept this paper for its contribution on methods. It would be even better if my concerns could be addressed. Edit: after seeing the review of Reviewer 3, I find the proposed method seems to be the same as Online EWC and I have downgraded the rating. The authors should address these concerns.