Generative Models from the perspective of Continual Learning

1 INTRODUCTION . Learning in a continual fashion is a key aspect for cognitive development among biological species ( Fagot & Cook , 2006 ) . In Machine Learning , such learning scenario has been formalized as a Continual Learning ( CL ) setting ( Srivastava et al. , 2013 ; Nguyen et al. , 2017 ; Seff et al. , 2017 ; Shin et al. , 2017 ; Schwarz et al. , 2018 ) . The goal of CL is to learn from a data distribution that change over time without forgetting crucial information . Unfortunately , neural networks trained with backpropagation are unable to retain previously learned information when the data distribution change , an infamous problem called `` catastrophic forgetting '' ( French , 1999 ) . Successful attempts at CL with neural networks have to overcome the inexorable forgetting happening when tasks change . In this paper , we focus on generative models in Continual Learning scenarios . Previous work on CL has mainly focused on classification tasks ( Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ; Shin et al. , 2017 ; Schwarz et al. , 2018 ) . Traditional approaches are regularization , rehearsal and architectural strategies , as described in Section 2 . However , discriminative and generative models strongly differ in their architecture and learning objective . Several methods developed for discriminative models are thus not directly extendable to the generative setting . Moreover , successful CL strategies for generative models can be used , via sample generation as detailed in the next section , to continually train discriminative models . Hence , studying the viability and success/failure modes of CL strategies for generative models is an important step towards a better understanding of generative models and Continual Learning in general . We conduct a comparative study of generative models with different CL strategies . In our experiments , we sequentially learn generation tasks . We perform ten disjoint tasks , using commonly used benchmarks for CL : MNIST ( LeCun et al. , 1998 ) , Fashion MNIST ( Xiao et al. , 2017 ) and CIFAR10 ( Krizhevsky et al. , 2009 ) . In each task , the model gets a training set from one new class , and should learn to generate data from this class without forgetting what it learned in previous tasks , see Fig . 1 for an example with tasks on MNIST . We evaluate several generative models : Variational Auto-Encoders ( VAEs ) , Generative Adversarial Networks ( GANs ) , their conditional variant ( CVAE ans CGAN ) , Wasserstein GANs ( WGANs ) and 1https : //github.com/anonymous-authors-2018/Generative_models_from_the_ perspective_of_Continual_learning Wasserstein GANs Gradient Penalty ( WGAN-GP ) . We compare results on approaches taken from CL in a classification setting : finetuning , rehearsal , regularization and generative replay . Generative replay consists in using generated samples to maintain knowledge from previous tasks . All CL approaches are applicable to both variational and adversarial frameworks . We evaluate with two quantitative metrics , Fréchet Inception Distance ( Heusel et al. , 2017 ) and Fitting Capacity ( Lesort et al. , 2018 ) , as well as visualization . Also , we discuss the data availability and scalability of CL strategies . Our contributions are : • Evaluating a wide range of generative models in a Continual Learning setting . • Highlight success/failure modes of combinations of generative models and CL approaches . • Comparing , in a CL setting , two evaluation metrics of generative models . We describe related work in Section 2 , and our approach in Section 3 . We explain the experimental setup that implements our approach in Section 4 . Finally , we present our results and discussion in Section 5 and 6 , before concluding in Section 7 . 2 RELATED WORK . 2.1 CONTINUAL LEARNING FOR DISCRIMINATIVE MODELS . Continual Learning has mainly been applied to discriminative tasks . On this scenario , classification tasks are learned sequentially . At the end of the sequence the discriminative model should be able to solve all tasks . The naive method of fine-tuning from one task to the next one leads to catastrophic forgetting ( French , 1999 ) , i.e . the inability to keep initial performance on previous tasks . Previously proposed approaches can be classified into four main methods . The first method , referred to as rehearsal , is to keep samples from previous tasks . The samples may then be used in different ways to overcome forgetting . The method can not be used in a scenario where data from previous tasks is not available , but it remains a competitive baseline ( Rebuffi et al. , 2017 ; Nguyen et al. , 2017 ) . Furthermore , the scalability of this method can also be questioned because the memory needed to store samples grows linearly with the number of tasks . The second method employs regularization . Regularization constrains weight updates in order to maintain knowledge from previous tasks and thus avoid forgetting . Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) has become the standard method for this type of regularization . It estimates the weights ’ importance and adapt the regularization accordingly . Extensions of EWC have been proposed , such as online EWC ( Schwarz et al. , 2018 ) . Another well known regularization method is distillation , which transfers previously learned knowledge to a new model . Initially proposed by Hinton et al . ( 2015 ) , it has gained popularity in CL ( Li & Hoiem , 2017 ; Rebuffi et al. , 2017 ; Wu et al. , 2018b ; Shin et al. , 2017 ) as it enables the model to learn about previous tasks and the current task at the same time . The third method is the use of a dynamic architecture to maintain past knowledge and learn new information . Remarkable approaches that implement this method are Progressive Networks ( Rusu et al. , 2016 ) , Learning Without Forgetting ( LWF ) ( Li & Hoiem , 2016 ) and PathNet ( Fernando et al. , 2017 ) . The fourth and more recent method is generative replay ( Shin et al. , 2017 ; Venkatesan et al. , 2017 ) , where a generative model is used to produce samples from previous tasks . This approach has also been referred to as pseudo-rehearsal . 2.2 CONTINUAL LEARNING FOR GENERATIVE MODELS . Discriminative and generative models do not share the same learning objective and architecture . For this reason , CL strategies for discriminative models are usually not directly applicable to generative models . Continual Learning in the context of generative models remains largely unexplored compared to CL for discriminative models . Among notable previous work , Seff et al . ( 2017 ) successfully apply EWC on the generator of Conditional-GANs ( CGANS ) , after observing that applying the same regularization scheme to a classic GAN leads to catastrophic forgetting . However , their work is based on a scenario where two classes are presented first , and then unique classes come sequentially , e.g the first task is composed of 0 and 1 digits of MNIST dataset , and then is presented with only one digit at a time on the following tasks . This is likely due to the failure of CGANs on single digits , which we observe in our experiments . Moreover , the method is shown to work on CGANs only . Another method for generative Continual Learning is Variational Continual Learning ( VCL ) ( Nguyen et al. , 2017 ) , which adapts variational inference to a continual setting . They exploit the online update from one task to another inspired from Bayes ’ rule . They successfully experiment with VAEs on a single-task scenario . While VCL has the advantage of being a parameter-free method . However , they experiment only on VAEs . Plus , since they use a multi-head architecture , they use specific weights for each task , which need task index for inference . A second method experimented on VAEs is to use a student-teacher method where the student learns the current task while the teacher retains knowledge ( Ramapuram et al. , 2017 ) . Finally , VASE ( Achille et al. , 2018 ) is a third method , also experimented only on VAEs , which allocates spare representational capacity to new knowledge , while protecting previously learned representations from catastrophic forgetting by using snapshots ( i.e . weights ) of previous model . A different approach , introduced by Shin et al . ( 2017 ) is an adaptation of the generative replay method mentioned in Section 2.1 . It is applicable to both adversarial and variational frameworks . It uses two generative models : one which acts as a memory , capable of generating all past tasks , and one that learns to generate data from all past tasks and the current task . It has mainly been used as a method for Continual Learning of discriminative models ( Shin et al. , 2017 ; Venkatesan et al. , 2017 ; Shah et al. , 2018 ) . Recently , Wu et al . ( 2018a ) have developed a similar approach called Memory Replay GANs , where they use Generative Replay combined to replay alignment , a distillation scheme that transfers previous knowledge from a conditional generator to the current one . However they note that this method leads to mode collapse because it could favor learning to generate few class instances rather than a wider range of class instances . 3 APPROACH . Typical previous work on Continual Learning for generative models focus on presenting a novel CL technique and comparing it to previous approaches , on one type of generative model ( e.g . GAN or VAE ) . On the contrary , we focus on searching for the best generative model and CL strategy association . For now , empirical evaluation remain the only way to find the best performing combinations . Hence , we compare several existing CL strategies on a wide variety of generative models with the objective of finding the most suited generative model for Continual Learning . In this process , evaluation metrics are crucial . CL approaches are usually evaluated by computing a metric at the end of each task . Whichever method that is able to maintain the highest performance is best . In the discriminative setting , classification accuracy is the most commonly used metric . Here , as we focus on generative models , there is no consensus on which metric should be used . Thus , we use and compare two quantitative metrics . The Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) is a commonly used metric for evaluating generative models . It is designed to improve on the Inception Score ( IS ) ( Salimans et al. , 2016 ) which has many intrinsic shortcomings , as well as additional problems when used on a dataset different than ImageNet ( Barratt & Sharma , 2018 ) . FID circumvent these issues by comparing the statistics of generated samples to real samples , instead of evaluating generated samples directly . Heusel et al . ( 2017 ) propose using the Fréchet distance between two multivariate Gaussians : FID = ‖µr − µg‖2 + Tr ( Σr + Σg − 2 ( ΣrΣg ) 1/2 ) , ( 1 ) where the statistics ( µr , Σr ) and ( µg , Σg ) are the activations of a specific layer of a discriminative neural network trained on ImageNet , for real and generated samples respectively . A lower FID correspond to more similar real and generated samples as measured by the distance between their activation distributions . Originally the activation should be taken from a given layer of a given Inception-v3 instance , however this setting can be adapted with another classifier in order to compare a set of models with each other ( Li et al. , 2017 ; Lesort et al. , 2018 ) . A different approach is to use labeled generated samples from a generator G ( GAN or VAE ) to train a classifier and evaluate it afterwards on real data ( Lesort et al. , 2018 ) . This evaluation , called Fitting Capacity of G , is the test accuracy of a classifier trained with G ’ s samples . It measures the generator ’ s ability to train a classifier that generalize well on a testing set , i.e the generator ’ s ability to fit the distribution of the testing set . This method aims at evaluating generative models on complex characteristics of data and not only on their features distribution . In the original paper , the authors annotated samples by generating them conditionally , either with a conditional model or by using one unconditional model for each class . In this paper , we also use an adaptation of the Fitting Capacity where data from unconditional models are labelled by an expert network trained on the dataset . We believe that using these two metrics is complementary . FID is a commonly used metric based solely on the distribution of images features . In order to have a complementary evaluation , we use the Fitting Capacity , which evaluate samples on a classification criterion rather than features distribution . For all the progress made in quantitative metrics for evaluating generative models ( Borji , 2018 ) , qualitative evaluation remains a widely used and informative method . While visualizing samples provides a instantaneous detection of failure , it does not provide a way to compare two wellperforming models . It is not a rigorous evaluation and it may be misleading when evaluating sample variability .

Summary: This paper evaluates and compares various methods for learning GANs in a Continual Learning setting, i.e., only some of the classes are available during training. It evaluates different continual learning methods including rehearsal, EWC and generative replay applied to training several deep generative models, like GAN, CGAN, WGAN, WGAN-GP, VAE and CVAE on MNIST, Fashion MNIST and CIFAR. The authors conclude with these experimental results that generative replay is the most effective method for such a setting, and found it is difficult to generate CIFAR10 images that can be classified successfully by an image classifier.