CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks

1 Introduction . Lifelong learning is an innate human characteristic ; we continuously acquire knowledge and upgrade our skills . We also accumulate our previous experiences to learn a new task efficiently . However , learning new tasks rarely affects our performance on already-learned tasks . For example , after learning one programming language , if we learn a new such language it is rare that our skill on the first deteriorates . In fact , the knowledge of the previous task ( s ) often speeds up learning of the subsequent task ( s ) . On the other hand , attaining lifelong learning in deep neural networks is still a challenge . Naive implementation of continual learning with deep learning models suffers from catastrophic forgetting [ 1 , 2 , 3 ] , which makes lifelong or continual learning ( CL ) difficult . Catastrophic forgetting refers to the situation when a model exhibits a decline in its performance on previously learned tasks , after it learns a new task . Several prior works [ 1 , 3 , 2 , 4 , 5 , 6 , 7 , 8 ] have been proposed to address catastrophic forgetting in deep neural networks , with most of them focusing on classification problems . In contrast , continual learning for unsupervised deep generative models , such as generative adversarial networks ( GAN ) [ 9 ] and variational auto-encoders ( VAE ) [ 10 ] is relatively less studied so far . Some recent work [ 6 , 11 , 12 , 13 ] has tried to address catastrophic forgetting in GANs . Among these , the generative-replay-based approach [ 6 , 11 ] usually works well only when the number of tasks is small . As the number of tasks ∗Equal contribution . † Currently with Google Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) become large , the model starts generating unrealistic samples that are not suitable for generative replay . A regularization-based approach for continual learning in GANs [ 12 ] often gives sub-optimal solutions . In particular , instead of learning the optimal weights for each task , it learns a mean weight for all tasks , since a single set of weights tries to generalize across all tasks . Therefore , these approaches mix the sample generation of various tasks and generate blurry and unrealistic samples . Also , continually learning new tasks on a fixed size network is not feasible since , after learning a few tasks , the model capacity exhausts and the model becomes unable to learn future tasks . Recently , expansion-based CL models [ 14 , 15 , 16 , 17 , 7 , 18 , 19 , 20 , 21 , 22 ] have shown promising results for discriminative ( supervised ) continual learning settings . These approaches are dynamic in nature and allow the number of network parameters to grow to accommodate new tasks . Moreover , these approaches can also be regularized appropriately by partitioning the previous and new task parameters . Therefore , unlike regularization-based approaches , the model is free to adapt to novel tasks . However , despite their promising performance , the excessive growth in the number of parameters [ 15 , 19 , 20 , 21 , 22 ] and floating-point ( FLOP ) requirements are critical concerns . In this work , we propose a simple expansion-based approach for GANs , which continually adapts to novel tasks without forgetting the previously-learned knowledge . The proposed approach considers a base model with global parameters and corresponding global feature maps . While learning each novel task , the base model is expanded to consider a task-specific feature transformation which efficiently adapts a global feature map to a task-specific feature map in each layer . Even though the total number of parameters increases due to the additional local/task-specific parameters , this feature-map transformation approach allows the leveraging of efficient architecture design choices ( e.g. , groupwise and pointwise convolution ) to obtain compact-sized task-specific parameters , which controls the growth and keeps the proposed model compact . We empirically observe that our feature map transformation is highly efficient , simple and effective , and shows more promising results compared to the weight space transformation based approaches [ 13 ] . In addition , our approach enables leveraging the task similarities . In particular , learning a new task that is similar to previous tasks should require less effort compared to learning a very different task ; if we already know statistics and linear algebra , learning the subject machine learning is more easy compared to learning a completely different subject , e.g. , computer architecture . Most existing CL approaches ignore the task-similarity information during the continual adaptation . We find that learning a novel task by initializing its task-specific parameters with the task-specific parameters of the most similar previous task significantly boosts performance . To this end , we learn a compact embedding for each task using the mean of the Fisher information matrix ( FIM ) per filter , and use it to measure task similarity . We observe that considering the task similarity information for parameter initialization not only boosts the model performance but can also be useful to reduce the number of task-specific parameters . While FIM has been used in regularization-based approaches for continual learning [ 2 ] , we show how it can be used in expansion based methods like ours . To show the efficacy of the proposed model , we conduct extensive experiments in various settings on real-world datasets . We show that the proposed approach can sequentially learn a large number of tasks without catastrophic forgetting , while incurring much smaller parameter and FLOP growth compared to the existing continual-learning GAN models . Further , we show that our approach is also applicable to the generative-replay-based discriminative continual learning ( e.g. , for classification problems ) . We empirically show that the pseudo-rehearsal provided by the proposed approach shows promising results for generative-replay-based discriminative models . Also , we conduct experiments to demonstrate the effectiveness of considering the task similarity in continual image generation , which we believe can lead to a promising direction for continual learning . Our contribution can be summarized as follows : • We propose an efficient feature-map-based transformation approach to continually adapt to new tasks in a task sequence , without forgetting the knowledge from previous tasks . • We propose a parallel combination of groupwise and pointwise 3× 3 and 1× 1 filters for efficient adaptation to novel tasks , and require significantly fewer parameters for adaptation . • We propose an approach that leverages the Fisher information matrix to measure task similarities for expansion-based continual learning . We empirically demonstrate its superior performance on several datasets . • We demonstrate the efficacy of the proposed model for continual learning of GANs and generative-replay-based discriminative tasks . We observe that , with much less parameter and FLOP growth , our approach outperforms existing state-of-the-art methods . 2 Lightweight Continual Adapter . Notation and Problem Setting : We assume a set of tasks T1 , T2 , . . . , TK to be learned , and they arrive in a sequence . At a particular time t , data corresponding to the only current task Tt are available . The data for the tth task are given as Tt = { xi , yi } Nti=1 where ( xi , yi ) represents data samples and class labels , respectively . For unconditional data generation , we assume yi = 1 , and for the conditional generation yi ∈ Y , where Y is the label set for the task Tt . The proposed approach consists of a base modelM with generator parameters θ , which serve as the global parameters . The global parameters at layer l are denoted by θl . Each layer ofM also contains the task-specific parameters which perform task-specific feature-map transformation . Let φtl denote the task-specific parameter for the lth layer during the training of task t. We use the GAN framework as a base generative model , based on the architecture proposed by [ 23 ] , although it is applicable to other GAN architectures as well . The base modelM comprises the discriminator ( D ) and generator ( G ) . Since our main aim is to learn a good generator for each task , our continual learning approach considers only the generator to have task-specific adapters . The discriminator parameters ( θd ) can be modified using each new task ’ s data . We learn the global parameters from a base task using the GAN objective [ 23 ] and learn the task specific parameters φt as we adapt to a new task t : min φt max θd Ex∼ptdata [ log ( D ( x ; θd ) ) ] + Ez∼pz ( z ) [ log ( 1−D ( G ( z ; θ , φ t ) ; θd ) ) ] +R ( θd , φ t ) ( 1 ) where E denotes expectation and R ( ) is the regularization term defined in [ 23 ] . We propose an efficient expansion-based approach to mitigate catastrophic forgetting in GANs while learning multiple tasks in a sequential manner . The proposed adapter module is fairly lightweight and easy to use in any standard model for continual adaptation . The added adapter in the standard architecture may , however , destabilize GAN training . To address this issue , we add a residual adapter in the transformed feature space that results in smooth training for the modified architecture . The parameters in efficient adapters and residual bias serve as local/task-specific parameters and are specific to a particular task . The details of the proposed adapter and residual bias are provided in Sections 2.1 and 2.2 , respectively . 2.1 Efficient Adapter . We propose task-specific adapter modules to continually adapt the model to accommodate novel tasks , without suffering from catastrophic forgetting for the previous tasks . The proposed approach leverages a layerwise transformation of the feature maps of the base model , using the adapter modules to learn each novel task . While training the task sequence , the global parameters corresponding to base modelM remain fixed . Let F tl ∈ Rm×n×c be the feature map obtained at the lth layer upon applying the global parameters θl of the lth layer during training of task t , i.e. , F tl = fθl ( F t l−1 ) ( 2 ) Here fθl represents convolution and F t l−1 is the previous layer ’ s feature map . Our objective is to transform the feature map F tl that is obtained at the l th layer , but not yet task-specific , to a taskspecific feature map using the task-specific adapters . Our task-specific adapters comprise two types of efficient convolutional operators . We combine both in a single operation to reduce layer latency . 3× 3 Groupwise Convolutional Operation Let fg be the 3×3 groupwise convolutional operation ( 3× 3−GCO ) [ 24 ] with a group size k at a particular layer l , and defined by local parameters φtlg . Therefore , we have each convolutional filter ci ∈ R3×3×k in contrast to the standard convolutional filter cs ∈ R3×3×c . Here k c , and therefore each filter ci is highly efficient and requires ck times fewer parameters and FLOPs . We apply the function fg on the feature map obtained by ( 2 ) . For the tth task , let the new feature map obtained after applying fg be M tg , which can be written as M tg = f g φtlg ( F tl ) ( 3 ) Here M tg ∈ Rm×n×c is the feature map of the same dimension as the feature map obtained by ( 2 ) . Therefore , we can easily feed the obtained feature map to the next layer without any architectural modifications . Note that the standard convolution cs accumulates the information across all c channels of the feature map , and in each channel its receptive field is 3× 3 . If we reduce the group size to k , then the new convolution ci covers the same receptive field but it accumulates the information across only k c channels . The above process is highly efficient but reduces the model ’ s performance , since it suffers from a non-negligible information loss by capturing fewer channels . To mitigate this potential information loss , we further leverage a specialized FLOP- and parameter-efficient convolutional filter , as described below . 1× 1 Groupwise Convolutional Operation Pointwise convolution ( PWC ) is widely used for efficient architecture design [ 25 , 26 ] . The standard PWC contains filters of size 1 × 1 × c. It is 9 times more efficient as compared to the 3 × 3 convolution but still contains significant number of parameters . To increase the efficiency of the PWC , we further divide it into z groups . Therefore , each convolutional filter ai contains the filter of size 1× 1× z and , since z c , it reduces the FLOP requirements and number of parameters cz× as compared to the standard PWC . Also , since z > k , it captures more feature maps and overcomes the drawback of the 3× 3−GCO . Let fp be the 1× 1 groupwise convolutional operational ( 1× 1-GCO ) and φtlp be the lth layer local parameter given by the 1× 1 groupwise convolutional operation . Assume that , after applying fp , we obtain the feature map M tp , which can be written as M tp = f p φtlp ( F tl ) ( 4 ) Here M tp ∈ Rm×n×c also has the same dimension as the incoming feature map , and therefore we can easily apply this input to the next layer without any significant changes to the base architecture . Parallel Combination Note that local feature map adapters , i.e . ( 3 ) and ( 4 ) , can be applied in a parallel manner and the joint feature map for task t can be written as M tl =M t g ⊕ βM tp ( 5 ) Here β = 1 when we use 1 × 1 groupwise convolution operation , otherwise β = 0 , and ⊕ is the element-wise addition . Therefore , using β , we can trade-off the model ’ s parameter and FLOP growth with its performance . In this combination , M tg is obtained by filters that capture a bigger receptive field and M tp is obtained by filters that capture the information across longer feature-map sequences . Therefore , the combination of the two helps increase a model ’ s performance without a significant increase in the model ’ s parameters and FLOPs . Like most of the efficient architecture models [ 26 , 27 ] , we can apply the 3× 3 and 1× 1 convolutional operation in a sequential manner , denoted as : M tl = f p φtlp ( fg φtlg ( F tl ) ) . Here M t l requires two sequential layers in the model architecture and execution of the second layer is followed by the first layer , which limits the parallelization of the model . Therefore , throughout the paper , we follow ( 5 ) , i.e. , the parallel combination . Again , note that the proposed adapter is added only to the generator network since the discriminator model ’ s parameter can be discarded once the model for each task is trained .

Summary: The paper presents an approach for continual learning in GANs by learning new task-specific modules over a base network. This work explores the contributions of different continual learning approaches for generative models and proposes an expansion-based model which learns new task-specific modules for each new task. These modules augment the features of the base task layer. To prevent instability issues, the model also creates additional residual pathways. To choose the parameters for a new task, a task similarity-based initialization is proposed which initializes the new task network parameters with the learnt parameters of the previous task which is most similar to the current task. Experiments are performed on conditional and unconditional image tasks, and ablations are performed on the effectiveness of the proposed modules.