CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks

1 Introduction . Lifelong learning is an innate human characteristic ; we continuously acquire knowledge and upgrade our skills . We also accumulate our previous experiences to learn a new task efficiently . However , learning new tasks rarely affects our performance on already-learned tasks . For example , after learning one programming language , if we learn a new such language it is rare that our skill on the first deteriorates . In fact , the knowledge of the previous task ( s ) often speeds up learning of the subsequent task ( s ) . On the other hand , attaining lifelong learning in deep neural networks is still a challenge . Naive implementation of continual learning with deep learning models suffers from catastrophic forgetting [ 1 , 2 , 3 ] , which makes lifelong or continual learning ( CL ) difficult . Catastrophic forgetting refers to the situation when a model exhibits a decline in its performance on previously learned tasks , after it learns a new task . Several prior works [ 1 , 3 , 2 , 4 , 5 , 6 , 7 , 8 ] have been proposed to address catastrophic forgetting in deep neural networks , with most of them focusing on classification problems . In contrast , continual learning for unsupervised deep generative models , such as generative adversarial networks ( GAN ) [ 9 ] and variational auto-encoders ( VAE ) [ 10 ] is relatively less studied so far . Some recent work [ 6 , 11 , 12 , 13 ] has tried to address catastrophic forgetting in GANs . Among these , the generative-replay-based approach [ 6 , 11 ] usually works well only when the number of tasks is small . As the number of tasks ∗Equal contribution . † Currently with Google Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) become large , the model starts generating unrealistic samples that are not suitable for generative replay . A regularization-based approach for continual learning in GANs [ 12 ] often gives sub-optimal solutions . In particular , instead of learning the optimal weights for each task , it learns a mean weight for all tasks , since a single set of weights tries to generalize across all tasks . Therefore , these approaches mix the sample generation of various tasks and generate blurry and unrealistic samples . Also , continually learning new tasks on a fixed size network is not feasible since , after learning a few tasks , the model capacity exhausts and the model becomes unable to learn future tasks . Recently , expansion-based CL models [ 14 , 15 , 16 , 17 , 7 , 18 , 19 , 20 , 21 , 22 ] have shown promising results for discriminative ( supervised ) continual learning settings . These approaches are dynamic in nature and allow the number of network parameters to grow to accommodate new tasks . Moreover , these approaches can also be regularized appropriately by partitioning the previous and new task parameters . Therefore , unlike regularization-based approaches , the model is free to adapt to novel tasks . However , despite their promising performance , the excessive growth in the number of parameters [ 15 , 19 , 20 , 21 , 22 ] and floating-point ( FLOP ) requirements are critical concerns . In this work , we propose a simple expansion-based approach for GANs , which continually adapts to novel tasks without forgetting the previously-learned knowledge . The proposed approach considers a base model with global parameters and corresponding global feature maps . While learning each novel task , the base model is expanded to consider a task-specific feature transformation which efficiently adapts a global feature map to a task-specific feature map in each layer . Even though the total number of parameters increases due to the additional local/task-specific parameters , this feature-map transformation approach allows the leveraging of efficient architecture design choices ( e.g. , groupwise and pointwise convolution ) to obtain compact-sized task-specific parameters , which controls the growth and keeps the proposed model compact . We empirically observe that our feature map transformation is highly efficient , simple and effective , and shows more promising results compared to the weight space transformation based approaches [ 13 ] . In addition , our approach enables leveraging the task similarities . In particular , learning a new task that is similar to previous tasks should require less effort compared to learning a very different task ; if we already know statistics and linear algebra , learning the subject machine learning is more easy compared to learning a completely different subject , e.g. , computer architecture . Most existing CL approaches ignore the task-similarity information during the continual adaptation . We find that learning a novel task by initializing its task-specific parameters with the task-specific parameters of the most similar previous task significantly boosts performance . To this end , we learn a compact embedding for each task using the mean of the Fisher information matrix ( FIM ) per filter , and use it to measure task similarity . We observe that considering the task similarity information for parameter initialization not only boosts the model performance but can also be useful to reduce the number of task-specific parameters . While FIM has been used in regularization-based approaches for continual learning [ 2 ] , we show how it can be used in expansion based methods like ours . To show the efficacy of the proposed model , we conduct extensive experiments in various settings on real-world datasets . We show that the proposed approach can sequentially learn a large number of tasks without catastrophic forgetting , while incurring much smaller parameter and FLOP growth compared to the existing continual-learning GAN models . Further , we show that our approach is also applicable to the generative-replay-based discriminative continual learning ( e.g. , for classification problems ) . We empirically show that the pseudo-rehearsal provided by the proposed approach shows promising results for generative-replay-based discriminative models . Also , we conduct experiments to demonstrate the effectiveness of considering the task similarity in continual image generation , which we believe can lead to a promising direction for continual learning . Our contribution can be summarized as follows : • We propose an efficient feature-map-based transformation approach to continually adapt to new tasks in a task sequence , without forgetting the knowledge from previous tasks . • We propose a parallel combination of groupwise and pointwise 3× 3 and 1× 1 filters for efficient adaptation to novel tasks , and require significantly fewer parameters for adaptation . • We propose an approach that leverages the Fisher information matrix to measure task similarities for expansion-based continual learning . We empirically demonstrate its superior performance on several datasets . • We demonstrate the efficacy of the proposed model for continual learning of GANs and generative-replay-based discriminative tasks . We observe that , with much less parameter and FLOP growth , our approach outperforms existing state-of-the-art methods . 2 Lightweight Continual Adapter . Notation and Problem Setting : We assume a set of tasks T1 , T2 , . . . , TK to be learned , and they arrive in a sequence . At a particular time t , data corresponding to the only current task Tt are available . The data for the tth task are given as Tt = { xi , yi } Nti=1 where ( xi , yi ) represents data samples and class labels , respectively . For unconditional data generation , we assume yi = 1 , and for the conditional generation yi ∈ Y , where Y is the label set for the task Tt . The proposed approach consists of a base modelM with generator parameters θ , which serve as the global parameters . The global parameters at layer l are denoted by θl . Each layer ofM also contains the task-specific parameters which perform task-specific feature-map transformation . Let φtl denote the task-specific parameter for the lth layer during the training of task t. We use the GAN framework as a base generative model , based on the architecture proposed by [ 23 ] , although it is applicable to other GAN architectures as well . The base modelM comprises the discriminator ( D ) and generator ( G ) . Since our main aim is to learn a good generator for each task , our continual learning approach considers only the generator to have task-specific adapters . The discriminator parameters ( θd ) can be modified using each new task ’ s data . We learn the global parameters from a base task using the GAN objective [ 23 ] and learn the task specific parameters φt as we adapt to a new task t : min φt max θd Ex∼ptdata [ log ( D ( x ; θd ) ) ] + Ez∼pz ( z ) [ log ( 1−D ( G ( z ; θ , φ t ) ; θd ) ) ] +R ( θd , φ t ) ( 1 ) where E denotes expectation and R ( ) is the regularization term defined in [ 23 ] . We propose an efficient expansion-based approach to mitigate catastrophic forgetting in GANs while learning multiple tasks in a sequential manner . The proposed adapter module is fairly lightweight and easy to use in any standard model for continual adaptation . The added adapter in the standard architecture may , however , destabilize GAN training . To address this issue , we add a residual adapter in the transformed feature space that results in smooth training for the modified architecture . The parameters in efficient adapters and residual bias serve as local/task-specific parameters and are specific to a particular task . The details of the proposed adapter and residual bias are provided in Sections 2.1 and 2.2 , respectively . 2.1 Efficient Adapter . We propose task-specific adapter modules to continually adapt the model to accommodate novel tasks , without suffering from catastrophic forgetting for the previous tasks . The proposed approach leverages a layerwise transformation of the feature maps of the base model , using the adapter modules to learn each novel task . While training the task sequence , the global parameters corresponding to base modelM remain fixed . Let F tl ∈ Rm×n×c be the feature map obtained at the lth layer upon applying the global parameters θl of the lth layer during training of task t , i.e. , F tl = fθl ( F t l−1 ) ( 2 ) Here fθl represents convolution and F t l−1 is the previous layer ’ s feature map . Our objective is to transform the feature map F tl that is obtained at the l th layer , but not yet task-specific , to a taskspecific feature map using the task-specific adapters . Our task-specific adapters comprise two types of efficient convolutional operators . We combine both in a single operation to reduce layer latency . 3× 3 Groupwise Convolutional Operation Let fg be the 3×3 groupwise convolutional operation ( 3× 3−GCO ) [ 24 ] with a group size k at a particular layer l , and defined by local parameters φtlg . Therefore , we have each convolutional filter ci ∈ R3×3×k in contrast to the standard convolutional filter cs ∈ R3×3×c . Here k c , and therefore each filter ci is highly efficient and requires ck times fewer parameters and FLOPs . We apply the function fg on the feature map obtained by ( 2 ) . For the tth task , let the new feature map obtained after applying fg be M tg , which can be written as M tg = f g φtlg ( F tl ) ( 3 ) Here M tg ∈ Rm×n×c is the feature map of the same dimension as the feature map obtained by ( 2 ) . Therefore , we can easily feed the obtained feature map to the next layer without any architectural modifications . Note that the standard convolution cs accumulates the information across all c channels of the feature map , and in each channel its receptive field is 3× 3 . If we reduce the group size to k , then the new convolution ci covers the same receptive field but it accumulates the information across only k c channels . The above process is highly efficient but reduces the model ’ s performance , since it suffers from a non-negligible information loss by capturing fewer channels . To mitigate this potential information loss , we further leverage a specialized FLOP- and parameter-efficient convolutional filter , as described below . 1× 1 Groupwise Convolutional Operation Pointwise convolution ( PWC ) is widely used for efficient architecture design [ 25 , 26 ] . The standard PWC contains filters of size 1 × 1 × c. It is 9 times more efficient as compared to the 3 × 3 convolution but still contains significant number of parameters . To increase the efficiency of the PWC , we further divide it into z groups . Therefore , each convolutional filter ai contains the filter of size 1× 1× z and , since z c , it reduces the FLOP requirements and number of parameters cz× as compared to the standard PWC . Also , since z > k , it captures more feature maps and overcomes the drawback of the 3× 3−GCO . Let fp be the 1× 1 groupwise convolutional operational ( 1× 1-GCO ) and φtlp be the lth layer local parameter given by the 1× 1 groupwise convolutional operation . Assume that , after applying fp , we obtain the feature map M tp , which can be written as M tp = f p φtlp ( F tl ) ( 4 ) Here M tp ∈ Rm×n×c also has the same dimension as the incoming feature map , and therefore we can easily apply this input to the next layer without any significant changes to the base architecture . Parallel Combination Note that local feature map adapters , i.e . ( 3 ) and ( 4 ) , can be applied in a parallel manner and the joint feature map for task t can be written as M tl =M t g ⊕ βM tp ( 5 ) Here β = 1 when we use 1 × 1 groupwise convolution operation , otherwise β = 0 , and ⊕ is the element-wise addition . Therefore , using β , we can trade-off the model ’ s parameter and FLOP growth with its performance . In this combination , M tg is obtained by filters that capture a bigger receptive field and M tp is obtained by filters that capture the information across longer feature-map sequences . Therefore , the combination of the two helps increase a model ’ s performance without a significant increase in the model ’ s parameters and FLOPs . Like most of the efficient architecture models [ 26 , 27 ] , we can apply the 3× 3 and 1× 1 convolutional operation in a sequential manner , denoted as : M tl = f p φtlp ( fg φtlg ( F tl ) ) . Here M t l requires two sequential layers in the model architecture and execution of the second layer is followed by the first layer , which limits the parallelization of the model . Therefore , throughout the paper , we follow ( 5 ) , i.e. , the parallel combination . Again , note that the proposed adapter is added only to the generator network since the discriminator model ’ s parameter can be discarded once the model for each task is trained .

Summary: The paper proposed to learn a set of global and task-specific parameters to get task-specific feature maps for different tasks. And the proposed method leverages the Fisher information matrix to measure task similarities for parameter initialization for expansion-based continual learning. The proposed method is demonstrated on unconditional generation tasks and label-conditioned generation tasks.