Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation

1 INTRODUCTION . It is well-known that neural networks ( NNs ) suffer from catastrophic forgetting ( CF ) ( McCloskey & Cohen , 1989 ) , which refers to the phenomenon that when learning a sequence of tasks , the learning of each new task may cause the NN to forget the models learned for the previous tasks . Without solving this problem , an NN is hard to adapt to lifelong or continual learning , which is important for AI . Problem Statement : Given a sequence of supervised learning tasks T = ( T1 , T2 , . . . , TN ) , we want to learn them one by one in the given sequence such that the learning of each new task will not forget the models learned for the previous tasks . In recent years , many approaches ( often called continual learning ) have been proposed to lessen the effect of CF ( Chen & Liu , 2018 ; Parisi et al. , 2018 ) , e.g. , dynamically expandable network ( DEN ) ( Yoon et al. , 2018 ) , learning without forgetting ( LWF ) ( Li & Hoiem , 2016 ) ) , elastic weight consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , incremental moment matching ( IMM ) ( Lee et al. , 2017 ) , gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) , generative replay ( GR ) ( Shin et al. , 2017 ) , etc . These existing studies except DEN and LWF focused on learning a model parameterized by a single joint set of parameters θ∗ which is assumed to work well for all tasks . We call them joint parameterization ( JP ) methods . DEN and LWF require constantly increasing the number of parameters and thus can result in a huge and complex model . JP methods , however , suffer from accuracy deterioration . Assume we have two tasks A and B that need to be learned sequentially . Let θ∗A and θ ∗ B be the optimal parameters for Task A and Task B respectively when each of them is learned individually . Let θ∗ be the joint parameters learned by the existing approaches to perform Tasks A and B in sequence . Inevitably , θ∗ is different from θ∗A and/or θ∗B and is highly likely to result in more errors for the two tasks than θ ∗ A and θ ∗ B individually . ∗Equal contribution †Corresponding author In this paper , we propose a different approach , called Parameter Generation and Model Adaptation ( PGMA ) , to dealing with CF and to significantly reducing accuracy deterioration . PGMA does not learn a joint parameter set θ∗ . Instead , it learns a parameter generator f ( · ) and a shared parameter set θ0 . The key idea of PGMA is as follows : The overall classification network , called the solver S , has two disjoint subsets/parts of parameters . The first subset is θ0 , which is shared by all tasks learned so far . The second subset is just a place holder H that will be filled by parameters generated by f ( · ) for each test instance . That is , given a test instance , a set of parameters will be generated by the learned parameter generator f ( · ) to replace H . Solver S then combines θ0 and the generated parameters for the test instance to classify it . Clearly , unlike existing JP approaches , we do not have a network with a set of fixed parameters for S. The idea is that θ0 contains the common features of all tasks , and the generated parameters for each test instance adapt S for each test instance in order to classify it . Since f ( · ) and the shared parameters θ0 will change during training for each new task , forgetting can occur for previous tasks . To deal with it , in training f ( · ) and S for each task Ti , in addition to the training data of Ti , a small number of replayed samples will be generated by a data generation network for the previous tasks to ensure that the knowledge learned for previous tasks remain stable/unforgotten . Compared with the existing generative replay methods , our method does not need labels for the replayed data because we use the replayed data only to constrain the training of S and f ( · ) rather than to treat them as surrogates of labeled training data from previous tasks . Further , since in the existing replay methods , the labels are produced by the current learned network itself , the labels can be noisy and biased , which may result in errors being accumulated and propagated to subsequent tasks . Our method does not have this problem . Apart from reducing the effect of accuracy deterioration of the existing approaches , the proposed approach also has some other advantages . First , no parameter increase or network expansion is needed to learn new tasks . Second , no previous data needs to be stored to enable the system to remember the previously learned models or knowledge . Experiments conducted using two image datasets ( MNIST and CIFAR-10 ) and two text datasets ( DBPedia ontology ( Lehmann et al. , 2015 ) and THUCNews ( Li et al. , 2006 ) ) show that the proposed approach PGMA works well for different scenarios and different types of datasets , and outperforms the existing strong baselines markedly . 2 PROPOSED PGMA FRAMEWORK . Let the sequence of supervised learning tasks be T = ( T1 , T2 , . . . , TN ) . Each task Ti is represented by Ti = { xij , yij |j ∈ ( 1 , . . . , Ni ) } , where xij is the j-th example/sample of Ti , and yij is its label . We use ( xt , yt ) to denote a test instance/example . To simplify the notation , we will just use ( xi , yi ) to denote a training example from task Ti , omitting the second subscript j . Note that in the paper , we use the terms example , sample , and instance interchangeably . The proposed parameter generation and model adaptation ( PGMA ) architecture has three main components : • Solver S : It is the main classification model . As mentioned in Section 1 , the parameter set of S consists of two subsets , θ0 that is shared by all tasks ( and instances ) and H , a parameter place holder , which will be replaced by the generated parameters set pi ( or pt ) for each training ( or testing ) example xi ( or xt ) . pi ( or pt ) basically serves to adapt the solver S to classify the example in training or testing . This is the key idea of our approach . We adopt this parameter split as several studies ( Yoon et al. , 2018 ; Li & Hoiem , 2016 ; Kirkpatrick et al. , 2017 ) have shown that only part of parameters of a neural network needs to be adjusted when learning a new task . See Section 2.2 • Dynamic Parameter Generator ( DPG ) f ( · ) : It takes the embedding zi ( or zt ) of each input training ( or testing ) example xi ( or xt ) to generate the parameters pi ( or pt ) for solver S. Note that we use zi ( or zt ) rather than the raw data xi ( or xt ) because the raw data ’ s dimension can be very high . The embedding zi ( or zt ) as a low dimensional dense representation reduces the mapping space for DPG and thus reduces the difficulty in its parameter generation . See Section 2.2 . • Data Generator ( DG ) : It has two functions . The main function is to generate a set of replayed data or samples { x′m } Mm=1 using its decoder DGD for previous tasks to deal with catastrophic forgetting . The other function is to generate the embedding zi ( or zt ) of each input training ( or testing ) example xi ( or xt ) using its encoder DGE . See Section 2.3 . 2.1 OVERALL APPROACH OF PGMA . We work backward by describing testing first before training . Given a test instance xt , DGE first generates its embedding zt , which is fed to f ( · ) to generate a set of parameters pt . Solver S then takes xt as input and uses the trained/learned shared parameters θ0 and pt to classify xt . θ0 contains the common features of all tasks learned so far . pt simply adapts S for xt in order to classify xt . For training , the pipeline of the proposed PGMA framework is shown in Figure 1 . Given a new task Ti with its data ( xi , yi ) , solver S and DPG f ( · ) are jointly trained to learn Ti and also not to forget the previously learned tasks . In each iteration , a set of parameters pi is generated by the current DPG f ( zi , µ ) for each training instance xi ( zi being its embedding ) , where µ is the set of parameters of the DPG network , which is trained . In training the solver S and DPG for the new task Ti , both f ( · ) and the shared parameters θ0 will change , which can cause forgetting in DPG for previous tasks . To keep DPG remembering the acquired knowledge for previous tasks , we minimize the variation of certain layers ’ output caused by the changes of θ0 and f ( · ) using the set of replayed samples { x′m } Mm=1 generated by DG . PGMA has two training components : ( 1 ) DPG and solver S ( DPG & S for short ) , i.e. , DPG is optimized together with solver S. ( 2 ) DG . The two components have their respective objective functions ( see Sections 2.2 and 2.3 ) , and are trained alternately . This is because DPG takes input zi ( which is produced by DG ) to generate parameters , and alternating training ensures consistent convergence rates of DG and DPG . 2.2 DYNAMIC PARAMETER GENERATOR ( DPG ) AND SOLVER S IMPLEMENTATION . Several neural networks can be used to implement DPG for parameter generation , e.g. , convolutional neural network ( CNN ) , recurrent neural network ( RNN ) and multilayer perceptron ( MLP ) . The objective of this paper is not to explore all possible implementations . We found that a straightforward implementation using MLP can already achieve good results . Formally , DPG can be written as : pi = f ( zi , µ ) = σ ( wDzi + b ) ( 1 ) where σ is the activation function , and wD and b are the parameters of DPG and denoted by µ . For the implementation of solver S , several deep learning networks can be used as well . Again , a MLP is employed in this work . Each layer of the MLP is a perceptron and can be formalized by output = σ ( wxinput ) , where xinput denotes the input of the particular perceptron . We also call a perceptron a basic unit 1 . In general , for each basic unit k of the solver S , we can have a shared portion of the parameters θ0 , k and a generated portion of the parameters pi , k , i.e. , θ∗i = combine ( θ0 , pi ) = { [ w∗i , k ] } Kk=1 = { [ θ0 , k ; pi , k ] } Kk=1 ( 2 ) where combine ( . ) is concatenation and K is the number of basic units that the solver has . In our case , K is the number of hidden layers ( basic units ) of the solver MLP . It is important to note that the above is the most general case . In practice , there is no need to adapt all the basic units in the solver , but only a subset of them . In our experiments ( Section 3 ) , we will see that adapting only the final layer of the solver MLP can already achieve good results . 1This is because most of the existing networks can be regarded as the composition of multiple basic units , even for those with complex structures , e.g. , seq2seq ( Sutskever et al. , 2014 ) , R-Net ( Wang et al. , 2017 ) , Resnet ( He et al. , 2016 ) and Densenet ( Huang et al. , 2017 ) . Further , complex structures give us more freedom to decide which part of the parameters should be generated . However , one may still ask whether the proposed combination method has sufficient capacity to adjust the solver S in general because only part of the parameters are generated . Actually , pik can affect all dimensions of the output of its corresponding basic unit k : w∗i , kxinput = [ θ0 , k ; pi , k ] xinput = θ0 , kx 1 input + pi , kx 2 input ( 3 ) where x1input and x 2 input are the block vectors of xinput . pi , kx 2 input can be regarded as the bias and can adapt the output vector to any point in the vector space . To train DPG & S , we use cross entropy loss ( Lce ) . The objective function of the solver S including DPG f ( zi , µ ) and θ0 for learning each new classification task Ti is defined as : minimize µ , θ0 Lce ( S ( xi , θ∗i ) , yi ) s.t . M∑ m=1 ||R ( x′m , θ∗i ) −R ( x′m , θ∗i−1 ) || < r ( 4 ) where Lce is the cross entropy loss , R ( · ) denotes the output of all basic units ( which can be any layer or a set of layers2 ) in solver S , which we will introduce later , and θ∗i is the whole set of parameters of S ( θ∗i = combine ( pi , θ0 ) ) representing the adaptation of S. We can see that the generated replayed samples x′m are used as constraints to alleviate DPG & S ’ s forgetting . Specially , we extend the knowledge distillation loss ( Hinton et al. , 2015 ) to the general situation . That is , to keep the past learned knowledge , the output of the basic units in the solver should not change much when learning a new task with the help of the generated data . If we do not consider the activation function , the constraints in Eq . 4 can also be written as : min M∑ m=1 K∑ k=1 ||w∗i , kx′m , k −w∗i−1 , kx′m , k|| ( 5 ) where K again denotes the number of basic units and M denotes the number of replayed samples . The basic unit with a smaller k is at the relative lower layer of the solver network.3 x′m , k is the input of the kth basic unit and is calculated through forward propagation , except x′m,1 = DGD ( z sample m , θ ′ d ) which is the initial replayed sample x′m generated by DG ( before optimizing the current task Ti ) . w∗i , k is the combined parameter introduced in Section 2.2 . Two questions may be asked about Eqs . 4 and 5 . ( 1 ) Since the replayed data x′ ( denoting all x′m ) is used in Eqs . 4 and 5 , does it play the same role as the original data x from previous tasks ? ( 2 ) What is the impact of the constraints on learning the new task ? We answer the two questions now . Question 1 : The answer is positive . Since the purpose of the constraints is to maintain the learned effect of the old data , clearly , using the original old data x in the constraints is better . However , we proved the positive answer to question 1 , which is given in Appendix . Question 2 : In calculating Eq . 5 , if we stack all { x′m , k } Mm=1 ( column vectors ) into a matrix X′k , Eq . 5 can be regarded as constraining w∗i , kX ′ k to remain unchanged . If X ′ k is a row low rank matrix , w∗i , k can be trained to fit the new tasks . Otherwise , especially if X ′ k is a row full rank matrix , w ∗ i , k can not be trained and therefore can not learn new tasks . However , benefiting from dynamic parameter generation , our approach will not suffer from this problem . That is because w∗i , k is specially generated through DPG to perform well with input X′k . And the new parameters will be generated to fit new tasks while w∗i , k is able to remain unchanged .

Summary: This paper proposes a Dynamic Parameter Generator (DPG) that given a test input modifies the parameters of a classification model. They also propose to regularize the training using a Data Generator (DG) to slow down catastrophic forgetting. DG is used to constrain the training that the internal representations of data generated by DG does not rapidly change. DG removes the need for storage of data or labels.