Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation

1 INTRODUCTION . It is well-known that neural networks ( NNs ) suffer from catastrophic forgetting ( CF ) ( McCloskey & Cohen , 1989 ) , which refers to the phenomenon that when learning a sequence of tasks , the learning of each new task may cause the NN to forget the models learned for the previous tasks . Without solving this problem , an NN is hard to adapt to lifelong or continual learning , which is important for AI . Problem Statement : Given a sequence of supervised learning tasks T = ( T1 , T2 , . . . , TN ) , we want to learn them one by one in the given sequence such that the learning of each new task will not forget the models learned for the previous tasks . In recent years , many approaches ( often called continual learning ) have been proposed to lessen the effect of CF ( Chen & Liu , 2018 ; Parisi et al. , 2018 ) , e.g. , dynamically expandable network ( DEN ) ( Yoon et al. , 2018 ) , learning without forgetting ( LWF ) ( Li & Hoiem , 2016 ) ) , elastic weight consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , incremental moment matching ( IMM ) ( Lee et al. , 2017 ) , gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) , generative replay ( GR ) ( Shin et al. , 2017 ) , etc . These existing studies except DEN and LWF focused on learning a model parameterized by a single joint set of parameters θ∗ which is assumed to work well for all tasks . We call them joint parameterization ( JP ) methods . DEN and LWF require constantly increasing the number of parameters and thus can result in a huge and complex model . JP methods , however , suffer from accuracy deterioration . Assume we have two tasks A and B that need to be learned sequentially . Let θ∗A and θ ∗ B be the optimal parameters for Task A and Task B respectively when each of them is learned individually . Let θ∗ be the joint parameters learned by the existing approaches to perform Tasks A and B in sequence . Inevitably , θ∗ is different from θ∗A and/or θ∗B and is highly likely to result in more errors for the two tasks than θ ∗ A and θ ∗ B individually . ∗Equal contribution †Corresponding author In this paper , we propose a different approach , called Parameter Generation and Model Adaptation ( PGMA ) , to dealing with CF and to significantly reducing accuracy deterioration . PGMA does not learn a joint parameter set θ∗ . Instead , it learns a parameter generator f ( · ) and a shared parameter set θ0 . The key idea of PGMA is as follows : The overall classification network , called the solver S , has two disjoint subsets/parts of parameters . The first subset is θ0 , which is shared by all tasks learned so far . The second subset is just a place holder H that will be filled by parameters generated by f ( · ) for each test instance . That is , given a test instance , a set of parameters will be generated by the learned parameter generator f ( · ) to replace H . Solver S then combines θ0 and the generated parameters for the test instance to classify it . Clearly , unlike existing JP approaches , we do not have a network with a set of fixed parameters for S. The idea is that θ0 contains the common features of all tasks , and the generated parameters for each test instance adapt S for each test instance in order to classify it . Since f ( · ) and the shared parameters θ0 will change during training for each new task , forgetting can occur for previous tasks . To deal with it , in training f ( · ) and S for each task Ti , in addition to the training data of Ti , a small number of replayed samples will be generated by a data generation network for the previous tasks to ensure that the knowledge learned for previous tasks remain stable/unforgotten . Compared with the existing generative replay methods , our method does not need labels for the replayed data because we use the replayed data only to constrain the training of S and f ( · ) rather than to treat them as surrogates of labeled training data from previous tasks . Further , since in the existing replay methods , the labels are produced by the current learned network itself , the labels can be noisy and biased , which may result in errors being accumulated and propagated to subsequent tasks . Our method does not have this problem . Apart from reducing the effect of accuracy deterioration of the existing approaches , the proposed approach also has some other advantages . First , no parameter increase or network expansion is needed to learn new tasks . Second , no previous data needs to be stored to enable the system to remember the previously learned models or knowledge . Experiments conducted using two image datasets ( MNIST and CIFAR-10 ) and two text datasets ( DBPedia ontology ( Lehmann et al. , 2015 ) and THUCNews ( Li et al. , 2006 ) ) show that the proposed approach PGMA works well for different scenarios and different types of datasets , and outperforms the existing strong baselines markedly . 2 PROPOSED PGMA FRAMEWORK . Let the sequence of supervised learning tasks be T = ( T1 , T2 , . . . , TN ) . Each task Ti is represented by Ti = { xij , yij |j ∈ ( 1 , . . . , Ni ) } , where xij is the j-th example/sample of Ti , and yij is its label . We use ( xt , yt ) to denote a test instance/example . To simplify the notation , we will just use ( xi , yi ) to denote a training example from task Ti , omitting the second subscript j . Note that in the paper , we use the terms example , sample , and instance interchangeably . The proposed parameter generation and model adaptation ( PGMA ) architecture has three main components : • Solver S : It is the main classification model . As mentioned in Section 1 , the parameter set of S consists of two subsets , θ0 that is shared by all tasks ( and instances ) and H , a parameter place holder , which will be replaced by the generated parameters set pi ( or pt ) for each training ( or testing ) example xi ( or xt ) . pi ( or pt ) basically serves to adapt the solver S to classify the example in training or testing . This is the key idea of our approach . We adopt this parameter split as several studies ( Yoon et al. , 2018 ; Li & Hoiem , 2016 ; Kirkpatrick et al. , 2017 ) have shown that only part of parameters of a neural network needs to be adjusted when learning a new task . See Section 2.2 • Dynamic Parameter Generator ( DPG ) f ( · ) : It takes the embedding zi ( or zt ) of each input training ( or testing ) example xi ( or xt ) to generate the parameters pi ( or pt ) for solver S. Note that we use zi ( or zt ) rather than the raw data xi ( or xt ) because the raw data ’ s dimension can be very high . The embedding zi ( or zt ) as a low dimensional dense representation reduces the mapping space for DPG and thus reduces the difficulty in its parameter generation . See Section 2.2 . • Data Generator ( DG ) : It has two functions . The main function is to generate a set of replayed data or samples { x′m } Mm=1 using its decoder DGD for previous tasks to deal with catastrophic forgetting . The other function is to generate the embedding zi ( or zt ) of each input training ( or testing ) example xi ( or xt ) using its encoder DGE . See Section 2.3 . 2.1 OVERALL APPROACH OF PGMA . We work backward by describing testing first before training . Given a test instance xt , DGE first generates its embedding zt , which is fed to f ( · ) to generate a set of parameters pt . Solver S then takes xt as input and uses the trained/learned shared parameters θ0 and pt to classify xt . θ0 contains the common features of all tasks learned so far . pt simply adapts S for xt in order to classify xt . For training , the pipeline of the proposed PGMA framework is shown in Figure 1 . Given a new task Ti with its data ( xi , yi ) , solver S and DPG f ( · ) are jointly trained to learn Ti and also not to forget the previously learned tasks . In each iteration , a set of parameters pi is generated by the current DPG f ( zi , µ ) for each training instance xi ( zi being its embedding ) , where µ is the set of parameters of the DPG network , which is trained . In training the solver S and DPG for the new task Ti , both f ( · ) and the shared parameters θ0 will change , which can cause forgetting in DPG for previous tasks . To keep DPG remembering the acquired knowledge for previous tasks , we minimize the variation of certain layers ’ output caused by the changes of θ0 and f ( · ) using the set of replayed samples { x′m } Mm=1 generated by DG . PGMA has two training components : ( 1 ) DPG and solver S ( DPG & S for short ) , i.e. , DPG is optimized together with solver S. ( 2 ) DG . The two components have their respective objective functions ( see Sections 2.2 and 2.3 ) , and are trained alternately . This is because DPG takes input zi ( which is produced by DG ) to generate parameters , and alternating training ensures consistent convergence rates of DG and DPG . 2.2 DYNAMIC PARAMETER GENERATOR ( DPG ) AND SOLVER S IMPLEMENTATION . Several neural networks can be used to implement DPG for parameter generation , e.g. , convolutional neural network ( CNN ) , recurrent neural network ( RNN ) and multilayer perceptron ( MLP ) . The objective of this paper is not to explore all possible implementations . We found that a straightforward implementation using MLP can already achieve good results . Formally , DPG can be written as : pi = f ( zi , µ ) = σ ( wDzi + b ) ( 1 ) where σ is the activation function , and wD and b are the parameters of DPG and denoted by µ . For the implementation of solver S , several deep learning networks can be used as well . Again , a MLP is employed in this work . Each layer of the MLP is a perceptron and can be formalized by output = σ ( wxinput ) , where xinput denotes the input of the particular perceptron . We also call a perceptron a basic unit 1 . In general , for each basic unit k of the solver S , we can have a shared portion of the parameters θ0 , k and a generated portion of the parameters pi , k , i.e. , θ∗i = combine ( θ0 , pi ) = { [ w∗i , k ] } Kk=1 = { [ θ0 , k ; pi , k ] } Kk=1 ( 2 ) where combine ( . ) is concatenation and K is the number of basic units that the solver has . In our case , K is the number of hidden layers ( basic units ) of the solver MLP . It is important to note that the above is the most general case . In practice , there is no need to adapt all the basic units in the solver , but only a subset of them . In our experiments ( Section 3 ) , we will see that adapting only the final layer of the solver MLP can already achieve good results . 1This is because most of the existing networks can be regarded as the composition of multiple basic units , even for those with complex structures , e.g. , seq2seq ( Sutskever et al. , 2014 ) , R-Net ( Wang et al. , 2017 ) , Resnet ( He et al. , 2016 ) and Densenet ( Huang et al. , 2017 ) . Further , complex structures give us more freedom to decide which part of the parameters should be generated . However , one may still ask whether the proposed combination method has sufficient capacity to adjust the solver S in general because only part of the parameters are generated . Actually , pik can affect all dimensions of the output of its corresponding basic unit k : w∗i , kxinput = [ θ0 , k ; pi , k ] xinput = θ0 , kx 1 input + pi , kx 2 input ( 3 ) where x1input and x 2 input are the block vectors of xinput . pi , kx 2 input can be regarded as the bias and can adapt the output vector to any point in the vector space . To train DPG & S , we use cross entropy loss ( Lce ) . The objective function of the solver S including DPG f ( zi , µ ) and θ0 for learning each new classification task Ti is defined as : minimize µ , θ0 Lce ( S ( xi , θ∗i ) , yi ) s.t . M∑ m=1 ||R ( x′m , θ∗i ) −R ( x′m , θ∗i−1 ) || < r ( 4 ) where Lce is the cross entropy loss , R ( · ) denotes the output of all basic units ( which can be any layer or a set of layers2 ) in solver S , which we will introduce later , and θ∗i is the whole set of parameters of S ( θ∗i = combine ( pi , θ0 ) ) representing the adaptation of S. We can see that the generated replayed samples x′m are used as constraints to alleviate DPG & S ’ s forgetting . Specially , we extend the knowledge distillation loss ( Hinton et al. , 2015 ) to the general situation . That is , to keep the past learned knowledge , the output of the basic units in the solver should not change much when learning a new task with the help of the generated data . If we do not consider the activation function , the constraints in Eq . 4 can also be written as : min M∑ m=1 K∑ k=1 ||w∗i , kx′m , k −w∗i−1 , kx′m , k|| ( 5 ) where K again denotes the number of basic units and M denotes the number of replayed samples . The basic unit with a smaller k is at the relative lower layer of the solver network.3 x′m , k is the input of the kth basic unit and is calculated through forward propagation , except x′m,1 = DGD ( z sample m , θ ′ d ) which is the initial replayed sample x′m generated by DG ( before optimizing the current task Ti ) . w∗i , k is the combined parameter introduced in Section 2.2 . Two questions may be asked about Eqs . 4 and 5 . ( 1 ) Since the replayed data x′ ( denoting all x′m ) is used in Eqs . 4 and 5 , does it play the same role as the original data x from previous tasks ? ( 2 ) What is the impact of the constraints on learning the new task ? We answer the two questions now . Question 1 : The answer is positive . Since the purpose of the constraints is to maintain the learned effect of the old data , clearly , using the original old data x in the constraints is better . However , we proved the positive answer to question 1 , which is given in Appendix . Question 2 : In calculating Eq . 5 , if we stack all { x′m , k } Mm=1 ( column vectors ) into a matrix X′k , Eq . 5 can be regarded as constraining w∗i , kX ′ k to remain unchanged . If X ′ k is a row low rank matrix , w∗i , k can be trained to fit the new tasks . Otherwise , especially if X ′ k is a row full rank matrix , w ∗ i , k can not be trained and therefore can not learn new tasks . However , benefiting from dynamic parameter generation , our approach will not suffer from this problem . That is because w∗i , k is specially generated through DPG to perform well with input X′k . And the new parameters will be generated to fit new tasks while w∗i , k is able to remain unchanged .

Summary: This paper proposes a method for continual learning. The model has three components: a) a data generator to be used at training time to replay past examples, b) a parameter generator that takes the input observation to produce parameters for c) the actual classifier. The authors demonstrate the method on simple datasets with a stream of 2 or 3 tasks.