Activity Regularization for Continual Learning

1 INTRODUCTION . Human has the ability to continuously learn , accumulate knowledge in their lifetime and be able to retrieve , consolidate previously acquired skills whenever necessary . Such kind of ability is referred as continual learning , which is a fundamental capability contributing to the skills development and specialization in humankind . Conventional machine learning , especially neural networks , however , are not capable of learning continuously from solving one task to another without forgetting the previous knowledge . This phenomenon is known as the catastrophic forgetting or catastrophic inference which has been a long-standing challenge in machine learning and AI for years . To overcome the catastrophic forgetting problem , a deep neural network on one hand should not be allowed to change too drastically in order to avoid forgetting the previous knowledge ; on the other hand , the same network also should be flexible enough to learn and acquire new knowledge . Such condition is referred to as the stability-plasticity dilemma ( Abraham & Robins , 2005 ) which has posed a great challenge in continual learning . In literature , continual learning has been extensively studied in many fields , ranging from robotics , machine learning , to neuroscience and cognitive science . In this work , we investigate the problem of continual learning from machine learning perspective . In particular , we show that catastrophic forgetting can be alleviated by optimizing the multi-task learning objective in which popular approaches such as Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , Synaptic Intelligence ( SI ) ( Zenke et al. , 2017 ) and Gradient Episodic Memory ( Lopez-Paz et al. , 2017 ) ( GEM ) can be viewed as special cases . We show that by varying the losses in the approximated multi-task learning objective , we can recover a family of regularization based approaches for continual learning . Then , we analyze the weakness of two popular approaches , EWC and GEM , and show that label noise or mistakes in the memory can result in dramatic changes in the optimal decision boundary of previous tasks , leading to catastrophic forgetting . To overcome the above problem , we propose a novel technique called “ Activity Regularization ” ( AR ) that penalizes changes in the model ’ s prediction on its learned information . AR works by utilizing a memory to store some of previous training samples ; Then , when learning a new task , AR penalizes the KL-divergence between the current model ’ s and the previous optimal model ’ s predictions on the corresponding memory . This objective is to ensure that the new model will be consistent with the previous optimal ones on the old tasks , thus avoiding catastrophic forgetting . We further develop a stochastic version of AR that randomly samples a task and apply AR in each minibatch update . Stochastic AR allows the network to be more flexible when acquiring new knowledge while still alleviating catastrophic forgetting . We evaluate our method on popular continual learning benchmarks and achieve promising results compared with state-of-the-art methods . 2 CONTINUAL LEARNING BY APPROXIMATED MULTI-TASK LEARNING . Basic notations . We first present notations used in this work . We use subscripts to denote task index , for example , T1 , D1 , are the first task and first training data respectively . Superscripts denote sample in a data set , we omit subscripts of the samples if the task is known , for example , Dt = { x ( n ) , y ( n ) } Nn=1 means data set Dt consists of n training samples . We denote a model that is trained on task t and the current model as θt and θ respectively . Finally , θk denotes the k–th parameter of θ . Consider the continual learning problem where the learner θ receives a sequence of T tasks T = { T1 , T2 , . . . , TT } , each of which consists of a training data Dt . At any given time t , only task Tt are presented to the learner θ and it has to learn to solve the current task without access to the training data of any previous tasks . The goal of continual learning is to train the learner such that it can solve the not only current task Tt but also all the previous tasks Tk < t . To avoid forgetting previous knowledge , when learning task Tt , the learner θ minimize the multitask learning objective as L̃t ( θ ) = Lt ( θ , Dt ) + λt−1Lt−1 ( θ , Dt−1 ) + . . .+ λ1L1 ( θ , D1 ) , ( 1 ) where the loss Li ( θ , Di ) is usually the negative log-likelihood − ∑ x , y∈Di log p ( y|x ; θ ) of the corresponding training data and λi regulates the importance of task Ti to all the tasks observed so far . In continual learning , at time t , we do not have access to the data Dk < t to evaluate the loss Lk < t , thus , we need to approximate the past losses Lk < t ( θ , Dk ) when minimizing Lt ( θ , Dt ) . A common choice is to employ Laplace propagation ( Eskin et al. , 2004 ) of the negative log posterior − log p ( θ|Di ) ≈ − log p ( θ∗i |Di ) + 1 2 ( θ − θ∗i ) TFi ( θ − θ∗i ) , ( 2 ) where θ∗i is a mode of the log posterior and Fi is the empirical Fisher information matrix evaluated at θ = θ∗i . If we assume an uniform prior on θ , the posterior coined with the likelihood 1 , thus by substituting eqn . 2 to 1 , the objective for training task Tt becomes L̃t ( θ ) = Lt ( θ , Dt ) + ∑ i < t λk ( θ − θ∗i ) TFi ( θ − θ∗i ) , ( 3 ) where the terms − log p ( θ∗i |Di ) are omitted because they are constant with respect to θ . According to eqn . 3 , we can minimize the negative log-likelihood of θ on the previous tasks without explicitly storing the training samples because the data distribution are already captured in the Fisher information . Eqn . 3 results in a wide range of methods for continual learning , each of which proposes a different method to approximate the Fisher information matrix . For example , Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) assumes the Fisher is diagonal and approximates it using the identity : Ft ≈ diag ( N∑ n=1 ( ∇θ log p ( y ( n ) t |x ( n ) t ; θ ) ) 2∣∣∣∣ θ=θ∗t ) ( 4 ) Instead of using the Fisher ’ s identity , Synaptic Intelligence ( SI ) ( Zenke et al. , 2017 ) directly measures the sensitivity of a parameter θkt to the loss function at task Tt through out the whole training trajectory . Zenke et al . ( 2017 ) also shows that this estimation is equivalent to measuring the full Fisher matrix on whole data set under certain choices of the loss function . Another approach to achieve the approximated multi-task learning objective in eqn . 1 is based on the observation that at the beginning of task Tt , we initialize θ = θ∗t−1 , whose value already minimized the lost L̃t−1 . From this , if we can consider eqn . 1 as minimizing the loss Lt ( θ ) with the constraints that previous losses are not allowed to increase after every mini-batch update , which is the GEM objective . In particular , GEM optimize the following problem Minimizeθ , Dt Lt ( θ ) subject to Lk ( θ , Mk ) ≤ Lk ( θ∗t−1 , Mk ) , ∀k < t , ( 5 ) 1With constant prior : p ( θ ) = const , ∀θ , by Bayes ’ s rule : p ( θ|Di ) ∝ p ( θ ) p ( Di|θ ) ∝ p ( Di|θ ) . where the loss Lk is evaluated at the memoryMk that stores some of the training samples from task Tk . GEM further converts the constraints in eqn . 5 into the gradient constraints of the losses and solve them using quadratic programming . Finally , the type of losses Lk < t can be flexibly changed to other loss rather than the negative loglikelihood . For example , Learning without Forgetting ( LwF ) ( Li & Hoiem , 2017 ) proposes a form of distillation loss by minimizing the KL-divergence of past models and current model on the newly observed tasks . By changing the losses Lk < t , we can recover different solutions for continual learning in the literature . 3 ACTIVITY REGULARIZATION FOR CONTINUAL LEARNING . 3.1 LIKELIHOOD MATCHING IN APPROXIMATED MULTI-TASK LEARNING . While EWC and GEM are two popular methods for continual learning , we argue that they are sensitive to misclassified samples selected in the memory or used to estimate the Fisher information . Let us consider a scenario of learning two tasks T1 and T2 where we store the first optimal model θ∗1 and use a sample { x1 , y1 } to estimate the Fisher information or storing in the memory . For EWC , by applying eqn . 2 with this scenario , we have : ( θ − θ∗1 ) TF1 ( θ − θ∗1 ) ≈ log p ( θ∗1 |x1 ) − log p ( θ|x1 ) , ( 6 ) again we assume uniform prior on θ , thus minimizing LHS of eqn . 6 is equivalent to finding θ that has the same log-likelihood as θ∗1 on x1 . For GEM , the constraint is to penalize whenever θ that has higher negative log-likelihood than θ∗1 on x1 . Now assume y1 = [ 1 , 0 , 0 ] and the predictions of the two models θ∗1 and θ are [ 0.1 , 0.9 , 0.1 ] and [ 0.1 , 0.1 , 0.9 ] , that is , both models make mistakes on x1 . As a result , θ∗1 and θ have the same log-likelihood value on x1 , and thus there are no penalty on θ in both EWC and GEM . However , these two models are very different from each other since they make different mistakes on x1 . In general , if we have too many misclassified samples or label noise , the decision boundary on the previous task might be altered despite minimizing the EWC or GEM objective . This phenomenon happens based on an assumption that we usually do not train the model to have 0 error on the training set to avoid overfitting and we only use a relatively small data portion to estimate the Fisher or store as memory . Therefore , EWC and GEM might not correctly penalize the current model from changing the past tasks ’ optimized decision boundaries . 3.2 ACTIVITY REGULARIZATION FOR CONTINUAL LEARNING . To tackle the aforementioned problem , we propose to minimize the KL-divergence between the current model and the previous optimal models on the corresponding tasks . This can also be interpreted as distilling the knowledge from θ∗i to θ on the corresponding data Di . In particular , for each task Ti , we store some of the samples xi and the prediction of the optimal model fθ∗i ( xi ) in a memoryMi ; then the Activity Regularization ( AR ) objective for task Tt is L̃t ( θ ) = Lt ( θ ) + ∑ i < t ∑ x∈Mi λiKL ( 1 τ fθ∗i ( x ) ∥∥∥∥ 1τ fθ ( x ) ) , ( 7 ) where τ is the temperature used in softmax outputs . Our goal is to ensure that the new model θ behaves similarly with θ∗t on the previous task Tt , thus we call this constraint as activity regularization . Algorithm 1 gives a summary of the proposed Activity Regularization based algorithm for continual learning . We refer to this algorithm as “ Deterministic Activity Regularization ” ( DAR ) which imposes regularization constraints for all the previous tasks at every learning iteration . Remark . Similar to approximated multi-task learning methods , DAR employs a regularization constraint on every of the past tasks when learning a new one , which requires calculating the KLdivergence resulting in an additional forward pass on the memory , and thus results in additional computational cost when training the models . Moreover , the number of regularizers for each task is equal to the number of previously observed tasks , which may considerably restrict the model ’ s plasticity to acquire new knowledge . That is , by enforcing too many constraints on the current model for reducing forgetting , we sacrifice its ability to learn new information . Algorithm 1 Deterministic Activity Regularization ( DAR ) algorithm for Continual Learning . 1 : Initialize the model and memory : θ , M0 ← ∅ , select temperature τ 2 : for t = 1 , . . . , T do 3 : Observe the data set Dt 4 : Mt ← add random samples from Dt to the memory . 5 : for k = 1 , . . . , niter do 6 : Sample D = { x ( n ) , y ( n ) } Nn=1 a batch from training data Dt 7 : gθ ← ∇θ ∑ x , y∈D Lt ( θ , ( x , y ) ) + ∑ i < t ∑ x∈Mi λiKL ( 1 τ fθ∗i ( x ) ∥∥∥∥ 1τ fθ ( x ) ) 8 : θ ← θ − α · SGD ( θ , gθ ) 9 : return θ

Summary: The authors proposed a new regularizer for continual learning to tackle the catastrophic forgetting problem. The proposed method minimizes the KL-divergence between the prediction of previous models and current models on the stored samples of previous tasks. The idea is straightforward and sounds technical. Experiments show the effectiveness of the methods compared to state-of-the-art. Although the idea sounds interesting and the experiments look promising, the novelty of the paper seems to be limited. In addition, the experiments setting needs to be improved as well. In the following, you have detailed comments.