Revisiting Gradient Episodic Memory for Continual Learning

1 INTRODUCTION . Catastrophic forgetting ( McCloskey & Cohen , 1989 ) is a common phenomenon in deep learning that the model performance over past tasks can be harmed by the training process of a current task when we employ one single neural network to learn consecutive tasks . Continual learning ( Ring , 1994 ) , also known as lifelong learning ( Thrun , 1994 ) , is a specific research field in AI that focuses on avoiding or alleviating catastrophic forgetting . It is well known that humans and large primates can continually learn new skills and accumulate knowledge throughout their lifetime ( Fagot & Cook , 2006 ) . Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) is a significant method for continual learning . The basic idea is to modify each gradient of the current task such that the losses at the memories ( data examples ) from previously solved tasks will not increase , hence allowing positive backward knowledge transfer . The problem is formulated as a Quadratic Program with inequality constraints , and then solving its dual problem provides an efficient computation for a small number of dual variables . GEM outperforms two state-of-the-art methods , Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) and iCARL ( Rebuffi et al. , 2017 ) on variants of the MNIST and CIFAR100 datasets . Despite the elegance of its mathematical formulation and the computational efficiency in solving the dual problem , the GEM model has following drawbacks for practical use . 1 . The episodic memory is simply populated with the last m = M/T examples from each task in their current implementation , where M is the total budget size of memory locations for totally T tasks . The authors of GEM mentioned that better memory update strategies could be employed to obtain a representative set of examples for past tasks , such as building a coreset per task ( Lucic et al. , 2018 ) . 2 . The inequality constraints of gradients appear to be quite restrictive , requiring that the gradient update can not increase any loss of all previously solved tasks . When two tasks are competing or conflicting in the gradient update directions , especially with a large number of past tasks , these constraints have to be violated and the feasible solution might be null . 3 . The gradient constraints only assure to avoid catastrophic forgetting , but might not promote positive backward knowledge transfer in some scenarios . For instance , we may obtain a gradient vector orthogonal to all gradients of previous tasks in the constrained optimization problem . In this case , GEM is greedy in minimizing the loss of the current task , overlooking the need of backward transfer to previous tasks . An ideal model for continual learning should consider a better trade-off between optimizing performance of the current task and backward transfer to previously solved tasks . To address these issues , in this paper we propose three techniques that improve the performance of GEM . These techniques come from successful practice in Support Vector Machines ( SVMs ) and deep learning approaches . In concrete , we use the notion of margins to select representative data points as episodic memories . We also leverage slack variables to allow violations of the constraints , and aggressively encourage backward knowledge transfer to previous tasks . Experiments on MNIST and CIFAR100 datasets corroborate the improvements of our techniques over the original GEM model . Also our methods increase little overhead in computational efforts . The paper is organized as follows . In Section 2 , we first survey several recent methods on continual learning and explain why we focus on improving GEM . Section 3 offers a brief review of the GEM algorithm . In Section 4 , we elaborate our three improvements and discuss the underlying motivations . Section 5 presents the experimental results and comparison analysis , and Section 6 concludes the paper . 2 RELATED WORK . According to Xu & Zhu ( 2018 ) , recent methods for continual learning can be roughly classified into two large categories , distinguished by whether the network architecture changes through a continuum of tasks . The first category attempts to modify the network architecture to alleviate catastrophic forgetting . Fernando et al . ( 2017 ) proposed a neural network model called PathNet , in which there are ten or twenty modules in each layer and for one task only three or four modules are dynamically chosen in each layer using an evolutionary approach . Progressive neural networks ( Rusu et al. , 2016 ) aims at accommodate the new coming task by dynamically expand the architecture with a fixed size of nodes or layers , leading to an extremely large network structure with high redundancy . In Dynamically Expandable Network ( DEN ) ( Yoon et al. , 2017 ) this issue is slightly alleviated by employing group sparsity regularization when expanding the networks . However many hyperparameters are involved , including various regularization and thresholds , and careful tuning is challenging . Xu & Zhu ( 2018 ) proposed Reinforced Continual Learning ( RCL ) approach that searches for the best neural architecture for each coming task via reinforcement learning strategies . Like other expandable networks , this approach still suffers from higher model complexity than non-expandable networks . Also reducing its training time is a major concern . Another category maintain a fixed network architecture with large capacity through learning . Oquab et al . ( 2014 ) proposed a weight transfer method by freezing early layers while cloning and finetuning later layers on the new task . In “ synaptic ” memory methods ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) , learning rates are adjusted to minimize updates in weights important for previous tasks . Lee et al . ( 2017 ) proposed to incrementally match the moment of the posterior distribution of the neural network trained on the first and the second task , respectively . In Zeng et al . ( 2018 ) an Orthogonal Weights Modification ( OWM ) was proposed to ensure that new learning processes will not interfere with previously learned tasks , but can not enable backward knowledge transfer . In Shin et al . ( 2017 ) ; Kemker & Kanan ( 2018 ) previously acquired knowledge can be acquired by replaying of pseudo-data generated by Generative Adversarial Networks ( GANs ) or generative autoencoders , without revisiting actual past data for the coming task . Several “ episodic ” memory approaches ( Jung et al. , 2016 ; Li & Hoiem , 2016 ; Triki et al. , 2017 ; Rebuffi et al. , 2017 ) aim at storing and replaying examples from past tasks to keep predictions invariant via distillation ( Hinton et al. , 2015 ) . GEM ( Lopez-Paz & Ranzato , 2017 ) belongs to this class of episodic memory approaches , but unlike prior work , allows for backward transfer . Comparing to these recent approaches on continual learning , GEM is relatively lightweight , computationally efficient and capable of achieving high accuracies . However , there is still room to improve GEM with respect to the limitations stated in Section 1 . 3 A BRIEF REVIEW OF GEM . Most of the literature on continual learning ( Rusu et al. , 2016 ; Fernando et al. , 2017 ; Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ) focused on a traditional setting : ( 1 ) the number of tasks is small ; ( 2 ) the number of examples per task is large ; ( 3 ) the learner allows for several passes over the examples per task ; ( 4 ) the average performance across all tasks serves as the only metric . In contrast , a “ more human-like ” setting was considered in GEM : ( 1 ) the number of tasks is large ; ( 2 ) the number of training examples is small for each task ; ( 3 ) the learner observes the examples of per task only once ; ( 4 ) knowledge transfer in backward and forward directions is assessed . Therefore following this new setting , the learner will observe the continuum of data in a manner of example by example ( x1 , t1 , y1 ) , . . . , ( xi , ti , yi ) , . . . , ( xn , tn , yn ) , where each triplet ( xi , ti , yi ) contains a input vector , a task index , and a target vector . Note that any order on the tasks can be allowed . The optimizing problem of GEM ( Lopez-Paz & Ranzato , 2017 ) is min w 1 2 ‖g − w‖22 s.t . 〈w , gk〉 ≥ 0 , ∀k < t , ( 1 ) where g is the gradient vector calculated on the batch data of the current task , and gk here is the gradient on the episodic memory Mk of the past task k. Note that the episodic memory Mk stores a subset of the observed examples from task k. If the gradient update g of the current batch fulfills all the constraints , the learner directly uses it to update like the usual gradient descent method . Otherwise , we should find a projection of g in L2-norm distance to satisfy any constraint related each previous task , and then use the solution w to update the network weights . We obtain a standard form of Quadratic Program ( QP ) by expanding the squared objective and discarding the constant term gT g : min w 1 2 wTw − gTw s.t . Gw ≥ 0 , ( 2 ) where G = ( g1 , . . . , gt−1 ) . The primal QP problem can be translated into its dual problem ( Dorn , 1960 ) , which is the final optimizing problem : min α 1 2 αTGGTα + ( Gg ) Tα s.t . α ≥ 0 , ( 3 ) We can recover the projected gradient update as w∗ = GTα∗ + g where α∗ is the solution to the dual problem . Note that solving the dual problem is computationally efficient since there are only t− 1 variables in α , much smaller than the number of weights in a neural network . 4 OUR IMPROVEMENTS . We propose three improvements to GEM : a technique to choose examples which can represent the past tasks better — support examples ; a technique to tolerant some slight violation in gradient constraints on past tasks — soft gradient constraints ; a technique to aggressively encourage gradient descend over previous tasks — positive backward transfer . 4.1 SUPPORT EXAMPLES . The first improvement is inspired from the idea of margins in Support Vector Machines ( SVMs ) ( Williams , 2003 ) . Due to the KKT theorem , a small set of support vectors obtained from a Quadratic Program determines the decision boundary of SVM for binary classification . In the context of continual learning , the decision is crucial to identify what samples should be chosen in the episodic memory , since we can only store a limited budget of representative samples from previous task . In SVMs support vectors arise naturally as a solution from the QP problem of SVM ; however such optimization problem does not apply to neural networks . In GEM a simple method is adopted , that is , the last m examples of task t are stored into the episodic memory Mt . Also the examples are shuffled randomly , which means that the storage is essentially random and ad hoc . In this work , we propose an idea of choosing samples based on the notion of margins . We assume that the problem to solve here is multi-class classification , so the prediction from the network ’ s forward propagation can be available . Suppose the output prediction h ( x , t , y ) on a triplet example ( x , t , y ) is computed through a softmax activation function , with value falling in the range [ 0 , 1 ] . The following gives the definition of margins : margin : = h ( x , t , y ) −max y′ 6=y h ( x , t , y′ ) . ( 4 ) When the classifier h makes a correct prediction , the margin is positive . Otherwise when h misclassifies the example , we have margin ≤ 0 . A margin value indicates the confidence of the prediction of an triplet example : larger the margin is in magnitude , more confidence we have in the prediction . In order to alleviate catastrophic forgetting , the most important thing is to ensure that the examples of past tasks can be correctly classified to retain the prediction performance . Usually we have a much smaller budget for the episodic memory than the number of examples per task . For instance , in experiment on CIFAR100 , the number of total samples is 2500 per task , while the episodic memory is 256 per task . On the one hand , we can not afford to assign the memory to the examples whose margins are smaller than 0 , which means they could not be classified correctly in the past task or even these examples are outliers . On the other hand , for the examples whose margins are relatively large , indicating that the network can classify them with a strong confidence , these example should not be put into the memory either . We define the Support Examples as those examples being classified correctly but with a limited confidence . In experiments we simply use a confidence interval to select the support examples ( we specify the interval as hyperparameters per task ) . In this way , the episodic memory can be used more efficiently , and at the same time , we argue that the chosen support examples can represent the past tasks better .

Summary: Paper proposes three improvements upon the gradient episodic memory (GEM) method [Lopez-Paz,2017]. The three improvements address 1. A way of selecting which exemplars to store (originally random), 2. The strictness of the constraints is loosened by considering slack vectors, 3. The update is improved by promoting positive backward transfer, not only limiting the gradient to the constraints imposed by exemplars of previous tasks, but aiming to improve also for previous tasks by preferring gradients which have high cosine similarity with the gradients on exemplars. Results on CIFAR 100 and MNIST permuted are presented. Noteworthy is the improved backward transfer on CIFAR 100 when compared with GEM.