AFEC: Active Forgetting of Negative Transfer in Continual Learning

1 Introduction . The ability to continually learn numerous tasks from dynamic data distributions is critical for deep neural networks , which needs to remember the old tasks by avoiding catastrophic forgetting [ 18 ] while effectively learn each new task by improving forward knowledge transfer [ 17 ] . Due to the dynamic data distributions , forward knowledge transfer might be either positive or negative , and is difficult to determine without accessing to the old training samples . If the forward knowledge transfer is negative , i.e. , learning a new task from the old knowledge is worse than learning the new task on a randomly-initialized network [ 36 , 17 ] , then precisely remembering the old tasks will severely interfere with the learning of the new task , thus decreasing the performance of continual learning . ⇤Corresponding author : J. Zhu and Y. Zhong . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . By contrast , biological neural networks can effectively learn a new experience on the basis of remembering the old experiences , even if they conflict with each other [ 18 , 5 ] . This advantage , called memory flexibility , is achieved by active forgetting of the old knowledge that interferes with the learning of a new experience [ 29 , 5 ] . The latest data suggested that the underlying mechanism of biological active forgetting is to regulate the learning-triggered synaptic expansion and synaptic convergence ( Fig . 1 , see Appendix A for neuroscience background and our biological data ) . Specifically , the biological synapses expand additional functional connections to learn a new experience together with the previously-learned functional connections ( synaptic expansion ) . Then , all the functional connections are pruned to the amount before learning ( synaptic convergence ) . Inspired by the biological active forgetting , we propose to actively forget the old knowledge that interferes with the learning of new tasks without significantly increasing catastrophic forgetting , so as to benefit continual learning . Specifically , we adopt Bayesian continual learning and actively forget the posterior distribution that absorbs all the information of the old tasks with a forgetting factor to better learn each new task . Then , we derive a novel method named Active Forgetting with synaptic Expansion-Convergence ( AFEC ) , which is formally consistent with the underlying mechanism of biological active forgetting at synaptic structures . Beyond regular weight regularization approaches [ 12 , 1 , 35 , 2 ] , which selectively penalize changes of the important parameters for the old tasks , AFEC dynamically expands parameters only for each new task to avoid potential negative transfer from the main network , while the forgetting factor regulates a penalty to selectively merge the main network parameters with the expanded parameters , so as to learn a better overall representation of both the old tasks and the new task . We extensively evaluate AFEC on continual learning of CIFAR-10 regression tasks , a variety of visual classification tasks , and Atari reinforcement tasks [ 10 ] , where AFEC achieves the state-of-theart ( SOTA ) performance . We empirically validate that the performance improvement results from effectively improving the learning of new tasks without increasing catastrophic forgetting . Further , AFEC can be a plug-and-play method that significantly boosts the performance of representative continual learning strategies , such as weight regularization [ 12 , 1 , 35 , 2 ] and memory replay [ 21 , 9 , 6 ] . Our contributions include : ( 1 ) We draw inspirations from the biological active forgetting and propose a novel approach to actively forget the old knowledge that interferes with the learning of new tasks for continual learning ; ( 2 ) Extensive evaluation on a variety of continual learning benchmarks shows that our method effectively improves the learning of new tasks and achieves the SOTA performance in a plug-and-play way ; and ( 3 ) To the best of our knowledge , we are the first to model the biological active forgetting and its underlying mechanism at synaptic structures , which suggests a potential theoretical explanation of how the underlying mechanism of biological active forgetting achieves its function of forgetting the past and continually learning conflicting experiences [ 29 , 5 ] . 2 Related Work . Continual learning needs to minimize catastrophic forgetting and maximize forward knowledge transfer . Existing work in continual learning mainly focuses on mitigating catastrophic forgetting . Representative approaches include : weight regularization [ 12 , 1 , 35 , 2 ] , which selectively penalizes changes of the previously-learned parameters ; parameter isolation [ 24 , 10 ] , which allocates a dedicated parameter subspace for each task ; and memory replay [ 21 , 28 , 9 ] , which approximates and recovers the old data distributions through storing old training data , their embedding or learning a generative model . In particular , Adaptive Group Sparsity based Continual Learning ( AGS-CL ) [ 10 ] proposed to regularize the group sparsity with separation of the important nodes for the old tasks to prevent catastrophic forgetting , which takes advantages of weight regularization and parameter isolation , and achieved the SOTA performance on various continual learning benchmarks . Several studies suggested that forward knowledge transfer is critical for continual learning [ 17 , 4 ] , which might be either positive or negative due to the dynamic data distributions . Although it is highly nontrivial to mitigate potential negative transfer while overcoming catastrophic forgetting , the efforts that specifically consider this challenging issue are limited . [ 3 ] developed a method to mitigate negative transfer when fine-tuning tasks on a pretrained network . For the scenario where the old tasks can be learned again , [ 26 ] learned an additional active column to better exploit potential positive transfer . [ 22 ] tried to maximize transfer and minimize interference from a memory buffer containing a few old training data . Similarly , [ 6 , 16 , 33 ] attempted to more effectively balance stability and plasticity with the memory buffer in class incremental learning , while [ 32 ] stored and updated the old features . By contrast , since pretraining or old training data might not be available in continual learning , we mainly focus on a more restrict yet realistic setting that a neural network incrementally learns a sequence of tasks from scratch , without storing old training data . Further , we extend our method to the scenarios where pretraining or memory buffer can be used , as well as the scenarios other than classification tasks , such as regression tasks and reinforcement tasks . 3 Method . In this section , we first describe the framework of Bayesian continual learning [ 12 , 20 ] . Under such framework , we propose an active forgetting strategy , which is formally consistent with the underling mechanism of biological active forgetting at synaptic structures . 3.1 Basics of Bayesian Continual Learning . Continual learning needs to remember the old tasks and learn each new task effectively . Let ’ s consider a simple case that a neural network with parameter ✓ continually learns two independent tasks , task A and task B , from their training datasets DtrainA and D train B [ 12 ] . The training dataset of each task is only available when learning the task . Bayesian Learning : After learning DtrainA , the posterior distribution p ( ✓|DtrainA ) = p ( DtrainA |✓ ) p ( ✓ ) p ( DtrainA ) incorporates the knowledge of task A . Then , we can get the predictive distribution for the test data of task A : p ( DtestA |DtrainA ) = Z p ( DtestA |✓ ) p ( ✓|DtrainA ) d✓ . As the posterior p ( ✓|DtrainA ) is generally intractable ( except very special cases ) , we must resort to approximation methods , such as the Laplace approximation [ 12 ] or other approaches of approximate inference [ 20 ] . Let ’ s take Laplace approximation as an example . If p ( ✓|DtrainA ) is smooth and majorly peaked around the mode ✓⇤A = argmax✓ log p ( ✓|DtrainA ) , we can approximate it with a Gaussian distribution whose mean is ✓⇤A and covariance is the inverse Hessian of the negative log posterior ( detailed in Appendix B.1 ) . Bayesian Continual Learning : Next , we want to incorporate the new task into the posterior , which uses the posterior p ( ✓|DtrainA ) as the prior of the next task [ 12 ] : p ( ✓|DtrainA , DtrainB ) = p ( DtrainB |✓ ) p ( ✓|DtrainA ) p ( DtrainB ) . ( 1 ) Then we can test the performance of continual learning by evaluating p ( DtestA , D test B |DtrainA , DtrainB ) = Z p ( DtestA , D test B |✓ ) p ( ✓|DtrainA , DtrainB ) d✓ . ( 2 ) Similarly , p ( ✓|DtrainA , DtrainB ) can be approximated by a Gaussian using Laplace approximation whose mean is the mode of the posterior : ✓⇤A , B = argmax ✓ log p ( ✓|DtrainA , DtrainB ) ( 3 ) = argmax ✓ log p ( DtrainB |✓ ) + log p ( ✓|DtrainA ) log p ( DtrainB ) | { z } const. . ( 4 ) This MAP estimation is also known as the Elastic Weight Consolidation ( EWC ) [ 12 ] : LEWC ( ✓ ) = LB ( ✓ ) + 2 X i FA , i ( ✓i ✓⇤A , i ) 2 , ( 5 ) where LB ( ✓ ) is the loss for task B and i is the label of each parameter . FA is the Fisher Information matrix ( FIM ) of ✓⇤A on D train A ( the computation is detailed in Appendix B.1 ) , which indicates the “ importance ” of parameter i for task A . The hyperparameter explicitly controls the penalty that selectively merges each ✓i to ✓⇤A , i to alleviate catastrophic forgetting . 3.2 Active Forgetting with Synaptic Expansion-Convergence . However , if precisely remembering task A interferes with the learning of task B , e.g. , task A and task B are too different , it might be useful to actively forget the original data , similar to the biological strategy of active forgetting . Based on this inspiration , we introduce a forgetting factor and replace p ( ✓|DtrainA ) that absorbs all the information of DtrainA with a weighted product distribution [ 8 , 19 ] : pm ( ✓|DtrainA , ) = p ( ✓|DtrainA ) ( 1 ) p ( ✓ ) Z , ( 6 ) where we use m to denote that we are ‘ mixing ’ p ( ✓|DtrainA ) and p ( ✓ ) to produce the new distribution pm . Z is the normalizer that depends on , which keeps pm ( ✓|DtrainA , ) following a Gaussian distribution if p ( ✓|DtrainA ) and p ( ✓ ) are both Gaussian ( detailed in Appendix B.2 ) . When ! 0 , pm will be dominated by p ( ✓|DtrainA ) and remember all the information about task A . When ! 1 , pm will actively forget all the information about task A . Modified from Eqn . ( 2 ) , our target becomes : p ( DtestA , D test B |DtrainA , DtrainB , ) = Z p ( DtestA , D test B |✓ ) p ( ✓|DtrainA , DtrainB , ) d✓ . ( 7 ) We first need to determine , which decides how much information from task A is forgotten to maximize the probability of learning task B well . A good should be as follows : ⇤ = argmax p ( DtrainB |DtrainA , ) = argmax Z p ( DtrainB |✓ ) pm ( ✓|DtrainA , ) d✓ . ( 8 ) Since the integral is difficult to solve , we can make a grid search to determine , which should be between 0 and 1 . Next , p ( ✓|DtrainA , DtrainB , ) can also be approximated by a Gaussian using Laplace approximation ( the proof is detailed in Appendix B.3 ) , and the MAP estimation is ✓⇤A , B = argmax ✓ log p ( ✓|DtrainA , DtrainB , ) = argmax ✓ ( 1 ) ( log p ( DtrainB |✓ ) + log p ( ✓|DtrainA ) ) + log p ( ✓|DtrainB ) + const .. ( 9 ) Then we obtain the loss function of Active Forgetting with synaptic Expansion-Convergence ( AFEC ) : LAFEC ( ✓ ) =LB ( ✓ ) + 2 X i FA , i ( ✓i ✓⇤A , i ) 2 + e 2 X i Fe , i ( ✓i ✓⇤e , i ) 2 . ( 10 ) ✓⇤e are the optimal parameters for the new task and Fe is the FIM of ✓⇤e ( the computation is detailed in Appendix B.1 ) . As shown in Fig . 2 , we first learn a set of expanded parameters ✓e with LB ( ✓e ) to obtain ✓⇤e and Fe . Then we can optimize Eqn . ( 10 ) , where two weight-merging regularizers selectively merge ✓i with ✓⇤A , i for the old tasks and ✓ ⇤ e , i for the new task . The forgetting factor is integrated into a hyperparameter e / / ( 1 ) to control the penalty that promotes active forgetting . Therefore , derived from active forgetting of the original posterior in Eqn . ( 6 ) , we obtain an algorithm that dynamically expands parameters to learn a new task and then selectively converges the expanded parameters to the main network . Intriguingly , this algorithm is formally consistent with the underlying mechanism of biological active forgetting ( the neuroscience evidence is detailed in Appendix A ) , which also expands additional functional connections for a new experience ( synaptic expansion ) and then prunes them to the amount before learning ( synaptic convergence ) . As the proposed active forgetting is integrated into the third term , our method can be used in a plug-and-play way to improve continual learning ( detailed in Appendix E , F ) . Here we use Laplace approximation to approximate the intractable posteriors , which can be other strategies of approximate inference [ 20 ] in further work . Note that ✓⇤e and Fe are not stored in continual learning , and the architecture of the main network is fixed . Thus , AFEC does not cause additional storage cost compared with regular weight regularization approaches such as [ 12 , 35 , 1 , 2 ] . Further , it is straightforward to extend our method to continual learning of more than two tasks . We discuss it in Appendix B.4 with a pseudocode . Now we conceptually analyze how AFEC mitigates potential negative transfer in continual learning ( see Fig . 2 ) . When learning task B on the basis of task A , regular weight regularization approaches [ 12 , 35 , 1 , 2 ] selectively penalize changes of the old parameters learned for task A , which will severely interfere with the learning of task B if they conflict with each other . In contrast , AFEC learns a set of expanded parameters only for task B to avoid potential negative transfer from task A . Then , the main network parameters selectively merge with both the old parameters and the expanded parameters , depending on their contributions to the overall representations of task A and task B .

Summary: The paper proposed a continual learning method that attempts to promote better forward transfer among similar tasks. Towards this, the authors attempt to suppress the negative knowledge, the knowledge that impedes learning a new task, from the previous task while learning a new task. This is done by forgetting such knowledge from the previous task without increasing the forgetting catastrophically. The authors claim that such “active forgetting” is biologically inspired. Similar to the Elastic Weight Consolidation (EWC), the authors derive the learning objective, by applying the laplace approximation to the posterior of a new task. The only difference is that the posterior of the previous task is replaced by a weighted product distribution, where a weight of ‘\beta’ is introduced to capture active forgetting. This change allowed them to derive a new regularization penalty whereby a separate model is trained for a new task and then subsequently merged with that of the previous one. It is in this merging that the negative knowledge of the previous task is eliminated. Experimentally, the authors show strong performance on regression, classification and RL tasks. The authors further show that their method can be added to other regularization- and memory-based methods to improve their performance.