AFEC: Active Forgetting of Negative Transfer in Continual Learning

1 Introduction . The ability to continually learn numerous tasks from dynamic data distributions is critical for deep neural networks , which needs to remember the old tasks by avoiding catastrophic forgetting [ 18 ] while effectively learn each new task by improving forward knowledge transfer [ 17 ] . Due to the dynamic data distributions , forward knowledge transfer might be either positive or negative , and is difficult to determine without accessing to the old training samples . If the forward knowledge transfer is negative , i.e. , learning a new task from the old knowledge is worse than learning the new task on a randomly-initialized network [ 36 , 17 ] , then precisely remembering the old tasks will severely interfere with the learning of the new task , thus decreasing the performance of continual learning . ⇤Corresponding author : J. Zhu and Y. Zhong . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . By contrast , biological neural networks can effectively learn a new experience on the basis of remembering the old experiences , even if they conflict with each other [ 18 , 5 ] . This advantage , called memory flexibility , is achieved by active forgetting of the old knowledge that interferes with the learning of a new experience [ 29 , 5 ] . The latest data suggested that the underlying mechanism of biological active forgetting is to regulate the learning-triggered synaptic expansion and synaptic convergence ( Fig . 1 , see Appendix A for neuroscience background and our biological data ) . Specifically , the biological synapses expand additional functional connections to learn a new experience together with the previously-learned functional connections ( synaptic expansion ) . Then , all the functional connections are pruned to the amount before learning ( synaptic convergence ) . Inspired by the biological active forgetting , we propose to actively forget the old knowledge that interferes with the learning of new tasks without significantly increasing catastrophic forgetting , so as to benefit continual learning . Specifically , we adopt Bayesian continual learning and actively forget the posterior distribution that absorbs all the information of the old tasks with a forgetting factor to better learn each new task . Then , we derive a novel method named Active Forgetting with synaptic Expansion-Convergence ( AFEC ) , which is formally consistent with the underlying mechanism of biological active forgetting at synaptic structures . Beyond regular weight regularization approaches [ 12 , 1 , 35 , 2 ] , which selectively penalize changes of the important parameters for the old tasks , AFEC dynamically expands parameters only for each new task to avoid potential negative transfer from the main network , while the forgetting factor regulates a penalty to selectively merge the main network parameters with the expanded parameters , so as to learn a better overall representation of both the old tasks and the new task . We extensively evaluate AFEC on continual learning of CIFAR-10 regression tasks , a variety of visual classification tasks , and Atari reinforcement tasks [ 10 ] , where AFEC achieves the state-of-theart ( SOTA ) performance . We empirically validate that the performance improvement results from effectively improving the learning of new tasks without increasing catastrophic forgetting . Further , AFEC can be a plug-and-play method that significantly boosts the performance of representative continual learning strategies , such as weight regularization [ 12 , 1 , 35 , 2 ] and memory replay [ 21 , 9 , 6 ] . Our contributions include : ( 1 ) We draw inspirations from the biological active forgetting and propose a novel approach to actively forget the old knowledge that interferes with the learning of new tasks for continual learning ; ( 2 ) Extensive evaluation on a variety of continual learning benchmarks shows that our method effectively improves the learning of new tasks and achieves the SOTA performance in a plug-and-play way ; and ( 3 ) To the best of our knowledge , we are the first to model the biological active forgetting and its underlying mechanism at synaptic structures , which suggests a potential theoretical explanation of how the underlying mechanism of biological active forgetting achieves its function of forgetting the past and continually learning conflicting experiences [ 29 , 5 ] . 2 Related Work . Continual learning needs to minimize catastrophic forgetting and maximize forward knowledge transfer . Existing work in continual learning mainly focuses on mitigating catastrophic forgetting . Representative approaches include : weight regularization [ 12 , 1 , 35 , 2 ] , which selectively penalizes changes of the previously-learned parameters ; parameter isolation [ 24 , 10 ] , which allocates a dedicated parameter subspace for each task ; and memory replay [ 21 , 28 , 9 ] , which approximates and recovers the old data distributions through storing old training data , their embedding or learning a generative model . In particular , Adaptive Group Sparsity based Continual Learning ( AGS-CL ) [ 10 ] proposed to regularize the group sparsity with separation of the important nodes for the old tasks to prevent catastrophic forgetting , which takes advantages of weight regularization and parameter isolation , and achieved the SOTA performance on various continual learning benchmarks . Several studies suggested that forward knowledge transfer is critical for continual learning [ 17 , 4 ] , which might be either positive or negative due to the dynamic data distributions . Although it is highly nontrivial to mitigate potential negative transfer while overcoming catastrophic forgetting , the efforts that specifically consider this challenging issue are limited . [ 3 ] developed a method to mitigate negative transfer when fine-tuning tasks on a pretrained network . For the scenario where the old tasks can be learned again , [ 26 ] learned an additional active column to better exploit potential positive transfer . [ 22 ] tried to maximize transfer and minimize interference from a memory buffer containing a few old training data . Similarly , [ 6 , 16 , 33 ] attempted to more effectively balance stability and plasticity with the memory buffer in class incremental learning , while [ 32 ] stored and updated the old features . By contrast , since pretraining or old training data might not be available in continual learning , we mainly focus on a more restrict yet realistic setting that a neural network incrementally learns a sequence of tasks from scratch , without storing old training data . Further , we extend our method to the scenarios where pretraining or memory buffer can be used , as well as the scenarios other than classification tasks , such as regression tasks and reinforcement tasks . 3 Method . In this section , we first describe the framework of Bayesian continual learning [ 12 , 20 ] . Under such framework , we propose an active forgetting strategy , which is formally consistent with the underling mechanism of biological active forgetting at synaptic structures . 3.1 Basics of Bayesian Continual Learning . Continual learning needs to remember the old tasks and learn each new task effectively . Let ’ s consider a simple case that a neural network with parameter ✓ continually learns two independent tasks , task A and task B , from their training datasets DtrainA and D train B [ 12 ] . The training dataset of each task is only available when learning the task . Bayesian Learning : After learning DtrainA , the posterior distribution p ( ✓|DtrainA ) = p ( DtrainA |✓ ) p ( ✓ ) p ( DtrainA ) incorporates the knowledge of task A . Then , we can get the predictive distribution for the test data of task A : p ( DtestA |DtrainA ) = Z p ( DtestA |✓ ) p ( ✓|DtrainA ) d✓ . As the posterior p ( ✓|DtrainA ) is generally intractable ( except very special cases ) , we must resort to approximation methods , such as the Laplace approximation [ 12 ] or other approaches of approximate inference [ 20 ] . Let ’ s take Laplace approximation as an example . If p ( ✓|DtrainA ) is smooth and majorly peaked around the mode ✓⇤A = argmax✓ log p ( ✓|DtrainA ) , we can approximate it with a Gaussian distribution whose mean is ✓⇤A and covariance is the inverse Hessian of the negative log posterior ( detailed in Appendix B.1 ) . Bayesian Continual Learning : Next , we want to incorporate the new task into the posterior , which uses the posterior p ( ✓|DtrainA ) as the prior of the next task [ 12 ] : p ( ✓|DtrainA , DtrainB ) = p ( DtrainB |✓ ) p ( ✓|DtrainA ) p ( DtrainB ) . ( 1 ) Then we can test the performance of continual learning by evaluating p ( DtestA , D test B |DtrainA , DtrainB ) = Z p ( DtestA , D test B |✓ ) p ( ✓|DtrainA , DtrainB ) d✓ . ( 2 ) Similarly , p ( ✓|DtrainA , DtrainB ) can be approximated by a Gaussian using Laplace approximation whose mean is the mode of the posterior : ✓⇤A , B = argmax ✓ log p ( ✓|DtrainA , DtrainB ) ( 3 ) = argmax ✓ log p ( DtrainB |✓ ) + log p ( ✓|DtrainA ) log p ( DtrainB ) | { z } const. . ( 4 ) This MAP estimation is also known as the Elastic Weight Consolidation ( EWC ) [ 12 ] : LEWC ( ✓ ) = LB ( ✓ ) + 2 X i FA , i ( ✓i ✓⇤A , i ) 2 , ( 5 ) where LB ( ✓ ) is the loss for task B and i is the label of each parameter . FA is the Fisher Information matrix ( FIM ) of ✓⇤A on D train A ( the computation is detailed in Appendix B.1 ) , which indicates the “ importance ” of parameter i for task A . The hyperparameter explicitly controls the penalty that selectively merges each ✓i to ✓⇤A , i to alleviate catastrophic forgetting . 3.2 Active Forgetting with Synaptic Expansion-Convergence . However , if precisely remembering task A interferes with the learning of task B , e.g. , task A and task B are too different , it might be useful to actively forget the original data , similar to the biological strategy of active forgetting . Based on this inspiration , we introduce a forgetting factor and replace p ( ✓|DtrainA ) that absorbs all the information of DtrainA with a weighted product distribution [ 8 , 19 ] : pm ( ✓|DtrainA , ) = p ( ✓|DtrainA ) ( 1 ) p ( ✓ ) Z , ( 6 ) where we use m to denote that we are ‘ mixing ’ p ( ✓|DtrainA ) and p ( ✓ ) to produce the new distribution pm . Z is the normalizer that depends on , which keeps pm ( ✓|DtrainA , ) following a Gaussian distribution if p ( ✓|DtrainA ) and p ( ✓ ) are both Gaussian ( detailed in Appendix B.2 ) . When ! 0 , pm will be dominated by p ( ✓|DtrainA ) and remember all the information about task A . When ! 1 , pm will actively forget all the information about task A . Modified from Eqn . ( 2 ) , our target becomes : p ( DtestA , D test B |DtrainA , DtrainB , ) = Z p ( DtestA , D test B |✓ ) p ( ✓|DtrainA , DtrainB , ) d✓ . ( 7 ) We first need to determine , which decides how much information from task A is forgotten to maximize the probability of learning task B well . A good should be as follows : ⇤ = argmax p ( DtrainB |DtrainA , ) = argmax Z p ( DtrainB |✓ ) pm ( ✓|DtrainA , ) d✓ . ( 8 ) Since the integral is difficult to solve , we can make a grid search to determine , which should be between 0 and 1 . Next , p ( ✓|DtrainA , DtrainB , ) can also be approximated by a Gaussian using Laplace approximation ( the proof is detailed in Appendix B.3 ) , and the MAP estimation is ✓⇤A , B = argmax ✓ log p ( ✓|DtrainA , DtrainB , ) = argmax ✓ ( 1 ) ( log p ( DtrainB |✓ ) + log p ( ✓|DtrainA ) ) + log p ( ✓|DtrainB ) + const .. ( 9 ) Then we obtain the loss function of Active Forgetting with synaptic Expansion-Convergence ( AFEC ) : LAFEC ( ✓ ) =LB ( ✓ ) + 2 X i FA , i ( ✓i ✓⇤A , i ) 2 + e 2 X i Fe , i ( ✓i ✓⇤e , i ) 2 . ( 10 ) ✓⇤e are the optimal parameters for the new task and Fe is the FIM of ✓⇤e ( the computation is detailed in Appendix B.1 ) . As shown in Fig . 2 , we first learn a set of expanded parameters ✓e with LB ( ✓e ) to obtain ✓⇤e and Fe . Then we can optimize Eqn . ( 10 ) , where two weight-merging regularizers selectively merge ✓i with ✓⇤A , i for the old tasks and ✓ ⇤ e , i for the new task . The forgetting factor is integrated into a hyperparameter e / / ( 1 ) to control the penalty that promotes active forgetting . Therefore , derived from active forgetting of the original posterior in Eqn . ( 6 ) , we obtain an algorithm that dynamically expands parameters to learn a new task and then selectively converges the expanded parameters to the main network . Intriguingly , this algorithm is formally consistent with the underlying mechanism of biological active forgetting ( the neuroscience evidence is detailed in Appendix A ) , which also expands additional functional connections for a new experience ( synaptic expansion ) and then prunes them to the amount before learning ( synaptic convergence ) . As the proposed active forgetting is integrated into the third term , our method can be used in a plug-and-play way to improve continual learning ( detailed in Appendix E , F ) . Here we use Laplace approximation to approximate the intractable posteriors , which can be other strategies of approximate inference [ 20 ] in further work . Note that ✓⇤e and Fe are not stored in continual learning , and the architecture of the main network is fixed . Thus , AFEC does not cause additional storage cost compared with regular weight regularization approaches such as [ 12 , 35 , 1 , 2 ] . Further , it is straightforward to extend our method to continual learning of more than two tasks . We discuss it in Appendix B.4 with a pseudocode . Now we conceptually analyze how AFEC mitigates potential negative transfer in continual learning ( see Fig . 2 ) . When learning task B on the basis of task A , regular weight regularization approaches [ 12 , 35 , 1 , 2 ] selectively penalize changes of the old parameters learned for task A , which will severely interfere with the learning of task B if they conflict with each other . In contrast , AFEC learns a set of expanded parameters only for task B to avoid potential negative transfer from task A . Then , the main network parameters selectively merge with both the old parameters and the expanded parameters , depending on their contributions to the overall representations of task A and task B .

Summary: This paper proposes a new approach for continual learning that actively forgets the old knowledge when learning the new tasks. The approach uses Bayesian continual learning with synaptic expansion-convergence for active forgetting. The paper evaluates the new method on several continual learning benchmarks, including CIFAR10 regression, visual classification, and Atari reinforcement tasks.