TRGP: Trust Region Gradient Projection for Continual Learning

1 INTRODUCTION . Human beings can continuously learn different new tasks without forgetting the learnt knowledge of old tasks in their lifespan . Aiming to achieve this remarkable capability for the deep neural networks ( DNNs ) , continual learning ( CL ) ( Chen & Liu , 2018 ) has garnered much attention in recent years . Nevertheless , many existing CL methods still leave the DNN vulnerable to forget the knowledge of old tasks when learning new tasks . Such a phenomenon is known as ‘ Catastrophic Forgetting ’ ( McCloskey & Cohen , 1989 ) , which has become one of the major challenges for CL . Many approaches ( e.g. , ( Rusu et al. , 2016 ; Li & Hoiem , 2017 ; Dhar et al. , 2019 ; Guo et al. , 2020 ; Zeng et al. , 2019 ) ) have been proposed to address the forgetting issue , which can be generally divided into two classes depending on the network architecture , i.e. , expansion methods and nonexpansion methods . In order to understand the fundamental limit of a fixed capacity neural network , we focus on non-expansion methods in this work . The basic idea for non-expansion methods is to constrain the gradient update either explicitly or implicitly when learning the new task , so as to minimize the introduced interference to old tasks . For example , the regularization-based methods ( e.g. , ( Kirkpatrick et al. , 2017 ; Serra et al. , 2018 ) ) penalize the modification on the most important weights of old tasks through model regularizations ; experience-replay based methods ( e.g. , ( Shin et al. , 2017 ; Chaudhry et al. , 2019 ) ) constrain the gradient directions by replaying the data of old tasks during learning of new tasks , in the format of either real data or synthetic data from generative models ; and orthogonal-projection based methods ( e.g. , ( Farajtabar et al. , 2020 ; Saha et al. , 2021 ) ) update the model with gradients in the orthogonal directions of old tasks , without the access to old task data . In particular , the recently proposed Gradient Projection Memory ( GPM ) ( Saha et al. , 2021 ) has demonstrated superior performance compared to other approaches . To sufficiently minimize the interference to old tasks , most existing non-expansion methods ( particularly the orthogonal-projection based methods ) , often put restrictive constraints on the optimization space of the new task , which may throttle the learning performance for the new task . A plausible conjecture is that such a scenario is likely to occur when the new task is strongly correlated with old tasks , and in this study we provide evidence to support this conjecture . The underlying rationale is as follows : The weights that are important to the new task are also important to the old tasks strongly correlated with the new task , which are often frozen to address the forgetting in the existing methods ; however , they should be updated in the learning of the new task . To tackle this challenge , a key insight is that for a new task that is strongly correlated with old tasks , although the model optimization space could be more restrictive , there should be better forward knowledge transfer from the correlated old tasks to the new task . With this insight , we propose an innovate continual learning approach to facilitate the forward knowledge transfer without forgetting . The main contributions can be summarized as follows : ( 1 ) Inspired by ( Schulman et al. , 2015 ) , we introduce a novel notion of ‘ trust region ’ based on the norm of gradient projection onto the subspace spanned by task inputs , which selects the old tasks strongly correlated to the new task in a layer-wise and single-shot manner . Intuitively , the new task and the selected old tasks in the trust region have similar input features for the corresponding layer . ( 2 ) We propose a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the trust region through a scaled weight projection . Particularly , a scaling matrix is learnt in each layer for the new task to scale the weight projection onto the subspace of old tasks in the trust region , in order to reuse the frozen weights of old tasks without modifying the model . ( 3 ) Building on the introduced trust region , scaled weight projection , and a module to construct task input subspace , we develop a continual learning approach , trust region gradient projection ( TRGP ) , that jointly optimizes the scaling matrices and the model for the new task . To mitigate the forgetting issue further , the model is updated along the directions orthogonal to the subspaces of old tasks . ( 4 ) We evaluate TRGP on standard CL benchmarks using various network architectures . Compared to related state-of-the-art approaches , TRGP achieves substantial performance improvement on all benchmarks , and demonstrates universal improvement on all tasks . The superior performance indicates that TRGP can effectively promote the forward knowledge transfer while alleviating forgetting . 2 RELATED WORK . Expansion-based methods . Expansion-based methods ( e.g. , ( Rusu et al. , 2016 ; Li & Hoiem , 2017 ; Rosenfeld & Tsotsos , 2018 ; Hung et al. , 2019 ; Yoon et al. , 2017 ; Li et al. , 2019 ; Veniat et al. , 2020 ) ) dynamically expand the network capacity to reduce the interference between the new tasks and the old ones . Progressive Neural Network ( PNN ) ( Rusu et al. , 2016 ) expands the network architecture for new tasks and preserves the weights of old tasks . Learning Without Forgetting ( LWF ) ( Li & Hoiem , 2017 ) splits the model layers into two parts , i.e. , the shared part co-used by all tasks , and the task-specific part which grows for new tasks . Dynamic-Expansion Net ( DEN ) ( Yoon et al. , 2017 ) and Compacting-Picking-Growing ( CPG ) ( Hung et al. , 2019 ) combine the strategies of model compression/pruning , weight selection and model expansion . In order to find the optimal structure for each of the sequential tasks , Reinforced Continual Learning ( RCL ) ( Xu & Zhu , 2018 ) leverages reinforcement learning and ( Li et al. , 2019 ) adapts architecture search . APD ( Yoon et al. , 2020 ) adds additional task-specific parameters for each task and selectively learns the task-shared parameters . Regularization-based methods . This category of methods ( e.g. , ( Kirkpatrick et al. , 2017 ; Lee et al. , 2017 ; Chaudhry et al. , 2018a ; Dhar et al. , 2019 ; Ritter et al. , 2018 ; Schwarz et al. , 2018 ; Zenke et al. , 2017 ) ) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their important weights . Notably , to determine the weight importance , Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) leverages Fisher information matrix , HAT ( Serra et al. , 2018 ) learns hard attention masks . MAS ( Aljundi et al. , 2018 ) evaluates the model outputs sensitivity to the inputs in an unsupervised manner . Memory-based methods . Depending on if data of old tasks is utilized when learning new tasks , memory-based methods can be further divided into the following two categories . 1 ) Experiencereplay based methods . This class of methods replays the old tasks data along with the current task data to mitigate catastrophic forgetting . Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) and Averaged GEM ( A-GEM ) ( Chaudhry et al. , 2018b ) alter the current gradient based on the gradient computed with data in the memory . A unified view of episodic memory based methods is proposed in ( Guo et al. , 2020 ) , based on new approaches are developed to balance between old tasks and the new task . Tiny episodic memory is considered in ( Chaudhry et al. , 2019 ) and meta- learning is leveraged in ( Riemer et al. , 2018 ) . 2 ) Orthogonal-projection based method . To eliminate the need of storing data of old tasks , recently a series work ( Zeng et al. , 2019 ; Farajtabar et al. , 2020 ; Saha et al. , 2021 ) updates the model in the orthogonal direction of old tasks , and has shown remarkable performance . Particularly , Orthogonal Weight Modulation ( OWM ) ( Zeng et al. , 2019 ) learns a projector matrix to multiply with the new gradients . Orthogonal Gradient Descent ( OGD ) ( Farajtabar et al. , 2020 ) stores the gradient directions of old tasks and projects the new gradients on the directions orthogonal to the subspace spanned by the old gradients . Gradient Projection Memory ( GPM ) ( Saha et al. , 2021 ) stores the bases of the subspaces spanned by old task data and projects the new gradients on the directions orthogonal to these subspaces . 3 PROBLEM FORMULATION . Continual learning . Consider the setting where a sequence of tasks T = { t } Tt=1 arrives sequentially . Each task t has a dataset Dt = { ( xt , i , yt , i ) } Nti=1 with Nt sample pairs , where xt , i is the input vector and yt , i is the label vector . Consider a fixed capacity neural network with L layers , and the set of weights is denoted as W = { W l } Ll=1 , whereW l is the layer-wise weight for layer l. Given the data input xt , i for task t , denote xlt , i as the input of layer l and x 1 t , i = xt , i . The output x l+1 t , i for layer l is computed as xl+1t , i = f ( W l , xlt , i ) , where f is the operation of the network layer . Following ( Saha et al. , 2021 ) , we denote xlt , i as the representations of xt , i at layer l. When learning a task t , we only have access to dataset Dt . Let L ( W , { ( xt , i , yt , i ) } ) = Lt ( W ) denote the loss function for training , e.g. , mean squared and cross-entropy loss , and Wt denote the model after learning task t. Orthogonal-projection based methods . To minimize the interference to old tasks , recently a series of studies ( Zeng et al. , 2019 ; Farajtabar et al. , 2020 ; Saha et al. , 2021 ) has been carried out to update the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks . In what follows , we briefly introduce the main ideas through a basic case with two tasks 1 and 2 . Denote the subspace spanned by the inputs of task 1 for layer l as Sl1 and the learnt model for task 1 as W1 = { W l1 } Ll=1 . It is clear that xl1 , i ∈ Sl1 . When learning task 2 , the modelW l1 will be modified in the direction orthogonal to Sl1 , by either multiplying the gradient∇W lL2 with a projector matrix ( e.g , ( Zeng et al. , 2019 ) ) , or projecting the gradient∇W lL2 onto the orthogonal direction to Sl1 ( e.g. , ( Saha et al. , 2021 ) ) . Let ∆W l1 denote the model change after learning task 2 . It follows immediately that ∆W l1x l 1 , i = 0 , and the modelW l 2 for task 2 isW l 2 = W l 1 + ∆W l 1 . Therefore , for task 1 : W l2x l 1 , i = ( W l 1 + ∆W l 1 ) x l 1 , i = W l 1x l 1 , i + ∆W l 1x l 1 , i = W l 1x l 1 , i , ( 1 ) which indicates that no interference is introduced to task 1 after learning task 2 , thereby addressing the forgetting issue . Such an analysis can be generalized to a sequence of tasks . When would orthogonal projection hinder the learning of a new task ? Orthogonal projection provides a promising solution to address the forgetting in continual learning . However , by modifying the model only in the orthogonal direction to the input space of old tasks , the optimization space of learning the new task could be more restrictive , resulting in compromised performance of the new task . To get a more concrete sense , consider the following basic examples with two tasks 1 and 2 . ( Toy example 1 ) Suppose task 1 has dataset D1 = { ( xi , yi ) } Ni=1 and task 2 has dataset D2 = { ( −xi , yi ) } Ni=1 , where only the sign is changed for the input vectors . Consider the case where two tasks share the same classifier ( Saha et al. , 2021 ) . It is clear that for the l-th layer , the subspace spanned by { xli } Ni=1 of task 1 is same with the subspace spanned by { −xli } Ni=1 of task 2 , i.e. , Sl1 = S l 2 , given the learnt model W l 1 for task 1 . Based on the fact that stochastic gradient descent updates lie in the subspace spanned by the data input ( Zhang et al. , 2021 ; Saha et al. , 2021 ) , it follows that the gradient ∇W lL2 ∈ Sl2 , such that ∇W lL2 ∈ Sl1 . Therefore , the projection of ∇W lL2 onto the orthogonal direction to Sl1 is 0 , which means that the model W l 1 will not be updated when learning task 2 , i.e. , W l2 = W l 1 . However , the optimal model for task 2 should be W l 2 = −W l1 , becauseW l1x l i achieves the minimum loss for the label yi after learning task 1 . ( Toy example 2 ) Suppose the input subspace of task 1 is orthogonal to that of task 2 , i.e. , Sl1 ⊥ Sl2 . It follows that the projection of∇W lL2 onto the orthogonal direction to Sl1 is indeed equal to∇W lL2 . Consequently , updating the model for task 2 based on orthogonal projection will not only introduce no interference to task 1 , but also move along the direction of steepest descent for task 2 . Motivated by these examples , a plausible conjecture is that naive orthogonal projection could possibly compromise the learning performance of the new task that is strongly correlated with old tasks , especially when the correlation is “ negative ” as in the toy example 1 . In this study , we advocate to characterize the task correlation through the correlation between the input subspaces for two tasks . As illustrated in Fig . 1 , when the subspace is 2-dimensional , two tasks are weakly correlated if their input subspaces are nearly orthogonal , and strongly correlated if their subspaces are nearly parallel .

Summary: **Summary:** The paper focuses on gradient projection (GP) for incremental learning. The authors motivate their work by stating that while approaches based on GP lead to superior performance in overcoming catastrophic forgetting, they suffer from a significant drawback. In GP, once one calculates the subspaces spanned by layerwise inputs for a task, say Task A, the network is then forced to only update the weights in an orthogonal direction to these subspaces for learning a subsequent task, say Task B (hence keeping important parameters for Task A intact and overcome catastrophic forgetting). However, suppose Task B and Task A are similar (an extreme case is when Task B is the continuation of Task A!). In that case, we know that the essential parameters for Task A are likely to be important for Task B and that the network could benefit by continuing to update the important parameters for Task A, which is not allowed in GP algorithms. This behavior has two consequences: 1) intransigence, i.e., the network won't be able to learn Task B as effectively as possible, and 2) the network will have reduced backward transfer. The backward transfer issue is apparent in the extreme case when Task B is a continuation of Task A, and the network's performance on Task A would have improved if it was able to learn Task B using the important weights for Task A!  The paper addresses this issue, with a simple, yet practical, solution. For a new task, the authors calculate the correlation between the subspaces calculated for old tasks and the new task (layerwise), keep track of the most correlated previous tasks for each layer, and denote them as (layerwise) Trust Regions. Next, the authors propose a scaled weight projection that allows for unfreezing the important parameters in the Trust Region of the new task, while learning this task, and learn the corresponding weights as part of the optimization process. Finally, the authors report results on Permuted MNIST (PMNIST), CIFAR100 Split, CIFAR100 Sup, and a sequence of 5-Datasets with 10-class classification which includes CIFAR-10, MNIST, SVHN, not-MNIST, and Fashion MNIST, in comparison with GP methods, regularization-based methods, and memory replay method. They show consistent improvement in accuracy, and more interestingly in backward transfer.