Gradient Projection Memory for Continual Learning

1 INTRODUCTION . Humans exhibit remarkable ability in continual adaptation and learning new tasks throughout their lifetime while maintaining the knowledge gained from past experiences . In stark contrast , Artificial Neural Networks ( ANNs ) under such Continual Learning ( CL ) paradigm ( Ring , 1998 ; Thrun & Mitchell , 1995 ; Lange et al. , 2021 ) forget the information learned in the past tasks upon learning new ones . This phenomenon is known as ‘ Catastrophic Forgetting ’ or ‘ Catastrophic Interference ’ ( Mccloskey & Cohen , 1989 ; Ratcliff , 1990 ) . The problem is rooted in the general optimization methods ( Goodfellow et al. , 2016 ) that are being used to encode input data distribution into the parametric representation of the network during training . Upon exposure to a new task , gradient-based optimization methods , without any constraint , change the learned encoding to minimize the objective function with respect to the current data distribution . Such parametric updates lead to forgetting . Given a fixed capacity network , one way to address this problem is to put constraints on the gradient updates so that task specific knowledge can be preserved . To this end , Kirkpatrick et al . ( 2017 ) , Zenke et al . ( 2017 ) , Aljundi et al . ( 2018 ) , Serrà et al . ( 2018 ) add a penalty term to the objective function while optimizing for new task . Such term acts as a structural regularizer and dictates the degree of stability-plasticity of individual weights . Though these methods provide resource efficient solution to the catastrophic forgetting problem , their performance suffer while learning longer task sequence and when task identity is unavailable during inference . Approaches ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ) that store episodic memories of old data essentially solve an optimization problem with ‘ explicit ’ constraints on the new gradient directions so that losses for the old task do not increase . In Chaudhry et al . ( 2019b ) the performance of old task is retained by taking gradient steps in the average gradient direction obtained from the new data and memory samples . To minimize interference , Farajtabar et al . ( 2020 ) store gradient directions ( instead of data ) of the old tasks and optimize the network in the orthogonal directions to these gradients for the new task , whereas Zeng et al . ( 2018 ) update gradients orthogonal to the old input directions using projector matrices calculated iteratively during training . However , these methods either compromise data privacy by storing raw data or utilize resources poorly , which limits their scalability . 1Our code is available at https : //github.com/sahagobinda/GPM In this paper , we address the problem of catastrophic forgetting in a fixed capacity network when data from the old tasks are not available . To mitigate forgetting , our approach puts explicit constraints on the gradient directions that the optimizer can take . However , unlike contemporary methods , we neither store old gradient directions nor store old examples for generating reference directions . Instead we propose an approach that , after learning each task , partitions the entire gradient space of the weights into two orthogonal subspaces : Core Gradient Space ( CGS ) and Residual Gradient Space ( RGS ) ( Saha et al. , 2020 ) . Leveraging the relationship between the input and the gradient spaces , we show how learned representations ( activations ) form the bases of these gradient subspaces in both fully-connected and convolutional networks . Using Singular Value Decomposition ( SVD ) on these activations , we show how to obtain the minimum set of bases of the CGS by which past knowledge is preserved and learnability for the new tasks is ensured . We store these bases in the memory which we define as Gradient Projection Memory ( GPM ) . In our method , we propose to learn any new task by taking gradient steps in the orthogonal direction to the space ( CGS ) spanned by the GPM . Our analysis shows that such orthogonal gradient descent induces minimum to no interference with the old learning , and thus effective in alleviating catastrophic forgetting . We evaluate our approach in the context of image classification with miniImageNet , CIFAR-100 , PMNIST and sequence of 5-Datasets on a variety of network architectures including ResNet . We compare our method with related state-of-the-art approaches and report comparable or better classification performance . Overall , we show that our method is memory efficient and scalable to complex dataset with longer task sequence while preserving data privacy . 2 RELATED WORKS . Approaches to continual learning for ANNs can be broadly divided into three categories . In this section we present a detailed discussion on the representative works from each category , highlighting their contributions and differences with our approach . Expansion-based methods : Methods in this category overcome catastrophic forgetting by dedicating different subsets of network parameters to each task . With no constraint on network architecture , Progressive Neural Network ( PGN ) ( Rusu et al. , 2016 ) preserves old knowledge by freezing the base model and adding new sub-networks with lateral connections for each new task . Dynamically Expandable Networks ( DEN ) ( Yoon et al. , 2018 ) either retrains or expands the network by splitting/duplicating important units on new tasks , whereas Sarwar et al . ( 2020 ) grow the network to learn new tasks while sharing part of the base network . Li et al . ( 2019 ) with neural architecture search ( NAS ) find optimal network structures for each sequential task . RCL ( Xu & Zhu , 2018 ) adaptively expands the network at each layer using reinforcement learning , whereas APD ( Yoon et al. , 2020 ) additively decomposes the parameters into shared and task specific parameters to minimize the increase in the network complexity . In contrast , our method avoids network growth or expensive NAS operations and performs sequential learning within a fixed network architecture . Regularization-based methods : These methods attempt to overcome forgetting in fixed capacity model through structural regularization which penalizes major changes in the parameters that were important for the previous tasks . Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) computes such importance from diagonal of Fisher information matrix after training , whereas Zenke et al . ( 2017 ) compute them during training based on loss sensitivity with respect to the parameters . Additionally , Aljundi et al . ( 2018 ) compute importance from sensitivity of model outputs to the inputs . Other methods , such as PackNet ( Mallya & Lazebnik , 2018 ) uses iterative pruning to fully restrict gradient updates on important weights via binary mask , whereas HAT ( Serrà et al. , 2018 ) identifies important neurons by learning attention masks that control gradient propagation in the individual parameters . Saha et al . ( 2020 ) using a PCA based pruning on activations ( Garg et al. , 2020 ) partition the parametric space of the weights ( filters ) into core and residual ( filter ) spaces after learning each task . The past knowledge is preserved in the frozen core space , whereas the residual space is updated when learning the next task . In contrast to these methods , we do not ascribe importance to or restrict the gradients of any individual parameters or filters . Rather we put constraints on the ‘ direction ’ of gradient descent . Memory-based methods : Methods under this class mitigate forgetting by either storing a subset of ( raw ) examples from the past tasks in the memory for rehearsal ( Robins , 1995 ; Rebuffi et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; b ; Riemer et al. , 2019 ) or synthesizing old data from generative models to perform pseudo-rehearsal ( Shin et al. , 2017 ) . For instance , Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) avoids interference with previous task by projecting the new gradients in the feasible region outlined by previous task gradients calculated from the samples of episodic memory . Averaged-GEM ( A-GEM ) ( Chaudhry et al. , 2019a ) simplified this optimization problem to projection in one direction estimated by randomly selected samples from the memory . Guo et al . ( 2020 ) propose a unified view of episodic memory-based CL methods , that include GEM and A-GEM and improves performance over these methods utilizing loss-balancing update rule . Additionally , Experience Replay ( ER ) ( Chaudhry et al. , 2019b ) and Meta-Experience Replay ( MER ) ( Riemer et al. , 2019 ) mitigate forgetting in online CL setup by jointly training on the samples from new tasks and episodic memory . All these methods , however , rely on the access to old data which might not be possible when users have concern over data privacy . Like all the memory-based methods we also use a storage unit which we call GPM . However , we do not save any raw data in GPM , thus satisfy data privacy criterion . Our method is closely related to recently proposed Orthogonal Gradient Descent ( OGD ) ( Farajtabar et al. , 2020 ) and Orthogonal Weight Modulation ( OWM ) ( Zeng et al. , 2018 ) . OGD stores a set of gradient directions in the memory for each task and minimizes catastrophic forgetting by taking gradient steps in the orthogonal directions for new tasks . In contrast to OGD , we compute and store the bases of core gradient space from network representations ( activations ) which reduces the memory requirement by orders of magnitude . Moreover , OGD is shown to work under locality assumption for small learning rates which limits its scalability in learning longer task sequences with complex dataset . Since our method does not use gradient directions ( like OGD ) to describe the core gradient spaces , we do not need to obey such assumptions , thus can use higher learning rates . On the other hand , OWM reduces forgetting by modifying the weights of the network in the orthogonal to the input directions of the past tasks . This is achieved by multiplying new gradients with projector matrices . These matrices are computed from the stored past projectors and the inputs with recursive least square ( RLS ) method at each training step . However , such an iterative method not only slows down the training process but also shows limited scalability in end-to-end task learning with modern network architectures . Like OWM , we aim to encode new learning in the orthogonal to the old input directions . In contrast to iterative projector computation in OWM , we identify a low-dimensional subspace in the gradient space analyzing the learned representations with SVD in one-shot manner at the end of each task . We store the bases of these subspaces in GPM and learn new tasks in the orthogonal to these spaces to protect old knowledge . We quantitatively show that our method is memory efficient , fast and scalable to deeper networks for complex long sequence of tasks . 3 NOTATIONS AND BACKGROUND . In this section , we introduce the notations used throughout the paper and give a brief overview of SVD for matrix approximation . In section 4 , we establish the relationship between input and gradient spaces . In section 5 we show the steps of our algorithm that leverage such relationship . Continual Learning : We consider supervised learning setup where T tasks are learned sequentially . Each task has a task descriptor , τ ∈ { 1 , 2 .... , T } with a corresponding dataset , Dτ = { ( xi , τ , yi , τ ) nτi=1 } having nτ example pairs . Let ’ s consider an L layer neural network where at each layer network computes the following function for task τ : xl+1i , τ = σ ( f ( W l τ , x l i , τ ) ) . ( 1 ) Here , l = 1 , ... L , σ ( . ) is a non-linear function and f ( . , . ) is a linear function . We will use vector notation for input ( xi , τ ) in fully connected layers and matrix notation for input ( Xi , τ ) in convolutional layers . At the first layer , x1i , τ = xi , τ represents the raw input data from task τ , whereas in the subsequent layers we define xli , τ as the representation of input xi , τ at layer l. Set of parameters of the network is defined by , Wτ = { ( W lτ ) Ll=1 } , where W0 denotes set of parameters at initialization . Matrix approximation with SVD : SVD can be used to factorize a rectangular matrix , A = UΣV T ∈ Rm×n into the product of three matrices , where U ∈ Rm×m and V ∈ Rn×n are orthogonal , and Σ contains the sorted singular values along its main diagonal ( Deisenroth et al. , 2020 ) . If the rank of the matrix is r ( r ≤ min ( m , n ) ) , A can be expressed as A = ∑r i=1 σiuiv T i , where ui ∈ U and vi ∈ V are left and right singular vectors and σi ∈ diag ( Σ ) are singular values . Also , k-rank approximation to this matrix can be expressed as , Ak = ∑k i=1 σiuiv T i , where k ≤ r and its value can be chosen by the smallest k that satisfies ||Ak||2F ≥ th||A||2F . Here , ||.||F is the Frobenius norm of the matrix and th ( 0 < th ≤ 1 ) is the threshold hyperparameter .

Summary: The paper proposes one of the most scalable approaches to sequential continual learning with known task boundaries and related tasks, while taking steps towards enforcing data privacy and removing some of the task label constraints. At all levels in expressive deep models, SVD is used on learned representations to identify important bases of task gradient spaces and a memory is populated with such directions. Learning progresses only in directions orthogonal to gradient memory. Several recent evaluation methodologies are used to empirically validate the approach with significant success.