Continual Learning with Filter Atom Swapping

1 INTRODUCTION . Humans keep acquiring new concepts without forgetting crucial ones in the past . To endow intelligent agents with the same ability of long-term knowledge accumulation , continual learning ( CL ) has been intensively studied in recent years . In continual learning , an agent learns from a sequence of tasks , with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones , therefore to avoid catastrophic forgetting . The recent advances of CL mainly follow several directions . One popular category among them is to maintain an external memory of the original images ( Robins , 1995 ; Rebuffi et al. , 2017 ) , synthesized images ( Shin et al. , 2017 ) , or parameter gradients ( Lopez-Paz & Ranzato , 2017 ) for archiving the past . These memory-based methods often suffer from heavy memory footprints , while still forgetting about the previous tasks to some extent . Motivated by the literature on subspace modeling of tasks ( Evgeniou & Pontil , 2007 ; Maurer et al. , 2013 ; Zhang & Yang , 2021 ; Romera-Paredes et al. , 2013 ; Kumar & Daume III , 2012 ) , in this paper , we propose to learn for each task a new filter subspace for each convolutional layer , i.e. , hundreds of parameters as filter atoms , but keep subspace coefficients shared across tasks . In other words , in a CNN , we enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms . Then , we perform continual learning by simply swapping filter atoms for each task . The effectiveness of our approach is empirically validated and further explained theoretically with an excess risk bound analysis . With the proposed approach , we can faithfully remember the past by only maintaining a small footprint atom memory to archive task-specific filter atoms . Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefficients with the task-specific atoms , which can be retrieved efficiently from the atom memory . Thus , the introduced filter atom memory permits historical knowledge to be recalled with a guarantee against forgetting . While comparing with state-of-the-art memory-based CL methods , our approach requires only storing for each task ∗This work is supported by DARPA TAMI program under No . HR00112190038 . some tiny size of filter atoms , which in total are typically much smaller than the size of exemplars in most memory-based methods ( Rebuffi et al. , 2017 ; Prabhu et al. , 2020 ) , and therefore potentially supports continual learning on a large scale . Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance in different continual learning settings : First , intertask ensemble utilizes the relevant past knowledge to boost the present task performance . To ensure that ensemble with past models can affect the current task positively , we only select a relevant subset of past models . With a life-long learning scenario in mind , we choose to assess task relevancy simply based on the filter subspace distance , which can be on-the-fly computed here via the Grassmann distance ( Absil et al. , 2004 ) among task-specific filter subspaces . Second , intra-task ensemble can be adopted in class-incremental setting to help task prediction with the minimal-entropy criterion . Usually , ensemble members are instantiated as independent CNNs , and their learning and inference are conducted separately for dissimilarity . However , this will lead to significant increase in training and inference time and memory usage . We address this problem by creating within a task multiple virtual members in a single CNN model by simply maintaining several groups of filter atoms in each layer . In this way , different intra-task members are integrated into a single network , and its learning and inference can be conducted efficiently with group convolution . We validate our simple yet effective approach on several continual learning benchmarks such as MNIST , CIFAR100 , and miniImageNet using both class-incremental and task-incremental settings , and observe competitive results against state-of-the-art methods on all benchmarks with far less memory usage . We summarize our contributions as follows , • We learn for each task a new filter subspace for each layer , and keep subspace coefficients shared across tasks . • We maintain a small footprint filter atom memory that can faithfully archive past knowledge with a guarantee against forgetting in a highly scalable way . • We adopt inter-task ensemble for the present task by recalling past models based on an on-the-fly calculated task relevancy in task-incremental settings . • We propose intra-task ensemble for class-incremental settings by creating multiple virtual members in a single CNN model through different groups of filter atoms per layer . 2 MOTIVATION . We are motivated by the literature on task subspace modeling ( Evgeniou & Pontil , 2007 ; Maurer et al. , 2013 ; Zhang & Yang , 2021 ; Romera-Paredes et al. , 2013 ; Kumar & Daume III , 2012 ) , where it is commonly assumed that task parameters lie in a low dimensional subspace , so that tasks can be modeled as a set of latent basis tasks and their linear combinations . The latent basis tasks and the respective linear combinations are often obtained via alternative optimization by fixing one and optimizing the other ( Kumar & Daume III , 2012 ) . In our continual learning setting , we model tasks using convolutional neural networks ( CNNs ) . Following ( Qiu et al. , 2018 ) , we decompose a convolutional filter Wi ∈ Rc×c ′×k×k for the i-th layer over m filter atoms Di ∈ Rm×k×k , linearly combined by coefficient αi ∈ Rc×c ′×m , where c and c′ are the number of input and output channels , k is the spatial size of each atom . This can be written as Wi = αi ×Di . Note that this decomposition distributes a filter Wi into two imbalanced parts : αi for channel mixing with mcc′ parameters , and light-weight Di for spatial filtering with only mk2 entries . In all , we use W = α×D to denote the filters decomposition in a model with l convolutional layers , where W = { Wi } li=1 , α = { αi } li=1 , D = { Di } li=1 indicate all filters , coefficients , and atoms respectively . It is easy to observe that , within each CNN layer , we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through filter atoms Di and their linear combinations as atom coefficients αi . The linear combination coefficients αi are learned on the first task jointly with the first group of atoms , and then atoms for subsequent tasks are alternatively optimized by fixing αi . As illustrated in the subsequent sections , this seemingly under-fitting method not only enables an efficient way to faithfully archive past models with a guarantee against forgetting , but also supports efficient inter-task and intra-task model ensemble to further improve the performance . 3 METHODOLOGY . We consider the problem of learning T tasks sequentially . Formally , we denote the data distribution that associates with the t-th task as Dt = ( X t , Yt ) , t ∈ { 1 , 2 , ... , T } , from which a dataset Dt = { xit , yit } Nt i=1 is sampled for training . The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data Dt from previous tasks t ≤ T ( Delange et al. , 2021 ) : T∑ t=1 E ( X t , Yt ) [ L ( F t ( X t ; θ ) , Yt ) ] , ( 1 ) where L denotes the risk function , F t ( · ; θ ) is the model for task t with parameter θ . Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in an external memoryM = { θt } Tt=1 given F t ( · ; θ ) = F ( · ; θt ) , thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory . However , such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence . We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional filters in a CNN into task-specific filter atoms and task-shared coefficients . Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting . This proposed approach allows efficient inter-task and intra-task ensemble to further boost performance . 3.1 A SCALABLE APPROACH AGAINST FORGETTING . In CNNs , catastrophic forgetting occurs when a model learned from a sequence of past tasks is updated in favor of the current task , resulting in significant performance degradation . A straightforward solution , as mentioned before , is to archive Wt in an external memoryM = { Wt } Tt=1 , and the representation space of any previous model can be faithfully recalled by memory retrieval . However , as deep CNNs contain great amount of parameters in Wt , this simple solution scales poorly with the number of tasks T . On the other hand , storing part of parameters , or small subsets of data for parameter flashback can not avoid forgetting completely ( Sarwar et al. , 2019 ; Yoon et al. , 2018 ) . This dilemma can be resolved with the proposed filter atom decomposition , as shown in Fig . 1 . With the filter decomposition described in Sec . 2 , by storing task-specific filter atoms Dt into memory , and enforcing a task-shared coefficients α , the model archives the entire knowledge for each time point . We refer to the memory for storing atoms as the atom memory , MD = { Dt } Tt=1 , Dt ∈ Rm×k×k , with each Dt learned in the t-th task with empirical risk minimization , arg min Dt Nt∑ i=1 L ( F ( xti ; α , Dt ) , yti ) . ( 2 ) The task-shared coefficients α are learned on the first task jointly with the first group of atoms , arg min D1 , α N1∑ i=1 L ( F ( x1i ; α , D1 ) , y1i ) . ( 3 ) In this way , we can guarantee that the statistical risk for a previous task t at any time point remains the same as we can recall faithfully the past model by multiplying the stored atoms with the task-shared coefficients , E ( X t , Yt ) [ L ( F t ( X t ; θ ) , Yt ) ] = E ( X t , Yt ) [ L ( F ( X t ; α×Dt ) , Yt ) ] . ( 4 ) Atom memory scalability . The proposed atom memory stores a group of atoms per task , which is scalable with increasing number of tasks . Formally , consider a l-layer CNN model with associated filters W = { Wi } li=1 . As mentioned in Sec . 2 , each filter can be decomposed as Wi = αi ×Di . The group of atoms for task t can then be denoted as Dt = { Dti } li=1 , which requires a size of lmk2 in storing parameters per task . This typically introduces only a few hundred of parameters for each task to be stored in the atom memory , which potentially supports continual learning on a large scale . Details for scalability comparison are shown in Sec . 5.2.1 . Analysis of excess risk bound . With task-shared coefficients α , the model for each new task may seemingly expect some degree of underfitting . However , as demonstrated in Section 5 , we still observe superior results over the state-of-the-art continual learning methods on all benchmarks we have evaluated . To understand this , we here theoretically analyze the excess risk bound for new tasks in the continual learning setting . For each tasks t ∈ { 1 , 2 , ... , T } , model F t ( · ; θ ) consists of representation function φt and prediction function wt ( wt ∈ Rk ) . The representation function φt maps an input x to feature space Z ⊆ Rk . For the analysis purpose , we assume φt is just one convolution layer , which can be decomposed as φt = α×Dt . Therefore , using the training samples from task t , we can solve the following optimization problem from Eq . ( 2 ) : min 1 2N t ‖yt − ( α×Dt ( xt ) ) +wt‖2 + λ 2 ‖α×Dt‖2F + λ 2 ‖wt‖2F , ( 5 ) where we use the mean square error as our loss , ( · ) + is ReLU activation ( z ) + = max { 0 , z } . The shared coefficients α are learned on the first task jointly with the first group of atoms . Similar to ( Du et al. , 2020 ) , we also assume that there is a ground-truth optimal representation function φt , ∗ and prediction function wt , ∗ for task t. Assumption 1 ( subgaussian input ) . There exists ρ > 0 such that , for all t ∈ { 1 , 2 , ... , T } , the random vector x̄ ∼ p̄t is ρ2-subgaussian . The p̄t is the distribution of samples in task t. Assumption 2 ( oracle network ) . Assume for task t ∈ { 1 , 2 , ... , T } that yt = ( α∗×Dt , ∗ ( xt ) ) +wt , ∗+ zt is generated by an oracle network with parameters α∗ , Dt , ∗ , and wt , ∗ . Noise term zt ∼ N ( 0 , σ2I ) . Excess risk bound . We can bound the excess risk of our learned model on the task t , i.e. , how much our learned model ( α̂ , D̂t , ŵt ) performs worse than the optimal model ( α∗ , Dt , ∗ , wt , ∗ ) on the task t as follows : ER ( α̂ , D̂t , ŵt ) = LDt ( α̂ , D̂ t , ŵt ) − LDt ( α∗ , Dt , ∗ , wt , ∗ ) ≤ σR̄ · Õ ( √ Tr ( Σ ) + √ ‖Σ‖2√ N t ) + ρ4R̄2 · Õ ( Tr ( Σ ) + ‖Σ‖2 N t ) where R̄ = 12‖α ∗ ×Dt , ∗‖2F + 12‖w t , ∗‖2F , Σ = Ex∼p [ xxᵀ ] , N t is the number of training samples . Õ is the big O notation , and LDt is the expected loss with the data distribution Dt . The detailed analysis is provided in the Appendix A .

Summary: The paper proposes a continual learning algorithm that enforces the convolutional filter in each layer to a low-rank filter subspace defined by a small set of filter atoms. For each task, each convolutional layer is defined by a new filter subspace but subspace coefficients are shared among the tasks. The algorithm is validated on multiple benchmark datasets.