Continual Learning with Filter Atom Swapping

1 INTRODUCTION . Humans keep acquiring new concepts without forgetting crucial ones in the past . To endow intelligent agents with the same ability of long-term knowledge accumulation , continual learning ( CL ) has been intensively studied in recent years . In continual learning , an agent learns from a sequence of tasks , with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones , therefore to avoid catastrophic forgetting . The recent advances of CL mainly follow several directions . One popular category among them is to maintain an external memory of the original images ( Robins , 1995 ; Rebuffi et al. , 2017 ) , synthesized images ( Shin et al. , 2017 ) , or parameter gradients ( Lopez-Paz & Ranzato , 2017 ) for archiving the past . These memory-based methods often suffer from heavy memory footprints , while still forgetting about the previous tasks to some extent . Motivated by the literature on subspace modeling of tasks ( Evgeniou & Pontil , 2007 ; Maurer et al. , 2013 ; Zhang & Yang , 2021 ; Romera-Paredes et al. , 2013 ; Kumar & Daume III , 2012 ) , in this paper , we propose to learn for each task a new filter subspace for each convolutional layer , i.e. , hundreds of parameters as filter atoms , but keep subspace coefficients shared across tasks . In other words , in a CNN , we enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms . Then , we perform continual learning by simply swapping filter atoms for each task . The effectiveness of our approach is empirically validated and further explained theoretically with an excess risk bound analysis . With the proposed approach , we can faithfully remember the past by only maintaining a small footprint atom memory to archive task-specific filter atoms . Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefficients with the task-specific atoms , which can be retrieved efficiently from the atom memory . Thus , the introduced filter atom memory permits historical knowledge to be recalled with a guarantee against forgetting . While comparing with state-of-the-art memory-based CL methods , our approach requires only storing for each task ∗This work is supported by DARPA TAMI program under No . HR00112190038 . some tiny size of filter atoms , which in total are typically much smaller than the size of exemplars in most memory-based methods ( Rebuffi et al. , 2017 ; Prabhu et al. , 2020 ) , and therefore potentially supports continual learning on a large scale . Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance in different continual learning settings : First , intertask ensemble utilizes the relevant past knowledge to boost the present task performance . To ensure that ensemble with past models can affect the current task positively , we only select a relevant subset of past models . With a life-long learning scenario in mind , we choose to assess task relevancy simply based on the filter subspace distance , which can be on-the-fly computed here via the Grassmann distance ( Absil et al. , 2004 ) among task-specific filter subspaces . Second , intra-task ensemble can be adopted in class-incremental setting to help task prediction with the minimal-entropy criterion . Usually , ensemble members are instantiated as independent CNNs , and their learning and inference are conducted separately for dissimilarity . However , this will lead to significant increase in training and inference time and memory usage . We address this problem by creating within a task multiple virtual members in a single CNN model by simply maintaining several groups of filter atoms in each layer . In this way , different intra-task members are integrated into a single network , and its learning and inference can be conducted efficiently with group convolution . We validate our simple yet effective approach on several continual learning benchmarks such as MNIST , CIFAR100 , and miniImageNet using both class-incremental and task-incremental settings , and observe competitive results against state-of-the-art methods on all benchmarks with far less memory usage . We summarize our contributions as follows , • We learn for each task a new filter subspace for each layer , and keep subspace coefficients shared across tasks . • We maintain a small footprint filter atom memory that can faithfully archive past knowledge with a guarantee against forgetting in a highly scalable way . • We adopt inter-task ensemble for the present task by recalling past models based on an on-the-fly calculated task relevancy in task-incremental settings . • We propose intra-task ensemble for class-incremental settings by creating multiple virtual members in a single CNN model through different groups of filter atoms per layer . 2 MOTIVATION . We are motivated by the literature on task subspace modeling ( Evgeniou & Pontil , 2007 ; Maurer et al. , 2013 ; Zhang & Yang , 2021 ; Romera-Paredes et al. , 2013 ; Kumar & Daume III , 2012 ) , where it is commonly assumed that task parameters lie in a low dimensional subspace , so that tasks can be modeled as a set of latent basis tasks and their linear combinations . The latent basis tasks and the respective linear combinations are often obtained via alternative optimization by fixing one and optimizing the other ( Kumar & Daume III , 2012 ) . In our continual learning setting , we model tasks using convolutional neural networks ( CNNs ) . Following ( Qiu et al. , 2018 ) , we decompose a convolutional filter Wi ∈ Rc×c ′×k×k for the i-th layer over m filter atoms Di ∈ Rm×k×k , linearly combined by coefficient αi ∈ Rc×c ′×m , where c and c′ are the number of input and output channels , k is the spatial size of each atom . This can be written as Wi = αi ×Di . Note that this decomposition distributes a filter Wi into two imbalanced parts : αi for channel mixing with mcc′ parameters , and light-weight Di for spatial filtering with only mk2 entries . In all , we use W = α×D to denote the filters decomposition in a model with l convolutional layers , where W = { Wi } li=1 , α = { αi } li=1 , D = { Di } li=1 indicate all filters , coefficients , and atoms respectively . It is easy to observe that , within each CNN layer , we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through filter atoms Di and their linear combinations as atom coefficients αi . The linear combination coefficients αi are learned on the first task jointly with the first group of atoms , and then atoms for subsequent tasks are alternatively optimized by fixing αi . As illustrated in the subsequent sections , this seemingly under-fitting method not only enables an efficient way to faithfully archive past models with a guarantee against forgetting , but also supports efficient inter-task and intra-task model ensemble to further improve the performance . 3 METHODOLOGY . We consider the problem of learning T tasks sequentially . Formally , we denote the data distribution that associates with the t-th task as Dt = ( X t , Yt ) , t ∈ { 1 , 2 , ... , T } , from which a dataset Dt = { xit , yit } Nt i=1 is sampled for training . The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data Dt from previous tasks t ≤ T ( Delange et al. , 2021 ) : T∑ t=1 E ( X t , Yt ) [ L ( F t ( X t ; θ ) , Yt ) ] , ( 1 ) where L denotes the risk function , F t ( · ; θ ) is the model for task t with parameter θ . Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in an external memoryM = { θt } Tt=1 given F t ( · ; θ ) = F ( · ; θt ) , thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory . However , such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence . We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional filters in a CNN into task-specific filter atoms and task-shared coefficients . Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting . This proposed approach allows efficient inter-task and intra-task ensemble to further boost performance . 3.1 A SCALABLE APPROACH AGAINST FORGETTING . In CNNs , catastrophic forgetting occurs when a model learned from a sequence of past tasks is updated in favor of the current task , resulting in significant performance degradation . A straightforward solution , as mentioned before , is to archive Wt in an external memoryM = { Wt } Tt=1 , and the representation space of any previous model can be faithfully recalled by memory retrieval . However , as deep CNNs contain great amount of parameters in Wt , this simple solution scales poorly with the number of tasks T . On the other hand , storing part of parameters , or small subsets of data for parameter flashback can not avoid forgetting completely ( Sarwar et al. , 2019 ; Yoon et al. , 2018 ) . This dilemma can be resolved with the proposed filter atom decomposition , as shown in Fig . 1 . With the filter decomposition described in Sec . 2 , by storing task-specific filter atoms Dt into memory , and enforcing a task-shared coefficients α , the model archives the entire knowledge for each time point . We refer to the memory for storing atoms as the atom memory , MD = { Dt } Tt=1 , Dt ∈ Rm×k×k , with each Dt learned in the t-th task with empirical risk minimization , arg min Dt Nt∑ i=1 L ( F ( xti ; α , Dt ) , yti ) . ( 2 ) The task-shared coefficients α are learned on the first task jointly with the first group of atoms , arg min D1 , α N1∑ i=1 L ( F ( x1i ; α , D1 ) , y1i ) . ( 3 ) In this way , we can guarantee that the statistical risk for a previous task t at any time point remains the same as we can recall faithfully the past model by multiplying the stored atoms with the task-shared coefficients , E ( X t , Yt ) [ L ( F t ( X t ; θ ) , Yt ) ] = E ( X t , Yt ) [ L ( F ( X t ; α×Dt ) , Yt ) ] . ( 4 ) Atom memory scalability . The proposed atom memory stores a group of atoms per task , which is scalable with increasing number of tasks . Formally , consider a l-layer CNN model with associated filters W = { Wi } li=1 . As mentioned in Sec . 2 , each filter can be decomposed as Wi = αi ×Di . The group of atoms for task t can then be denoted as Dt = { Dti } li=1 , which requires a size of lmk2 in storing parameters per task . This typically introduces only a few hundred of parameters for each task to be stored in the atom memory , which potentially supports continual learning on a large scale . Details for scalability comparison are shown in Sec . 5.2.1 . Analysis of excess risk bound . With task-shared coefficients α , the model for each new task may seemingly expect some degree of underfitting . However , as demonstrated in Section 5 , we still observe superior results over the state-of-the-art continual learning methods on all benchmarks we have evaluated . To understand this , we here theoretically analyze the excess risk bound for new tasks in the continual learning setting . For each tasks t ∈ { 1 , 2 , ... , T } , model F t ( · ; θ ) consists of representation function φt and prediction function wt ( wt ∈ Rk ) . The representation function φt maps an input x to feature space Z ⊆ Rk . For the analysis purpose , we assume φt is just one convolution layer , which can be decomposed as φt = α×Dt . Therefore , using the training samples from task t , we can solve the following optimization problem from Eq . ( 2 ) : min 1 2N t ‖yt − ( α×Dt ( xt ) ) +wt‖2 + λ 2 ‖α×Dt‖2F + λ 2 ‖wt‖2F , ( 5 ) where we use the mean square error as our loss , ( · ) + is ReLU activation ( z ) + = max { 0 , z } . The shared coefficients α are learned on the first task jointly with the first group of atoms . Similar to ( Du et al. , 2020 ) , we also assume that there is a ground-truth optimal representation function φt , ∗ and prediction function wt , ∗ for task t. Assumption 1 ( subgaussian input ) . There exists ρ > 0 such that , for all t ∈ { 1 , 2 , ... , T } , the random vector x̄ ∼ p̄t is ρ2-subgaussian . The p̄t is the distribution of samples in task t. Assumption 2 ( oracle network ) . Assume for task t ∈ { 1 , 2 , ... , T } that yt = ( α∗×Dt , ∗ ( xt ) ) +wt , ∗+ zt is generated by an oracle network with parameters α∗ , Dt , ∗ , and wt , ∗ . Noise term zt ∼ N ( 0 , σ2I ) . Excess risk bound . We can bound the excess risk of our learned model on the task t , i.e. , how much our learned model ( α̂ , D̂t , ŵt ) performs worse than the optimal model ( α∗ , Dt , ∗ , wt , ∗ ) on the task t as follows : ER ( α̂ , D̂t , ŵt ) = LDt ( α̂ , D̂ t , ŵt ) − LDt ( α∗ , Dt , ∗ , wt , ∗ ) ≤ σR̄ · Õ ( √ Tr ( Σ ) + √ ‖Σ‖2√ N t ) + ρ4R̄2 · Õ ( Tr ( Σ ) + ‖Σ‖2 N t ) where R̄ = 12‖α ∗ ×Dt , ∗‖2F + 12‖w t , ∗‖2F , Σ = Ex∼p [ xxᵀ ] , N t is the number of training samples . Õ is the big O notation , and LDt is the expected loss with the data distribution Dt . The detailed analysis is provided in the Appendix A .

Summary: This paper introduces a model for continual learning based on the decomposition of linear filters into low-rank components, called atoms. Specifically, the authors decompose convolutional filters shaped (c,c',k,k) into two components: i) alpha, shaped (c,c',m) and D, shaped (m,k,k). The former is learned on the first task and then frozen, whereas for D every task has its own and they are do not conflict during optimization. On top of that, the authors envision two different ensembling schemes that improve performances. i) First, in task-incremental settings, they retrieve atoms from the task of interest and from similar tasks as well (based on SVD decomposition of D matrices and Grassman distance) and ensemble them. ii) Furthermore, in class-incremental settings, they explicitly setup multiple atoms per task, building a task ensemble. During inference, all ensembles are queried and their predictive variance is used to "discover" the relevant task, for which a prediction is carried out. Experiments are carried out on 3 datasets in both task-incremental and class-incremental learning settings.