Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

1 INTRODUCTION . Continual learning ( Thrun , 1995 ) , or lifelong learning , is a learning scenario where a model is incrementally updated over a sequence of tasks , potentially performing knowledge transfer from earlier tasks to later ones . Building a successful continual learning model may lead us one step further towards developing a general artificial intelligence , since learning numerous tasks over a long-term time period is an important aspect of human intelligence . Continual learning is often formulated as an incremental / online multi-task learning that models complex task-to-task relationships , either by sharing basis vectors in linear models ( Kumar & Daume III , 2012 ; Ruvolo & Eaton , 2013 ) or weights in neural networks ( Li & Hoiem , 2016 ) . One problem that arises here is that as the model learns on the new tasks , it could forget what it learned for the earlier tasks , which is known as the problem of catastrophic forgetting . Many recent works in continual learning of deep networks ( Li & Hoiem , 2016 ; Lee et al. , 2017 ; Shin et al. , 2017 ; Kirkpatrick et al. , 2017 ; Riemer et al. , 2019 ; Chaudhry et al. , 2019 ) tackle this problem by introducing advanced regularizations to prevent drastic change of network weights . Yet , when the model should adapt to a large number of tasks , the interference between task-specific knowledge is inevitable with fixed network capacity . Recently introduced expansion-based approaches handle this problem by expanding the network capacity as they adapt to new tasks ( Rusu et al. , 2016 ; Fang et al. , 2017 ; Yoon et al. , 2018 ; Li et al. , 2019 ) . These recent advances have largely alleviated the catastrophic forgetting , at least with a small number of tasks . However , to deploy continual learning to real-world systems , there are a number of issues that should be resolved . First , in practical scenarios , the number of tasks that the model should train on may be large . In the lifelong learning setting , the model may even have to continuously train on an unlimited number of tasks . Yet , conventional continual learning methods have not been verified for their scalability to a large number of tasks , both in terms of effectiveness in the prevention of catastrophic forgetting , and efficiency as to memory usage and computations ( See Figure 1 ( a ) , and ( b ) ) . Another important but relatively less explored problem is the problem of task order sensitivity , which describes the performance discrepancy with respect to the task arrival sequence ( See Figure 1 ( c ) ) . The task order that the model trains on has a large impact on the individual task performance as well as the final performance , not only because of the model drift coming from the catastrophic forgetting but due to the unidirectional knowledge transfer from earlier tasks to later ones . This order-sensitivity could be highly problematic if fairness across tasks is important ( e.g . disease diagnosis ) . To handle these practical challenges , we propose a novel continual learning model with Additive Parameter Decomposition ( APD ) . APD decomposes the network parameters at each layer of the target network into task-shared and sparse task-specific parameters with small mask vectors . At each arrival of a task to a network with APD , which we refer to as APD-Net , it will try to maximally utilize the task-shared parameters and will learn the incremental difference that can not be explained by the shared parameters using sparse task-adaptive parameters . Moreover , since having a single set of shared parameters may not effectively utilize the varying degree of knowledge sharing structure among the tasks , we further cluster the task-adaptive parameters to obtain hierarchically shared parameters ( See Figure 2 ) . This decomposition of generic and task-specific knowledge has clear advantages in tackling the previously mentioned problems . First , APD will largely alleviate catastrophic forgetting , since learning on later tasks will have no effect on the task-adaptive parameters for the previous tasks , and will update the task-shared parameters only with generic knowledge . Secondly , since APD does not change the network topology as existing expansion-based approaches do , APD-Net is memory-efficient , and even more so with hierarchically shared parameters . It also trains fast since it does not require multiple rounds of retraining . Moreover , it is order-robust since the task-shared parameters can stay relatively static and will converge to a solution rather than drift away upon the arrival of each task . With the additional mechanism to retroactively update task-adaptive parameters , it can further alleviate the order-sensitivity from unidirectional knowledge transfer as well . We validate our methods on several benchmark datasets for continual learning while comparing against state-of-the-art continual learning methods to obtain significantly superior performance with minimal increase in network capacity while being scalable and order-robust . The contribution of this paper is threefold : • We tackle practically important and novel problems in continual learning that have been overlooked thus far , such as scalability and order robustness . • We introduce a novel framework for continual deep learning that effectively prevents catastrophic forgetting , and is highly scalable and order-robust , which is based on the decomposition of the network parameters into shared and sparse task-adaptive parameters with small mask vectors . • We perform extensive experimental validation of our model on multiple datasets against recent continual learning methods , whose results show that our method is significantly superior to them in terms of the accuracy , efficiency , scalability , as well as order-robustness . 2 RELATED WORK . Continual Learning The literature on continual ( lifelong ) learning ( Thrun , 1995 ) is vast ( Ruvolo & Eaton , 2013 ) as it is a long-studied topic , but we only mention the most recent and relevant works . Most continual deep learning approaches are focused on preventing catastrophic forgetting , in which case the retraining of the network for new tasks shifts the distribution of the learned representations . A simple yet effective regularization is to enforce the representations learned at the current task to be closer to ones from the network trained on previous tasks ( Li & Hoiem , 2016 ) . A more advanced approach is to employ deep generative models to compactly encode task knowledge ( Shin et al. , 2017 ) and generate samples from the model later when learning for a novel task . Kirkpatrick et al . ( 2017 ) , and Schwarz et al . ( 2018 ) proposed to regularize the model parameter for the current tasks with parameters for the previous task via a Fisher information matrix , to find a solution that works well for both tasks , and Lee et al . ( 2017 ) introduces a moment-matching technique with a similar objective . Serrà et al . ( 2018 ) proposes a new binary masking approach to minimize drift for important prior knowledge . The model learns pseudo-step function to promote hard attention , then builds a compact network with a marginal forgetting . But the model can not expand the network capacity and performs unidirectional knowledge transfer thus suffers from the order-sensitivity . Lopez-Paz & Ranzato ( 2017 ) ; Chaudhry et al . ( 2019 ) introduces a novel approach for efficient continual learning with weighted update according to the gradients of episodic memory under single-epoch learning scenario . Nguyen et al . ( 2018 ) formulates continual learning as a sequential Bayesian update and use coresets , which contain important samples for each observed task to mitigate forgetting when estimating the posterior distribution over weights for the new task . Riemer et al . ( 2019 ) addresses the stability-plasticity dilemma maximizing knowledge transfer to later tasks while minimizing their interference on earlier tasks , using optimization-based meta-learning with experience replay . Dynamic Network Expansion Even with well-defined regularizers , it is nearly impossible to completely avoid catastrophic forgetting , since in practice , the model may encounter an unlimited number of tasks . An effective way to tackle this challenge is by dynamically expanding the network capacity to handle new tasks . Dynamic network expansion approaches have been introduced in earlier work such as Zhou et al . ( 2012 ) , which proposed an iterative algorithm to train a denoising autoencoder while adding in new neurons one by one and merging similar units . Rusu et al . ( 2016 ) proposed to expand the network by augmenting each layer of a network by a fixed number of neurons for each task , while keeping the old weights fixed to avoid catastrophic forgetting . Yet , this approach often results in a network with excessive size . Yoon et al . ( 2018 ) proposed to overcome these limitations via selective retraining of the old network while expanding each of its layer with only the necessary number of neurons , and further alleviate catastrophic forgetting by splitting and duplicating the neurons . Xu & Zhu ( 2018 ) proposed to use reinforcement learning to decide how many neurons to add . Li et al . ( 2019 ) proposes to perform an explicit network architecture search to decide how much to reuse the existing network weights and how much to add . Our model also performs dynamic network expansion as the previous expansion-based methods , but instead of adding in new units , it additively decomposes the network parameters into task-shared and task-specific parameters . Further , the capacity increase at the arrival of each task is kept minimal with the sparsity on the task-specific parameters and the growth is logarithmic with the hierarchical structuring of shared parameters . 3 CONTINUAL LEARNING WITH ADDITIVE PARAMETER DECOMPOSITION . In a continual learning setting , we assume that we have sequence of tasks { T1 , . . . , TT } arriving to a deep network in a random order . We denote the dataset of the tth task as Dt = { xit , yit } Nt i=1 , where xit and y i t are i th instance and label among Nt examples . We further assume that they become inaccessible after step t. The set of parameters for the network at step t is then given as Θt = { θlt } , where { θ l t } represents the set of weights for each layer l ; we omit the layer index l when the context is clear . Then the training objective at the arrival of task t can be defined as follows : minimizeΘt L ( Θt ; Θt−1 , Dt ) +λR ( Θt ) , whereR ( · ) is a regularization term on the model parameters . In the next paragraph , we introduce our continual learning framework with task-adaptive parameter decomposition and hierarchical knowledge consolidation . Additive Parameter Decomposition To minimize the effect of catastrophic forgetting and the amount of newly introduced parameters with network expansion , we propose to decompose θ into a task-shared parameter matrix σ and a task-adaptive parameter matrix τ , that is , θt = σ ⊗Mt + τ t for task t , where the masking variableMt acts as an attention on the task-shared parameter to guide the learner to focus only on the parts relevant for each task . This decomposition allows us to easily control the trade-off between semantic drift and predictive performance of a new task by imposing separate regularizations on decomposed parameters . When a new task arrives , we encourage the shared parameters σ to be properly updated , but not deviate far from the previous shared parameters σ ( t−1 ) . At the same time , we enforce the capacity of τ t to be as small as possible , by making it sparse . The objective function for this decomposed parameter model is given as follows : minimize σ , τ t , vt L ( { σ ⊗Mt + τ t } ; Dt ) + λ1‖τ t‖1 + λ2‖σ − σ ( t−1 ) ‖22 , ( 1 ) where L denotes a loss function , σ ( t−1 ) denotes the shared parameter before the arrival of the current task t , ‖ · ‖1 indicates an element-wise ` 1 norm defined on the matrix , and λ1 , λ2 are hyperparameters balancing efficiency catastrophic forgetting . We use ` 2 transfer regularization to prevent catastrophic forgetting , but we could use other types of regularizations as well , such as Elastic Weight Consolidation ( Kirkpatrick et al. , 2017 ) . The masking variableMt is a sigmoid function with a learnable parameter vt , which is applied to output channels or neurons of σ in each layer . We name our model with decomposed network parameters , Additive Parameter Decomposition ( APD ) . The proposed decomposition in ( 1 ) makes continual learning efficient , since at each task we only need to learn a very sparse τ t that accounts for task-specific knowledge that can not be explained with the transformed shared knowledge σ ⊗Mt . Thus , in a way , we are doing residual learning with τ t. Further , it helps the model achieve robustness to the task arrival order , because semantic drift occurs only through the task-shared parameter that corresponds to generic knowledge , while the task-specific knowledge learned from previous tasks are kept intact . In the next section , we introduce additional techniques to achieve even more task-order robustness and efficiency . Order Robust Continual Learning with Retroactive Parameter Updates We observe that a naive update of the shared parameters may induce semantic drift in parameters for the previously trained tasks which will yield an order-sensitive model , since we do not have access to previous task data . In order to provide high degree of order-robustness , we impose an additional regularization to further prevent parameter-level drift without explicitly training on the previous tasks . To achieve order-robustness in ( 1 ) , we need to retroactively update task adaptive parameters of the past tasks to reflect the updates in the shared parameters at each training step , so that all previous tasks are able to maintain their original solutions . Toward this objective , when a new task t arrives , we first recover all previous parameters ( θi for task i < t ) : θ∗i = σ ( t−1 ) ⊗M ( t−1 ) i + τ ( t−1 ) i and then update τ 1 : t−1 by constraining the combined parameter σ ⊗Mi + τ i to be close to θ∗i . The learning objective for the current task t is then described as follows : minimize σ , τ1 : t , v1 : t L ( { σ ⊗Mt + τ t } ; Dt ) + λ1 t∑ i=1 ‖τ i‖1 + λ2 t−1∑ i=1 ‖θ∗i − ( σ ⊗Mi + τ i ) ‖22 . ( 2 ) Compared to ( 1 ) , the task-adaptive parameters of previous tasks now can be retroactively updated to minimize the parameter-level drift . This formulation also constrains the update of the task-shared parameters to consider order-robustness . Algorithm 1 Continual learning with Additive Parameter Decomposition input Dataset D1 : T and hyperparameter λ , m , s , K = k output σ ( T ) , v1 : T , σ̃1 : K , and τ 1 : T 1 : Let σ ( 1 ) = θ1 , and optimize for the task 1 2 : for t = 2 , ... , T do 3 : for i = 1 , ... , t− 1 do 4 : Restore θ∗i = σ ( t−1 ) ⊗M ( t−1 ) i + τ̃ ( t−1 ) i 5 : end for 6 : Minimize ( 3 ) to update σ and { τ i , vi } ti=1 7 : if t mod s = 0 then 8 : Initialize k new random centroids , { µg } Kg=K−k+1 9 : Group all tasks into K disjoint sets , { Gg } Kg=1 10 : for g = 1 , ... , K do 11 : Decompose { τ̃ i } i∈Gg into σ̃g and { τ i } i∈Gg 12 : end for 13 : Delete old σ̃ and K = K + k 14 : end if 15 : end for Hierarchical Knowledge Consolidation The objective function in ( 2 ) does not directly consider local sharing among the tasks , and thus it will inevitably result in the redundancy of information in the task-adaptive parameters . To further minimize the capacity increase , we perform a process called hierarchical knowledge consolidation to group relevant task-adaptive parameters into taskshared parameters ( See Figure 2 ) . We first group all tasks into K disjoint sets { Gg } Kg=1 using K-means clustering on { τ i } ti=1 , then decompose the task-adaptive parameters in the same group into locally-shared parameters σ̃g and task-adaptive parameters { τ i } i∈Gg ( with higher sparsity ) by simply computing the amount of value discrepancy in each parameter as follows : • If max { τ i , j } i∈Gg −min { τ i , j } i∈Gg ≤ β , then { τ i , j } i∈Gg = 0 and σ̃g , j = µg , j • Else , σ̃g , j = 0 , where τ i , j denotes the jth element of the ith task-adaptive parameter matrix , and µg is the cluster center of group Gg . We update the locally-shared parameters σ̃g after the arrival of every s tasks for efficiency , by performing K-means clustering while initializing the cluster centers with the previous locally-shared parameters σ̃g for each group . At the same time , we increase the number of centroids to K + k to account for the increase in the variance among the tasks . Our final objective function is then given as follows : minimize σ , τ1 : t , v1 : t L ( { σ ⊗Mt + τ t } ; Dt ) + λ1 t∑ i=1 ‖τ i‖1 + λ2 t−1∑ i=1 ‖θ∗i − ( σ ⊗Mi + τ̃ i ) ‖22 , where τ̃ i = τ i + σ̃g for i ∈ Gg . ( 3 ) Algorithm 1 describes the training of our APD model . Selective task forgetting In practical scenarios , some of earlier learned tasks may become irrelevant as we continually train the model . For example , when we are training a product identification model , recognition of discontinued products will be unnecessary . In such situations , we may want to forget the earlier tasks in order to secure network capacity for later task learning . Unfortunately , existing continual learning methods can not effectively handle this problem , since the removal of some features or parameters will also negatively affect the remaining tasks as their parameters are entangled . Yet , with APDs , forgetting of a task t can be done by dropping out the task adaptive parameters τ t. Trivially , this will have absolutely no effect on the task-adaptive parameters of the remaining tasks .

Summary: The paper addresses continual learning challenges such as catastrophic forgetting and task-order robustness by introducing a new hybrid algorithm that uses architecture growth as well as parameter regularization where parameters of each layer are decomposed into task-specific and task-private parameters. They also use a simple trick to The authors perform experiments on Split CIFAR100, CIFAR100 Superclass, Omniglot, and a sequence of 3 datasets (SVHN,CIFAR10,CIFAR100). The maximum number of tasks in the experiments is 100 for Omniglot-rotation. The authors show superior performance to EWC (a regularization-based method), P&C (architecture-based method), DEN (architecture-based method), PGN (architecture-based method), RCL (architecture-based method), etc.