CPR: Classifier-Projection Regularization for Continual Learning

1 INTRODUCTION . Catastrophic forgetting ( McCloskey & Cohen , 1989 ) is a central challenge in continual learning ( CL ) : when training a model on a new task , there may be a loss of performance ( e.g. , decrease in accuracy ) when applying the updated model to previous tasks . At the heart of catastrophic forgetting is the stability-plasticity dilemma ( Carpenter & Grossberg , 1987 ; Mermillod et al. , 2013 ) , where a model exhibits high stability on previously trained tasks , but suffers from low plasticity for the integration of new knowledge ( and vice-versa ) . Attempts to overcome this challenge in neural network-based CL can be grouped into three main strategies : regularization methods ( Li & Hoiem , 2017 ; Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Nguyen et al. , 2018 ; Ahn et al. , 2019 ; Aljundi et al. , 2019 ) , memory replay ( Lopez-Paz & Ranzato , 2017 ; Shin et al. , 2017 ; Rebuffi et al. , 2017 ; Kemker & Kanan , 2018 ) , and dynamic network architecture ( Rusu et al. , 2016 ; Yoon et al. , 2018 ; Golkar et al. , 2019 ) . In particular , regularization methods that control model weights bear the longest history due to its simplicity and efficiency to control the trade-off for a fixed model capacity . In parallel , several recent methods seek to improve the generalization of neural network models trained on a single task by promoting wide local minima ( Keskar et al. , 2017 ; Chaudhari et al. , 2019 ; Pereyra et al. , 2017 ; Zhang et al. , 2018 ) . Broadly speaking , these efforts have experimentally shown that models trained with wide local minima-promoting regularizers achieve better generalization and higher accuracy ( Keskar et al. , 2017 ; Pereyra et al. , 2017 ; Chaudhari et al. , 2019 ; Zhang et al. , 2018 ) , and can be more robust to weight perturbations ( Zhang et al. , 2018 ) when compared to usual training methods . Despite the promising results , methods that promote wide local minima have yet to be applied to CL . In this paper , we make a novel connection between wide local minima in neural networks and regularization-based CL methods . The typical regularization-based CL aims to preserve important weight parameters used in past tasks by penalizing large deviations when learning new tasks . As ∗Corresponding author ( E-mail : tsmoon @ snu.ac.kr ) shown in the top of Fig . 1 , a popular geometric intuition ( as first given in EWC ( Kirkpatrick et al. , 2017 ) ) for such CL methods is to consider the ( uncertainty ) ellipsoid of parameters around the local minima . When learning new tasks , parameter updates are selected in order to not significantly hinder model performance on past tasks . Our intuition is that promoting a wide local minima—which conceptually stands for local minima having a flat , rounded uncertainty ellipsoid—can be particularly beneficial for regularization-based CL methods by facilitating diverse update directions for the new tasks ( i.e. , improves plasticity ) while not hurting the past tasks ( i.e. , retains stability ) . As shown in the bottom of Fig . 1 , when the ellipsoid containing the parameters with low-error is wider , i.e. , when the wide local minima exists , there is more flexibility in finding a parameter that performs well for all tasks after learning a sequence of new tasks . We provide further details in Section 2.1 . Based on the above intuition , we propose a general , yet simple patch that can be applied to existing regularization-based CL methods dubbed as Classifier-Projection Regularization ( CPR ) . Our method implements an additional regularization term that promotes wide local minima by maximizing the entropy of the classifier ’ s output distribution . Furthermore , from a theory standpoint , we make an observation that our CPR term can be further interpreted in terms of information projection ( I-projection ) formulations ( Cover & Thomas , 2012 ; Murphy , 2012 ; Csiszár & Matus , 2003 ; Walsh & Regalia , 2010 ; Amari et al. , 2001 ; Csiszár & Matus , 2003 ; Csiszár & Shields , 2004 ) found in information theory . Namely , we argue that applying CPR corresponds to projecting a classifier ’ s output onto a Kullback-Leibler ( KL ) divergence ball of finite radius centered around the uniform distribution . By applying the Pythagorean theorem for KL divergence , we then prove that this projection may ( in theory ) improve the performance of continual learning methods . Through extensive experiments on several benchmark datasets , we demonstrate that applying CPR can significantly improve the performance of the state-of-the-art regularization-based CL : using our simple patch improves both the stability and plasticity and , hence , achieves better average accuracy almost uniformly across the tested algorithms and datasets—confirming our intuition of wide local minima in Fig . 1 . Furthermore , we use a feature map visualization that compares methods trained with and without CPR to further corroborate the effectiveness of our method . 2 CPR : CLASSIFIER-PROJECTION REGULARIZATION FOR WIDE LOCAL MINIMUM . In this section , we elaborate in detail the core motivation outlined in Fig . 1 , then formalize CPR as the combination of two regularization terms : one stemming from prior regularization-based CL methods , and the other that promotes a wide local minima . Moreover , we provide an information-geometric interpretation ( Csiszár , 1984 ; Cover & Thomas , 2012 ; Murphy , 2012 ) for the observed gain in performance when applying CPR to CL . We consider continual learning of T classification tasks , where each task contains N training samplelabel pairs { ( xtn , ytn ) } Nn=1 , t ∈ [ 1 , · · · , T ] with xtn ∈ Rd , and the labels of each task has Mt classes , i.e. , ytn ∈ [ 1 , · · · , Mt ] . Note that task boundaries are given in evaluation time ; i.e. , we consider a task-aware setting . We denote fθ : Rd → ∆M as a neural network-based classification model with softmax output layer parameterized by θ . 2.1 MOTIVATION : INTRODUCING WIDE LOCAL MINIMA IN CONTINUAL LEARNING . Considering the setting of typical regularization-based CL ( top of Fig . 1 ) , we denote θ∗i as parameters that achieve local minima for a specific task i and θ̂i is that obtained with regularization terms . Assuming that θ∗1 is learnt , when learning task 2 , an appropriate regularization updates the parameters from θ∗1 to θ̂2 instead of θ ∗ 2 , since θ̂2 achieves low-errors on both tasks 1 and 2 . However , when the low-error regimes ( ellipsoids in Fig . 1 ) are narrow , it is often infeasible to obtain a parameter that performs well on all three tasks . This situation results in the trade-off between stability and plasticity in regularization-based CL ( Carpenter & Grossberg , 1987 ) . Namely , stronger regularization strength ( direction towards past tasks ) brings more stability ( θ̂13 ) , and hence less forgetting on past tasks . In contrast , weaker regularization strength ( direction towards future tasks ) leads to more plasticity so that the updated parameter θ̂23 performs better on recent tasks , at the cost of compromising the performance of past tasks . A key problem in the previous setting is that the parameter regimes that achieve low error for each task are often narrow and do not overlap with each other . Therefore , a straightforward solution is to enlarge the low-error regimes such that they have non-empty intersections with higher chance . This observation motivates us to consider wide local minima for each task in CL ( bottom of Fig . 1 ) . With wide local minima for each task , a regularization-based CL can more easily find a parameter , θ̂3 , that is close to the the local minimas for each task , i.e. , { θ∗i } 3i=1 . Moreover , it suggests that once we promote the wide local minima of neural networks during continual learning , both the stability and plasticity could potentially be improved and result in simultaneously higher accuracy for all task — which is later verified in our experimental ( see Sec . 3 ) . In the next section , we introduce the formulation of wide local minima in CL . 2.2 CLASSIFIER PROJECTION REGULARIZATION FOR CONTINUAL LEARNING . Regularization-based continual learning Typical regularization-based CL methods attach a regularization term that penalizes the deviation of important parameters learned from past tasks in order to mitigate catastrophic forgetting . The general loss form for these methods when learning task t is LtCL ( θ ) = L t CE ( θ ) + λ ∑ i Ωt−1i ( θi − θ t−1 i ) 2 , ( 1 ) where LtCE ( θ ) is the ordinary cross-entropy loss function for task t , λ is the dimensionless regularization strength , Ωt−1 = { Ωt−1i } is the set of estimates of the weight importance , and { θ t−1 i } is the parameter learned until task t− 1 . A variety of previous work , e.g. , EWC ( Kirkpatrick et al. , 2017 ) , SI ( Zenke et al. , 2017 ) , MAS ( Aljundi et al. , 2018 ) , and RWalk ( Chaudhry et al. , 2018 ) , proposed different ways of calculating Ωt−1 to measure weight importance . Single-task wide local minima Several recent schemes have been proposed ( Pereyra et al. , 2017 ; Szegedy et al. , 2016 ; Zhang et al. , 2018 ) to promote wide local minima of a neural network for solving a single task . These approaches can be unified by the following common loss form LWLM ( θ ) = LCE ( θ ) + β N N∑ n=1 DKL ( fθ ( xn ) ‖g ) , ( 2 ) where g is some probability distribution in ∆M that regularizes the classifier output fθ , β is a trade-off parameter , andDKL ( ·‖· ) is the KL divergence ( Cover & Thomas , 2012 ) . Note that , for example , when g is uniform distribution PU in ∆M , the regularization term corresponds to entropy maximization proposed in Pereyra et al . ( 2017 ) , and when g is another classifier ’ s output fθ′ , Eq . ( 2 ) becomes equivalent to the loss function in Zhang et al . ( 2018 ) . CPR : Achieving wide local minima in continual learning Combining the above two regularization terms , we propose the CPR as the following loss form for learning task t : LtCPR ( θ ) = L t CE ( θ ) + β N N∑ n=1 DKL ( fθ ( x t n ) ‖PU ) + λ ∑ i Ωt−1i ( θi − θ t−1 i ) 2 , ( 3 ) where λ and β are the regularization parameters . The first regularization term promotes the wide local minima while learning task t by using PU as the regularizing distribution g in ( 2 ) , and the second term is from the typical regularization-based CL . Note that this formulation is oblivious to Ωt−1 and , hence , it can be applied to any state-of-the-art regularization-based CL methods . In our experiments , we show that the simple addition of the KL-term can significantly boost the performance of several representative state-of-the-art methods , confirming our intuition on wide local minima for CL given in Section 2.1 and Fig 1 . Furthermore , we show in the next section that the KL-term can be geometrically interpreted in terms of information projections ( Csiszár , 1984 ; Cover & Thomas , 2012 ; Murphy , 2012 ) , providing an additional argument ( besides promoting wide local minima ) for the benefit of using CPR in continual learning .

Summary: The paper proposes to add a KL-divergence regularization to the objective of regularized continual learning in order to encourage the output prediction to be close to a uniform distribution over classes (i.e., increasing the entropy). They argue that this regularization makes the local minima flat and thus less prone to forgetting. They try to build a theoretical connection using results from information projection but there is still a large gap. In experiments, they show on several benchmarks that applying the KL divergence regularization to different regularization-based continual learning brings improvements.