Continual Learning with Recursive Gradient Optimization

1 INTRODUCTION . In many application scenarios , one needs to learn a sequence of tasks without access to historical data , called continual learning . Although variants of stochastic gradient descent ( SGD ) have made a significant contribution to the progress made by neural networks in many fields , these optimizers require the mini-batches of data to satisfy the independent identically distributed ( i.i.d . ) assumption . In continual learning , the violation of this requirement leads to significant degradation of performance on previous tasks , called catastrophic forgetting . Recent works attempt to tackle this issue by modifying the training process from a variety of perspectives . Memory-based approaches use extra memory to store some samples ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2020 ) , gradients ( Chaudhry et al. , 2019a ; 2020 ; Saha et al. , 2021 ) , or their generative models ( Shin et al. , 2017 ; Shen et al. , 2020 ) to modify future training process . The memory for replay leads to a linear-increased space complexity with respect to the number of tasks . Expansionbased approaches select the network parameters dynamically ( Yoon et al. , 2018 ; Rosenbaum et al. , 2018 ; Serra et al. , 2018 ; Kaushik et al. , 2021 ) , add additional components as new tasks arrive ( Rusu et al. , 2016 ; Fernando et al. , 2017 ; Alet et al. , 2018 ; Chang et al. , 2019 ; Li et al. , 2019 ) , or use larger networks to generate network parameters ( Aljundi et al. , 2017 ; Yoon et al. , 2019 ; von Oswald et al. , 2019 ) . These methods reduce interference between tasks by additional task-specific parameters . Single-Task Learning ( STL ) can also be regarded as an expansion-based method which trains a network for each task separately . Regularization-based approaches encourage important parameters to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Yin et al. , 2020 ) or constraining the direction of parameter update ( Farajtabar et al. , 2019 ; Chaudhry et al. , 2019a ; Saha et al. , 2021 ) . Our method is also regularization-based which combines the advantages of loss penalty and gradient constraint . In this work , we focus on continual learning in a fixed-capacity network without data replay . We aim to minimize the expected increment of the total loss of past tasks without reducing the performance of the current task . To this end , we propose an upper bound of quadratic loss estimation and design a recursive optimization procedure to modify the direction of the gradient to the optimal solu- tion under this upper bound . In addition , we introduce trace normalization process to guarantee the learning rate during the training process according to the principle of current-task-first ( CFT ) . This normalization process makes our approach compatible with the vast majority of existing models and learning strategies well-designed for single-task solutions . As the gradient modification process is independent of data samples and previous parameters , our optimizer can be used directly in most deep architecture networks as typical single-task optimizers like SGD . Further , to reduce the interference between tasks , we develop a feature encoding strategy to represent the multi-modal structure of the network without additional parameters . A virtual feature encoding layer ( FEL ) which randomly permutes the output feature maps using integer task descriptor as seed is attached after each real layer . Thus , each task obtains a specific virtual structure under same network parameters . Since the parameter space of the network is not changed , such a strategy will not change the fitting ability of the neural network . Experimental validations on several continual learning benchmarks show that the proposed method has significantly less forgetting and higher accuracy than existing fixedcapacity baselines . We achieve state-of-the-art performance on 20-split-CIFAR100 ( 82.22 % ) and 20-split-miniImageNet ( 72.62 % ) . In addition to minimizing forgetting , this method has comparable or better performance than single-task learning which handles all tasks individually . 2 PRELIMINARIES . Consider K sequentially arrived supervised learning tasks { Tk|k ∈ [ K ] } , where [ N ] : = { 1 , 2 , · · · , N } for any positive integer N . In each task Tk , there are nk data points { ( xk , i , yk , i ) |i ∈ [ nk ] } sampled from an unknown distribution Dk . Let X , Y , W be the space of inputs , targets and model parameters . By denoting the predictor as f ( x , k ) : X × [ K ] → Y , the loss function of θ ∈ W associated with data point ( x , y ) and task identifier k can be expressed as l ( f ( θ ; x , k ) , y ) : W → R , and the empirical loss function of task Tk is defined as : Lk ( θ ) = 1 nk nk∑ i=1 l ( f ( θ ; k , xk , i ) , yk , i ) ( 1 ) In the continual learning scenario studied in this work , the parameter space W remains a fixed size , and an integer task descriptor is provided at both training and testing time . Without access to past samples , we use a second-order Taylor expansion to estimate the loss function of previous tasks.Let θ∗j be the optimal parameter of Lj ( θ ) generated by the gradient descent process according to ∇θ∗jLj = 0 . For a new model parameter θ in the neighborhood of θ ∗ j , the loss of previous task Tj ( j < k ) can be estimated as : Lj ( θ ) = Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ( 2 ) where Hj : = ∇2Lj ( θ∗j ) is the Hessian matrix . 3 PROBLEM FORMULATION & SOLUTION . In our fixed-capacity continual learning setting , finding an appropriate joint solution that works well across the task sequence is the core goal . To this end , we introduce a novel continual learning optimization problem and corresponding iterative optimization strategy . 3.1 OPTIMIZATION PROBLEM . In this paper , we formalize forgetting as the the increment of old task losses . As mentioned in Section 2 , the total loss of tasks before Tk , denoted as Fk , can be estimated by : Fk ( θ ) = k−1∑ j=1 Lj ( θ ) ≈ k−1∑ j=1 [ Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ] ( 3 ) As Equation ( 3 ) need the explicit value of previous model parameters , Fk is too expensive to be an optimization target in continual learning . We turn to a more concise form which we refer to as recursive least loss ( RLL ) : FRLLk ( θ ) : = 1 2 ( θ − θ∗k−1 ) T ( k−1∑ j=1 Hj ) ( θ − θ∗k−1 ) ( 4 ) In Appendix A.2 , we prove that FRLLk and Fk are equivalent for optimization if all previous tasks are fully-trained . Based on the conclusions above , the optimization problem during task Tk is formalized as : θ∗k : min θ FRLLk ( θ ) , subject to∇Lk ( θ ) = 0 ( 5 ) FRLLk has the same form as the regularization term in many regularization-based methods . The optimization goal of these methods are variants of Lk ( θ ) +λFRLLk ( θ ) , derived from Bayesian posterior approximation with Gaussian prior ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ) or approximation of the KL-divergence used in natural gradient descent ( Amari , 1998 ; Ritter et al. , 2018 ; Tseran et al. , 2018 ) . The Bayesian methods try to estimate and minimize the overall loss function , while our method prioritizes the performance of the current task by ∇Lk ( θ ) = 0 and minimizes the expected forgetting of the past tasks FRLLk ( θ ) . 3.2 GRADIENT MODIFICATION . For the newest task Tk , the optimal solution where ∇Lk ( θ ) = 0 should be obtained by stochastic gradient descent started from the former optimal model parameter θ∗k−1 at the end of task Tk−1 . Using subscript ·i to represent the parameters of step i , and the initial state θ0 = θ∗k−1 , the single step update can be expressed as : θi = θi−1 − ηi∇Lk ( θi−1 ) Assume that the pre-set learning rate ηi is small enough to ignore the higher order terms , the loss function after the one-step update can be expressed as : Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) T∇Lk ( θi−1 ) If we hope to solve the task Tk only , updating the parameter θ according to the gradient above is enough . However , as mentioned above , such a method will encourage the neural network to gradually forget the old tasks . Therefore , we modify the update direction to minimize the expectation of forgetting . To this end , we introduce a new positive definite symmetric matrix P with appropriate dimensions to modify the gradients ( g → Pg ) . The modified one-step update is : { θi = θi−1 − ηiP∇Lk ( θi−1 ) Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 6 ) To maintain the pre-set learning rate during the continual learning problem and avoid repetitive selection of hyper-parameters , we impose an additional constraint on the trace of the projection matrix and prove the corresponding convergence rate consistency theorem 1 in Appendix A.1 . Theorem 1 ( convergence rate consistency ) . Under the constraint of trace ( P ) =dim ( P ) , the expectation of the learning rate for unknown isotropic distribution is the same as the original optimizer . For common network structures , we provide the space complexity of matrix P and the time complexity of the corresponding gradient modification process in Appendix B.2 . As described above , the only extra memory of our approach is the projection matrix P , which contains the information of previous tasks . This allows our approach to be a space-invariant method different with typical memory-based or expansion-based continual learning method . As the performance of our method is identified with the choice of P , the following problem is : How to find a good projection matrix ? We will answer this question in the following parts . 3.3 APPROXIMATE SOLUTION . The next step is to find a solution for problem ( 5 ) . When the training process on Tk is finished , the final state θ∗k and the residual loss can be obtained from the accumulation of one-step update : θ∗k = θ ∗ k−1 − nk∑ i=1 ηiP∇Lk ( θi−1 ) Lk ( θ ∗ k ) = Lk ( θ ∗ k−1 ) − nk∑ i=1 ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 7 ) In this way , for a given sample sequence and initial value θ∗k−1 , the result of θ ∗ k depends on P . The optimization problem 5 on θ∗k is transformed into an optimization problem on P . However , the relationship between FRLLk ( θ ∗ k ) and P is too complicated to be used in optimization process . To tackle this problem , we propose an upper bound of FRLLk ( θ ∗ k ) as a practical optimization target . Theorem 2 ( upper bound ) . Denote σ̂m ( · ) as the symbol for maximum eigenvalue and ηm as the maximum single-step learning rate , the recursive least loss has an upper bound : FRLLk ( θ ∗ k ) ≤ 1 2 nkηmσ̂m ( PH̄ ) Lk ( θ ∗ k−1 ) ( 8 ) where H̄ = ∑k−1 j=1 Hj is defined as the sum of the Hessian matrices of all old tasks . We prove theorem 2 in Appendix A.3 . Discarding the constant terms , we get an alternative optimization problem for projection matrix P : P : { min P σ̂m ( PH̄ ) subject to trace ( P ) = dim ( P ) ( 9 ) We provide a detailed solution of this problem in Appendix A.4 . The normalized solution is : P = dim ( H̄ ) trace ( H̄−1 ) H̄−1 ( 10 ) The normalized projection procedure can be described as : finding a new gradient that has similar effects on the current task to minimize the upper bound of the old task losses . Our optimizer only modifies the direction of the gradient without reducing the search region , which will guarantee the fitting ability of the network consistent throughout the task sequence .

Summary: The paper addresses a long-standing problem of deep neural networks where new training data for new tasks arrive continuously and data does not stay forever (continual learning). The network has to learn to cater for all the tasks without forgetting what it learnt for an older task as data from older tasks are no longer available. The authors came up with a formulation where learning will mean minimizing the (expectation of) forgetting and forgetting is formalized as increase of losses corresponding to the old tasks. The authors then move on to find a Taylor series expression of forgetting and an equivalent Recursive Least Loss (RLL) formulation of the loss (aka forgetting). Next the authors introduce a modification in the direction of gradient update of normal SGD that makes sure to minimize forgetting. The modification implied the introduction of a positive definite matrix and the optimization problem transformed into an optimization problem on finding the optimal positive definite matrix minimizing the RLL. As the relation between RLL and the positive definite matrix is not straightforward, an upper bound of RLL is proposed and the optimal solution is found under this upper bound. One more interesting proposal is a virtual feature encoding layer (FEL) that applies task-specific random rearrangement to the input feature maps. FEL eliminates possible interference between tasks by decorrelating features among old tasks without expanding the memory footprint. The experiments show great improvement over sota approaches on benchmark datasets and sometimes (rotated MNIST, Split ImageNet) even goes beyond supposed upperbound performance given by single task learning showing evidence of positive transfer.  Overall, I liked the approach, the presentation and the experimental results with only a few minor concerns detailed below.