Continual Learning with Recursive Gradient Optimization

1 INTRODUCTION . In many application scenarios , one needs to learn a sequence of tasks without access to historical data , called continual learning . Although variants of stochastic gradient descent ( SGD ) have made a significant contribution to the progress made by neural networks in many fields , these optimizers require the mini-batches of data to satisfy the independent identically distributed ( i.i.d . ) assumption . In continual learning , the violation of this requirement leads to significant degradation of performance on previous tasks , called catastrophic forgetting . Recent works attempt to tackle this issue by modifying the training process from a variety of perspectives . Memory-based approaches use extra memory to store some samples ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2020 ) , gradients ( Chaudhry et al. , 2019a ; 2020 ; Saha et al. , 2021 ) , or their generative models ( Shin et al. , 2017 ; Shen et al. , 2020 ) to modify future training process . The memory for replay leads to a linear-increased space complexity with respect to the number of tasks . Expansionbased approaches select the network parameters dynamically ( Yoon et al. , 2018 ; Rosenbaum et al. , 2018 ; Serra et al. , 2018 ; Kaushik et al. , 2021 ) , add additional components as new tasks arrive ( Rusu et al. , 2016 ; Fernando et al. , 2017 ; Alet et al. , 2018 ; Chang et al. , 2019 ; Li et al. , 2019 ) , or use larger networks to generate network parameters ( Aljundi et al. , 2017 ; Yoon et al. , 2019 ; von Oswald et al. , 2019 ) . These methods reduce interference between tasks by additional task-specific parameters . Single-Task Learning ( STL ) can also be regarded as an expansion-based method which trains a network for each task separately . Regularization-based approaches encourage important parameters to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Yin et al. , 2020 ) or constraining the direction of parameter update ( Farajtabar et al. , 2019 ; Chaudhry et al. , 2019a ; Saha et al. , 2021 ) . Our method is also regularization-based which combines the advantages of loss penalty and gradient constraint . In this work , we focus on continual learning in a fixed-capacity network without data replay . We aim to minimize the expected increment of the total loss of past tasks without reducing the performance of the current task . To this end , we propose an upper bound of quadratic loss estimation and design a recursive optimization procedure to modify the direction of the gradient to the optimal solu- tion under this upper bound . In addition , we introduce trace normalization process to guarantee the learning rate during the training process according to the principle of current-task-first ( CFT ) . This normalization process makes our approach compatible with the vast majority of existing models and learning strategies well-designed for single-task solutions . As the gradient modification process is independent of data samples and previous parameters , our optimizer can be used directly in most deep architecture networks as typical single-task optimizers like SGD . Further , to reduce the interference between tasks , we develop a feature encoding strategy to represent the multi-modal structure of the network without additional parameters . A virtual feature encoding layer ( FEL ) which randomly permutes the output feature maps using integer task descriptor as seed is attached after each real layer . Thus , each task obtains a specific virtual structure under same network parameters . Since the parameter space of the network is not changed , such a strategy will not change the fitting ability of the neural network . Experimental validations on several continual learning benchmarks show that the proposed method has significantly less forgetting and higher accuracy than existing fixedcapacity baselines . We achieve state-of-the-art performance on 20-split-CIFAR100 ( 82.22 % ) and 20-split-miniImageNet ( 72.62 % ) . In addition to minimizing forgetting , this method has comparable or better performance than single-task learning which handles all tasks individually . 2 PRELIMINARIES . Consider K sequentially arrived supervised learning tasks { Tk|k ∈ [ K ] } , where [ N ] : = { 1 , 2 , · · · , N } for any positive integer N . In each task Tk , there are nk data points { ( xk , i , yk , i ) |i ∈ [ nk ] } sampled from an unknown distribution Dk . Let X , Y , W be the space of inputs , targets and model parameters . By denoting the predictor as f ( x , k ) : X × [ K ] → Y , the loss function of θ ∈ W associated with data point ( x , y ) and task identifier k can be expressed as l ( f ( θ ; x , k ) , y ) : W → R , and the empirical loss function of task Tk is defined as : Lk ( θ ) = 1 nk nk∑ i=1 l ( f ( θ ; k , xk , i ) , yk , i ) ( 1 ) In the continual learning scenario studied in this work , the parameter space W remains a fixed size , and an integer task descriptor is provided at both training and testing time . Without access to past samples , we use a second-order Taylor expansion to estimate the loss function of previous tasks.Let θ∗j be the optimal parameter of Lj ( θ ) generated by the gradient descent process according to ∇θ∗jLj = 0 . For a new model parameter θ in the neighborhood of θ ∗ j , the loss of previous task Tj ( j < k ) can be estimated as : Lj ( θ ) = Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ( 2 ) where Hj : = ∇2Lj ( θ∗j ) is the Hessian matrix . 3 PROBLEM FORMULATION & SOLUTION . In our fixed-capacity continual learning setting , finding an appropriate joint solution that works well across the task sequence is the core goal . To this end , we introduce a novel continual learning optimization problem and corresponding iterative optimization strategy . 3.1 OPTIMIZATION PROBLEM . In this paper , we formalize forgetting as the the increment of old task losses . As mentioned in Section 2 , the total loss of tasks before Tk , denoted as Fk , can be estimated by : Fk ( θ ) = k−1∑ j=1 Lj ( θ ) ≈ k−1∑ j=1 [ Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ] ( 3 ) As Equation ( 3 ) need the explicit value of previous model parameters , Fk is too expensive to be an optimization target in continual learning . We turn to a more concise form which we refer to as recursive least loss ( RLL ) : FRLLk ( θ ) : = 1 2 ( θ − θ∗k−1 ) T ( k−1∑ j=1 Hj ) ( θ − θ∗k−1 ) ( 4 ) In Appendix A.2 , we prove that FRLLk and Fk are equivalent for optimization if all previous tasks are fully-trained . Based on the conclusions above , the optimization problem during task Tk is formalized as : θ∗k : min θ FRLLk ( θ ) , subject to∇Lk ( θ ) = 0 ( 5 ) FRLLk has the same form as the regularization term in many regularization-based methods . The optimization goal of these methods are variants of Lk ( θ ) +λFRLLk ( θ ) , derived from Bayesian posterior approximation with Gaussian prior ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ) or approximation of the KL-divergence used in natural gradient descent ( Amari , 1998 ; Ritter et al. , 2018 ; Tseran et al. , 2018 ) . The Bayesian methods try to estimate and minimize the overall loss function , while our method prioritizes the performance of the current task by ∇Lk ( θ ) = 0 and minimizes the expected forgetting of the past tasks FRLLk ( θ ) . 3.2 GRADIENT MODIFICATION . For the newest task Tk , the optimal solution where ∇Lk ( θ ) = 0 should be obtained by stochastic gradient descent started from the former optimal model parameter θ∗k−1 at the end of task Tk−1 . Using subscript ·i to represent the parameters of step i , and the initial state θ0 = θ∗k−1 , the single step update can be expressed as : θi = θi−1 − ηi∇Lk ( θi−1 ) Assume that the pre-set learning rate ηi is small enough to ignore the higher order terms , the loss function after the one-step update can be expressed as : Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) T∇Lk ( θi−1 ) If we hope to solve the task Tk only , updating the parameter θ according to the gradient above is enough . However , as mentioned above , such a method will encourage the neural network to gradually forget the old tasks . Therefore , we modify the update direction to minimize the expectation of forgetting . To this end , we introduce a new positive definite symmetric matrix P with appropriate dimensions to modify the gradients ( g → Pg ) . The modified one-step update is : { θi = θi−1 − ηiP∇Lk ( θi−1 ) Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 6 ) To maintain the pre-set learning rate during the continual learning problem and avoid repetitive selection of hyper-parameters , we impose an additional constraint on the trace of the projection matrix and prove the corresponding convergence rate consistency theorem 1 in Appendix A.1 . Theorem 1 ( convergence rate consistency ) . Under the constraint of trace ( P ) =dim ( P ) , the expectation of the learning rate for unknown isotropic distribution is the same as the original optimizer . For common network structures , we provide the space complexity of matrix P and the time complexity of the corresponding gradient modification process in Appendix B.2 . As described above , the only extra memory of our approach is the projection matrix P , which contains the information of previous tasks . This allows our approach to be a space-invariant method different with typical memory-based or expansion-based continual learning method . As the performance of our method is identified with the choice of P , the following problem is : How to find a good projection matrix ? We will answer this question in the following parts . 3.3 APPROXIMATE SOLUTION . The next step is to find a solution for problem ( 5 ) . When the training process on Tk is finished , the final state θ∗k and the residual loss can be obtained from the accumulation of one-step update : θ∗k = θ ∗ k−1 − nk∑ i=1 ηiP∇Lk ( θi−1 ) Lk ( θ ∗ k ) = Lk ( θ ∗ k−1 ) − nk∑ i=1 ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 7 ) In this way , for a given sample sequence and initial value θ∗k−1 , the result of θ ∗ k depends on P . The optimization problem 5 on θ∗k is transformed into an optimization problem on P . However , the relationship between FRLLk ( θ ∗ k ) and P is too complicated to be used in optimization process . To tackle this problem , we propose an upper bound of FRLLk ( θ ∗ k ) as a practical optimization target . Theorem 2 ( upper bound ) . Denote σ̂m ( · ) as the symbol for maximum eigenvalue and ηm as the maximum single-step learning rate , the recursive least loss has an upper bound : FRLLk ( θ ∗ k ) ≤ 1 2 nkηmσ̂m ( PH̄ ) Lk ( θ ∗ k−1 ) ( 8 ) where H̄ = ∑k−1 j=1 Hj is defined as the sum of the Hessian matrices of all old tasks . We prove theorem 2 in Appendix A.3 . Discarding the constant terms , we get an alternative optimization problem for projection matrix P : P : { min P σ̂m ( PH̄ ) subject to trace ( P ) = dim ( P ) ( 9 ) We provide a detailed solution of this problem in Appendix A.4 . The normalized solution is : P = dim ( H̄ ) trace ( H̄−1 ) H̄−1 ( 10 ) The normalized projection procedure can be described as : finding a new gradient that has similar effects on the current task to minimize the upper bound of the old task losses . Our optimizer only modifies the direction of the gradient without reducing the search region , which will guarantee the fitting ability of the network consistent throughout the task sequence .

Summary: The paper proposes a continual learning approach based on recursive gradient optimization. To this end, a projection matrix is derived for the gradient modification. This matrix, P, is computed incrementally and updated by integrating the Hessian on each task locally.