Continual Learning with Recursive Gradient Optimization

1 INTRODUCTION . In many application scenarios , one needs to learn a sequence of tasks without access to historical data , called continual learning . Although variants of stochastic gradient descent ( SGD ) have made a significant contribution to the progress made by neural networks in many fields , these optimizers require the mini-batches of data to satisfy the independent identically distributed ( i.i.d . ) assumption . In continual learning , the violation of this requirement leads to significant degradation of performance on previous tasks , called catastrophic forgetting . Recent works attempt to tackle this issue by modifying the training process from a variety of perspectives . Memory-based approaches use extra memory to store some samples ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2020 ) , gradients ( Chaudhry et al. , 2019a ; 2020 ; Saha et al. , 2021 ) , or their generative models ( Shin et al. , 2017 ; Shen et al. , 2020 ) to modify future training process . The memory for replay leads to a linear-increased space complexity with respect to the number of tasks . Expansionbased approaches select the network parameters dynamically ( Yoon et al. , 2018 ; Rosenbaum et al. , 2018 ; Serra et al. , 2018 ; Kaushik et al. , 2021 ) , add additional components as new tasks arrive ( Rusu et al. , 2016 ; Fernando et al. , 2017 ; Alet et al. , 2018 ; Chang et al. , 2019 ; Li et al. , 2019 ) , or use larger networks to generate network parameters ( Aljundi et al. , 2017 ; Yoon et al. , 2019 ; von Oswald et al. , 2019 ) . These methods reduce interference between tasks by additional task-specific parameters . Single-Task Learning ( STL ) can also be regarded as an expansion-based method which trains a network for each task separately . Regularization-based approaches encourage important parameters to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Yin et al. , 2020 ) or constraining the direction of parameter update ( Farajtabar et al. , 2019 ; Chaudhry et al. , 2019a ; Saha et al. , 2021 ) . Our method is also regularization-based which combines the advantages of loss penalty and gradient constraint . In this work , we focus on continual learning in a fixed-capacity network without data replay . We aim to minimize the expected increment of the total loss of past tasks without reducing the performance of the current task . To this end , we propose an upper bound of quadratic loss estimation and design a recursive optimization procedure to modify the direction of the gradient to the optimal solu- tion under this upper bound . In addition , we introduce trace normalization process to guarantee the learning rate during the training process according to the principle of current-task-first ( CFT ) . This normalization process makes our approach compatible with the vast majority of existing models and learning strategies well-designed for single-task solutions . As the gradient modification process is independent of data samples and previous parameters , our optimizer can be used directly in most deep architecture networks as typical single-task optimizers like SGD . Further , to reduce the interference between tasks , we develop a feature encoding strategy to represent the multi-modal structure of the network without additional parameters . A virtual feature encoding layer ( FEL ) which randomly permutes the output feature maps using integer task descriptor as seed is attached after each real layer . Thus , each task obtains a specific virtual structure under same network parameters . Since the parameter space of the network is not changed , such a strategy will not change the fitting ability of the neural network . Experimental validations on several continual learning benchmarks show that the proposed method has significantly less forgetting and higher accuracy than existing fixedcapacity baselines . We achieve state-of-the-art performance on 20-split-CIFAR100 ( 82.22 % ) and 20-split-miniImageNet ( 72.62 % ) . In addition to minimizing forgetting , this method has comparable or better performance than single-task learning which handles all tasks individually . 2 PRELIMINARIES . Consider K sequentially arrived supervised learning tasks { Tk|k ∈ [ K ] } , where [ N ] : = { 1 , 2 , · · · , N } for any positive integer N . In each task Tk , there are nk data points { ( xk , i , yk , i ) |i ∈ [ nk ] } sampled from an unknown distribution Dk . Let X , Y , W be the space of inputs , targets and model parameters . By denoting the predictor as f ( x , k ) : X × [ K ] → Y , the loss function of θ ∈ W associated with data point ( x , y ) and task identifier k can be expressed as l ( f ( θ ; x , k ) , y ) : W → R , and the empirical loss function of task Tk is defined as : Lk ( θ ) = 1 nk nk∑ i=1 l ( f ( θ ; k , xk , i ) , yk , i ) ( 1 ) In the continual learning scenario studied in this work , the parameter space W remains a fixed size , and an integer task descriptor is provided at both training and testing time . Without access to past samples , we use a second-order Taylor expansion to estimate the loss function of previous tasks.Let θ∗j be the optimal parameter of Lj ( θ ) generated by the gradient descent process according to ∇θ∗jLj = 0 . For a new model parameter θ in the neighborhood of θ ∗ j , the loss of previous task Tj ( j < k ) can be estimated as : Lj ( θ ) = Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ( 2 ) where Hj : = ∇2Lj ( θ∗j ) is the Hessian matrix . 3 PROBLEM FORMULATION & SOLUTION . In our fixed-capacity continual learning setting , finding an appropriate joint solution that works well across the task sequence is the core goal . To this end , we introduce a novel continual learning optimization problem and corresponding iterative optimization strategy . 3.1 OPTIMIZATION PROBLEM . In this paper , we formalize forgetting as the the increment of old task losses . As mentioned in Section 2 , the total loss of tasks before Tk , denoted as Fk , can be estimated by : Fk ( θ ) = k−1∑ j=1 Lj ( θ ) ≈ k−1∑ j=1 [ Lj ( θ ∗ j ) + 1 2 ( θ − θ∗j ) THj ( θ − θ∗j ) ] ( 3 ) As Equation ( 3 ) need the explicit value of previous model parameters , Fk is too expensive to be an optimization target in continual learning . We turn to a more concise form which we refer to as recursive least loss ( RLL ) : FRLLk ( θ ) : = 1 2 ( θ − θ∗k−1 ) T ( k−1∑ j=1 Hj ) ( θ − θ∗k−1 ) ( 4 ) In Appendix A.2 , we prove that FRLLk and Fk are equivalent for optimization if all previous tasks are fully-trained . Based on the conclusions above , the optimization problem during task Tk is formalized as : θ∗k : min θ FRLLk ( θ ) , subject to∇Lk ( θ ) = 0 ( 5 ) FRLLk has the same form as the regularization term in many regularization-based methods . The optimization goal of these methods are variants of Lk ( θ ) +λFRLLk ( θ ) , derived from Bayesian posterior approximation with Gaussian prior ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ) or approximation of the KL-divergence used in natural gradient descent ( Amari , 1998 ; Ritter et al. , 2018 ; Tseran et al. , 2018 ) . The Bayesian methods try to estimate and minimize the overall loss function , while our method prioritizes the performance of the current task by ∇Lk ( θ ) = 0 and minimizes the expected forgetting of the past tasks FRLLk ( θ ) . 3.2 GRADIENT MODIFICATION . For the newest task Tk , the optimal solution where ∇Lk ( θ ) = 0 should be obtained by stochastic gradient descent started from the former optimal model parameter θ∗k−1 at the end of task Tk−1 . Using subscript ·i to represent the parameters of step i , and the initial state θ0 = θ∗k−1 , the single step update can be expressed as : θi = θi−1 − ηi∇Lk ( θi−1 ) Assume that the pre-set learning rate ηi is small enough to ignore the higher order terms , the loss function after the one-step update can be expressed as : Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) T∇Lk ( θi−1 ) If we hope to solve the task Tk only , updating the parameter θ according to the gradient above is enough . However , as mentioned above , such a method will encourage the neural network to gradually forget the old tasks . Therefore , we modify the update direction to minimize the expectation of forgetting . To this end , we introduce a new positive definite symmetric matrix P with appropriate dimensions to modify the gradients ( g → Pg ) . The modified one-step update is : { θi = θi−1 − ηiP∇Lk ( θi−1 ) Lk ( θi ) = Lk ( θi−1 ) − ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 6 ) To maintain the pre-set learning rate during the continual learning problem and avoid repetitive selection of hyper-parameters , we impose an additional constraint on the trace of the projection matrix and prove the corresponding convergence rate consistency theorem 1 in Appendix A.1 . Theorem 1 ( convergence rate consistency ) . Under the constraint of trace ( P ) =dim ( P ) , the expectation of the learning rate for unknown isotropic distribution is the same as the original optimizer . For common network structures , we provide the space complexity of matrix P and the time complexity of the corresponding gradient modification process in Appendix B.2 . As described above , the only extra memory of our approach is the projection matrix P , which contains the information of previous tasks . This allows our approach to be a space-invariant method different with typical memory-based or expansion-based continual learning method . As the performance of our method is identified with the choice of P , the following problem is : How to find a good projection matrix ? We will answer this question in the following parts . 3.3 APPROXIMATE SOLUTION . The next step is to find a solution for problem ( 5 ) . When the training process on Tk is finished , the final state θ∗k and the residual loss can be obtained from the accumulation of one-step update : θ∗k = θ ∗ k−1 − nk∑ i=1 ηiP∇Lk ( θi−1 ) Lk ( θ ∗ k ) = Lk ( θ ∗ k−1 ) − nk∑ i=1 ηi ( ∇Lk ( θi−1 ) ) TP∇Lk ( θi−1 ) ( 7 ) In this way , for a given sample sequence and initial value θ∗k−1 , the result of θ ∗ k depends on P . The optimization problem 5 on θ∗k is transformed into an optimization problem on P . However , the relationship between FRLLk ( θ ∗ k ) and P is too complicated to be used in optimization process . To tackle this problem , we propose an upper bound of FRLLk ( θ ∗ k ) as a practical optimization target . Theorem 2 ( upper bound ) . Denote σ̂m ( · ) as the symbol for maximum eigenvalue and ηm as the maximum single-step learning rate , the recursive least loss has an upper bound : FRLLk ( θ ∗ k ) ≤ 1 2 nkηmσ̂m ( PH̄ ) Lk ( θ ∗ k−1 ) ( 8 ) where H̄ = ∑k−1 j=1 Hj is defined as the sum of the Hessian matrices of all old tasks . We prove theorem 2 in Appendix A.3 . Discarding the constant terms , we get an alternative optimization problem for projection matrix P : P : { min P σ̂m ( PH̄ ) subject to trace ( P ) = dim ( P ) ( 9 ) We provide a detailed solution of this problem in Appendix A.4 . The normalized solution is : P = dim ( H̄ ) trace ( H̄−1 ) H̄−1 ( 10 ) The normalized projection procedure can be described as : finding a new gradient that has similar effects on the current task to minimize the upper bound of the old task losses . Our optimizer only modifies the direction of the gradient without reducing the search region , which will guarantee the fitting ability of the network consistent throughout the task sequence .

Summary: This paper introduces a gradient-based approach to continual learning in neural networks called Recursive Gradient Optimization (RGO). RGO modifies the gradient direction at each update by multiplying it with a projection matrix P that is designed to minimize the increase in loss on previously encountered tasks. RGO is theoretically designed to prioritise performance on the current task and, among the optimal solutions for the current task, find the one that causes least interference with previous tasks - thus it assumes that the network is overparameterised. The derivation of the method starts by approximating the continual learning loss with the “recursive least loss”, which involves a sum of the Hessians of the previous tasks and assumes that tasks are fully trained and their solutions are close by to each other. It is then shown that the recursive least loss can be upper-bounded by an expression involving the projection matrix P, via which the solution for P that minimizes this upper bound is derived. It is shown that the expectation of the step size can be preserved by guaranteeing that trace(P)=dim(P) and by introducing random task-specific permutations at each layer - this preserves the “current-task-first” principle and reduces the need for hyperparameter tuning. Experiments are run on a number of standard continual learning image classification benchmarks, demonstrating an extremely strong performance of RGO in comparison to competing methods, sometimes surpassing the performance of a baseline that trains a separate model for each task.