Formalizing the Generalization-Forgetting Trade-off in Continual Learning

1 Introduction . In continual learning ( CL ) , we incrementally adapt a model to learn tasks ( defined according to the problem at hand ) observed sequentially . CL has two main objectives : maintain long-term memory ( remember previous tasks ) and navigate new experiences continually ( quickly adapt to new tasks ) . An important characterization of these objectives is provided by the stability-plasticity dilemma [ 9 ] , where the primary challenge is to balance network stability ( preserve past knowledge ; minimize catastrophic forgetting ) and plasticity ( rapidly learn from new experiences ; generalize quickly ) . This balance provides a natural objective for CL : balance forgetting and generalization . Traditional CL methods either minimize catastrophic forgetting or improve quick generalization but do not model both . For example , common solutions to the catastrophic forgetting issue include ( 1 ) representation-driven approaches [ 49 , 25 ] , ( 2 ) regularization approaches [ 27 , 2 , 34 , 16 , 48 , 48 , 24 , 37 , 10 , 43 ] , and ( 3 ) memory/experience replay [ 31 , 32 , 11 , 12 , 17 ] . Solutions to the generalization problem include representation-learning approaches ( matching nets [ 45 ] , prototypical networks [ 42 ] , and metalearning approaches [ 18 , 19 , 8 , 47 ] ) . More recently , several approaches [ 35 , 16 , 46 , 23 , 48 , 15 ] have been introduced that combine methods designed for quick generalization with frameworks designed to minimize forgetting . The aforementioned CL approaches naively minimize a loss function ( combination of forgetting and generalization loss ) but do not explicitly account for the trade-off in their optimization setup . The first work to formalize this trade-off was presented in meta-experience replay ( MER ) [ 38 ] , where the forgetting-generalization trade-off was posed as a gradient alignment problem . Although MER provides a promising methodology for CL , the balance between forgetting and generalization is enforced with several hyperparameters . Therefore , two key challenges arise : ( 1 ) lack of theoretical tools that study the existence ( under what conditions does a balance point between generalization and forgetting exists ? ) and stability ( can this balance be realistically achieved ? ) of a balance point 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and ( 2 ) lack of a systematic approach to achieve the balance point . We address these challenges in this paper . We describe a framework where we first formulate CL as a sequential decision-making problem and seek to minimize a cost function summed over the complete lifetime of the model . At any time k , given that the future tasks are not available , the calculation of the cost function becomes intractable . To circumvent this issue , we use Bellman ’ s principle of optimality [ 4 ] and recast the CL problem to model the catastrophic forgetting cost on the previous tasks and generalization cost on the new task . We show that equivalent performance on an infinite number of tasks is not practical ( Lemma 1 and Corollary 1 ) and that tasks must be prioritized . To achieve a balance between forgetting and generalization , we pose the trade-off as a saddle point problem where we designate one player for maximizing the generalization cost ( player 1 ) and another for minimizing the forgetting cost ( player 2 ) . We prove mathematically that there exists at least one saddle point between generalization and forgetting for each new task ( Theorem 1 ) . Furthermore , we show that this saddle point can be attained asymptotically ( Theorem 2 ) when player strategies are chosen as gradient ascent-descent . We then introduce balanced continual learning ( BCL ) , a new algorithm to achieve this saddle point . In our algorithm ( see Fig . 1 for a description of BCL ) , the generalization cost is computed by training and evaluating the model on given new task data . The catastrophic forgetting cost is computed by evaluating the model on the task memory ( previous tasks ) . We first maximize the generalization cost and then minimize the catastrophic forgetting cost to achieve the balance . We compare our approach with other methods such as elastic weight consolidation ( EWC ) [ 27 ] , online EWC [ 40 ] , and MER [ 38 ] on continual learning benchmark data sets [ 21 ] to show that BCL is better than or comparable to the state-of-the-art methods . Moreover , we also show in simulation that our theoretical framework is appropriate for understanding the continual learning problem . The contributions of this paper are ( 1 ) a theoretical framework to study the CL problem , ( 2 ) BCL , a method to attain balance between forgetting and generalization , and ( 3 ) advancement of the state of the art in CL . 2 Problem Formulation . We use R to denote the set of real numbers and N to denote the set of natural numbers . We use ‖.‖ to denote the Euclidean norm for vectors and the Frobenius norm for matrices , while using bold symbols to illustrate matrices and vectors . We define an interval [ 0 , K ) , K ∈ N and let p ( Q ) be the distribution over all the tasks observed in this interval . For any k ∈ [ 0 , K ) , we define a parametric model g ( . ) with yk = g ( xk ; θk ) , where θk is a vector comprising all parameters of the model with xk ∈ X k. Let n be the number of samples and m be the number of dimensions . Suppose a task at k|k ∈ [ 0 , K ) is observed and denoted as Qk : Qk ∼ p ( Q ) , where Qk = { X k , ` k } is a tuple with X k ∈ Rn×m being the input data and ` k quantifies the loss incurred by X k using the model g for the task at k. We denote a sequence of θk as uk : K = { θτ ∈ Ωθ , k ≤ τ ≤ K } , with Ωθ being the compact ( feasible ) set for the parameters . We denote the optimal value with a superscript ( ∗ ) ; for instance , we use θ ( ∗ ) k to denote the optimal value of θk at task k. In this paper we use balance point , equilibrium point , and saddle point to refer to the point of balance between generalization and forgetting . We interchange between these terms whenever convenient for the discussion . We will use ∇ ( j ) i to denote the gradient of i with respect to j and ∆i to denote the first difference in discrete time . An exemplary CL problem is described in Fig . 1 where we address a total of K = 3 tasks . To particularize the idea in Fig . 1 , we define the cost ( combination of catastrophic cost and generalization cost ) at any instant k as Jk ( θk ) = γk ` k + ∑k−1 τ=0 γτ ` τ , where ` τ is computed on task Qτ with γτ describing the contribution of Qτ to this sum . To solve the problem at k , we seek θk to minimize Jk ( θk ) . Similarly , to solve the problem in the complete interval [ 0 , K ] , we seek a θk to minimize Jk ( θk ) for each k ∈ [ 0 , K ] . In other words we seek to obtain θk for each task such that the cost Jk ( θk ) is minimized . Therefore , the optimization problem for the overall CL problem ( overarching goal of CL ) is provided as the minimization of the cumulative cost Vk ( uk : K ) = ∑K τ=k βτJτ ( θτ ) such that V ( ∗ ) k , is given as V ( ∗ ) k = minuk : KVk ( uk : K ) , ( 1 ) with 0 ≤ βτ ≤ 1 being the contribution of Jτ and uk : K being a weight sequence of length K − k. Within this formulation , two parameters determine the contributions of tasks : γτ , the contribution of each task in the past , and βτ , the contribution of tasks in the future . To successfully solve the optimization problem , Vk ( uk : K ) must be bounded and differentiable , typically ensured by the choice of γτ , βτ . Lemma 1 ( full statement and proof in Appendix A ) states that equivalent performance can not be guaranteed for an infinite number of tasks . Furthermore , Corollary 1 ( full statement and proof in Appendix A ) demonstrates that if the task contributions are prioritized , the differentiability and boundedness of Jτ ( θτ ) can be ensured . A similar result was proved in [ 28 ] , where a CL problem with infinite memory was shown to be NP-hard from a set theoretic perspective . These results ( both ours and in [ 28 ] ) demonstrate that a CL methodology can not provide perfect performance on a large number of tasks and that tasks must be prioritized . Despite these invaluable insights , the data corresponding to future tasks ( interval [ k , K ] ) is not available , and therefore Vk ( uk : K ) can not be evaluated . The optimization problem in Eq . ( 1 ) naively minimizes the cost ( due to both previous tasks and new tasks ) and does not provide any explicit modeling of the trade-off between forgetting and generalization . Furthermore , uk : K , the solution to Eq . ( 1 ) is a sequence of parameters , and it is not feasible to maintain uk : K for a large number of tasks . Because of these three issues , the problem is theoretically intractable in its current form . We will first recast the problem using tools from dynamic programming [ 29 ] , specifically Bellman ’ s principle of optimality , and derive a difference equation that summarizes the complete dynamics for the CL problem . Then , we will formulate a two-player differential game where we seek a saddle point solution to balance generalization and forgetting . 3 Dynamics of Continual Learning . Let V ( ∗ ) k = minuk : K ∑K τ=k βτJτ ( θτ ) ; the dynamics of CL ( the behavior of optimal cost with respect to k ) is provided as ∆V ( ∗ ) k = −minθk∈Ωθ [ βkJk ( θk ) + ( 〈∇θkV ( ∗ ) k , ∆θk〉+ 〈∇xkV ( ∗ ) k , ∆xk〉 ) ] . ( 2 ) The derivation is presented in Appendix A ( refer to Proposition 1 ) . Note that V ( ∗ ) k is the minima for the overarching CL problem in Eq . ( 2 ) and ∆V ( ∗ ) k represents the change in V ( ∗ ) k upon introduction of a new task ( we hitherto refer to this as perturbations ) . Zero perturbations ( ∆V ( ∗ ) k = 0 ) implies that the introduction of a new task does not impact our current solution ; that is , the optimal solution on all previous tasks is optimal on the new task as well . Therefore , the smaller the perturbations , the better the performance of a model on all tasks , thus providing our main objective : minimize the perturbations ( ∆V ( ∗ ) k ) . In Eq . 2 , ∆V ( ∗ ) k is quantified by three terms : the cost contribution from all the previous tasks and the new task Jk ( θk ) ; the change in the optimal cost due to the change in the parameters 〈∇θkV ( ∗ ) k , ∆θk〉 ; and the change in the optimal cost due to the change in the input ( introduction of new task ) 〈∇xkV ( ∗ ) k , ∆xk〉 . The first issue with the cumulative CL problem ( Eq . ( 1 ) ) can be attributed to the need for information from the future . In Eq . ( 2 ) , all information from the future is approximated by using the data from the new and the previous tasks . Therefore , the solution of the CL problem can directly be obtained by solving Eq . ( 2 ) using all the available data . Thus , minθk∈Ω [ H ( ∆xk , θk ) ] yields ∆V ( ∗ ) k ≈ 0 for β > 0 , with H ( ∆xk , θk ) = βkJk ( θk ) + 〈∇θkV ( ∗ ) k , ∆θk〉 + 〈∇xkV ( ∗ ) k , ∆xk〉 . Essentially , minimizing H ( ∆xk , θk ) would minimize the perturbations introduced by any new task k. In Eq . ( 2 ) , the first and the third term quantify generalization and the second term quantifies forgetting . A model exhibits generalization when it successfully adapts to a new task ( minimizes the first and the third term in Eq . ( 2 ) ) . The degree of generalization depends on the discrepancy between the previous tasks and the new task ( numerical value of the third term in Eq . ( 2 ) ) and the worst-case discrepancy prompts maximum generalization . Quantification of generalization is provided by ∆xk that summarizes the discrepancy between subsequent tasks . However , ∆xk = xk+1 − xk , and xk+1 is unknown at k. Therefore , we simulate worst-case discrepancy by iteratively updating ∆xk through gradient ascent in order to maximize H ( ∆xk , θk ) ; thus maximizing generalization . However , large discrepancy increases forgetting , and worst-case discrepancy yields maximum forgetting . Therefore , once maximum generalization is simulated , minimizing forgetting ( update θk by gradient descent ) under maximum generalization provides the balance . To formalize our idea , let us indicate the iteration index at k by i and write ∆xk as ∆x ( i ) k and θk as θ ( i ) k with H ( ∆xk , θk ) as H ( ∆x ( i ) k , θ ( i ) k ) ( for simplicity of notation , we will denote H ( ∆x ( i ) k , θ ( i ) k ) as H whenever convenient ) . Next , we write min θ ( i ) k ∈Ωθ [ H ( ∆x ( i ) k , θ ( i ) k ) ] = min θ ( i ) k ∈Ωθ [ βkJk ( θ ( i ) k ) + 〈∇θ ( i ) k V ( ∗ ) k , ∆θ ( i ) k 〉+ 〈∇x ( i ) k V ( ∗ ) k , ∆x ( i ) k 〉 ] ≤ min θ ( i ) k ∈Ωθ [ βkJk ( θ ( i ) k ) + 〈∇θ ( i ) k V ( ∗ ) k , ∆θ ( i ) k 〉+ max ∆x ( i ) k ∼p ( Q ) 〈∇ x ( i ) k V ( ∗ ) k , ∆x ( i ) k 〉 ] ≤ min θ ( i ) k ∈Ωθ max ∆x ( i ) k ∼p ( Q ) [ H ( ∆x ( i ) k , θ ( i ) k ) ] . ( 3 ) In Eq . ( 3 ) , we seek the solution pair ( ∆x ( ∗ ) k , θ ( ∗ ) k ) ∈ ( Ωθ , Ω∆x ( ∗ ) k ) , where ∆x ( ∗ ) k maximizes H ( maximizing player , player 1 ) while θ ( ∗ ) k minimizes H ( minimizing player , player 2 ) where ( Ωθ , Ω∆x ( ∗ ) k ) are the feasible sets for ∆x ( i ) k and θ ( i ) k respectively . The solution is attained , and ( ∆x ( ∗ ) k , θ ( ∗ ) k ) is said to be the equilibrium point when it satisfies the following condition : H ( ∆x ( ∗ ) k , θ ( i ) k ) ≥ H ( ∆x ( ∗ ) k , θ ( ∗ ) k ) ≥ H ( ∆x ( i ) k , θ ( ∗ ) k ) . ( 4 )

Summary: The paper theoretically investigates the trade-off between generalization and forgetting in continual learning (CL). The authors prove the existance of a saddle point in the generalization/forgetting space; they show that it is stable under certain conditions; and propose a practical algorithm called balanced continual learning (BCL) for finding it. Finally, the authors compare their approach experimentally against the state of the art in CL.