Formalizing the Generalization-Forgetting Trade-off in Continual Learning

1 Introduction . In continual learning ( CL ) , we incrementally adapt a model to learn tasks ( defined according to the problem at hand ) observed sequentially . CL has two main objectives : maintain long-term memory ( remember previous tasks ) and navigate new experiences continually ( quickly adapt to new tasks ) . An important characterization of these objectives is provided by the stability-plasticity dilemma [ 9 ] , where the primary challenge is to balance network stability ( preserve past knowledge ; minimize catastrophic forgetting ) and plasticity ( rapidly learn from new experiences ; generalize quickly ) . This balance provides a natural objective for CL : balance forgetting and generalization . Traditional CL methods either minimize catastrophic forgetting or improve quick generalization but do not model both . For example , common solutions to the catastrophic forgetting issue include ( 1 ) representation-driven approaches [ 49 , 25 ] , ( 2 ) regularization approaches [ 27 , 2 , 34 , 16 , 48 , 48 , 24 , 37 , 10 , 43 ] , and ( 3 ) memory/experience replay [ 31 , 32 , 11 , 12 , 17 ] . Solutions to the generalization problem include representation-learning approaches ( matching nets [ 45 ] , prototypical networks [ 42 ] , and metalearning approaches [ 18 , 19 , 8 , 47 ] ) . More recently , several approaches [ 35 , 16 , 46 , 23 , 48 , 15 ] have been introduced that combine methods designed for quick generalization with frameworks designed to minimize forgetting . The aforementioned CL approaches naively minimize a loss function ( combination of forgetting and generalization loss ) but do not explicitly account for the trade-off in their optimization setup . The first work to formalize this trade-off was presented in meta-experience replay ( MER ) [ 38 ] , where the forgetting-generalization trade-off was posed as a gradient alignment problem . Although MER provides a promising methodology for CL , the balance between forgetting and generalization is enforced with several hyperparameters . Therefore , two key challenges arise : ( 1 ) lack of theoretical tools that study the existence ( under what conditions does a balance point between generalization and forgetting exists ? ) and stability ( can this balance be realistically achieved ? ) of a balance point 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and ( 2 ) lack of a systematic approach to achieve the balance point . We address these challenges in this paper . We describe a framework where we first formulate CL as a sequential decision-making problem and seek to minimize a cost function summed over the complete lifetime of the model . At any time k , given that the future tasks are not available , the calculation of the cost function becomes intractable . To circumvent this issue , we use Bellman ’ s principle of optimality [ 4 ] and recast the CL problem to model the catastrophic forgetting cost on the previous tasks and generalization cost on the new task . We show that equivalent performance on an infinite number of tasks is not practical ( Lemma 1 and Corollary 1 ) and that tasks must be prioritized . To achieve a balance between forgetting and generalization , we pose the trade-off as a saddle point problem where we designate one player for maximizing the generalization cost ( player 1 ) and another for minimizing the forgetting cost ( player 2 ) . We prove mathematically that there exists at least one saddle point between generalization and forgetting for each new task ( Theorem 1 ) . Furthermore , we show that this saddle point can be attained asymptotically ( Theorem 2 ) when player strategies are chosen as gradient ascent-descent . We then introduce balanced continual learning ( BCL ) , a new algorithm to achieve this saddle point . In our algorithm ( see Fig . 1 for a description of BCL ) , the generalization cost is computed by training and evaluating the model on given new task data . The catastrophic forgetting cost is computed by evaluating the model on the task memory ( previous tasks ) . We first maximize the generalization cost and then minimize the catastrophic forgetting cost to achieve the balance . We compare our approach with other methods such as elastic weight consolidation ( EWC ) [ 27 ] , online EWC [ 40 ] , and MER [ 38 ] on continual learning benchmark data sets [ 21 ] to show that BCL is better than or comparable to the state-of-the-art methods . Moreover , we also show in simulation that our theoretical framework is appropriate for understanding the continual learning problem . The contributions of this paper are ( 1 ) a theoretical framework to study the CL problem , ( 2 ) BCL , a method to attain balance between forgetting and generalization , and ( 3 ) advancement of the state of the art in CL . 2 Problem Formulation . We use R to denote the set of real numbers and N to denote the set of natural numbers . We use ‖.‖ to denote the Euclidean norm for vectors and the Frobenius norm for matrices , while using bold symbols to illustrate matrices and vectors . We define an interval [ 0 , K ) , K ∈ N and let p ( Q ) be the distribution over all the tasks observed in this interval . For any k ∈ [ 0 , K ) , we define a parametric model g ( . ) with yk = g ( xk ; θk ) , where θk is a vector comprising all parameters of the model with xk ∈ X k. Let n be the number of samples and m be the number of dimensions . Suppose a task at k|k ∈ [ 0 , K ) is observed and denoted as Qk : Qk ∼ p ( Q ) , where Qk = { X k , ` k } is a tuple with X k ∈ Rn×m being the input data and ` k quantifies the loss incurred by X k using the model g for the task at k. We denote a sequence of θk as uk : K = { θτ ∈ Ωθ , k ≤ τ ≤ K } , with Ωθ being the compact ( feasible ) set for the parameters . We denote the optimal value with a superscript ( ∗ ) ; for instance , we use θ ( ∗ ) k to denote the optimal value of θk at task k. In this paper we use balance point , equilibrium point , and saddle point to refer to the point of balance between generalization and forgetting . We interchange between these terms whenever convenient for the discussion . We will use ∇ ( j ) i to denote the gradient of i with respect to j and ∆i to denote the first difference in discrete time . An exemplary CL problem is described in Fig . 1 where we address a total of K = 3 tasks . To particularize the idea in Fig . 1 , we define the cost ( combination of catastrophic cost and generalization cost ) at any instant k as Jk ( θk ) = γk ` k + ∑k−1 τ=0 γτ ` τ , where ` τ is computed on task Qτ with γτ describing the contribution of Qτ to this sum . To solve the problem at k , we seek θk to minimize Jk ( θk ) . Similarly , to solve the problem in the complete interval [ 0 , K ] , we seek a θk to minimize Jk ( θk ) for each k ∈ [ 0 , K ] . In other words we seek to obtain θk for each task such that the cost Jk ( θk ) is minimized . Therefore , the optimization problem for the overall CL problem ( overarching goal of CL ) is provided as the minimization of the cumulative cost Vk ( uk : K ) = ∑K τ=k βτJτ ( θτ ) such that V ( ∗ ) k , is given as V ( ∗ ) k = minuk : KVk ( uk : K ) , ( 1 ) with 0 ≤ βτ ≤ 1 being the contribution of Jτ and uk : K being a weight sequence of length K − k. Within this formulation , two parameters determine the contributions of tasks : γτ , the contribution of each task in the past , and βτ , the contribution of tasks in the future . To successfully solve the optimization problem , Vk ( uk : K ) must be bounded and differentiable , typically ensured by the choice of γτ , βτ . Lemma 1 ( full statement and proof in Appendix A ) states that equivalent performance can not be guaranteed for an infinite number of tasks . Furthermore , Corollary 1 ( full statement and proof in Appendix A ) demonstrates that if the task contributions are prioritized , the differentiability and boundedness of Jτ ( θτ ) can be ensured . A similar result was proved in [ 28 ] , where a CL problem with infinite memory was shown to be NP-hard from a set theoretic perspective . These results ( both ours and in [ 28 ] ) demonstrate that a CL methodology can not provide perfect performance on a large number of tasks and that tasks must be prioritized . Despite these invaluable insights , the data corresponding to future tasks ( interval [ k , K ] ) is not available , and therefore Vk ( uk : K ) can not be evaluated . The optimization problem in Eq . ( 1 ) naively minimizes the cost ( due to both previous tasks and new tasks ) and does not provide any explicit modeling of the trade-off between forgetting and generalization . Furthermore , uk : K , the solution to Eq . ( 1 ) is a sequence of parameters , and it is not feasible to maintain uk : K for a large number of tasks . Because of these three issues , the problem is theoretically intractable in its current form . We will first recast the problem using tools from dynamic programming [ 29 ] , specifically Bellman ’ s principle of optimality , and derive a difference equation that summarizes the complete dynamics for the CL problem . Then , we will formulate a two-player differential game where we seek a saddle point solution to balance generalization and forgetting . 3 Dynamics of Continual Learning . Let V ( ∗ ) k = minuk : K ∑K τ=k βτJτ ( θτ ) ; the dynamics of CL ( the behavior of optimal cost with respect to k ) is provided as ∆V ( ∗ ) k = −minθk∈Ωθ [ βkJk ( θk ) + ( 〈∇θkV ( ∗ ) k , ∆θk〉+ 〈∇xkV ( ∗ ) k , ∆xk〉 ) ] . ( 2 ) The derivation is presented in Appendix A ( refer to Proposition 1 ) . Note that V ( ∗ ) k is the minima for the overarching CL problem in Eq . ( 2 ) and ∆V ( ∗ ) k represents the change in V ( ∗ ) k upon introduction of a new task ( we hitherto refer to this as perturbations ) . Zero perturbations ( ∆V ( ∗ ) k = 0 ) implies that the introduction of a new task does not impact our current solution ; that is , the optimal solution on all previous tasks is optimal on the new task as well . Therefore , the smaller the perturbations , the better the performance of a model on all tasks , thus providing our main objective : minimize the perturbations ( ∆V ( ∗ ) k ) . In Eq . 2 , ∆V ( ∗ ) k is quantified by three terms : the cost contribution from all the previous tasks and the new task Jk ( θk ) ; the change in the optimal cost due to the change in the parameters 〈∇θkV ( ∗ ) k , ∆θk〉 ; and the change in the optimal cost due to the change in the input ( introduction of new task ) 〈∇xkV ( ∗ ) k , ∆xk〉 . The first issue with the cumulative CL problem ( Eq . ( 1 ) ) can be attributed to the need for information from the future . In Eq . ( 2 ) , all information from the future is approximated by using the data from the new and the previous tasks . Therefore , the solution of the CL problem can directly be obtained by solving Eq . ( 2 ) using all the available data . Thus , minθk∈Ω [ H ( ∆xk , θk ) ] yields ∆V ( ∗ ) k ≈ 0 for β > 0 , with H ( ∆xk , θk ) = βkJk ( θk ) + 〈∇θkV ( ∗ ) k , ∆θk〉 + 〈∇xkV ( ∗ ) k , ∆xk〉 . Essentially , minimizing H ( ∆xk , θk ) would minimize the perturbations introduced by any new task k. In Eq . ( 2 ) , the first and the third term quantify generalization and the second term quantifies forgetting . A model exhibits generalization when it successfully adapts to a new task ( minimizes the first and the third term in Eq . ( 2 ) ) . The degree of generalization depends on the discrepancy between the previous tasks and the new task ( numerical value of the third term in Eq . ( 2 ) ) and the worst-case discrepancy prompts maximum generalization . Quantification of generalization is provided by ∆xk that summarizes the discrepancy between subsequent tasks . However , ∆xk = xk+1 − xk , and xk+1 is unknown at k. Therefore , we simulate worst-case discrepancy by iteratively updating ∆xk through gradient ascent in order to maximize H ( ∆xk , θk ) ; thus maximizing generalization . However , large discrepancy increases forgetting , and worst-case discrepancy yields maximum forgetting . Therefore , once maximum generalization is simulated , minimizing forgetting ( update θk by gradient descent ) under maximum generalization provides the balance . To formalize our idea , let us indicate the iteration index at k by i and write ∆xk as ∆x ( i ) k and θk as θ ( i ) k with H ( ∆xk , θk ) as H ( ∆x ( i ) k , θ ( i ) k ) ( for simplicity of notation , we will denote H ( ∆x ( i ) k , θ ( i ) k ) as H whenever convenient ) . Next , we write min θ ( i ) k ∈Ωθ [ H ( ∆x ( i ) k , θ ( i ) k ) ] = min θ ( i ) k ∈Ωθ [ βkJk ( θ ( i ) k ) + 〈∇θ ( i ) k V ( ∗ ) k , ∆θ ( i ) k 〉+ 〈∇x ( i ) k V ( ∗ ) k , ∆x ( i ) k 〉 ] ≤ min θ ( i ) k ∈Ωθ [ βkJk ( θ ( i ) k ) + 〈∇θ ( i ) k V ( ∗ ) k , ∆θ ( i ) k 〉+ max ∆x ( i ) k ∼p ( Q ) 〈∇ x ( i ) k V ( ∗ ) k , ∆x ( i ) k 〉 ] ≤ min θ ( i ) k ∈Ωθ max ∆x ( i ) k ∼p ( Q ) [ H ( ∆x ( i ) k , θ ( i ) k ) ] . ( 3 ) In Eq . ( 3 ) , we seek the solution pair ( ∆x ( ∗ ) k , θ ( ∗ ) k ) ∈ ( Ωθ , Ω∆x ( ∗ ) k ) , where ∆x ( ∗ ) k maximizes H ( maximizing player , player 1 ) while θ ( ∗ ) k minimizes H ( minimizing player , player 2 ) where ( Ωθ , Ω∆x ( ∗ ) k ) are the feasible sets for ∆x ( i ) k and θ ( i ) k respectively . The solution is attained , and ( ∆x ( ∗ ) k , θ ( ∗ ) k ) is said to be the equilibrium point when it satisfies the following condition : H ( ∆x ( ∗ ) k , θ ( i ) k ) ≥ H ( ∆x ( ∗ ) k , θ ( ∗ ) k ) ≥ H ( ∆x ( i ) k , θ ( ∗ ) k ) . ( 4 )

Summary: This paper considers the stability-plasticity dilemma in continual learning. The authors develop an algorithm which addresses catastrophic forgetting while trying to improve generalisation on new tasks.  The paper introduces a new cost function for continual learning, which takes into account the model's performance on previous, current and future tasks. An approximation, H, to this cost function, which involves only the data that has been encountered so far, is optimised. The optimisation is phrased as a 2-player game, where player 1 tries to maximise H by creating worst-case dissimilar inputs for future tasks. Player 2 minimises the H by optimising the model's parameters. The authors show that the two players would eventially reach a single solution (an equilibrium). Catastrophic forgetting is prevented by maintaining a subset of previously encountered datasets and using it when optimising H.  Contributions: - A novel formulation of the cost function of continual learning (CL) - A theoretically justified approach to CL. - An implementation of their approach - Experiments which show that the resulting method outperforms the many baselines.