Formalizing the Generalization-Forgetting Trade-Off in Continual Learning
Year: 2022

Abstract:
We formulate the continual learning problem via dynamic programming and model the trade-off between catastrophic forgetting and generalization as a two-player sequential game. In this approach, player 1 maximizes the cost due to lack of generalization whereas player 2 minimizes the cost due to increased catastrophic forgetting. We show theoretically and experimentally that a balance point between the two players exists for each task and that this point is stable (once the balance is achieved, the two players stay at the balance point). Next, we introduce balanced continual learning (BCL), which is designed to attain balance between generalization and forgetting, and we empirically demonstrate that BCL is comparable to or better than the state of the art.

heading: 1 Introduction
text: In continual learning (CL), we incrementally adapt a model to learn tasks (defined according to the problem at hand) observed sequentially. CL has two main objectives: maintain long-term memory (remember previous tasks) and navigate new experiences continually (quickly adapt to new tasks). An important characterization of these objectives is provided by the stability-plasticity dilemma [9], where the primary challenge is to balance network stability (preserve past knowledge; minimize catastrophic forgetting) and plasticity (rapidly learn from new experiences; generalize quickly). This balance provides a natural objective for CL: balance forgetting and generalization.
Traditional CL methods either minimize catastrophic forgetting or improve quick generalization but do not model both. For example, common solutions to the catastrophic forgetting issue include (1) representation-driven approaches [49, 25], (2) regularization approaches [27, 2, 34, 16, 48, 48, 24, 37, 10, 43], and (3) memory/experience replay [31, 32, 11, 12, 17]. Solutions to the generalization problem include representation-learning approaches (matching nets [45], prototypical networks [42], and metalearning approaches [18, 19, 8, 47]). More recently, several approaches [35, 16, 46, 23, 48, 15] have been introduced that combine methods designed for quick generalization with frameworks designed to minimize forgetting.
The aforementioned CL approaches naively minimize a loss function (combination of forgetting and generalization loss) but do not explicitly account for the trade-off in their optimization setup. The first work to formalize this trade-off was presented in meta-experience replay (MER) [38], where the forgetting-generalization trade-off was posed as a gradient alignment problem. Although MER provides a promising methodology for CL, the balance between forgetting and generalization is enforced with several hyperparameters. Therefore, two key challenges arise: (1) lack of theoretical tools that study the existence (under what conditions does a balance point between generalization and forgetting exists?) and stability (can this balance be realistically achieved?) of a balance point
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
and (2) lack of a systematic approach to achieve the balance point. We address these challenges in this paper.
We describe a framework where we first formulate CL as a sequential decision-making problem and seek to minimize a cost function summed over the complete lifetime of the model. At any time k, given that the future tasks are not available, the calculation of the cost function becomes intractable. To circumvent this issue, we use Bellman’s principle of optimality [4] and recast the CL problem to model the catastrophic forgetting cost on the previous tasks and generalization cost on the new task. We show that equivalent performance on an infinite number of tasks is not practical (Lemma 1 and Corollary 1) and that tasks must be prioritized. To achieve a balance between forgetting and
generalization, we pose the trade-off as a saddle point problem where we designate one player for maximizing the generalization cost (player 1) and another for minimizing the forgetting cost (player 2). We prove mathematically that there exists at least one saddle point between generalization and forgetting for each new task (Theorem 1). Furthermore, we show that this saddle point can be attained asymptotically (Theorem 2) when player strategies are chosen as gradient ascent-descent. We then introduce balanced continual learning (BCL), a new algorithm to achieve this saddle point. In our algorithm (see Fig. 1 for a description of BCL), the generalization cost is computed by training and evaluating the model on given new task data. The catastrophic forgetting cost is computed by evaluating the model on the task memory (previous tasks). We first maximize the generalization cost and then minimize the catastrophic forgetting cost to achieve the balance. We compare our approach with other methods such as elastic weight consolidation (EWC) [27], online EWC [40], and MER [38] on continual learning benchmark data sets [21] to show that BCL is better than or comparable to the state-of-the-art methods. Moreover, we also show in simulation that our theoretical framework is appropriate for understanding the continual learning problem. The contributions of this paper are (1) a theoretical framework to study the CL problem, (2) BCL, a method to attain balance between forgetting and generalization, and (3) advancement of the state of the art in CL.

heading: 2 Problem Formulation
text: We use R to denote the set of real numbers and N to denote the set of natural numbers. We use ‖.‖ to denote the Euclidean norm for vectors and the Frobenius norm for matrices, while using bold symbols to illustrate matrices and vectors. We define an interval [0,K),K ∈ N and let p(Q) be the distribution over all the tasks observed in this interval. For any k ∈ [0,K), we define a parametric model g(.) with yk = g(xk;θk), where θk is a vector comprising all parameters of the model with xk ∈ X k. Let n be the number of samples and m be the number of dimensions. Suppose a task at k|k ∈ [0,K) is observed and denoted as Qk : Qk ∼ p(Q), where Qk = {X k, `k} is a tuple with X k ∈ Rn×m being the input data and `k quantifies the loss incurred by X k using the model g for the task at k. We denote a sequence of θk as uk:K = {θτ ∈ Ωθ, k ≤ τ ≤ K}, with Ωθ being the compact (feasible) set for the parameters. We denote the optimal value with a superscript (∗); for instance, we use θ(∗)k to denote the optimal value of θk at task k. In this paper we use balance
point, equilibrium point, and saddle point to refer to the point of balance between generalization and forgetting. We interchange between these terms whenever convenient for the discussion. We will use ∇(j)i to denote the gradient of i with respect to j and ∆i to denote the first difference in discrete time.
An exemplary CL problem is described in Fig. 1 where we address a total of K = 3 tasks. To particularize the idea in Fig. 1, we define the cost (combination of catastrophic cost and generalization cost) at any instant k as Jk(θk) = γk`k + ∑k−1 τ=0 γτ `τ , where `τ is computed on task Qτ with γτ describing the contribution of Qτ to this sum. To solve the problem at k, we seek θk to minimize Jk(θk). Similarly, to solve the problem in the complete interval [0,K], we seek a θk to minimize Jk(θk) for each k ∈ [0,K]. In other words we seek to obtain θk for each task such that the cost Jk(θk) is minimized. Therefore, the optimization problem for the overall CL problem (overarching goal of CL) is provided as the minimization of the cumulative cost Vk(uk:K) = ∑K τ=k βτJτ (θτ ) such that V (∗) k , is given as
V (∗) k = minuk:KVk(uk:K), (1)
with 0 ≤ βτ ≤ 1 being the contribution of Jτ and uk:K being a weight sequence of length K − k. Within this formulation, two parameters determine the contributions of tasks: γτ , the contribution of each task in the past, and βτ , the contribution of tasks in the future. To successfully solve the optimization problem, Vk(uk:K) must be bounded and differentiable, typically ensured by the choice of γτ , βτ . Lemma 1 (full statement and proof in Appendix A) states that equivalent performance cannot be guaranteed for an infinite number of tasks. Furthermore, Corollary 1 (full statement and proof in Appendix A) demonstrates that if the task contributions are prioritized, the differentiability and boundedness of Jτ (θτ ) can be ensured. A similar result was proved in [28], where a CL problem with infinite memory was shown to be NP-hard from a set theoretic perspective. These results (both ours and in [28]) demonstrate that a CL methodology cannot provide perfect performance on a large number of tasks and that tasks must be prioritized.
Despite these invaluable insights, the data corresponding to future tasks (interval [k,K]) is not available, and therefore Vk(uk:K) cannot be evaluated. The optimization problem in Eq. (1) naively minimizes the cost (due to both previous tasks and new tasks) and does not provide any explicit modeling of the trade-off between forgetting and generalization. Furthermore, uk:K , the solution to Eq. (1) is a sequence of parameters, and it is not feasible to maintain uk:K for a large number of tasks. Because of these three issues, the problem is theoretically intractable in its current form.
We will first recast the problem using tools from dynamic programming [29], specifically Bellman’s principle of optimality, and derive a difference equation that summarizes the complete dynamics for the CL problem. Then, we will formulate a two-player differential game where we seek a saddle point solution to balance generalization and forgetting.

heading: 3 Dynamics of Continual Learning
text: Let V (∗)k = minuk:K ∑K τ=k βτJτ (θτ ); the dynamics of CL (the behavior of optimal cost with respect to k) is provided as
∆V (∗) k = −minθk∈Ωθ [ βkJk(θk) + ( 〈∇θkV (∗) k ,∆θk〉+ 〈∇xkV (∗) k ,∆xk〉 )] . (2)
The derivation is presented in Appendix A (refer to Proposition 1). Note that V (∗)k is the minima for the overarching CL problem in Eq. (2)and ∆V (∗)k represents the change in V (∗) k upon introduction of a new task (we hitherto refer to this as perturbations). Zero perturbations (∆V (∗)k = 0) implies that the introduction of a new task does not impact our current solution; that is, the optimal solution on all previous tasks is optimal on the new task as well. Therefore, the smaller the perturbations, the better the performance of a model on all tasks, thus providing our main objective: minimize the perturbations (∆V (∗)k ). In Eq. 2, ∆V (∗) k is quantified by three terms: the cost contribution from all the previous tasks and the new task Jk(θk); the change in the optimal cost due to the change in the parameters 〈∇θkV (∗) k ,∆θk〉; and the change in the optimal cost due to the change in the input (introduction of new task) 〈∇xkV (∗) k ,∆xk〉.
The first issue with the cumulative CL problem (Eq. (1)) can be attributed to the need for information from the future. In Eq. (2), all information from the future is approximated by using the data from the new and the previous tasks. Therefore, the solution of the CL problem can directly be obtained by solving Eq. (2) using all the available data. Thus, minθk∈Ω [ H(∆xk,θk) ] yields ∆V (∗)k ≈ 0 for β > 0, with H(∆xk,θk) = βkJk(θk) + 〈∇θkV (∗) k ,∆θk〉 + 〈∇xkV (∗) k ,∆xk〉. Essentially, minimizing H(∆xk,θk) would minimize the perturbations introduced by any new task k.
In Eq. (2), the first and the third term quantify generalization and the second term quantifies forgetting. A model exhibits generalization when it successfully adapts to a new task (minimizes the first and the third term in Eq. (2)). The degree of generalization depends on the discrepancy between the previous tasks and the new task (numerical value of the third term in Eq. (2)) and the worst-case discrepancy prompts maximum generalization. Quantification of generalization is provided by ∆xk that summarizes the discrepancy between subsequent tasks. However, ∆xk = xk+1 − xk, and xk+1 is unknown at k. Therefore, we simulate worst-case discrepancy by iteratively updating ∆xk through gradient ascent in order to maximize H(∆xk,θk); thus maximizing generalization. However, large discrepancy increases forgetting, and worst-case discrepancy yields maximum forgetting. Therefore, once maximum generalization is simulated, minimizing forgetting (update θk by gradient descent) under maximum generalization provides the balance.
To formalize our idea, let us indicate the iteration index at k by i and write ∆xk as ∆x (i) k and θk as θ (i) k with H(∆xk,θk) as H(∆x (i) k ,θ (i) k ) (for simplicity of notation, we will denote H(∆x (i) k ,θ (i) k ) as H whenever convenient). Next, we write
min θ (i) k ∈Ωθ
[ H(∆x
(i) k ,θ (i) k ) ] = min θ (i) k ∈Ωθ [ βkJk(θ (i) k ) + 〈∇θ(i)k V (∗) k ,∆θ (i) k 〉+ 〈∇x(i)k V (∗) k ,∆x (i) k 〉 ]
≤ min θ (i) k ∈Ωθ
[ βkJk(θ
(i) k ) + 〈∇θ(i)k V (∗) k ,∆θ (i) k 〉+ max
∆x (i) k ∼p(Q)
〈∇ x
(i) k
V (∗) k ,∆x (i) k 〉 ]
≤ min θ (i) k ∈Ωθ max ∆x (i) k ∼p(Q)
[ H(∆x
(i) k ,θ (i) k ) ] .
(3) In Eq. (3), we seek the solution pair (∆x(∗)k ,θ
(∗) k ) ∈ (Ωθ,Ω∆x(∗)k ), where ∆x (∗) k maximizes
H (maximizing player, player 1) while θ(∗)k minimizes H (minimizing player, player 2) where (Ωθ,Ω∆x(∗)k ) are the feasible sets for ∆x(i)k and θ (i) k respectively. The solution is attained, and (∆x (∗) k ,θ (∗) k ) is said to be the equilibrium point when it satisfies the following condition:
H(∆x (∗) k ,θ (i) k ) ≥ H(∆x (∗) k ,θ (∗) k ) ≥ H(∆x (i) k ,θ (∗) k ). (4)

heading: 3.1 Theoretical Analysis
text: Figure 2: Illustration of the proofs. ∆x (player 1) is the horizontal axis, and the vertical axis indicates θ (player 2) where the curve indicates H. If we start from the red circle for player 1 (player 2 is fixed at the blue circle), H is increasing (goes from a grey circle to a red asterisk) with player 1 reaching the red asterisk. Next, start from the blue circle (θ is at the red asterisk), the cost decreases.
With our formulation, two key questions arise: Does our problem setup have an equilibrium point satisfying Eq. (4)? and how can one attain this equilibrium point? We answer these questions with Theorems 1 and 2, respectively. Full statements and proofs are provided in Appendix A.
To illustrate the theory, we refer to Fig. 2, where the initial values for the two players are characterized by the pair {θ(i)k (blue circle),∆x (i) k (red circle)} and the cost value at {θ(i)k ,∆x (i) k } is indicated by H(∆x(i)k ,θ (i) k ) (the grey circle on the cost curve (the dark blue curve)). Our proofing strategy is as follows. First, we fix θ(.)k ∈ Ωθ and construct a neighborhood
Mk = {Ωx,θ(.)k }. Within this neighborhood we prove in Lemmas 2 and 4 that if we search for ∆x(i)k through gradient ascent, we can converge to a local maximizer, and H is maximizing with respect to ∆x(i)k . Second, we let ∆x (.) k ∈ Ωx be fixed, and we search for θ(i)k through gradient descent. Under this condition, we demonstrate two ideas in Lemmas 3 and 5: (1) we show that H is minimizing in the neighborhood Nk : Nk = {Ωθ,∆x(.)k }; and (2) we converge to the local minimizer in the neighborhood Nk. Third, in the union of the two neighborhoodsMk ∪Nk, (proven to be nonempty according to Lemma 6), we show that there exists at least one local equilibrium point (Theorem 1); that is, there is at least one balance point.
Theorem 1 (Existence of an Equilibrium Point). For any k ∈ [0,K], let θ(∗)k ∈ Ωθ, be the minimizer of H according to Lemma 5 and define M(∗)k = {Ωx,θ (∗) k }. Similarly, let ∆x (∗) k ∈ Ωx, be the maximizer of H according to Lemma 4 and define N (∗)k = {∆x (∗) k ,Ωθ}. Further, letM (∗) k ∪N (∗) k be nonempty according to Lemma. 6, then (∆x(∗)k ,θ (∗) k ) ∈M (∗) k ∪N (∗) k is a local equilibrium point.
We next show that this equilibrium point is stable (Theorem 2) under a sequential play. Specifically, we show that when player 1 plays first and player 2 plays second, we asymptotically reach a saddle point pair (∆x(∗)k ,θ (∗) k ) for H. At this saddle point, both players have no incentive to move, and the game converges.
Theorem 2 (Stability of the Equilibrium Point). For any k ∈ [0,K], ∆x(i)k ∈ Ωx and θ (i) k ∈ Ωθ be the initial values for ∆x (i) k and θ (i) k respectively. Define Mk = {Ωx,Ωθ} with H(∆x(i)k ,θ (i) k ) given by Proposition 2. Let ∆x (i+1) k − ∆x (i) k = α (i) k × (∇∆x(i)k H(∆x (i) k ,θ (.) k ))/‖∇∆x(i)k H(∆x (i) k ,θ (.) k )‖2) and θ (i+1) k − θ (i) k = −α (i) k × ∇ θ (i) k H(∆x (.) k ,θ (i) k ). Let the existence of an equilibrium point be given by Theorem 1, then, as a consequence of Lemmas 2 and 3, (∆x(∗)k ,θ (∗) k ) ∈Mk is a stable equilibrium point for H given .
In this game, the interplay between these two opposing players (representative of generalization and forgetting, respectively) introduces the dynamics required to play the game. Furthermore, the results presented in this section are local to the task. In other words, we prove that we can achieve a balance between generalization and forgetting for each task k (neighborhoods are task dependent, and we achieve a local solution given a task k). Furthermore, our game is sequential; that is, there is a leader (player 1) and a follower (player 2). The leader (∆x(i)k ) plays first, and the follower (θ (i) k ) plays second with complete knowledge of the leader’s play. The game is directed by ∆x(i)k , and any changes in the task (reflected in ∆x(i)k ) will shift the input and thus the equilibrium point. Consequently, the equilibrium point varies with respect to a task, and one will need to attain a new equilibrium point for each shift in a task. Without complete knowledge of the tasks (not available in a CL scenario), only a local result is possible. This highlights one of the key limitations of this work. Ideally, we would like a balance between forgetting and generalization that is independent of tasks. However, this would require learning a trajectory of the equilibrium point (How does the equilibrium point change with the change in the tasks?) and is beyond the scope of this paper. One work that attempts to do this is [39], where the authors learn a parameter per task. For a large number of tasks, however, such an approach is computationally prohibitive.
These results are valid only under certain assumptions: (1) the Frobenius norm of the gradient is bounded, always positive; (2) the cost function is bounded and differentiable; and (3) the learning rate goes to zero as i tends to infinity. The first assumption is reasonable in practice, and gradient clipping or perturbation strategies can be used to ensure it. The boundedness of the cost (second assumption) can be ensured by prioritizing the contributions of the task (Lemma 1 and Corollary 1). The third assumption assumes a decaying learning rate. Learning rate decay is a common strategy and is employed widely. Therefore, all assumptions are practical and reasonable.

heading: 3.2 Balanced Continual Learning
text: Equipped with the theory, we develop a new CL method to achieve a balance between forgetting and generalization. By Proposition 2, the cost function can be upper bounded as H(∆x(i)k ,θ (i) k ) ≤
βkJk(θ (i) k ) + (Jk(θ (i+ζ) k )− Jk(θ (i) k )) + (Jk+ζ(θ (i) k )− Jk(θ (i) k )), where Jk+ζ indicates ζ updates on player 1 and θ(i+ζ)k indicates ζ updates on player 2.
α (i) k ∇∆xkE[H(∆x (i) k ,θ (i) k )]
‖∇∆xkH(∆x (i) k ,θ (i) k )‖2︸ ︷︷ ︸
Player 1
,
−α(i)k ×∇θkE[H(∆x (∗) k ,θ (i) k ))]︸ ︷︷ ︸
Player 2
,
(5) The strategies for the two players ∆xk,θk are chosen in Eq. (5) with E being the expected value operator. We can approximate the required terms in our update rule (player strategies) using data samples (batches). Note that the approximation is performed largely through one-sided finite difference, which may introduce an error and is another potential drawback. The pseudo code of the BCL is shown in Algorithm 1. We define a new task array DN (k) and a task memory array DP (k) ⊂ ∪k−1τ=0Qτ (samples from all previous tasks). For each batch bN ∈ DN (k), we sample bP from DP (k), combine to create bPN (k) = bP (k) ∪ bN (k), and perform a sequential play. Specifically, for each task the first player initializes xPNk = bPN (k) and performs ζ updates on xPNk through gradient ascent. The second player, with complete knowledge of the first player’s strategy, chooses the best play to reduce H(∆x(i)k ,θ (i) k ). To estimate player 2’s play, we must estimate different terms in H(∆x(i)k ,θ (i) k ). This procedure involves three steps. First, we use the first player’s play and approximate (Jk+ζ(θ (i) k ) − Jk(θ (i) k )). Second, to approximate (Jk(θ (i+ζ) k ) − Jk(θ (i) k )) : (a), we copy θ̂ into θ̂B (a temporary network) and perform ζ updates on θ̂B; and (b) we compute Jk(θ (i+ζ) k ) using θ̂B(k + ζ) and evaluate (Jk(θ (i+ζ) k )− Jk(θ (i) k )). Third, equipped with these approximations, we compute H(∆x(i)k ,θ (i) k ) and obtain the play for the second player. Both these players perform the steps repetitively for each piece of information (batch of data). Once all the data from the new task is exhausted, we move to the next task.
3.3 Related Work Algorithm 1: BCL Initialize θ, DP , DN while k = 1, 2, 3, ...K do
j = 0 while j < ρ do
Get bN ∈ DNk Get bP ∈ DPk Get bPN = bP ∪ bN Copy bPN into xPNk i = 0 while i+ 1 <= ζ do
Update xPNk with Jk(θk) using gradient ascent
i = i+1 Calculate Jk+ζ(θ (i) k )− Jk(θ (i) k ) Copy θ(i)k into θ B k i = 0 while i+ 1 <= ζ do Update θBk with Jk(θ B k )
i = i+1 Calculate (Jk(θBk )− Jk(θ (i) k )) Calculate H(∆x(i)k ,θ (i) k ) Update θ(i)k using gradient descent
j= j+1 Update DP with DN
Traditional solutions to the CL focus on either the forgetting issue [39, 49, 47, 7, 27, 50, 2, 31, 32, 11] or the generalization issue [45, 42, 18, 19, 8]. Common solutions to the forgetting problem involve dynamic architectures and flexible knowledge representation such as [39, 49, 47, 7], regularization approaches including [27, 50, 2] and memory/experience replay [31, 32, 11]. Similarly, quick generalization to a new task has been addressed through few-shot and one-shot learning approaches such as matching nets [45] and prototypical network [42]. More recently, the field of metalearning has approached the generalization problem by designing a metalearner that can perform quick generalization from very little data [18, 19, 8].
In the past few years, metalearners for quick generalization have been combined with methodologies specifically designed for reduced forgetting [22, 3]. For instance, the approaches in [22, 3] adapt the modelagnostic metalearning (MAML) framework in [18] with robust representation to minimize forgetting and provide impressive results on CL. However, both these approaches require a pretraining phase for learning representation. Simultaneously, Gupta et al. [20] introduced LA-MAML—a metalearning approach where the impact of learning rates on the CL problem is reduced through the use of per-parameter learning rates. LA-MAML [20] also introduced episodic memory to address the forgetting issue. Other approaches also have attempted to model both generalization and forgetting. In [16], the gradients from new tasks are projected onto a subspace that is orthogonal to the older tasks, and forgetting is minimized. Similarly, Joseph and Balasubramanian [23] utilized a Bayesian framework to consolidate learning across previous and current tasks, and Yin et al. [48] provided a framework for approx-
imating loss function to summarize the forgetting in the CL setting. Furthermore, Abolfathi et al. [1] focused on sampling episodes in the reinforcement learning setting, and Elrahimi et al. [15] introduced a generative adversarial network-type structure to progressively learn shared features assisting reduced forgetting and improved generalization. Despite significant progress, however, these methods [22, 3, 20, 16, 23, 48, 1, 15] are still inherently tilted toward maximizing generalization or minimizing forgetting because they naively minimize the loss function. Therefore, the contribution of different terms in the loss function becomes important. For instance, if the generalization cost is given more weight, a method would generalize better. Similarly, if forgetting cost is given more weight, a method would forget less. Therefore, the resolution of the trade-off inherently depends on an hyperparameter.
The first work to formalize the trade-off in CL was MER, where the trade-off was formalized as a gradient alignment problem. Similar to MER, Doan et al. ([13]) studied forgetting as an alignment problem. In MER, the angle between the gradients was approximated by using Reptile [36], which promotes gradient alignment by reducing weight changes. On the other hand, [13] formalized the alignment as an eigenvalue problem and introduced a PCA-driven method to ensure alignment. Our approach models this balance as a saddle point problem achieved through stochastic gradient such that the saddle point (balance point or equilibrium point) resolves the trade-off.
Our approach is the first in the CL literature to prove the existence of the saddle point (the balance point) between generalization and forgetting given a task in a CL problem. Furthermore, we are the first to theoretically demonstrate that the saddle point can be achieved reasonably under a gradient ascent-descent game. The work closest to ours is [15], where an adversarial framework is described to minimize forgetting in CL by generating task-invariant representation. However, [15] is not model agnostic (the architecture of the network is important) and requires a considerable amount of data at the start of the learning procedure. Because of these issues, [15] is not suitable for learning in the sequential scenario.
Several attempts have been reported in the literature to theoretically analyze different aspects of CL. For instance, Benzing [6] attempted to unify regularization methods such as [27, 50, 2, 31]. On the other hand, Benamin et al. [5] provided generalization (in the context of training to test data generalization) guarantees with orthogonal gradient descent, and Yin et al. [48] provided generalization (in the context of training to test data generalization) analysis while regularizing with a second-order approximation of the loss functions. Although these works provide important results, they focus on quantifying catastrophic forgetting or generalization but do attempt to model the trade-off in any manner.

heading: 4 Experiments
text: We use the CL benchmark [21] for our experiments and retain the experimental settings (hyperparameters) from [21, 44]. For comparison, we use the split-MNIST, permuted-MNIST, and split-CiFAR100 data sets while considering three scenarios: incremental domain learning (IDL), incremental task learning (ITL), and incremental class learning (ICL). The splitting and permutation strategies when applied to the MNIST or CiFAR100 data set can generate task sequences for all three scenarios (illustrated in Figure 1 and Appendix: Figure 2 of [21]). For comparing our approach, we use three baseline strategies—standard neural network with Adam [26], Adagrad [14], and SG—and use L2-regularization and naive rehearsal (which is similar to experience replay). For CL approaches, we use EWC [27], online EWC [40], SI [50], LwF [30], DGR [41], RtF [44], MAS [2], MER [38], and GEM [33]. We utilize data preprocessing as provided by [21]. Additional details on experiments can be found in Appendix B and [21, 44]. All experiments are conducted in Python 3.4 using the pytorch 1.7.1 library with the NVIDIA-A100 GPU for our simulations.
Comparison with the state of the art: The results for our method are summarized in Table 1 and 2. The efficiency for any method is calculated by observing the average accuracy (retained accuracy (RA) [38]) at the end of each repetition and then evaluating the mean and standard deviation of RA across different repetitions. For each method, we report the mean and standard deviation of RA over five repetitions of each experiment. In each column, we indicate the best-performing method in bold. For the split-MNIST data set, we obtain 99.52± 0.07 for ITL, 98.71± 0.06 for IDL, and 97.32 ± 0.17 for ICL. Similarly, with the permuted-MNIST data set, we obtain 97.41 ± 0.01 for ITL, 97.51± 0.05 for IDL, and 97.61± 0.01 for ICL. Furthermore, with the split-CiFAR100 data
set, we obtain 81.82 ± 0.17 for ITL, 62.11 ± 0.00 for IDL, and 69.27 ± 0.03 for ICL. BCL is the best-performing methodology for all cases (across both data sets) except RtF for ITL (0.14% drop) with the split-MNIST data set.
Generally, ITL is the easiest learning scenario [21], and all methods therefore perform well on ITL (the performance is close). For the ITL scenario with the split-MNIST data set, BCL is better than most methods; but several methods, such as naive rehearsal, naive rehearsal-C, RtF, and DGR, attain close RA values (less than 1% from BCL). Note that both DGR and RtF involve a generative model pretrained with data from all the tasks. In a sequential learning scenario, one cannot efficiently train generative models because data corresponding to all the tasks is not
available beforehand. Although RtF provides improved performance for split-MNIST (ITL), the improvement is less than 1% (not significant). In fact, RtF performance is poorer for BCL in ICL by 4.76% (a significant drop in performance) and in IDL by 1.4%.
Two additional observations can be made about the split-MNIST data set. First, Adagrad, SGD, and L2 achieve better performance than does Adam. Therefore, in our analysis Adagrad appears more appropriate although Adam is popularly used for this task. Second, naive rehearsal (both naive rehearsal and naive rehearsal-C approaches) achieves performance equivalent to the state-of-the-art methods with similar memory overhead. Furthermore, naive rehearsal performs much better than online EWC and SI, especially in the ICL scenario. These limitations indicate that regularizationdriven approaches are not much better than baseline models and in fact perform poorer than methods involving memory (naive rehearsals, MER, BCL, etc.). In [28], it was shown theoretically that memory-based approaches typically do better than regularization-driven approaches, as is empirically observed in this paper, too. Another interesting observation is that EWC and online EWC require significant hyperparameter tuning, which would be difficult to do in real-world scenarios. Other regularization-based methods, such as SI and MAS, also suffer from the same issue.
The observations from the split-MNIST carry forward to the permuted-MNIST data set. Moreover, RA values for the permutation MNIST data set are better for the split-MNIST data set across the board, indicating that the permutation MNIST data set presents an easier learning problem. Similar to the observations made with the split-MNIST data set, BCL is better than all methods for the permuted-MNIST dataset, with naive rehearsal and RtF providing RA values that are close (less than 1%). The only methodology in the literature that attempts to model the balance between forgetting and generalization is MER, an extension of GEM. From our results, we observe that BCL is better than
MER in all cases (Split-MNIST–2.4% improvement for ITL, 6.55% improvement for IDL, 4.12% improvement for ICL and Permuted-MNIST–0.26% improvement for ITL, 1.4% improvement for IDL and 5.9% improvement for ICL).
The substantial improvements obtained by BCL are more evident in Table 2 where the results on the split-CiFAR100 data set are summarized. BCL is clearly the best-performing method. The next best-performing method is naive rehearsal-C, where BCL improves performance by 3.41% for ITL, 10.3% for IDL, and 17.99% for ICL. Other observations about the regularization methods and the rest of the baseline methods carry forward from Table 1. However, one key difference is that while AdaGrad is observed to be better than Adam for the MNIST data set, Adam is comparable to SGD and Adagrad for split-CiFAR100. In summary, BCL is comparable to or better than the state of the art in the literature for both the MNIST and CiFAR100 data sets.
0 200 400 600 800 1000 1200 1400 1600 i (update iteration at each task k)
10
5
0
5
10
15
20 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6
〈 ∇θkV (∗) k ,∆θk 〉 Jk(θk) + 〈 ∇xkV (∗) k ,∆xk 〉 ∇V ∗k
(a)
0 50 100 150 200 250 300 i (update iteration at task k= 3)
5
0
5
10
15
(b)
data set within the ICL setting, are illustrated in Fig. 3a. These tasks are introduced every 300 update steps. From Fig. 3a we make two important observations. First, as soon as tasks 1, 3, 4, and 6 are introduced, all three curves (red, blue, and green) indicate a bump. Second, when tasks 2 and 5 are introduced, there is no change. On a closer look at task 3 in Fig. 3b, we observe that when the task 3 is introduced, the blue curve exhibits a large positive bump (the introduction of the new tasks increases the first and the third terms in Eq. (2)). The increase implies that task 3 forced the model to generalize and increased forgetting on tasks 1 and 2 (observed by the increase in the green curve). To compensate for this increase, we require the model θk to behave adversarially and introduce a large enough negative value in 〈∇θkV (∗) k ,∆θk〉 (red curve) to cancel out the increase in the blue curve. In Fig. 3b the red curve demonstrates a large negative value (expected behavior) and eventually (as i increases) forces the blue curve (by consequence, green: the sum of red and blue) to move toward zero (the model compensates for the increase in forgetting). As observed, the blue and the red curves behave opposite to each other and introduce a push-pull behavior that stops only when the two cancel each other and the sum (green) is zero. Once the sum has reached zero, there is no incentive for the
red and green to be nonzero, and therefore they remain at zero; thus, all three curves (green, red, and blue) remain at zero once converged until a task 4 is introduced (when there is another bump, as seen in Fig. 3a). However, this increase in the blue curve is not observed when tasks 2 and 5 are introduced. When new tasks are similar to the older tasks, it is expected that ∆V (∗)k = 0, as is observed in Fig. 3a.
All of these observations are fully explained by Eq. 2, which illustrates that the solution to the CL problem is obtained optimally only when ∆V (∗)k = 0 (observed in Figs. 3a and 3b). The term ∆V
(∗) k is quantified by Jk(θk), 〈∇θkV (∗) k ,∆θk〉 and 〈∇xkV (∗) k ,∆xk〉. Our theory suggests that there exists an inherent trade-off between different terms in Eq. 2. Therefore every time a new task is observed, it is expected that Jk(θk) + 〈∇xkV (∗) k ,∆xk〉 increases (increase in blue curve when tasks 1, 3, 4, and 6 are introduced) and 〈∇θkV (∗) k ,∆θk〉 compensates to cancel this increase (red curve exhibits a negative jump). In Theorems 1 and 2 we demonstrate the existence of this balance point (for each task, as i increases, ∆V (∗)k tends to zero, as observed in Fig. 3b) and ∆V (∗) k remains zero (the balance point is stable, proved in Theorem 2) until a new task increases forgetting. Furthermore, our theory claims the existence of a solution with respect to each task. This is also observed in Fig. 3a as, for each task, there is an increase in cost, and BCL quickly facilitates convergence. These observations indicate that our dynamical system in Eq. 2 accurately describes the dynamics of the continual learning problem. Furthermore, the assumptions under which the theory is developed are practical and are satisfied by performing continual learning on the permuted MNIST problem.

heading: 5 Conclusion
text: We developed a dynamic programming-based framework to enable the methodical study of key challenges in CL. We show that an inherent trade-off between generalization and forgetting exists and must be modeled for optimal performance. To this end, we introduce a two-player sequential game that models the trade-off. We show in theory and simulation that there is an equilibrium point that resolves this trade-off (Theorem 1) and that this saddle point can be attained (Theorem 2). However, we observe that any change in the task modifies the equilibrium point. Therefore, a global equilibrium point between generalization and forgetting is not possible, and our results are valid only in a neighborhood (defined given a task). To attain this equilibrium point, we develop BCL and demonstrate state-of-the-art performance on a CL benchmark [21]. In the future, we will extend our framework for nonEuclidean tasks.

heading: 6 Broader Impact
text: Positive Impacts: CL has a wide range of applicability. It helps avoids retraining, and it improves the learning efficiency of learning methods. Therefore, in science applications where the data is generated sequentially but the data distribution varies with time, our theoretically grounded method provides the potential for improved performance. Negative Impacts: Our theoretical framework does not have direct adverse impacts. However, the potential advantages of our approach can improve the efficiency of adverse ML systems such as fake news, surveillance, and cybersecurity attacks.
Acknowledgments and Disclosure of Funding
This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357 and by a DOE Early Career Research Program award. We are grateful for the computing resources from the Joint Laboratory for System Evaluation and Leadership Computing Facility at Argonne. We also are grateful to Dr. Vignesh Narayanan, assistant professor, University of South Carolina, and Dr. Marieme Ngom, Dr. Sami Khairy –postdoctoral appointees, Argonne National Laboratory, for their insights.

Review summaries:
The paper theoretically investigates the trade-off between generalization and forgetting in continual learning (CL). The authors prove the existance of a saddle point in the generalization/forgetting space; they show that it is stable under certain conditions; and propose a practical algorithm called balanced continual learning (BCL) for finding it. Finally, the authors compare their approach experimentally against the state of the art in CL.

The paper proposed a novel idea of formalizing the generality-forgetting trade-off in continual learning via a two-player sequential game. Theoretical and empirical results are represented to support the authors' claims. The main contribution of the paper is to formally discuss the forgetting and generalization trade-off theoretically.

This paper considers the stability-plasticity dilemma in continual learning. The authors develop an algorithm which addresses catastrophic forgetting while trying to improve generalisation on new tasks.  The paper introduces a new cost function for continual learning, which takes into account the model's performance on previous, current and future tasks. An approximation, H, to this cost function, which involves only the data that has been encountered so far, is optimised. The optimisation is phrased as a 2-player game, where player 1 tries to maximise H by creating worst-case dissimilar inputs for future tasks. Player 2 minimises the H by optimising the model's parameters. The authors show that the two players would eventially reach a single solution (an equilibrium). Catastrophic forgetting is prevented by maintaining a subset of previously encountered datasets and using it when optimising H.  Contributions: - A novel formulation of the cost function of continual learning (CL) - A theoretically justified approach to CL. - An implementation of their approach - Experiments which show that the resulting method outperforms the many baselines. 

The paper attempts to present a theoretical framework for modelling the trade-off between generalization (adaptation to future tasks, plasticity) and forgetting (stability on previous tasks). Towards this, the paper proposes an objective function that minimizes the weighted loss on all the past (implemented using episodic memory), present (implemented using the dataset of the current task) and future tasks (approximated using preturbing the datasets of the past and current tasks). The perturbation to this loss is then approximated in the local neighbourhood of the current task. The optimization of the loss is posed as a minimax game between the two players, where player 1 maximizes the generalization by maximizing the discrepancy between the successive tasks, and player 2 minimizes the forgetting by adapting the model parameters against that discrepancy. The authors show that if player 1 starts first, the equilibrium solution of this game exists in the local neighbourhood that is defined using the current task. The proposed approach achieves strong results against several baselines on MNIST and CIFAR-100 variants used for continual learning.