Formalizing the Generalization-Forgetting Trade-off in Continual Learning
Authors: Krishnan Raghavan
Prasanna Balaprakash
Venue: NeurIPS 2021 Poster (2021)
Keywords: Continual learning
dynamic programming
game theory
differential equations

Abstract: We formulate the continual learning (CL) problem via dynamic programming and model the trade-off between catastrophic forgetting and generalization as a two-player sequential game. In this approach, player 1 maximizes the cost due to lack of generalization whereas player 2 minimizes the cost due to catastrophic forgetting. We show theoretically that a balance point between the two players exists for each task and that this point is stable (once the balance is achieved, the two players stay at the balance point). Next, we introduce balanced continual learning (BCL), which is designed to attain balance between generalization and forgetting and empirically demonstrate that BCL is comparable to or better than the state of the art.

Formalizing the Generalization-Forgetiin Trade-Off in Continual Learning
R. Krishnan' / and Prasanna Balaprakash
Mathematics and Computer Science Division
2 LLeaderhhip Computing Facility
Argonne National Laboratory
kraghavva.ppalapraee anl.gov
Abstract
We formulate the continual learning problem via dynamic programming and model the trade-off between catastrophic forgetting and generalization as a two-player sequential game. In this approach, player maximizes the cost due to lack of generalization whereas player minimizes the cost due to increased catastrophic forgetting. We show theoretically and experimentally that a balance point between the two players exists for each task and that this point is stable (once the balance is achieved, the two players stay at the balance point). Next, continual learning (BCL), which is designed to attain balance between generaliza- tion and forgetting, and we empirically demonstrate that BCL is comparable to better than the state of the art.
introduce balanced
Introduction
In continual learning (CL), we incrementally adapt a model to learn tasks (defined according to the problem at hand) observed sequentially. CL has main objectives: maintain long-term memory (remember previous tasks) and navigate new experiences continually (quickly adapt to new tasks). An important characterization of these objectives is provided by the stability-plasticity dilemma [9]. where the primary challenge to balance network stability (preserve past knowledge; minimize catastrophic forgetting) and plasticity (rapidly learn from new experiences; generalize quickly). This balance provides a natural objective for CL: balance forgetting and generalization.
Traditional CL. methods either minimize catastrophic forgetting or improve quick generalization but do not model both. For example, common solutions to the catastrophic forgetting issue include representation-drivee
(1)
approaches 49 25 (2) regularization approaches [27 34
[6
4%
4g
24
37 10 43
[7]. Solutions to the generalization problem include representation-leammii approaches (matching nets 445) prototypical networks 42 and metalearning approaches [18
and (3) memory/expeerience replay [3 32
19
g (47). More recently, several approaches [55] 16
46 23 48
15] have been introduced that combine methods designed for quick generalization with frameworks designed to minimize forgetting.
The aforementioned CL approaches naively minimize loss function (combination of forgetting and generalization loss) but do not explicitly account for the trade-off in their optimization setup. The first work to formalize this trade-off the forgetting generalization trade-off was posed as a gradient alignment problem. Although MER provides promising methodology for CL. the balance between forgetting and generalization is enforced with several hyperparameters. Therefore, two key challenges arise: (1) lack of theoretical tools that study the existence (under what conditions does balance point between generalization and forgetting exists?) and stability (can this balance be realistically achieved?) of balance point
presented in meta-experience replay (MER) (38). where
forgettinggeenerlizattoo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and (2) lack of a systematic approach to achieve the balance point. We address these challenges in this paper.
We describe framework where we first formulate CL. as a sequential decision-making problem and seek minimize cost function summed over the complete lifetime of the model. At any time k, given that the future tasks are not available, the calculation of the cost function becomes intractable. To circumvent this issue, we use Bellman's principle of optimality 4 and recast the CL problem to model the catastrophic forgetting cost on the previous tasks and generalization cost on the new task. We show that equivalent performance on an infinite number of tasks is not practical (Lemma and Corollary 1) and that tasks must be prioritized. To achieve balance between forgetting and
Grudient Aucent
Gradient
Z Model
Descent
Model e
K=
Inpot Sample
T All Tasks data
81k -1 Lion
Task
OKk - 2) Lion and Cat
Data
Step Maximize
Step Minimize
0)+3)) Lion, Cat and Dog
Generalization
Catastrophic Forgetting
Player
Player
Figure I: (left) Exemplary CL problem: the lifetime of the model can be split into three intervals. At k = 1 we seek to recognize lions; at l = 2 we seek to recognize hoth lions and cats; and at k = 3 we seek to recognize cats, lions, and dogs. (right) Illustration of the proposed method: our methodology comprises interplay between two players. The first player maximizes generalization by simulating maximum discrepancy between two tasks. The second player minimizes forgetting by adapting to maximum discrepancy.
generalization, we pose the trade-off as a saddle point problem where we designate one player for maximizing the generalization cost (player 1) and another for minimizing the forgetting cost (player 2). We prove mathematically that there exists at least one saddle point between generalization and forgetting for each new task (Theorem 1). Furthermore, we show that this saddle point can be attained asymptotically (Theorem 2) when player strategies are chosen as gradient ascent-descent. We then introduce balanced continual learning (BCL), new algorithm to achieve this saddle point. In our algorithm (see Fig. [ for a description of BCL), the generalization cost is computed by training and evaluating the model on given new task data. The catastrophic forgetting cost is computed by evaluating the model on the task memory (previous tasks). We first maximize the generalization cost and then minimize the catastrophic forgetting cost to achieve the balance. We compare our approach with other methods such as elastic weight consoliaation (EWC) (27), online EWC (40) and MER 3 on continual learning benchmark data sets [21] show that BCL is better than or comparable to the state-of-the-arr methods. Moreover, we also show simulation that our theoretical framework is appropriate for understanding the continual learning problem. The contributions of this paper are (1) theoretical framework to study the CL problem, (2) BCL, a method to attain balance between forgetting and generalization, and (3) advancement of the state of the art in CL.
2 Problem Formulation
We use IR to denote the set of real numbers and N to denote the set of natural numbers. We use ||-|| to denote the Euclidean norm for vectors and the Frobenius norm for matrices, while using bold symbols to illustrate matrices and vectors. We define an interval j0, K), K E N and let p( ② be the distribution over all the tasks observed in this interval. For any lo E [0, K), we define a parametric model g()) with y = g( 0x), where 0₂ is a vector comprising all parameters of the model with Ct. E X Let be the number of samples and m be the number of dimensions. Suppose task at k l: [0, K) is observed and denoted as 2x Qx ~ (()), where 2, = {xx &. 4x) is a tuple with X's E nomm being the input data and l quantifies the loss incurred by Xx using the model for the task at k. We denote a sequence of 0 ur.k (0, De, k < T < K}, with Zs being the compact (feasible) set for the parameters. We denote the optimal value with a superscript (+); for instance, use of to denote the optimal value of 0 at task k. In this paper use balance

---

point, equilibrium point, and saddle point to refer to the point of balance between generalization and forgetting. We interchange between these terms whenever convenient for the discussion. We will use Voy to denote the gradient of with respect to j and Ai to denote the first difference in discrete time.
An exemplary CL problem is described in Fig. [ where we address total of K = 3 tasks. To particularize the idea in Fig. we define the cost (combination of catastrophic cost and generalization cost) at any instant l as Ju (0x) ) = "n + Σナ= 7rl,, where & is computed on task 2, with 7/ describing the contribution of 2 to this sum.
To solve the problem at we seek 0% to minimize Jx (0x)... Similarly, solve the problem in the complete interval [0, K]. we seek 02 to minimize Jx,00, for each k E [0, K]. In other words we seek to obtain 0₂ for each task such that the cost (+(OO) is minimized. Therefore, the optimization problem for the overall CL problem (overarching goal of CL) is provided as the minimization of the
cumulative cost V& (Uk:K)
is given Dr=k ( such that k
K
vi(-) = minuxx V& (u(u:));
(1)
with 0 S B, S being the contribution of J, and ui:K being weight sequence of length K k.
Within this formulation, two parameters determine the contributions of tasks: "r. the contribution of each task in the past, and 3r, the contribution of tasks in the future. To successfully solve the optimization problem, Via (u::k) must be bounded and differentiable, typically ensured by the choice of 7r 3,. Lemma 1 (full statement and proof in Appendix A) states that equivalent performance cannol be guaranteed for an infinite number of tasks. Furthermore, Corollary (full statement and proof in Appendix A) demonstrates that if the lask contributions are prioritized, the differentiability and boundedness of Jr (0+) )) can be ensured. A similar result was proved in (28), where a CL problem with infinite memory was shown be NP-hard from set theoretic perspective. These results (both ours and in (28) demonstrate that a CL methodology cannot provide perfect performance on large number of tasks and that tasks must be prioritized.
Despite these invaluable insights, the data corresponding to future tasks (interval [ls, K) is not available, and therefore (u cannot be evaluated. The optimization problem in Eq. ① naively minimizes the cost (due to both previous tasks and new tasks) and does not provide any explicit modeling of the trade-off between forgetting and generalization. Furthermore, Uk:K, the solution to Eq. (1) is sequence of parameters, and is not feasible to maintain ug:K for a large number of tasks. Because of these three issues, the problem is theoretically intractable in its current form.
We will first recast the problem using tools from dynamic programming [29], specifically Bellman's principle of optimality, and derive difference equation that summarizes the complete dynamics for the CL problem. Then, we will formulate a two-player differential game where we seek a saddle point solution to balance generalization and forgetting.
3 Dynamics of Continual Learning
Let = minu ΣK= 3 (0,); the dynamics of CL (the behavior of optimal cost with respect to k) provided as
Av = -mino [Bx + ((Vo, V A.x)) + (Ve v(()) Aax))].
(2)
The derivation is presented in Appendix A (refer to Proposition 1). Note that (+)
g is the minima for
the overarching CL problem in Eq. (2 and A AV(()) represents the change in Vi upon introduction
of new task (we hitherto refer to this as perturbations). Zero perturbations
(AV( = 0) implies that the introduction of new task does not impact our current solution; that is, the optimal solution on all previous tasks optimal on the new task as well. Therefore, the smaller the perturbations, the better the performance of model on all tasks, thus providing our main objective: minimize the
perturbations (AV (A)())) ). In Eq. A (v(*) is quantified by three terms: the cost contribution from all the previous tasks and the new task (e();); the change in the optimal cost due to the change in the parameters (Vo, k AOt): and the change in the optimal cost due the change in the input
(introduction of new task) [ xx (9) Azx).
3

---

The first issue with the cumulative CL problem (Eq. T can be attributed to the need for information from the future. In Eq. ( all information from the future is approximated by using the data from the new and the previous tasks. Therefore, the solution of the CI. problem can directly be obtained by solving Eq. (2) using all the available data. Thus, mingLen [ (A 0k) yields AV for > O, with ((A 0&)) (Ox) + (Vo, vgd") A0x) + (V V/(1) Arx)). Essentially, minimizing (Azz, 0x) would minimize the perturbations introduced by any new task k.
In Eq. (2. the first and the third term quantify generalization and the second term quantifies forgetting. model exhibits generalization when it successfully adapts to new task (minimizes the first and the third term in Eq. 2 The degree of generalization depends on the discrepancy between the previous tasks and the new task (numerical value of the third term in Eq. (2) and the worst-case discrepancy prompts maximum generalization. Quantification of generalization is provided by AæR that summarizes the discrepancy between subsequent tasks. However, AT₃ = Ix+1 - Zk. and #R+1 is unknown at k. Therefore, we simulate worst-case discrepancy by iteratively updating Aæx through gradient ascent in order to maximize H H( (Axx, ))); thus maximizing generalization. However, large discrepancy increases forgetting, and worst-case discrepancy yields maximum forgetting. Therefore, once maximum generalization is simulated, minimizing forgetting (update 0₂ by gradient descent) under maximum generalization provides the balance.
To formalize our idea, let us indicate the iteration index i by and write Arg as AzY ( with H (AæR, 0s) as H (Ax (6) o(() (for simplicity of notation, we will denote 11 (A ( as whenever convenient). Next, we write
(i) and Og as
(AA (9) (
min [B + (V V 2 (V (g v/( Aa(
min Ene
t Eno
[B (0)
(*)
v ( A
< min E
mar
Am (p)))
S min
max Aal (2) ~
H aa
E
(3)
In Eq. we seek the solution pair (Aml ( (Ro, Aaat)) where Aa(") maximizes e(*) minimizes H (minimizing player, player 2) where (La, G (l)()) are the feasible sets for Ar and @ respectively. The solution is attained, and
H (maximizing player, player 1) while 0
(Aah)) (+) e said to be the equilibrium point when it satisfies the following condition:
H H(A
(+) k 000) Z H( .00 2 11 11 (2) (
(4)
3.1 Theoretical Analysis
With our formulation, two key questions arise: Does our problem setup have equilibrium point satisfying Eq. 44h and how can one at- tain this equilibrium point? We answer these questions with Theorems and 2, respectively. Full statements and proofs are provided in Ap- pendix A.
#! pit) 4,50
Ax
To illustrate the theory, we refer to Fig. [ where the initial values for the two players are characterized by the pair ( circle), Aa (red circle))
and the cost value at {0).) Ax(") is indicated by (AA o(()) (the grey circle on the cost curve (the dark blue curve)). Our proofing strategy is as follows. First, we
Figure 2: Illustration of the proofs. Ax (player 1) is the horizontal axis, and the vertical axis indicates (player 2) where the curve indicates HI. If we start from the red circle for player (player 2 is fixed at the blue circle), H is increasing (goes from a grey circle to a red asterisk) with player reaching the red asterisk. Next, start from the blue circle (0 is the red asterisk), the cost decreases.
fix o E Le and construct neighborhood
4

---

Within this neighborhood we prove in Lemmas and 4 that we search for A through gradient ascent, we can converge to local maximizer, and H is maximizing with respect to A Second, we let Ax, (.) S2g be fixed, and we search for ( through gradient descent. Under this condition, we demonstrate two ideas in Lemmas and 5: (1) we show that H is minimizing in the neighborhood N₁ .V& = {22g A { and (2) converge to the local minimizer in the neighborhood. Mi. Third, in the union of the two neighhorhoods, Ma U.2, (proven to be nonempty according to Lemma 6), we show that there exists at least one local equilibrium point (Theorem 1); that is, there is at least one balance point.
MR = (2g, 2
):
Theorem 1 (Existence of an Equilibrium Point). For any ls E [0, K), ler o E Ag. be the minimizer
of H according to Lemma and define M = {Sa,, ) Similarty, let Axy E Qg be the maximizer of H according to Lemma 4 and define v((1) = (Aa (+), Sla). Further, let M UN((
be nonempty according to Lemma. 6, then (Ark (+)
k E MM UN is local equilibrium point.
We next show that this equilibrium point is stable (Theorem 2) under a sequential play. Specifically, we show that when player plays first and player 2 plays second, we asymptotically reach saddle point pair (Az(") ( for H. At this saddle point, both players have no incentive to move, and the game converges.
Theorem 2 (Stability of the Equilibrium Point). For any E 10, Ark E
Og and (
Qg be the initial values for Aa (i) and o respectively. {La, Oo) with H(Amk) (1) o given by Proposition 2. Let Aaki (i+1) - -Ar (1)
Define
Mr
( x (V (a H(Aak) (0) I)/II Aa (Ax (6) (( and ((i+1) - - = -o -0k x
k
V oo H(Aa O()). Let the existence of an equilibrium point be given by Theorem 1, then, as consequence of Lemmas 2 and 3, (Aal (t) o(r)) E EMy is a stable equilibrium point for H given.
In this game, the interplay between these two opposing players (representative of generalization and forgetting, respectively) introduces the dynamics required to play the game. Furthermore, the results presented in this section are local to the task. In other words, we prove that we can achieve a balance between generalization and forgetting for each task l (neighborhoods are task dependent, and we achieve local solution given a task k). Furthermore, our game is sequential; that is. there is a leader (player 1) and follower (player 2). The leader (Ax( plays first, and the follower (o(") plays second with complete knowledge of the leader's play. The game is directed by Aak" and any changes in the task (reflected in Ax(")) will shift the input and thus the equilibrium point. Consequently, the equilibrium point varies with respect to task, and one will need to attain new cquilibrium point for each shift in a task. Without complete knowledge of the tasks (not available in a CL scenario), only a local result is possible. This highlights one of the key limitations of this work. Ideally, we would like a balance between forgetting and generalization that is independent of tasks. However, this would require learning a trajectory of the equilibrium point (How does the equilibrium point change with the change in the tasks?) and is beyond the scope of this paper. One work that attempts do this is (39), where the authors learn a parameter per task. For a large number of tasks, however, such an approach is computationally prohibitive.
These results are valid only under certain assumptions: (1) the Frobenius norm of the gradient is bounded, always positive; (2) the cost function is bounded and differentiable; and (3) the learning rate goes to zero as i tends to infinity. The first assumption is reasonable in practice, and gradient clipping or perturbation strategies can be used to ensure it. The boundedness of the cost (second assumption) can be ensured by prioritizing the contributions of the task (Lemma 1 and Corollary 1). The third assumption assumes a decaying learning rate. Learning rate decay is a common strategy and is employed widely. Therefore, all assumptions are practical and reasonable.
3.2 Balanced Continual Learning
Equipped with the theory, we develop a new CL method to achieve a balance between forgetting and generalization. By Proposition 2, the cost function can be upper bounded as H (Ax .o() <
(1)
5

---

( k + (J=(0 Jx )= on player 1 and ( indicates { updates on player 2.
+ (00))((0)) J=(( )) where Ju+s indicates C updates
The strategies for the two players Aæg, Os are chosen in Eq. (5) with E being the expected value operator. We can approximate the required terms in our update rule (player strategies) using data samples (batches). Note that the ap- proximation is performed largely through one-sided finite difference, which may introduce an error and is another po- tential drawback. The pseudo code of the BCL is shown in Algorithm|l
(1) k
V Da. (A O 2
Aa H (Aa (1) k
Player
(5)
x Vo, E &
I We define a new task array (Ik) and a task
Player 2
memory array (kk C - 2- (samples from all previous tasks). For each batch by E Dk:(k) sample bp from p (kk), combine create (p (k)) U byb (k), and perform sequential play, Specifically, for each task the first player initializes =x = bp (k) and performs Ç updates on PN through gradient ascent. The second player, with complete knowledge of the first player's strat- Tk
) To estimate player 2's play, we must estimate egy, chooses the best play to reduce H (Ax i
different terms in H (A (6) ) This procedure involves three steps. First, we use the first player's play and approximate (Ju+c (e( Jz (O Second, to approximate (0( - Ju (0%) )) (a), we copy Ô into 0B (a temporary network) and perform updates on OB: and (b) we compute J&(O ()) using (k + () and evaluate (J&(O - Jz (o())). )). Third, equipped with these ap- proximations, we compute H (AA (1) .(())) and obtain the play for the second player. Both these players perform the steps repetitively for each piece of information (batch of data). Once all the data
from the new task is exhausted, move to the next task.
3.3 Related Work
Algorithm 1: BCL
Traditional solutions to the CL focus on either the for- getting issue (39 49
Initialize 0, 0pD DN while k = 1, ...K do j= 0
47
7 27
50
2
31
11 or 8 Common
32
the generalization issue [45
42
88
19
solutions to the forgetting problem involve dynamic
while j < p do
chitectures and flcxible knowledge representation such 47 7, regularization approaches including
Get bN E D
B9
49
Get bp E D
[7 Similarly, quick generalization to new task has been addressed through few-shot and one-shot learning ap- proaches such as matching nets (45) and prototypical network [42). More recently, the field of metalearning has approached the generalization problem by design- ing a metalearner that can perform quick generaliza- tion from very little data [18 19 8)
50 2 and memorylexperience replay [I 32] I1]
Get bpx = bp by
Copy bpN into o PN
i= 0 while i <= ( do
Update x._" with J& (0x)
using gradient ascent i=it1
Calculate Jk+C (++c((()))) - J4 (0)
Copy @ into 0
In the past few years, metalearners for quick gener- alization have been combined with methodologies specifically designed for reduced forgetting [2 3] For instance, the approaches in [2 adapt the model- agnostic metalearning (MAML) framework in [88 with robust representation to minimize forgetting and provide impressive results on CI. However, both these approaches require pretraining phase for learning representation. Simultancously, Gupta et al. (20) intro- duced LA-MAML metalcarning approach where the impact of learning rates on the CL problem is re- duced through the use of per-paramette learning rates.
i= 0 while + <<= Ç do
Update 0 with Jn ( i=i+1
Calculate (0
B ( Calculate H1 (Ax x())
Update @ using gradient descent j=j+1
Update Dr with DN
LA-MAML (20 also introduced episodic memory to address the forgetting issue. Other approaches also have attempted to model both generalization and forgetting. [16], the gradients from new tasks are projected onto a subspace that is orthogonal to the older tasks, and forgetting is mini- mized. Similarly, Joseph and Balasubramanian (3) utilized Bayesian framework to consolidate
learning across previous and current tasks, and Yin et al. [48] provided a framework for approx-

---

imating loss function to summarize the forgetting in the CL setting. Furthermore, Abolfathi et al. [ focused on sampling episodes in the reinforcement learning setting, and Elrahimi et al. [T5] introduced a generative adversarial network-type structure to progressively learn shared features assisting reduced forgetting and improved generalization. Despite significam significant progress, however, these 23 48, [5] are still inherently tilted toward maximizing generalization or minimizing forgetting because they naively minimize the loss function. Therefore, the contribution of different terms in the loss function becomes important. For instance, if the generalization cost is given more weight, a method would generalize better. Similarly, if forgetting cost is given more weight, method would forget less. Therefore, the resolution of the rradeooff inherently depends on an hyperparameter.
methods (22 3 20
16
The first work to formalize the trade-off in CL was MER, where the trade-off was formalized as gradient alignment problem. Similar to MER, Doan et al. ((13) studied forgetting as an alignment problem. In MER, the angle between the gradients was approximated by using Reptile [36), which promotes gradient alignment by reducing weight changes. On the other hand, [3] formalized the alignment an eigenvalue problem and introduced PCA-driven method to ensure alignment. Our approach models this balance as a saddle point problem achieved through stochastic gradient such that the saddle point (balance point or equilibrium point) resolves the trade-off.
Our approach is the first in the CL literature to prove the existence of the saddle point (the balance point) between generalization and forgetting given a task in CL problem. Furthermore, we are the first to theoretically demonstrate that the saddle point can be achieved reasonably under a gradient ascent-descent game. The work closest to ours is [[.5] where an adversarial framework is described to minimize forgetting in CL by generating task-invariant representation. However, [55] is not model agnostic (the architecture of the network is important) and requires considerable amount of data at the start of the learning procedure. Because of these issues, [1] is not suitable for learning in the sequential scenario.
Several attempts have been reported in the literature to theoretically analyze different aspects of CL. For instance, Benzing [6] attempted to unify regularization methods such as (27 50
2
31
On the other hand, Benamin et al. (5] provided generalization (in the context of training to test data generalization) guarantees with orthogonal gradient descent, and Yin et al. generalization (in the context of training to test data generalization) analysis while regularizing with a second-order approximation of the loss functions. Although these works provide important results, they focus on quantifying catastrophic forgetting or generalization but do attempt to model the trade-off in any manner.
48 provided
4 Experiments
We use the CL benchmark 21 for our experiments and retain the experimental settings (hyperparame- ters) from [21] (4.. For comparison, we use the split-MNIST, permuted-MNIST and split-CiFARIO0 100 data sets while considering three scenarios: incremental domain learning (IDL), incremental task learning (ITL), and incremental class learning (ICL). The splitting and permutation strategies when applied to the MNIST or CiFARI00 data set can generate task sequences for all three scenarios (il- lustrated in Figure 1 and Appendix: Figure 2 of (21). For comparing our approach, we use three baseline stradgges--tanddrdd neural network with Adam (6).. Adagrad [14,, and SG and use Ia-regularization and naive rehearsal (which is similar to ex experience replay). For CL approaches, we use EWC 27 online EWC [40), SI [50], LwF [30, DGR 411, RtF 44, MAS 2], MER (38), and GEM 33 We utilize data preprocessing as provided by 21 Additional details on experiments can be found in Appendix B and 12 44). All experiments are conducted in Python 3.4 using the pytorch 1.7.1 library with the NVIDIA-A100 GPU for our simulations.
Comparison with the state of the art: The results for our method are summarized in Table and The efficiency for any method is calculated by observing the average accuracy (retained accuracy (RA) 38) at the end of each repetition and then evaluating the mean and standard deviation of RA across different repetitions. For each method, we report the mean and standard deviation of RA over five repetitions of each experiment. in bold. For the split-MNIST data set, we obtain 99.52 + 0.07 for ITT.,, 98.71 + 0.06 for IDL, and 97.32 + 0.17 for ICI. Similarly, with the peNMuted-MNIST data set, we obtain 97.41 + 0.01 for ITL, 97.51 + 0.05 for IDL, and 97.61 + 0.01 for ICL. Furthermore, with the Sp0it-CiFAR100 data
each column, we indicate the best-performing method
7

---

Table I: Performance of our approach compared with other methods in the literature. We record the mean and standard deviation of the retained accuracy for the different methods. The best scores in bold.
split-MNIST
permuted-MNN
Method
Incremertaa task learning [ML)]
Incretienaal domain leaming [ILL]
Incremental learning [CL]]
Incremental task learning
Incremental domain leamning
Incremental class learning [CC]]
[TTL]
[DLL]
Adum
95.52 + 2.14
54.75
1 2.06
19.72
0.03
93.42 0.56
77.87
1.27
14.02 1.25
SGD
97.65 +: 0.28
62.80
+ 0.34
19.36
0.02
90.95 +: 0.20
78.17
+: 1.16
12.82 +: 0.95
Adagrdd
98.37 + 0.22
57.59
+ 2.54
19.59
0.17
92.45
+ 0.16
91.59
+ 0.46
29.09
= 1.48
Lg
0.69
3.91.
22.92
91.87
0.38
13.92
1.79
Naive reheumal
98. 32 + 0.10
94.85
+ 0.80
90.88
= 0.70
96.23
4 0.04
95.84
0.06
96.25
= 0.10
Naive reheanal-C
99.41 0.04
97.13
+ 0.37
94.92
0.63
97.13 0.03
96.75
0.03
97.24
0.05
FWC
96.59 0.00
57.31 = 1.07
19.70
0.14
95.38
0.33
80.54 = 0.52
26.32
4.32
Online EWC
99.01 + 0.112
58.25 + 1.23
19.68
0.05
95.15 + 0.49
93.47
0.01
42.58 = 6.50
SI
99.10 0.16
64.63
1.67
94.35
0.51
91.12
58.52
4.20
MAS
98.58 + 0.14
61.98 = 7.17
0.34 19.70
91.74 = 0.52
93.22 - 0.80
50.81
2.92
GEM
98.32 + 0.08
97.37 + 0.22
93.04
0.05
95.44 + 0.96
96.86 - 0.02
96.72 = 0.03
DGR RIF
99.47 0.03
95.74 0.23
91.24
92.52 0.08
92.19
0.09
99.66 + 0.03
97.31
+ 0.11
92.56
0.21
97.
+ 0.01
97.06 + 0.02
96.23 = 0.04
MER
97.12 0.10
92.16
0.35
93.20
0.12
97.15
0.08
96.11 # 0.31
91.71
0.03
BCL (With Game)
99.52 0.07
98.71
1: 0.06
97.32
0.17
97
0.01
97.51
0.05
97.61
0.01
BCL (Witout Game)
97.73 + 0.03
96.43
# 0.29
91.88
=: 0.55
96.16 + 0.03
96.08 + 0.06
95.96
0.06
set, we obtain 81.82 + 0.17 for ITL, 62.11 + 0.00 for IDL, and 69.27 + 0.03 for ICL. BCL is the
best-performing methodology for all cases (across both data sets) except RıF for ITL (0.14% drop) with the split-MNIST data set.
Generally, ITL is the casiest learn- ing scenario [2I), and all methods therefore perform well on ITL (the performance is close). For the ITL scenario with the split-MNIST data set, BCL is better than most methods; but several methods, such as naive re- hearsal, naive rehearsal-C, RIF, and DGR, attain close RA values (less than 1% from BCL). Note that both DGR and RtF involve a generative model pretrained with data from all the tasks. In a sequential learning scenario, one cannot efficiently train generative models because data cor- responding to all the tasks is not available beforehand. Although RtF provides improved performance for split-MNIST (TTL), the improvement is less than 1% (not significant). In fact, RtF performance is poorer for BCL in by 4.76% (a significant drop in performance) and in IDL. by 1.4%.
Table 2: Performance of BCL for the split-iFARR000 data set. We
record the retained accuracy for the different methods. We obtained RA scores for all methods except BCL from 21
CPARIOCIA
Method
Incremental task learning
Incremental
Incremencal class leaming
domain lesrning
Adam
30.53 0.58
19.65 + 0.14
17.20 0.06
SGD
43.77 +: 1.15
19.17 + 0.12
17.18 +: 0.12
Adaagnad
36.27 0.43
19.06 + 0.14
15.83
0.20
1.30
19.96
+
17.12
0.04
Naive rehearsal
70.20
0.17
35.94
0.39
34.23
0.19
Nuive rchetesal-CC
78.41 + 0.37
51.81
+ 0.18
51.28 +: 0.17
EWC
61.11 4 1.43
19.76
+ 0.12
19.70
1: U.14
Online EWC
63.22 0.97
20.03
1 0.10
17.16
0.09
64.81 1.00
20.26
0.09
17.26
0.11
MAS
64.77
0.78
19.99
+ 0.16
17.07
0.12
BCL/WWth Game)
81.2 1 0.17
62.11 1 0,00
69.27 1 0.03
BCL/Without Game!
51.82 +
Two additional observations can be made about the split-MNIST data set. First, Adagrad, SGD, and L2 achieve better performance than does Adam. Therefore, in our analysis Adagrad appears more appropriate although Adam is popularly used for this task. Second, naive rehearsal (both naive rehearsal and naive rehearsal-C -C approaches) achieves performance equivalent to the state-of-the-ar methods with similar memory overhead. Furthermore, naive rehearsal performs much better than online EWC and SI, especially in the ICL scenario. These limitations indicate that regularization- driven approaches are not much better than baseline models and in fact perform poorer than methods involving memory (naive rehearsals, MER, BCL,, etc.). 228, it was shown theoretically that memory-based approaches typically do better than aegularization-driee approaches, as is empirically observed in this paper, too. Another interesting observation is that EWC and online EWC require significant hyperparameter tuning, which would be difficult to do in real-wool world scenarios. Other regularization-bast methods, such as SI and MAS, also suffer from the same issue.
The observations from the -NiSTMMNIST carry forward to the permuted- -NNIST data set. Moreover, RA values for the permutation MNIST data set are better for the split-MNIST data set across the board, indicating that the permutation MNIST data set presents an easier learning problem. Similar to the observations made with the Split-MNIST data set, BCL better than all methods for the permuted-MNIST dataset, with naive rehearsal and RtF providing RA values that are close (less than 1%). The only methodology in the literature that attempts to model the balance between forgetting and generalization is MER, extension of GEM. From our results, observe that BCL is better than
8

---

MER in all cases (SplitMMNIST- 2.4% improvement for ITL, 6.55% improvement for IDL, 4.12% improvement for ICL and Comuued-MNIISTOO2255 improvement for ITL. 1.4% improvement for IDI. and 5.9% improvement for ICI.).
The substantial improvements obtained by BCL more evident in Table 2 where the results on the split-CiFAR 100 data set are summarized. BCL is clearly the berr-rrni method. The next best-performing method naive rehearsal-C, where BCL improves performance by 3.41% for ITL, 10.3% for IDL, and 17.99% for ICL. Other observations about the regularization methods and the rest of the baseline methods carry forward from Table T However, one key difference that while AdaGrad is observed to be better than Adam for the MNIST data set, Adam is comparable to SGD and Adagrad for split- CiFARI00 In summary, BCL is comparable to or better than the state of the art in the literature for both the MNIST and CiFAR 100 data sets.
Do need game to achieve this perfor- mance? To provide additional insights into the benefits of the game, we compare RA values with and without the game. In our setup, Axy aims increasing the cost, and 0% aims at re- ducing the cost. If we hold the play for Ax then the dynamics required to play the game do not exist, thus providing a method that can perform CL without the game. Therefore, we induce the absence of game by fixing A (f) and perform each of the nine experiments for five repetitions (ICL, IDL. and ITL for split- MNIST, permuted-MNIS. We summarize these results in the last two rows of Tables and 2 Consequently, we make two observations. First: even without the game, achieve RA values comparable to the state of the art This is true for both the MNIST and Ci- FAR data sets. Second, with the introduction of the game, across the board (at least by 1% with MNIST). The difference is clearer with the CiFAR100 data set where we observe a substantial improvement in RA values (at least 10%).
1 .66)
J.00] (V,, 1L" .Arr)
VV2
20
15
10
-5
-10
200
600
800
1200
(update iteration at each task
(a)
and split-CiFAR 100).
15
5
0-
observe improved RA values
-5
50
100
150
200
250
300
(update iteration at task 3-
(b)
Does the theory appropriately model the con- tinual learning problem? In Fig. Ba we plot the
Figure 3: (Top) Progression of different terms in Eq. 2 with respect to update iterations, where the index the axis is calculated à x 300. The tasks boundaries (at what i the tasks are introduced) are illustrated through shades of grey, (down) Illustration of the cost at k = 3.
progression of (Os) + (V.r, ( Aar) (blue curve), (Ve, (-()) A0x)) (red curve), and the sum, namely, AV (green curve). A total of six tasks, sampled from the permutation MNIST data set within the ICL setting, are illustrated in Fig. 3a These tasks are introduced every 300 update steps. From Fig. Ba we make two important observations. First, as soon as tasks 1, 3, 4, and 6 are introduced, all three curves (red, blue, and green) indicate bump. Second, when tasks 2 and 5 are introduced, there no change. On a closer look at task 3 in Fig. 3b we observe that when the task 3 is introduced, the blue curve exhibits a large positive bump (the introduction of the new tasks increases the first and the third terms in Eq. (2). The increase implies that task 3 forced the model to generalize and increased forgetting on tasks and 2 (observed by the increase in the green curve). To compensate for this increase, we require the model 0 to behave adversarially and introduce large
enough negative value in (Ve, (2("), AOk) (red curve) to cancel out the increase in the blue curve. In Fig. (3b the red curve demonstrates large negative value (expected behavior) and eventually (as increases) forces the blue curve (by consequence, green: the sum of red and blue) to move toward zero (the model compensates for the increase in forgetting). As observed, the blue and the red curves behave opposite to each other and introduce push- pull behavior that stops only when the two cancel each other and the sum (green) is zero. Once the sum has reached zero, there no incentive for the

---

red and green to be nonzero, and therefore they remain at zero; thus, all three curves (green, red, and blue) remain at zero once converged until task 4 introduced (when there is another bump. as seen in Fig. Ga). However, this increase in the blue curve is not observed when tasks 2 and 5 introduced. When new tasks are similar the older tasks, it is expected that AV observed in Fig. 3a
as is
All of these observations are fully explained by Eq. which illustrates that the solution to the CL problem obtained optimally only when A AV,
(observed Figs.
3a and 3bp. The term
AV(") is quantified by Jx((x) (Vo, vi+) A0&) and (V.,, vi(*) Arx). Our theory suggests that there exists an inherent trade-off between different terms in Eq. Therefore every time a new task is observed, it is expected that Js (0x) ) + (V v((",, Arx) increases (increase in blue curve when tasks
1. 3, 4, and 6 are introduced) and (Va, l A compensates to cancel this increase (red curve exhibits negative jump). In Theorems and we demonstrate the existence of this balance point (for each task, as í increases, Á & tends to zero, as observed in Fig. 3b) and AV," remains zero (the balance point is stable, proved in Theorem 2) until new task increases forgetting. Furthermore, our theory claims the existence of a solution with respect to each task. This is also observed in Fig. 3a as, for each task, there is increase in cost, and BCL quickly facilitates convergence. These observations indicate that our dynamical system in Eq. P Qaccurately describes the dynamics of the continual learning problem. Furthermore, the assumptions under which the theory is developed practical and are satisfied by performing continual learning on the permuted MNIST problem.
(*))
5 Conclusion
We developed a dynamic programming- framework to enable the methodical study of key challenges in CL. We show that an inherent trade-off between generalization and forgetting exists and must be modeled for optimal performance. To this end, we introduce two-player sequential game that models the trade-off. We show in theory and simulation that there is equilibrium point that resolves this trade-off (Theorem I) and that this saddle point can be attained (Theorem 2). However, we observe that any change in the task modifies the equilibrium point. Therefore, a global equilibrium point between generalization and forgetting is not possible, and our results are valid only in neighborhood (defined given task). To attain this equilibrium point, develop BCL and demonstrate state-of of-the-art performance on CL benchmark [21]. In the future, we will extend our framework for nonEuclidean tasks.
6 Broader Impact
Positive Impacts: CL has a wide range of applicability. It helps avoids retraining, and it improves the learning efficiency of learning methods. Therefore, in science applications where the data is generated sequentially but the data distribution varies with time, our theoretically grounded method provides the potential for improved performance. Negative Impacts: Our theoretical framework does not have direct adverse impacts. However, the potential advantages of our approach can improve the efficiency of adverse ML systems such fake news, surveillance, and cybersecurity attacks.
Acknowledgments and Disclosure of Funding
This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, under Contract DE-AC02-06CHI12 1357 and by a DOE Early Career Research Program award. We are grateful for the computing resourees from the Joint Laboratory for System Evaluation and Leadership Computing Facility at Argonne. We also are grateful to Dr. Vignesh Narayanan, assistant professor, University of South Carolina, and Dr. Marieme Ngom, Dr. Sami Khairy -postdoctoral appointees, Argonne National Laboratory, for their insights.
References
[1] Elmira Amirloo Abolfathi, Jun Luo, Peyman Yadmellat, and Kasra Rezaee. CoachNet: adversarial sampling approach for reinforcement learning, 2021.
10

---

[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny. Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154, 2018.
[3] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09577 099001, 2020.
[4] Richard E Bellman. Adaptive control processes: a guided tour. Princeton university press,
2015.
[5] Mchdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006. 11942, 2020.
[6] Frederik Benzing. Unifying regularisation methods for continual learning. 2021.
[7] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: approach to continual learning, 2021.
[8] Massimo Caccia, Pau Rodríguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam H. Laradji, Irina Rish, Alexande Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: A new approach to continual learning. ArXiv, abs/2003.05555 005566,, 2020.
[9] Gail Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1):54-
115, 1987.
[10] Arslan Chaudhry, Naeemullah Khan, Puneet K Dokania, and Philip HS Torr. Continual learning in low-rank orthogonal subspaces. arXiv preprint arXiv:2010.11633, 2020.
[11] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint 1rXiv:1902.1044: 2019.
[12] Arslan Chaudhry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint /9Xiv:/902. 10486, 2019.
[13] Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusscau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the NTK overlap matrix. In International Conference on Artificial Intelligence and Statistics, pages 1072-1080. PMLR, 2021.
[14] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7), 2011.
[15] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning, 2020.
[16] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning, 2019.
[17] Enrico Fini, Stéphane Lathuilière, Enver Sangineto, Moin Nabi, and Elisa Ricci. Online continual learning under extreme memory constraints. In European Conference on Computer Vision, pages 720-733.. Springer, 2020.
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1126-1135 JMLR. org, 2017.
[19] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. arXiv preprint 1rXiv:1902.08434 2019.

---

[20] Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-MAML: Look-ahead meta learning for continual learning, 2020.
[21] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. In NeurIPS Continual learning Workshop, 2018.
[22] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pages 1818-1828, 2019.
[23] K J Joseph and Vineeth N Balasubramanian. Meta-consolidation for continual learning, 2020.
[24] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. arXiv e-prints, pages arXiv-2003, 2020.
[25] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. Advances in Neural Information Processing Systems, 33, 2020.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[27] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabska-Barwinst et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114((3):3521 -3526, 2017.
[28] Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is NP-hard. In International Conference on Machine Learning, pages 5327-5337. PMLR, 2020.
[29] Frank L Lewis, Draguna Vrabie, and Vassilis L Syrmos. Optimal control. John Wiley & Sons, 2012.
[30] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12):2935 2947, 2017.
[31] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3-4):293-321, 1992.
[32] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pages 6467-6476, 2017.
[33] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. arXiv preprint arXiv: 080000884, 2017.
[34] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning, arXiv preprint arXiv:2006.069) 2020.
[35] Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based RL, 2019.
[36] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
[37] Pingbo Pan, Siddharth Swaroop. Alexander Immer, Runa Eschenhagen, Richard E Turner, and Mohammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. arXiv preprint arXiv:2004.14071 2020.
[38] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 2018.
12

---

[39] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer. James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, arXiv preprini arXiv: 0666600667,, 2016.
[40] Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Brabska-Barwinska Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528 9928-553.. PMLR, 2018.
[41] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. arXiv preprint axXiv/705.088900 2017.
[42] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087. 2017.
[43] Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with Gaussian processes. arXiv preprint arXiv: /901. 1356, 2019.
[44] Gido M. de Ven and Andreas S. Tolins. Generative replay with feedback connections general strategy for continual learning, 2019.
[45] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29:3630-3638, 2016.
[46] Risto Vuorio. Dong- Yeon Cho, Daejoong Kim, and Jiwon Kim. Meta continual learning. CoRR, abs/1 806.06928, 2018.
[47] Huaxiu Yao, Longkai Huang, Ying Wei, Li Tian, Junzhou Huang, and Zhenhui Li. Don't overlook the support set: Towards improving generalization in meta-learning arXiv preprint arXiv:2007. J 13040, 2020.
[48] Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, and Alex Mott. Optimization and general- ization of regularization-base continual learning: loss approximation viewpoint, 2020.
[49] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.00 01547, 2017.
[50] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of Machine Learning Research, 70:3987, 2017.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Fig (right)
(b) Did you describe the limitations of your work? [Yes] Refer Section 3.1 and Section 3.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Refer Section Broader Impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Refer to the theorem statements in section. 3.1
(b) Did you include complete proofs of all theoretical results? [Yes] Refer to proofs in Appendix A
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] Refer to supplementary materials
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Refer Appendix in supplementary materials and [2]]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Five repetitions and mean and standard deviations
reported. Refer to Table. [
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Refer the Results section assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We used the continual learning benchmark by 21;; we cite their paper.
4. If you using existing assets (c.g., code, data, models) curating/releasing
(b) Did you mention the license of the assets? [Yes] No licence information was provided
by (2
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you are using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
The submitted manuscript has been created by UChicago Argonne, I.IC,, Operator of Argonne National Laboratory ("Argonne"). Argonne, a U.S. Department Energy Of- fice of Science laboratory, is operated under Contract No. DE-AC02-06CHII33 U.S. Government retains for itself. and others acting on its behalf, paid-up nonexclu- sive, irrevocable worldwide license said article reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on be- half of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan.
The
14


Reviews:
[{"review_id": "uZPG0anE8Ju", "reviewer": "Reviewer_jBBb", "summary": "The paper attempts to present a theoretical framework for modelling the trade-off between generalization (adaptation to future tasks, plasticity) and forgetting (stability on previous tasks). Towards this, the paper proposes an objective function that minimizes the weighted loss on all the past (implemented using episodic memory), present (implemented using the dataset of the current task) and future tasks (approximated using preturbing the datasets of the past and current tasks). The perturbation to this loss is then approximated in the local neighbourhood of the current task. The optimization of the loss is posed as a minimax game between the two players, where player 1 maximizes the generalization by maximizing the discrepancy between the successive tasks, and player 2 minimizes the forgetting by adapting the model parameters against that discrepancy. The authors show that if player 1 starts first, the equilibrium solution of this game exists in the local neighbourhood that is defined using the current task. The proposed approach achieves strong results against several baselines on MNIST and CIFAR-100 variants used for continual learning. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n- The paper is written very well and the generalization-forgetting trade-off is well-motivated. \n- There is a dearth of theory papers in continual learning and this presents a very nice theory for analyzing the trade-off between forgetting and generalization. \n- Empirical results are quite strong. \n\nNegatives:\n\nWhile I don’t have many problems with the paper, I do like to ask a couple of questions. \n\n- Modelling generalization: The paper argues that the degree of generalization depends on the discrepancy between the previous and new tasks. Hence in maximizing that discrepancy, and subsequently training your model to account for that discrepancy, we maximize the generalization. I could see why as an optimization objective such an adversarial training would encourage the model to adapt to potentially different tasks. But you can’t say that a model would generalize better to more different tasks, in fact it is the opposite. If the two tasks are similar then the model has a better chance of generalizing (it's a test time concept). Maybe the authors would want to rephrase the lines 121-125 to correctly reflect this. \n\n- Time constraints: It seems that the proposed approach BCL is quite time consuming. Could the authors please provide the timing comparison of their approach with the other baselines?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "l6FIfzj1xL", "reviewer": "Reviewer_aHYF", "summary": "This paper considers the stability-plasticity dilemma in continual learning. The authors develop an algorithm which addresses catastrophic forgetting while trying to improve generalisation on new tasks.\n\nThe paper introduces a new cost function for continual learning, which takes into account the model's performance on previous, current and future tasks. An approximation, H, to this cost function, which involves only the data that has been encountered so far, is optimised. The optimisation is phrased as a 2-player game, where player 1 tries to maximise H by creating worst-case dissimilar inputs for future tasks. Player 2 minimises the H by optimising the model's parameters. The authors show that the two players would eventially reach a single solution (an equilibrium).\nCatastrophic forgetting is prevented by maintaining a subset of previously encountered datasets and using it when optimising H.\n\nContributions:\n- A novel formulation of the cost function of continual learning (CL)\n- A theoretically justified approach to CL.\n- An implementation of their approach\n- Experiments which show that the resulting method outperforms the many baselines.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclaimer: I did not verify the proofs in the appendix.\n\nIt was interesting for me to read the new CL cost function. The analysis of the cost function and the proposed 2-player approach was also informative.\nThe experiments evaluate different CL settings (class-incremental, domain-incremental and task-incremental), which I think is useful for determining the overall worth of a proposed new method.\n\n\nWeaknesses: \n[Experiments] I'd be curious to see experiments with mixed datasets (e.g. CIFAR10 and MNIST), which would better illustrate whether your method can generate to diverse input domains\n\nNote: I'm not sure that the baselines used in the experiments are state-of-the-art.\n\nNote: [Clarity] It appears that Algorithm 1 has j=0 and i=0 swapped. Needs to be corrected.\n\nQuestions regarding the cost function H on line 206: \nq1: When approximating H for player 2, you copy over the parameters in a new network \\theta_B and optimise them. Afterwards, you use the predictions of this optimised network to calculate H. However, these predictions are not differentiable wrt the original \\theta , are they? If this is the case, why calculate it at all?\nq2: H includes J_k(\\theta) a number of times: H = \\beta * J_k(\\theta) + ... - J_k(\\theta) + ... -J_k(\\theta) = J_k(\\theta) *(\\beta - 2) + ... .\nIf \\beta < 2,  when player 2 is minising H, wouldn't it try to increase J_k(\\theta) in order to minimise H? In this case, I find it a bit confusing why the method works so well?\nI looked at your code as well to try to find the answers to the above questions, but it's not clear to me. Looking at the code, it appears that you are using \\beta = 1.\n\nq3: How computationally intensive would you say your method is, compared to others?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "hOnefg5r2o", "reviewer": "Reviewer_Me2K", "summary": "The paper theoretically investigates the trade-off between generalization and forgetting in continual learning (CL). The authors prove the existance of a saddle point in the generalization/forgetting space; they show that it is stable under certain conditions; and propose a practical algorithm called balanced continual learning (BCL) for finding it. Finally, the authors compare their approach experimentally against the state of the art in CL.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, the paper is original and contains some very interesting ideas (i.e., the generalization/forgetting trade-off and using gradient ascent to improve generalization).\n\nThe text is mostly clear, but I think the authors should try to simplify their mathematical notation, as it is very dense and very difficult to wrap your head around it.\n\nThe theoretical results and the algorithm presented in the paper are significant for non-task-free CL settings. \n\nI feel that the experiments of the paper are insufficient. Two-thirds of the experimental work are based on the MNIST dataset which is extremely simplistic and cannot be considered to give trustworthy benchmarks. The Permuted MNIST benchmark, in particular, has been (in my opinion, justly) criticized by other works (e.g., Farquhar and Gal, 2018) for not being realistic enough. I recommend that the authors include at least one or two more split-type benchmarks in their experimental work (e.g., CIFAR-10 and/or mini-ImageNet). I would also recommend studying the effect of memory size on the behavior of the proposed algorithm using one of the 'harder' datasets.\n\nFinally, I think that the paper will benefit a lot by if the authors were to describe some intuitive interpretations of the main theoretical results of their work.\n\n\nMinor Issues:\n\n- Line 96: There is a \"Third\" left there (probably after re-writing the paragraph) that needs to be removed.\n- Line 134: In line 3 of Eq. (3), insert a large enough gap between the min and max operators so that their corresponding subscripts are clearly seperated.\n- Line 210: Regarding the expectation operator, you have used an italic E in the formula but a regular E in the text. In any case, I would recommend using a blackboard bold typeface for the expectation operator. \n\n\nQuestions:\n\n- What is the effect of memory size on the BCL game?\n- Do you think BCL can be expanded for task-free CL?\n- Do we really need to prioritize tasks? Why not use the expectation of all task losses (which will be bounded given the conditions stated in the paper)?\n- Why define the loss function to be a weighted sum of H, where H is a weighted sum of task losses? Why not just use a weighted sum of task losses directly?\n- What is the intuition behind the selected order of play (Δx first, θ afterwards)? \n- Have you experimentally tried using the reverse order of play? I am curious if the results would be different as Theorem 2 only discusses the selected order of play but not the reverse.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FlFHemFgSG1", "reviewer": "Reviewer_dfQR", "summary": "The paper proposed a novel idea of formalizing the generality-forgetting trade-off in continual learning via a two-player sequential game. Theoretical and empirical results are represented to support the authors' claims. The main contribution of the paper is to formally discuss the forgetting and generalization trade-off theoretically.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of formalizing continual learning problem as a two-player game is novel and interesting. The theorems are interesting, but the flow of the paper is not quite clear, which makes the details hard to fully understand. The paper has put a lot of emphasis on its theoretical contribution, however, it is not quite clear in their experiments. Showing the average prediction accuracy in benchmark datasets, although necessary, could not fully support the theoretical side. When talking about trade-off theoretically, authors should think about ways to reveal it experimentally also.  For example, several metrics such as average forgetting and learning curve area [1] could be leveraged to depict the actual trade-off between forgetting and generalization. Also, in Table 2, showing more recent methods instead of earlier methods that are not particularly designed for incremental class learning might be a better choice.\n\n\n[1] EFFICIENT LIFELONG LEARNING WITH A-GEM, Chaudhry et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the balance between stability-plasticity tradeoff as a two player sequential game and​​ show theoretically that a balance point between the two players exists for each task and that this point is stable. This leads to a theoretically justified approach, balanced continual learning (BCL), which has empirical performance that is comparable to or better than the state of the art CL approaches.

Overall, the paper addresses an important question, provides a new perspective of formulating the cost, provides theoretical justification as well as empirical evaluation for the proposed method. The reviewers raised some points that were satisfactorily addressed in the rebuttal, which helped improved their understanding of where the paper stands. 

Decision: Accept (Poster)