Continual evaluation for lifelong learning: Identifying the stability gap
Authors: Matthias De Lange
Gido M van de Ven
Tinne Tuytelaars
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Continual learning
lifelong learning
incremental learning
evaluation metrics

Abstract: Time-dependent data-generating distributions have proven to be difficult for gradient-based training of neural networks, as the greedy updates result in catastrophic forgetting of previously learned knowledge. Despite the progress in the field of continual learning to overcome this forgetting, we show that a set of common state-of-the-art methods still suffers from substantial forgetting upon starting to learn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery. We refer to this intriguing but potentially problematic phenomenon as the stability gap. The stability gap had likely remained under the radar due to standard practice in the field of evaluating continual learning models only after each task. Instead, we establish a framework for continual evaluation that uses per-iteration evaluation and we define a new set of metrics to quantify worst-case performance. Empirically we show that experience replay, constraint-based replay, knowledge-distillation, and parameter regularization methods are all prone to the stability gap; and that the stability gap can be observed in class-, task-, and domain-incremental learning benchmarks. Additionally, a controlled experiment shows that the stability gap increases when tasks are more dissimilar. Finally, by disentangling gradients into plasticity and stability components, we propose a conceptual explanation for the stability gap.

Published as a conference paper at ICLR 2023
CONTINUAL EVALUATION FOR LIFELONG LEARNING: IDENTIFYING THE STABILITY GAP
Matthias De Lange, Gido M. van de Ven & Tinne Tuytelaars'
KU Leuven
ABSTRACT
lime-dependent data-generating distributions have proven to be difficult for gradient-based training of neural networks, the greedy updates result in catas- trophic forgetting of previously learned knowledge. Despite the progress in the field of continual learning overcome this forgetting. we show that a set of common state-of-the-ar methods still suffers from substantial forgetting upon starting to learn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery. We refer to this intriguing but potentially problematic phenomenon as the stability gap. The stability gap had likely remained under the radar due to standard practice in the field of evaluating continual learning models only after each task. Instead, we establish framework for continual evaluation that uses per-iteration evaluation and define new set of metrics to quantify worst-case performance. Empirically we show that experience replay, constraint- based replay, Knowledge-distillaion and parameter regularization methods are all prone to the stability gap; and that the stability gap can be observed in class-, task,, and domain- gemmain-nncremetall learning benchmarks. Additionally, a controlled experiment shows that the stability gap increases when tasks are more dissimilar. Finally, by disentangling gradients into plasticity and stability components, we propose a conceptual explanation for the stability gap.
INTRODUCTION
The fast convergence in gradient- optimization has resulted in many successes with highly overparameterized neural networks (Krizhevsky et al.) 2012: Mnih et al.) 2013 Devlin et al. 2018). In the standard training paradigm, these results are conditional on having static data-generating distri- bution. However, when non-stationarity is introduced by a time-varying data-generating distribution, the gradient-based updates greedily overwrite the parameters of the previous solution. This results in catastrophic forgetting (French) 1999) and is one of the main hurdles in continual or lifelong learning.
Continual learning often presented as aspiring to learn the way humans learn, accumulating instead of substituting knowledge. To this end, many works have since focused alleviating catastrophic forgetting with promising results, indicating such learning behavior might be tractable for artificial neural networks (De Lange al., [2021] Parisi et al. 2019). In contrast, this work surprisingly identifies significant forgetting is still present on task transitions for standard state-of- the-art methods based on experience replay, constraint-basee replay, knowledge distillation, and parameter regularization, although the observed forgetting is transient and followed by recovery phase. We refer this phenomenon as the stability gap.
Contributions in this work are along three main lines, with code publicly available 1 First, we define a framework for continual evaluation that evaluates the learner after each update. This framework is designed to enable monitoring of the worst-case performance of continual learners from the perspective of agents that acquire knowledge over their lifetime. For this we propose novel principled metrics such as the minimum and worst-case accuracy (min-ACC and WC-ACC).
Second, we conduct an empirical study with the continual evaluation framework, which leads to identifying the stability gap, illustrated in Figure in variety of methods and settings. An
"Contact at: www.:/tttaaaa..e... htt.aaaee..naaaa....e..eeeeeeeeo tinee, twytelytteleee laars] Sovee. be htt.///rtt////aaaaaocccc//tttttt

---

Published as a conference paper at ICLR 2023
ablation study evaluation frequency indicates continual evaluation is necessary means to surface the stability gap, explaining why this phenomenon had remained unidentified so far. Additionally, we find that the stability gap is significantly influenced by the degree of similarity of consecutive tasks in the data stream.
Third, propose conceptual analysis to help explain the stability gap, by disentangling the gradients based on plasticity and stability. We do this for several methods: Experience Replay (Chaudhry et al. 2019b), GEM (Lopez-Paz Ranzato 2017). EWC (Kirkpatrick et al. (017). SI (Zenke et al. 2017). and LwF (Li & Hoiem (017)) Additional experiments with gradient analysis provide supporting evidence for the hypothesis.
Implications of the stability gap. (i) Continual evaluation is important, especially for safety-critical applications, as representative continual learning methods falter in maintaining robust performance during the learning process. (ii) There is a risk that sudden distribution shifts may be exploited by adversaries that can control the data stream to momentarily but substantially decrease performance. (īii) Besides these practical implications, the stability gap itself is scientifically intriguing phe- nomenon that inspires further research. For example, the stability gap suggests current continual learning methods might exhibit fundamentally different learning dynamics from the human brain.
The stability gap: substantial forgetting followed by recovery upon learning new tasks in sfate-of-the-art continual learning methods. Continual evaluation at every iteration (orange curve) reveals the stability gap, remaining unidentified with standard task-oriented evaluation (red dia- monds). Shown is the accuracy on the first task, when a network using Experience Replay sequentially learns the first five tasks of iasseincrementll Split- Split-Minimmagent.. More details in Figure
Figure 1:
Task Accuracy
60
40
20
200
400
600
800
1000
iterations
PRELIMINARIES ON CONTINUAL LEARNING
The continual or lifelong learning classification objective is learn function f X Y with parameters 8. mapping the input space X' to the output space V. from a non-stationary data stream S ((x,)) y); (x, y) ... (x.y ), where data tuple (x X Y),, is sampled from data- generating distribution D which depends on time f. While standard machine learning assumes a static data-generating distribution, continual learning introduces the dependency
time variable This time-dependency introduces trade-off between adaptation to the current data-generating distribution and retaining knowledge acquired from previous ones, also referred as the stability-plasticin trade-off (Grossberg 1982).
Tasks. The data stream often assumed be divided in locally stationary distributions, called tasks. We introduce discrete task identifier l to indicate the k-th task Tj with locally stationary data- generating distribution Dz. Additionally, time variable assumed discrete, indicating the overall iteration number in the stream, and f1211 indicates the overall iteration number at the end of task Tk.
Learning continually. During the training phase, a learner continuously updates f based on new tuples (x. y); from the data stream (De Lange & Tuytelaars 2021). Optimization follows empirical risk minimization over the observed training sets DesR, as the learner has no direct access data- generating distributions D The negative log-likelihood objective we would ideally optimize while leamingg task Ti
min C. - E Ex,Y)DD [y_ log f (x: 0)]
A key challenge for continual learning is to estimate this objective having only the current task's training data Di available, while using limited additional compute and memory resources.
2

---

Published as a conference paper at ICLR 2023
3 A FRAMEWORK FOR CONTINUAL EVALUATION
The learner continually updating f has a counterpart, the continual evaluator, that tracks the perfor- mance of f over time. Not only the data observed by the learner in data stream 8 may change over time, but also the distribution on which the learner is evaluated. This is reflected in a time dependency on the evaluator's data stream Sy (De Lange & Tuytelaars 2021). The evaluator is configured by three main factors: i) the evaluation periodicity Peval determining how often f is evaluated; ii) the process to construct Sg; iii) the set of evaluation metrics.
Evaluation periodicity. The typical approach to evaluation in continual learning is task-based evaluation: the performance of model f is evaluated only after finishing training on new task. As an alternative, here we define continual evaluation as evaluating the performance of f every Peval training iterations. For the most fine- fine-grained evaluation Peval is set to 1, the setting used in the following unless mentioned otherwise. The effect of increasing Peval can be observed in Figure 2 and is studied in more detail in Appendix B
Defining the evaluation stream. The desirable behavior of the model can be disentangled in performance on a set of evaluation tasks. We define a set of N evaluation tasks TE = {Eo, ..., Ex }. where each evaluation task E; has data set D sampled from data-generating distribution Dr The set of evaluation tasks can be extended at any point in time: TE t TE U {1x+1]. Following literature (van de Ven et al., 2020), we assume a new evaluation task Eg is added upon each encountered training task 7 in S. Typically, the evaluation tasks are chosen to match the tasks in the training stream S, where the evaluation data D E. may consist a subset of the observed task's training data DR. or more commonly, consists of a held-out evaluation set to test the generalization performance with D3 n DEA O. As the set of evaluation tasks expands over the training stream, we provide relaxations regarding computational feasibility in Appendix B
Continual evaluation metries The third factor of the evaluator is the set of evaluation metrics used to assess the performance of the learner. To date, metrics have been defined mainly assuming evaluation on task transitions and focusing on the final performance of the learner. We advocate that continual worst-case performance is important for continual learners in the real world. Therefore, we propose new metrics to quantify worst-case performance (WC-ACC, min-ACC, WF") and metrics that are also applicable to task-agnostic data streams (WF", WP"",, We focus on classification metrics, with accuracy (the percentage of correctly classified instances) considered the main performance metric. Using ft to indicate the version of the model after the t-th overall training iteration, the accuracy of evaluation task Fgg at this iteration is denoted as A Ek, ft). We provide taxonomy of metrics based on plasticity and stability in the following.
3.1 STABILITY-BSEED METRICS
To measure stability of the learner, we aim to quantify how well knowledge from previously observed tasks Tex is preserved while learning new task Ta. Average forgetting (FORG) (Chaudhry et al.) 2018) averages the accuracy difference for the most recent model Styzzi compared to Jt just after
learning 7; with evaluation task F,, and is defined as -ー Σ. k1 A(E,, Jtmay) A A S( ). Large forgetting indicates the phenomenon of forgeetnniic while negative forgetting indicates knowledge transfer from new previous tasks.
For worst-case performance it is desirable to have an absolute measure on previous task performance. We define the average minimum accuracy (min-ACC) current training task Tj as the average absolute minimum accuracy over previous evaluation tasks E, after they have been learned:
min-ACCT, k: - Σ min fr), Vtpa < n St
(2)
where the iteration number ranges from after the task is learned until current iteration t. This gives a worst-case measure of how well knowledge is preserved in previously observed tasks T In the following, report the min-ACC for the last task, omitting the dependency on T2 for brevity.
Furthermore, we introduce a more general stability metric that does not assume bask- data stream, and is therefore also applicable to data incremental learning. We define Windowed-
3

---

Published as a conference paper at ICLR 2023
Forgetting (WF") based window of w consecutive accuracy evaluations averaged over the evaluation set Tg. For a single evaluation task E, the maximal accuracy decrease in the window wi and the task-specific Windowed- Hundowed-Forgettig WF,, are defined at current iteration as
A E.E.
max (A(F;; fm. - A(F fn)), Vm, E +1,1 mn
(3)
WF max A Vn S
Averaging the metric over all N evaluation tasks results in single metric WF." = -NN-E> WF,,,, As identifies the maximal observed performance drop in the window, it is considered a worst-case metric. Evaluators that can afford linear space complexity can consider the full history at iteration / with WF'. To account for both fast forgetting and forgetting on a larger scale with constant memory, we instantiate WF and WF100.
3.2 PLASTICITY-BANE BASED METRICS
Plasticity refers in this work to the learner's ability to acquire new knowledge from the current data-generating distribution Dx. The current task accuracy measures plasticity A (Ea, f+) with
t E | 11. ]. Other metrics proposed in literature are the few-shot measure Learning Curve Area (Chaudhry al. (019a) and zero-shot Forward Transfer (Lopez-Paz Ranzato, 2017). Previous metrics depend on the current learning task Ti and are therefore not directly applicable to task- agnostic data streams. As the counterpart for WF". we introduce the more generally applicable Windowed Plasticity (WP"), defined over all N evaluation tasks by the maximal accuracy increase
A w.+
in window of size Appendix defines A (Eq. 10) A,E, but with constraint m >> n.
I.E,
WP Σ
max A ME n R,E,
Vn Sn
3.3 STABILITY-PLASTICIII TRADE-OFF BASED METRICS
The main target in continual learning is to find a balance between learning new tasks Tp and retaining knowledge from previous tasks Tek: commonly referred to as the stability-lassiiiity trade- off (Grossberg) 1982). Stability-plasticity trade-off metrics provide single metric to quantify this balance. The standard metric for continual learning is the Average Accuracy (ACC) that after learning task Tk averages performance over all evaluation tasks: ACCT, E A(E;, St Stpry) provides a measure of trade-off by including the accuracy of both current evaluation task Fx (plasticity) and all previous evaluation tasks Ecc (stability). The ACC is measured only at the final model ftrr, and is negligent of the performance between task transitions. Therefore, we propose the Worst-case Accuracy (WC-ACC) as the trade-off between the accuracy on iteration t of current task Th and the worst-case metric min-ACCI; (see Eq. (2) for previous tasks:
WC-ACC, = A(FE, + - min-ACCT,
This metric quantifies per iteration the minimal accuracy previous tasks retain after being Iearned, and the accuracy the current task. WC-ACC gives lower bound guarantee on ACC that established over iterations. Evaluating after learning task TR. we conclude the following lower bound:
WC-ACC (Ir ACCT,
4 IDENTIFYING THE STABILITY GAP WITH CONTINUAL EVALUATION
We start by conducting empirical study confined to Experience Replay (ER) (Chaudhry et al. 2019b), as has been shown to prevail over regularization- domain-incrementall learning (van de Ven et al. 2022). ER stores small subset of samples in memory buffer, which are revisited later on by sampling aini-batches that are partly new data, partly memory data. Due to its simplicity and effectiveness against catastrophic forgetting, it has been widely adopted in literature (Rebuffi et al. 2017: Chaudhry et al., 2019b De Lange & Tuytelaars 2021 Chaudhry al. 2018)
methods, especially for class- and
4

---

Published as a conference paper at ICLR 2023
Figure Average accuracy (mean=SD over 5 seeds) on the first task when using ER. Experiments are class-incremental (a-c) or nomain-incremmntaa (d). Evaluation periodicity ranges from standard evalu- ation on task transitions (red diamonds) to per-iteration continual evaluation (Peval 1). Per-iteration evaluation reveals sharp, transient drops in performance when learning new tasks: the stability gup. Vertical lines indicate task transitions; horizontal lines the min-ACC averaged over seeds.
Pevll
Peul
100
Task-based
(a) Split-MNIST
(b) Split-CIFARI0
100
100
Accuracy
Aocurry
95-
0
500
1000
1500
2000
500
1000
1500
2000
iterations
ilerations
(c) Mait-Minilmagenet
(d) Mini- DomainNet
Accuracy
Accuracy
20-
0
0-
1000
3000
200
400
iterations
iterations
Datasets. For experiments on class-incremental learning we use three standard datasets: MNIST (LeCun & Cortes 2010) consists of grayscale handwritten digits, CIFARIO (Kizhevsky
2009) contains images from range of vehicles and animals, and Minilmagenet (Vinyals
et al. 2016) is subset of Imagenet (Russakovsky et al. 2015). Split-MNIST, Split-CIFAR10,
and Split-Minilmagenet are defined by respectively splitting the data in 5, and 20 tasks based on the 10, 10, and 100 classes. For domain-incrementtal learning
consider drastic domain changes Mini-DomainNet (Zhou et al. 2021), a scaled-down subset of 126 classes of DomainNet (Peng al. 2019) with over 90k images, considering domains: clipart, painting, real, sketch.
Setup. We employ continual evaluation with evaluation periodicity in range Peval
{1, 10, 102, 103) and subset size 1k per evaluation task, based on our feasibility analysis in Ap- pendix B For reference with literature, task- transtrannttonn based metries ACC and FORG follow standard evaluation with the entire test set. Split-MNIST uses an MLP with 2 hidden layers of 400 units. Split- CIFAR1O, Split-Minilmagenet and Net use slim version of Resnet18 8 (Lopez-Paz & Ranzato 2017). SGD optimization is used with 0.9 momentum. To make sure our worst-case analysis applies the best-case configuration for ER, we run gridsearch over different hyperparameters and select the entry with the highest subility-plasticii trade-off metric ACC on the held-out evaluation data (Lopez-Paz & Ranzato 2017). Details for all experiments can be found in Appendix Cand code
is available here: https: github..... com wtt.coot/mmttttcce wwwtta/ttaaaccc Cttp:/ooniinnn coooniinual. Eva vavallln luat ion
Qualitative analysis with continual evaluation. Figure illustrates the first task accuracy curves for ER over the 4 benchmarks. The red markers on task transitions indicate the standard evaluation scheme in continual learning. We find that continual evaluation (Peval 1) reveals significant temporary forgetting between task transitions, both for the class- and domain-incrementte bench- marks. After the significant performance drops, partial recovery follows, making the task -k-nssantiionn based metrics such as ACC and FORG bad worst-case performance estimators. We refer to this phenomenon of transient, significant forgetting when learning new tasks the stability gap.
Quantifying worst-case performance. The qualitative analysis allows observation of the stability gap from the accuracy curves over time. However, performance analysis also requires a quantifying measure for the worst-case performance. For class-incremental Split-Minilmagentt, report results
5

---

Published as a conference paper at ICLR 2023
for continual evaluation metrics in Table from which two important observations be made. First, we confirm that the standard metric ACC measured on task transitions is negligent to the stability gap, whereas worst-case performance metrics min-ACC and WC-ACC clearly indicate the sheer performance loss. Second, our metrics indicate similarly to Figure 2 that fine-grained evaluation periodicity is crucial to identify the stability gap. Table shows that the sharp performance drops are phased out as min-ACC and WC-ACC both increase with evaluation periodicity Peval and large-scale forgetting WF decreases, Table 3 in Appendix reports similar results on the other benchmarks. Additionally, Appendix reports results for the stability gap for various ER buffer sizes, speech modality experiment, and for online continual learning.
Table Our newly proposed continual evaluation metrics on plass-incremental Split-Minllmageeet for range of evaluation periodicities Peval- Standard continual learning metrics 32.9 (ACC) and 32.3 # 1.0 (FORG). Results over seeds reported as mean (=DD).
Trade-off
Stability
Plasticity
Peval
WC-ACC
min-ACC
WF'0
WF
Wp
Wp
10"
0.2
56.6 +
1.
4..
67.6 0.7
10a
5.0 0.4
1.4
60.9 + 0.5
6.6
6.3
68.1 0.4
10a
6.7
3.1
58.8 1.8.0.. 0.6
0.7
0..0
6..2
10a
7.1111
3.611.
57.7
0.3
0.81
0.9
4.1 THE INFLUENCE OF TASK SIMILARITY ON THE STABILITY GAP
When the domain shift between subsequent tasks in the stream increases, the interference of the objectives is expected to lead to higher forgetting (FORG) and hence lower average accuracy (ACC) over tasks. However, the effect on the stability gap remains unclear.
Experiment setup. We set up controlled domain-incremmnnal learning experiment with Notated-MNIST, where each task constitutes the entire MNIST dataset with increasing but fixed rotation transform of e degrees. The increasing task-specific rotation results controllable domain shifts over tasks in the input space. To avoid ambiguity between digits 6 and 9, constrain the total rotation to not exceed 180 We consider a cumulative fixed rotation by á degrees, leading to do. (Øo + Ø), and (øo 2Φ) for the three tasks, with do to -90 for the initial task's rotation. To increase the task dissimilarity, we consider a range of increasing relative rotations [10", 30", 60%, 60", We study ER with buffer capacity of 1k samples.
Results. Table Table@ contirms the decrease in ACC as the distribution shift between tasks increases. Between the easiest (ø- 10) and hardest (Φ 80) sequence, ACC declines with 6.6%. However, the effect on the stability gap is substantially larger as the min-ACC drops from 94.3% to 69.2%, 25.1% decrease. Similarly, FORG between Φ 60 and ø S0 indicates only 1.1% more forgetting, whereas the WF10 shows decline of 5.0%. This indicates both that i) the standard metrics fail to capture the effects of the stability gap, and ii) the stability gap increases significantly for larger distribution shifts, As additional confirmation, the accuracy curves for the first task in Figure 3 show qualitatively that the stability gap increases substantially with increasing rotations.
Table 2: Task similarit results for ER in Rotated-MNIST. Three subsequent tasks are constructed by rotating MNIST images by degrees per task. We decrease task similarity by increasing the rotation from = 10 to o = 80 Results over 5 seeds reported as mean (=SD).
Trade-off
Stability
Plasticity
ACC
WC-ACC
min-ACC
FORG
WF
100
W
Wp
10" 95.3 96.6
94.3
0.7.0.3
3.5
3.6
20.9
3..7
30
035
91.840.4
8..3
4.641.
5.1
6.0
28.2+11
41.3
60
9..6=0.6
8..5
7..3
8.720.8
12.8114
13.5
50.8
67.240.9
80
9..0=0.5
78.4
69.2
9.8
17.8+20
18.6
57.3
7..6

---

Published as a conference paper at ICLR 2023
Figure Controlled task similarity experiment with task rotation angle o in Rotated-MNIST. Accuracy curves for ER in the first task are reported as mean (#SD) over 5 seeds. The two vertical lines indicate task transitions. Horizontal lines indicate the min-ACC averaged over seeds.
100
90
Acrtarcy
80
80
00
30
Φ 10
601
50
100
200
300
400
300
600
700
itrations
5 CONCEPTUAL ANALYSIS OF THE STABILITY GAP
To establish conceptual grounding for the stability gap phenomenon, disentangle the continual learning gradients of the objective L in a-weighed plasticity and stability gradients
VL = a Eplastiiity (1 Latabiliity
(8)
In gradient- based optimization of model f. V Cllsttiiyy aims improve performance on the current task and V Cstabllityy maintains performance on past tasks. In the viewpoint of the plastiity trade-off (Grossberg 1982), continual learning methods aim to balance both gradient terms. However, due to task-oriented evaluation and lack of continual evaluation, progress in this balance has only been verified on the task transitions.
Finetuning optimizes solely for the current task Tg and is not concerned with stability, with || Cabilill O. The lack of a stability term results in greedy updates for the currently observed distribution shifts. Our continual evaluation framework indicates that severe forgetting of past tasks occurs already during the first few training iterations on new task (see Appendix D.3).
data, resulting in forgetting
Experience replay (ER) learns concurrently from data of the new task and a subset of previous task data sampled from experience buffer M of size M. The loss term Laabiliy is obtained by revisiting previous task samples in M. In contrast to finetuning, this results in a stability gradient Laaaiility designed to prevent catastrophic forgetting.
Forgetting. Our further analysis of the Liaability gradient dynamics indicate a low gradient norm during the early phase of training on new task. We first consider data stream with two subsequent tasks. When training starts on T2. f presumably has converged for the first task Ti. resulting in [[V x O. Therefore, directly after the task transition, we indeed have //V the replayed samples are exclusively from Ti. Generalizing this to longer task sequences requires not only nearly zero gradients for the previous task, but for all previous task data in the entire replay buffer M. This has been empirically confirmed by Verwimp et al. (202I) indicating ER to consistently converge to near-zero gradients for M. We demonstrate similar findings for Split-MNIST in Figure
0 because
(ee)) ((eh)). Due to the imbalance of the stability and plasticity gradients, the plasticity gradient will dominate and potentially result in forgetting. Our empirical findings indeed confirm for ER that significant drops in accuracy occur directly after the task transition, shown in Figure
Recovery. Upon the initial steps of learning a new task with the greedy V Eplasticity updates, parameters change with nearly no gradient signal from replay buffer M. These updates in turn lead to increasing gradient norm of V Luabiliyy allowing the samples in M to be re-learned.
In contrast to the prior belief of V Luability maintaining prior knowledge, we find that stability is at least partially preserved by means of relearning, leading to the recovery of prior knowledge. This is confirmed by experiments in Section on five class- and domain-incremental benchmarks.
7

---

Published as a conference paper at ICLR 2023
5.1 FINDING A STABILITY GAP IN OTHER CONTINUAL LEARNING METHODS
Our analysis of disentangling the gradients based on plasticity and stability also applies to other representative methods continual learning. Additionally, we provide empirical evidence for the stability gap in these methods, focusing on constraint-baser replay method GEM (Lopez-Paz)
& Ranzato 2017). knowledge-distillttie method LwF (Li & Hoiem 2017). and parameter
regularization methods EWC (Kirkpatrick et al. 2017) and SI (Zenke et al. 2017). The details of the experiments are Appendix C
Figure 4: GEM and ER accuracy curves (a-d) on class-incremental Split-MNIST for the first four tasks, and the per-iteration 1.2-norms of current Suabiliy (e-h). Results reported as mean (+SD) over 5 seeds, with horizontal lines representing average min-ACC. Vertical lines indicate the start of a new task. Note that the x-axis scale varies over (a-d) and (e-h) are zooms of the first 50 iterations.
(a) Ti
(b) T2
(c) Ta
T4
100
Armracy
04-
00-
ET
20-
GEM
500
10:00
1500
2000
500
1800
15540
2000
1000
1960
2000
1500
1500
2000
2000
Stenolinns
iletalions
ileradãone
iteealiocas
(c) zoom Ti
(f) zoom T2
g zoom T3
(b) zoom T4
500
0300
720
530
540
950
300
1200
1020
1020
1420
11490
14.00
1400
1.00
1000
3003
1900
1930
1940
Heeraioons
itscutooss
sterations
Gradient-constrained replay. Gradient Episodic memory (GEM) (Lopez-Pan Paz & Ranzato 2017 exploits a memory buffer M similar to ER, divided over K equally sized task buffers Mr. However, instead of directly optimizing the objective for the samples in M, their task-specific gradients Ok = vL(M)) are used to compose a set of constraints. The constraints (9t, 9n) 2 0. Vn < attempt to prevent loss increase on the k previous tasks, with 9r = V plastiicity the gradient of the current observed sample (x,y): in task Tg. Current gradient 9: is projected to the closest gradient satisfying the constraint set, obtained by Quadratic Programming. We reformulate the projection to obtain the conditional stability gradient:
0,
if (gt, gn) 2 0, Vn < k
V Estability
(9)
( 9t. otherwise
As the GEM constraints explicitly attempt to prevent an increase in previous task losses, || Sstabilitt only zero if the current gradient update gi lies within the feasible region of the constraint set or is near-zero with only slightly violated constraints || gt R O. As in the dot-product constraints the sign determined solely based on the gradient angles, satisfying them is independent of the norm of L/V || on task transitions. This suggests that GEM might enable avoiding alleviating the stability gap.
However, empirically we find that also GEM is prone to the stability gap (Figure 4. Compared to ER, the stability gap of GEM is significantly larger, indicated by the large discrepancy in the horizontal lines representing average min-ACC. On the task transitions for MNIITM Figure 4(e-h) shows that GEM has significant Laahiilyy compared to ER (determined following Eq. [9). indicating large violations of the constraints. However, especially for task transitions T3 and Ta we observe the gradients to drop to near-zero magnitude, resulting in few updates mostly based on V Eplssticity
Distillation-baser methods (Li & Hoiem, 2017 Rannen al. 2017) prevent forgetting of previous task T by penalizing changes in the output distribution for samples in the current task x E Dr-
8

---

Published as a conference paper at ICLR 2023
This results in regularization objective Luaaiiay
(xx)) that distills knowledge via KL-divergence (Hinton et al. 2015) from the previous task model at TT the current model. Before the first update on a new task, the previous and current task models are identical, leading to
IIV Stabilill caused by perfectly matching distributions, which could lead a stability gap.
Parameter regularization methods (Kirkpatrick et al. 20177; Zenke et al. 2017; Aljundi et al.
2018 Kao al. (02T). also referred to as model-prior based methods, penalize changes in important parameters for previous tasks with regularization objective Littailiity = (0 - 0- S2 (0 0*), where x|0
parameter importance is defined by the penalty matrix R ||0 and weighed by the change of the model's ' parameters 9 w.r.t. the previous task solution 6 = 6 - For the first updates on a new
task, the parameters f are still close the previous task solution 0 resulting in V Sstbbittyl || Ri
and are even identical for the first update, resulting in Luabilly = 0.
Figure almp@reialllll confirms the stability gap for representative distillation-basse (LwF) and param- eter regularization methods (EWC, SI) in two settings: first, miskiincremental Split-MNIST, where each task is allocated a separate elassifier (Figure 5 (a-d);; second, Rotated-MNIST from Section (4. with lowest task similarity (Φ 80") between the tasks. Setup details are reported in Appendix C
Figure 5: Distillation-
(LwF) and parameter regularization methods (EWC, SI) accuracy curves on the first four tasks (a-d) of insk-incrementl Split-MNIST, with separate classifier head per task. Rotated-MNIST (ø = 80%) for the first
The same methods are considered for two tasks (e-f) out of three. Results reported as mean (+SD) over 5 seeds, with horizontal lines representing average min-ACC. Vertical lines indicate the start of a new task.
(a) Split-MNIST (Ti)
(b) Split-MNIST (T2)
(c) Split-MNIST (Ta)
(d) Split- (TA)
Accuneee
50-
Lx/F
85--
60-
EWC
5000
750
1010
201
000
900
2000
000
1000
000
900 Htrratons
900
1000
itetalions
Eterotiooe
Sterations
(c) Rotated-MNisg (Ti)
(I) Rotated-MNIST (T>)
80-
Acottaty
00-
20
200
(40
700
Besalins
CONCLUSION
This work proposed novel framework for continual evaluation with metrics that enable measuring worst-case performance and are applicable to task-agnostic data streams. Our evaluation framework identified shortcomings of the standard task-oriented evaluation protocol for continual learning, as we identified a striking and potentially problematic phenomenon: the stability gap. In our study on seven continual learning benchmarks. we showed that upon starting to learn new task, various state-of-the-at continual learning methods suffer from a significant loss in performance on previously learned tasks that, intriguingly, is often recovered later on. We found the stability gap increasing significantly as subsequent tasks are more dissimilar. To provide insight into what might be causing the stability gap, we formalized a conceptual analysis the gradients, disentangled into plasticity and stability terms. This led to consistent findings for experience replay, knowledge- distillation, and parameter regularization methods, but left some open questions for constraint-bart based replay. Interesting directions for future work include mechanisms to overcome the stability gap and connections to biological neural networks: when learning something new, does the brain suffer from transient forgetting as well?

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This project has received funding from the ERC project KeepOnLeeaming (reference number 101021347), the KU Leuven CI project Macchina, and the European Union's Horizon 2020 re- search and innovation program under the Marie Sklodowska-Curie grant agreement No 101067759.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, pp. 139-154, 2018.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. Proceedings NeurIPS 2019, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurIPS, pp. 11816 1 11825, 2019b.
Johannes Buchner. Synthetic speech commands: public dataset for single-word speech recogni- tion. Kaggle Dataset, 2017. URL http:: kaggle. com wtwaaattttttesaeeeeeeeeeeeeeeeee
syrthetic setpret---ppeeccccccoomm-----ddd dataset
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky,
New insights reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022. URL httpt:t::///o.....t. htttps://openreveee...t net/forum?
Id=N8MaByOOUTF
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, pp. 532-547,
2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a- a-GMM. In International Conference on Learning Representations, 2019a.
URL http:ttts///ppeeeeeeeee
net /aee/forre forum? id=Hkf2 id=Hkf2_scCFFX
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: 2019b.
Aristotelis Chrysakis and
Moens. Online continual learning from imbalanced data.
Proceedings of Machine Learning Research, 2020.
Andrea Cossu, Antonio Carta, Vincenzo Lomonaco, and Davide Bacciu. Continual learning for recurrent neural networks: An empirical evaluation. Neural Networks, 143:607-627, 2021.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. Proceedings of the JEEE/CVF International Conference on Computer 2021.
Vision, pp.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10.1100/TPAMM.221..3077..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8/0: 04.005 2018.
10

---

Published as a conference paper at ICLR 2023
Enrico Fini, Stéphane Lathuiliere, Enver Sangineto. Moin Nabi, and Elisa Ricci. Online continual learning under extreme memory constraints. In European Conference on Computer Vision, pp. 720-735. Springer. 2020.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-133) 135, 1999.
Stephen Grossberg. Studies of mind and brain: neural principles of learning, perception, development, cognition, and motor control. Boston studies in the philosophy science 70. Reidel, Dordrecht, 1982. ISBN 9027713598
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
Ta-Chu Kao, Kristopher Jensen, Gido van de Ven, Alberto Bernacchia, and Guillaume Hen- nequin. Natural continual learning: success is a journey, not (just) a destination, In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 28067-280799 Curran As- sociates, Inc., 2021. URL http::s//pprcceeeiinngg neerui
neurips wtt.iiss..cccpppeee hctppcccccppper/2///11ee..
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabssa-BBrwiinsh et al. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 20161 1835, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances Neural Information Processing Systems, volume 25, Curran Asso- ciates, Inc., 2012. URL https: htttss:/ http:///ccccccccccc//eeeesc wwwpriiinnnnnneuunsneeeeeeeeeeee neurips cc/ pttpppape/e///2///eeeeeeee oac.aaaaaaetbceeeeeeeeeeeeea-s-s
pdf
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann
lecun. www/exdb/ebe/eppooop /
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido M van de Ven, et al. Avalanche: an end-to-end library for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3600-3610, 2021.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in neural information processing systems, pp. 6467-6476, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue.
11

---

Published as a conference paper at ICLR 2023
E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035, Curran Associates, Inc., 2019. URL http:///ww/./////aaa.......aaa.a 9015
library..
pdf
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE International Conference on Computer Vision. pp. 1406-1415, 2019.
Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. ICCV, pp. 1320-1328, 2017.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):21 1-21111222222 2015.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 3xXiv:606.046771 2016.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, pp- 4548-4557. PMLR, 2018
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyumwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 9630-9633 2021.
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired inspired replay for continual learning with artificial neural networks. Nature Communications, 11:4069, 2020.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4:1 1-1185-1197, 2022.
Vinay Kumar Verma, Kevin Liang, Nikhil Mehta, Piyush Rai, and Lawrence Carin. Efficient feature transformations for discriminative and generative continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13865-13875, 2021.
Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9385-9394, October 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637-3645, 2016.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995. JMLR. org, 2017.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. JEEE Transactions Image Processing, 30:8008-800 2021.
12

---

Published as a conference paper at ICLR 2023
SUPPLEMENTAL MATERIAL
Supplemental materials include a discussion on the limitations and societal impact of this work (Appendix (A), an empirical study and discussion on computational feasibility for continual evaluation (Appendix [ the detailed reproducibility details for all experiments (Appendix C.. and additional empirical evidence in Appendix
A LIMITATIONS AND SOCIETAL IMPACT
Computational complexity of continual evaluation. Both time and space complexity are important factors for continual learning with constrained resources. The space complexity for continual evaluation retains linear increase with the number of tasks it as for standard evaluation. However, the time complexity changes from task-based periodicity to a per-iteration periodicity. The ratio of computational increase for a single task Tę can be defined as f.y - where f|T. f/TT-1l is the number of iterations for that task. Albeit the increase in time complexity, due to our empirical findings, we advocate the use of continual evaluation when possible for analysis of the learning behavior and especially for safety-critical applications. We refer to Appendix B to enable tractable continual evaluation.
Poal
Parameter isolation methods (De Lange et al. 2021) are not considered in our empirical study for the stability gap, typically fixed parts of the model are allocated to specific tasks (Rusu et al. 2016 Verma et al. 2021 Serra et al. 2018: Mallya & Lazebnik, (2018). The task identifier is typically assumed available for inference, allowing activate only the subnetwork for that task. This allows to prevent any changes and hence forgetting for observed tasks, albeit impeding backward transfer. Therefore, for methods where no changes in performance of learned tasks is allowed, stability gap cannot be present either.
Societal impact. A continual learning agent operational in the real world may lead to unpredictable results the agent may observe malicious data points, learn from highly biased data, or learn undesirable behavior that diverges from its original goals. Such evolution can remain undetected with sparse or even no evaluation of the learner. Additionally, the sudden distribution shifts may be exploited by adversaries to momentarily but significantly decrease performance. Therefore, we deem continual evaluation important for fine-grained monitoring of the agent's learning behavior.
B TRACTABLE CONTINUAL EVALUATION
The challenge in continual evaluation is twofold. First, the high evaluation frequency is computation- ally demanding. Second, the evaluator's set of locally stationary test distributions may expand with the number of tasks. Furthermore, discuss practical considerations for real- weal-world applications by using training data for evaluation and parallelism.
Evaluation periodicity. A first relaxation for feasible continual evaluation to increase the per-update evaluation periodicity Peval 1 larger values. We perform an analysis for Peval E [1,10, 1 1 on 4 benchmarks for Experience Replay (ER). We follow the full setup and datasets discussed Section Section4I that continual evaluation reveals catastrophic drops in performance after task tran- sitions. Table shoows for Aplit-Minilmagenee that these sharp drops are phased out min-ACC and WC-ACC both increase with evaluation periodicity Peval, and large-scale forgetting W00 decreases. This phenomenon is illustrated in Figure 2 where further increasing periodicity up to the standard evaluation task transitions becomes entirely negligent of these performance drops. Therefore, continual evaluation of our experiments in the main paper adopt Peval = 1 unless mentioned otherwise.
I and Appendix C Important for this analysis are our findings in
Additionally to the Split-Minilmagent MNIST. Split- Aplit-CIFAR10, 10, and Mini-DomainNet in Table 3 We find consistent conclusions as for Moilmmmiiilmggenet, i.e. the stability gap is unobserved for larger evaluation periodicity, especially notable for Peval E {100, 1000) which are indicated in bold. The subsample size of the evaluation sets is 1000 for the other experiments in this work.
ER results, additionally report the results for Split-
13

---

Published as a conference paper at ICLR 2023
We noticed that results in Split- Aplit-CIFARIO have large variance in min-ACC (and hence WC-ACC) for Peval = 100. As the Split-CIFAR10 accuracy is noisy over the iterations as indicated in Figure 2 we found in some rums for Peval 100 to completely miss the aceuracy drop to zero of T4 for 2 out of the 5 seeds. This dependency on initialization seed stresses the importance of smaller Peval to reduce dependency the exact point of evaluation.
Table 3: Full results Split-MNIST, Split-CIFARIO Split-Minilmagentt and Mini-DomainNNt in continual evaluation metrics for a range of evaluation periodicities Peval Results over 5 seeds reported as mean (+SD). The Spli--Minilmagenet subset of the full results reported in the main paper in Table 1
Trade-off
Stability
Plasticity
Assi
WC-ACC
min-ACC
WF'
WF
WP 10
Wp 100
MPNSS
7..7=3.5
73.0
18.4
21.2
84.9
95.5 1.5
10
81.3 3.9
77.4
16.4
17.2+111
94.5
102
92.3
91.3+0.2
4.6
6.0
9..1
97.2
10a
92.3 0.2
91.3
4.6
6.0
97.1 1.4
97.2
Spli--IFARRO
17.9=0.9
0.0
71.1
76.0
57.7
81.4
10
18.
0.1
71.01
1.74
7..
89.2 1.0.
102
17.4
2.0
6..
3.2
86.9 1.8
87.1
10s
17.0=111
0.4
66.0
1.7
86.0 1.9
16.1
Split Ipli-Meinilmagenet
4.1
0.5+0.2
56.6
64.6+11
49.3
0..6
5.0
1.4005
60.9
63.0
65.3
68.1
10a
6.7
3.1104
58.8
6..
6.70
67.2 1.8
10a
7.1111
3.6111
57.7
59.3
6..1
6.9
Mini-Domain
1
13.7=10
9.6+10
14
17.8
10.2
16.9
10
15.1
11.0117
14.8
15.7
1..9
2..0
10a
18
15.8113
10.0
10.0
20.61110
21.0
10a
18.9=1.8
16.5 1.9
10.0
10.0
21.0 1.6
21.0
Subsampling the evaluation sets. A second relaxation for feasible continual evaluation is to control the number of evaluation samples. The set of evaluation tasks TE often grows linearly over time. This is unavoidable if we want to provide data-based performance guarantees per learned task. Nonetheless, we can reduce the absolute computation time by limiting the number of samples per evaluation set. With Peval and uniformly subsampling each evaluation task's 's dataset D F, Table indicates for Split- Split-CIFAR 10, and Split-Minilmagenee that a sample size of 1k provides a good approximation for using the entire test set ('All".. A smaller sample size of 100 induces more noise in the performance estimate, resulting in larger deviations compared to the entire dataset, especially notable for WF and Wp!00.
Instead of the uniform sampling, alternative but more complex sampling scheme would also be possible to restrain the number of evaluation samples. For example by measuring task similarity (e.g. via task2vec (Achille et al. (2019) and merging test data of similar tasks in fixed capacity bins. A Although this would be applicable for real-world learners, our analysis we opt for the straightforward subsampling avoid confounding factors rooted in the task similarity procedure.
Exploiting training data. A disadvantage of extracting held-out evaluation data from the data stream is that this data cannot be used by the learner. Focusing on Split-Minilmagenet, Table shows the continual evaluation of ER with training subsets compared to the actual evaluation data, with similar ACC 31.5 0.5 for the train set and even slightly higher 32.7 1.3 for the test set. We find that worst-case stability metric min-ACC is very similar on both sets, and catastrophic forgetting in WF0 is significantly higher for the training data. This suggests it can serve as good (over)estimator to monitor the stability of the learner.
Parallellism. In real-world continual evaluation, parallelism can be exploited by separating the learner and evaluator in different processing units. The evaluator captures snapshot from the model
14

---

Published as a conference paper at ICLR 2023
Table 4: Continual evaluation metrics for different sample sizes of the evaluation sets and comparison to training set performance on 3 benchmarks with Peral = 1. "All" indicates the full task test sets. Results over 5 seeds reported mean (+SD). The 10 in bold indicates the setting we applied for the other experiments in this work.
Trade-off
Stability
Plasticity
Sample size
WC-ACC
min-ACC
WP'
WP
Wp
Wp100
Split-MNIST
Eval
1.4
72.3
20.8 1..2
95.0
10a
77.7
43
18.4
14 31.2
95.5 1.5
10a
75.1
39.4
25.0
27.8
98.0= 86
Train
10a
77.6
22.0 4..2
85.2
95.7
Split-CIFARII
Eval
All
18.1
2.0
73.0
77.8
81.5 3..
10a
17.8
0.0
75.7 71.3
82.0 2.72
10a
18.
0.0
4..
81.7
2..4
86.8
Train
10a
19.3
2.0 0.0
75.8
79.8
57.81228
81.8 2.5
Taiigeeiilmaageet
Eval
0.1
0.2
0.7
6.6
48.3
67.8
10a
4.1
0.5
56.6100
64
1..3
67.61
10a
3.9
0.3
60.8
71.6
51.5
74.3
Train
10a
5.4
0.5
73.8
7.a
50.4
78.0
for evaluation, while the learner continues the continual optimization process. In this scenario, the periodicity Peval is dependent on the evaluator's processing time.
C REPRODUCIBILITY DETAILS
Datasets and transforms. For class-incremental learning, Split-MNIST, Split-CIFARIO, and Split- Minilmagenet split their data in 5, 5, and 20 tasks based on their 10, 10, and 100 classes respectively. Each of data associated the training and test sets, to create training and evaluation stream of sequential tasks. The order of the class- groupings to create tasks is sequential, e.g. in Split-MNIST we use the digits {0, 1) for Ti, () for T2. up to {8, 9} for TT. MNIST (LeCun & Cortes 2010) consists of grayscale handwritten digits with (28 x 28) inputs. The dataset contains about 70k images from which 60k training and 10k test images. CIFAR10 (Krizhevsky
set of classes then constitutes a task. This split is performed for both
al. 2009) contains 50k training and 10k test images from a range of vehicles and animals with colored (32 x 32) inputs. Mini-Imaggenet (Vinyals et al.) 2016) is 100-class subset of Imagenet (Russakovsky al., 2015). with colored inputs resized to (84 x 84). Each class contains 600 images, from which 500 are used for training and 100 for testing. For domain-incremenntll learning consider drastic domain changes in
(Zhou et al. 20211. scaled-down subset of 126 classes of DomainNet (Peng et al.) 2019) with over 90k images, considering domains: clipart, painting, real, sketch. Inputs from all datasets are normalized and Mimii-Domiineet resizes the original DomainNet images with their smaller size to 96 using bilinear interpolation, followed by a center crop to fixed (3 96 96) inputs. For all datasets, all used transforms are deterministic.
Gridsearch procedure. All experiments for Split-MNIST and Split-IIFAR10 perform a gridsearch over hyperparameters, with each run averaged over initialization seeds. The result with highest ACC is selected as best entry, following (Lopez-Paz & Ranzato 2017). For computational feasibility in the larger benchmarks, Split-Minilmagene and Mini-DomainNet the gridsearch is performed over a single seed and the top entry is averaged over 5 initialization seeds. For all experiments, the learning rate 77 for the gradient-bart (0.1, 0.01, 0.001, 0.0001). ). A fixed batch size is used for all benchmarks, with 128 for the larger-scale Molit-Mimilmagenet and Sini-DomainNet. and 256 for the smaller Split-MNIST and Split-CIFARIO Sethod-specific and other continual learning hyperparameters discussed below.
updates is considered as hyperparameter
the set 17 E
15

---

Published as a conference paper at ICLR 2023
Continual evaluation setup. We employ continual evaluation with Peval = 1 and subset size Ik per evaluation task, based on our feasibility analysis in Appendix [B] For reference with literature, task-transition based metrics ACC and FORG follow standard evaluation with the entire test set.
Architectures and optimization. Split-MNIST uses an MLP with 2 hidden layers of 400 units. Split-CIFAR10, Salit-Minilmagenet and
use slim version of Resnet18 (Lopez-Par Paz & Ranzato 2017 De Lange& & Tuytelaars) 2021). SGD optimization is used with 0.9 momentum. Split-MNIS T, Split-CIFARIO TO and Split-Minilmagened are configured for 10 epochs per task, which roughly equals to 500, 400,and 200 iterations per task with the defined batch sizes. Mini--oomiinNtt has 300 training iterations per task to balance compute equally over the tasks.
Results for identifying the stability gap in Section ( are obtained using Experience Replay (ER) where we consider the loss weighing hyperparameter a E {0.1, 0.3, 0.5, 0.9) as in Eq. 8 The above defined batch size indicates the number of new samples in the mini-batch. Additionally another batch of the same size is sampled from the replay memory, which is concatenated to the new data, following (Chaudhry et 2019b] Verwimp et al.) 2021). Class-based reservoir sampling is used as in (De Lange & Tuytelaars| (021) to determine which samples are selected for storage in the buffer M with tixed total capacity of [M] samples. We indicate the selected hyperparameters (17, a, (M) per dataset here: Split-MNIST (0.01, 0.3, 2. 109), Split-CIFARIO (0.1, 0.7, 10*), Split-Minilaagnett (0.1, 0.5, 10*), Mini-DomainNet (0.1, 0.3, 109).
Results in Section 5.1 use the ER results obtained from Section 4 intro). GEM (Lopez-Paz &
Ranzato 2017) uses the author's standard bias parameter of 0.5, with best learning rates 1) 0.01 and = 0.001 for Split-MNIST and Split-IIARR10 respectively. The task-incremental Split-MNIST setup is based on (van de Ven al. 2022) with Adam optimizer, batch size 128 and r) = 0.001, but with 200 iterations per task. As it task-incremental, each task is allocated a separate classifier head and at inference, the corresponding classifier used for given sample. Other details remain the same as our class-incremental Split-MNIST setup. The best regularization strength À is chosen based on highest ACC, for SI E {0.1, 10, 100, 1000), for EWC X E {105, 10 10 107), for LwF X E [1, 2, and distillation temperature (0.5, 1,
are based on the best ACC results for ER reported in Section 4 see above for details. In the experiment for different subsample sizes, the evaluation and training subsets are obtained from the entire set through a uniform sampling per evaluation.
Results Appendix B
Used codebases and equipment. The experiments were based the Avalanche frame- work (Lomonaco et al. 2021) in Pytorch (Paszke et al., 2019). All results were performed on compute cluster with range of NVIDIA GPU's.
Definition of maximal delta increase A A We define in the main paper the maximal delta
decrease A Wi,- t. and due to space constraints report the similar definition of maximal delta decrease F.E. here in Appendix as:
w max (())) - A(E,, fw)), Vm, n E -- - w + t] F.
(10)
m>n
10.- Note that the equation is identical to L. (Eq. 3 but replacing the constraint m < n with m >> n.
D ADDITIONAL RESULTS
D.1 THE STABILITY GAP IN ONLINE CONTINUAL LEARNING WITH ER
The main results in the paper are based on 10 epochs per task for the class-incremental benchmarks. In online continual learning, or streaming continual learning, each sample in the stream is only observed once (Shim et al. 2222 Fini et al. 2000 Chrysakis & Moens, 2020 De Lange & Tuytelaars 2021] Caccia et 2022). We show for Spli-MNIST and Split-CIFAR in Figure that the stability gap persists for ER in the online continual learning setup.
16

---

Published as a conference paper at ICLR 2023
Additionally, Table shows that our proposed metrics successfully identify the stability gaps in the online continual learning streams. For example in Split-MNIST, the WF indicates the sheer performance drops of nearly 40% whereas the standard task-based FORG only indicates 5% forgetting. Further, in Split-CIFAR 10 an ACC is maintained of 41.8%, whereas the min-ACC shows that all tasks drop at least once in the stream.
Figure 6: Online Continual Learning only samples each instance from the stream once, resembling streaming setup. Reports ER aceuracy curves for the first four tasks on Split-MNIST (a-d) and Split- CIFARIO (e-h), as mean (+SD) over 5 seeds, with horizontal lines representing average min-ACC. Vertical lines indicate the start new task.
(a) Split- (T:)
(b) Split- SpNit- (T2)
(c) Split-MNIST (Ta)
Split-MNIST (T.)
100
Acrnary
00
70
ER
50
100
153
200
50
114)
150
2000
100
150
160
1.50
500
220
:tenations
iterations
iterations
iteentiooss
(c) Split-CTFARIO (T1)
Split-CFFAROO (T2)
(g) Split- Split-CIFARIO (13)
(h) CIFARIC (7:)
100
ER
Accurayy
60-
41-
300)
1000
1200
2000
300]
1050
1060
2000
1000
1250
1500
1730
2000
1230
1400
1000
1900
2000
Sterations
itcrataoes
iterations
Sterations
Table Online Continual Learning results for ER. Results are reported mean (=SD) over seeds.
Trade-off
Stability
Plasticity
ACC
WC-ACC
min-ACC
FORG
WF
WF100
Wp
Wp100
Split- ONIt-MNIST
91.610.8 0.8
58.9
50.8188 0.0
5.3 55.8 35.813.
1..9
4..
86.7
2.0
95.8
Split- Opli-CCIFARIO
13.0
18.2 1.77
71.5
78.5
55.9 1.7
81.4288
D.2 THE STABILITY GAP FOR VARIOUS ER MEMORY SIZES
In the main paper, ER is considered for a single fixed memory size of M samples. Figure 7 shows the class-incremental Split-MNIST results for range of memory sizes M {500, 1000, 2000, | where |S|| stores all seen samples in the stream. The benchmark setup and ER hyperparameters are identical to Section Three key observations can be made. First, the stability gap persists for all memory sizes, even when storing all samples with M = |S] which approximates joint learning on the observed tasks. Second, with decreasing memory size M the recovery phase deteriorates after the sheer performance drop in the stability gap. This is to be expected larger memory sizes enable better approximation of the joint distribution. Third, although storing samples results in higher overall ACC, Figure indicates smaller performance drop in the stability gap. This is especially noteable for T. to TĄ right after learning the task when all memory sizes have similar performance. Overfitting larger memory sizes is more challenging and might take more iterations per task. Therefore, with reference to our conceptual analysis, the gradient norms on task transitions may be higher for larger M and hence decreasing the drop in the stability gap.
17

---

Published as a conference paper at ICLR 2023
Figure 7: ER accuracy curves on Split-MNIST for the first four tasks, for memory sizes M (500, 1000, 2000, }S]} where |S| stores all seen samples in the stream. Results reported as mean (+:DD over 5 seeds, with horizontal lines representing average min-ACC. Vertical lines indicate the start of a new task.
(a) Ti
(b) 72
(c) Ta
(d) 74
100
50-
M
500
M
1000
07-
[5]
60
10:0
1800
2000
300
15500
1000
1900
16000
2200
D.3 WORST-CASE PERFORMANCE ANALYSIS OF FINETUNING
We examine the finetuning method that naively trains on the new task data with stochastic gradient descent (SGD). Finetuning has repeatedly been shown to be prone to catastrophic forgetting (De Lange al. 2021 van de Ven et al., 2022). We indicate with continual evaluation and our proposed metrics that large forgetting occurs in window of only a few iterations. We use the same experimental setup with 0.9 momentum as described for the four benchmarks Section but with plain SGD instead of ER.
the catastrophic and prompt forgetting window of updates with large values of WF 10 Additionally, the worst-case performance min-ACC shows for the class-incremental learning benchmarks that the accuracy declines to zero for previously learned tasks, and on average to 4.9% for the domain-incrementa. Mini-DomainNer
Table Finetuning results with continual evaluation for four benchmarks, reported as mean (+SD)
over 5 seeds.
Trade-off
Stability
Plasticity
ACC
WC-ACC
min-ACC
FORG
WF10
WP100
Wp
Wp100 100
Split-NNIS Split-CIFAR10
19.5 0.1
19.7
0.0
99.7 0.1
8..
4.4
96.9 #
0.6
17.8
0.0
+
46.6
0..4
62.0
1.5
Split-Minide tatemimmllmgeeet
3.110.1
3.310.1
0.0
54.910.
28.4
52.9 1.6.6.0..
20.2
52.8
Doti-Domainet
1.
10.2
4.9213
13.4
16.7
19.6
10.5
1..9
D.4 THE STABILITY GAP WITH CLASSIFICATION OF SPOKEN WORDS
Figure 8: Per-task accuracy curves (for task to 4) for ER while learning the first five tasks of the speech recognition experiment, performed in class- insseinncemmnntll manner. Reported is the mean (+:DD over seeds. Task transitions are indicated by dashed grey vertical lines.
Tusk
Tesk
Tusk
Tesk
100
80-
Acouracy
60-
20-
200
800
400
600
600
800
Iteration:
Iberation
Itaration
Iteration
To test whether the stability gap can also be observed domains other than image classification, we perform continual learning experiment based on the Synthetic Speech Commands dataset (Buchner) 2017). which contains over 30,000 audio clips of 30 different spoken words. To turn this speech recognition dataset into continual learning experiment, closely follow Cossu et al. (2021) The
18

---

Published as a conference paper at ICLR 2023
dataset is split up into different tasks such that each task contains all the training samples of two words, and these tasks are presented to the algorithm one after the other. We use the same sequence of tasks as Cossu et al. (202I). The experiment is performed according the class-incremental learning scenario (i.e., task identities are not provided at test time), so the algorithm must learn to distinguish between all words encountered far.
Following Cossu et al. 2021), ). we used an MLP with a single hidden layer containing 1024 units with ReLU activations, followed by softmax output layer. Training was done using the Adam optimizer with learning rate 0.0001. Each task was trained for 200 iterations with mini-batch size 256. We used the same preprocessing steps as Cossu et al. (202T): for each audio clip 40 Mel coefficients were extracted using 25 ms sliding window with 10 ms stride, resulting in fixed-length sequences with 101 time steps. These sequences where aggregated over time, so that each sample fed the network consisted of 40 x 101 = 4040 input features.
On this class-incrementll learning experiment we tested the method ER, using a memory budget of 100 examples per class. Figure displays the per-task accuracy curves for the first five tasks, displaying clear stability gaps after every task transition, thus confirming that the stability gap phenomenon is not specific to the domain of image classification.
D.5 CORRELATION CATASTROPHIC FORGETTING AND STABILITY GAP METRICS
Figure 9: Correlation analysis between task-based forgetting (catastrophic forgetting measure) and windowed-forgetting (stability gap measure). After each newly learned task, the FORG and WT are included for all evaluation tasks. Pearson correlation coefficient p shows strong correlation for easier benchmark Split- SNIIT-MNIST, but is closer to zero for the more challenging benchmarks. Correlation determined on the averages over seeds (bold entries), light entries illustrate per-seed results.
(a) Split-MNIST p 0.86
(b) Split-CIFARIO - 0.37
(c) Split Mplii-Minilmagenett p 0.61
35
30
25
PORG
BORG
15
10
10
23
30
40
60
70 80
90
30 40
50
60
Two of our proposed metrics directly measure properties of the stability gap. WF10 measures how steep the initial relative drop is after learning a task and min-ACC measures the absolute minimum performance. With correlation analysis we hope to gain additional insights w.r.t. the standard measure for catastrophic forgetting (FORG).
On each task transition in the learning stream, we compare the FORG with the WF for each of the evaluation tasks separately. This approach can compare per evaluation task how steep the stability gap is, and what the effect is for catastrophic forgetting. The experiment considers evaluation data for ER from the analysis in Figure the main paper. We also considered the min-ACC as measure for the stability gap, but as it often drops to zero accuracy after the first task, it doesn't 't provide further insights.
Figure 9 indicates results for our correlation study on Split-MNIST, Split-CIFAR10, and Split- Minilmagenet. The easier SpLIt-MNIST shows a high Pearson correlation coefficient = 0.86. However, the linear relation is less prominent for the more challenging Split-CIFAR10 and Split- Minilmagenet with p -0.37 and = 0.61. For the latter two, we specifically observe formation of columns. As the 2F measures maximum drop for the evaluation task due to the stability gap, the accuracy can still recover with ER when converging for the new task. However, over time when learning more tasks in the stream, the accuracy recovery of the evaluation task declines. This
19

---

Published as a conference paper at ICLR 2023
results in entries with constant WF whereas FORG declines. This relation between the two metrics can also be qualitatively observed in the accuracy curves Figure 2 the main paper.
D.6 ADDITIONAL GEM RESULTS FOR SPLIT-MNIST AND SPLIT-CIFARIO
The main paper shows results for Split- accuracy curves and zooms of the stability gradient norms on task transitions. In Figure 10 we provide the full V Latabliiyy norm results during the learning process. For the analysis of the stability gap, the zoom the task transitions is most informative, whereas the full results give overview of gradient magnitude towards convergence of the tasks. GEM and ER are reported for the best ACC, both with learning rate 0.01 (see details in Appendix above). Compared to GEM, ER cxhibits larger stability gradient norms over nearly the entire learning trajectory. This indicates that towards convergence of the task, the mini-batch gradients mostly satisfy the GEM constraints with gradient angles I. 90", hence resulting in projection vectors with small magnitude. Although using the gradients of the replay samples directly for ER converges to low gradient norms, there is notable difference for the near-zero values for GEM where these gradients are used for angle- angle-based constraints.
Additionally the results on Split-MNIST, Figure 11 provides the accuracy curves for the first four tasks on Split-CIFAR 10. These results are in correspondence with literature (De Lange & Tuytelaars 2021 Aljundi et al., 2219b a). where GEM is not effective for the more difficult class-increment Split-CIFAR benchmark. Notable the sharp drop to near-zero accuracy on task transitions (stability gap), which subsequently peaks and then declines rapidly.
Figure 10: GEM and ER per-iteration L2-norms of current V Lasbility Split-MNIST. Results reported as mean (DD over 5 seeds. Vertical lines indicate the start new task.
GEM
1000
1500
2000
inerations
Figure 11: GEM and ER accuracy curves for the first four tasks of Split-CIFARI reported as mean (+SD) over 5 seeds. The min-ACC averaged over seeds is zero for both methods.
(a) Ti
(b) 72
(c) T3
T4
ER
50-
GEM
Ascimey
60
20-
10.00
1500
1000
2000
1000
1250
1500
1730
2000
1200
1400
1000
1800
2000
Itarations
Itorationa
D.7 ADDITIONAL LWF RESULTS FOR MINI-DOMAINNET
Besides the empirical analysis on Split-MNIST and
we demonstrate the stability
gap for Learning without Forgetting (LwF) (Li & Hoiem 2017) in the domain-increment Mini- DomainNet (Figure 12 The stability gaps are especially notable for T; and Tz in Figure 12 a b). Figure 12]e) reports the stability gradient norms, with focus the fask-transitions in Figure f2 The results confirm the zero gradient norms on task transitions, followed by recovery with increasing gradient norms.
20

---

Published as a conference paper at ICLR 2023
Figure 12: LwF and ER accuracy curves (a-d) Mini-DomainNet for the four domains, and the per-iteration 1.2-norms of current Vasability (e-g). Results are reported as mean (+SD) over seeds, with horizontal lines representing average min-ACC. Vertical lines indicate the start of task. Note that the x-axis scale varies over (a-d), (e) overviews the full learning trajectory, and (fig) are zooms in on the first few iterations for the last two tasks.
(a) Ti
(b) T2
(c) T3
Ta
30
Armauuy
20
RR
5000
1000
5000
750 iterationn
10010
600
:terations
Ieration:
Inanationn
(e) LwF 7i T4
(f) LwF Z00m T3
(g) LwF zoom Ta
1.0
|| Cainiottll
100-
10-
0.5
0.3
0.5
200
400
iterations
21


Reviews:
[{"review_id": "zt3Ivxp4Bl", "reviewer": "Reviewer_Jyqa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper try to show that common state-of-the-art methods in continual learning suffer from substantial forgetting upon starting to\nlearn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery. The main contributions can be summarized as :\n\n(1) This work defines a framework for continual evaluation that evaluates the learner after each update.\n\n(2) This work conducts an empirical study with the continual evaluation framework, which leads to identifying the stability gap for Experience Replay.\n\n(3) This work proposes a conceptual analysis as a hypothesis for causing the stability gap, by disentangling the gradients based on plasticity and stability.", "summary_of_the_review": "marginally below the acceptance threshold", "strength_and_weaknesses": "Strengths:\n\n(a) This paper is well-written and easy to read.\n\n(b) The phenomenon found in this work is easy to follow.\n\nWeaknesses:\n\n(a) Some notations are confusing. What  is $|T_{i}|$? In Section 3.2, what is the maximal accuracy increase (lack of detailed formulations)?\n\n(b) My main concern is about the phenomenon---stability gap. This work claims: ``we show that common state-of-the-art methods still suffer from substantial forgetting upon starting to learn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery.'' I am afraid that this phenomenon can not cover most kinds of baselines in CL and only exists in replay-based methods. The phenomenon is trivial in replay-based methods and easy to be explained. The model reaches at a local minimum in the previous task and tries to approach to next task's local minimum quickly in the early several iterations, which leads to the substantial forgetting upon starting to learn new tasks. As replay-based methods replay previous experience, the model learns previous tasks' skills in the following iterations. These lead to the phase of performance recovery. The baselines  discussed in this paper is mainly replay-based (ER and GEM), and the phenomenon in LwF is not well explained.  I think this phenomenon is more likely to not exist in other kinds of baselines---regularization-based (EWC, SI and MAS) and parameter isolation based (PackNet and HAT). For example, the work [1] plots a similar accuracy curve of EWC in Fig.3(a) where there is no stability gap.\n\n[1] Kirkpatrick J, Pascanu R, Rabinowitz N, et al. Overcoming catastrophic forgetting in neural networks[J]. Proceedings of the national academy of sciences, 2017, 114(13): 3521-3526.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MPaWTHP-ski", "reviewer": "Reviewer_xcu2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes stability gap as a concept to measure the transient forgetting that happens for continual learning. It measures the trade-off between the knowledge conservation by learning new tasks (retaining knowledge) and getting new knowledge for the new task. The relation of stability gap with similarity between tasks has bee explored and its analysis also has been conducted.", "summary_of_the_review": "This paper proposes a concept that tries to measure the tradeoff between forgetting and getting new knowledge in continual learning which is very important in such learnings. The analysis of this concept also helps to better understand it. Overall motivation and idea and experiments are convincing, the only concern is its analysis that is only limited to image-based datasets and some of the experiments are conducted for synthetic datasets that are not close to real-case scenarios.", "strength_and_weaknesses": "The idea of measuring the stability-gap which shows how much information is retained and how much new information is learnt after each iteration is a very interesting concept in continual learning since  by learning new tasks the model tries to recover its knowledge after losing it in first timestamps for that task. \nThe paper strengths are:\n1) The good explanation of the problem and the solution.\n\n2) Proposing a concept that tries to measure the tradeoff between forgetting and getting new knowledge after each iteration. \n\n3) The analysis of the proposed stability gap and its dependency with tasks' similarity make the idea more promising.\n\nSome weaknesses that I can mention for this work are:\n\n1) The idea and analysis have been implemented only for image-based datasets while the continual learning happens for other domains such as texts. The shortage of experiments and analysis of the metric on such datasets should be addressed to show the generalizability of the stability gap for different domains.\n\n2) To measure the relation between stability gap and tasks differences, a synthetic dataset is created by rotating images, however studying this relation on more real-cases tasks can be more beneficial and important to show the relevance.\n\nThis is a minor question and some explanations can make it more clear: according to Table 1 stability gap changes by increasing the evaluation periodicity, therefore what is the best periodicity that should be used in such evaluations, is it domain dependent?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KCRvMJPEPQ", "reviewer": "Reviewer_q7YA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper pointed out the shortcomings in the existing approaches for continual learning, mainly catastrophic forgetting of the previous task during new task adoption. It proposes per-iteration continual evaluation with new metrics that enable measuring worst-case performance. This work empirically studied five existing continual works and shows that they suffer from a significant performance loss on previous tasks. The stability gap increases significantly as subsequent tasks are more dissimilar. It also explains the stability-plasticity trade-off by proposing a new metric evaluation for continual learning. The conceptual analysis is provided for causing the stability gap by disentangling the gradients based on plasticity and stability.\n\n\n\n\n\n\n\n\n\n", "summary_of_the_review": "The paper has pointed out the interesting shortcomings in the existing continual learning approaches and proposed a new evaluation framework to address them. I encourage the authors to include my concerns raised in the weaknesses subsection. For more detail, please see the strength and weaknesses section.", "strength_and_weaknesses": "Strengths:\n1- The paper identifies the issues that occur in the existing task \nincremental continual learning approaches- and tries to address them by providing empirical experiments and detailed analysis.\n\n2- It proposes a framework for continual evaluation that evaluates the learner after each update and provides an ablation study to prove why evaluation frequency indicates continual evaluation is necessary to surface the stability gap.\n\n3- It provides an empirical study with the continual evaluation framework to identify the stability gap for Experience Replay.\n\n4- To quantify the stability gap in contrast to existing metrics in the existing approaches, it proposes novel metrics such as the minimum and worst-case accuracy (min-ACC and WC-ACC). It shows that the stability gap is significantly influenced by the degree of similarity of consecutive tasks in the data stream.\n\n5-  It also provided a conceptual analysis for causing the stability gap and conducted the experiments for several existing approaches such as Experience Replay (Chaudhry et al., 2019b), GEM (Lopez-Paz & Ranzato, 2017), and LwF (Li & Hoiem, 2017) for supporting the proposed hypothesis.\n\nWeaknesses:\n1- In the experimental setup for MNIST, as you mentioned (on page 6), \"each task constitutes the entire MNIST dataset.\" Is it mean that each task consists of all class examples (0,1...9)? If yes, then how is it valid for continual learning? Because there should be new class examples in the next task. How a buffer capacity affects ER performance?\n2- How your new evaluation setup work for online continual learning approaches such as [a],[b] and[c]? Is the newly proposed metrics (worst case and Min) helpful in identifying this stability gap?\n3- The extreme case for continual learning is where the data examples are available only once, unlike the replay-based approaches, where the old data may be available for future training [d]. It would be interesting to apply the proposed framework to this challenging continual setup [d].\n\n\n[a]- Online Class-Incremental Continual Learning with Adversarial Shapley Value, AAAI-2021.\n[b]- Online Continual Learning under Extreme Memory Constraints, ECCV 2020.\n\n[c]- Online continual learning from imbalanced data, ICML 2020.\n[d]- Efficient feature transformations for discriminative and generative continual learning, CVPR 2021.\n\n\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FX_d25ZPGl", "reviewer": "Reviewer_FwH6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work is motivated by a problem that current continual learning metrics are mainly coarse-grained (e.g., task-based), which can lead to information loss. For instance, monitoring task-based metrics cannot show the stability gap (substantial but temporary forgetting upon learning a new task). The stability gap is defined as the sudden drop in performance right after facing a new task. \n\n\nFurther, the authors propose various metrics for continual learning that focus on worst-case performance, which can be helpful in practical scenarios. Finally, the paper briefly studies the implications of the stability gap from the task similarity perspective. Also, it shows that this phenomenon exists in the presence of other CL algorithms.", "summary_of_the_review": "Overall, While there is room for improvement in a more in-depth analysis of the stability gap, I find this work a good first step that can contribute significantly to the field. Hence, I recommend acceptance.", "strength_and_weaknesses": "### Strengths\n- The stability gap is an interesting problem in continual learning.\n- The provided metrics are helpful for some sensitive scenarios.\n- The experiments are well-desigend.\n\n\n### Weakness\n- The authors do not propose a mechanism to alleviate the stability gap (and potentially catastrophic forgetting). However, I believe as a first step, the analysis of Sec. 5 is enough to motivate others to work on this problem.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%