Overcoming the Stability Gap in Continual Learning
Authors: Md Yousuf Harun
Christopher Kanan
Venue: Accepted by TMLR ()

Abstract: Pre-trained deep neural networks (DNNs) are being widely deployed by industry for making business decisions and to serve users; however, a major problem is model decay, where the DNN's predictions become more erroneous over time, resulting in revenue loss or unhappy users. To mitigate model decay, DNNs are retrained from scratch using old and new data. This is computationally expensive, so retraining happens only once performance significantly decreases. Here, we study how continual learning (CL) could potentially overcome model decay in large pre-trained DNNs and greatly reduce computational costs for keeping DNNs up-to-date. We identify the ``stability gap'' as a major obstacle in our setting. The stability gap refers to a phenomenon where learning new data causes large drops in performance for past tasks before CL mitigation methods eventually compensate for this drop. We test two hypotheses to investigate the factors influencing the stability gap and identify a method that vastly reduces this gap. In large-scale experiments for both easy and hard CL distributions (e.g., class incremental learning), we demonstrate that our method reduces the stability gap and greatly increases computational efficiency. Our work aligns CL with the goals of the production setting, where CL is needed for many applications.

Published in Transactions on Machine Learning Research (09/2024)
Overcoming the Stability Gap in Continual Learning
Md Yousuf Harun Rochester Institute of Technology
ml1023@rit.cm edu
Christopher Kanan
www.ann@cccrrochsttte edu
University of Rochester
Reviewed on OpenReview: https: // openreview. net/ forum? id= o2uEEuuUmm
Abstract
Pre-trained deep neural networks (DNNs) are being widely deployed by industry for making business docisions and to serve users; however, a major problem is model decay, where the DNN's predictions become more erroncous over time, resulting in revenue loss or unhappy users. To mitigate model decay, DNNs are retrained from scratch using old and new data. This is computationally expensive, retraining happens only once performance significantly decreases. Here, we study how continual learning (CL) could potentially overcome model decay in large pre-trained DNNs and greatly reduce computational costs for keeping DNNs up-to-date. We identify the "stability gap" as a major obstacle in our setting. The stability gap refers to a phenomenon where learning new data causes large drops in performance for past tasks before CL mitigation methods eventually compensate for this drop. We test two hypotheses to investigate the factors influencing the stability gap and identify method that vastly reduces this gap. In large-scale experiments for both easy and hard CL distributions (e.g., class incremental learning), we demonstrate that our method reduces the stability gap and greatly increases computational efficiency. Our work aligns CL with the goals of the production sctting, where CL is needed for many applications
1 Introduction
Deep neural nelworks (DNNs) are now widely deployed the industry; however, a major problem for many companies is model decay, where the predictive performance of DNN degrades over time (Zhou et al., 2020). This is primarily caused by concept drift (Tsymbal, 2004; Gama et al., 2014; Lu et al., 2018), where the nature of the predicted larget variables changes over time, e.g., for a classifier, this would correspond to the introduction of new categories or an expanded definition individual classes. When model decay is detected, most companies employ offline retraining (i.e., batch or joint retraining). where model is retrained from scratch with a combination of old and new data (Egg, 2021). This is very expensive, so model monitoring is used to determine when offline training is required (Mäkinen al., 2021). Despite frequent offline training, deployed models still suffer from accuracy drops of up 40% (Mallick et al., 2022).
Continual learning (CL) is promising solution for preventing model decay (Huyen, 2022a;b; Jain & Shenoy, 2022), where in CT. the goal is to update DNN incrementally with new data. while preserving the previous knowledge (Parisi et al., 2019). In a GrubHub study, this enabled them to avoid model decay and provided a 45x decrease in training costs compared to daily offline training (Egg, 2021). To do this, GrubHub employed online updates on new samples, Online updates can cause catastrophic forgetting of past knowledge due to concopt drift (McCloskey & Cohen, 1989), but due to the rapidly changing preferences of their customers, forgetting past knowledge was desirable. However, catastrophie forgetting is unacceptable for many industry applications because past knowledge must be retained. For CL to be widely adopted by industry, it must be shown to inhibit model decay in pre-trained models, provide significant computational benefits, work for
Code is available at httpe://yonsuuf007 http://yoosur077giihhhh.. wttggt.iitsggaaai...eeeee io/sgmsite

---

Published in Transactions on Machine Learning Research (09/2024)
78.5
78.0
78.0
Task Accuracy [96]
Old Task Accuracy [86]
77.5
77.5
77.0
77.0
mml
76.5
76.5
76.0
76.0
75.5
75.0
0
600
1200
1800
2400
3000
75.5
0
100
200
300
400
500
600
Iterations (Updating New Task)
#lterations (Updating on New Task)
(a) Stability gap over all rehearsal sessions
(b) Stability gap as an average over all rehearsals
An overview of stability gap phenomenon. The stability gap is a phenomenon that occurs in CL when learning new data, where accuracy on previously learned data (Y-axis) drops significantly as a function of training iterations when new distribution introduced (X-axis). Fig. (a) illustrates this behavior during CIL, where network pre-trained ImageNet-1IK, IK, learns 365 new classes from Places365-LT over five rehearsal sessions. Each rehearsal session involves 600 iterations that combine samples from the old and new tasks. A gray dotted vertical line marks the end of rehearsal session or task transition. When rehearsal begins, accuracy on the old task for the conventional rehearsal drops dramatically before slowly recovering, although it fails to recover the original performance on the old data. The traditional measures of catastrophic forgetting focus on performance at task transitions (red diamonds), ignoring significant forgetting that occurs during the lcarning process between task transitions. Fig. (b) shows the stability gap in the learning curve averaged over five rehearsal sessions. In this work, we attempt to mitigate the stability gap.
Figure 1:
arbitrary shifts in concepts, and ideally be effective as offline training. The majority of CL algorithms deviate from these criteria, i.c., they impose constraints on storage that are irrelevant to industry problems, shun pre-trained models, perform worse than ofline retraining, design algorithms for only a single concept drift distribution (e.g., class incremental learning), and/or are more computationally expensive than offline retraining (Harun et al., 2023a;b; Prabhu et al., 2023b;a; Verwimp et al., 2024; Lee et al., 2023).
Here, we study using CL to mitigate model decay and to introduce new concepts into DNN. Initially, studied updating ImageNet-IIK pre-trained models with new classes in class incremental learning (CIL) experiments. We used computationally constrained cumulative rehearsal to mitigate forgetting, where re- hearsal involves mixing old samples with new ones to prevent forgetting, and cumulative rehearsal does this using all previously observed samples. Computational constraints were enforced by limiting the total number of rehcarsal updates permitted. However, we found that CL's stability gap was a major obstacle to our goal (De Lange et al., 2023), as shown in Fig. 1.
The stability gap refers to the observation that when a model updated on new data, accuracy on the classes observed in earlier batches plummets before rehearsal other CL methods gradually recover old performance. In the typical CIL setting we adopted, samples arrive in batches where each batch contains classes only within that batch. As seen in Fig. 1, the performance plummets when learning a new batch during the rchcarsal sessions and does not fully recover. The stability gap can be seen as transient forgetting due to introducing a new task. Typically, catastrophic forgetting refers to the performance drops observed at the end of learning new tasks (at task transitions). In contrast, the stability gap refers to the performance drops observed over learning steps (between task transitions). In our work, we seek to understand why the stability gap happens in our scenario and how to effectively mitigate it, enabling using fewer rehearsal updates while achieving higher accuracy.
2

---

Published in Transactions on Machine Learning Research (09/2024)
We test two hypotheses to examine the stability gap in CIL with pre-trained DNNs:
1. The stability gap is increased in part due to having a large loss at the output layer for the new classes. To test this hypothesis, we study two methods to mitigate the large loss 2 in the output layer for the new classes. The first method is to initialize the output layer in a data-driven approach rather than randomly initializing the output units responsible for the new classes. The second method is specialized form of soft targets for the network, rather than the typical hard targets used for the network, where these soft targets are designed to improve performance for the new classes while minimally perlurbing others.
The stability gap is increased in part due excessive network plasticity. We test this hypothesis by controlling the level of plasticity in network layers in a dynamic manner. For hidden layers, we test this hypothesis using LoRA (Hu et al., 2021), which reduces the number of trainable parameters in the hidden layers of the network. For rehearsal methods, after each rehearsal ses- sion, these weights are folded into the original network weights. For the output layer, we test this hypothesis by freezing the output units for classes seen in earlier batches during rehearsal.
In our experiments, we find that both hypotheses are supported. This leads us to develop a combinod method that greatly reduces the stability gap for both CIL and other distributions.
This paper makes the following major contributions:
We are the first to study overcoming the stability gap. We test the aforementioned hypotheses in CIL and discover that they both play an important role. We propose novel metrics to measure the stability, plasticity, and continual knowledge gaps.
2. We are the first to study mainlaining or improving performance on ImageNet-1K while learning additional classes, an aspect not investigated in previous research employing ImageNet-1K pre- trained backbones (Wang et al., 2022b;a; Smith et al., 2023; Gao et al., 2023; McDonnell et al., 2024). We study this for both the CIL and IID (independent and identically distributed) distributions. These experiments conducted by combining TmageNet-1K with Places365 or Places365-LT (a combined 1365 total classes).
3. We develop a method that greatly mitigates the stability gap and significantly improves computa- tional efficiency. Our method requires 16.7x fewer network updates than a jointly-trained (upper bound) model. Tn terms of TFLOPs, our method provides 31.9x speedup (see Fig. 5). For the IID CL distribution, the method achieves backward transfer, where learning the new dataset helps to improve ImageNet-1K accuracy.
4. We show that our method effective and we integrate it into existing CL methods. Specifically, it performs well in conjunction with the rehearsal methods Vanilla Rehearsal, DERpp, GDumb, and REMIND well as the non-rehearsal method LwF.
2 Background
Many methods have been proposed to learn continuously from non-stationary datasets in continual learn- ing (see Zhou et al. (2023) for review). These methods can be broadly divided into three categories: 1) Rehearsal-based methods store reconstruct subset of old data to rehearse alongside new data while learning a new batch (Chaudhry et al., 2019; Hou et al., 2019; Rebuffi et al., 2017; Wu et al., 2019), 2) Regularization-based methods constrain weight updates by adding additional regularization in the loss function (Aljundi et al., 2018; Chaudhry et al., 2018; Dhar et al., 2019; Kirkpatrick et al., 2017), and 3) Parameter-isolation based methods allocate multiple sets of parameters or multiple copies of the model to different incremental batches (Douillard et al., 2021; Yan al., 2021; Yoon et al., 2020). Since parameter
2 Referring to the loss when we only train the output layer (I.e., final classification layer) and freeze other layers.
3

---

Published in Transactions on Machine Learning Research (09/2024)
isolation methods do not allow backward transfer, the stability gap does not apply to them De Lange et al. (2023).
While most of the CL research community has focused on mitigating catastrophic forgetting (MeCloskey & Cohen, 1989), there is a growing body of literature demonstrating its ability to reduce the amount of compute needed to update the network (Ghunaim et al., 2023; Harun et al., 2023a;bçe; Prabhu et al., 2023b; Verwimp et al., 2024). Recently, a major obstacle to this objective has been identified in CL: The stability gap (De Lange et al., 2023). However, it has been studied on small datasets, c.g., CIFAR, using randomly initialized DNNs. This setting is not aligned with studying model decay in industrial settings that involve many-class data streams and preserving performance in large pre-trained DNNs.
Traditionally, the Imaining-from-scruulc CL would start with a randomly initialized DNN that had no pre- training. With the prevalence of large pre-trained models, recent CL research has started integrating these models into the learning process. This has resulted a growing number of CL works that utilize large pre-trained vision transformer (ViT) models, which have been pre-trained on ImageNet-1K 21K (Wang et al., 2022b;a; Smith et al., 2023; Gao et al., 2023; McDonnell et al., 2024). Moreover, many earlier works use pre-training for CL (Belouadah & Popescu, 2019; Hou et al., 2019; Castro et al., 2018; Rebuffi et al., 2017; Hayes et al., 2020; Douillard et al., 2020; Harun et al., 2023b) and emphasize the significance of pre-training in the context of CL (Lee et al., 2023; Mehta et al., 2023; Ostapenko et al., 2022; Ramasesh et al., 2021b; Gallardo et al., 2021). However, as demonstrated in several works (Wang et al., 2022a; Mirzadeh et al., 2022), using pre-trained models does not naively enhance CL performance and effectively leveraging vre-trained models for CL remains an open question. In this work, we mainly focus on overcoming the stability gap in CL with pre-trained models.
ImageNel-
Almost all CL methods focus on the catastrophic forgetting problem with evaluations occurring discrete batch or task transitions (Hayes et al., 2018; Chaudhry et al., 2018) and fail to capture the stability gap that occurs immediately after a new task is introduced (Fig. 1). De Lange et al. (2023) demonstrates the stability gap occurs for a variety of CL methods, including rehearsal (Chaudhry et al., 2019), GEM (Lopez-Pad Paz & Ranzato, 2017), EWC (Kirkpatrick et al., 2017)), and LwF (Li & Hoiem, 2017). To quantify the stability gap, they propose continual evaluation metrics based on worst-case performance ie., the largest drop in accuracy on old batches. However, their metrics do not enable comparing different CL models because they only quantily the largest drop relative to the same model's best performance, thus making it impossible to quantify the impact of different training methodologies aimed at mitigating the stability gap. We address this limitation using novel metries that are normalized against a jointly-trained universal upper bound (see Sec. 3.2). We also illustrate this phenomenon using synthetic examples in Appendix F to build intuitions. We study overcoming the stability gap with both rehearsal and non-rehearsal methods, with a focus on rehearsal due to its effectiveness (van de Ven al., 2022; Zhou et al., 2023).
3 Continual Learning Protocol
In this work, we attempt to align CL with the industry setting, where CL updates knowledge in pre-trained models prevent model decay when new concepts are introduced while preserving performance on the original classes. Likewise, because the industry requires maintaining high accuracy, we focus rehearsal- based CL; nonetheless, demonstrate that mitigation strategies can be used for non-rehearsal methods that use regularization and knowledge distillation. In this section, we formalize our CL framework and define the metrics we use to measure the stability gap.
3.1 Formal Continual Learning Setting
To align our work with addressing model decay, we assume CL begins with pre-trained model capable of K-way classification. The goal is to incorporate new data, which may have additional classes, into the model, while preserving or improving performance on the original K classes. For this purpose, we use ImageNet- IK pro-trained models (K 1000). While ImageNet-1K pre-trained models are commonly employed in CL systems, previous studies do not attempt to maintain or measure performance on ImageNet- 1K dataset itself (Wang et al., 2022b;a; Smith et al., 2023; Gao et al., 2023; McDonnell et al., 2024). Our research
4

---

Published in Transactions on Machine Learning Research (09/2024)
uniquely explores this aspect in CL. Upon deployment, the pre-trained model is exposed to sequence of data batches over N - learning sessions, i.e., {S2,... ,w),, where S is the data used for pre-training. The j'th session consists of a batch of m, labeled training samples, i.e., Sj = {(x,, Vy) an instance of category E Y and Y is the label space of task j. In the CIL sctting, each S contains non-overlapping classes, i.e., Yy = 0 for i H The total number of samples in the entire sequence is M M=E; my- When learning new batch Sj, the learner can access Sj and any stored data from previous batches S13-1- During lest time, the learner is evaluated on lest data from all seen classes, Y3 = Y1 U... Yg- The batch (or task) identifier j cannot be exploited during test time.
my where Xj is
To efficiently adapt to a large-scale data stream in a real-world setting, a CL system should not increase compute cost over time. For all learning sessions j after pre-training, it given a fixed compute budget B corresponding to the number of SGD model updates i.c., B U x b, where U and b denote the number of training iterations and the number of training samples per iteration respectively.
3.2 Measuring the Stability Gap
Popular CL metrics focus measuring performance after each S, is learned. They do not permit fine- grained analysis for a) preserving old knowledge, b) acquiring new knowledge and c) balancing both during training. To study the stability gap in manner that enables us to test our hypotheses across models trained with different strategics, we created metrics that mcasure these criteria: 1) the stability gap, SA (criterion a) 2) the plasticity gap, Pa (criterion b), and 3) the continual knowledge gap, CKA (criterion c). Each metric asks: How much performance does a learner lack on previously observed, recently observed, or all observed data compared to joint model (upper bound) when learning new data?
For the j'th learning session, denote evaluation sets old, new, and all seen data by Eig-1, E,, and Frg; respectively. Ai is the accuracy of the current model 0; on batch evaluation set E at training iteration i, and L is the total number of iterations. Ajoint is the final accuracy of a joint model (0joins) trained jointly on all data. We define the stability gap as
SA = 1
1
Σ
sggh, where soold 1 Σ
A,(EI.j-1,0.) 0c)
(1)
N
Ajoint (E1,--1, 0joint)
1=1
Similarly, the plasticity gap is
1
1
A. 02)
PA=1-
Spew; where Quew
(2)
L
Abeat (Lis, Obest)
J=2
where Abest stands for the best accuracy achieved by the best CL model (Obest) at any time during training. And finally, we define the continual knowledge gap as
N 25 where 229tt j=2
L A, 0,)
CKA = N
Σ 1=1
(3)
I
AEii. Ojoine)
Qj records CL performance compared Ajoint or Abest- After learning all N batches, Ω scores are averaged to indicate average performance gain. The first batch is excluded since that is used for pre-training. For all metrics, smaller Sa, Pa, and CKs indicate better performance. When A, Ajoint and A, = Abest for all L iterations, Sa, PA. and CKA become zero which is desirable. A negative value means knowledge transfer between new and old batches, which is desirable. These metrics apply for offline CL (i.e., incremental-batthh CL) and online CL with any data distributions, including CIL and IID.
4 Stability Gap Mitigation Methods
This scction describes the methods we use to test our hypotheses regarding the factors that increase the stability gap. We use our evaluation metrics, SA, PA, and CKA to validate the efficacy of proposed mitigation methods. We include additional discussion in Appendix H.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Weight Initialization. In CIL, the output units for new classes are typically randomly initialized causing those units to produce a high loss during backpropagation. We hypothesize that using data-driven initial- ization for new class units will reduce the loss and therefore reduce the stability gap, Sa. To test this, we initialize them to the mean of unit length embeddings for that class, i.c.,
h
We
(4)
where Wa E Idd is the output layer weight vector for class k; hj E IRd is the j'th embedding from the penultimate layer, and V is the mumber of samples from class in the batch. Intuitively, this approach aims to align the initial weights more closely with the distribution of the new class's data, facilitating faster and more effective learning adjustments during the initial phase of CL.
For classification, models are often trained with hard targets, i.e., at training iteration i a one-hot vector t; with a '1' in position k corresponding to the correct class. We hypothesize hard target training is partially responsible for the stability gap. Intuitively, hard targets are one-hot. encoded and enforce strict. inter-class independence despite several classes sharing distributional similarities. This property of hard targets, therefore, also causes a large initial loss when learning new classes. Soft targets, on the other hand, can help the network retain the joint inter-class distributions, which further ameliorates the perturbation of learned classes. To test this, we use soft targets constructed such that the model's predictions on previously learned classes are largely preserved.
Hard vs. Dynamic Soft Targets.
At learning iteration i, let P (k|x;; 0.) be the model's output softmax probabilities for sample x; from class k given the model's current parameters 0, and the predicted class be v/² = arg max P (k|x,;;0.) We maintain a running average vector ug E R of the softmax probabilities for each class that is updated when an example from class & is observed, i.e.,
UR + P (k)x;;;))
U
Ck +1
where Cj is a counter for class k that is subsequently increased by 1, and us is initialized to a uniform distribution prior to the running updates. Subsequently, soft targets t, for iteration i are constructed by setting t; <- uy and then setting the element for the correct class to 1, i.e., t; < 1. If oo / k, then we also set t; [v4] < 1/K. Subsequently, t, is normalized to sum to 1 and used to update the network. This strategy results in targets that minimally perturb the network and smaller loss values. In Appendix. G, we
illustrate the process of updating soft targets.
Limiting Hidden Layer Plasticity Using LoRA. To accumulate knowledge over time, most CL ap- proaches update the entire network. Given that each batch of data in CL is relatively small, we hypothesize that this leads to excessively perturbing hidden representations, leading to a larger stability gap. To test this hypothesis, we constrain the number of trainable parameters in hidden representations using a network adaptor. While there are various network adaptors (Han et al., 2024) that restrict network plasticity, we use low-rank adaptation (LoRA) (Hu et al., 2021) due to its simplicity and effoctiveness. Specifically, we inject LoRA weights into the linear layers of the network, and only these parameters and the output. layer are updated, which greatly reduces the number of trainable parameters.
For batch j. let W E R be previously learned linear layer. At the start of each learning session, we reparameterize this layer by replacing W with
(6) where B E Rdxr and A E R are the LoRA adapter parameters with rank < min(d, Only B and A are plastic, with A initialized with random Gaussian values and B initialized to a zero matrix, so BA = 0 at the beginning of the learning session. At the end of the session, the LoRA parameters are folded into the network, Le., W - e.. In LoRA experiments, only the output layer and the LoRA parameters are plastic.
e W + BA,
Limiting Output Layer Plasticity via Targeted Freezing. In CIL, large changes in the network's representations for old classes increase the stability gap. While LoRA restricts plasticity in hidden represen- tations, we hypothesize that restricting plasticity in the out layer could also be helpful for CIL. Therefore,
6

---

Published in Transactions on Machine Learning Research (09/2024)
we study freezing output layer weights for classes previously learned in earlier batches. For rehearsal meth- ods, samples from classes seen in earlier batches have the hidden layers updated as usual. We refer to this technique as old output class freezing (OOCF).
Combining Mitigation Methods & SGM. We independently evaluate each of the stability gap mitigation methods. Additionally, we evaluate them in combination. We refer to the method that combines dynamic soft targets, weight initialization, OOCF, and LoRA as SGM (Stability Gap Mitigation). Soft targets and weight initialization prevent higher loss at the output layer to enhance stability. OOCF and LoRA restrict plasticity in the network to targeted locations so that existing representations are minimally perturbed.
5 Main Experiments: Hypothesis Evaluation
We design our experiments to emulate using CL to mitigale model decay. Hence, we use an ImageNet-1K pre-trained DNN that is progressively updated with new data and classes. As shown in Fig. and Fig. 4, the stability gap is present when vanilla rehearsal is employed.
5.1 Experimental Setup
Bounding Compute. In real-world scenario, a continual learner must adapt to large-scale data stream, which may not be fcasible if more computation is required over time. Recently, many works have advocated computational efficiency for CL (Prabhu et al., 2023b; Harun et al., 2023b;a;;; Verwimp el al., 2024; Zhang et al., 2023). In our experiments, we bound compute by a fixed number of training iterations or SGD steps.
Rehcarsal. Bocause our goal to understand and mitigate the stability gap, our main results use rchearsal and assume the learner has access to all previously observed data, with no constraints on storage. This is aligned with finding better alternative to periodically retraining from scratch as more data is acquired, which is commonly done in industry where the computational budget depends compute to a far greater extent than data storage (Prabhu et al., 2023b;a).
Continual Learning Procedure. We aim to study CL in an industry-like setting, requiring a large-scale data stream with numerous object categories. However, it is difficult to find a suitable dataset well-curated and suitable for large-scale CL experiments. Therefore, we construct large-scale data stream by combining ImageNet-11 with another dataset, Places365-LT, which is variant of the Places365 dataset. Places365 is challenging (Liu et al., 2019) and is widely used for out-of-distribution (OOD) detection with ImageNet pre-trained models (Zhu et al., 2022).
During CL, the model sequentially learns 5 incremental batches of data from Places365-LT. In the CIL ordering, each CL batch contains 73 catogories, and in the IID ordering each CL batch has 12500 examples. During rehearsal, the model is updated over 600 minibatches, where the minibatch consists of 128 samples where 50% are selected randomly from the current CL batch and 50% from data seen in earlier CL batches and ImageNet-1K. We study two CL orderings for Places365-LT: 1) CIL ordering where cach batch has classes exclusively seen in that batch (maximum concept drift), and 2) IID ordering where each batch contains examples from randomly sampled classes (minimal concept drift). These are two opposite extreme situations in CL. Although the stability gap is not known to occur in the IID CL setting, this setting is critical to demonstrating the algorithm's generality. To measure the stability gap and other metrics, we assess performance during rehearsal every 50 training iterations, where the test set consists of the ImageNet- IK validation set and all of the classes from Places365-LT from the current and prior CL batches. Additional implementation and dataset details are given in Appendix C and B.
Network Architecture. We choose a network architecture that is amenable to LoRA and performs well in offline training with ImageNet-1K compared to similar-sized DNNs. In our main results, we study CL using the ConvNeXt V2-Femto (Woo et al., 2023) CNN that has been pre-trained on ImageNet- using a fully convolutional masked autoencoder framework followed by supervised fne-tuning on ImageNet 1K. While ResNet18 is widely used in CL, it under-performs other lightweight CNNs in CL (Hayes & Kanan, 2022; Harun et al., 2023b). ConvNeXtV22Femtt has 5.2M parameters, which is 2x less than ResNet18's 11.6M parameters, and it outperforms R8 by absolute 8.47% in final top-1 accuracy on Iet
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Hypothesis Evaluation (CIL). Results after learning ImageNet-1K followed by Places365-LT over 5 rehearsal sessions. 1l denotes average accuracy (%) over rehearsal sessions and is final accuracy (%) on all 1365 classes. o stands for final accuracy (%) on ImageNet-IK only, #P denotes the trainable parameters in Millions. The best and 2nd best values indicated in bold and underlined respectively.
Method
#P4
SA 4
PA 4
CKA 4
o t
# t
t
Joint Model (Upper Bound)
5.08
77.58
70.69
Naive Finetune (Lower Bound)
5.08
0.743
0.474
0.739
16.77
14.03
15.60
Output Layer Only
0.53
0.026
0.494
0.032
76.10
71.23
67.89
Vanilla Rehearsal
5.08
0.028
0.393
0.033
76.06
71.52
67.67
Dynamic Soft Targets
5.08
0.022
0.397
0.029
76.26
71.78
68.24
Weight Initialization
5.08
0.024
0.097
0.020
76.69
72.43
69.22
OOCF
5.08
0.026
0.376
0.032
75.95
71.57
67.94
LoRA
1.45
0.018
0.316
0.019
76.92
72.74
69.19
SGM (Combined Method)
1.45
0.006
0.082
0.002
77.64
73.70
70.30
It is worth noting that LoRA is not appropriate for CNN architectures (e.g., ResNet) that lack I x 1 convolutional layers, but it can be used with ViT, ConvNeXtV1, ConvNeXtV2, and other DNN architectures with these linear layers. Fach block of ConvNeXt consists of one 2D convolutional layer and two 1 x 1 convolutional layers. For experiments using LoRA, the 1 x 1 convolutional layers in ConvNeXt blocks are modified to incorporate LoRA's weights using Equation 6. The number of trainable LoRA weights is 0.92M, which is much less than the total number of hidden layer parameters (5.08M). Based on prior work, early layers in the network are universal feature extractors and are little altered during CL (Ramasesh et al., 2021a; Ebrahini et al., 2020; Pellegrini et al., 2020; Harun et al., 2024), we freeze the first blocks of the CNN in all experiments, leaving the remaining 8 blocks plastic (97.7% of the parameters) which consist of 5.08M parameters.
5.2
Experimental Results
We describe our findings on how the proposed method (SGM) mitigates the stability gap and enhances learning efficiency in an unlimited storage setting with rehearsal in CIL.
5.2.1
Evaluating Our Hypotheses
Here, we describe the results for evaluating our two hypotheses.
Baselines. In our experiments, we evaluate each of the methods in Sec. 4 individually and the combined SGM method. As baselines, we compare SGM with rehearsal against vanilla rehearsal, which uses unlimited storage for rehearsal without any additional components, as well as joint models (upper bound). The joint models are jointly trained on ImageNet and the CL batches seen up to the current batch. We also include naive finetuning which is vanilla variant without rehearsal and serves lower bound. Finally, we compare with the output layer only which updates the output layer with rehearsal while freezing all other layers.
Results. Our main CIL results are given in Table 1. SGM with rehearsal shows the greatest reduction in the stability gap (Sx), plasticity gap (Pa), and continual knowledge gap (CKs). also performs best other metrics. Of its components, LoRA reduces the stability gap the most; however, the stability gap for LoRA is 3x higher than SGM. We next turn to examining the support for our two hypotheses.
Hypothesis 1. Our first hypothesis was that the stability gap in CIL is caused by having large loss at the output layer due to the new classes, which we tested by using weight initialization and dynamic soft targets. Both methods are effective at achieving the goal of reducing the initial loss, especially weight initialization (see Fig. 2a). As shown in Table 1, both methods reduce the stability gap. We observe that weight initialization also greatly reduces the plasticity gap. Fig. 2b shows the average performance on ImageNet- during the rehearsal sessions for hard vs. soft targets, which reveals that hard targets increase the stability gap to a greater extent than dynamic soft targets.
8

---

Published in Transactions on Machine Learning Research (09/2024)
14
Hard Targets
Vanilla w/ Weight Inil
Dynamic Soft Targets
Vanilia w/ Solt Targets
12
Weight Initialization
Top-1 Accuracy (96)
77.5
Vanilla w/ Hard Targets
Top-1 Accuracy [96
76
10
LoRA
77.0
74
Output Layer Only
Vanilla
76.5
Entire Network
72
76.0
2
70
200
400
600
75.5
200
400
600
0
200
400
600
Number Iterations
Number of Iterations
Number of Iterations
(a) Loss at output layer
(b) Hard vs soft largets
(c) Plastic layers
Figure 2: Mitigation methods averaged over rehearsal sessions during CIL. (a) The loss on new classes
when only training the output layer, which reveals soft. largets and data-driven weight initialization greatly reduce the initial loss. (b) Accuracy on ImageNet-1K for hard vs. soft targets, which shows that soft targets reduce the stability gap. (c) Network plasticity increases the stability gap.
Table 2: Average CIL Results. Results after learning ImageNet-11K followed by Places365-LT over 5 rehearsal sessions. denotes average accuracy (%) over rehearsal sessions and is final accuracy (%) on all 1365 classes. a stands for final accuracy (%) on ImageNet- 1K only. #P denotes the trainable parameters in Millions. We report the average over 6 data orderings with standard deviation (+).
Method #P1
SA 1
PA 4
CKs
a'
P
Joint
5.08
77.58
70.69
Vanilla
5.08
0.020
=0 0017
0.385 +0 0091
0.031
+0 0015
75.81
+0.1585
71.68
10.1236
67.94
+0.1721
Output Layer
0.53
0.021
10.0011
0.473 10.0250
0.032
10.0009
75.79
10.2875
71.28
10.1410
67. 67.68 10.2710
SGM
1.45
0.001
=0.0012
0.087 =0.0082
0.002 =0.0007
77.70 =0.0509
73.71 =0.0763
70.31 =0.0882
Hypothesis 2. Our second hypothesis was that the stability gap in CIL is caused by excessive network plasticity, which we tested by using LoRA and OOCF. Fig. 2c shows the average results on ImageNet-1KK across the 5 rehearsal sessions for LoRA vs. when only the output layer, top 8 blocks (vanilla), or entire network are trainable. This reveals that plasticity plays a major role in the stability gap; however, this does not translate directly into the number of trainable parameters since LoRA includes the output layer but exhibits smaller decrease in performance than training only the output layer. Unlike others, LoRA fully recovers old performance. As seen in Table 1, both OOCF and especially LoRA reduce the stability gap.
Interim Conclusions. Our experimental results support both hypotheses. LoRA mitigates the stability and continual knowledge gaps the most. Our data-driven weight initialization greatly improves the plasticity. This aligus with the prior study that suggests that weight. initialization is critical for plasticity in DNN (Lyle et al., 2023). We find that SGM, a method that combines the approaches used to test these hypotheses, greatly reduces the stability gap.
CIL with Multiple Data Orderings. To ensure the robust ness of our findings, we extend our experiments to multiple data orderings. In addition to studying SGM and vanilla rehearsal, we examine only training the output layer during rehearsal. The averaged results across 6 data orderings are given in Table 2. We find that SGM consistently mitigates the stability gap. Compared to vanilla, SGM provides 20 more stability, 4.4x more plasticity, and 15.5x more continual knowledge. SGM also outperforms training only the output layer in all criteria. This indicates that updating representations in hidden layers besides the output layer using SGM is critical for learning new and retaining old knowledge. Moreover, SGM balances stability and plasticity (see Fig. 6).
9

---

Published in Transactions on Machine Learning Research (09/2024)
5.2.2 Learning Efficiency
80
400
One of our goals in studying CL and the stability gap is to enable more computationally efficient training. To study whether SGM achieves this goal, we evaluated performance during CIL, where we measured performance on old and new classes every 10 iterations. For new classes, we measured the number of updates and FLOPs (floating-poin) operations)
23.70
22.74
350
70
29.45
21.92
Number of Updates [K]
17.95
300
60
Vanilla TFLOPs
Vanilla Updates
TFLOPS
250
needed to achieve 99% of the best accuracy.
SGM Updates
49.11
As shown in Fig. 3, SGM learns new classes much faster than vanilla rehearsal, and on average, it is 61.8% more efficient in terms network updates. Moreover, because the curve shows a decreasing trend as learning progresses this means that SGM becomes a more efficient learner over time, with fewer updates and TFLOPs for the current. batch than pre- vious batches being needed. For old class performance, measured the number of iterations during rehearsal required to recover the performance of a joint model (upper bound) on ImageNett-IK for vanilla and SGM. As shown Table 3, SGM requires far fewer iterations to recover old performance whereas vanilla rehcarsal (without SGM) fails recover old performance fully. Compared to joint model, SGM provides 16.7x speedup in number of network updates and 31.9x speedup in TFLOPs (see Fig. 5). Moreover, SGM requires 18x less training time than the joint model on the samne hardware.
50
47.40
SGM TFLOPs
200
44.59
49.86
40
150
43.22
100
30
73
146
219
292
365
Number of New Classes
Figure 3: Speed of acquiring new knowl- edge. SGM requires fewer updates and TFLOPs than vanilla to reach 99% of the best accuracy new classes (highlighted).
Table 3: Speed of Recovering Old Knowledge. SGM compared to vanilla (without SGM) for the number of iterations needed to recover 97%, 98%, 99%, 100% of the accuracy ImageNet-1K for joint model as each of the 5 tasks (denoted by T, where i E {1,5}) from Places365-LT is learned during CIL. A hyphen indicates the model did not recover performance whereas zero means there was no stability gap.
Recovery
With SGM
Without SGM
T1
T2
T3
T;
T1
T2
T3
T4
T;
100%
260
0
70
50
80
99%
110
0
70
0
70
98%
90
0
70
70
450
110
97%
90
0
70
70
450
360
6 Additional Experiments: SGM's Generality
This section expands on the versatility and efficacy of the SGM method by exploring its application across various settings and constraints. First, we show how SGM performs the IID CL setting in Sec. 6.1. Next, we study storage-constrriined rehearsal both offline CL. (Sec. 6.2) and online CL (Sec. 6.3) scenarios with CIL. data ordering. Finally, we summarize additional supporting studies in Sec. 6.4. Additional implementation details and dataset details are given in Appendix C and B.
6.1 IID Continual Learning
To understand SGM with rehearsal would be useful for other CL data distributions, we examine its behavior in an IID ordering where each of the CL batches contains randomly sampled classes from Places365-LT datasct. In this experiment, we use ConvNexttV2 learning (SSL). During IID CL, the model sequentially learns incremental batches of data from Places365. LT where each incremental batch contains 12500 examples. During rehearsal, the model is updated over 600
pre-trained on ImageNet-1K using self-supervised
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: IID CL Results. Results after learning ImageNet- followed by Places365-LI over rehearsal sessions. j denotes average accuracy (%) over rehearsal sessions and a is final accuracy (%) on all 1365
classes. a stands for final accuracy (%) on ImageNet- 1K only, #P denotes trainable parameters in Millions. We report the average over 5 data orderings with standard deviation (+)).
Method
#P1 4
SA
PA J
CKA +
+
Joint 5.08
77.58
70.69
Vanilla 5.08
0.014
+0.0004
0.173 +0.0056
0.034 +0.0005
76.12 1007
68.45
+0.0470
68.77 #0.0517
SGM
1.45
-0.004 +0.0005
0.131 +0.0017
0.003 +000004
77.84 +0002344
70.81 #0.0151
71.23 +0.0664
Table 5: Offline CL. SGM's generality results for CIL on combination of ImageNet-1K and Places365- Standard. SGMf and SGM$ denote variants of DERpp and GDumb respectively when SGM is integrated with them. j denotes average accuracy (%) over rehearsal sessions and a is final accuracy (%) on all 1365 classes. #P denotes the trainable parameters in Millions.
Method
#P +
Storage Constraint. 192K Samples
Storage Constraint 24K Samples
Sat +
Pa
CK& + p
0: f
Sat +
Pa +
CKA +
#
o
Joint
5.08
65.37
65.37
DERpp
5.08
0.142
0.109
0.126
62.53
53.38
0.209
0.109
0.187
57.25
44.74
SGMf
1.45
0.071
0.091
0.061
67.41
57.28
0.086
0.095
0.074
66.29
55.98
GDumb
5.08
0.133
0.110
0.120
62.85
54.72
0.224
0.114
0.202
55.54
43.10
SGMt
1.45
0.053
0.095
0.046
68.50
59.24
0.073
0.098
0.065
66.88
57.18
minibatches, where the minibatch consists of 128 samples with 50% randomly selected from the current CL batch and 50% drawn from data seen in earlier CL batches and ImageNet-1K.
In this experiment, we omit storage constraints to solely focus on mitigation methods without the influence of other variables. Our results are summarized in Table 4. In terms of final accuracy, SGM achieves a final accuracy of 71.23%, outperforming vanilla rehearsal's 's 68.77% accuracy. Surprisingly, SGM even surpasses the jointly trained model's 70.69% accuracy, resulting in negative stability gap, which indicates knowledge transfer from new classes to old classes. In contrast, we found there was a small stability gap in CIL ordering (see Table 2), likely due to the dissimilarity among subsequent batches.
6.2 Storage Constrained Offline Continual Learning
To study SGM's efficacy under storage constraints, we combine it with two popular rehearsal methods, DERpp (Buzzega et al., 2020) and GDumb (Prabhu et al., 2020), under varied storage constraints while using identical configurations. For this, we use ConyNeXtV2 SST. To study varied storage constraints on large-scale data stream, we combine the TmageNet-1K (1.2M images) dataset with another large-scale dataset, Places365-Standare (1.8M images). This allows us to test more restrictive storage constraints on a data stream consisting of 3 million images.
pre-trained on ImageNet-1K using
After being pre-trained on ImageNet-IK,, a model sequentially learns incremental batches of data (73 classes per batch) from Places365-Standard in offline CL with CIL ordering. Each rehearsal is compute- constrained where the model is updated over 1200 minibatches. Each minibatch consists of 256 samples where 50% are selected randomly from the current CL batch and 50% from data seen in earlier CL batches and ImageNet-1K. During learning a new batch, the model rehearses old data from storage which is bounded by a maximun number of samples e.g., 192K and 24K corresponding to 6.4% and 0.8% of the entire dataset (ImageNet and Places combined), respectively. As shown in Table 5, SGM improves each method's per- formance in all criteria. When the storage bounded by 24K samples, SGM improves final accuracy by absolute 11.24% (DERpp) and 14.08% (0s%%GoDo (GDumb) and provides 2.4x and 3.1x more stability for DERpp and GDumb, respectively.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Online CL. Results when SGM combined with REMIND (denoted by SGMf) for CIL on ImageNet-IK and CUB-200. μ denotes average accuracy (%) over rehearsal sessions and final accuracy (%) on all 1200 classes, #P denotes trainable parameters in Millions.
Method
#P
Storage Constraint 80K Samples
Storage Constraint 20K Samples
Sa
PA $
CKA +
μ
t
SA +
PA
CKA
a
Joint
5.08
75.99
75.99
REMIND
5.08
0.146
0.837
0.162
64.15
62.35
0.166
0.844
0.183
62.58
60.37
SGM
1.45
0.034
0.675
0.049
72.81
69.67
0.042
0.695
0.057
72.20
69.01
6.3 Storage Constrained Online Continual Learning
We also assess SGM's efficacy in an online CL setting using a stale-of-the-art online CL method, RE- MIND (Hayes et al., 2020). Using ConvNexttV- with the CIL data ordering, where we combine SGM with REMIND. Since performing online CL experi- ments on large-scale datasets is computationally expensive, we choose small dataset, CUB-200 as the 2nd dataset after ImageNet-IK. CUB-200 is more challenging than other small datasets (Lee et al., 2023). After being pre-trained on ImageNet-IK, a model learns CUB-200 in sample-by-an by-sapll manner, i.e., after roceiving a new sample, it takes one SGD stop using 51 samples (50 old samples and 1 new samplc). Storage is constrained by a maximum number of samples, i.e., 80K and 20K, corresponding to 6.2% and 1.5% of the entire dataset (ImageNet and CUB combined), respectively. As shown in Table 6, SGM combined with REMIND outperforms standalone REMIND by large margins in all metrics. When storage is bounded by 20K samples, SGM improves REMIND's final accuracy by absolute 8.64% and reduces REMIND's stability gap by a factor of 3.95x.
we conduct stosage- CL experiments
6.4 Supporting Studies
We include additional supporting studies in the Appendix and summarize the findings here.
Non-Rehearsal Methods. While non-rehearsal methods are less performant for CL, it is interesting to study SGM when combined with non-rchearsal methods to determinc if our findings are consistent. During CIL of ImageNet-1I and Places365-LT, SGM also benefits a non-rehearsal method, LwF (Li Hoiem, 2017) with an absolute gain of 35.24% in final accuracy and 2.6x reduction in the stability gap (see Appendix E).
Class-Balanced Rehearsal. Since our main experiments are based conventional rehearsal without class balance, also conduct experiments using class balanced rehearsal. Compared to vanilla rehearsal, SGM reduces the stability gap by 7.3x when both use class-balanced rehearsal. SGM also achieves continual knowledge transfer (see Appendix D.2).
Supervised Pre- Pre-Training. In our main results, SGM reduces the stability gap using the self-supervised pre-trained ConvNeXtV2 model, and self-supervised models are known to perform better CL (Gallardo et al., 2021). We also conduct experiments with the supervised pre-trained NeXVV (Liu et al., 2022) to examine how SGM performs without a self-supervised backbone. We find that SGM is effective without the self-supervised backbone. Compared to vanilla. rehearsal, SGM achieves 6x more stability, 3.9x more plasticity, and 35x more continual knowledge transfer when both use ConvNeXtV1 (sce Appendix D.3).
Vision Transformers. We also study the behavior of SGM with ViTs, which now rival CNNs. Compared to vanilla rehearsal, SGM reduces the stability gap by 2.4x when both use a pre-trained ViT (see Appendix D.4).
7 Limitations & Future Work
Our work is built upon the assumption that a well-trained DNN is provided and performance on the original tasks must be maintained or improved. If this assumption violated, as seen some CL papers that do not start with pre-trained models, SGM's dependence on LoRA would impair its utility. Although most of the components of SGM, such as weight initialization, dynamic soft targets, and OOCF, can be readily applied to
12

---

Published in Transactions on Machine Learning Research (09/2024)
DNNs trained from scratch, LoRA cannot be used because DNN layers without LoRA would remain frozen and unlearned. Applying SGM in this setting would require variant of ToRA where LoRA could be used for all layers from the start of training such that the DNN rivaled jointly trained model without LoRA. We used pre-trained ConvNeXtVI, ConvNeXtV2,, and MobileViT DNNs, which are amenable to LoRA. Future work could investigate different LoRA variants and network adapters (Han el al., 2024) with SGM. would also be interesting to study the efficacy of SGM for meeting the needs of CL for embedded devices Hayes & Kanan (2022).
Our study focused image classification. Future work could explore SGM's efficacy for regression, object detection (Acharya et al., 2020; 2019), reasoning Hayes & Kanan (2021), and semantic segmentation (Zhang et al., 2022). CL methods developed only on small datasets often do not scale to larger datasets such as ImageNet-1K (Zhou et al., 2023), so we studied a combination of ImageNet- 1K and Places365. Due to computational limitations and lack of readily available large-scale image classification datasets, we could not study CL for more than 1365 classes. In futuure work, asscssing how well SGM scales with increasing dataset sizes and class counts would be valuable. We hope our work will inspire future research to find better approaches to prevent model decay due to concept drift.
8 Conclusion
Although there has been remarkable progress in mitigating catastrophic forgetting, almost all CL methods remain unsuited for practical applications such addressing model decay (Verwimp ct al., 2024; Harun et al., 2023a). To eliminate the need for frequent retraining of deployed DNNs, CL has to combat model decay, mitigate the stability gap, and be more computationally efficient than retraining from scratch in large-scale CL settings. Focusing on this scenario, we showed that SGM meets these criteria. With the growing energy usage of deep learning models (Luccioni et al., 2022; Patterson et al., 2021; Wu et al., 2022), reducing the need for retraining from scratch could significantly contribute to reducing carbon emissions associated with training DNNs. Our work aligns CL with these real-world challenges, and we hope it encourages the CL community to focus them.
Acknowledgments
This work was supported in part by NSF awards # 19096696, #2326491, #2125362, and #2317706. The views and conclusions contained herein are those of the authors and should not be interpreted as representing any sponsor's s official policies or endorsements. We thank Jhair Gallardo, Junyu Chen, Shikhar Srivastava, and Robik Shrestha for their feedback and comments on the manuscript.
References
Manoj Acharya, Karan Jariwala, and Christopher Kanan. Vqd: Visual query detection in natural scenes. In NAACL, 2019.
Manoj Acharya, Tyler L Hayes, and Christopher Kanan. Rodeo: Replay for online object detection. In BMVC, 2020.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 339-154, 2018.
Eden Belouadah and Adrian Popescu. I12m: Class incremental lcarning with dual memory. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 583 592, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in newral information processing systems. 33:15920- 15930, 2020.
Francisco M Castro, Manuel J Marin-Jiméncz, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End- lo-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp- 233-248, 2018.
Arslan Chaudhry, Punect K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Ricmannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532 547, 2018
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories, Xrviv preprint arXiv:1902. 10486, 2019, 2019.
Matthias De Lange, Gido de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. In ICLR, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl ar arXiv:1810.044 2018.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5138 5146, 2019.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Dision-ECCV 2020-16th 16th European conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, volume 12365, pp. 102. Springer, 2020.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. arXiv preprint arXiv:2111.11390 2021.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part XI 16, pp. 386 402. Springer, 2020.
Alex Egg. Ouline learning for recommendations at grubhub. In Proceedings of the 15th ACM Conference Recommender Systems, pp. 569-571, 2021.
Jhair Gallardo, Tyler I, Hayes, and Christopher Kanan. Self-supervised training enhances online continual learning. In British Machine Vision Conference (BMVC), 2021.
João Gama, Indré Zliobaité, Albert Bifet, Mykola Pochenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaplation. ACM compuling surveys (CSUR), 46(4):1-33 37, 2014.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, A uni- fiod contimual learning framework with general parametor-efficient tuning. International Conference on Computer Vision (ICCV), 2023.
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation online continual learning: A new paradigm. In CVPR, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, and Christopher Kanan. How efficient are today's continual learning algorithms? In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2430 2435, June 2023a.
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, Ronald Kemker, and Christopher Kanan. STESTA: Efficient online continual learning with sleep. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL
Md Yousuf Harun, Jhair Gallardo, and Christopher Kanan. Grasp: A rehcarsal policy for efficient online continual learning. arXiv preprint arXie:2308.196/ 18646, 2023c.
Md Yousuf Harun, Kyungbok Lee, Jhair Gallardo, Giri Krishnan, and Christopher Kanan. What variables affect out-of-distributio generalization in pretrained models? arXiv preprint arXiv:2405.1501, 2024.
Tyler L Hayes and Christopher Kanan. Selective replay enhances learning in online continual analogical reasoning. In CLVISON, pp. 3502 3512, 2021.
Tyler I. Hayes and Christopher Kanan. Online continual learning for embedded devices. In CoLLAs, 2022.
Tyler Hayes, Ronald Kemker, Nathan D Cahill, and Christopher Kanan. New metrics and experimental paradigms for continual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2031-20344 2018.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent. calastrophic lorgetting. In European, Conference on Compuler Vision, pp. 166-483. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEFE international conference computer vision, pp. 1026 1034, 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incre- mentally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion, pp. 831 839, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0m 096855 2021.
Chip Huyen. Designing machine learning systems. O'Reilly Media, Inc.", 2022a.
Chip Huyen. Introduction to streaming for data scientists. https://huyenchh com /2002/2000//3 stream- -twppppessiii---orraaaaaaaaaaaae html, 2022b.
Nishant Jain and Pradeep Shenoy. Instanco-conditional timescales of decay for non-stationary learning.
arXiv e-prints, pp. MrXiv-2211. 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335) -3326 2017.
Kuan-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally continual learning? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6485-6493, 2023.
Zhizhong Li and Derck Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 2947, 2017.
Zhuang Liu, Hanzi Mao, Chao Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A con- vnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2022.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in an open world. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2537 2546, 2019.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Xiv preprint arXiv:1706.088. 08840, 2017.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, (1(12):2346-2388 2018.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. arXie preprinl arXiv:2211.0200 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity in neural networks. In International Conference on Machine Learning, pp. 23190-241 29211... PMLR, 2023.
Sasu Mäkinen, Henrik Skogström, Eero Laaksonen, and Tommi Mikkonen. Who needs mlops: What data sci- entists seek to accomplish and how can mlops help? In 2021 IEEE/ACM 1st Workshop on AI Engineering- Software Engineering for AI (WAIN), pp. 109-112. IEEE, 2021.
Ankur Mallick, Kevin Hsieh, Behnaz Arzani, and Gauri Joshi. Matchmaker: Data drift mitigation in machine learning for large-scale systems. Proceedings of Machine Learning and Systems, 4:77-94, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp- 109-165. Flsevier, 1989.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ran- pac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2024.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-frieudly vision transformer. In International Conference on Learning Representations.
Sanket Vaibhav Mchta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training lifelong learning. Journal of Machine Learning Research, ((214):1-50 2023.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascamu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXie preprint arXiv:2202. 00275, 2022.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodriguez, Md Rifat Arefin, Arthur Douillard, Trina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. In Conference on Lifelong Learning Agents, pp. 60-91. PMLR, 2022.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54 71, 2019.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Mand Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiw:2104.1033 2021.
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real- time continual learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10203-10209. IEEE, 2020.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 524-540, 2020.
Ameya Prabhu, Zhipeng Cai, Puncet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online con- tinual learning without the storage constraint. arXio preprint (ll:2305.0925. 2023a.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puncet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Vinay Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden repre- sentations and lask semantics. In ICLR, 2021a.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgelting in neural networks. In International Conference Learning Representations, 2021b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 2015.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Casiante-Bonilla, Donghyun Kim, Assaf Ar- belle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11909 119199, 2023.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. arxiv. arXic preprint arXiv:1708.0711 2017.
Alexey Tsymbal. The problem of concept drift: definitions and related work. Computer Science Department, Trinity College Dublin, 106(2):58, 2004.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, pp. 1-13, 2022.
Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhirecsha Kudithipudi, Christoph H Lampert, et al. Continual learning: Applications and the road forward. TMLR, 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognétion, pp. 139-149, 2022b.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Sain- ing Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint arXiv:2301.00800 2023.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795-811 2022.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEFE/CVV CVV Conference on Computer Vision and Pattern Recognition, pp. 374 382, 2019.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremen- tal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
pp.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Eighth International Conference on Learning Regresenlations, ICLR 2020. ICLR, 2020.
Chang- Bin Zhang, Jia Wen Xiao, Xialei Liu, Ying- Chen, and Ming-Ming Cheng. Representation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 7053 7064, 2022.
Wenxuan Zhang, Youssef Mohamed, Bernard Ghanem, Philip Torr, Adel Bibi, and Mohamed Elhoseiny. Continual learning on a diet: Learning from sparsely labeled streams under constrained computation. The Twelfth International Conference on Learning Representations, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6); 1452-1464, 2017.
Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class- incremental learning: A survey, arXiv preprint arXiv:230..0364 03648, 2023.
Yue Zhou, Yue Yu, and Bo Ding. Towards mlops: case study ml pipeline platform. In 2020 International conference artificial intelligence and computer engineering (ICAICF), pp. 494 500- IEFE, 2020.
Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of-distribution detection with typical features. Advances Neural Information Processing Systems, 55:20758-20769) 2022.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Appendix
A
Implementation Details and Additional Results
We organize additional supporting experimental findings as follows:
Appendix provides details on the datasels used in this paper.
Appendix C provides additional implementation and training details for all of the methods.
Appendix provides additional CIL experiments and results with rehearsal, including an analysis of learning curves, studying alternative sampling strategies for rehearsal, using non-self-superviseed backbone CNN, using vision transformer, and using balanced dataset. We find that SGM works well across these experiments and analysos compared to vanilla rehearsal.
Appendix E studies the behavior of our stability gap mitigation method when used with Learning without Forgetting (LwP), a popular regularization method used in CL. We find that our method greatly improves results, illustrating that the mitigation strategy is not specific to rehearsal.
Appendix demonstrates that our evaluation metrics facilitate a more accurale and meaningful comparison between different CL models.
Appendix G illustrates the process of updating soft targets.
Appendix H includes additional discussion.
B
Dataset Details
In this paper, we use two large-scale datasets (ImageNet-11 and Places36--Sanndard) and two small- scale datasets (Places365-LT and CUB-200). ImageNet-1K (Russakovsky al., 2015) has 1.28 million images from 1000 categories, each with 732 1300 training images and 50 validation images. Places365- LT (Liu et al., 2019) is a long-tailed dataset with an imbalanced class distribution. It is long-tailed variant of the Places-2 dataset (Zhou et al., 2017). Places365-LT has 365 classes and 62500 training images with to 4980 images per class. For its test set, we use the Places365-LT validation set from (Liu et al., 2019) which consists of a lotal of 7300 images with a balanced distribution of 20 images per class. Places365- Standard (Zhou al., 2017) has over 1.8 million training images from 365 classes with 3068 5000 images per class. We use the validation set consisting of 100 images per class to test the models. CUB-200 (Wah et al., 2011) has RGB images of 200 bird species with 5994 training images and 5794 test images.
C Additional Implementation Details
In this section, provide additional implementation details for the models.
Main Experiments. For both CIL and IID experiments, we train SGM with rehcarsal, vanilla rehearsal, and output layer only using cross-entropy loss for 600 iterations per rehearsal session. During each ileration model is updated on 128 samples. All methods use the same ConyNeXtV2 backbone 3 use AdamW optimizer with weight decay of 0.05 and initial learning rates of 10-3 (SGM and vanilla) and 10-2 (output layer only). The learning rate is reduced in earlier layers by a layer-wise decay factor of 0.9. The learning rate scheduler is not applied for vanilla and output layer only due to poor performance. On the other hand, SGM uses OneCycle learning rate scheduler (Smith & Topin, 2017). The joint model (upper bound) is trained for 12500 iterations on all data i.e., ImageNet-1K 1K and Places365-LT combined using an initial learning rate of 10-1 without a scheduler, For all experiments, we set the rank of the LoRA weight matrices to 48. In all cases, all metrics are based on Top-1 accuracy (%). Most CL experiments including those in Sec. 5, and Sec. 6.1 adhere to the aforementioned settings unless otherwise noted.
3Pre-trained weights are available here: https: http:/ghht....ceeeeaoaeareeraa/a
19

---

Published in Transactions on Machine Learning Research (09/2024)
Storage Constrained Offline CL with DERpp and GDumb. We describe settings used in Sec. 6.2 where we combine SGM with DERpp and GDumb while using identical settings e.g., the same ImageNet- 1K pre-trained ConvNeXt V2 Femnto network and the same optimizer settings. Each model pre-trained on ImageNet-1K learns Places365-Standare in 5 batches subsoquently (73 categories per batch). Each rehearsal session performs a lotal of 1200 iterations with 256 samples per iteration. DERpp employs distillation and regularization along with rehearsal to prevent catastrophic forgetting. It regularizes loss on old samples and uses an additional distillation loss on logits of old samples for promoting consistency. We set coelficients a = 0.1 and [ 0.9 for distillation and regularization respectively. GDumb randomly removes sample from the largest class when the buffer reaches its maximum capacity and maintains class-balanced buffer. For all methods, the buffer is bounded by a maximum number of samples (80% ImageNet-1K + 20% Places365- Standard). DERpp, GDumb, and SGM use an initial learning rate of 1 x 10-3, 1 x 10-3, and 1.5 x 10-3, respectively, for batch size 256. The joint model (upper bound) uses an initial learning rate of 10-2 and 12K iterations with 256 samples per itcration. We assess performance during rehearsal every 100 mini-batches to compute the metrics.
Class-balanced Rehearsal. For class-balanced rehearsal experiments in Appendix D.2, SGM with re- hearsal and vanilla rehearsal use an initial learning rate of 10-3 and 10-4, respectively.
Non-SSL Backbone CNN. For non-SSL, backbone experiments with ConyNeXt V1-Tiny (Liu et al., 2022) in Appendix D.3, initial learning rates for SGM with rehearsal, vanilla rehearsal, and joint model are 4x 10 10-3, 3 x 10-3, and 10 respectively. ConvNeXt V1-Tiny has been pre-trained on Image using supervised learning"
VIT Backbone. For VIT backbone experiments in Appendix D.4, we select MobileViT-Small (Mehta & Rastegari) (5.6M) pre-trained on ImageNet-1K using supervised learning 9. For universal feature extraction, we freeze the first 8 blocks including stem, 6 MobielNetV2 blocks, and 1 MobileViT block. We keep the remaining blocks (1 MobileNet V2 block and 2 MobileViT blocks) and layers (1 CNN layer and linear layer) plastic which correspond to 96.4% of the total parameters. We apply LoRA (rank-48) to query, key, and value projection matrices in the self-attention module of MobileViT transformer blocks. All methods use the AdamW optimizer with a weight decay of 0.01. Vanilla rehearsal and SGM with rehearsal use an initial learning rate of 3 x 10-3 and 4 x 10-3, respectively. The initial learning rate for the joint model is 10-2. Places365-LT data is learned over 5 rehearsal sessions (73 classes per session) where each session includes 1200 iterations with 32 samples per iteration. The joint model is trained for 25K iterations with 64 samples per iteration. All other settings are identical to those in the main experiments.
Balanced (Non-LT) Dataset. For experiments with a balanced datasel in Appendix D.5, SGM with rehearsal and vanilla rehearsal use an initial learning rate of 1.5 10-3 3 and 10-3, respectively. Fach model is pre-trained on ImageNet-1K and then learns Placee365-Slandard in 5 batches subsequently (73 categories per batch). Each rehearsal session has total of 1200 iterations with 256 samples per iteration. We set f f= 0.5 for compute budget. Hence, at the end of CL. the total number of SGD model updates is 50% of the total number of samples in the entire datasct (ImageNet and Placos-Standard combined). We assess performance during rehearsal every 100 minibatches to compute the metrics. The joint model uses an initial learning rate of 10-2 and 12K iterations with 256 samples per iteration.
Storage Constrained Online CL with REMIND. In Appendix 6.3, we use identical settings and hyper- parameters for both REMIND and REMIND + SGM methods. We use ImageNet- pre-trained Con V2 Femto with similar network configurations and LoRA configurations use the AdamW optimizer and REMIND's ' default per-class learning rate scheduler. We set the initial Iearn- ing rate to 1 x 10 the final learning rate to 1 x 10-5, and weight decay to 0.05. Following REMIND, we perform rehearsal with a mini-batch of 51 samples including 50 old samples and new sample. Fach method learns the CUB-200 dataset in cample-by-samplee manner. For all methods, the buffer is bounded by a maximum number of samples (75% - 93% ImageNet).
used in main experiments. We
Non-rehearsal Methods. In Appendix E, LwF has similar configurations as vanilla rehearsal except for the initial learning rate (6 x 10-5). LwF + SGM has similar configuration as SGM with rehearsal except
'The pre-trained weights are available here: https://github. wwwp/ppppp//.ccc/c/ccccoope cynets
The pre-trained weights are available here: httpps:/gittab.cc///cceeeeed http:///...ccbbeeeeer/////a.aeee
20

---

Published in Transactions on Machine Learning Research (09/2024)
78.5
ImageNet-1K Accuracy [%6]
78.0
77.5
77.0
SGM
Vanilla
mmm
76.5
Joint Model
76.0
75.5
75.0
600
1200
1800
2400
3000
Number of Iterations
Figure 4: Stability gap over all rehearsal sessions. After pre-training on ImageNet-IK, the model learns 365 new classes from Places365-LT over five rehearsal sessions (600 iterations per rehearsal session). SGM quickly recovers old performance at the beginning of CL whereas vanilla. fails to obtain full recovery. After each rehearsal session (vertical dotted gray line), the final top-1 accuracy (%) is highlighted by diamond (SGM), star (joint model), and square (vanilla). The joint model (upper bound) is jointly trained on ImageNet and seen CL batches from the Places dataset.
the initial learning rate (2 x 10-4). During each iteration model is updated on 64 new samples without any rehearsal of old samples. Our mitigation approaches e.g., dynamic soft targets, data-driven weight initialization, OOCF, and LoRA are applied for LwF similarly they are applied for rehearsal methods in main experiments.
Weight Initialization. For SGM, we use our data-driven weight initialization (Sec. 4) to initialize the weights the final output layer. For other compared methods, we use He initialization (He et al., 2015) to initialize the weights in the final output layer.
All other settings adhere to the above-mentioned general settings for the main experiments unless otherwise mentioned. Hyperparameters are tuned to maximize performance for each method.
Compute. We ran all experiments the same hardware with single GPU (NVIDIA RTX A5000).
D Additional CIL Analysis & Experiments
In this section, we conduct additional analysis of the CIL experiments in the main results as well present additional experiments. Implementation details and dataset details are given in Appendix C and B.
D.1
Qualitative Analysis
In this section, we present a qualitative analysis to see how SGM mitigates the stability gap and enhances computational efficiency.
D.1.1 Stability Gap Over All Rehearsal Sessions
In our main text, our figures are averaged across rehearsal sessions. In Fig. 4, we instead present all of the learning curves in sequence, where we denote when the next batch containing new classes is received.
When rehearsal begins, accuracy on ImageNet-1K for vanilla rehcarsal drops dramatically and gradually decreases throughout the rehearsal sessions. In the end, vanilla fails recover the original performance using total of 3000 iterations. Tn contrast, SGM shows better performance throughout rehearsal sessions
21

---

Published in Transactions on Machine Learning Research (09/2024)
78.0
78.0
Rehearsal with SGM
Rehearsal SGM
ImageNet Top-1 Accuracy
77.8
Conventional Rehearsal
ImageNet Top-1 Accuracy
77.8
Conventional Rehearsal
Upper Bound (x=12.5k)
Upper Bound (x=8.24k)
77.6
77.6
16.7x Speedup
31.9x Speedup
77.4
77.4
77.2
77.2
77.0
77.0
76.8
0
500
1000
1500
2000
2500
3000
76.8
500
1000
1500
2000
Number Iterations
TFLOPs
(a) Network Updates
(b) TFLOPs
Figure 5: Computational efficiency. Our method, SGM, provides a 16.7x speedup in the number of net- work updates and 31.9x speedup TFLOPss compared to joint model (upper bound) with the combined 1365 class dataset (ImageNet-IK and Places365-LT combined). For SGM and conventional rehearsal, we show the stability gap in the learning curve averaged over rehearsal sessions.
78
SGM
SGM
Vanilla
76
Vanilla
Top-1 Accuracy [96]
Top-1 Accuracy [96]
76
Top-1 Accuracy [%6]
74
SGM
74
Vanilla
72
72
20
68
1073 1146
1219 1292 1365
1073 1146
1219
1292 1365
1073 1146 1219 1292 1365
Number Classes
Number Classes Seen
Number Classes Seen
(a) Accuracy on new classes
(b) Accuracy on old classes
(c) Accuracy on all classes
Figure 6: Stabillty-plasticity. After pre-training on ImageNet-IK, the model learns 365 new classes from Places365-LT over five batches (73 new classes per batch) in CIL setting. The accuracy is averaged over 6 runs and shaded region indicates standard deviation.
with reduced stability gap and full recovery compared to the joint model. Models are evaluated every 10 iterations. After each rehearsal session, SGM outperforms vanilla and matches or exceeds the accuracy of the joint model (upper bound).
D.1.2 Computational Efficiency
To mcasure computational efficiency, consider two criteria: 1) number of network updates (Sec. 3.1) and 2) FLOPs (Doating-poinl operations). For FLOPs analysis, use DeepSpeed with the same GPU across compared models. We perform computational analysis for CIL on combination of ImageNet-Il -K and Places365-LT datasets.
As shown in Fig. 5, SGM provides a 16.7x speedup in network updates and a 31.9x speedup in TFLOPs compared to a joint model (upper bound). SGM fully recovers the original performance on ImageNet-Ii 1K using 600 iterations per rehearsal session whereas conventional rehearsal fails to do that using 3000 iterations per rehearsal session.
Ohttps://github http://hhhhho.coo/tttttr////eeaa
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 7: Class Balanced Rehearsal. model pre-trained on ImageNet-1K learns Places365-LT 5 rehcarsal sessions. Here μ denotes average accuracy (%) over rehearsal sessions and a is final accuracy (%) on 1365 classes. # #P denotes total number of trainable parameters in Millions.
Method
#P4
Sa +
Pa L
CK +
a f
Joint Model (Upper Bound)
5.08
70.69
Vanilla Rehearsal
5.08
0.022
0.316
0.021
72.24
69.03
SGM (Ours)
1.45
0.003
0.089
-0.003
74.00
70.61
Table 8: CIL without Self-Supervised Pre-Training. This table shows results from ConvNeXt V1- Tiny pre-trained on ImageNet-1K using supervised learning, which then learns Places365-LT in rehearsal sessions in CIL setting. Here y denotes average accuracy (%) over rehearsal sessions and is final accuracy (%) on 1365 classes. #P denotes total trainable parameters in Millions.
Method
#P
Sa +
PA
CK&t
f
Joint Model (Upper Bound)
27.00
74.16
Vanilla Rehearsal
27.00
0.030
0.396
0.035
74.73
70.67
SGM (Ours)
3.53
0.005
0.102
0.001
77.48
73.92
D.1.3 Stability
In Fig. 6, we also illustrate the model's accuracy on new, old, and all classes in all rehearsal sessions where SGM achicves higher accuracy than vanilla. This indicates that SGM consistently improves the model's plasticity (Fig. Ga), stability (Fig. 6b), and knowledge accumulation (Fig. Gc).
D.2 Class Balanced Uniform Sampling for Rehearsal
In our main results, we sampled randomly during rehearsal without balancing for each class. However, prior work has shown that class-balanced random sampling works significantly better than unbalanced uniform sampling for long-tailed datasets (Harun et al., 2023b). We conducted CIL experiments to examine this in our unlimited storage for rehearsal setup where we learned ImageNet-1K followed by Places365-LT.
Table shows that using class-balance rehearsal, SGM improves performance most criteria compared to previous results without class balance (Table 1). When both vanilla and SGM use class balanced rehearsal, SGM outperforms vanilla by 7.3x in stability gap, 3.6x in plasticity gap and provides continual knowledge transfer (CKA < 0).
D.3 Analysis with a Non-Self- Seef--Sluluuiieee Backbone CNN
Much of deep learning has moved toward self-supervised pre-training prior to supervised fine-tuning, espe- cially in foundation models (Devlin et al., 2018; Brown et al., 2020; Ramesh et al., 2021), since this has been shown to reduce overfitting on the pretext dataset. used for self-supervised learning and to generalize better to downstream tasks. In the main text, we used the self-supervised ConvNeXtV2 architccture. This may have enabled our system achieve higher results on Places365-LT than if the CNN was initialized from ImageNet- 1K with supervised learning. To determine if our general trends for the methods hold, we conducted another experiment with ConvNeX:V1-Ting (29M), which is pre-trained on ImageNet- without self-supervision. We conducted CIL experiments to examine ConvNeXtV1 models in our unlimited storage for rehearsal setup where we learned ImageNet-1K followed by Places365-LTT.
Experimental results Table demonstrate that SGM with a supervised backbone mitigates the stability gap and enhances performance in all criteria. Therefore the efficacy of SGM does not depend upon self- supervised pre-training.
23

---

Published in Transactions on Machine Learning Research (09/2024)
D.4 Analysis with using& Vision Transformer Backbone
In this section, we study the behavior of the system for VIT model pre-trained with supervised learning, For this, we select a lightweight transformer, MobileViT-Small (Mehta & Rastegari). MobileViT learns local and global representations using convolutions and transformers, respectively. It has a total of 5.6 million parameters and top-1 accuracy of 78.4% on ImageNet-1K3 We conducted CIL experiments to examine MobileViT models in our unlimited storage for rehearsal setup where we learned ImageNet-1K followed by Places365-LT.
Table 9 shows the comparison between vanilla and SGM when they have the same MobileViT backbone. SGM shows better performance in all criteria using 3.8x fewer parameters than vanilla rehearsal.
Table 9: Vision Transformer Backbone. A model pre-trained on ImageNet- 1K learns Places365-LT in 5 rehearsal sessions. Here y denoles average accuracy (%) over rehearsal sessions and a is final accuracy (%) 1365 classes. #P denotes the total number of trainable parameters in Millions.
Method
#P
Sa +
Pa 1
CKA
f
Joint Model (Upper Bound)
4.97
69.10
Vanilla Rehearsal
4.97
0.039
0.434
0.046
70.18
66.35
SGM (Ours)
1.30
0.016
0.140
0.016
72.09
67.96
D.5 Balanced (Non-LT) Dataset
In real-world setting, data distribution commonly long-tailed and imbalanced, hence we used Places365- LT dataset in the main results. However, our analysis holds for balanced and non-LT datasets well. We study this using Places365-Standari mitigation methods without the influence of other variables. Results in Table 10 show that SGM outperforms vanilla rehearsal in all criteria.
We omit storage constraints in this experiment to solely focus on
Table 10: Non-LT Dataset. A model pre-trained on ImageNet-1K learns Places365-Standaae in 5 rehearsal sessions in CIL setting. Here a denotes average accuracy (%) over rehearsal sessions and final accuracy (%) on 1365 classes. #P denotes the total number of trainable paramcters in Millions.
Method
#P
SA 4
PA
CKA 4
t
Joint Model (Upper Bound)
5.08
65.37
Vanilla Rehearsal
5.08
0.078
0.201
0.082
66.37
56.63
SGM (Ours)
1.45
0.054
0.091
0.047
68.47
59.21
E SGM Enhances Non-rehearsal Methods
We hepoteesized that SGM would be helpful for non-rehearsal methods as well. We, therefore, study SGM using Learning without Forgetting (LwF) (Li & Hoiem, 2017), which pioneered using knowledge distillation in CL (Zhou et al., 2023). Instead rehearsal, LwF stores a copy of the model before learning the new CL. batch to update the model with distillation. LwF has been shown to reduce catastrophic forgetting a range of CL scenarios, although it and other regularization-basde methods have not been shown to be effective in the CIL setting (Zhou et al., 2023).
We conducted an experiment compare vanilla LwF with version of LwF that uses SGM without re- hearsal during CIL of ImageNet-1K and Places365-LT. For this, we use ConvNeettV2-Fomto pre-trained on ImageNet-11K using SSL. After being pre-trained on ImageNet-1K, the model sequentially learns 5 in- cremental batches of data (73 classes per batch) from Places365-LT in offline CL with CIL ordering. The training is compute-constrained where the model is updated over 600 minibatches. Each minibatch consists of 64 new samples and rehearsal of old samples is omitted. Additional implementation details are given in A Appendix C.
24

---

Published in Transactions on Machine Learning Research (09/2024)
80
ImageNet Top-1 Acc. [%6]
LWF + SGM
LWF
60
40
20
1000
2000
3000
Number of Iterations
Figure 7: Comparison with non-rchcarsal method. The Y-axis shows average accuracy (%) of 6 runs with standard deviation (shaded region). The network is trained on ImageNet-1K and then learns 365 new classes from Places-I.T over five batches (73 new classes and 600 iterations per batch). When a new batch arrives, accuracy on ImageNet-IK for LwF plummets. LwF fails to recover performance and ends up with a large stability gap. In contrast, LwF with SGM does not plummet like LwF and shows better performance throughout the CL phase with a significantly reduced stability gap.
Overall results are given in Table 11 and learning curve is given in Fig. 7. As expected based prior results, rehearsal methods vastly outperform LwF; however, we find that SGM provides enormous benefit to LwF in terms of reducing the stability gap, resulting in increased accuracy.
Table 11: Comparison with non-rehearsal method. A model pre-trained on ImageNet-113 learns Places365-LT in 5 rehearsal sessions in CIL setting, Results are averaged over 6 runs. Here μt denotes average accuracy (%) over rehearsal sessions and a is final accuracy (%) on 1365 classes. #P denotes total trainable parameters Millions. For the non-rehearsal baseline, we select LwF that regularizes the model basod knowlodge distillation.
Method
#PL
SA 4
PA 4
CK 4
#
a t
Joint Model (Upper Bound)
5.08
70.69
Vanilla Rehearsal
5.08
0.020
0.385
0.031
71.68
67.94
SGM (Ours)
1.45
0.001
0.087
0.002
73.71
70.31
LwF
5.08
0.605
0.450
0.607
24.04
4.76
LwF + SGM (Ours)
1.45
0.236
0.072
0.235
54.87
40.00
F Reasons for New Metrics
In the main text in Sec. 3.2, we proposed new metrics that are normalized against a joint model (upper bound). Unlike earlier metrics (De Lange et al., 2023), our metrics enable a more significant and accurate comparison between different CL models. Moreover, our metrics can be used to analyze whether the CL model meeting the needs of the industry catch up the joint model retrained from scratch when new data becomes available.
In this section, we illustrate two cases using synthetic examples in Fig. 8 to build intuitions. In each plot, we show the drop in old task accuracy when updating models on a new task. We compare two CL models in terms of the stability gap i.e., drop in old task accuracy, to find a better model.
Case 1. The metric proposed earlier work (De Lange et al., 2023) measures the stability gap of a model by evaluating the largest drop in old task performance compared to the same model's performance. According their metric, CL model is considered be better than CL model 2 exhibits a smaller drop in accuracy for the old task, shown in Fig. 8(a). However, in contrast to this observation, it can be observed
25

---

Published in Transactions on Machine Learning Research (09/2024)
70
70
Old Task Accuracy (96
Old Task Accuracy (%)
60
40
50
20
20
20
70
300
300
600
# Iteration (Updating on new task)
# Iteration (Updating on new task)
CL Model
(a) Their Metric
(b) Our Metric
CL Model
Joint Model
70
70
Old Task Accuracy (96)
65
Old Task Accuracy (%)
65
55
55
50
50
65
55
15
300
600
300
600
# Iteration (Updating on new task)
Iteration (Updating on new task) (d) Our Metric
(c) Their Metric
Figure 8: Efficacy of our metric. We present a series of synthetic plots that display the performance of model old task as it learns new task. The largest drop in accuracy the old task is marked by dotted line with arrow. This drop in accuracy is also displayed sub-figures (a) and (b), our metric delivers different outcome from theirs (prior work). In sub-figures (c) and (d), unlike their metric, our metric can distinguish between two CL models.
nunber for ease of comparison. In
that CL model 2 performs better than CL model as shows a smaller drop in old task accuracy compared to CL model while employing the joint model (upper bound) as a universal reference point. This is captured by our metric and can be seen in Fig. 8(b).
Case 2. We demonstrate that the metric proposed in previous work (De Lange et al., 2023) is unable to differentiate between two Cl. models that show the same decline in old task performance, depicted in Fig. 8(c). In contrast, our metric, as illustrated in Fig. 8(d), is capable of identifying that CL model 2 (with a smaller drop) performs better than CL model 1 (with a larger drop). From this, it is evident that different CL models cannot be compared without universal upper bound.
26

---

Published in Transactions on Machine Learning Research (09/2024)
Hard targets (one-hot encoding)
Soft targets
Probability
Probability
1
3
000
K
0 1
2
3
000 K
Input
DNN
Soft targets
Class 2
Classes:
2 000 K
0.10
Softmax Output
0.02
0.81
0.01
Update targets
Figure 9: Process of updating soft targets. Here we illustrate the update mechanism of our dynamic soft targets. For each class k we maintain a probability distribution over total classes, denoted by up E TRK During iteration i, we update this vector ul the model (0t) correctly predicts the class k. the following iteration i 1, uf will serve as soft targets to train the model, l.+1- Unlike hard targets, soft targets do not enforce strict inter-class independence and prevent the stability gap.
G Soft Targets Update
In the main text in Sec. we proposed dynamic soft targets to prevent the stability gap. In this section, we illustrate how we update the soft targets in Fig. 9. Soft targets are updated using the model's predicted probabilities.
Additional Discussion
What is the connection between evaluation metrics and stability gap mitigation approaches?
In the main text in Sec. 3.2, we formalized our evaluation metrics to measure the stability gap (SA), plasticity gap (Pa), and continual knowledge gap (CKa). We found that our proposed mitigation methods in Sec. effectively mitigated the stability, plasticity, and continual knowledge gaps when evaluated using metrics from Sec. 3.2. Therefore, the evaluation metrics can be regarded the tools to mcasure the efficacy of the proposed mitigation methods.
Mitigation methods were designed to prevent factors contributing to the stability gap such as large loss at the output layer and excessive network plasticity. All three metries were used in all evaluations and experiments in the main paper and the Appendix.
27


Reviews:
[{"review_id": "uq2tN232wA", "reviewer": "Reviewer_KC3q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The paper proposes new metrics to measure the stability, plasticity, and continual knowledge gaps. \n2. The stability gap can be mitigated with the proposed weight initialization, soft targets, partly updating the neural networks with LoRA, and further limiting the plasticity by freezing output layer weights of previously seen classes.\n\n### Weaknesses\n1. The related work discussion is insufficient, and I am somehow not convinced by the motivation of the work. In my opinion, the stability gap is specifically limited to rehearsal methods, which may not be an obvious problem in other previous methods like continual learning methods combining meta-learning in online fashion [1,2,3] or dynamic architecture methods that create new network pathways or submodules for the new classes [4,5].\n2. In addition to the proposed possible reasons (e.g., the large number of parameters to be updated) for the stability gap phenomenon, the learning rate itself can cause a large drop when updating the model with new data. However, such reason is not considered. \n3. Some notations are not clear (see required changes), some related setting need more detailed introduction for broader audience, and the writing could be further improved.\n\n[1] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018.\n\n[2] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 33:16532–16545, 2020.\n\n[3] Chen, Qi, et al. \"On the stability-plasticity dilemma in continual meta-learning: theory and algorithm.\" _Advances in Neural Information Processing Systems_ 36 (2024).\n\n[4] Yoon, Jaehong, et al. \"Lifelong learning with dynamically expandable networks.\" _arXiv preprint arXiv:1708.01547_ (2017).\n\n[5] Zhenyi Wang, Li Shen, Tiehang Duan, Donglin Zhan, Le Fang, and Mingchen Gao. Learning to learn and remember super long multi-domain task sequence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7982–7992, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the \"stability gap phenomenon\" in continual learning (CL) that occurs at the early stage of the CL phase, where the accuracy of previous data drops drastically when updating the model with the new data. Normally, the CL method, like rehearsal, will retrain the model with both the old and new data, where the performance on previous data will recover to a satisfactory level after a long-time retraining phase. Hence, the authors consider this phenomenon as a major obstacle to efficient CL. They propose two hypotheses as the possible reason for this phenomenon and a method to reduce the stability gap. The proposed method is tested on large-scale experiments to illustrate its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a formal mathematical description for the Class Incremental Learning setting (CIL). \n2. In the text above eq(5), $P(x_i;\\theta_i)$ is defined as the output softmax of sample $x_i$ from class $k$, I guess it should be $P(y_i=k|x_i;\\theta_i)$.\n3. Why the softmax probabilities for **each class** $u_k$ is a vector in $R^K$? Isn't it a real value and all the classes form the vector?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FAfV6HbCYo", "reviewer": "Reviewer_tFwu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The paper introduces three insightful metrics: stability gap, plasticity gap, and continual knowledge gap.\n\n+ The experimental results are comprehensive and robustly support the proposed hypotheses.\n\n\n\nWeaknesses:\n\n- The motivation of the paper is unclear. The description makes it difficult to distinguish between the stability gap and catastrophic forgetting, as both result in performance drops when introducing new tasks. For instance, it is not evident whether the method proposed in [1] would be applicable in reducing the stability gap.\n\n- The connection between the methods discussed in Section 4 and the theoretical metrics introduced in Section 3 is weak. Although the continual knowledge gap is formally defined with a mathematical formulation, it is not utilized in any subsequent analysis in Section 4.\n\n[1] Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the issue of model decay in large pre-trained deep neural networks (DNNs) by employing continual learning (CL). Specifically, the authors identify the stability gap that leads to forgetting in past tasks. They propose two hypotheses that explain the causes of the stability gap and subsequently introduce methods to mitigate this gap.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ Discuss the differences between stability gap and forgetting in the literature with more details.\n\n+ Clearly link Sec 4 to Sec 3 and show how the metrics in Sec 3 are used.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J8JD2LwqZh", "reviewer": "Reviewer_5mcy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem the work focuses on in relevant for both industry and academia, paper is well and clearly written, and the experiments are satisfactory in showing the benefits of the proposed approaches. \n\nEach component of the method does not seem to be particularly novel, especially the most impactful LoRA, but the overall method is well motivated, validated and explained. I did not understand what is meant by \"hypothesis-driven\" experiments. Specifically, the paper writes two hypothesis, which are that high losses *cause* the stability gap and that high plasticity *causes* the stability gap. Then, the paper proposes additions to the baseline *inspired* by these hypothesis, and show that they improve over the baseline. However, I do not think this shows that the hypothesis were confirmed, because plasticity and loss can be *correlated* with the stability gap, not causing it. So I think it is very important that the authors either explain what they mean by \"hypothesis-driven\" experiments or remove the nomenclature. \n\nDue to my last concern, I am claiming \"no\" to the \"claims and evidence\" question, even though I am open for clarification from the authors, or a change in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work focuses on continual learning, where the task changes while a model is already deployed, and the model needs to learn the new task while not losing overall performance. The work proposes several components to mitigate the performance drop: dynamic soft targets and weight initialization to attempt at reducing the loss of the model, and OOCF and LoRA to attempt at reducing the plasticity of the model. All the components are shown to effectively reduce the performance drop and computational overhead, and are combined into the SGM method that further achieves these benefits.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not see how the results validate the hypothesis, only that the proposed method (which is indeed inspired by the hypothesis) performs well, so I think there needs to be a clarification to this point, or the text altered. It is very important for me that this point is addressed.\n\nMinor:\n- Why are the values on the x-axis of Figures 1a and 1b different?", "broader_impact_concerns": "I do not have broader impact concerns to attribute to this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an important topic of continual learning and its impact on the model delay occurring in deep learning. The paper identified a new performance metric of the stability gap, based on which the paper proposed novel method to mitigate such a gap and improve the training performance. I think this paper will be a good contribution to the community and hence recommend the acceptance of the paper. One reviewer has a concern on the connection between evaluation metrics and stability gap mitigation approaches. The authors have addressed this in their response. I suggest the authors to add their response to their final version of the paper.
Decision: Accept (poster)